InvestAI thinking about LLMs

One way to think about LLMs is as a clever language compression scheme. We humans do this all the time. In fact, the ability to zero in on precisely what most matters about a question is a sign of intelligence. It’s why we make students write essays.

“What is energy?” is a huge question with answers that can range from entire books, libraries, institutions dedicated to the subject. Or you can reply with Einstein’s \(e=mc^2\).

The JPEG format used by digital cameras is one example. Although your iPhone captures photos that contain millions of pixels, only professional photographers bother with the RAW output that keeps every single one of them intact. By reducing the range of colors and smearing together parts of the image (like a blue sky) that look the same, a JPEG image can be a tenth the size of the original and even experts are unable to tell the difference without careful examination.

Similarly, ChatGPT takes around 200 million documents plus your question and “compresses” all of that into a few paragraphs that you think of as “the answer”.

But nobody will claim that JPEG compression is “intelligence”. Why not? Because although we admit that JPEG has greatly reduced the size of the image, it doesn’t understand the image. It took one aspect of the image – the pixels, their colors, and relationships to one another – but those are arguably the least interesting parts of the picture. When I look at a picture, I care about its subject: the people and other objects and how they relate to one another. Sometimes I care more about what’s not in the picture.

There has long been a contest¹ to see how far you can compress Wikipedia. A key constraint is that it must be lossless, i.e. it must preserve every character in its exact original form. So far the winning entry has been able to take a 1GB version down to 112MB.

Footnotes

1. The Hutter Prize, a 500’000€ Prize for Compressing Human Knowledge, by Deep Mind scientist Marcus Hutter. See also his interview with Lex Fridman