Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
> You just can't compress the entire human knowledge into a 30GB file
Fortunately, that isn’t necessary! What LLMs need is a level of fluency with key concepts so that they can (1) make effective use of retrieval tools and (2) understand the material in the context window. 30GB-sized models can absolutely store enough knowledge to do this.
Here is an example from the field of law. Most lawyers who have litigated contract disputes in California know about Civil Code § 1717, which makes any contract providing for attorney fees to a prevailing party mutual, so that even if the contract was written to be one-sided, it won’t be enforced that way. It’s a simple enough concept, but there are many more particulars to it, such as what happens when the fee provision is only written to apply to part of the contract. (Answer: it depends on other facts.)
When a lawyer recognizes that they’re in a situation where § 1717 is relevant, the first thing they will do is pull the statute and read it, because nobody has it memorized. And they don’t need to.
> (1) make effective use of retrieval tools and (
A downside is that you can't just download a lot of that knowledge, vs with the weights the copyright infringement has been outsourced to the lab. Nor can you just search for the info because the internet as a whole is increasingly aggressive at blocking anything that looks like an AI agent.
I'd love to see more retrieval powered local AI-- I think it's an area that open source development could excel. ... but there are advantages of having the knowledge in the weights!
Perhaps what needs happen is for someone to make an "ultrapedia", an AI restatement of a huge library of reference works-- created expressly for the purpose of being a locally stored corpus for AI agents.
I wonder how much more effective LLMs would be at general knowledge if you just download wikipedia and set up an MCP for it.
> You just can't compress the entire human knowledge into a 30GB file.
...I'm just asking questions here... how sure are we of this?
If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.
If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.
(mumbles) Shannon entropy... Kolmogorov complexity... something, something...
On a more serious note, it depends on your cutoff for "entire human knowledge". It's easy to prove for a generous interpretations of "entire human knowledge" that it can't be done, but hard for something like "all useful human knowledge".
IDK, 30GB is a lot of data when we're talking about text!
Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB.
I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class <insert name of preferred professional knowledge worker> in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).
I think a specialized model could squeeze all you need to know about a certain profession in 30GB. But not all professions at once, which is what these models try to do.
Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.
Yea to be clear I think >70% of the information is not profession specific.
I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.
The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.
My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.
The human genome is 800MB, (and 500MB with compression according to GPT) so that part is easy.
I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".
Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.
Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.
Wikipedia (text) is about 25gigs compressed. I think that's a reasonable starting point.
Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic extensively uses synthetic text in training. You could probably get good 50tile, 90tile, 99 etc numbers just from the size of the training materials on Anthropic servers.
English Wikipedia is around 25GB compressed