Lossy compression of the internet
andrej-karpathy's core metaphor for a pre-trained LLM: pre-training chops the internet into tokens and compresses it into roughly a terabyte of parameters, "a lossy, probabilistic zip file." You get the gist, not the exact text. Recall scales with how often something appears online, and knowledge stops at the pre-training cutoff (months to a year old).
In How I use LLMs it becomes the model's self-introduction: "Hi, I'm ChatGPT. I am a one-terabyte zip file. My knowledge comes from the internet, which I read in its entirety about six months ago and I only remember vaguely… my winning personality was programmed, by example, by human labelers at OpenAI." Post-training attaches "a smiley face" (the persona) but adds little knowledge.
Practical consequences:
- Common, stable facts (caffeine in an Americano, DayQuil ingredients) are safe to ask directly, but still worth spot-checking.
- A bare model has no calculator, browser, or interpreter. Anything recent, niche or exact needs a tool, which is the logic of tool-selection-by-freshness.
- It pairs with params-vs-context-memory: the zip file is the vague memory, and the context window is the part you control.