Wiki / concepts / wiki

Parameters vs context memory

andrej-karpathy's two-memory-systems model: knowledge in a model's parameters is "vague recollection" โ€” like something you read a month ago, quality proportional to training-data frequency (lossy-compression-of-the-internet) โ€” while the context window is "working memory," directly accessible to attention.

Practical corollaries:

  • Paste, don't recall โ€” summarizing a chapter goes better with the chapter in the prompt than from the model's memory.
  • Tools refresh working memory โ€” web search exists to move facts from the world into context; that's the second hallucination mitigation after trained "I don't know" responses.
  • The mechanism under smart-zone-dumb-zone: working memory is precious and finite, and matt-pocock's clear-don't-compact discipline is about keeping it deterministic. Karpathy names the same open problem from the research side: context windows alone won't scale to long multimodal tasks โ€” some analog of "updating parameters while you sleep" (test-time training) is missing.

Hygiene corollary from How I use LLMs: start a new chat whenever you switch topic. Tokens in the window are "a precious resource": stale ones distract the model and make each next token slightly more expensive. Every tool (search, upload, interpreter) is a way of writing exact text into working memory; see tool-selection-by-freshness.

Agent-memory version (michael-ryan, Stanford CS329Z, Oct 2026): each of the agent-memory-types can be parametric or retrieved into context. Pre-trained knowledge is semantic memory in the weights and instruction following is procedural; episodic memory doesn't really live there. When the two conflict, context usually wins unless something was trained in very deeply. Fine-tuning facts in takes thousands of documents and is lossy, so small sets of facts go in context, while for skills an RL environment can make the model much better (report).

Background: model-parameters covers what the weights are and why a conversation never changes them.

Source: report ยท report

Linked from

Agent memory typesAndrej KarpathyLossy compression of the internetModel parameters (weights)Sleep-time computeSmart zone / dumb zoneTime to first tokenTool selection by freshness