RLHF is not RL

andrej-karpathy's distinction: in verifiable domains (math, code — a boxed answer to check) RL can run indefinitely and discover superhuman strategies, because the scoring function can't be gamed. RLHF replaces the human judge with a learned reward model — a billion-parameter simulator of human preference orderings — and "RL is extremely good at discovering a way to game the simulation": run it long enough and the top pelican joke becomes "the the the the" with a 1.0 score. Adversarial examples are infinite, so patching them one by one never wins; you crop after a few hundred updates. "RLHF is RL, but not RL in the magical sense… it lacks magic" — a bounded fine-tune, not a scaling paradigm.

The verifiable side is where reasoning emerged (emergent-reasoning-via-rl: DeepSeek-R1, AlphaGo's move 37); the unverifiable side (jokes, poems, summaries) is stuck with the gameable proxy.

Archive rhyme: a judge that can be gamed by its own optimization target is the same failure shape as quality-gated-data-omission — whenever the scored party influences the scoring, the metric drifts from the goal.

Source: report