Why LLMs Can Reason: Compression Builds the Map, Iteration Does the Thinking

There is a story the industry tells itself about why language models can reason, and it fits on a sticker: scale is all you need. Make the model bigger, feed it more of the internet, and intelligence condenses out like dew. It is a convenient story — it justifies the capex — and I no longer believe it is the right one. At least not on its own.

After years of putting these systems into production — hospitals, agents, document pipelines — and two decades watching machine learning grow up, here is the claim I actually defend when someone asks me why a next-word predictor can plan a database migration:

Reasoning in LLMs does not arise from parameter count alone. It arises because repeated learned transformations through transformer layers build increasingly abstract, compressed, semantic representations — of the text stream and, through it, partially of the world. By privileging relations and structure over surface form, training carves out a latent space in which analogy, generalization and a degree of planning become possible — and ordinary generation is better described as a reconstruction from that representation than as retrieval of stored text.

Scale is the enabler; compression is the mechanism — that is the synthesis this essay defends. The distinction sounds academic until you notice it predicts most of what has happened in this field over the past four years — including the parts that surprised the scale maximalists. This is the long version: where the idea comes from, what the evidence looks like in mid-2026, where it is genuinely weak, and what it says about what comes next.

Three clarifications before we start, so you know what kind of text you are reading. First: no single experiment establishes the full account presented here. The individual pieces are better grounded than the synthesis — predictive training corresponds to compression (a theorem); finite capacity creates pressure toward shared structure (observed in controlled capacity experiments); internal representations contain abstract, causally usable variables (observed, through method-dependent instruments); and additional serial computation improves many reasoning tasks (supported by formal expressivity results, controlled experiments and deployed systems). The thesis of this essay is that these findings are best understood as parts of one mechanism. That last step is my synthesis, and I will mark it as such along the way. Second, a working definition, since “reasoning” gets used for everything: by reasoning I mean the use of internal representations to carry out input-dependent transformations whose result is not directly retrieved as a memorized continuation — especially where the task requires combining relations across multiple dependent steps. Third, because “compression” will carry just as much weight: throughout this essay it means the construction of progressively more abstract internal representations that preserve predictive structure while discarding irrelevant detail — not merely shrinking a file or a parameter count. (A later section separates the three technical senses hiding under that one word.)

Act I: when humans built the representations

Machine learning before 2012 was mostly humans doing the understanding and machines doing the arithmetic. You sat down with domain experts and hand-crafted features — edge detectors for images, TF-IDF weights for text, cepstral coefficients for audio — and then a comparatively simple classifier drew a boundary in the space you had designed. The intelligence lived in the representation, and the representation was authored by people.

Then AlexNet won the 2012 ImageNet competition with a top-5 error of 15.3% against 26.2% for the best hand-engineered pipeline, and the field flipped. The lesson usually drawn from that moment is Rich Sutton’s bitter lesson: “general methods that leverage computation are ultimately the most effective, and by a large margin.” True. But there is a more precise reading that the sticker version drops: what the computation bought, concretely, was learned representations. Every leap since has been a change in how machines re-describe their input, not merely a change in how much hardware we throw at them.

The first crack: meaning has geometry

In 2013, word2vec made famous what Mikolov’s group had first observed with recurrent-network vectors that same year: train a small network on nothing but word co-occurrence, and the resulting vectors organize themselves so that vector(“King”) − vector(“Man”) + vector(“Woman”) lands near vector(“Queen”). Nobody programmed that. Semantic relations — gender, tense, capital-of — fell out as directions in a vector space, purely from co-occurrence statistics. J.R. Firth’s 1957 dictum — “you shall know a word by the company it keeps” — turned out to be an engineering recipe.

Honesty requires a footnote the popular telling omits: Linzen (2016) showed that without the standard trick of excluding the input words from the search, the raw arithmetic returns one of the inputs in ~98% of cases. The geometry is real but noisy — a preview of a pattern we will see again and again: the structure exists, and it is less clean than the demo suggests.

Static word vectors had an obvious ceiling — one vector per word, no context. The decade that followed was a search for machinery that could build contextual representations: seq2seq (2014) squeezed whole sentences through a single vector; attention (Bahdanau, 2014) let the decoder look back at everything; and in June 2017, “Attention Is All You Need” distilled the idea into the Transformer — “dispensing with recurrence and convolutions entirely.” The base model had 65 million parameters — 213 million for the “big” variant that set the headline scores. Everything since GPT-2 is, in its core block, that machine scaled up.

The transformer is an engine of repeated re-description

Strip away the mystique and a decoder-only transformer does one thing, over and over. Tokens become vectors. Each layer reads the running state — what Anthropic’s interpretability team later named the residual stream, “a communication channel” that every component reads from and writes to — then attention exchanges information across positions, an MLP transforms features locally, and the update is added back to the stream. Each pass is a learned transformation: linear maps (the attention paper itself calls its query/key/value maps “projections”), softmax, nonlinearities, normalization — a re-encoding of the state into a form more useful for prediction — not a tidy change of basis; it is free to discard, mix and overwrite. GPT-3 did it 96 times per token. This is the “repeated transformation” of my thesis, and it is concrete: the model applies learned operations to its own internal state, again and again.

What do those iterated re-descriptions build? The measurement tools are imperfect — probing classifiers can flatter the prober, and the logit lens is a biased instrument that its successor, the tuned lens, had to correct — but several independent lines now agree on the shape:

• The input is rapidly re-encoded. Viewed through the model’s own output head, intermediate states “almost never look like the input” — tokens are converted at once into a different representation which is then smoothly refined toward a prediction. Careful caveat: not looking like the input is not the same as the input being gone — in a residual architecture, information about token identity can remain recoverable deep into the network (the author’s own addendum to that post found input tokens still ranked in the hundreds — not the ten-thousands — deep into the network, most strongly for rare tokens). What changes is the working format: iterative inference over a re-encoded state, not storage-and-lookup.
• The middle is the most abstract place in the network. Anthropic’s early description — early layers “de-tokenize,” middle layers deal in abstract concepts, late layers “re-tokenize” — was labelled preliminary by its own authors, but it keeps being rediscovered with better instruments. Wendler et al. (2024) traced Llama-2 processing non-English text through an abstract middle-layer “concept space” that leans toward English before re-materializing in the target language. Skean et al. (ICML 2025) found intermediate layers beat the final layer by up to 16% as embeddings across 32 tasks — the most generally useful summary of the input lives mid-network — and the mid-network compression bottleneck behind it is a byproduct of autoregressive training rather than of the data modality. (That bottleneck, at least, is measured directly, as an entropy and effective-rank dip in the activations themselves — not merely inferred from task utility.) (Useful is not the same as measurably more compressed — that reading is my interpretation through the compression lens, and embedding quality alone does not prove it.)
• The exit behaves like a reconstruction stage. Not decompression in the codec sense — nothing inverts an earlier encoding — but a conversion from distributed features into one concrete next-token distribution. Lad et al. hypothesise the last stretch of the network as prediction ensembling and residual sharpening.

Funnel in, funnel out — my compression-lens reading of these results: surface shed on the way up, structure used in the middle, surface re-synthesized on the way down. The claim “early = syntax, middle = semantics” in its strong pipeline form is contested (replications have pushed back on the neat staircase); the broader pattern — rapid re-encoding, abstract intermediate representation, late prediction shaping — has substantially wider support.

Prediction is compression — literally, in one sense

Why would next-token prediction produce structure at all? Because the training objective is a compression objective. This is a theorem, not a slogan: any probability model can be turned into a lossless compressor via arithmetic coding, and DeepMind’s “Language Modeling Is Compression” (ICLR 2024) states it plainly — “current language model training protocols use a maximum-compression objective” (meaning: minimizing log-loss minimizes expected code length under the model’s own distribution — not that any given model achieves maximal compression in an absolute sense). Their measurements make it concrete: Chinchilla 70B, trained almost entirely on text, compresses Wikipedia to 8.3% of its size (the strongest classical compressor they test manages 23.0%; purpose-built context-mixing compressors reach ~11% on the Hutter Prize’s identical file) — and, with no image or audio training to speak of, compresses ImageNet patches to 48.0% versus PNG’s 58.5% whole-file (61.7% chunk-matched) and LibriSpeech audio to 21.0% versus FLAC’s 30.3% (figures from the paper’s results table; its abstract and introduction quote slightly lower numbers that appear nowhere in that table). A text model beats purpose-built codecs on modalities it barely saw — plausibly because the statistical structure it extracted transfers, though a duller reading also fits: patches and waveforms are locally redundant, and a good sequence model exploits redundancy wherever it finds it. My reading is the first; the measurement alone does not decide.

(The honest caveat, from the same paper: count the 140 GB of weights as part of the compressed artifact and the story only works on corpora of terabyte scale — so the headline rate is not evidence of parsimony at the 1 GB scale measured. Compression-as-intelligence is a statement about amortizing one model across enormous data — which is what pretraining is.)

One distinction matters enough to spell out, because the word “compression” is doing three different jobs in this essay. Predictive compression is about the code length a model assigns to data — that one is mathematically exact. Parameter compression is about how the regularities of a huge corpus get amortized into finite weights — strongly supported by the capacity measurements below. Representational compression is the claim that internal states preserve semantic structure while discarding surface detail — and that is the interpretive step this essay argues for, on the interpretability evidence in the next section. The first two do not automatically prove the third; keeping them separate is what keeps the argument honest.

None of this is new as an idea. Solomonoff, Kolmogorov and the minimum-description-length tradition formalized, from the 1960s on, the tight relationship between short descriptions, explanatory structure and prediction: models that capture reusable regularities tend both to compress past observations and to predict future ones well. Marcus Hutter took it seriously enough to fund the Hutter Prize — launched in 2006 at 50,000€ for compressing 100 MB of Wikipedia, scaled tenfold in 2020 to 500,000€ for 1 GB — on the explicit rationale that “being able to compress well is closely related to intelligence.” And Ilya Sutskever spent his 2023 Berkeley talk (recording) arguing that compression is the missing theory of unsupervised learning: a good enough compressor of two concatenated datasets must exploit the structure they share — which is why predicting internet text teaches you something transferable about the world that generated it.

Now add the decisive constraint: the models are far too small to store their training data verbatim. Careful measurement (Morris et al., 2025) estimates the memorization capacity of GPT-style transformers at roughly 3.6 bits per parameter — measured by eliminating generalization on synthetic data, with text-trained models from 500K to 1.5B parameters validating the resulting scaling laws, all under their specific experimental definition. Extrapolate that to a 70B model (an extrapolation, not a physical law) and you get on the order of 30 GB of effective memorization capacity — against training corpora in the tens of terabytes. Even granting generous error bars, verbatim storage of the training set is off by two to three orders of magnitude; what the model can still do is memorize a substantial, highly non-uniform subset, concentrated on duplicated text. The same paper watched what happens at the boundary: models memorize until capacity fills, and then generalization takes over — “once the model can no longer memorize datapoints individually, it is forced to share information between datapoints to save capacity, which leads to generalization.” Compression — in the parameter sense — is not a design choice. It is forced.

You can watch the same endpoint reached in miniature — under a different pressure, weight decay rather than capacity saturation — in the grokking experiments: train a tiny transformer on modular arithmetic and it first memorizes (training accuracy perfect within a thousand steps, validation at chance), then — around a thousand times more optimization later — validation snaps to 100%. Nanda et al. reverse-engineered what happened: the network had built a genuine algorithm (a discrete Fourier transform turning modular addition into rotation), and the famous sudden jump is just weight decay deleting the lookup table after the circuit already exists. Ablate the circuit’s five key frequencies and the model collapses to chance; ablate the other 95% of frequencies and it slightly improves. Lookup table out, reusable rule in. It is not a complete model of pretraining — how universal grokking is outside algorithmic toy tasks is still debated — but it is the cleanest miniature demonstration we have of optimization pressure replacing memorization with a compact algorithm.

This is why “reconstruction, not retrieval” is the right frame for generation — with one honest asterisk. Verbatim memorization exists: it concentrates on hyper-duplicated text (one 2025 study extracted ~91% of Harry Potter and the Sorcerer’s Stone from Llama 3.1 70B at a permissive extraction-probability threshold, while most books showed almost none on the same measure — the paper’s headline finding is that most models do not memorize most books, in whole or in part), and extraction attacks are real. Ordinary generation is therefore better understood as conditional reconstruction from learned structure than as verbatim retrieval — granting that the boundary between memorization, reconstruction and pattern generalization is not directly measurable for every output — and the failure modes are diagnostic. A pure retrieval system typically fails by missing or misranking its evidence. A generative reconstruction system has an additional failure mode: it can manufacture a fluent, plausible completion even when the necessary evidence was never stored. This framework naturally explains an important class of hallucinations — the plausible fill — while other causes (prompting, sampling, reward-model incentives, distribution shift, calibration) contribute their own. OpenAI’s own analysis derives that class from the statistics: facts seen once in training (“singletons”) cannot be reliably reconstructed, so the model synthesizes something distributionally reasonable instead — and our accuracy-only benchmarks then reward the confident guess over the honest “I don’t know.” Andrej Karpathy said it best back in 2023: “in some sense, hallucination is all LLMs do. They are dream machines. … It looks like a bug, but it’s just the LLM doing what it always does.”

We opened the hood — what is actually in there

Until a few years ago, all of the above was inference from behavior. The reason I hold the thesis with some confidence in 2026 is that interpretability grew up and started looking directly. One honesty note before the tour: direct measurements of representational compression — activations measurably discarding surface detail — remain thinner than the evidence for abstraction. The clearest instances are the mid-network entropy dip noted above and the grokking circuit of the previous section, where the compression is literal and countable. Most of what follows establishes abstraction and causal structure; I read it through the compression lens, and say so where I do.

The model represents more features than it has dedicated dimensions. Anthropic’s superposition work framed — and demonstrated in toy networks — the hypothesis that networks “want to represent more features than they have neurons”, packing them into overlapping directions — “a small neural network may be able to noisily simulate a sparse larger model.” When sparse autoencoders were pointed at a production model (Claude 3 Sonnet, 2024), out came millions of features — among them one for the Golden Gate Bridge that fired on its Wikipedia opening in six languages and on images of the bridge. Clamp it to 10× its maximum and the model starts self-identifying as the bridge. Two calibrations: an SAE is one useful decomposition of the superposed state, not a guaranteed canonical inventory of “what the model knows” — but the steering results show that at least some of these directions are behaviorally load-bearing, not mere probe artifacts.

Structured, usable state — the “world model” evidence, carefully stated. A GPT trained only on Othello move sequences develops an internal board state — linearly readable once you know the right frame (“mine/yours”, not “black/white”, 99.5% probe accuracy), and editable: flip a square in the latent board, and the model’s legal-move predictions follow the edited position (0.10 average errors, versus 2.7 with no intervention). The same holds for chess models (99.6% square accuracy — plus a coarse readable skill signal: 90.5% at separating weak from strong players, versus 69% on a random-init model). Gurnee & Tegmark found literal linear maps of the world in Llama-2’s activations (linear-probe R² of 0.91 for world-scale place coordinates in the 70B model; finer-grained maps like New York City decode far worse). And the effects are causal at the concept level too: MIT’s semantic hub work showed that intervening in English changes model behavior on other surfaces — arithmetic (over 90% of instances steered as intended at the strongest intervention setting), Python code (67%), even images (over 80%) — weakest for cross-lingual sentiment on Chinese, where the shift sits near seed noise. Shared causal directions underneath, which I read as one semantic space — noting that this sits not-yet-reconciled beside Wendler’s English-leaning concept space.

Planning and multi-step structure. Anthropic’s attribution-graph work (2025) traced individual computations through Claude 3.5 Haiku and found, among other things: the model activates candidate rhyme features — “rabbit” and “habit” — on the newline token before a poem’s second line even begins, holding alternatives open (the paper’s actual signature of planning); suppress “rabbit” and it re-plans around “habit”; asked for the capital of the state containing Dallas, the attribution method identifies a causally relevant intermediate “Texas” representation on the way to “Austin” — swap that latent Texas for California and out comes Sacramento (method-dependent evidence, but causal, not correlational); and concepts are represented in shared circuits across languages, with the shared fraction markedly higher in the larger of the two models compared. Independent probing work (Physics of Language Models) found the same thing on math problems — GPT-2-scale models on synthetic grade-school corpora: before generating a single token, the model has already computed internally which quantities the solution will need. And in 2026, Anthropic’s natural-language autoencoders sharpened the picture further — through an instrument its authors call a blackbox by construction, to be read for themes rather than individual claims: models carry load-bearing content they never verbalize, such as beliefs about being evaluated and reasoning about how they will be graded, none of which reaches the visible text. The chain of thought you read is not the whole of the reasoning; the visible text is a partial rendering of what happens in the representation. (The attribution-graph work had already shown the uncomfortable corollary: ask the model how it added two numbers and it recites the schoolbook carry algorithm — while the traced mechanism is a parallel approximate-plus-exact circuit it has no introspective access to. And a scope note: lookahead within a poem’s line is not a database-migration plan — the distance between the two is the reliability gap I return to below.)

And the honest counterweight. Decodable does not mean coherent. Vafa et al. trained transformers on turn-by-turn routes between real Manhattan taxi origin–destination pairs; the models passed both standard diagnostics (~100% valid next-token predictions, >90% state-probe recovery) while finding true shortest paths 97% of the time — and the reconstructed street maps were still a mess of physically impossible flyovers, with the shortest-path-trained models’ routing collapsing from 99% valid to 8% under a 10% rate of forced detours. The same metrics applied to actual LLMs land the same way: GPT-4 solves their logic puzzles while scoring 0.21 on state-equivalence. Their 2025 follow-up is even sharper: a model that predicts orbital trajectories with R² > 0.9999 has not thereby learned Newton’s law — such models behave, in the authors’ words, “as if they develop task-specific heuristics that fail to generalize” — and frontier reasoning models given the same in-context physics recover nonsense force laws too. Even Othello-GPT’s tidy board state, an interim interpretability report finds, looks to be assembled from many local decision rules — the authors’ best guess is a bag of heuristics rather than one crisp algorithm. So the defensible claim — and the one I actually make — is this: LLMs build causally load-bearing internal variables that carry real task-relevant structure. What is not established is that those variables respect the full equivalence structure of the domain, or compose into a clean, transferable simulator of the world. It is a compressed statistical model — genuinely semantic, genuinely usable, and genuinely brittle off-distribution.

So what does size actually buy?

Here is where the research of the last two years has, in my reading, made the most progress — by splitting the question.

Knowledge storage tracks parameter count. Allen-Zhu & Li’s capacity measurements (on controlled synthetic data) land on ~2 bits of stored knowledge per parameter as a near-universal constant in the well-trained regime — independent of depth, width, even the presence of MLP layers. Parameters provide the storage medium — and the machinery that operates on it.

Reasoning depends strongly on serial computation depth — iteration — not on parameter count alone. The cleanest way to put the split: parameter count expands the repertoire and fidelity of representations and computations the model can implement; serial depth determines how many dependent transformations it can execute on a given input. The evidence for this split stacks from theory to practice:

• Theory. Fixed-depth transformers are provably confined to shallow circuit classes (TC⁰-style bounds, under precision assumptions) — classes that exclude genuinely serial problems under standard, widely believed, but unproven complexity conjectures. Chain of thought escapes the limit precisely because each emitted token writes intermediate results back into the input, converting output length into computation depth: Merrill & Sabharwal showed polynomially many CoT steps lift transformers to full polynomial-time power (under mild architectural assumptions), and Li et al. that T steps simulate any circuit of size T. Merrill & Sabharwal’s 2025 result is the cleanest statement: growing depth logarithmically solves problem families that no amount of extra width — and no logarithmic amount of CoT — can touch. (These are expressivity results under formal assumptions about precision and architecture, with the separations conditional on standard complexity conjectures — not performance laws for production models. Their value is isolating why serial depth can matter in principle.)
• Controlled experiments. In the Physics of Language Models math suite, a 20-layer, 576-dim model outperforms a 4-layer, 1920-dim model with twice the parameters — and layer-by-layer probes show why: deeper layers resolve dependencies further from the question. Depth acts as a serial computation budget for resolving dependent parts of the plan.
• Same weights, more iterations. The recurrent-depth model of Geiping et al. (2025): 3.5B parameters, trained explicitly for weight-shared recurrence, loops its middle block at inference. At 1 iteration it scores ~0 on GSM8K math (CoT scoring); at 32 iterations, ~35% — same parameters, different amount of thinking. (You cannot do this to an ordinary transformer; the iteration budget has to be trained in — and a model trained to loop 32 times is off-distribution at one iteration, so read the curve as the shape of iteration’s value, not a clean per-step price.) Its ancestor, the Universal Transformer (2019), cut bAbI reasoning error from 15.2 to 0.21 against a same-size vanilla transformer (with its adaptive per-position halting; 0.23 without). Meta’s COCONUT takes it further — feeding the hidden state back as input so the model reasons in latent space without emitting tokens at all, beating token-CoT on search-heavy logic tasks with 3–10× fewer tokens (while, honestly, still losing to CoT on grade-school math).
• Tiny recursive models. The Tiny Recursive Model — 7M parameters, a two-layer network iterating on its own answer — hits ~45% on ARC-AGI-1 public eval, above DeepSeek R1 (671B) and o3-mini on those tasks. Necessary honesty: these are per-task-trained, transductive puzzle solvers, not general reasoners, and the ARC Prize’s own forensic analysis of the predecessor HRM found the celebrated brain-inspired hierarchy contributed almost nothing — a vanilla transformer came within ~5 points — while the under-documented outer refinement loop drove much of the performance, alongside per-task training and heavy augmentation. Which refutes the architecture mysticism cleanly — and supports, less cleanly, the mechanism I care about: iteration. Even the TRM author concedes “why recursion helps so much compared to using a larger and deeper network remains to be explained.” My favorite datapoint in this family: CompressARC (3rd-place Paper Award, ARC Prize 2025) solves 20% of the ARC-AGI-1 public evaluation set with ~76K parameters and no pretraining at all — by directly minimizing the description length of each puzzle, at inference time (a fourth sense of compression, and a revealing one: description length minimized per instance rather than amortized across a corpus). Compression as reasoning, with the parameter count of a 1990s model — though note this case bends my own slogan: here compression is not the map, it is the search.
• Reasoning procedures are compressible. DeepSeek’s R1 distillation re-encodes the teacher’s reasoning patterns into the fresh parameters of a student 400× smaller: the 1.5B model beats GPT-4o and Claude 3.5 Sonnet on competition math (28.9 vs 9.3/16.0 on AIME’24 — a math-specific win; it falls well behind on science knowledge and code) — and, in the controlled comparison, distilling the procedure into a 32B base far outperforms running RL on that same base (72.6 vs 47.0 on AIME). This weakens the idea that raw parameter count is the direct mechanism of reasoning — the procedure fits in a tiny fraction of the mass that discovered it — without making capacity irrelevant: the student rides on the teacher’s expensive discovery, and keeps it within a narrower envelope. And what this does not show is that mass is dispensable for discovering the procedure — only for running it. The split may be acquisition-versus-execution rather than storage-versus-computation; the evidence here cannot tell those apart.

Where does that leave emergence — the observation that abilities appear “suddenly” at scale? Genuinely contested: the famous emergent-abilities curves partly dissolve into metric artifacts (of 39 BIG-Bench metrics, at most 5 show emergence, and two discontinuous metrics account for over 92% of claims), yet later work found real thresholds when you index by pretraining loss instead of parameter count. Both can be true — my synthesis, and untested as far as I know: capability grows smoothly in the representation; task success snaps discontinuously when the last link in a serial chain becomes reliable. Note that loss-indexed thresholds, if anything, cut toward the scale-centric reading.

So: scale buys capacity — room for features that would otherwise interfere in superposition, coverage of the long tail of knowledge, more reliable individual steps, a lower loss floor. Those are real, and I am not giving them back. But the mechanism that turns stored structure into solved problems is iterated transformation in representation space: architectural depth in the base model, and chains of thought or recurrence at inference time. Parameters provide the substrate; iterated transformation performs the computation — that is the split as I read the evidence.

The counter-evidence worth taking seriously

A thesis you cannot falsify is a slogan, so let me state the strongest attacks and what they actually cut.

• The Reversal Curse. Train a model that “A is B” and it does not learn “B is A” — Berglund et al. measured reversed-direction recall at effectively zero (0.0% and 0.1% across the two directions), with the correct answer’s log-probability indistinguishable from a random name’s. And a 2026 workshop study shows that bidirectional training fixes the behavior while still not producing a direction-agnostic representation — the two directions remain stored as distinct entries. This kills the strong version of the thesis — a clean relational knowledge graph would be symmetric. What survives is important: the latent space is built along the arrow of prediction. It is a compressed model of the text stream, not of the world directly — asymmetries of the corpus become asymmetries of the mind.
• Embers of autoregression. McCoy et al.: GPT-4 deciphers shift-1 ciphers at 82% and shift-3 at 76% — but shift-2 at 2%. Identical algorithm; shifts 1, 3 and 13 are simply the three most common shift levels in internet text — and, asked to answer directly, the only three GPT-4 can do. (Chain-of-thought prompting lifts the others substantially; the frequency-shaped ranking survives it.) Competence tracks distribution, not task structure — exactly what a statistical compression of text would have predicted, had anyone thought to ask before McCoy did, and exactly what a symbolic reasoner forbids.
• Fragility under irrelevant detail. Apple’s GSM-Symbolic: renaming entities cost a few points, changing the numbers cost more — and adding one seemingly relevant but inconsequential clause cratered small models’ scores by up to 65 points. Notably, the stronger of the two reasoning models tested (o1-preview) lost only 17.5 — a drop the authors still call significant — and stayed flat as problems grew harder; o1-mini did not match that (−29.1). Their 2025 sequel, “The Illusion of Thinking”, found reasoning models collapse past a complexity threshold on puzzles like Tower of Hanoi — even when handed the algorithm. The rebuttal war that followed (including a comment paper whose first version listed “C. Opus (Anthropic)” as first author — removed under arXiv policy, the human author crediting Claude with “the bulk of the writing” — and whose reference list cites the Apple paper under DeepSeek-R1’s arXiv ID; the field has a sense of humor) established that some of the collapse was evaluation artifact — river-crossing instances that were mathematically unsolvable and, more arguably, token budgets. The residue after both sides: execution of long serial procedures inside the latent space is a real, current weakness. Iteration helps; it does not yet make the model a reliable computer. And note the awkwardness for my own account: I defined reasoning by the multi-step case, and the multi-step case is where iteration remains least reliable. What I can defend is that iteration is the resource that moves these tasks at all; whether the remaining gap is immaturity or a missing mechanism is genuinely open.
• The chain of thought is not faithful. Anthropic’s own measurements: models that exploited reward hacks verbalized it in under 2% of cases in most test environments; models nudged by planted hints acknowledged them 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time. This is fatal for “read the reasoning to trust the reasoning” — but note what it implies for the thesis: the real computation is in the latent space, and the verbal trace is a rationalization layer. That makes representation-centric interpretability more urgent — a separate question from whether my thesis is true. And two things are true at once here, worth keeping apart: the emitted tokens are causally load-bearing — they are the scratchpad that buys serial depth — while being an unreliable report of the latent computation that produced them. Unfaithfulness is a failure of the report, not of the mechanism.
• Stochastic parrots. The 2021 position — a language model is “a system for haphazardly stitching together sequences of linguistic forms … without any reference to meaning” — was a defensible reading of the evidence available in 2021. Against interventions on latent board states, steerable cross-lingual features and causally verified plans, it no longer is. What the critique got permanently right is the training-distribution dependence — the “embers” above.
• The deflationary reading. The strongest live objection is not that anything above is false but that “compression” adds nothing: capabilities come from scale, data quality and RL against verifiers, and the representational story is bookkeeping with no extra predictive content. I cannot refute that with the evidence in this essay — the evidence is compatible with it. What would separate the two is measurement, which is what the dated predictions at the end attempt. Until they resolve, the honest status of this synthesis is a framework that organises the evidence, not a mechanism that has beaten its rivals.

My synthesis: every one of these failures is a failure characteristic of reconstruction from compressed, distribution-shaped representations. Directionality, frequency sensitivity, confabulated detail, plausible-but-wrong fills — human reconstructive memory exhibits a strikingly similar family of errors — suggesting that lossy, reconstruction-based memory systems may share characteristic failure modes, without implying that brains and transformers share an architecture. But a caution about what this buys: every failure above is equally well predicted by the deflationary reading — next-token statistics over text, no commitment to semantic structure at all. Consistency with my thesis is therefore not confirmation of it. What the counter-evidence genuinely rules out is the crisp inner symbolic engine; between my account and the deflationary one, this section cannot decide. That is what the predictions at the end are for.

Where we stand, July 2026

The last two years are, at minimum, strikingly consistent with this account — one product launch at a time.

The reasoning-model era — o1 (September 2024), DeepSeek R1’s open-weights counterpart (January 2025), extended thinking in Claude, GPT-5’s router, Gemini’s Deep Think — is, in mechanism terms, the monetization of test-time computation: additional generated steps, sampled candidates, search or verification let the system perform more serial computation before committing to an answer. (The mechanisms differ — ordinary chain of thought does not refine one hidden state; it emits tokens that feed the next forward pass — but the resource being purchased is the same: dependent computation steps.) It turned out you could induce the behavior with a thousand training examples and steer it by literally appending “Wait” to force another round of reflection. Accuracy improves smoothly with thinking tokens (the published curves are plotted against log-scaled compute) — until it doesn’t: by early 2025 an “overthinking” literature had formed, documenting models talking themselves out of correct answers. Iteration is a dial, not a free lunch.

The benchmarks tell the same story with dates on it — provided you compare like with like, because evaluation split and reasoning-effort setting move these numbers as much as model identity does. On ARC-AGI-2’s Semi-Private evaluation (the split behind ARC Prize’s main leaderboard column), the frontier moved from Claude Opus 4.5 at 37.6% (64K thinking budget) and Gemini 3 Pro at 31.1% in November 2025 to GPT-5.6 Sol at 92.5% and Claude Opus 5 at 90.4% as of late July 2026, each at Max reasoning effort — the same Opus 5 scores 88.3% at High effort on the same split, which is why unlabeled comparisons mislead. ARC-AGI-3, the interactive benchmark launched on March 25, 2026 with the line “Humans score 100%. Frontier AI scores 0.51%,” is where the gap stays visible — and where the ranking reshuffles: Claude Opus 5 (High) leads at 30.2%, while GPT-5.6 Sol (Max), the ARC-AGI-2 leader, reaches 7.8% and Grok 4.5 sits below 0.4%. Saturating a static benchmark evidently does not transfer to an interactive one. And worth stating plainly what this costs my own account: whatever compression-plus-iteration buys, at the frontier it currently delivers under a third of human performance on an interactive benchmark. Necessary, on this evidence; nowhere near sufficient. And the ARC Prize team’s diagnosis (Chollet and colleagues) of what these systems now automate is precise: domains where “sufficient task knowledge coverage exists in the pretraining corpus” and “the task provides a verifiable feedback signal.” Knowledge coverage from compression; capability from iterating against a verifier — my translation, not his. His remaining complaint — “we still need … methods to separate knowledge and reasoning” — reads, to my eye, as a restatement of the capacity-versus-computation split.

Meanwhile the economics confirm that capability is not glued to mass. METR’s task-horizon metric — the length of task, in expert-human time, that a model completes at 50% reliability — doubled roughly every seven months across 2019–2025 in METR’s original study, and faster since 2023 — about four months on their current revised suite. Their latest published estimate (May 2026) puts Claude Mythos Preview (early) at a 50% horizon around seventeen hours — a point estimate past the threshold where METR itself cautions that “measurements above 16 hrs are unreliable with our current task suite,” with a confidence interval spanning roughly 8 to 55 hours and the point excluded from METR’s own trend fit — while the same model’s 80%-reliability horizon is about three hours. That seventeen-versus-three spread is the production reliability gap expressed in a single pair of numbers. The inference cost of hitting a fixed benchmark level has been falling by roughly an order of magnitude per year (measured 2021–2024 at fixed MMLU thresholds); the open-weights frontier (GLM-5.2 at 753B total but 40B active parameters, Kimi K3 at 2.8T/104B) runs on mixture-of-experts sparsity — a related economy at the architectural level: maintain enormous total capacity, activate a small subset per token. And the gap that actually hurts in production is no longer intelligence but reliability under compounding steps — a reconstruction system’s error rate, compounded.

What comes next

If the mechanism is compression plus iteration, the roadmap almost writes itself — and the field’s most expensive bets in 2025–26 line up with it.

• Reasoning moves into the latent space. Emitting English tokens to think is paying tax to the tokenizer. COCONUT-style continuous thoughts, recurrent-depth models, and production diffusion LLMs (Mercury arrived in 2025; Mercury 2, the first reasoning diffusion model, in February 2026 — refine a sketch of the whole answer in parallel) all point the same direction: buy iterations in representation space directly. I expect frontier models to think less and less in words.
• World models and grounding. Yann LeCun left Meta after twelve years to found AMI Labs, which closed a $1.03B seed in March 2026 — the bet being JEPA: predict in representation space, not pixel or token space. DeepMind’s Genie 3 generates playable, physically consistent worlds — for minutes at a time, today. Strip the marketing and they are half rebuke, half agreement with this essay: LeCun’s premise is that token prediction alone will not deliver a robust world model — the Vafa results above say he has a point — while the shared conclusion is the one I have been arguing: prediction and iteration belong in representation space.
• Experience and memory. Sutskever’s NeurIPS line — “pre-training as we know it will unquestionably end,” with data as what he called the “fossil fuel” of AI — and Silver & Sutton’s “era of experience” converge on the same gap: models compress a fixed fossil corpus and then stop learning. Today’s patch is scaffolding — persistent file-based memory (a headline feature of Anthropic’s Claude Fable 5 launch) — but the research frontier (test-time training, sparse memory finetuning) is about writing experience back into weights without destroying the compression already there. Continual learning is, to my eyes, the largest remaining architectural gap.
• Small, recursive and specialized. TRM, CompressARC and the distillation results say narrow reasoning does not need frontier mass. Expect a widening split between huge sparse generalists and tiny iterative specialists running on-prem — with distillation as the bridge that moves procedures downmarket a year after they appear at the frontier.
• The age of research. Sutskever again, November 2025: another 100× of scale would change things, he grants — but he does not believe it would transform them. “It’s back to the age of research again, just with big computers.” His stated puzzle — models “generalize dramatically worse than people” from equivalent exposure — is a compression-efficiency question. Humans build deeper abstractions from radically less data; nobody yet knows what loss function gets machines there. That, not another order of magnitude of parameters, is the open problem.

What this means if you build things

• Stop equating capability with parameter count. Route by task, re-evaluate quarterly — the mid-tier keeps absorbing last quarter’s frontier (Opus 5 made that explicit).
• Treat thinking budgets as an engineering dial. Roughly log-linear gains, then an overthinking tail where models talk themselves out of correct answers. Measure where your tasks saturate instead of defaulting to max — and remember the dial has several settings: longer chains, parallel samples with a vote, or a verifier pass are different trades of latency for reliability.
• Expect reconstruction, and design for it. Verbatim fidelity needs retrieval bolted on (RAG, tools); confident confabulation needs verifiers. You would not trust a brilliant colleague’s memory for a contract clause; don’t trust a compression artifact’s.
• Score abstention in your evals, or you are training for confabulation. The hallucination-incentives result applies to your internal benchmarks too: accuracy-only scoring rewards the model (and the team) that guesses. The comparison in OpenAI’s accompanying post makes it vivid — a model that abstains 52% of the time posts a 26% error rate, while an older model that abstains just 1% of the time posts 75%, for two points more accuracy. Give “I don’t know” partial credit and penalize wrong answers harder than silence.
• Reliability is compounding arithmetic. 95% per step is 36% over twenty steps (assuming independent steps and no retries — real agent loops violate both, in both directions). The fix is checkpoints, verification and narrow scopes, not a bigger model.
• Distillation is a deployment strategy, not just a research trick. Reasoning procedures discovered at the frontier compress into models a fraction of the size within a year — which is how narrow, on-prem, data-residency-friendly reasoning becomes viable. But the wins are domain-shaped (R1’s 1.5B distill beats GPT-4o on competition math while falling well behind on science knowledge and code), so validate on your distribution, not the leaderboard’s.
• Watch interpretability become tooling. Attribution graphs are open source; steering and feature-level control are crossing from papers into ops. “MRI for AI” (Amodei’s phrase) is, on his own bet, a 5–10 year track — a bet he pairs with the worry that AI may move faster than that — and worth tracking, because it is the audit layer for everything above.

What would prove this wrong

A synthesis that cannot fail is a slogan, so let me end the argument the way arguments should end: with dated predictions. If the account in this essay is right, the following should hold. Each names what its failure costs — and one stake binds them all: if three or more come out the wrong way, the synthesis is not a mechanism, it is a vocabulary, and I will say so here.

• Depth keeps beating width at matched budgets (check by end-2027). On multi-step reasoning suites, deeper models should keep outperforming wider-but-shallower twins at matched parameter count and matched total inference compute, within the aspect-ratio band where both train stably. If equalizing inference compute closes the gap, the iteration half of this thesis fails.
• Latent reasoning reaches the frontier (by end-2028). A production frontier model whose primary reasoning mechanism is latent iteration rather than token emission should match the best token-CoT systems on interactive benchmarks like ARC-AGI-3 at materially lower cost. If frontier reasoning is still overwhelmingly token-mediated by then, “the reasoning happens in the representation” is in trouble.
• Distillation stays asymmetric — on a balanced diet (testable today). Distill on a corpus with equal shares of rare-entity factual QA and reasoning traces: students should retain procedures disproportionately better than facts. If retention is symmetric once the distillation mix is balanced, the storage-versus-computation split is wrong.
• Abstractions deepen with scale, method held fixed (by end-2027). With dictionary size and sparsity held fixed per parameter across a model family’s scale ladder, the fraction of steering-verified features that transfer across three or more surface domains should rise with scale — and Vafa-style coherence scores should improve materially at frontier scale on open-ended domains. Flat curves on both refute the representational half.
• Directional failures survive pure scaling (testable today). On a synthetic corpus where facts appear in one direction only, reversed-direction recall should stay near the random-name baseline across a 10× parameter sweep with the objective unchanged. If scale alone dissolves the reversal curse, my directionality story is wrong.
• Compression, not just abstraction — the one that tests the title. At matched final loss, models trained with an explicit description-length or bottleneck pressure should show measurably more transferable abstract structure — by an activation-level compression metric, not task utility — than models trained without it. If the pressure buys nothing once loss is matched, “compression” was a redescription of training rather than the mechanism, and the title of this essay is wrong.

I will happily lose on any of these. That is what makes them worth writing down.

The sticker version

Twenty years of machine learning compress — fittingly — into three sentences. We stopped hand-crafting representations and let prediction force better ones into existence. Prediction at scale turned out to be compression, and the evidence increasingly suggests that compression under a capacity constraint builds hierarchical, semantic, causally usable structure — partial, directional, distribution-shaped, but real. And reasoning turned out to live not in the parameter count alone but, above all, in how many times you let the machine re-describe the problem to itself before it answers — scale improving the capacity, coverage and reliability of that process; compression shaping the representations; iteration supplying the multi-step computation.

Scale bought the substrate. Compression built the map. Iteration does the thinking. The next era belongs to whoever teaches the map to keep drawing itself.

Sources and further reading

Foundations. word2vec (Mikolov et al., 2013) · Attention Is All You Need (Vaswani et al., 2017) · The Bitter Lesson (Sutton, 2019) · Scaling Laws (Kaplan et al., 2020) · Chinchilla (Hoffmann et al., 2022)

Compression and memorization. Language Modeling Is Compression (Delétang et al., ICLR 2024) · The Hutter Prize · Sutskever, An Observation on Generalization (2023) · How Much Do Language Models Memorize? (Morris et al., 2025) · Grokking (Power et al., 2022) · Progress Measures for Grokking (Nanda et al., ICLR 2023) · Why Language Models Hallucinate (Kalai et al., 2025)

Inside the model. A Mathematical Framework for Transformer Circuits (2021) · Toy Models of Superposition (2022) · Scaling Monosemanticity (2024) · On the Biology of a Large Language Model (2025) · Natural Language Autoencoders (2026) · Othello-GPT (Li et al., ICLR 2023) · LLMs Represent Space and Time (Gurnee & Tegmark, 2023) · The Semantic Hub Hypothesis (Wu et al., ICLR 2025) · The Platonic Representation Hypothesis (Huh et al., 2024) · Evaluating the World Model Implicit in a Generative Model (Vafa et al., 2024)

Depth, iteration and small models. Expressive Power of Transformers with CoT (Merrill & Sabharwal) · A Little Depth Goes a Long Way (2025) · Physics of LMs 3.3: Knowledge Capacity (2024) · Physics of LMs 2.1: Hidden Reasoning (2024) · Recurrent-Depth Latent Reasoning (Geiping et al., 2025) · COCONUT (Hao et al., 2024) · Tiny Recursive Models (2025) · CompressARC / ARC-AGI Without Pretraining (Liao & Gu, 2025) · ARC Prize HRM Analysis (2025) · DeepSeek-R1 (2025)

Counter-evidence. The Reversal Curse (Berglund et al., 2023) · Embers of Autoregression (McCoy et al., 2023) · GSM-Symbolic (Mirzadeh et al., 2024) · The Illusion of Thinking (Shojaee et al., 2025) · Reasoning Models Don’t Always Say What They Think (2025) · Stochastic Parrots (Bender et al., 2021)

Now and next. ARC Prize 2025 Technical Report (Chollet et al., 2026) · METR Time Horizons · Sutskever on Dwarkesh (Nov 2025) · The Era of Experience (Silver & Sutton, 2025) · Genie 3 (DeepMind) · The Urgency of Interpretability (Amodei, 2025) · A Survey on Latent Reasoning (2025)

First published July 28, 2026. Revision 7 — July 29, 2026: deep multi-agent audit complete — thesis restated to match the evidence, synthesis markers at every thesis-stating sentence, the deflationary rival named and conceded standing, dated falsifiable predictions including a direct test of the title claim, and every quotation and number re-verified against primary sources. If you do not see this note, you are reading a cached copy.