Model Thinking
You do not need to understand transformers to work effectively with LLMs — but you do need a mental model accurate enough to predict when they will shine and when they will confidently fail. This page builds that model with no math, on the principle from Abstraction: every non-trivial abstraction leaks, and “ask in English, get an answer” is the leakiest abstraction in modern computing. What follows is the one layer beneath.
The core loop: predict the next token
Strip away everything else and an LLM does exactly one thing, over and over: given all the text so far, produce a probability estimate for what small chunk of text comes next, pick one, append it, repeat.
- Tokens are those chunks — roughly word-fragments. “Computational” might be two tokens; common words are one; rare names shatter into several. The model reads tokens and writes tokens; it has never seen a letter or a word the way you do. This is why classic stumbles like counting the letters in a word are architectural, not stupidity: you are asking about units the model literally does not perceive.
- Probability, not lookup. The model is not searching a database of facts. It has compressed the patterns of an enormous training corpus into billions of numeric weights, and it reconstructs plausible continuations from those patterns. When the pattern coverage is dense — common code, mainstream knowledge, standard formats — reconstruction is astonishingly accurate. When coverage is thin, the machinery runs identically and produces something shaped like an answer.
- Sampling. At each step the model usually does not take the single most probable token but samples from the distribution (the knob is often called temperature). That is why the same prompt yields different answers on different runs — the variability is a setting, not a mood.
Context windows: the model’s working memory
The context window is the bounded stretch of tokens the model can attend to in a single exchange — your prompt, any documents you pasted, the conversation so far, and its own ongoing answer, all sharing one budget.
Practical consequences:
- Nothing persists outside it. The model has no memory between conversations unless the application re-inserts history. It “remembers” earlier chat turns only because they are literally re-sent every time.
- Attention is uneven. Even within the window, material in the middle of very long contexts tends to get less effective attention than the beginning and end — burying the key instruction on page 40 of pasted material is asking for it to be missed.
- Overflow is silent. When conversations exceed the window, older content is truncated or summarized away. The model does not announce this; it simply stops being influenced by what it can no longer see. Long-session drift — the agent “forgetting” constraint you set an hour ago — is usually this, not caprice.
Working implication: treat context as a scarce, curated resource. Put the load-bearing instructions and data in explicitly, close to where they are needed, and restate what matters in long sessions.
Hallucination: the leak, named
Hallucination is fluent output not grounded in fact — invented citations, plausible-but-fake API functions, confident biography errors. Given the mechanism above, it needs no exotic explanation: the model’s job is to continue text plausibly, and when it lacks the pattern to be right, plausible is what remains.
Recent research sharpened this from vibe to mechanism. OpenAI’s 2025 analysis (Kalai et al.) argues hallucination is a statistically inevitable byproduct of how models are trained and evaluated: benchmarks score confident answers above honest abstentions, so training pressure systematically favors the well-phrased guess over “I do not know.” Hallucination, in other words, is not a bug being patched out next quarter; it is a trade-off being managed.
Predictable high-risk zones, worth memorizing:
- Specifics over gists — exact numbers, dates, quotes, URLs, case citations, function signatures. The gist is usually right; the pointer-like details are where fabrication concentrates.
- Thin training coverage — niche domains, recent events, your private codebase, anything past the model’s training cutoff.
- Leading questions — “explain why X causes Y” invites a fluent explanation whether or not X causes Y. The model completes your pattern.
- Long chains — each reasoning step is sampled; small per-step error rates compound across a long derivation.
The practitioner’s summary
Hold these seven statements and you have the working mental model:
- It predicts tokens; everything else is emergent from that.
- It reconstructs from patterns; it does not look things up.
- Dense-pattern territory → excellent; sparse territory → confident improvisation, same tone.
- Its memory is the context window — bounded, curated by you, silently truncated.
- Randomness in output is sampling, a dial rather than a defect.
- Fluency and confidence carry near-zero information about correctness.
- Therefore: ground it (supply the facts in context), constrain it (specify tightly, per Problem Formulation), and verify it (per Verification Thinking).
Notice these are the classical pillars in new clothes: model thinking is abstraction discipline applied to a stochastic component. Practitioners who hold this model stop being surprised by failure modes and start designing around them — which is precisely the difference between using AI and being used by it.