Table of Contents
What happens in the second before the reply appears
Type “Who won the 1998 Tour de France?” into a chatbot and the answer arrives in about a second, streaming in word by word. Nothing is being looked up. There is no database of race results sitting behind a search box. The model produced that name the same way it produces every other word in the sentence: by working out what should probably come next.
That single idea explains most of what chatbots are good at and a fair amount of what they get wrong. Here is the whole chain, from the moment you hit enter to the final full stop.
It starts with next-word prediction
Large language models are trained on an ocean of text: web pages, books, code repositories, forum threads, instruction manuals. The training objective is almost boring. Take a passage, hide the next token, ask the model to guess it, measure how wrong it was, and nudge the internal numbers a fraction in the right direction. Repeat a few trillion times.
Nobody hands the model a list of facts. It builds one indirectly, because the only way to get reliably good at guessing the next token is to absorb grammar, tone, formatting conventions, and the way ideas tend to follow each other.
Step one: your question gets chopped into tokens
Models never see letters. Text is sliced into tokens, usually chunks of three or four characters. “Unbelievable” might become un + believ + able. Common words survive as single units; rare ones get broken into pieces.
This is why a model can whiff on a question like how many r’s are in “strawberry.” It isn’t looking at letters at all, just two or three token IDs that happen to represent the word. If terms like token, embedding and context window are unfamiliar, this glossary of 57 AI terms explained in plain language is a good place to start.
Each token is then converted into a vector, a list of several thousand numbers that positions the word in a kind of mathematical space. Similar meanings land near each other, which is why “doctor” and “nurse” sit closer together than “doctor” and “trampoline.”
Step two: attention works out which words matter
Those vectors pass through dozens of transformer layers. At each one, an attention mechanism lets every token look back at the tokens before it and decide how much to borrow from each.
Take the sentence “The trophy didn’t fit in the suitcase because it was too big.” To resolve it, the model weighs the earlier nouns and lands on the trophy. Swap big for small and the answer flips to the suitcase. That is attention doing its job, one layer at a time, refining meaning rather than just recording it.
Why long conversations get fuzzy
Every model has a context window, the maximum number of tokens it can hold in view at once. Early versions managed around 4,000. Current ones handle anywhere from 128,000 to over a million, which sounds limitless until you realise attention is not spread evenly across all of it. Details buried 40 messages back carry less weight than the paragraph you just typed, and anything outside the window is simply gone. If a chatbot forgets a constraint you set an hour ago, that’s the reason.
Step three: scoring every possible next word
After the last layer, the model produces a score for every token in its vocabulary. That vocabulary might contain 100,000 entries. Those raw scores get squashed into probabilities that sum to 1.
Feed in “The capital of France is” and you might get something like: Paris 92%, located 1.4%, Lyon 0.3%, with the remaining points scattered across a long tail. The model then picks from that distribution. At temperature 0 it always takes the top choice, which is why the same prompt gives the same answer. Crank it to 0.9 and a slightly less obvious word sometimes wins, which is where variety and occasional strangeness come from. If you’re building with the API rather than a chat window, you set that dial yourself, and this practical guide to building with GPT through the OpenAI API walks through how parameters like temperature and top-p change output.
Step four: write one token, then start over
The chosen token gets appended to the input, and the entire process runs again from scratch. This loop is what “autoregressive” means. The model has no plan for the rest of the paragraph, no outline tucked away in memory. It’s re-reading everything including its own output, one step at a time.
At roughly 50 to 80 tokens per second, a 300-word reply means several hundred full passes through the network. It also explains a quirk you have probably noticed: if the first sentence takes a wrong turn, everything after it bends to fit that mistake.
Where the facts actually live
Knowledge is spread across the model’s weights, the billions of numbers adjusted during training. It isn’t stored as rows in a table. Ask the same fact three different ways and you may get three slightly different phrasings, because each one is generated fresh from statistical patterns rather than retrieved. The training loop that turns raw text into those weights is worth understanding if you want the mechanics behind this, but the short version is that the model compresses language into tendencies, not encyclopaedia entries.
So why do chatbots invent things?
Because the training objective rewards plausible continuations, not true ones. When the pattern is well represented in the data, plausible and true usually line up. When it isn’t, the model fills the gap with something that sounds right. In 2023 a New York lawyer submitted a brief citing six court decisions that did not exist; the chatbot had produced convincing names, docket numbers and quotes because that is what a citation-shaped passage looks like. Fake references, invented statistics and confidently wrong dates all come from the same place. AI hallucinations are proving stubbornly persistent precisely because they aren’t a bug bolted onto the system. They’re the same mechanism that makes the system work.
The extra layers stacked on top
A raw base model will happily continue any text you give it, including text you’d rather it didn’t. The assistant you actually talk to has been through more:
- Instruction tuning. Tens of thousands of human-written examples teach the model to answer questions rather than continue them.
- Preference training. Humans rank pairs of responses, and the model learns to lean toward the ones people preferred: clearer, more cautious, better formatted.
- A system prompt. Hidden instructions set persona, tone and refusals before your message is even read.
- Retrieval and tools. Some products fetch live pages or run a calculator and paste the results into the context window, so the answer partly comes from outside the weights.
This is why two products running the same underlying model can feel completely different. One pipes in fresh search results; the other doesn’t.
What this means when you’re the one typing
- Supply the raw material. Paste the document, the data, the exact question. The model can only reason over what’s in front of it.
- Ask for structure. A numbered list of requirements gets followed far more reliably than a vague request for “something good.”
- Treat specifics as drafts. Names, dates, figures and citations should be checked before they leave your screen.
- Push back once. Regenerating with more context usually beats repeating the same prompt and hoping.
- Start fresh threads for new topics. Old context dilutes new instructions.
What changes as models get smaller
The loop stays identical. What’s shifting is where it runs. Distillation, quantisation and better training data have pushed capable models down to a few billion parameters, small enough to sit on a laptop or a phone. A 7B model that fits in 4GB of memory won’t match a frontier system on a hard reasoning task, but it handles summarising, drafting and routine extraction without sending a single word off your device. The trend toward smaller, cheaper models is reshaping what gets built and where, and it means the answer to “how did the chatbot produce that?” is increasingly happening on hardware you own, in the same four steps, at the same speed, for free.
