What is a Large Language Model?

A large language model is a neural network that has read an enormous amount of text and learned one skill extremely well: given some text, predict what comes next. Everything else it does, answering questions, writing code, translating, making jokes, is a consequence of that single skill scaled up to enormous size.

That sounds too simple to produce ChatGPT. The rest of this guide is about why it isn't.

A model does not read words or letters. Text is first chopped into tokens: short chunks of characters that sit somewhere between a syllable and a word. "Understanding" might be several tokens, "cat" is probably one. Models like the ones behind ChatGPT work with vocabularies of roughly 50,000 to 200,000 tokens.

From the model's point of view, your entire conversation is a long sequence of tokens, and its whole job is to continue that sequence.

This is the core loop of every language model, including the large ones: read the tokens so far, compute a probability for every token that could come next, pick one, append it, repeat.

Notice that each choice changes the probabilities for the next one. Generation is not a script being replayed, it is a chain of decisions, each one made fresh.

Before a model can answer anything, it is trained by showing it billions of sentences with the next token hidden, and asking it to guess. Every wrong guess nudges millions of internal numbers, called weights, slightly toward the right answer. Repeat that trillions of times and the model absorbs grammar, facts, styles, code and the rhythm of how people actually write.

After pretraining comes tuning: the model is shown examples of helpful, honest behaviour and refined with human feedback, so that the raw next-token predictor becomes something that tries to be useful when you talk to it.

"It's just next-token prediction" is technically true and usually a bad explanation. To predict text well, a model cannot just memorise phrases. It has to compress the regularities of language: how syntax fits together, which facts tend to co-occur, how a proof unfolds, how a joke is set up.

That compression is why a model trained purely to predict can do things nobody explicitly taught it: translate, summarise, write working code. The abilities emerge from the scale of the prediction problem, not from a rulebook.

A language model predicts plausible text, not verified truth. When it reaches the edge of what it knows, it does not stop. It produces the most likely continuation anyway, in the same confident tone as everything else. That is what a hallucination is.

The strange behaviours are the interesting part: models that leak their training data, that change personality mid-conversation, that fail at arithmetic a calculator from 1980 could do. Studying where prediction breaks is how we learn what these systems actually are.

This guide is the text version of the first minutes of AI Visualized, a course that teaches the inside of language models through interactive instruments you can touch, break and zoom into. If you want to see how researchers investigate the strange behaviours, read about current research.