Large Language Models
How LLMs Work
By Sajid
An Interactive Visual Guide
From raw text to the next word — scroll through nine steps that turn a sentence into a prediction.
Text to Tokens
A model can't read letters the way we do. First, your text is chopped into small chunks called tokens — whole words, word-pieces, or even single characters.
Each token is mapped to a number (an ID) from a fixed vocabulary. This is the raw input the model actually sees.
Splitting into word-pieces instead of whole words keeps the vocabulary small (tens of thousands of tokens) while still letting the model spell out any rare or never-seen word from its parts. It's also why API usage is billed per token, not per word.
Embeddings
Token IDs are just labels — they carry no meaning. So each token is converted into an embedding: a long list of numbers (a vector) that captures its meaning.
These vectors are learned during training, so similar words end up with similar numbers.
A typical embedding has hundreds or thousands of dimensions — each one a learned "dial" that quietly measures some property of meaning. The model also adds a position signal here, so it knows the order the tokens arrived in, not just which tokens.
Vector Space & Similarity
Think of every embedding as a point in a high-dimensional space. Words with related meanings sit close together; unrelated words drift far apart.
Direction matters too — the path from "king" to "queen" mirrors the path from "man" to "woman."
"Closeness" is measured with math — usually the angle between two
vectors (cosine similarity). Because relationships live in the
directions between points, you can even do arithmetic:
king − man + woman lands near queen.
The Neural Network
We've turned words into vectors. But what actually transforms one vector into another — and where does the model's "knowledge" live? In a neural network: layers of simple units (neurons) joined by connections, each with a learned weight.
A neuron adds up its inputs, each scaled by a weight, and passes the result through a small non-linear function. Stack layers of these and the network can learn astonishingly complex patterns — the weights are what the model knows.
Those weights start random and are tuned over and over against mountains of text until the network's guesses get good. Everything ahead — attention and the feed-forward network — is built from exactly these weighted layers.
Self-Attention
Words mean different things in context. Attention lets each token look at every other token and decide which ones matter most for understanding it.
In "the river bank," attention links "bank" to "river" — resolving the ambiguity by weighing the surrounding words.
Mechanically, each token sends out a query ("what am I looking for?") and every token offers a key ("here's what I am"). The closer a query and key match, the more of that token's value gets mixed in. Models run several of these in parallel — multi-head attention — so they can track grammar, topic, and reference all at once.
Transformer Block
Attention is one piece of a larger unit: the transformer block. It pairs attention with a feed-forward network, plus normalization and residual connections.
Each block refines the representation of every token a little further.
The two halves play different roles: attention lets tokens share information, then the feed-forward network processes each token on its own to add knowledge and computation. Residual connections let the original signal skip past each step (so very deep stacks still train), and normalization keeps the numbers from blowing up.
Stacking Layers
One block isn't enough. Models stack dozens of blocks on top of each other, each building on the last.
Early layers catch grammar and syntax; deeper layers capture abstract meaning, reasoning, and intent.
The number of layers (depth) and the size of each vector (width) are the main dials of model size. More of both means more capacity for nuance — but also far more compute and memory. Large models stack dozens to over a hundred of these blocks.
Next Token Prediction
After all the layers, the model produces a probability for every token in its vocabulary — its best guess at what comes next.
It picks one, adds it to the text, and repeats. That loop, one token at a time, is how an LLM writes.
The final vector becomes a raw score for every token (a logit); softmax turns those scores into probabilities. How the model picks from them is your control: temperature and sampling (top-k / top-p) trade off between safe and predictable versus varied and creative.
How It Learns
Everything so far assumed the dials were already set. But the model starts knowing nothing — its billions of weights begin as random noise. So how does random noise become something that can finish your sentence? It practices the one game you just watched: predict the next word.
Show the model real text, hide the next word, and let it guess. At first the guess is nonsense. But we can measure exactly how wrong it was — a single number called the loss. The bigger the surprise, the bigger the loss.
Then comes the trick: the model traces that error backwards through every layer and nudges each weight a tiny amount in the direction that lowers the loss. One nudge barely matters. Repeat it across trillions of words and the dials settle into a configuration that predicts astonishingly well — grammar, facts, and reasoning all emerge from this one relentless loop.