Concept Map — How Machines Read & The Breakthrough

Session 2A

How Machines Read

From raw text to the limits of sequential models

Bag of Words

How did machines first process text?

Foundation
The simplest representation: count how many times each word appears, ignore order entirely. "The cat sat" and "sat cat the" are identical to a BoW model. It worked surprisingly well for spam detection and sentiment — but lost all grammar, sequence, and nuance.

Tokenisation

How does text become numbers?

Preprocessing
Before any math can happen, text is split into tokens — sub-word units from a fixed vocabulary. "unhappy" → ["un", "##happy"]. Each token gets a unique integer ID. Modern LLMs use Byte-Pair Encoding (BPE) to build ~50k-token vocabularies that balance coverage and efficiency.

Word Embeddings

How do numbers capture meaning?

Representation
Token IDs are looked up in a learned matrix to get dense vectors of hundreds of floats. The geometry encodes meaning: king − man + woman ≈ queen. Embeddings are the model's internal language — everything downstream operates on these vectors.

Sequential Models — the problem

Why did we need something fundamentally new?

Limitation
RNNs and LSTMs process tokens one at a time, left to right. Long-range dependencies fade. Parallelisation is impossible — token 512 must wait for token 511. This bottleneck in both memory and speed was the wall that demanded the Transformer.

Session 2B

The Breakthrough

Attention, Transformers, and how models are trained

"Attention is All You Need" — context

What was the 2017 paradigm shift?

Paper · Vaswani et al. 2017
Eight Google researchers proposed a model that uses no recurrence at all — only attention. Every token attends to every other token simultaneously. Training became massively parallelisable, unlocking the scale that produced GPT, BERT, and every major LLM since.

Attention mechanism — single head

How does one word look at all other words?

Core Mechanism
Each token projects into three vectors — Query (what am I looking for?), Key (what do I contain?), Value (what do I contribute?). Attention(Q,K,V) = softmax(QKᵀ/√d) · V. The dot product scores how relevant each token is; softmax turns scores into weights; values are summed.

Multi-Head Attention

Why does asking many questions simultaneously matter?

Parallelism
Rather than one attention pass, run h heads in parallel — each with its own Q, K, V projections. One head might track syntactic agreement, another coreference, another semantic similarity. Outputs are concatenated and projected back to the model dimension. GPT-4 uses 128 heads.

3 Layers — general neural network

What is the basic building block of any neural network?

Foundations
Any neural net: input → hidden layer(s) → output. Each layer is a matrix multiply + bias + non-linearity (ReLU, GELU). The Transformer's MLP is exactly this — a 2-layer network applied independently to each token position, widening to 4× the hidden size then projecting back.

Transformer Block

How does attention + feed-forward + normalisation combine into one unit?

Architecture
One block = LayerNorm → Multi-Head Attention → residual add → LayerNorm → MLP → residual add. Residual connections let gradients flow cleanly during training. Layer norm stabilises activations. This pattern repeats N times.

Stacking — why depth matters

Why 96 blocks and not 3?

Scale
Early layers detect surface patterns (syntax, word order). Middle layers build semantic relationships. Deep layers handle abstract reasoning and world knowledge. Depth multiplies representational capacity far more efficiently than width alone. GPT-4: ~96 layers. GPT-2 small: 12.

Full Transformer Architecture

How does the complete model connect end to end?

System View
Token IDs → Embedding table → + Positional encoding → [Transformer Block] × N → Final layer norm → Linear projection to vocab size → Softmax → probabilities. During autoregressive generation the output token is appended and fed back in.

Pre-training

How did the model learn language from scratch?

Training Phase 1
Next-token prediction on trillions of tokens of internet text — books, code, Wikipedia. The model minimises cross-entropy loss: predict the next word, compare to the real one, adjust weights. No labels needed. The signal is the text itself. This instils language, facts, and reasoning.

Supervised Fine-Tuning

How is that general knowledge shaped into specific behaviour?

Training Phase 2
After pre-training, the model is fine-tuned on curated human-written examples (instruction → ideal response pairs) so it behaves helpfully, safely, and follows instructions. RLHF (Reinforcement Learning from Human Feedback) further aligns the model using human preference rankings as a reward signal.