How LLMs Actually Work: Tokenization, Attention, and Next-Token Prediction, With Code

Strip away the marketing language and an LLM is doing one mechanical thing at inference time: given everything written so far, predict a probability distribution over what token comes next, pick one, append it, and repeat. That's it. Everything that looks like reasoning, planning, or "understanding" is what emerges from that loop running thousands of times in sequence.

This isn't a claim to take on faith. By the end of this tutorial you'll have tokenized real text, hand-built the attention mechanism in raw NumPy, and watched an actual model predict tokens one at a time — with print statements at every step showing you exactly what's happening. No hand-waving, no pretrained weights you have to trust blindly.

What You'll Build

• Real text turned into token IDs and back, using the same tokenizer OpenAI-family models use
• A toy embedding lookup showing how a token ID becomes a vector
• Scaled dot-product self-attention, implemented from scratch in NumPy on a 4-token example, with causal masking
• A live generation loop against a real model, comparing greedy decoding to sampling

Step 1: Tokenization

Models don't see words. They see tokens — subword pieces from a fixed vocabulary, each mapped to an integer ID. Install tiktoken, the tokenizer library behind OpenAI's models (cl100k_base below is the GPT-3.5/GPT-4-era encoding; newer OpenAI models have since moved to o200k_base with a larger vocabulary — the exact token IDs shift between encodings, but the byte-pair-encoding mechanism this tutorial is demonstrating is identical, and shared across most modern LLMs regardless of vendor):

pip install tiktoken

1import tiktoken
2
3enc = tiktoken.get_encoding("cl100k_base")
4
5text = "Unbelievable how tokenization works."
6ids = enc.encode(text)
7print(ids)
8# [1844, 32898, 24694, 1268, 4037, 2065, 4375, 13]
9
10for token_id in ids:
11    print(token_id, "->", repr(enc.decode([token_id])))

Run it and look at the output. "Unbelievable" doesn't come back as one token — it splits into pieces like "Un", "belie", "vable". Common short words like "how" and "works" each get exactly one token (note the leading space baked into " how" and " works" — that's how this tokenizer marks word boundaries, instead of a separate space token). There's no dictionary of "words" here at all, just a fixed table of ~100k byte-sequences learned to compress text efficiently.

Two things fall out of this immediately: token count isn't word count (which is why API pricing and context limits are quoted in tokens, not words), and enc.decode(enc.encode(text)) == text always holds — tokenization is lossless, just not human-shaped.

Step 2: Embeddings

A token ID is just an integer — useless to matrix math on its own. The first thing a real model does is look that ID up in an embedding matrix: a table of shape [vocab_size, d_model] where row i is a learned vector representing token i. "Learned" is the key word — in a real model these vectors come from training on enormous amounts of text; every embedding is a floating-point conclusion. Here we'll fake the training and just allocate random vectors, purely to make the mechanic concrete:

1import numpy as np
2
3vocab_size = 50000
4d_model = 8  # real models use hundreds to thousands of dimensions
5
6rng = np.random.default_rng(seed=0)
7embedding_matrix = rng.normal(size=(vocab_size, d_model))
8
9token_id = ids[0]
10vector = embedding_matrix[token_id]
11print(vector.shape)   # (8,)
12print(vector)

That's the whole operation: a row lookup. A sentence becomes a matrix of shape [sequence_length, d_model] — one row per token — by stacking these lookups. In a real model, these particular numbers encode learned relationships (similar tokens end up with similar vectors), but structurally it's nothing more than indexing into a table.

Step 3: Self-Attention, By Hand

This is the mechanism that makes transformers work, and it's small enough to write from scratch. The idea: for every token, look at every other token in the sequence and decide how much attention to pay to each one, then blend their information accordingly.

Take a toy sequence of 4 token vectors (pretend these came from Step 2) and compute scaled dot-product attention:

1import numpy as np
2
3rng = np.random.default_rng(seed=1)
4
5seq_len, d_model, d_k = 4, 8, 8
6X = rng.normal(size=(seq_len, d_model))  # 4 tokens, 8-dim vectors
7
8# Learned projection matrices — random here, trained in a real model
9W_q = rng.normal(size=(d_model, d_k)) * 0.1
10W_k = rng.normal(size=(d_model, d_k)) * 0.1
11W_v = rng.normal(size=(d_model, d_k)) * 0.1
12
13Q = X @ W_q   # what each token is "looking for"      -> (4, 8)
14K = X @ W_k   # what each token "offers" to be found   -> (4, 8)
15V = X @ W_v   # the actual content to blend in         -> (4, 8)
16
17scores = Q @ K.T / np.sqrt(d_k)   # (4, 4) — every token scored against every token
18
19def softmax(x, axis=-1):
20    e = np.exp(x - x.max(axis=axis, keepdims=True))
21    return e / e.sum(axis=axis, keepdims=True)
22
23weights = softmax(scores)         # (4, 4) — rows sum to 1
24output = weights @ V              # (4, 8) — blended per-token output
25
26print(np.round(weights, 3))

Run it and look at weights. Row i, column j is "how much token i attends to token j" — a number between 0 and 1, and each row sums to 1. Token 0 might put 0.6 of its attention on itself and 0.4 split across the others; token 3 might weight token 1 heavily because their vectors happen to have a high dot product. output is each token's original representation replaced with a weighted mix of every token's V vector, weighted by those scores. That's the entire mechanism: score every pair, normalize with softmax, blend.

One thing is missing before this matches a real generative model: causal masking. During generation, token 3 must not be allowed to attend to a token 4 that doesn't exist yet — the model only ever sees what's already been written. Block future positions by setting their scores to -inf before the softmax, which drives their weight to zero:

mask = np.triu(np.ones((seq_len, seq_len)), k=1).astype(bool)  # True above the diagonal
masked_scores = np.where(mask, -np.inf, scores)
causal_weights = softmax(masked_scores)

print(np.round(causal_weights, 3))

Compare the two weight matrices. In causal_weights, row 0 has weight only on column 0 (the first token can only see itself), row 1 has weight split across columns 0–1, and so on — a lower-triangular pattern. That's the constraint that makes left-to-right generation coherent: at each step, the model is only ever conditioning on the past.

Step 4: Stacking It Into a Model

You've now built the core operation. A real transformer stacks it:

• Multi-head attention runs several smaller attention operations like the one above in parallel (each with its own W_q/W_k/W_v), then concatenates the results — letting different heads specialize in different kinds of relationships (one might track syntax, another long-range references).
• Each attention layer is followed by a small feedforward network applied to every token independently.
• Dozens to hundreds of these attention-plus-feedforward blocks are stacked, each refining the per-token representations using the output of the last.
• The final layer projects the last token's vector back out to a vector of size vocab_size — the logits — one raw score per possible next token.

Nothing new happens conceptually after that. Softmax those logits and you have a probability distribution over the entire vocabulary for "what comes next."

Step 5: Next-Token Prediction, Live

Time to watch it happen against a real model instead of toy numbers. The easiest free path is a local model through Ollama, which exposes an OpenAI-compatible endpoint on localhost — swap base_url and api_key if you'd rather point this at the real OpenAI API.

1from openai import OpenAI
2
3client = OpenAI(
4    base_url="http://localhost:11434/v1/",
5    api_key="ollama",  # required by the SDK, ignored by Ollama
6)
7
8MODEL = "qwen3.5:9b"
9prompt = "The capital of France is"
10
11def complete(temperature: float) -> str:
12    resp = client.chat.completions.create(
13        model=MODEL,
14        messages=[{"role": "user", "content": prompt}],
15        max_tokens=5,
16        temperature=temperature,
17    )
18    return resp.choices[0].message.content
19
20print("greedy, run 1:", complete(temperature=0))
21print("greedy, run 2:", complete(temperature=0))
22print("sampled, run 1:", complete(temperature=1.0))
23print("sampled, run 2:", complete(temperature=1.0))

At temperature=0 the model always picks the single highest-probability next token — greedy decoding — so both greedy runs come back identical. At temperature=1.0, the model samples from the probability distribution instead of always taking the top pick, so the two sampled runs can genuinely differ even though the prompt didn't change. That randomness isn't the model being uncertain in some mystical sense — it's a weighted die roll over the same logits Step 4 described, with temperature controlling how flat or peaked that distribution is before sampling.

You can see the distribution directly by asking for log-probabilities on the top candidates — this needs logprobs/top_logprobs support, which Ollama's OpenAI-compatible endpoint doesn't implement (the fields are silently dropped rather than erroring cleanly), so point this one call at the real OpenAI API instead:

1from openai import OpenAI
2
3openai_client = OpenAI()  # reads OPENAI_API_KEY from the environment
4
5resp = openai_client.chat.completions.create(
6    model="gpt-4o-mini",
7    messages=[{"role": "user", "content": prompt}],
8    max_tokens=1,
9    temperature=0,
10    logprobs=True,
11    top_logprobs=5,
12)
13for candidate in resp.choices[0].logprobs.content[0].top_logprobs:
14    print(f"{candidate.token!r:12} logprob={candidate.logprob:.3f}")

That prints the model's actual top-5 candidates for the very next token and their log-probabilities — the literal output of the softmax-over-logits step from Step 4, for a real prompt instead of random matrices. No OpenAI key handy? The greedy-vs-sampled comparison above already made the same point without needing logprobs at all — this step is a bonus, not a requirement.

Common Issues

Confusing parameters with tokens — a "70B model" has 70 billion learned weights (the entries in matrices like the embedding table and every W_q/W_k/W_v/feedforward weight); a "4096-token context window" is how many tokens of conversation it can condition on at once. They're unrelated numbers that both happen to get called "how big is the model."

Assuming the model remembers between API calls — it doesn't. There's no session state on the server. Every call in a multi-turn conversation resends the entire message history; the model is stateless and re-reads everything from scratch each time, which is exactly what Step 5's messages list was doing.

Silent context overflow — push past the context window and older messages get dropped (or the request errors, depending on the client), often silently truncating from the start of the conversation. If a model suddenly "forgets" something from early in a long chat, count your tokens before assuming a bug.

Mistaking fluency for understanding — a confident, grammatically perfect wrong answer isn't evidence the model "understood" and then erred. It's the same next-token-prediction process that produces correct answers, applied to a case where the highest-probability continuation happened to be false. Fluency and correctness are computed by nothing in common.

Frequently Asked Questions

Do LLMs actually understand language, or just predict statistically?

Mechanically, every output token is a sample from a next-token probability distribution — that's the entirety of the forward pass you implemented above. Whether the internal representations that produce good predictions constitute "understanding" is a genuinely open research and philosophical question; what's not open to debate is the mechanism, which is exactly what Steps 3–5 showed you, no more and no less.

What is a "parameter" concretely?

Every number in every weight matrix the model uses — the embedding table, every W_q, W_k, W_v in every attention head, every feedforward layer's weights, across every stacked block. A model with "70 billion parameters" has 70 billion such floating-point numbers, all set during training and fixed at inference time (nothing you do while chatting with it changes them).

Why do bigger models generally perform better?

More parameters mean more capacity to store nuanced, disentangled patterns from training data instead of collapsing similar-but-distinct concepts together. In practice this trades off against training cost, inference speed, and the amount and quality of training data available — bigger isn't strictly better if the extra capacity isn't matched by enough data to use it, which is why data quality and model size are usually discussed together, not size alone.

What's the difference between pretraining and fine-tuning?

Pretraining is the expensive part: learning every parameter from scratch (or near-scratch) by predicting the next token across a massive, broad text corpus — this is where the model acquires general language and world-pattern knowledge, and it costs the compute you hear about in the headlines. Fine-tuning starts from those already-trained weights and continues training on a much smaller, focused dataset (like instruction-following examples or a specific domain) to adjust behavior — the same next-token-prediction mechanism, just applied briefly and narrowly on top of an already-capable model instead of building one from zero.

Official References

• Attention Is All You Need — the original transformer paper that introduced the attention mechanism built in Step 3
• tiktoken — the tokenizer library used in Step 1, with encoding details and vocabulary info
• OpenAI tokenizer reference — an interactive tool for seeing how any text tokenizes
• OpenAI API reference: logprobs — full parameter docs for temperature, logprobs, and top_logprobs used in Step 5

If you haven't already, Run an LLM Locally with Ollama sets up the model this tutorial's Step 5 talks to — install it first if localhost:11434 isn't answering.