How LLMs Work - For The Rest Of Us

· AI 
The capital of France is

An LLM will probably continue that sentence with Paris. How does it get from those five words to the next token?

I’ll use a simplified version of the transformer used for text generation. The token splits, IDs, and scores below are examples; they are not measurements from a particular model.

Table of Contents

Text Becomes Tokens

The model first breaks the prompt into tokens. A token can be a word, part of a word, punctuation, or even a leading space together with a word. Our sentence might be split like this:

"The"
" capital"
" of"
" France"
" is"

Each token has an ID in the tokenizer’s vocabulary. " France" and "France" can have different IDs because the space is part of the token. The ID tells the model which row of the embedding table to read:

"The"      -> id_101
" capital" -> id_6864
" of"      -> id_315
" France"  -> id_9822
" is"      -> id_374
The sample prompt split into five tokens, with a vocabulary lookup giving each token an ID

From Token IDs To Vectors

A token ID identifies a piece of text, but the transformer works with vectors. A vector is a list of numbers. Here is a tiny one:

[0.12, -0.88, 1.41, 0.03]

Real token vectors are much larger. The model has a learned embedding table that maps each token ID to a starting vector. So our five IDs become five vectors, one for each position in the prompt.

Token IDs from the France prompt being looked up in an embedding table to produce vectors

The labels in this drawing omit leading spaces to keep it readable.

The numbers in the embedding table were learned during training. " capital" gets a starting vector whether we are talking about a city, money, or an uppercase letter. Its token ID alone cannot tell the model which meaning fits " France".

The position matters too. The transformer needs to know that " France" comes after " capital". Some models add position information to the starting vectors; others use it when attention compares positions.

What Each Token Can See

A text-generating transformer predicts the next token from the tokens it already has. During this process, a token can use information from itself and earlier positions, but not from later positions. This rule is called the causal mask.

For our five-token example, the allowed positions look like this:

"The"      can use: "The"
" capital" can use: "The", " capital"
" of"      can use: "The", " capital", " of"
" France"  can use: "The", " capital", " of", " France"
" is"      can use: "The", " capital", " of", " France", " is"

The vector at " capital" cannot look ahead at " France" to settle which meaning of “capital” fits the sentence. The later positions can combine the two. By the time we get to " is", that position can use the entire prompt.

The Transformer

The model passes these five vectors through many transformer layers. If we write the shape as code, it looks something like this:

let vectors = startingVectors; // one vector per token
for (const layer of transformerLayers) {
  vectors = layer(vectors);
}

Each call takes five vectors and returns five updated vectors in the same order. After the last layer, the vector at " is" is used to score the next token.

Each layer does two main things. Attention lets a position use information from earlier positions. A feed-forward network then changes each position’s vector separately. Both are controlled by numbers learned during training, called weights. When you send the model a prompt, the weights stay fixed while the vectors change.

How Attention Works

Suppose we are at the final position, " is". Its vector started with that token, but predicting " Paris" needs information from " capital" and " France" too. Attention lets " is" use the positions allowed by the causal mask.

Imagine one part of the attention calculation gives these numbers:

"The"      0.05
" capital" 0.35
" of"      0.05
" France"  0.45
" is"      0.10

The numbers add up to 1. In this example, attention uses more information from " capital" and " France" than from the other positions when it updates the vector at " is".

The 0.45 next to " France" is an attention weight, not a 45% chance of " Paris". The model scores next tokens after the final layer.

Conceptual attention example in which the is position draws information from capital and France

The drawing highlights two earlier positions; " is" can also attend to itself.

Real models do several of these calculations in parallel, called attention heads, and combine the results. The next layer does it again with the updated vectors, so its attention numbers can be different.

How Do Query, Key, And Value Work Numerically?

An attention head makes three versions of each position’s vector: a Query to compare with other positions, a Key to compare it against, and a Value containing the information to combine. These are calculated using three sets of learned numbers. For " is" and " France", we can write:

q_is     = x_is     × W_Q
k_france = x_france × W_K
v_france = x_france × W_V

The Query for " is" is compared with each allowed Key. In scaled dot-product attention, the comparison with " France" is:

score(is, France) = dot(q_is, k_france) / sqrt(key_dimension)

The dot product gives a learned compatibility score. Dividing by the square root of the Key dimension keeps the scores from growing too large as vectors get wider. The causal mask excludes future positions before softmax turns the remaining scores into weights that add up to 1. Some models also change Queries and Keys according to position before this comparison; the equations above leave that out.

Then the block forms a weighted sum of the Values. For the toy weights above:

attention output at " is"
  = 0.05 × v_the
  + 0.35 × v_capital
  + 0.05 × v_of
  + 0.45 × v_france
  + 0.10 × v_is

This is one head’s output. The block combines the outputs of all its heads and projects the result back to the vector size expected by the layer.

What Feed-Forward Does

After attention, the vector at " is" contains information from earlier positions. The feed-forward network changes that vector again. It runs the same calculation for each position independently, much like calling vectors.map(feedForward).

At " is", it receives a vector already influenced by " capital" and " France". It returns another vector for the next layer. It does not go back and read earlier positions during this step; attention did that work.

Predicting The Next Token

After the last layer, the model uses the vector at " is" to predict the next token. It turns that vector into a list of scores, one for each token in its vocabulary. These scores are called logits.

This is the same vocabulary that supplied the token IDs at the start. " Paris" has its own score alongside " London", " a", punctuation, code fragments, and all the other candidates.

In the diagram, " Paris" scores much higher than " London". During training, the model adjusted its weights while predicting tokens in many different contexts. The trained weights produce these scores for the prompt we gave it; no one wrote a rule for this sentence.

The final vector at is scoring possible next tokens, with Paris receiving the highest sample score

The drawing omits leading spaces from the token labels.

If " Paris" is selected, its leading space gives us the text is Paris when it is appended:

"The capital of France is" + " Paris"
= "The capital of France is Paris"

The model then scores another token using this longer sequence. Text arrives one token at a time, and a token may be a word, part of a word, or punctuation.

The model can take the highest-scored token, or it can turn the scores into probabilities and sample from them. A setting called temperature changes how strongly sampling favors higher scores. So the same prompt can produce a different next token.

Why It Can Be Wrong

The score for " Paris" happens to line up with a true fact. The scoring process itself does not check a candidate against the world.

If a model suggests this for reading a JSON file using Node’s built-in fs module:

const fs = require("node:fs");
const config = fs.readJSON("config.json");

Run that code and you get:

TypeError: fs.readJSON is not a function

The fs API has readFileSync(), but no readJSON(). The method name looks reasonable enough for a model to suggest it. You can read the file and parse it with JSON.parse(fs.readFileSync("config.json", "utf8")).

This can happen with package names, method arguments, and citations too. Sampling is one way to get a different answer, but even the highest-scored answer can be wrong. If the model can inspect documentation or run the code, the error becomes part of its context and it has a chance to correct the suggestion.

Where Thinking Fits In

Some models generate intermediate tokens before giving a final answer. For 3:15pm + 2 hours and 50 minutes, a written calculation could go:

3:15pm + 2 hours  = 5:15pm
5:15pm + 50 minutes = 6:05pm

The later tokens can attend to 5:15pm because it is now part of the sequence. Each extra token goes through the same layers, giving the model another step and an intermediate value to work from. If it gets the first step wrong, the second step uses that wrong number.

What About KV Cache?

Without a cache, predicting the token after " Paris" would mean processing the five prompt tokens again along with " Paris". The next step would reprocess those six tokens plus the new one. Generation would do a lot of repeated work.

The causal mask helps here. Earlier positions cannot depend on a token that has not arrived yet. On the first pass over The capital of France is, the model saves some of attention’s calculations for each position. These saved results are called Keys and Values, giving the KV cache its name.

After " Paris" is selected, the model processes this new token and reuses the saved Keys and Values from the prompt. Each attention layer has its own cache. The cache saves repeated work, but it also takes more memory as the sequence grows.

Thanks for reading.