How LLMs Work - For The Rest Of Us

LLMs are fascinating. I’ve been using them for a while now, and it still surprises me that something like this actually works at all. And after spending time learning how they actually work, I ended up even more surprised.

In this blog, I will try to explain how LLMs work without using much math, although some basic understanding of a vector will help relate more. Let’s take an example and carry it across the article to understand all the different parts. When I typed:

The capital of France is

and the model answered Paris, what actually happened? Was it searching a database? Was it running a very large if/else tree? Was it remembering one exact sentence from its training data?

Chances are you already know that’s not what is happening. An LLM is not normal application logic. There is no giant handwritten rulebook inside the model doing something like:

if prompt asks "capital of France":
  return "Paris"

The better mental model is this:

An LLM turns text into token vectors, repeatedly enriches those vectors through transformer layers, and then uses the final vector of the last token to score the next token.

That sentence has a lot going on, let’s try to understand it piece by piece.

One thing before we dive in: I’m describing a classic, simplified version of a transformer here. Real models, especially newer ones, tweak plenty of details under the hood. The big picture still holds, just don’t assume every detail here applies exactly to every model out there.

Table of Contents

• Text Becomes Tokens
• What Is A Vector Here?
• The Model Is A Giant Vector-Moving Machine
• How Attention Works
• How Feed-Forward Works
• Features, Signals, And Patterns
• The Causal Mask
• Predicting The Next Token
• Why It Can Be Wrong
• What About KV Cache?
• The Full Flow

Text Becomes Tokens

The model doesn’t read our prompt the way we do. The first step is tokenization, where the text is broken into small chunks called tokens. A token is sometimes a full word, sometimes a piece of a word, and sometimes a space, a punctuation mark, a code symbol, or a part of a JSON key.

For our prompt, the tokens may roughly look like this:

"The"
" capital"
" of"
" France"
" is"

The thing to note here is that the space before a word can be part of the token, which means "capital" and " capital" may be two different tokens. This looks like a small detail, but it is important because the model is not directly working with words. It is working with token ids.

Tokenization is just convert this text into the list of ids that the model knows how to process. So our prompt becomes something like [token_1, token_2, token_3, token_4, token_5]. The exact ids depend on the tokenizer, but the important part is the shape of the transformation:

text -> tokens -> token ids

Now token ids are still not enough. The model needs numbers it can do math on, so each token id is converted into a vector.

What Is A Vector Here?

A vector is just a list of numbers. A very tiny fake vector may look like this:

[0.12, -0.88, 1.41, 0.03]

Real LLM vectors are much bigger, they usually have thousands of dimensions, not 4. The model has an embedding table that knows how to convert each token id into a vector. So our sentence goes through this shape:

"The"      -> token id -> vector
" capital" -> token id -> vector
" of"      -> token id -> vector
" France"  -> token id -> vector
" is"      -> token id -> vector

This is where people use the phrase vector space. In plain terms, each token becomes a point in a huge learned space, and tokens with related meanings tend to be placed in related directions and regions of that space.

For example, the token " France" starts with a rough meaning that sits near country-like things like Europe, the French language, Paris, maps, and national identity. The token " capital" starts near multiple possible meanings at once. It could be a city, a government, money, or an uppercase letter, depending on the context.

It is important to note that none of this is stored as English labels. The model doesn’t have a row somewhere saying "France" = country + Europe + Paris. It has numbers, and those numbers behave in a way where " France" is related to country-like things and " capital" is related to multiple possible meanings of capital.

So at the start, every token has a rough meaning and a point in that space.

The position of each token, i.e. where it sits in the sentence, first, second, third, and so on, matters, but the embedding table doesn’t know anything about that. " capital" maps to the same starting vector whether it’s the first word of the prompt or the last.

But order is important. "The capital of France is" and "is France of capital The" use the exact same five tokens, just shuffled, and only one of them means anything.

To address this, the model adds the position directly into the token’s own vector, number by number. So the token’s vector now carries its meaning and its position at the same time, and any system that reads that vector later (like attention) picks up on both automatically.

But the model doesn’t fully understand the sentence yet. That happens inside the layers.

The Model Is A Giant Vector-Moving Machine

Once the model has token vectors, it passes them through many transformer layers. A layer takes the current vectors and produces new vectors. After one layer, rough token vectors become slightly more contextual token vectors. After many layers, they become much richer contextual token vectors.

Let’s go back to our prompt. The vector for " capital" initially has multiple possible meanings like capital city, capital letter, financial capital, and capital punishment. Somewhere, the model has to settle on the capital city meaning. But here’s the catch, the model doesn’t do this by rewriting the " capital" vector with information from " France". In GPT-style models, a token can only gather information from the tokens before it (we will see why in the causal mask section). " capital" only sees "The capital", so its vector sharpens a little, but it never sees " France".

The later tokens are where the sentence-level meaning builds up. The vector for " France" can see The capital of, so it stops being the standalone concept of France and becomes France, whose capital is being talked about. And the final " is" vector can see the whole prompt, so it becomes the place where capital + of + France combine into the capital city of France.

This is the important part:

The model is not storing meaning as words. It is storing meaning as points and directions in a large learned vector space.

A transformer layer has two important pieces. Attention, which lets tokens gather information from other tokens, and a feed-forward network, which transforms each token vector further. Attention helps with context, the feed-forward part helps reshape what each token now represents. Together, layer after layer, the vectors keep changing.

So an LLM is basically a giant vector-moving machine, not a lookup table. It doesn’t look up "France" and return "Paris", it moves token vectors through learned transformations until the last token's vector strongly points toward a likely continuation.

We will come back to why the last token is important shortly

What Is A Weight?

We just used the term learned transformations. We also specifically mentioned the last token’s vector. Both of these details matter. First, let’s try to understand what learned transformations means.

A weight is a learned numeric setting inside the model. You can think of it like a tiny knob that controls how strongly one signal should affect another signal. If a signal should matter more, the learned settings strengthen it, if it should matter less, they weaken it. In the real model, this happens by multiplying and combining the numbers inside vectors.

Now here’s the thing, and this was a big Aha moment for me.

A model doesn’t have one giant bucket of a billion knobs where everything is mixed together. A billion-parameter model has billions of learned numbers, but those numbers are highly organized. They are stored in grids of numbers called matrices, and those matrices sit in specific parts of each transformer layer.

It’s not one giant control panel, though. It’s an assembly line. When a token vector enters a transformer layer, it goes through organized stations in a fixed order.

First, it goes through the attention station. This part uses the attention block’s learned weights to create Queries, Keys, and Values. These are the pieces that let a token gather useful context from other tokens. We will talk about Queries, Keys, and Values properly in the next section, it will make more sense after we go through an example.

Then the enriched vector goes through the feed-forward station. This part uses a different group of learned weights, it takes the context-rich vector and reshapes it further.

Each station’s output gets added back onto the vector that was already there. The attention station doesn’t replace the vector, it computes a correction and adds it on top. Same with the feed-forward station. This is usually called a residual connection (also called a skip connection).

A modern model can stack well over a hundred of these layers. If every layer fully replaced the vector, a small mistake early on would compound badly, and the token’s original identity could get lost somewhere around layer 40. Because each layer only adds a correction, the original signal survives the whole way through, and every layer only has to learn a small update instead of reconstructing everything from scratch.

After enough of this adding, the numbers inside the vector can drift into very large or very small ranges depending on what got added along the way. So each station also runs the vector through a quick normalization step before passing it on, basically checking how big the numbers have gotten and scaling them back down to a steady range, so nothing spirals out of control the deeper it goes. It doesn’t change what the vector means, it just keeps the math from misbehaving over so many layers.

It is worth noting that these stations don’t have the same number of knobs. In many transformer LLMs, the feed-forward part has more learned weights than the attention part.

The exact split depends on the model architecture, but the rough idea is that attention handles communication between tokens, and feed-forward does a lot of the per-token processing after that communication has happened.

So the token vector doesn’t touch all the knobs in the whole model at the same time. It moves through a sequence of learned transformations:

attention weights
-> feed-forward weights
-> next layer
-> attention weights
-> feed-forward weights
-> next layer

There are other details inside real transformer layers, but this is the useful mental model. The weights are partitioned by layer and by job. Some weights help with communication between tokens (attention), some weights help with processing the meaning of each token (feed-forward) after that communication has happened.

This is why the phrase learned transformations is so important. The model is not carrying an English meaning like France = country + Europe + Paris.

It is carrying a long list of numbers, and as the vector moves through these organized blocks of weights, those numbers are changed. When those numbers change, what the vector represents also changes.

For our prompt, the weights are what decide that money, uppercase letters, and punishment don’t fit next to " of France", and that capital city does. The attention station can help the final " is" vector gather useful context from "The", " capital", " of", and " France".

The feed-forward station can then further reshape that context-rich vector so it points more strongly toward a likely continuation.

There is no single knob called "Paris" or "capital city" though. The knob example is only a mental model. The real thing is a huge set of learned numeric settings, organized into different parts of the model, that together decide how vectors should be reshaped.

During training, the model learns these weights. During inference, when you are chatting with the model, those weights are fixed.

In other words, when you send a prompt to a normal LLM, you are not changing the model’s actual weights. You are only giving it context for this one generation.

How Attention Works

But how does a token know which other tokens to look at?

That’s where attention comes in. Attention is how tokens gather useful context from other tokens.

Let’s go back to our prompt:

The capital of France is

We said earlier that the vector for " capital" starts out ambiguous. It may mean a capital city, financial capital, or an uppercase letter. The model resolves this by letting later tokens look back at earlier tokens and gather their information. So let’s follow the final " is" token, the token that will eventually have to predict what comes next.

The common terms used in attention are Query, Key, and Value. These names sound more complicated than the idea. Roughly:

Query: what am I looking for?
Key: what kind of information do I have?
Value: what information can I share?

How Does A Token Become Query, Key, And Value?

When a token goes through attention, the model is not using if/else logic. It is using learned matrices to turn token vectors into three new vectors:

Query = what this token is looking for
Key   = what this token offers as a label
Value = what information this token can pass forward

Let’s use our prompt, The capital of France is, and follow the token " is".

Step 1: Start With Token Vectors

Each token already has a vector, which is just a list of numbers. These vectors are the current internal representations of those tokens.

x_is      = vector for " is"
x_capital = vector for " capital"
x_france  = vector for " France"

Step 2: Create Query, Key, And Value

Inside the attention layer, the model has learned weight matrices W_Q, W_K, and W_V. These are grids of learned numbers. The model combines token vectors with these matrices:

q_is      = x_is * W_Q
k_france  = x_france * W_K
v_france  = x_france * W_V

This creates q_is (what " is" is looking for), k_france (what " France" advertises), and v_france (what " France" can contribute). The other previous tokens like "The", " capital", and " of" get their own Keys and Values in the same way.

The important part is that W_Q, W_K, and W_V are different learned matrices, so they produce different versions of the same token vector. The same token can become a Query version, a Key version, and a Value version.

Step 3: Compare Query With Key

Now the model compares q_is with k_france using a dot product:

score = dot_product(q_is, k_france)

A dot product is basically a similarity score. If the Query from " is" matches the Key from " France" well, the score is high, which means " is" should pay attention to " France". If it doesn’t match well, the score is low, which means " France" is probably not very relevant to " is".

In the real model, " is" compares its Query against the Keys of many previous tokens. Then the scores are normalized, usually with softmax, so they become attention weights like:

The     -> 0.05
capital -> 0.40
of      -> 0.05
France  -> 0.50

Step 4: Use The Score To Pull Information

Now the model uses those scores to decide how much information to take from each token’s Value vector. If " France" gets a high score, its Value vector contributes a lot. If "The" gets a low score, its Value vector contributes only a little. Then the model adds these contributions together into one attention output. Conceptually:

attention output for " is"
=
0.05 * v_the
+ 0.40 * v_capital
+ 0.05 * v_of
+ 0.50 * v_france

So the " is" vector receives a lot of information from " capital" and " France".

Step 5: The Token Vector Gets Updated

That attention output is then used to update the current representation of " is". So " is" changes from a generic verb to something more like is, in a sentence asking for the capital city of France. In vector terms, the numbers inside the representation of " is" have changed, and because those numbers changed, what the vector represents also changed.

Step by step, that’s:

1. Create a Query for the current token.
2. Create Keys and Values for previous tokens.
3. Compare Query with Keys.
4. Use the match scores to blend the Values.
5. Update the current token vector with that blended information.

Attention uses learned weights to decide how tokens should pull information from other tokens: Query and Key decide the match, Value carries the information.

So the token " is" may create a Query like I am the point where the sentence continues, looking backward for what this sentence is about. The previous tokens present their Keys. "The" advertises determiner/article, " capital" advertises city/money/uppercase-letter, " of" advertises relation/possession, " France" advertises country/place/proper-noun. The Query from " is" compares with those Keys, and the best matches are " capital" and " France".

Then the model takes the Values from " capital" and " France" and blends that information into the vector for " is". So " is" no longer means just a generic verb. It becomes something closer to is, expecting the capital city of France next.

Notice what happened to the ambiguity of " capital" here. On its own, the token could mean a city, money, or an uppercase letter. But when its information is blended with the information from " France" inside the " is" vector, the combination behaves like capital city of France. The money and uppercase-letter meanings simply don’t fit that combination, so they fade.

It is worth noting that this is not one simple lookup. Every token can attend to many previous tokens, and the model has many attention heads, not just one (explained in more detail below).

One head may focus on grammar (which earlier words does this token depend on?), another may focus on meaning (which country is being talked about?), and another may focus on structure (which bracket or quote should match this one?). That’s why one token can be understood from many angles at the same time.

Attention Heads - More Context

When I first read about attention heads, my mental model was wrong. I thought one head means one match with one other token, and multiple heads exist so a token can match multiple tokens. Actually, a single head already compares one token against every previous token; multiple heads just mean the model runs that comparison several different learned ways at once as shown in the image below.

A head is one full attention system with its own Query, Key, and Value weight matrices. Inside one head:

1. Every token creates a Query
2. Every token creates a Key
3. Every token creates a Value
4. The current token's Query is compared against previous tokens' Keys
5. The matching scores decide how much of each previous token's Value to blend in

So even with one head, when the model is processing the final " is" in our prompt, it compares its Query against the Keys of "The", " capital", " of", and " France", gets a score for each, and blends their Values. The walkthrough we did above was exactly this, one head.

So what does multi-head attention add?

Instead of doing this once, the model does it many times in parallel. Each head has its own learned matrices:

Head 1: WQ1, WK1, WV1
Head 2: WQ2, WK2, WV2
Head 3: WQ3, WK3, WV3

Which means the same token vector becomes a different Query in each head:

q_is_head_1 = x_is * WQ1
q_is_head_2 = x_is * WQ2
q_is_head_3 = x_is * WQ3

For " is", the Queries may behave like:

Head 1: what is this sentence about?
Head 2: what type of token should come next?
Head 3: what grammar structure am I in?

The English words are our interpretation of the vector math, not something the model stores.

Each head then runs its own attention and lands on its own scores:

Head 1:
" is" attends mostly to " France"

Head 2:
" is" attends mostly to " capital"

Head 3:
" is" attends to nearby structure like "The" and " of"

Then all the head outputs are combined into one updated vector for " is". So one compact way to remember it is:

one head = one attention lens
many heads = many attention lenses

Each lens sees different relationships because each one has different learned weights.

Who decides how many heads?

Humans decide this when designing the model architecture, before training even starts:

token vector size = 4096
attention heads per layer = 32
each head works with 128 dimensions

So in a layer with 32 heads, a token gets 32 different attention lenses, 32 different learned query-types asked in parallel.

Who decides what the heads ask?

Training decides that. Nobody sits down and writes head 1 = grammar, head 2 = meaning. Each head starts with its own matrices, and during training the weights slowly adjust. Some heads end up becoming useful for certain patterns:

pronoun -> which noun does it refer to?
opening bracket -> where is the closing bracket?
adjective -> which word is it describing?
capital + country -> which relation is being completed?

This specialization is emergent, it is not manually assigned.

One thing to be careful about, a head’s question is a retrieval question, not a reasoning question. Attention asks which previous tokens should I read from?. What to do with the gathered information is the feed-forward network’s job, which we will cover in the next section.

Boiled down:

number of heads = chosen by the model designers
what each head learns to look for = learned during training
which tokens each head actually attends to = decided from the current prompt

Translated to English, attention is a token looking at the useful previous tokens and pulling their information into its own vector.

How Feed-Forward Works

Attention isn’t the whole story though. Feed-forward is a little harder to feel than attention. Attention has an obvious story. " is" looks back, finds " capital" and " France", and pulls information from them. Feed-forward doesn’t have a story like that.

This was the part that took me the longest to internalize.

If attention asks which other tokens should I read from?, feed-forward asks what does this enriched vector now mean? That is the core idea. Attention gathers useful context, feed-forward processes the token after that context has been gathered.

Let’s continue with our prompt. After attention, the vector for " is" carries mixed signals. It is a linking verb, the sentence is about a capital, the country is France, a factual statement is being completed.

The feed-forward network works on this updated " is" vector and tries to reshape it into a better version for the next layer. It’s asking given these signals together, which learned features should become stronger?

For example, it may strengthen signals like the capital here means a city and not money, the expected next token is the name of a city, the city belongs to France. These are human descriptions for directions and signals inside the vector, the model itself has no English labels for any of it.

This is the main difference between the two:

attention:
  moves information between tokens

feed-forward:
  processes the updated meaning inside each token

Attention mostly moves information around, like capital and France into is, or a function name into a later function call. Feed-forward processes the result. Capital + France means the likely answer type is a city, JSON key + value means this is structured data.

So a lot of the pattern processing happens inside feed-forward. For our prompt, attention helps the final " is" vector gather information from " capital" and " France".

Then feed-forward can process that combined information and strengthen signals like “this is a factual completion”, “the relation is capital-of”, “the country is France”, “the expected answer is a city”, “Paris should become more likely” (again, directions inside the vector, not literal labels).

In short, feed-forward takes a token vector that already has context, inspects what signals are present, and rewrites it into a better version for the next layer.

expand -> activate -> compress

before:
" is" = generic verb + pulled information from capital and France

after:
" is" = expecting the capital city of France next, ready for the next layer

Features, Signals, And Patterns

We have used words like features, signals, and patterns a few times now. These words are easy to mix up, so let’s make them more concrete.

A feature is a characteristic or a property. A signal is how strongly that feature is present. A pattern is a recognizable combination of signals.

So instead of thinking of a feature as a fixed object sitting somewhere inside the vector, think of it as a property that can be weakly or strongly active inside a vector. This distinction matters.

When I say the vector has a country signal or a city signal, I don’t mean there is a literal English label stored inside the model. I mean the numbers inside the vector are arranged in a way that behaves like that property is active.

Let’s take our prompt again:

The capital of France is

After attention and feed-forward processing, the vector for " France" may carry a strong country signal, a strong Europe signal, and a strong capital-being-asked signal, alongside a weak person-name signal and a weak French-language signal.

The weak signals are also important. The model is not usually picking one clean box like this is only a country. It is carrying many possible properties at different strengths. Some are strong, some are weak, some may become useful later.

Now here’s the important part. When multiple signals are active together, the model can recognize a larger pattern. Country + capital-of relation + factual completion adds up to the capital city of France, and that pattern can influence future tokens.

The model may make tokens like " Paris" much more likely, because the active pattern is the combination, the capital city of France, which is richer than capital or France alone.

This is also why attention and feed-forward are useful as two different jobs. Attention helps move useful signals between token vectors.

Feed-forward looks at the current token vector and asks, in a learned numeric way, which features are active? Which combinations matter? What should become stronger or weaker? What should be written back into the vector?

So when we say feed-forward does pattern processing, this is the kind of thing we mean. It detects useful combinations of active signals and rewrites the token vector so the next layer has a better representation to work with.

Let’s put this together for the full prompt. The final " is" in The capital of France is has gathered information from all the previous tokens, so its vector may carry signals like:

factual-completion signal: strong
country signal: strong
capital-relation signal: strong
expects-city-next signal: strong
France-context signal: strong

Together, that pattern makes " Paris" score highly, because the current prompt activates a learned combination of signals that points strongly toward " Paris" as the next token. There’s no rule like if country == France and relation == capital: return Paris being executed anywhere.

When we say the model has “seen” a pattern before, we do not always mean it memorized that exact sentence. More accurately, during training the weights learn useful combinations of signals, and during inference the current prompt temporarily activates some of those combinations.

The Causal Mask

In GPT-style models, tokens are not allowed to see future tokens. This is called causal masking. For our prompt, the visibility roughly looks like this:

"The"      can see: "The"
"capital"  can see: "The", "capital"
"of"       can see: "The", "capital", "of"
"France"   can see: "The", "capital", "of", "France"
"is"       can see: "The", "capital", "of", "France", "is"

The first token doesn’t know the full sentence. The last token does. This is why the final vector of the last token is so important.

The model does not merge all token vectors into one big final vector and then predict from that. Instead, because of attention and causal masking, the last token’s vector becomes the place where useful previous context has been gathered.

For our prompt, the final vector at " is" becomes something like I have seen “The capital of France is”, the next token is probably a city name, and the city is probably Paris (a human description of what the vector represents, not literal stored text).

Predicting The Next Token

Now the model has a final vector for the last token. The next step is vocabulary scoring. The model has a fixed vocabulary, which is basically a menu of possible tokens like " Paris", " London", " the", " function", "{", and tens of thousands more.

The model takes the final vector of the last token and gives a score to every possible next token.

For our prompt, the scores may look roughly like:

" Paris"  -> very high
" London" -> lower
" France" -> lower
" a"      -> low
" {"      -> almost zero

Then it chooses or samples one token, and it emits Paris. Now the text becomes The capital of France is Paris, and the model repeats the same process to predict the next token.

This is the generation loop:

take current text
turn it into tokens
convert tokens into vectors
move vectors through transformer layers
score the next token
choose one token
append it
repeat

That’s why LLMs stream text one piece at a time. They are not writing the whole answer in one internal step and then revealing it slowly, they are repeatedly choosing the next token.

Why It Can Be Wrong

The model scores likely next tokens, it does not automatically verify truth.

If the prompt points the model toward a wrong but plausible continuation, it may continue in that direction. For example, if the context strongly suggests a fake package name, fake API, or fake method, the model may produce something that looks very real.

It is doing exactly what it was built to do. Given this context, produce a likely continuation. That likely continuation is often correct because the model has learned a lot of useful patterns. But likely and true are not the same thing.

This is why hallucination is not a weird extra behavior bolted on the side. It falls naturally out of the mechanism. If the model has no way to check the world, it can only continue from the prompt, the conversation so far, and the patterns stored in its weights.

That’s where retrieval, tools, and code execution come in. If the model can search your docs, query a database, inspect your repo, or run tests, then it is no longer only relying on what is inside its weights and current context. The base mechanism is still next-token prediction, but the context is now grounded in fresh external information.

What About KV Cache?

There is one performance detail that is worth understanding because it explains why generation is fast enough to be usable. When a model generates text, it predicts one token, appends it, predicts the next token, appends it, and so on.

Naively, the model would have to recompute all previous tokens every time it generates a new token. That would be expensive.

But causal masking gives us a useful property:

Old tokens cannot see future tokens, so their attention data does not need to change when a new token is appended.

Suppose the prompt is The capital of France is. The first pass processes all prompt tokens once, this is often called the prefill phase. During this pass, the model saves the previous tokens’ Keys and Values at each layer. That saved memory is called the KV cache, a Key cache and a Value cache.

Now the model predicts Paris. To predict the token after "Paris", it doesn’t need to recompute everything from scratch. The old tokens (The, capital, of, France, is) already have their Key and Value data saved.

So the model mostly processes the newest token, while letting that newest token attend backward to the cached Keys and Values.

Put simply, old tokens have already done their work, so their attention information gets saved, and when a new token comes in, it just reuses that saved information instead of redoing it.

old tokens:
  already processed
  keys and values saved
  not recomputed

newest token:
  goes through the layers
  creates queries
  attends to cached keys and values
  becomes context-rich
  predicts the next token

KV cache does not make the model smarter. It makes generation faster.

The Full Flow

If we put everything together, an LLM generation step looks like this:

That is the core mechanism. The model does not execute a handwritten rule for The capital of France is. It moves vectors through learned transformations until the final " is" vector strongly points toward continuations like " Paris", and then " Paris" gets a high score from the vocabulary menu.

Thanks for reading.