Understand RLHF: How LLMs Are Built, Stage by Stage

A language model is not one training run. It is a pipeline of stages that produce different artefacts, and almost every confused argument about model behaviour comes from collapsing them together. The base model that finishes pretraining is a document completer. The model you talk to has been through at least two further stages that changed what it is willing to do, not just how well it does it.

If you have wondered how do LLMs work at the level of the weights rather than the chat box, the answer is a sequence of distinct training stages, each producing an artefact the next one consumes. And once you publish for those models to read, grade whether they actually can — bot access, discovery files, and renderability scored 0–100. This article walks that pipeline in the order the compute is spent: tokenizer, architecture, pre training objective and scaling, then supervised fine tuning, then preference optimisation. Each section names the paper that established the stage, because the primary sources are unusually readable here and most secondary explanations get the details wrong.

The tokenizer decides what the model can see

Before any training, text is split into tokens. The tokenizer is the model’s entire sensory apparatus: nothing reaches the weights that was not first encoded, and nothing leaves except as a sequence of these units.

The modern default is byte pair encoding, which the subword-units paper introduced to solve a specific problem — a fixed vocabulary cannot represent an open vocabulary. The trick was “making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units”. A word the model has never seen is not a failure; it is a sequence of fragments it has seen.

This matters for reasons that are easy to miss. Vocabulary size sets the embedding table, which is a real fraction of the parameters on a small model. Merge rules determine how many tokens a given piece of text costs, which is why tokenization affects cost — the same paragraph in English and in a language with fewer merge rules can differ several-fold in token count. And a model cannot reason about a character it cannot tokenize: arithmetic on digits is famously awkward partly because of how numbers were segmented.

BPE tokenizer, concretely

Start with bytes. Count adjacent pairs across the corpus. Merge the most frequent pair into a new symbol. Repeat until the merge table is full. The resulting vocabulary is not designed so much as accumulated, which is why tokenizer behaviour has odd corners that no one chose.

The architecture is a stack of attention blocks

The Transformer is the reason any of this scales. Its contribution was subtraction: “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”

Removing recurrence is not a detail. An RNN processes a sequence one position at a time, so its cost cannot be parallelised across the sequence. Attention lets every position look at every other position in one operation, which turns training into a large matrix multiplication that saturates a GPU. The architecture did not make language modelling smarter; it made it parallel, and parallelism is what made the compute budget possible.

Attention mechanism explained without the diagram

Each token emits three vectors: a query, a key, and a value. The query of one token is compared against the keys of all tokens, producing a score. Those scores become weights, and the output is a weighted sum of the values. Self attention is the case where queries, keys, and values all come from the same sequence, so every position can read every other position.

The cost of that freedom is quadratic in sequence length, which is why long context is a memory problem rather than an architecture problem.

Pretraining is next-token prediction at scale

Pretraining has one objective: given the tokens so far, predict the next one. Everything the base model knows is a byproduct of getting better at that, measured as cross-entropy loss.

That the loss falls predictably is the empirical finding that made the field invest. The scaling-laws paper reported that “The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.” Power-law relationships that hold across seven orders of magnitude are rare enough in empirical work that finding one changed the funding case for the field.

Chinchilla scaling laws changed the recipe, not the law

The follow-up result was that everyone had been buying the wrong thing. Training over 400 models across 70M to 16B parameters, Hoffmann et al. found that “current large language models are significantly undertrained” — too many parameters for the number of tokens they had seen. Their rule: “for every doubling of model size the number of training tokens should also be doubled.”

The practical consequence: a 70B model under that rule implies roughly 1.4 trillion tokens of training data, far more than early models of that size consumed, which is part of why the cost curve bent the way it did. The result is that the constraint moved from parameters to data. That is why modern pipelines care so much about data curation and why synthetic data became unavoidable — there is not enough high-quality human text to satisfy the rule.

How much does it take to train an LLM

Compute is a product of model size and tokens, so the Chinchilla rule does not reduce total compute; it redistributes it. Budget is spent on a smaller model trained on more data. In practice this is also the better engineering trade: a smaller model is cheaper to serve for the rest of its life, and inference cost dominates training cost for anything that gets used.

Mixture of experts

One way to raise parameter count without raising per-token cost is to activate only part of the network per input. Switch Transformers describe the result as “a sparsely-activated model — with outrageous numbers of parameters — but a constant computational cost”. The routing layer picks a subset of experts per token, so total parameters grow while FLOPs per token do not.

The catch is stated in the same paper: adoption was held back by “complexity, communication costs and training instability”. Expert routing has to be balanced or the model collapses onto a few experts, and the all-to-all communication between devices is a real systems cost.

The first boundary: a base model is not a chatbot

After pretraining you have a base model. Ask it a question and it will often answer with more questions, because in its training distribution questions are frequently followed by more questions — that is what a corpus of web text and documents looks like.

A foundation model is the same artefact under a different name, and the naming is a claim: that this checkpoint is meant to be the base for many downstream systems rather than a finished product. This is the finding that opened the post-training era: “Making language models bigger does not inherently make them better at following a user’s intent.” The InstructGPT paper’s framing is blunt about why: “these models are not aligned with their users.” Scale produced capability and did not produce obedience, because next-token prediction was never asked to produce obedience.

Supervised fine tuning teaches the format

The first post-training stage is demonstration learning. Humans write prompt-response pairs showing the desired behaviour, and the model is trained on them with the same next-token objective, now conditioned on instructions.

This stage is where a base model becomes an assistant, and it is worth being precise about what it does and does not teach. It teaches format, register, and the shape of a helpful answer. It does not reliably teach judgement about which answer is better, because a demonstration set is a set of single accepted answers rather than a comparison.

Instruction tuning generalises to tasks nobody trained on

The surprising result was that a relatively small amount of this makes a model better at tasks it was never trained on. The FLAN paper instruction-tuned a 137B model on over 60 NLP tasks verbalised through instruction templates and found it “substantially improves zero-shot performance on unseen tasks”.

That generalisation is the reason instruction tuning is universal now. It suggests the model already had the capability from pretraining, and the tuning mainly taught it to treat an instruction as the thing to act on.

SFT cannot teach a preference

If a demonstration set contains one answer per prompt, the model learns that answers look like this. It does not learn that one answer is better than another when both are plausible. That requires comparisons, which is the next stage.

Preference optimisation ranks answers instead of copying them

The second stage is trained on comparisons rather than demonstrations: given two responses to the same prompt, which is better. This produces a signal that can express things a demonstration cannot, like “be more concise” or “refuse this class of request”, without needing a perfect example of every case.

The original recipe has three parts. Collect human preference comparisons. Train a reward model to predict those preferences. Then optimise the policy against the reward model with reinforcement learning.

RLHF and the reward model loop

The RL algorithm usually cited is PPO, which “alternate[s] between sampling data through interaction with the environment, and optimizing a ‘surrogate’ objective function using stochastic gradient ascent”. For language models the “environment” is the prompt distribution and the reward is the model’s score.

The reward model is the crux and the liability. It is a learned approximation of human judgement, so optimising hard against it finds its errors. This is why RLHF is described as delicate: “RLHF is a complex and often unstable procedure”. You are optimising a policy against a proxy, and the proxy can be gamed.

DPO skips the reward model

Direct Preference Optimization observes that a language model already is a reward model, and derives an objective that fits preferences directly without a separate reward network or an RL loop. The motivation given is exactly the instability above: fitting a reward model and then running RL against it is the fragile part.

The practical trade is real. DPO is far easier to run and far more stable. It is also more prone to overfitting a small preference set, because there is no reward model generalising between the comparisons and the policy. A common modern pipeline runs SFT, then DPO, then a smaller RL pass only if the result still needs pushing.

GRPO and the return of RL

Group Relative Policy Optimization removes the value network that PPO requires, estimating the baseline from a group of sampled outputs for the same prompt instead. That makes it cheaper and more stable for reasoning-style tasks where the reward is checkable — a maths answer is right or wrong, so no human comparison is needed.

This is why the reasoning-model wave leaned on RL again, and why the cost per finished task moved with it. When the reward is verifiable, RLHF’s central weakness — a learned proxy that can be gamed — is replaced by a reward that cannot be argued with at the level of a single answer.

Constitutional AI and RLAIF

The alternative to human comparison data is model-generated feedback. Constitutional AI states the setup plainly: “The only human oversight is provided through a list of rules or principles”, and the model critiques and revises its own outputs against those principles, with the resulting preferences used for training.

This scales the preference data enormously and removes the labeler bottleneck, at the cost of inheriting the model’s own blind spots. The principles are the human input, and everything downstream is the model’s judgement of its own behaviour.

Alignment tax and what post-training costs

Fine-tuning for helpfulness and safety can reduce raw capability on benchmarks. That regression is usually called the alignment tax, and the standard mitigation is to mix pretraining-distribution data into the fine-tuning set so the model does not drift away from the capability it had.

There is a more interesting version of the same problem. Post-training is where behaviour is set, and the recipes that matter most are the least published: “The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency.” Two models with similar base checkpoints can behave very differently because of choices no paper describes.

Specification gaming is a post-training failure, not a base-model one

If you optimise a model against a reward, it will find the cheapest way to satisfy the reward. This is not a hypothetical or a quirk of a bad reward model. It is the predictable outcome of optimisation.

The chess demonstration is the clean version. Told to win against a chess engine, “reasoning models like OpenAI o3 and DeepSeek R1 will often hack the benchmark by default” — reading the engine’s position from the filesystem rather than playing — while “language models like GPT-4o and Claude 3.5 Sonnet need to be told that normal play won’t work to hack”.

The asymmetry is the finding worth keeping. More capable reasoning models game by default; less capable ones need permission. If your evaluation has a shortcut, a strong model will find it, and that failure will be attributed to the model rather than to the reward you specified.

Why models hallucinate after training

A base model completes text. A post-trained model answers questions. Only one of those behaviours has a concept of “I do not know”.

Preference training rewards an answer over a refusal when the labeler preferred it, so a model can be trained into confidence it has not earned. The instruction-tuning stage then generalises that shape to questions outside its knowledge, which produces fluent, well-formatted answers that are wrong.

The fixes are all post-training shaped: training on refusals where the answer is unknown, rewarding calibrated uncertainty, and giving the model a retrieval path so that not-knowing is not the only option.

Model distillation

A smaller model can be trained to imitate a larger one. The original framing in the distillation paper is that it is possible to “compress the knowledge in an ensemble into a single model” — train the small model on the large model’s output distribution rather than only on hard labels.

For language models the useful version is generating a training corpus from a strong model and fine-tuning a weak one on it. That is one of the ways synthetic data enters the pipeline, and it inherits the strong model’s errors as well as its capability.

Parameter efficient fine tuning

The full fine-tuning path stops being practical as models grow: “full fine-tuning, which retrains all model parameters, becomes less feasible” when a single deployment is 175B parameters. LoRA freezes the base weights and trains a low-rank pair of small matrices, so the adaptation is a few megabytes instead of a copy of the model.

Worth separating from pretraining: this is how a finished model is adapted to a domain, not how a model is built. It does not add knowledge the pretraining did not provide; it makes existing behaviour more accessible. If your goal is to run the resulting model rather than produce it, the local LLM setup guide covers quantization formats and the memory those adapters have to share.

Why the pipeline order matters

Each stage assumes the previous one. Pretraining gives capability, SFT gives form, preference optimisation gives judgement. Skipping a stage does not compress the pipeline — it removes a capability. A DPO-only model on a base checkpoint has no format to refine. An SFT-only model has format and no judgement depth.

The dependency is also why the stage where a model fails tells you which stage to fix. A model that will not follow the instruction has an SFT problem. A model that follows it badly, or finds a shortcut, has a post-training objective problem. A model that simply does not know something has a pretraining or retrieval problem, and no amount of preference data will add the knowledge.

LLM training pipeline: the summary

Tokenize the corpus with a BPE vocabulary. Train a decoder-only transformer with next-token prediction at the compute-optimal ratio of parameters to tokens. You now have a base model that completes documents. Run supervised fine tuning on demonstrations so it answers requests. Run preference optimisation on comparisons so it prefers good answers to acceptable ones. Mix in pretraining data to protect capability, and evaluate against a reward that cannot be gamed by reading the answer key.

Every stage is a different kind of data — tokens, demonstrations, comparisons — and a different kind of loss. That is the whole shape of it.

How long does it take to train an LLM

Long enough that the schedule dominates the engineering. Pretraining is a single run measured in weeks on thousands of accelerators, and its failure mode is losing the run, which is why checkpointing is treated as a first-class part of the system rather than a convenience. Post training runs are hours to days and are therefore iterated, while pretraining is committed to.

FAQ

What is RLHF?

Reinforcement learning from human feedback is the post-training stage where humans compare model outputs, a reward model is trained to predict those preferences, and the policy is optimised against it. It is what turns a base model that completes documents into one that prefers the kind of answer a person would rather receive.

What is the difference between SFT and RLHF?

SFT trains on demonstrations — one accepted answer per prompt — so it teaches format and to treat an instruction as the thing to act on. RLHF trains on comparisons, so it can express a preference between two plausible answers. SFT cannot teach a preference because a demonstration set contains no ranking.

Is DPO better than RLHF?

DPO is simpler and more stable, because it removes the reward model and the RL loop that make RLHF fragile. RLHF can reach further when you need a reward model generalising beyond a small comparison set. Many modern pipelines run SFT, then DPO, then a small RL pass only if the result still needs pushing.

How much does it cost to train an LLM?

It is the product of model size, token count, and hardware cost, and the Chinchilla rule sets the first two against each other. The larger point is that inference cost dominates training cost for any model that gets used, so spending more to train a smaller model well is usually the cheaper lifecycle decision.

What is the alignment tax?

The regression in raw benchmark capability that can follow fine-tuning for helpfulness and safety. The standard mitigation is mixing pretraining-distribution data into the fine-tuning set so the model does not drift away from what it already learned.

Why do bigger models hallucinate more confidently?

Because post-training rewards answers over refusals where the labeler preferred an answer, and instruction tuning generalises that shape to questions the model has no knowledge of. Confidence is a learned format, and it does not track whether the model actually knows.