RLHF and DPO explained: how language models learn to be helpful

A pretrained model predicts text; an assistant follows instructions. The bridge is alignment: instruction tuning, RLHF and DPO. How each works, how they differ, and their honest limitations.

Alignment turns a pretrained language model, which only predicts the next token, into an assistant that follows instructions helpfully and safely. It usually happens in stages: supervised instruction tuning on example conversations, then preference training, either RLHF, which trains a reward model and optimises against it with reinforcement learning, or DPO, which learns directly from preferred and rejected answers.

If you have ever wondered why a raw pretrained model rambles while a chat assistant answers your question, this is the reason. Alignment is also one of the most active research areas, and understanding it helps you reason about model behaviour, refusals and failure modes.

Why does a pretrained model need alignment?

Pretraining teaches a model to continue text the way the internet does. Ask it a question and it might continue with more questions, a forum argument or an unrelated paragraph. It has knowledge, but no reliable sense of what a helpful, honest, safe answer looks like. Alignment teaches that behaviour.

What is instruction tuning?

Supervised fine-tuning (SFT) trains the model on curated examples of instructions paired with good responses. It is ordinary fine-tuning, but on carefully written or selected conversations. SFT alone already transforms behaviour, and its quality depends heavily on the quality and diversity of the examples.

How does RLHF work?

This approach was described in research on training language models to follow instructions with human feedback, which became the template for many chat assistants.

What is DPO and why did it become popular?

Direct Preference Optimization starts from the same preference data but skips the separate reward model and reinforcement learning step. Its authors noted that RLHF is complex and often unstable; DPO instead uses a simple classification-style loss that increases the probability of preferred answers relative to rejected ones. It is simpler to implement and tune, which is why it spread quickly.

What are the limitations of alignment?

Where does preference data come from?

From people comparing answers, from experts writing ideal responses, and increasingly from AI-assisted labelling checked by humans. The quality and diversity of this data matter more than its size. For an Indian product, that means raters who understand local context, languages and expectations, otherwise the model learns preferences that do not fit its users.

What is constitutional or AI-assisted feedback?

Some approaches reduce the need for human ratings by having a model critique and revise answers against a written set of principles, then train on those preferences. These methods can scale feedback, but the principles and the checking still come from people, and the results still need evaluation.

Can you do preference tuning on a small budget?

Yes, at small scale. With an open model, parameter-efficient methods such as LoRA and a few thousand preference pairs, you can run DPO on modest hardware and see real behaviour changes. It is an excellent learning project because it connects data, training and evaluation. Our walkthrough of how small language models are trained shows where this fits in the pipeline.

What does an alignment project look like for a learner?

A practical project: take a small open model, write or collect a few hundred instruction examples for a narrow task such as answering course-fee questions politely and accurately, fine-tune with LoRA, then create preference pairs where one answer is clearly better and run DPO. Compare the base, SFT and DPO versions on the same test prompts, including attempts to make the model promise outcomes it should not. The before-and-after comparison teaches more than any lecture.

How does this connect to evaluation?

You cannot tell whether alignment helped without testing. Build a set of prompts covering helpfulness, honesty and safety, compare outputs before and after, and include adversarial prompts. Our guide to evaluating LLM applications covers how.

Where do you learn alignment hands-on?

Program Zero’s advanced LLM engineering phase includes an SFT pipeline, instruction-data curation, reward model training, PPO for RLHF, DPO, red-teaming and guardrail design, building on the earlier phase where you train a small GPT.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Bengaluru, Pune, Hyderabad and Noida, growing cities such as Lucknow, Kanpur, Visakhapatnam and Madurai, and smaller towns such as Siliguri, Jamshedpur, Durgapur and Cuttack. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading