Advanced8 min read

Instruction Tuning vs Preference Tuning

Instruction tuning teaches "this is the answer." Preference tuning teaches "this answer is better than that one." Do SFT first; in 2026 most teams then run DPO, not a full RLHF/PPO stack.

Explain like I'm new to AI

Two data shapes:

SFT / instruction tuning: one prompt, one gold completion. The model imitates. Great for JSON shape, tone, "always cite."
Preference tuning: one prompt, two completions, a label which is better. The model shifts toward chosen, away from rejected. Great for "less smarmy," "refuse this class of request," "shorter."

prompt → one gold reply

Show the model the answer you want. It copies the format, tone, and steps. First stage of almost every tune.

SFT teaches a target answer. DPO teaches a preference. Most 2026 pipelines do SFT first, then DPO — not PPO.

RLHF (reward model + PPO) is how frontier labs historically aligned chat models. It works. It is also a lot of machinery. DPO fits a similar goal with one offline training run and no separate reward model — that's why product teams standardized on SFT → DPO.

GRPO and verifier-based RL show up when you have a checkable reward (unit tests, math). Different tool, same idea: optimize preferences or scores after imitation.

Mental model

SFT is a worked example in a textbook. DPO is a teacher circling the better essay and crossing out the worse one. You still need the worked examples first or the model doesn't know the assignment.

How it works

SFT: minimize next-token loss on gold tokens. Data = demonstrations.

DPO: for each triple (x, y_w, y_l), increase likelihood of winner relative to loser vs a frozen reference model (usually the SFT checkpoint). No PPO loop.

RLHF: train reward model on preferences, then PPO to maximize reward with a KL penalty to stay near SFT. Labs still use variants at scale.

Watch verbosity: DPO can learn "longer = better" if winners were wordy. Length-control in eval.

Real-world example

SFT: 500 gold "claim summary" records with four headings. DPO: 300 pairs where the winner is terse and the loser dumps liability. After DPO, refusals and tone tighten without rewriting every gold label.

Technical explanation

Preference data is expensive. Synthetic pairs (LLM-as-judge) work if you audit them — judges have biases. Never DPO on raw "the model liked itself."

SFT on preference-style chat without a rejected side is still SFT. Mixing both losses in one run exists; conceptually keep the stages clear.

If you only have demonstrations, stop at SFT. Don't invent fake losers.

Common mistakes

Common mistake

Running DPO on a base model that never saw SFT for your task. Preferences without a stable imitation baseline wander.

Reward-hacking with a sloppy judge.
Skipping eval for sycophancy and length.

When to use it

SFT whenever you fine-tune. DPO when you can label pairwise quality or refusals. GRPO/RL when a program can score the answer.

When NOT to use it

No preference labels and SFT already hits the spec. Don't PPO for a JSON formatter.

Alternatives

Prompting and constitutions; RAG for facts; human HITL instead of baking refusals (sometimes safer to keep policy out of weights).

Quick quiz

Related concepts

LoRA and QLoRA — LoRA trains two small matrices instead of the frozen weight W; QLoRA also stores W in 4-bit so a 7B+ model fine-tunes on one GPU.
RAG vs Fine-tuning vs Long Context — RAG adds knowledge at answer time, fine-tuning bakes behavior into weights, and long context pastes everything into the prompt — each fits different problems.