From Models to Tools

Post-Training

In Module 1, we built the core loop of a language model: tokenize, embed, attend, predict. We saw that a base model generates fluent text, but it just completes whatever you give it. Ask it to classify a tweet, and it writes another tweet. It is not being uncooperative; it was trained to predict the next token, so that is exactly what it does.

Something happens between a base model and ChatGPT that turns a next-token predictor into something that answers questions, follows instructions, and refuses harmful requests. That something is post-training.

The Base-to-Assistant Gap

Give a base model the prompt “Classify this tweet as positive or negative: I love this weather! Answer:” and it might continue with another tweet, a news headline, or anything else that looks plausible as a continuation. An instruction-tuned model would respond with “positive” because it has learned, through post-training, that this kind of prompt expects a label.

Both models have the same knowledge: they were pre-trained on the same data. The difference is entirely in how they were taught to behave. Post-training is a pipeline of techniques that reshapes behaviour without fundamentally changing what the model knows.

Supervised Fine-Tuning (SFT)

The first stage: show the model curated (instruction, response) pairs and train it to imitate the responses. This is analogous to giving someone a style guide with worked examples: “when asked X, respond like Y.”

A concrete training example might look like this. Instruction: “Classify the following tweet as positive, negative, or neutral. Tweet: ‘Finally some sunshine after weeks of rain!’” Response: “Positive. The tweet expresses relief and happiness about an improvement in weather.” Thousands of such pairs, across diverse tasks and formats, teach the model what an “answer” looks like.

The scale involved is modest by pre-training standards. InstructGPT used roughly 13,000 demonstration examples for SFT—compare that to the hundreds of billions of tokens consumed during pre-training. SFT does not rewrite the model’s knowledge; it adjusts the surface layer of behaviour. The same cross-entropy loss used in pre-training drives learning, but now on curated examples instead of raw internet text.

After SFT, the model can follow instructions and produce formatted responses. But it has only a limited sense of which valid response is better than another. It has learned from high-quality demonstrations, but it cannot reliably distinguish between two on-topic responses that differ in subtle quality dimensions like depth, accuracy, or nuance.

For researchers, SFT also opens a door: if an off-the-shelf model performs poorly on your domain, you can fine-tune it on your own (instruction, response) pairs to improve task-specific performance. Module 3 covers when and how to pursue this.

Preference Optimization: Teaching Quality

The key idea: let humans define “quality” by comparing pairs of model outputs. For a given prompt, the model generates two responses. A human annotator decides which is better. This preference data is then used to train the model to favour responses that humans prefer.

Two main approaches exist. RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model on the preference data, then uses reinforcement learning to optimise the language model against that reward. DPO (Direct Preference Optimization) skips the reward model entirely: it directly optimises the language model on the preference pairs. DPO tends to produce comparable results with simpler training infrastructure, and is used by many recent open models including Llama 3 and Qwen 2.5.

In both approaches, the optimised model must stay close to the SFT model. Without this constraint, the model could discover degenerate strategies that exploit the preference signal: producing high-scoring but nonsensical text. The mathematical mechanism differs (a KL divergence penalty in RLHF, an implicit constraint in DPO), but the purpose is the same: keep the model useful while improving quality.

Safety Training & Constitutional AI

The final stage addresses safety: teaching the model when not to comply. Red-teaming involves human testers deliberately trying to elicit harmful outputs, and the model is then trained to refuse these inputs. Constitutional AI (Bai et al., 2022) takes a different approach: the model critiques its own outputs against a set of written principles (a “constitution”) and is trained to prefer the self-revised versions, reducing reliance on human annotators for safety feedback.

Safety training introduces an alignment tax: the model may become overly cautious, refusing benign requests that superficially resemble harmful ones. For researchers, this manifests as models declining to engage with sensitive topics even when the research purpose is legitimate.

What Post-Training Changes (And What It Doesn’t)

Post-training primarily reshapes behaviour rather than adding knowledge. If the base model has weak coverage of a domain, post-training alone is unlikely to fill that gap: the instruct version will just express its ignorance more politely, or (worse) confabulate an answer more convincingly.

For research: when a model gets a classification wrong, the problem is usually a capability limitation from pre-training or a prompting issue, not a post-training failure. Post-training determines the format (one-word answer vs. paragraph), the style (hedged vs. confident), and the boundaries (comply vs. refuse). It does not determine the accuracy.

But post-training also introduces systematic biases that matter for research. RLHF annotators encode their preferences into the reward model. The demographic composition of the annotator pool, their cultural context, and the guidelines they follow all shape what the model considers a “good” answer.

Resources

• Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback: the InstructGPT paper; foundational work on RLHF for language models.
• Rafailov et al. (2023), Direct Preference Optimization: eliminates the need for a separate reward model in preference learning.
• Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback: Anthropic’s approach to scalable safety alignment.
• Chip Huyen: “RLHF: Reinforcement Learning from Human Feedback”: accessible technical overview of the full pipeline.
• The Alignment Handbook (Hugging Face): practical guide to implementing alignment techniques.

Prompting

Post-training gave us a model that follows instructions. But which instructions? The quality of the prompt now determines the quality of the output, and here a new problem emerges: the same task, stated slightly differently, can produce substantially different results. For social scientists, this is not just a practical concern. A prompt is a measurement instrument: it defines what construct you measure, how reliably you measure it, and whether your results replicate.

Prompting as a Research Instrument

Consider the analogy to survey design. A survey question’s wording determines what respondents report. Leading questions produce biased responses. Ambiguous questions produce noisy responses. The same is true for prompts: a classification prompt that asks for “sentiment” measures something different from one that asks for “stance,” even when applied to the same text.

Just as survey methodology demands pre-testing, piloting, and reporting the exact question wording, prompt-based research should demand the same rigour. The prompt is the instrument. Report it fully, test its sensitivity, and validate it against known ground truth.

Anatomy of a Classification Prompt

A well-designed prompt has distinct structural components, each serving a specific function. The interactive diagram below breaks down a complete classification prompt. Click any highlighted region to see what it does, why it matters, and what happens if you leave it out.

The notebook exercises walk you through building prompts at each level of complexity: zero-shot (just a task description), few-shot (with labeled examples), and chain-of-thought (with reasoning steps). The interactive above shows the fully-specified version. The Minimal toggle shows what you lose when you strip it down.

Temperature & Sampling

Beyond the prompt itself, sampling parameters control how the model selects tokens from its predicted probability distribution. The most important parameter is temperature. Recall from Module 1 that the model produces logits which softmax converts into probabilities. Temperature modifies this step:

Drag the slider below and watch what happens to the distribution over candidate output tokens.

Two additional sampling parameters are common. Top-k sampling restricts the model to the k highest-probability tokens, zeroing out everything else before resampling. Top-p (nucleus) sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p (e.g., 0.9). Both methods prevent the model from selecting extremely unlikely tokens while preserving more diversity than greedy decoding.

For research classification tasks, set temperature to 0. Greedy decoding makes the output deterministic: the same prompt produces the same label every time, which is essential for reproducibility. Reserve higher temperatures for tasks where diversity matters, such as generating synthetic survey responses or brainstorming coding prompts.

Prompt Sensitivity

Here is the finding that should concern every social scientist using LLMs: small changes to prompt wording can produce large changes in classification results. Sclar et al. (2024) found that accuracy can swing by over 10 percentage points from semantically equivalent prompts. This is the LLM equivalent of question wording effects in survey research.

The visualisation below shows the same twelve tweets classified under five prompt variants. Notice which tweets are stable across all variants and which ones flip. Click any row to inspect the tweet and the failure pattern.

Structured Outputs

When building classification pipelines, the model’s response needs to be machine-readable. A response like “I would classify this as support because...” is useful for a human reader but breaks automated parsing. Structured output formats constrain the model to produce responses in a predictable schema.

Consider the difference. A free-text response might say: “This tweet expresses opposition to the policy, primarily through sarcastic framing. I would classify it as oppose.” A structured JSON response to the same prompt might return:

{
  "label": "oppose",
  "confidence": "high",
  "reasoning": "Sarcastic framing ('really changed the world') signals opposition."
}

The first version requires custom parsing that will inevitably break on edge cases. The second can be processed with a single line of code across 10,000 documents.

The simplest approach is to instruct the model to respond with a single word or a JSON object. Most API providers also offer JSON mode, which guarantees valid JSON output at the decoding level, and function calling (or “tool use”), which constrains the output to match a predefined schema. These work by modifying the token sampling process: at each generation step, tokens that would produce invalid JSON are masked out, ensuring the output always conforms to the requested structure. This eliminates the parsing failures that plague free-text classification at scale.

For research pipelines, structured outputs serve two purposes. First, reliability: when processing 10,000 documents, you cannot afford to manually fix malformed responses. Second, auditability: a JSON response with separate fields for “label” and “reasoning” lets you log both the classification and the model’s rationale, enabling systematic error analysis. Module 3 covers the full pipeline from structured outputs through batching and cost management.

Resources

• Brown et al. (2020), Language Models are Few-Shot Learners: the GPT-3 paper; establishes in-context learning as a paradigm.
• Sclar et al. (2024), Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design: systematic study of prompt sensitivity effects.
• Anthropic Prompt Engineering Guide: practical, detailed, regularly updated.
• OpenAI Prompt Engineering Guide: complementary perspective on prompt design.
• DSPy (Stanford NLP): framework for programmatic prompt optimization.

Reasoning

Even a well-designed prompt has limits. Standard prompting works well for clear-cut classification, but sarcastic tweets, implicit stance, ambiguous framing, or tasks that involve multiple logical steps often defeat direct prompts. The model produces a label in a single forward pass without any intermediate reasoning—and for hard cases, that is not enough.

In Module 1, we saw that model performance follows power laws with training compute, data, and parameters. But there is a second axis of scaling that operates at inference rather than training: letting the model think longer before answering. Chain-of-thought prompting and dedicated reasoning models exploit this axis, trading output tokens and latency for accuracy on tasks that require multi-step inference, disambiguation, or nuanced judgment.

Chain-of-Thought Prompting

Wei et al. (2022) showed that adding the phrase “think step by step” or providing worked examples with explicit reasoning dramatically improves performance on tasks requiring multi-step inference. Instead of asking for a direct answer, ask the model to show its work.

Why does this work? The intermediate tokens serve as a form of working memory. The model can “store” partial results in the generated text and attend back to them when producing the final answer. Without CoT, the model must compress all reasoning into a single forward pass. Kojima et al. (2022) showed that even simply appending “Let’s think step by step” without any examples can elicit this behaviour.

Reasoning Models & Test-Time Compute

A recent class of models takes chain-of-thought further: instead of relying on the user to prompt for reasoning, the model is trained to reason before answering. These include OpenAI’s o1 and o3, DeepSeek-R1 (open-weight, where reasoning emerged from reinforcement learning without explicit instruction), and Claude with extended thinking.

This represents a shift in how we think about model capability. In Module 1, we saw that performance follows power laws with training compute, data, and parameters. Reasoning models introduce test-time compute scaling: instead of making the model bigger, let it think longer. A smaller model that reasons for 30 seconds can sometimes match a larger model that answers instantly, at the cost of higher latency and more output tokens.

When to Use Reasoning: The Cost Trade-off

Reasoning is not always worth the cost. The gains are largest on ambiguous, multi-step tasks and negligible on clear-cut classification. The interactive below lets you explore the trade-off at different corpus sizes.

Resources

• Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: the foundational CoT paper.
• Kojima et al. (2022), Large Language Models are Zero-Shot Reasoners: the “Let’s think step by step” paper.
• DeepSeek-AI (2025), DeepSeek-R1: open-source reasoning model showing reasoning emerging from RL.
• Anthropic: Extended Thinking documentation: practical guide to using Claude’s reasoning mode.
• Lightman et al. (2023), Let’s Verify Step by Step: process reward models that evaluate each reasoning step.

Evaluating & Choosing Models

Post-training determines how a model behaves, prompting determines what you measure, and reasoning techniques extend accuracy on hard cases. But all of this assumes you have already chosen a model. Which one? The landscape includes hundreds of options across a wide range of capabilities, costs, and transparency properties. The choice is not just technical: it is a research design decision with consequences for reproducibility, data privacy, and budget—and unlike a survey instrument, the model you chose might silently change between your pilot and your final data collection.

Benchmarks: Useful but Insufficient

The field evaluates models using standardised benchmarks. MMLU (Hendrycks et al., 2021) tests broad knowledge across 57 subjects. HumanEval measures code generation. GPQA (Rein et al., 2023) tests graduate-level reasoning. Chatbot Arena (Chiang et al., 2024) ranks models by blind human preference on real conversations.

Think of benchmarks like standardised tests (GRE, LSAT): they correlate with something real, but they do not measure what you specifically care about. Three fundamental problems undermine them. Contamination: models may have seen benchmark questions during training, inflating scores. Gaming: providers can optimise specifically for benchmark performance without improving general capability. Construct validity: no standard benchmark measures your task. A high MMLU score does not mean the model will classify your political texts well. The benchmark that matters most is the one you build yourself.

Open vs. Closed Models

The choice between open-weight and closed (proprietary API) models involves trade-offs across five dimensions that map directly onto research design concerns.

Capability: frontier closed models currently lead on the hardest tasks, though the gap has narrowed substantially. For many social science classification tasks, the difference is small enough that other dimensions dominate the decision.

Cost: open models are free to use (you pay only for compute). At scale, self-hosting can be dramatically cheaper per token—a difference that matters when processing millions of documents.

Transparency: open models give you full access to weights and architecture. Closed models are opaque: you cannot inspect what training data was used, what safety filters are applied, or how the model was post-trained.

Data privacy: with open models, your data never leaves your infrastructure. This may be essential under IRB protocols, GDPR requirements, or when working with sensitive political, health, or legal data.

Reproducibility: this is the dimension researchers most often underestimate. Open model weights are fixed: a snapshot you download today produces the same outputs in two years. Closed models can change without notice. If you submit a paper in June using GPT-4 and reviewers try to replicate your results in December, the model behind the API may have been updated, retrained, or replaced entirely. Chen et al. (2023) documented substantial behaviour drift in GPT-4 and GPT-3.5 over just a few months, with accuracy on specific tasks changing by 10+ percentage points between API snapshots.

The term “open” itself exists on a spectrum. Fully open models release weights, training code, and training data (e.g., OLMo). Open-weight models release weights but not training data or code (e.g., Llama, Mistral). API-only models provide no access to internals (e.g., GPT-4, Claude). For reproducibility, any model whose weights you can download is a significant improvement over API-only access, even if the training data remains proprietary.

Cost-Capability Frontiers

The key insight for researchers: you rarely need the most capable model. If a 70B open model achieves 92% accuracy on your task and the frontier closed model achieves 95%, the open model may be the better choice when processing millions of documents, especially if data privacy is a concern. The frontier shifts rapidly: check current comparisons (e.g., Artificial Analysis) before committing to a model for a large project.

Building Task-Specific Evaluations

Because no standard benchmark measures your task, you need to build your own evaluation. The process follows the same logic as validating any measurement instrument:

1. Create a gold-standard set. Manually label a subset of your data (50–200 items is often sufficient for initial evaluation). Use multiple coders and compute inter-annotator agreement.

2. Pilot multiple models. Run your classification prompt on the gold-standard set with 2–4 candidate models. Compare accuracy, F1, and, crucially, patterns of disagreement with your human coders.

3. Assess failure modes. Where does the model disagree with humans? Are the errors random, or systematic? Systematic errors (e.g., all sarcastic tweets misclassified) suggest a model-task mismatch; random errors may be tolerable.

4. Document and iterate. Record the model version, prompt, temperature setting, and evaluation metrics. These are your instrument specifications: they belong in the methods section of your paper.

Step 2 mentions F1. For classification tasks, accuracy alone is misleading when classes are imbalanced. If 95% of your tweets are neutral, a model that labels everything “neutral” achieves 95% accuracy while being completely useless. Precision, recall, and F1 give a more honest picture:

For multi-class tasks, report macro-averaged F1 (average F1 across all classes, giving equal weight to rare categories) or per-class F1. When comparing model labels to human labels, treat the comparison as an inter-annotator agreement problem: Cohen’s kappa or Krippendorff’s alpha account for agreement that would occur by chance, giving a more rigorous measure than raw accuracy. Module 3 walks through computing these metrics on your own classification results.

Resources

• Hendrycks et al. (2021), Measuring Massive Multitask Language Understanding: introduces MMLU, the most widely cited LLM benchmark.
• Chiang et al. (2024), Chatbot Arena: Elo-based rankings from human comparisons.
• Rein et al. (2023), GPQA: expert-level questions resistant to memorisation.
• Chatbot Arena Leaderboard: live updated model rankings.
• Artificial Analysis: cost, speed, and quality comparisons across providers.
• Chen et al. (2023), How is ChatGPT’s behavior changing over time?: documents significant performance drift in closed API models over months.