From Models to Tools

Post-Training

In Day 1, we built the core loop of a language model: tokenize, embed, attend, predict. We saw that a base model generates fluent text: but it just completes whatever you give it. Ask it to classify a tweet, and it writes another tweet. It is not being uncooperative; it was trained to predict the next token, so that is exactly what it does.

Something happens between a base model and ChatGPT that turns a next-token predictor into something that answers questions, follows instructions, and refuses harmful requests. That something is post-training.

The Base-to-Assistant Gap

Give a base model the prompt “Classify this tweet as positive or negative: I love this weather! Answer:” and it might continue with another tweet, a news headline, or anything else that looks plausible as a continuation. An instruction-tuned model would respond with “positive” because it has learned, through post-training, that this kind of prompt expects a label.

Both models have the same knowledge: they were pre-trained on the same data. The difference is entirely in how they were taught to behave. Post-training is a pipeline of techniques that reshapes behaviour without fundamentally changing what the model knows.

Supervised Fine-Tuning (SFT)

The first stage: show the model curated (instruction, response) pairs: from as few as hundreds to hundreds of thousands: and train it to imitate the responses. This is analogous to giving someone a style guide with worked examples: “when asked X, respond like Y.”

After SFT, the model can follow instructions and produce formatted responses. But it has only a limited sense of which valid response is better than another: it has learned from high-quality demonstrations, but it cannot reliably distinguish between two on-topic responses that differ in subtle quality dimensions like depth, accuracy, or nuance. Something more is needed to teach fine-grained quality discrimination.

Reward Modeling & RLHF

The key idea behind Reinforcement Learning from Human Feedback (RLHF): let humans define “quality” by comparing pairs of model outputs. For a given prompt, the model generates two responses. A human annotator decides which is better. This preference data is then used to train a reward model: a separate neural network that predicts how much a human would prefer a given response.

Once the reward model is trained, the language model is optimised using Proximal Policy Optimization (PPO) to produce responses that score highly. But there is a critical constraint: the optimised model must not stray too far from the SFT model. Without this constraint, the model could discover degenerate strategies that exploit the reward model: producing high-scoring but nonsensical text. The KL divergence penalty prevents this.

The RLHF framework was developed by Christiano et al. (2017) for general reinforcement learning and adapted for language models by Ouyang et al. (2022) in the InstructGPT paper. The core innovation was showing that relatively modest amounts of human preference data: tens of thousands of comparisons, not millions: could dramatically improve model behaviour.

Direct Preference Optimization (DPO)

Rafailov et al. (2023) showed that the separate reward model in RLHF is not strictly necessary. Their key insight: the language model itself can serve as an implicit reward model. DPO directly optimises the model on preference pairs without training a separate reward model or running an RL loop.

Notation note: the DPO loss writes the policy as π_θ to emphasise that θ are the parameters being optimised: the same policy written as π in the RLHF objective above.

In practice, DPO tends to produce results comparable to RLHF with substantially simpler training infrastructure: no reward model to maintain, no RL training loop to stabilise. Many recent open models, including Llama 3 and Qwen 2.5, use DPO or its variants for preference alignment.

Safety Training & Constitutional AI

The final stage of post-training addresses safety: teaching the model when not to comply.

Red-teaming: human testers deliberately try to elicit harmful outputs (instructions for violence, discriminatory content, private information). The model is then trained to refuse these inputs.

Constitutional AI (Bai et al., 2022): the model critiques its own outputs against a set of written principles (a “constitution”) and is trained to prefer the self-revised versions. This reduces reliance on human annotators for safety-related feedback.

Safety training introduces an alignment tax: the model may become overly cautious, refusing benign requests that superficially resemble harmful ones. For researchers, this manifests as models declining to engage with sensitive topics: even when the research purpose is legitimate. Understanding this trade-off helps diagnose unexpected refusals in research workflows.

What Post-Training Changes (And What It Doesn’t)

Post-training primarily reshapes behaviour rather than adding knowledge. The vast majority of the model’s factual knowledge, language capabilities, and reasoning patterns come from pre-training. If the base model has weak coverage of a domain, post-training alone is unlikely to fill that gap: the instruct version will just express its ignorance more politely, or (worse) confabulate an answer more convincingly.

For research: when a model gets a classification wrong, the problem is usually a capability limitation from pre-training or a prompting issue, not a post-training failure. Post-training determines the format (one-word answer vs. paragraph), the style (hedged vs. confident), and the boundaries (comply vs. refuse). It does not determine the accuracy.

But post-training also introduces systematic biases that matter for research. RLHF annotators encode their preferences into the reward model. The demographic composition of the annotator pool, their cultural context, and the guidelines they follow all shape what the model considers a “good” answer. We return to this in the Social Science Applications section.

Resources

• Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback: the InstructGPT paper; foundational work on RLHF for language models.
• Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences: the original RLHF framework that InstructGPT builds upon.
• Rafailov et al. (2023), Direct Preference Optimization: eliminates the need for a separate reward model in preference learning.
• Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback: Anthropic’s approach to scalable safety alignment.
• Chip Huyen: “RLHF: Reinforcement Learning from Human Feedback”: accessible technical overview of the full pipeline.
• The Alignment Handbook (Hugging Face): practical guide to implementing alignment techniques.

Prompting

Post-training gave us a model that follows instructions. The quality of those instructions: the prompt: now determines the quality of the output. For social scientists, this is not just a practical concern. A prompt is a measurement instrument: it defines what construct you measure, how reliably you measure it, and whether your results replicate.

Prompting as a Research Instrument

Consider the analogy to survey design. A survey question’s wording determines what respondents report. Leading questions produce biased responses. Ambiguous questions produce noisy responses. The same is true for prompts: a classification prompt that asks for “sentiment” measures something different from one that asks for “stance,” even when applied to the same text.

Just as survey methodology demands pre-testing, piloting, and reporting the exact question wording, prompt-based research should demand the same rigour. The prompt is the instrument. Report it fully, test its sensitivity, and validate it against known ground truth.

Zero-Shot Prompting

The simplest approach: describe the task and provide the input, with no examples. The model relies entirely on its pre-trained knowledge and post-training to interpret the request.

Zero-shot prompting works well when the task is unambiguous and aligns with the model’s training distribution: standard sentiment analysis, language identification, simple factual questions. It tends to struggle with tasks that require a specific interpretation of categories, domain conventions, or nuanced distinctions (such as the difference between stance and sentiment).

Few-Shot Prompting & In-Context Learning

A striking property of large language models: they can learn new tasks from a handful of examples provided in the prompt itself. This is in-context learning, demonstrated at scale by Brown et al. (2020) in the GPT-3 paper.

In few-shot prompting, you include labeled examples before the target input. The model infers the classification pattern from these examples and applies it to the new input. This is not fine-tuning: no weights are updated. The model processes the examples as part of its context and adapts its behaviour accordingly.

Key practical considerations for few-shot prompting: balance the examples across classes (do not provide five “support” examples and one “oppose” example). Shuffle the order : models can be sensitive to which class appears last. And keep examples representative of the cases the model will encounter, not just easy ones.

System Prompts & Role Specification

Most API-based models accept a system prompt that frames the model’s identity and behavioural constraints before the user’s message. System prompts can set context (“You are a political science research assistant”), define output format (“Always respond with valid JSON”), and establish boundaries (“Do not provide your own opinion on the topic”).

For research applications, well-crafted system prompts can improve consistency by anchoring the model’s behaviour across many classifications. However, system prompts are not a guarantee : sufficiently unusual inputs can override them. They are best thought of as strong defaults, not hard constraints.

Structured Outputs

When building classification pipelines, you need machine-parseable output: not free-form text. Structured output means asking the model to respond in a specific format: JSON, XML, or a constrained schema.

Larger models comply with formatting instructions more reliably. Smaller models (under ~10B parameters) often produce nearly-valid JSON with small errors: a missing comma, an extra field. Modern APIs from major providers now offer guaranteed structured output modes that constrain the model’s generation to valid JSON conforming to a provided schema, eliminating parsing failures at the cost of slightly constrained generation.

Temperature & Sampling Parameters

Beyond the prompt itself, sampling parameters control how the model selects tokens from its predicted probability distribution. These settings directly affect output variability and reproducibility.

A related parameter, top-p (nucleus sampling), restricts token selection to the smallest set whose cumulative probability exceeds a threshold (e.g., 0.95). In practice, most research workflows set temperature to 0 and leave top-p at its default. The key point: always report your sampling parameters. A classification study run at temperature 0 and the same study run at temperature 0.7 can produce meaningfully different results, and failing to report this makes replication impossible.

Prompt Sensitivity & Reproducibility

Here is the finding that should concern every social scientist using LLMs: small changes to prompt wording can produce large changes in classification results.

Sclar et al. (2024) systematically tested how minor prompt variations: rephrasing the instruction, changing label names, reordering components: affect model outputs. They found that accuracy can swing by over 10 percentage points from seemingly equivalent prompts. This is the LLM equivalent of question wording effects in survey research.

Implications for research: if your results depend on how you phrase the prompt, you have a reproducibility problem. Best practice is to test multiple prompt variants, report the range of results, and select the prompt based on validation against a gold-standard set: not based on which variant happens to produce the best-looking numbers.

The interactive explorer below compares how the same tweet is classified under different prompting strategies. The third tab previews chain-of-thought prompting, which we cover in detail in the next section.

Resources

• Brown et al. (2020), Language Models are Few-Shot Learners: the GPT-3 paper; establishes in-context learning as a paradigm.
• Sclar et al. (2024), Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design: systematic study of prompt sensitivity effects.
• Anthropic Prompt Engineering Guide: practical, detailed, regularly updated.
• OpenAI Prompt Engineering Guide: complementary perspective on prompt design.
• Prompt Engineering Guide: comprehensive community-maintained resource covering all major techniques.
• DSPy (Stanford NLP): framework for programmatic prompt optimization.

Reasoning

Standard prompting: whether zero-shot or few-shot: works well for many tasks. But some cases require more: sarcastic tweets, implicit stance, ambiguous framing, or tasks that involve multiple logical steps. For these, asking the model to reason before answering can substantially improve accuracy.

This section traces the evolution from a simple prompting technique (chain-of-thought) to a new class of models specifically trained to reason: and connects both to the scaling laws from Day 1.

Chain-of-Thought Prompting

Wei et al. (2022) showed that adding the phrase “think step by step” or providing worked examples with explicit reasoning dramatically improves performance on tasks requiring multi-step inference. The idea is simple: instead of asking for a direct answer, ask the model to show its work.

Why does this work? One hypothesis: the intermediate tokens generated during reasoning serve as a form of working memory. The model can “store” partial results in the generated text and attend back to them when producing the final answer. Without CoT, the model must compress all reasoning into the implicit computation within a single forward pass: a much harder task.

Zero-Shot Chain-of-Thought

Kojima et al. (2022) discovered that simply appending “Let’s think step by step” to a prompt: without any examples: is often enough to elicit reasoning. This zero-shot CoT approach is remarkably effective for its simplicity: it requires no example engineering and adds minimal tokens to the prompt.

The trade-off is control. With few-shot CoT, you guide the model’s reasoning by showing what good reasoning looks like. With zero-shot CoT, the model decides its own reasoning strategy, which can be inconsistent across inputs.

Reasoning Models

A recent class of models takes the chain-of-thought idea further: instead of relying on the user to prompt for reasoning, the model is trained to reason before answering. These models generate an internal “thinking” process: sometimes visible to the user, sometimes hidden: before producing a final response.

Key examples include OpenAI’s o1 and o3 models, DeepSeek-R1 (open-weight, trained with reinforcement learning to develop reasoning strategies), and Claude with extended thinking. What makes these models notable is how reasoning emerges: DeepSeek-R1 showed that training a model with reinforcement learning : rewarding correct answers without specifying how to reason : can cause the model to independently develop chain-of-thought-like strategies.

Test-Time Compute Scaling

In Day 1, we saw that model performance follows power laws with respect to training compute, data, and parameters. Reasoning models introduce a different kind of scaling: test-time compute. Instead of making the model bigger or training it longer, you let it think longer on each input.

This represents a shift in how we think about model capability. Traditional scaling asked: how big should the model be? Test-time scaling asks: how long should the model think? The two are complementary. A smaller model that thinks for 30 seconds can sometimes match a larger model that answers instantly: at the cost of higher latency and more output tokens.

When to Use Reasoning

Reasoning: whether through CoT prompting or dedicated reasoning models: is not always worth the cost. The gains are largest on tasks that involve:

Ambiguity: sarcasm, irony, implicit stance, or texts where the surface meaning differs from the intended meaning. Multi-step inference: tasks requiring combination of multiple pieces of information. Complex categorisation: classification schemes with many categories or fine-grained distinctions.

For simple binary classification on clear-cut texts, standard prompting is usually sufficient and far cheaper. Running a reasoning model on 10,000 straightforward tweets is wasteful: the model will “think” about each one and reach the same answer it would have given instantly, at 10× the cost and latency.

Resources

• Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: the foundational CoT paper.
• Kojima et al. (2022), Large Language Models are Zero-Shot Reasoners: the “Let’s think step by step” paper.
• DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning: open-source reasoning model showing reasoning emerging from RL.
• OpenAI: “Learning to Reason with LLMs”: o1 announcement and technical context.
• Anthropic: Extended Thinking documentation: practical guide to using Claude’s reasoning mode.
• Lightman et al. (2023), Let’s Verify Step by Step: process reward models that evaluate each reasoning step.

Evaluating & Choosing Models

You now know how to use a model: post-training made it follow instructions, prompting lets you frame tasks precisely, and reasoning techniques handle harder cases. But which model? The landscape includes hundreds of options: open and closed, small and large, general and specialised: and the right choice depends on your specific research context.

Perplexity Revisited

In Day 1, we introduced perplexity as a measure of how well a language model predicts text.

Perplexity is the standard training metric for language models, and lower perplexity generally correlates with better text generation within a model family. But for downstream tasks: classification, information extraction, summarisation: perplexity is a poor predictor of performance. A model with lower perplexity is not necessarily better at classifying your political texts. The connection between language modelling ability and task-specific accuracy is mediated by model architecture, training data composition, post-training choices, and the nature of the task.

Perplexity is useful for comparing variants of the same model (e.g., different checkpoints during training) but unreliable for comparing across model families, architectures, or sizes. Treat it as necessary but insufficient.

The Benchmark Landscape

The field evaluates models using standardised benchmarks. Each captures a different dimension of capability:

MMLU (Hendrycks et al., 2021): 57-subject multiple-choice test covering STEM, humanities, and social sciences. The most widely cited general benchmark. Scores above ~90% are now common among frontier models, leading to concerns about ceiling effects.

HumanEval: code generation benchmark testing whether models can write correct Python functions from docstrings. Important for research workflows that involve code generation, but narrow in scope.

GPQA (Rein et al., 2023): graduate-level science questions designed to be difficult even for domain experts. Intended to measure deep reasoning, but the test set is small, making scores noisy.

Chatbot Arena (Chiang et al., 2024): human preference rankings from blind comparisons on real conversations. Arguably the most ecologically valid benchmark, since it reflects how users actually experience model quality. But it measures general helpfulness, not research-specific capability.

Benchmark Limitations

Every benchmark has fundamental limitations. Three deserve particular attention:

Contamination is endemic. Benchmark questions leak into training data, and models may have effectively memorised answers rather than demonstrating generalizable reasoning on novel problems. Newer benchmarks attempt to mitigate this with dynamic question sets, but the arms race between benchmark creators and training data curators is ongoing.

Gaming: model providers can optimise specifically for benchmark performance: through targeted fine-tuning, example selection, or architectural choices: without improving general capability. A model that scores well on MMLU is not guaranteed to handle your specific annotation task well.

Construct validity: this is the most important limitation for social scientists. No standard benchmark measures your task. MMLU tests broad knowledge; your research might require nuanced understanding of political rhetoric. Chatbot Arena tests conversational helpfulness; your research might require precise, consistent annotation. The benchmark that matters most is the one you build yourself.

Open vs. Closed Models

The choice between open-weight and closed (proprietary API) models involves real trade-offs across five dimensions:

Capability: frontier closed models (GPT-4o, Claude 3.5 Sonnet, Gemini) currently tend to outperform open models on the hardest tasks, though the gap has narrowed substantially. For many classification and extraction tasks, the best open models (Llama 3, Qwen 2.5, DeepSeek) perform comparably.

Cost: open models are free to use (you pay only for compute), while closed APIs charge per token. For large-scale annotation projects, the cost difference can be orders of magnitude. Self-hosting an open model on a GPU cluster is capital-intensive but can be dramatically cheaper per token at scale.

Transparency: open models give you full access to weights, architecture details, and (sometimes) training data documentation. Closed models are opaque: you cannot inspect why they produce a particular output. For research that requires understanding the instrument, transparency matters.

Data privacy: with open models, your data never leaves your infrastructure. With closed APIs, your data is sent to the provider’s servers. For sensitive data: survey responses, medical records, classified documents: this may be unacceptable under IRB protocols or data protection regulations.

Reproducibility: open model weights are fixed. You can pin a specific version and guarantee identical outputs indefinitely. Closed models can change without notice: the provider may update the model between when you run your study and when reviewers try to replicate it. For scientific work, this is a significant concern.

Cost-Capability Frontiers

The key insight for researchers: you rarely need the most capable model. If a 70B open model achieves 92% accuracy on your task and the frontier closed model achieves 95%, the open model may be the better choice for a project that requires processing millions of documents : especially if data privacy is a concern.

The cost-capability frontier shifts rapidly. A model that was frontier-quality six months ago may be outperformed by one that costs a fraction of the price today. Check current comparisons (e.g., Artificial Analysis) before committing to a model for a large project.

Building Task-Specific Evaluations

Because no standard benchmark measures your task, you need to build your own evaluation. The process follows the same logic as validating any measurement instrument:

1. Create a gold-standard set. Manually label a subset of your data (50–200 items is often sufficient for initial evaluation). Use multiple coders and compute inter-annotator agreement.

2. Pilot multiple models. Run your classification prompt on the gold-standard set with 2–4 candidate models. Compare accuracy, F1, and: crucially: patterns of disagreement with your human coders.

3. Assess failure modes. Where does the model disagree with humans? Are the errors random, or systematic? Systematic errors suggest a model-task mismatch; random errors may be tolerable.

4. Document and iterate. Record the model version, prompt, temperature setting, and evaluation metrics. These are your instrument specifications: they belong in the methods section of your paper.

Resources

• Hendrycks et al. (2021), Measuring Massive Multitask Language Understanding: introduces MMLU, the most widely cited LLM benchmark.
• Chiang et al. (2024), Chatbot Arena: An Open Platform for Evaluating LLMs through Human Preference: Elo-based rankings from human comparisons.
• Rein et al. (2023), GPQA: A Graduate-Level Google-Proof Q&A Benchmark: expert-level questions resistant to memorisation.
• Chatbot Arena Leaderboard: live updated model rankings based on human preference.
• Open LLM Leaderboard (Hugging Face): open model comparison across benchmarks.
• Artificial Analysis: cost, speed, and quality comparisons across providers and models.
• HELM (Stanford CRFM): holistic evaluation covering accuracy, calibration, fairness, and robustness.

Social Science Applications

The tools covered in this module: alignment, prompting, reasoning, evaluation: are already transforming social science methodology. This section highlights key papers that demonstrate how these techniques open new research possibilities and introduce new methodological challenges.

Alignment & Opinion Representation

Post-training does not just affect model style: it shapes what perspectives models represent. Understanding this is essential for any research that uses model outputs as data.

Prompting as Experimental Design

The prompting techniques covered in this module are not just practical tools: they are methodological choices with direct implications for research validity.

Reasoning & Analytical Complexity

Evaluation as Research Design