Deploying for Research

Fine-Tuning

Day 2 showed how prompting and reasoning techniques turn a language model into a classification instrument. For many tasks, this works well. But prompting has a ceiling: when your classification scheme is domain-specific, when the model makes systematic errors that better examples cannot fix, or when per-token costs make API-based prompting prohibitive at scale : fine-tuning adapts a model’s weights to your specific task.

The Decision Framework: Prompt, RAG, or Fine-Tune

Before investing in fine-tuning, ask whether it is necessary. Three approaches exist for improving model performance, and each fits different circumstances.

Prompting (covered in Day 2) is the right starting point when zero-shot or few-shot performance meets your validation threshold, when you need flexibility across multiple tasks, when you have little or no labeled data, or when the task aligns well with the model’s pre-training distribution.

Retrieval-Augmented Generation (RAG) is the right choice when the model needs domain knowledge that exceeds its training data or its context window: your corpus of parliamentary records, legal texts, or historical archives. RAG uses embeddings (from Day 1) to retrieve relevant documents and feed them to the model alongside the query. We cover RAG in Day 4.

Fine-tuning is the right choice when prompting produces systematic errors that more or better examples cannot resolve, when you need consistent output formatting across thousands of classifications, when you have labeled data (hundreds to thousands of examples), or when domain-specific language or conventions are not handled well by general-purpose models.

What Fine-Tuning Does

Fine-tuning continues training a pre-trained model on your labeled data. The model’s weights are adjusted to minimise prediction error on your examples. This is the same gradient-based optimisation from pre-training, but applied to a much smaller, task-specific dataset.

For decoder models (GPT, Llama, Qwen), fine-tuning typically uses Supervised Fine-Tuning (SFT): you provide (instruction, response) pairs: the same format you used for prompting in Day 2: and the model learns to produce those responses. The critical insight: the training data is just prompt–response pairs. The difference from few-shot prompting is that instead of the model seeing your examples at inference time, it internalises the patterns during training.

LoRA: Parameter-Efficient Fine-Tuning

Full fine-tuning updates every parameter in the model. For a 7B-parameter model, this means updating seven billion numbers : requiring multiple high-end GPUs and risking catastrophic forgetting, where the model loses general capabilities while learning your task.

The key observation behind LoRA: weight updates during fine-tuning tend to occupy a low-dimensional subspace. They do not need the full dimensionality of the weight matrix. LoRA exploits this by decomposing the update into two small matrices whose product approximates the full-rank update.

QLoRA (Dettmers et al., 2023) takes this further: it loads the frozen base model in 4-bit precision (NormalFloat4 quantisation), reducing memory by roughly 75% compared to 16-bit storage. The LoRA adapters themselves remain in full precision. This combination means a 7B model fits on a single consumer GPU with 16GB of VRAM, and even a 3B model can be fine-tuned on a free Google Colab T4 (15GB).

The output of LoRA fine-tuning is a small adapter file: typically tens of megabytes: that can be shared, versioned, and loaded on top of the base model. Multiple adapters can be trained for different tasks and swapped without reloading the base model.

Encoder Models for Classification

Everything covered so far: GPT, Claude, Llama, Qwen : uses a decoder-only architecture. These models read text left to right, generating one token at a time. They are designed for generation: writing text, following instructions, answering questions. But for pure classification : assigning a fixed label to a piece of text: a different architecture is often the better choice.

Encoder models (BERT, RoBERTa, DeBERTa) read the entire input simultaneously. Every token attends to every other token in both directions: forward and backward. In Day 1, we described how decoder models use causal masking so each token can only attend to preceding tokens. Encoder models remove this mask entirely: the representation of any word is informed by everything before and after it.

The classification mechanism is straightforward. The encoder processes the input and produces a contextualised vector for each token. A special [CLS] token: prepended to every input : aggregates information from the full sequence. A single linear layer maps this vector to a probability distribution over the label set.

The Encoder Lineage: BERT → RoBERTa → DeBERTa

BERT (Devlin et al., 2019) established the encoder paradigm. It introduced masked language modelling: randomly mask 15% of tokens and train the model to predict them from the surrounding context. This bidirectional pre-training produced representations that dramatically outperformed previous methods on classification, question answering, and other understanding tasks. BERT was, for several years, the standard tool for text classification in NLP research and social science.

RoBERTa (Liu et al., 2019) showed that BERT was substantially undertrained. By training longer, on more data, with larger batches, and with a refined masking strategy, RoBERTa achieved considerable improvements without changing the architecture. The lesson: training procedure matters as much as architecture.

DeBERTa (He et al., 2021) introduced two architectural innovations. Disentangled attention separates content and position into independent vectors, allowing the model to compute content-to-content, content-to-position, and position-to-content attention scores separately. This finer-grained attention captures more nuanced relationships. The enhanced mask decoder incorporates absolute position information at the prediction stage, addressing an ambiguity limitation in BERT’s approach.

DeBERTaV3 (He et al., 2023) replaced masked language modelling with ELECTRA-style pre-training, a more sample-efficient approach. The result: deberta-v3-base (184M parameters) and deberta-v3-small (44M parameters) currently represent the best general-purpose encoder models for classification. A 2025 comparison found that DeBERTaV3 remains superior to the newer ModernBERT in sample efficiency and overall benchmark performance, though ModernBERT offers advantages in long-context support.

Encoder vs. Decoder: When to Use Each

For social science researchers, the practical guidance is straightforward:

Use an encoder model when your task is classification with a fixed set of categories, you have labeled training data (hundreds to thousands of examples), you need fast inference (thousands of texts per minute), and you do not need the model to generate free text. For most annotation tasks in computational social science : sentiment analysis, stance detection, topic labeling, framing analysis: a fine-tuned DeBERTa is likely the best choice in terms of accuracy per compute dollar.

Use a decoder model (via LoRA or prompting) when you need the model to generate text (summaries, explanations, structured extractions), when your task requires flexible instructions, when you are working with multiple tasks simultaneously, or when you have no labeled training data and must rely on zero/few-shot prompting.

Recent comparative work supports this distinction. Widmann & Wich (2023) found that fine-tuned encoder models (BERT-family) remain state-of-the-art for many classification tasks, sometimes outperforming much larger prompted decoder models. A 2024 analysis in Language Resources and Evaluation confirmed that encoder-only architectures generally provide better efficiency-to-performance ratios for discriminative classification, while noting that fully fine-tuned large decoders can occasionally match or exceed encoder performance when sufficient data and compute are available.

Practical Considerations

Data requirements: For encoder fine-tuning, a few hundred labeled examples often produce strong results; a few thousand approach diminishing returns. For LoRA fine-tuning of decoders, similar quantities work, though the per-example signal may be weaker because the model must also learn the output format.

Hyperparameters: For encoders, the most critical parameter is learning rate: typically 2×10^-5 to 5×10^-5, with 2–4 training epochs and batch sizes of 16–32. For LoRA, common settings include rank 8–16, lora_alpha 16–32, and dropout 0.05–0.1. These are well-established defaults; the notebook walks through specific configurations.

Overfitting: With small datasets, the model can memorise training examples rather than learning generalisable patterns. Monitor validation loss during training: if it starts increasing while training loss continues to decrease, the model is overfitting. Use early stopping to halt training at the best validation checkpoint.

Evaluation: Always evaluate on a held-out test set that the model has never seen during training or validation. Report per-class precision, recall, and F1: not just overall accuracy. A model with 90% accuracy may achieve 95% on the majority class and 60% on the minority class, which is unacceptable for most research purposes.

Resources

• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models: the foundational LoRA paper; efficient fine-tuning by learning low-rank weight updates.
• Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized Language Models: 4-bit quantisation enabling fine-tuning on consumer GPUs.
• Devlin et al. (2019), BERT: Pre-training of Deep Bidirectional Transformers: the paper that established encoder-based NLP.
• He et al. (2021), DeBERTa: Decoding-enhanced BERT with Disentangled Attention: disentangled attention for better classification.
• He et al. (2023), DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training: the current best general-purpose encoder model.
• Liu et al. (2019), RoBERTa: A Robustly Optimized BERT Pretraining Approach: showed that BERT’s training was substantially suboptimal.
• Hugging Face PEFT library: LoRA, QLoRA, and other parameter-efficient methods.
• Hugging Face Transformers: step-by-step fine-tuning tutorials for both encoder and decoder models.
• Unsloth: 2× faster LoRA fine-tuning with lower memory usage.

Working at Scale

You have chosen your approach: prompted classification, a fine-tuned decoder, or a fine-tuned encoder. The next challenge is infrastructure: how to process thousands or millions of texts efficiently, reliably, and within budget.

Running a model interactively in a chat interface works for exploration. Moving to research-scale annotation requires programmatic access, structured outputs, error handling, and cost planning. This section covers the practical infrastructure that turns a working prototype into a research pipeline.

API Access

Major model providers (OpenAI, Anthropic, Google) expose their models through REST APIs with a common structure: you send a JSON request containing your messages and parameters, and receive a JSON response with the model’s output. Python client libraries wrap this into clean function calls.

The core abstraction is the messages array: a list of (role, content) pairs representing the conversation. A system message sets the model’s behaviour, a user message provides the input, and the model returns an assistant message with its response. This is the same structure you used in Day 2 for prompting: the API formalises it.

Key parameters that affect output quality and cost: temperature (0 for deterministic classification, higher for creative tasks), max_tokens (cap the response length: for binary classification, 10 tokens is more than enough), and model (the specific model version, which should be pinned for reproducibility).

Structured Outputs

When building classification pipelines, you need machine-parseable output: not free-form text. Asking a model to “respond with JSON” usually works for large models but can produce malformed output from smaller ones (a missing comma, an extra field, explanatory text outside the JSON).

For research pipelines, structured outputs are essential. Without them, you need extensive post-processing to extract labels from free-text responses, and edge cases (the model adding qualifications, refusing to classify, or producing unexpected formats) can silently corrupt your data. With schema-constrained output, every response is guaranteed to parse correctly.

Batching & Async Processing

Processing texts one at a time is straightforward but slow. A single API call takes 0.5–2 seconds for classification; at that rate, 10,000 texts take 1.5–5.5 hours. Three patterns speed this up:

Asynchronous requests with rate limiting: Send multiple requests concurrently, using a semaphore to cap the number of simultaneous connections (typically 10–50, depending on your rate limit). This can reduce processing time by 10–50×. The notebook demonstrates this pattern with Python’s asyncio.

Batch APIs: Anthropic and OpenAI offer dedicated batch endpoints where you submit a file of requests and receive results within hours. Anthropic’s Batch API provides a 50% cost reduction for non-urgent work. For large annotation projects where same-day results are acceptable, this is the most cost-effective approach.

Checkpointing: For any pipeline processing thousands of texts, save results incrementally. If the script crashes at text 8,000 of 10,000, you should be able to resume from where it stopped rather than re-running everything. This is not a performance optimisation: it is a reliability requirement. API connections fail, rate limits trigger, and machines restart.

Cost Estimation

API pricing is token-based: you pay per million input and output tokens. For classification tasks, the input is typically 50–200 tokens (instruction + text) and the output is 1–10 tokens (the label). The cost per classification is therefore dominated by input tokens.

For research budgeting: estimate your corpus size, multiply by the per-text cost for your chosen model, and add 20–30% for retries, prompt iteration, and validation runs. A project classifying 100,000 texts with a budget model costs under $10; the same project with a frontier model costs $100–200. Self-hosted open models eliminate per-token costs but require GPU access.

Self-Hosted Inference

When data privacy prevents sending text to external APIs, when per-token costs are prohibitive at scale, or when you need guaranteed reproducibility (no model version changes), self-hosted inference becomes necessary. You run the model on your own hardware and control the entire pipeline.

vLLM (Kwon et al., 2023) is the most widely used open-source inference engine. Its core innovation, PagedAttention, manages the KV cache (from Day 1) like virtual memory, reducing memory waste from 60–80% to under 4%. This allows serving 2–4× more concurrent requests on the same hardware. vLLM supports over 50 model architectures and exposes an OpenAI-compatible API, so existing code works without modification.

SGLang (Zheng et al., 2024) achieves even higher throughput through RadixAttention, which automatically discovers and reuses shared prefixes across requests. When many requests share the same system prompt or instruction prefix : as in classification pipelines: SGLang can be significantly faster than vLLM. Benchmarks show 29% higher throughput for standard serving and up to 4.6× faster for concurrent requests.

Ollama provides the simplest path to local inference. It wraps llama.cpp (efficient C++ inference) in a Docker-like interface: install, pull a model, and run it with a single command. Ollama is the right choice for researchers who want to experiment with local models without managing GPU infrastructure. It runs on consumer hardware: including Apple Silicon laptops : though throughput is substantially lower than dedicated GPU serving.

Resources

• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention: introduces vLLM and the PagedAttention mechanism.
• Zheng et al. (2024), SGLang: Efficient Execution of Structured Language Model Programs: RadixAttention for fast structured inference.
• Anthropic API documentation: structured outputs, batching, tool use, and message batches.
• OpenAI Cookbook: practical recipes for common API patterns.
• Ollama: the simplest way to run open models locally.
• Artificial Analysis: live cost, speed, and quality comparisons across providers and models.
• Groq, Together AI, Fireworks AI: inference providers for open models; useful for cost comparison.

Validation

Whether you use prompting, fine-tuning, or a dedicated encoder model, you face the same fundamental question: can you trust the results? A model that classifies 10,000 texts in minutes is useless if 15% of those classifications are systematically wrong in ways that bias your analysis.

The answer requires the same methodological rigour that social scientists apply to any measurement instrument. In content analysis, reliability is established by having multiple human coders label the same texts and measuring their agreement. An LLM-based classifier is another coder. It needs the same scrutiny.

The LLM as Another Coder

A common mistake: treating the model’s output as ground truth. When a model classifies a tweet as “support,” that is a prediction: not a fact. The same tweet might be ambiguous, sarcastic, or genuinely borderline. The model’s classification is one coder’s judgment, and it should be evaluated against the same standards you would apply to a human research assistant.

This framing has a practical consequence: your accuracy ceiling is set by human inter-coder agreement, not by 100%. If two well-trained human coders agree on 85% of texts (κ ≈ 0.70), expecting the model to exceed that level is unreasonable. The model should match or approximate human-level reliability: not surpass it.

Cohen’s Kappa: Beyond Accuracy

Raw accuracy: the percentage of correct classifications : is the most intuitive metric but also the most misleading. If 80% of your corpus expresses “support” and a model labels everything as “support,” it achieves 80% accuracy while being completely useless. The model has learned nothing; it exploits the class distribution.

A common interpretation scale: κ > 0.80 is “almost perfect” agreement, 0.60–0.80 is “substantial,” 0.40–0.60 is “moderate,” and below 0.40 is “fair” to “poor.” These thresholds are conventional guidelines, not hard rules: the appropriate threshold depends on the difficulty of the task and the consequences of misclassification.

Beyond Kappa: F1 and Krippendorff’s Alpha

F1 provides complementary information to kappa. While kappa measures overall agreement, per-class F1 reveals where the model fails. A model with high overall kappa but low F1 on the minority class is systematically missing cases that may be analytically important.

When your project involves more than two coders (e.g., the LLM, a research assistant, and you), or when some texts are coded by different subsets of coders, Krippendorff’s alpha is the more appropriate metric.

Qualitative Error Analysis

Aggregate metrics tell you how much the model gets wrong. They do not tell you why. Qualitative error analysis : reading through misclassified texts and categorising the failure modes: is essential for diagnosis and improvement.

Five error patterns recur across LLM classification tasks:

Sarcasm and irony: The surface text expresses one sentiment while the intended meaning is the opposite. Models that rely on keyword matching tend to miss this. Chain-of-thought prompting (from Day 2) can help, as can providing sarcastic examples in few-shot prompts.

Mixed signals: Texts that contain both supportive and opposing language: e.g., acknowledging a movement’s goals while criticising its methods. The model must decide which signal dominates. Clearer instructions about what to prioritise (author’s overall stance vs. individual statements) can reduce these errors.

Ambiguity: Texts that are genuinely too short or context-dependent to classify reliably. These are not model failures : they reflect real uncertainty in the data. Consider adding an “ambiguous” or “uncertain” category rather than forcing a binary choice.

Indirect stance: The author reports or quotes someone else’s position without stating their own. A tweet saying “Protesters claim the march was a success” is reporting, not supporting. Instruct the model to classify the author’s stance, not the reported stance.

Domain mismatch: The model misinterprets domain-specific language, abbreviations, or cultural references. This is most common with non-English text, subculture-specific language, or highly specialised terminology. Fine-tuning or providing domain context in the prompt can address this.

Systematic Biases

Beyond individual errors, LLMs exhibit systematic biases that affect classification at a population level. These are not random mistakes: they are directional tendencies that can corrupt aggregate statistics even when per-text accuracy appears acceptable.

Positional bias: When given ordered options (“support or oppose”), models tend to favour the first-listed option. Rotating label order across classifications and measuring whether results change can detect and partially mitigate this.

Sycophancy bias: Models tend toward the more “positive” or “agreeable” option. In stance detection, this manifests as over-predicting support. In sentiment analysis, it manifests as over-predicting positive sentiment. This is a direct consequence of RLHF training (from Day 2), where annotators tend to prefer agreeable outputs.

Cultural and political bias: Models trained predominantly on English-language Western data may misinterpret rhetorical conventions from other cultural contexts. They may also exhibit political leanings reflected in their training data and alignment process. For research on politically sensitive topics, this is a measurement validity concern that must be documented.

Temporal instability: Closed-source model providers update their models without notice. The exact same prompt may produce different results next month than it does today. For research that requires replicability, this is a fundamental problem. Pin model versions where possible, and document the exact version, date, and provider in your methods section. Open-weight models avoid this issue entirely.

Building a Validation Plan

Before deploying any classification pipeline, you need a validation plan. This is not optional: it is the methodological equivalent of pre-registering your survey instrument.

1. Create a gold-standard set. Manually label a random sample of your corpus. For binary classification, 200–500 texts typically provides stable kappa estimates. Use stratified sampling if your corpus has known subgroups (e.g., different sources, time periods, or topics). Include edge cases deliberately : they are what differentiate good classifiers from mediocre ones.

2. Establish the human ceiling. Have at least two humans (ideally including yourself) label the same subset. Their inter-coder agreement sets the ceiling: if humans agree at κ = 0.75, that is the maximum you can reasonably expect from the model.

3. Set a threshold. Decide in advance what level of agreement is acceptable for your research question. κ ≥ 0.70 is a common threshold in political science; some tasks may justify lower (exploratory) or require higher (high-stakes policy analysis).

4. Analyse failures qualitatively. Do not just report the number. Read the misclassified texts. Categorise the error patterns. Determine whether the errors are random (tolerable) or systematic (threatening to your analysis).

5. Document everything. Your methods section should report: the model name and version, the exact prompt, the temperature and sampling settings, the gold-standard size and sampling method, the kappa score, per-class F1, the qualitative error analysis, and any modifications made based on validation results. This is not optional transparency: it is the minimum for replicable research.

Resources

• Pangakis, Wolken & Fasching (2023), Automated Annotation with Generative AI Requires Validation: essential cautionary paper demonstrating that LLM annotation performance varies dramatically across tasks.
• Gilardi, Alizadeh & Haunss (2023), ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks: landmark comparison of LLM and human annotation quality, with important caveats.
• Törnberg (2024), ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages: zero-shot LLM classification on political stance.
• Scikit-learn classification metrics: Cohen’s κ, F1, precision, recall, and confusion matrix implementations.
• Krippendorff, K. (2019), Content Analysis: An Introduction to Its Methodology: the foundational text on annotation methodology and inter-coder reliability.
• Reiss (2023), Testing the Reliability of ChatGPT for Text Annotation and Classification: systematic assessment of reliability across annotation tasks.

Social Science Applications

The tools covered in this module: fine-tuning, scaled deployment, and rigorous validation: are the infrastructure that makes LLM-based research viable. This section highlights key papers that demonstrate how researchers are deploying these techniques and the methodological lessons they have learned.

LLM-Based Annotation at Scale

Fine-Tuning vs. Prompting: When Each Wins

The Validation Imperative

Reproducibility & Open Science