LLM Agents and Evaluation

Start here

In the invented example used below, a reader sends an assistant the title of a health article and asks whether it supports a conclusion they have already reached. One response confidently agrees and cites a review of twelve trials that does not exist; another asks to see the article’s findings and methods before drawing a conclusion. The first may feel more helpful because it offers a clear answer, but its reassurance rests on evidence the assistant does not have. Understanding why a reader might prefer it is a useful part of evaluation, alongside examining whether the response can support what it claims.

This distinction matters throughout LLM evaluation, from rating an assistant’s responses to testing an agent that runs machine-learning experiments. A score is evidence for a particular claim under a particular evaluation setup. Understanding that setup means knowing what the system was asked to do, how its work was judged, and which failures the test could reveal.

A useful framework separates human likeability, safety, broad base capabilities, and professional or complex-task capabilities. These dimensions overlap, but each needs its own evidence. Comparing preferred answers, evaluating health conversations, and testing research agents shows how the requirements change as systems take on more demanding work.

Original slide 6 The lecture’s four axes of evaluation

The core course introduces the background in Evaluating & Choosing Models and Agentic Workflows. The examples here develop those ideas further and can be followed without code.

Preference: a useful signal with a limited claim

Pairwise evaluation gives preference a concrete form: a person sees two responses to the same prompt and chooses one. Combining many such comparisons produces an estimate of relative preference across the sampled prompts and participating raters. Chatbot Arena uses this approach to compare assistants, capturing aspects of the experience that a narrowly defined task metric might miss.

Interpreting the result requires care because several qualities influence a choice. Formatting, fluency, length, and confidence may all matter, while a reader without the relevant expertise or source material may struggle to detect an error. Preference can therefore help explain why an assistant is appealing without establishing how accurately it performs a particular task.

When the judge is another model

Human judgement is costly to collect at scale, so evaluations also use an LLM-as-a-judge: a model prompted to rate another model’s response. This introduces a second system whose judgements need validation. The MT-Bench and Chatbot Arena paper documents limitations including position and verbosity bias, showing why the judge’s model, prompt, rubric, and access to evidence all matter.

For example, judging a literature summary simply by asking how helpful it is combines several properties that may deserve separate attention. A rubric could assess whether the supplied paper supports the summary’s claims, whether important qualifications survive, whether citations lead to relevant evidence, and whether the explanation suits its audience. These Oxford LLMs examples make the evaluation goal specific enough to inspect.

The next step is to check how well the judge applies those criteria. Agreement with expert raters on representative examples supports its use on similar material. Swapping answer order or changing the style of an answer while preserving its claims can also reveal sensitivity to features that should not determine the score. Together, these checks help define where the judge’s results are interpretable and where further scrutiny is needed.

Risk: the answer can be appealing and still fail

Preference becomes particularly consequential when a system is rewarded for agreement in situations that call for correction or restraint. Sycophancy describes excessive agreement with the user’s beliefs or preferences at the expense of an accurate or appropriate response. Research on sycophancy in language models finds that human preference feedback can favour agreement over truthfulness. An evaluation concerned with this failure needs to examine how the model handles mistaken premises, uncertainty, and disagreement.

In vulnerable conversations, these behaviours may have consequences beyond the quality of an individual answer. A study of potentially harmful chatbot conversations examines a small observational sample and motivates closer scrutiny of possible mental-health harms. Its findings need to be read within that study design: they do not establish causation or the prevalence of harm among chatbot users.

Other risks require their own evidence. A fabricated citation, a systematic error affecting one group, and assistance with dangerous misuse involve different failures, so they need different test cases and decision rules. Wider societal questions may require a different unit of analysis altogether. Labour-market exposure estimates, such as those in GPTs are GPTs, concern which work tasks might change as capabilities advance; they neither measure an individual model’s correctness nor establish realised job losses.

Even when the requirement is zero critical failures, a finite test can only establish what was observed under its particular conditions. The number of cases, their coverage, and the uncertainty around the result remain relevant. Medical evaluation brings these concerns together because both the nature of an error and its frequency affect how a result should be interpreted.

HealthBench: expertise becomes a rubric

An exam question can often be scored against a defined answer key, whereas a health conversation may require several judgements at once. A response can contain correct facts while omitting a crucial qualification, offering unsupported reassurance, or overlooking the user’s circumstances. HealthBench addresses this problem by evaluating a model’s next response in the context of a conversation, using criteria defined by physicians.

The HealthBench paper describes 5,000 multi-turn conversations with 48,562 unique rubric criteria, developed with 262 physicians. Each conversation has its own weighted rubric for the model’s next response. Rather than demanding one canonical wording, the rubric specifies behaviours that should occur and behaviours that should not.

According to the official introduction, a GPT-4.1-based grader checks responses against these criteria. Domain experts define what matters, while the model grader applies their criteria at scale; physician-authored or validated rubrics therefore do not imply that physicians independently graded every output. As with other model judges, understanding the results requires attention to grader validation and inspection of its errors.

How a conversational rubric changes the measurement

For the title-only article question, a domain rubric would examine whether the response acknowledges that the evidence is missing, asks for information that could resolve the question, and avoids inventing support for the reader’s conclusion. A response might satisfy some of these requirements while still introducing an unsupported claim. Separating the criteria makes that combination visible instead of letting the answer’s reassuring tone determine the whole judgement.

HealthBench weights its criteria and includes penalties for undesirable behaviour, so an omission or harmful statement can affect the result even when the response is otherwise well written. Those weights are part of the evaluation’s design: their relative sizes express how much different behaviours contribute to the score. The illustration below shows how this works with an invented rubric and three example responses.

Reading a score alongside the conversation, rubric, and criterion-level judgements makes the failure easier to diagnose. Instead of a general impression that the model performed badly, the evaluation can identify an unsupported statement, an omitted action, or an assumption that the response should have questioned.

Separate the overall score from specific failures

The same attention to individual criteria helps explain why HealthBench has several variants. Hard contains 1,000 challenging examples, while Consensus contains 3,671 examples evaluated with criteria filtered through further physician validation. They address different properties: the difficulty of the conversation and agreement about what an adequate response should do. Within these variants, narrower failure slices examine particular kinds of error, as documented in the GPT-5 system card, section 3.10.

Original slide 37 HealthBench, Hard and Consensus results

These slices use additional physician validation and report error rates, so lower values indicate better performance. Their different criteria and denominators also matter: comparing rates across slices is not equivalent to comparing models on the same test.

The Global Health slice illustrates why context belongs in an evaluation. A medically plausible response may still assume resources, local risks, or care pathways that the conversation does not establish. Similar problems arise when social scientists evaluate models across countries or languages: criteria developed for a dominant setting may overlook what an adequate response requires elsewhere.

Original slide 38 HealthBench failure slices

What HealthBench can establish

Taken together, these results provide evidence about responses to HealthBench’s conversation distribution under its rubric and grading procedure. Because scored text responses represent only part of a real-world care process, they do not directly measure patient outcomes or certify a system for clinical deployment.

The approach is useful beyond medicine because it gives domain expertise an explicit role in scoring. A historian might specify how an answer should handle conflicting primary sources, while a survey researcher might define which ambiguities a coding system must preserve. In either case, the evaluation needs to describe the required behaviours and critical failures, then establish whether its criteria work across the intended population.

From an answer to a research experiment

Domain criteria become more complicated when a system must carry out work as well as explain it. A research agent may need to inspect data, write code, run experiments, and produce a result that someone else can check. Evaluating this workflow requires evidence about its intermediate decisions and execution, beyond the quality of its final answer.

The distinction is easier to see by comparing tasks. An MMLU-style question tests knowledge through a constrained answer format, while a SWE-bench task asks a system to resolve an issue in a code repository, with executable tests providing part of the evidence. Completing an ML experiment places further demands on the system: it may choose a poor plan, struggle with its environment, exhaust its budget, or produce output that cannot be evaluated. Each benchmark captures a different part of this progression.

MLE-bench: the model operates inside a scaffold

MLE-bench examines this kind of practical work through 75 Kaggle-derived competitions adapted for local evaluation. An agent works on an ML engineering problem and produces a submission scored using the competition’s task metric, making its sequence of practical decisions part of the evaluation.

The language model operates within an agent scaffold: software that manages tools, state, execution, retries, and selection between candidate solutions. The scaffold can shape the search substantially. Sequential refinement revises a candidate over successive steps, whereas the branching experiment search used by AIDE explores multiple candidates. For example, separate branches can pursue different models or training choices rather than continuing to edit the same solution.

Those opportunities come with resource costs. More branches, retries, or runtime may improve the chance of finding a strong solution, so a comparison needs to account for how much search each system can perform. Holding the scaffold, tools, tasks, budget, and scoring constant helps isolate a model difference. When the aim is to compare complete systems, their designs and constraints instead become part of what the reported result describes.

AIRS-Bench: a valid run is only the first hurdle

AIRS-Bench extends the comparison to 20 research tasks drawn from machine-learning papers across several domains. Its repository provides task definitions and evaluation code, while the results reproduced here compare models operating within one-shot, sequential, and tree-search approaches.

An important distinction is whether an agent produces an evaluable submission at all, and how well that submission performs. AIRS-Bench captures the first with valid submission rate (VSR): the proportion of valid runs within each task, averaged across tasks. This task-level averaging matters because pooling all runs can give a different answer when tasks have different numbers of attempts. Average normalised score (ANS) then summarises task performance on a shared scale, allowing results measured by different task metrics to contribute to an aggregate.

The scoring convention first transforms distance from the task’s theoretical optimum using a negative logarithm, so equal proportional reductions in the remaining distance produce equal increases in the transformed score. On that transformed scale, the worst observed run anchors the normalised score at 0 and the cited literature reference anchors it at 1. Invalid or failed runs receive a normalised score of 0 and remain in the average, so ANS includes workflow failures as well as the quality of valid results. The resulting average depends on this transformation and its reference points; it cannot be interpreted as a mean accuracy or the fraction of tasks completed successfully.

In the historical comparison shown in slides 47–48, the average VSR is 65.8% and the average ANS is 0.352. Together they describe both the difficulty of reaching an evaluable submission and the performance achieved under this scoring convention. In particular, 0.352 is neither a 35.2% task-success rate nor an accuracy of 35.2%. Both figures belong to the configuration set and run conditions shown in the presentation.

Original slide 47 AIRS-Bench valid submission rates

Original slide 48 AIRS-Bench average normalised scores

The aggregate also leaves room for substantial differences between tasks. On some, evaluated agents exceeded the cited human reference result, even though performance across the full task set remained modest. The examples below show improvements in coreference resolution, molecular property prediction, and forecasting, each measured using its own task metric.

All three examples report Gemini 3.1 Pro agents and reproduce historical results from slide 50. Their metrics need to be read in the appropriate direction: accuracy increases, whereas mean absolute error decreases. Because relative improvements are computed against each reference value, they differ from percentage-point changes and should not be averaged across incompatible raw metrics. The reference results are also tied to the cited papers; establishing whether they remain state of the art would require a current comparison under equivalent conditions.

The Winogrande result also shows what the agent contributed. Gemini 3.1 Pro acted as the controller developing the solution, which fine-tuned Qwen2.5-14B using LoRA. The reported accuracy belongs to that resulting prediction model, rather than to Gemini answering the test questions directly. The agent’s work includes choosing and implementing a way to solve the task; its scaffold and resource budget affect which solutions it has the opportunity to develop.

Original slide 50 Selected AIRS-Bench task results and methods

These examples show why domain expertise remains necessary even when the result can be scored automatically. Across medicine, chemistry, social science, and other fields, someone must decide what constitutes meaningful progress and which interpretations a metric supports. An executable score makes part of that judgement reproducible without settling the scientific question on its own.

Why a ranking can change without a model changing

The judge, the rubric, and the agent scaffold each influence the results above. Another part of the evaluation can change a ranking even when the systems themselves stay fixed: the distribution of tasks. An aggregate score depends on which cases are included and how much each contributes. The following illustration isolates that effect by keeping each system’s performance within each type of case constant.

As the mix changes, so does the overall ranking. A benchmark can therefore suggest a different model choice from the one suited to your work if the two contain different proportions of cases. This helps explain the value of HealthBench’s urgent and global-health slices: a useful overall average may conceal performance differences that matter in a particular setting. For research agents, the relevant breakdown might instead separate classification, forecasting, and molecular prediction tasks. Reporting these alongside the aggregate makes its weighting easier to assess.

Variation also occurs between attempts at the same task. A long-running agent may fail before scoring or follow a different sequence of experiments on its next run. Repeated runs help reveal that variation, provided failed attempts remain in the record. A report of at least one success in five attempts describes a different level of reliability and resource use from success on a single attempt, so the selection rule belongs with the result.

Can we trust the evidence behind the score?

Accounting for the judge, task mix, and run conditions still leaves a question about the test material itself. As models improve and public benchmarks become widely used, a score may become less informative for two different reasons: the test may no longer distinguish strong systems, or exposure to its contents may undermine the claim that it measures generalisation.

The first problem is saturation. When many scores approach a test’s ceiling, a harder test can restore separation, provided its difficulty still represents the capability of interest. A task that is difficult for reasons unrelated to the intended use would offer little improvement.

The second is contamination, where exposure to test material weakens the evidence of performance on unseen cases. Establishing exposure can be difficult when the training corpus is unavailable. Even without known training exposure, public tasks may influence prompt design, scaffold development, or repeated selection of systems, making the history of benchmark use relevant to its interpretation.

Testing for exposure

One approach to investigating exposure is the black-box contamination study, which compares the likelihood a model assigns to examples in their original dataset order with the likelihood of shuffled orders. Under the paper’s assumptions, a difference can provide statistical evidence of contamination without access to the full training corpus.

Recovery of masked benchmark content and audits of training data examine other signals. Each supports a particular kind of exposure claim, while a failure to detect exposure leaves uncertainty rather than certifying a clean test. It is also necessary to distinguish learning a skill from seeing its test cases: training on translation examples, for instance, does not by itself establish exposure to a particular held-out translation benchmark.

LiveCodeBench: refresh the tasks, retain the conditions

LiveCodeBench addresses part of the exposure problem by collecting coding problems from periodic contests over time. Dated task windows allow comparisons on problems released after a relevant cutoff, reducing reliance on a fixed public test set. The paper describes its evaluation across coding scenarios.

New problems still need to be considered alongside the evaluation conditions. A model’s training cutoff may be unclear, tools may retrieve public solutions, and a widely used window may enter the development process. Release dates, permitted tools, and the specific task window therefore remain necessary context for reading the leaderboard.

The exposure question is one instance of a broader requirement: the evidence must match the claim being made. That requirement becomes particularly demanding when a domain result is used to support a claim about general intelligence. Different definitions of AGI can require different evidence, so a numerical threshold needs an argument linking the benchmark task to the proposed definition.

Bring the evaluation back to your research

Designing an evaluation for your own research brings these decisions together. Domain expertise determines which behaviours matter, which failures would undermine the work, and which contexts the test must cover. If an agent will carry out the task, the evaluation also needs a reproducible environment and a result that can be scored.

Consider an Oxford LLMs example: an agent extracts policy commitments from speeches and returns supporting passages. As with the title-only health question, the system needs to establish what the source actually supports before making a claim on its behalf. Before comparing systems, a researcher would need to specify the population of speeches and languages, define what counts as a commitment, and decide how to adjudicate ambiguous cases. Requiring a supporting passage makes it possible to distinguish missed commitments from unsupported extractions or fabricated quotations. Examining errors by group and document type would then show whether the aggregate conceals failures concentrated in particular parts of the material.

The same design needs to specify how the system is developed and tested. Keeping development cases separate from the final test, fixing the model, prompt, tools, scaffold, and budget before final evaluation, and repeating stochastic runs make the comparison easier to interpret. Recording these choices allows another researcher to reconstruct what was asked, what counted as success, and which limitations remain. The reference below collects them in one place.

Claim and intended use:
Population, sampling unit, and exclusions:
Task set version; development and final-test split:
Reference labels, rubric, and adjudication:
Model version, prompt, tools, and scaffold:
Budget, attempts, selection rule, and stopping rule:
Metrics, subgroup breakdowns, and critical failures:
Repeated runs, execution dates, and uncertainty:
Data exposure checks and remaining limitations:
Decision rule and location of run logs:

For the reader who supplied only an article title, a helpful response would explain what remains unknown and ask for evidence that could resolve it. The rubric makes those requirements explicit and distinguishes them from the appeal of a confident answer. Research agents extend the same need for evidence into a workflow whose execution, resources, and variation across attempts also affect the result. Keeping these observations connected to the intended use gives a favourable score a meaning that another researcher can examine and defend.

Original materials and sources

Tatiana’s original slides, 99 pages are the primary record of the lecture. The main presentation is pages 1–71; its appendix, pages 72–99, extends the discussion to the Turing test, agent environments, alignment, and licensing. The explanations and synthetic examples on this page were written and assembled by Oxford LLMs. Reproduced figures retain their historical results, while current source views are labelled separately. PDF references use page numbers.

• Preference and judging: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, accompanying slides 7–13.
• Medical conversation evaluation: HealthBench paper, official introduction, and dataset and grader, accompanying slides 37–38. The three failure slices are documented in the GPT-5 system card.
• ML engineering: Chan et al., MLE-bench, accompanying slides 39–42.
• Research agents: Lupidi et al., AIRS-Bench and benchmark repository, accompanying slides 45–51.
• Contamination: Oren et al., Proving Test Set Contamination in Black Box Language Models, cited in slide 61; Golchin and Surdeanu, Investigating Data Contamination in Modern Benchmarks, cited in slide 63.
• Refreshed coding tasks: LiveCodeBench paper, project, and evaluation code, accompanying slides 65–67.

Continue with Evaluating & Choosing Models for task-specific model selection, or Evaluating Agents for outcomes and execution traces. Return to Advanced lectures for the lecture collection.