The AI Respondents Challenge.

Skill. Accuracy above always guessing the majority answer, rescaled as (accuracy − majority share) / (1 − majority share) and averaged over target questions. 1 = every answer right; 0 = no better than majority guessing; negative = worse than majority guessing. The ceiling is 1 but the floor is not −1: on questions where one answer dominates, skill can fall well below −1, because there is more room under the majority guess than above it. Teams are ranked on this.

Alignment. One minus the distance between your predicted answer distribution and the true one, per question, averaged. The distance is an order-aware earth-mover distance: answer options sit at equal steps in scale order, and we sum the absolute gaps between the two cumulative distributions, divided by the number of steps. Range 0 to 1: 1 = identical distributions, 0 = all your mass at one end of the scale while the truth sits entirely at the other. Only valid labels enter this measure; invalid ones are already punished in skill.

F1-macro. Computed in two steps. Per question: all answered pairs are pooled across countries, an F1 score (the harmonic mean of precision and recall) is computed for each answer option, and these are averaged with equal weight to give the question's macro-F1. The board then shows the plain average of the per-question values. Range 0 to 1. Because every answer option counts equally, rare answers matter as much as common ones, which is exactly what accuracy misses.

Coverage. Share of respondent–question pairs answered, from 0 to 100%. Unanswered pairs count as wrong in skill, so predict everyone.

Hidden-question boards. The Hidden questions tab covers the four questions that were never announced as targets until their mid-week release. Same metrics, same truth data, but scored only over the hidden questions and only for teams that submitted them. The main boards always score the original nine questions, so submitting the hidden tier never changes your main-board standing by itself.

Held-out survey boards. The third tab scores two surveys released on Thursday with no training data at all: European Social Survey wave 11 and Latinobarometro 2023, predicted zero-shot. The pooled view ranks teams over all sixteen questions, both surveys weighing equally, split into countries WVS also covers and countries it never has. The ESS and Latinobarometro views show each survey's own boards with per-survey skill, alignment and F1. On these boards the baselines are labelled WVS-seen countries baselines: their base rates come from the countries WVS also covers, so on the unseen-population boards they show what majority guessing scores when the majorities were learned somewhere else.

Baselines. Majority class always picks each question's most common training answer; Random draw samples from training shares. Both score like teams. Note that skill 0 already marks a perfect majority guesser for a board's own countries, by construction. The baseline rows estimate those majorities from the training data instead, so on boards whose countries have no training data they sit below zero: that gap is the cost of assuming base rates transfer. Doing well on skill and alignment together is the goal.