Humanity's Last Exam Explained: Where 2,500 Frontier Questions Come From and How to Read the Scores

If you read model launches, one name keeps appearing: Humanity's Last Exam (HLE). Major vendors cite it as a reasoning-comparison item, and it has become shorthand for "frontier difficulty". But most citations give only a percentage -- where the questions came from, how 2,500 items were protected from contamination, why there are no-tools and with-tools rows, and what the dynamic HLE-Rolling variant implies. Based on the official HLE site (re-fetched and checked 2026-09-08) and a recent launch that cites it, this article unpacks the benchmark's origins and the correct way to cite its scores.

1. What HLE Is: A One-Page Résumé

HLE is produced by Scale AI and the Center for AI Safety (CAIS) as a cross-disciplinary frontier reasoning benchmark -- questions were authored by contributors from 500+ institutions across 50 countries -- mostly professors, researchers, and graduate degree holders -- spanning over a hundred subjects. Three hard facts: finalized on 2025-04-03 with 2,500 questions; published in Nature on 2026-01-28 (Nature 649, 1139-1146); the dataset is hosted as cais/hle, with an Arxiv paper, a GitHub repository, and a live submission dashboard.

2. Where the Questions Come From: An Anti-Contamination Design

HLE's standing rests not on difficulty but on clean questions. The items came from global contributor submissions and passed a complete quality pipeline:

1. Open contribution: the community broadly collected candidate questions;
2. Bug bounty for correction: a community-feedback expansion with bounties specifically flagged defective questions;
3. Removing searchable questions: flagged questions and questions whose answers could be found via web search were removed and replaced;
4. Finalization: 2,500 questions on 2025-04-03.

This pipeline answers a fundamental question: testing reasoning requires that the model cannot cheat by searching. No matter how hard a static bank is, it gets digested over time -- which is exactly why the dynamic variant in the next section exists.

3. HLE-Rolling: Keeping the Question Bank Fresh

On 2025-10-08 the official team launched the dynamic HLE-Rolling variant with a live submission dashboard. Rolling's questions rotate continuously, compressing the space for memorization. The implication for citations: always state whether a score is from the static version or Rolling -- the question sets differ and the numbers cannot be compared. It also signals a broader trend in evaluation: important benchmarks are moving from one-time finalization toward continuous freshness.

4. Reading the Scores: Conventions, Missing Cells, and Comparison Tables

The three most common traps in launch citations are all visible in one example -- the 2026-09-01 Claude Fable 5.1 announcement table:

Three reading points:

1. Dual conventions: no-tools and with-tools are independent rows, differing by 4.1 points for Fable 5.1 -- citations must carry the convention;

# Dual-convention spread for the same model (numbers from the 2026-09-01 table)
fable51_no_tools, fable51_with_tools = 60.9, 65.0
print(f"Fable 5.1 no-tools/with-tools spread: {fable51_with_tools - fable51_no_tools:.1f} pts")

2. Missing cells are unreported: GPT-5.6 Sol shows a dash on the HLE row -- unreported is not losing; find the model's own official publication when you need the number;
3. Single dimension: HLE measures cross-disciplinary reasoning only; it says nothing about coding, long context, or agentic capability -- read the whole comparison table, not one row.

# HLE citation self-check list
[ ] Convention: no-tools / with-tools
[ ] Version: static / HLE-Rolling
[ ] Date: as-of information for the score
[ ] Trial count and grading script version

The complete method for reading such tables (log cost axes, error margins, safeguard-intervention zeros) is in our companion piece, How to Read the Benchmark Charts in a Model Launch: Effort Curves, Log Cost Axes, and Safeguard Interventions.

5. Where HLE Fits in a Deep-Research Workflow

For teams doing deep research, HLE has two practical uses:

• A reference frame for frontier reasoning: on the "frontier difficulty" dimension, HLE provides a cross-vendor comparable anchor -- use it to shortlist which models deserve a slot in your candidate pool;
• A methodology template: its question governance (bounty-driven correction, replacing searchable items, continuous freshness) is a design template worth borrowing for your own evaluation sets.

But hold the boundary: HLE measures reasoning, not your task. Domain data, real samples, cost, and latency -- the final grounds for selection -- are irreplaceable by any benchmark (the broader quality-control picture: DeepResearch Output Quality Control: Model Selection, Prompt Tuning, and Citation Verification).

6. Common Mistakes and Troubleshooting

• Mixing no-tools and with-tools scores: a 4-point convention spread is enough to flip rankings.
• Mixing static and Rolling scores: different question sets carry different semantics.

# Citation format example (three elements present)
Fable 5.1 scored 65.0% on HLE (static, with tools) per the 2026-09-01 announcement

• Reading a dash as a loss: a dash means unreported, not zero.
• Concluding from one benchmark: a single-benchmark ranking is a single-dimension ranking; complete judgment needs a set of benchmarks plus real tasks.
• Citing scores without dates or versions: benchmarks evolve -- scores without as-of information decay into noise.

7. Next Steps

• The full fact briefing where HLE appears as a comparison item: OpenAI Ecosystem September 2 Briefing: Anthropic Launches Claude Fable 5.1 / Mythos 5.1, Benchmarks Include GPT-5.6 Sol.
• The complete method for reading launch benchmark charts: How to Read the Benchmark Charts in a Model Launch: Effort Curves, Log Cost Axes, and Safeguard Interventions.
• The quality-control overview for deep research: DeepResearch Output Quality Control: Model Selection, Prompt Tuning, and Citation Verification.