Measuring AI R&D Pace Inside a Frontier Lab: Anthropic's Three Self-Reported Metrics

How much of a frontier lab's AI research is now done by AI itself? On 2026-09-17 (the date per Anthropic's Newsroom index; the article page itself shows no date), Anthropic published "Measurements for understanding the pace of AI development inside frontier labs" (authors Marina Favaro and Phillie Wright), breaking its internal AI R&D automation level, agent oversight posture, and safety compute allocation into three sets of quotable numbers. This article restates each metric, the headline values, the methodology, and the limitations the authors state themselves (re-fetched and checked line by line on 2026-09-18, the day after publication). Every number is an official self-reported figure — we draw no comparisons with the OpenAI ecosystem.

1. Why measure R&D pace: the problem definition

The starting point: AI increasingly automates its own development, approaching "recursive self improvement" — a model fully building its successor. As the world debates pacing frontier AI, the authors want to narrow the information gap between labs and the public by making three things publicly countable: how automated the internal work has become, how the agents are overseen, and how much compute goes to safety.

The three metrics in one line each: the automation index answers "how much does AI do," agent oversight answers "how are they watched," and the safety compute share answers "how much compute goes to safety."

2. Metric one: the R&D Automation Index

The scale comes from Epoch AI's automation levels: AL0 (no AI involvement) to AL5 (fully autonomous, no human in the loop). Two levels matter most: AL3, "collaborates" — AI handles large chunks of work under close human direction; AL4, "leads" — AI completes most of a task end-to-end from a high-level prompt while a human supervises.

Current values (as of August 2026): Claude leads 26% of Anthropic's AI R&D work; more than 90% of work is at collaborates or above; no measured subset is fully autonomous.

Methodology: during each week of July 2026, 20% of staff were randomly sampled per department, their work broken down into about 15,000 granular tasks, and organized into a 542-node (378-leaf) task tree frozen as the baseline; Claude agents gather evidence, an independent Claude judge assigns automation levels, and tasks are weighted by person-time. Validation: the judge's exact agreement with humans was 59%, higher than human-to-human agreement at 35%; agreement within one level was 97%. A January 2026 comparison basket served as a stability check and showed no rise in novel tasks.

3. Metric two: oversight of AI agents

This metric splits into three sub-metrics: coverage (the share of agent actions passing through a monitor), review latency (time from action to review), and the escalation/blocking rate (the share of activities blocked or flagged).

Current values: on Anthropic's most-used internal platform, about 30,000 agents were doing research and engineering work at any one time; 100% of agent actions passed online monitors before execution; in August 2026 over a billion agent decisions were analyzed and 0.002% were blocked (roughly 1 in 47,000); blocked actions get human review within a week. On the offline side: about 100,000 transcripts are flagged per week, about 50 of which are escalated to human review — roughly 1-2 transcripts flagged per thousand.

Supporting design: each agent has a persistent identity that survives model upgrades, and an open shared messaging channel lets agents audit one another. METR has independently red-teamed the offline monitoring platform.

4. Metric three: the safety compute share

In the July 13-20, 2026 snapshot: about 6% of AI R&D compute and about 12% of AI-driven AI R&D compute went to safety work. How it was measured: existing monitoring tools plus Claude-based classifiers, sampling about 14% of nearly 10,000 runs that week (weighted toward compute-heavy runs); transcripts that could not be accessed were conservatively classified as AI R&D.

The authors' own warnings are strongest on this number: dual-purpose work counts as capabilities, and the safeguard classifiers are excluded from the safety side — so this is a deliberately conservative floor.

5. The limitations the authors state themselves

The limitations listed in the piece, restated one by one:

• Labels are best-effort and unverified;
• The compute snapshot covers a single week;
• Compute share measures spending, not effort — efficiency gains can shrink the share without any reduction in safety work;
• What counts as "safety" is judgment-dependent, and the authors argue the burden of proof should sit with developers;
• The safety compute measure is deliberately conservative (see section 4).

When quoting any number from this piece, carry "self-reported" and the matching caveat with it — the same discipline as reading any benchmark chart in a model launch.

6. Policy context: the RSP and third-party evaluators

The publication positions these metrics as a complement to capability evaluations under the Responsible Scaling Policy — the RSP page (current version 3.4, effective July 8, 2026) already includes AI R&D automation capability thresholds that trigger affirmative risk cases once crossed — and as aligned with the transparency obligations proposed in the Advanced AI Framework. On implementation, Anthropic plans to embed independent third-party evaluators from multiple organizations with internal access comparable to its own risk teams, tasked with verifying safety practices, reporting incidents, and monitoring the metrics.

7. How to read self-reported numbers like these

A three-step conservative reading. First, record the definition before the number: "leads 26%" only holds under the AL4 definition (end-to-end completion, human supervision); quoting it without the definition misleads. Second, read the interlocking numbers together: 26% goes with "90%+ at collaborates-or-above" and "none fully autonomous"; 0.002% goes with "100% coverage" and "100,000 weekly flags." Third, acknowledge the self-reported nature: the task tree and the judge are built in-house, and the stated limitations should travel with the numbers. For the general discipline of reading official numbers, see How to Read the Benchmark Charts in a Model Launch: Effort Curves, Log Cost Axes, and Safeguard Interventions.

8. Next steps

• Same-day companion piece: Anthropic's Life Sciences Verification Program (LSVP), Explained: Tiered Access for Biology Workloads — the other half of Anthropic's September 17 transparency and safety-mechanism push.
• How to Read the Benchmark Charts in a Model Launch: Effort Curves, Log Cost Axes, and Safeguard Interventions — applies the same "definition before number" discipline to any official figure.