Benchmark · ORCA-bench, public

Finding an incident's root cause, after learning from past incidents.

An on-call agent gets a vague complaint, the shop's source code and a frozen copy of its telemetry, and must name what broke and when. With notes learned from graded practice incidents, the same Haiku agent went from 6.3% to 35.2%, and invented far fewer causes.

Root causes found35.2% from 6.3%+28.9 pts · 95% CI +20.8 to +37.6

Invented root causes7.1% from 28.5%share of reports naming a wrong cause

Cost per task$0.024 from $0.03225% cheaper

Small model, big model's results

Haiku with notes scored higher than Sonnet 5.5 alone, at 1/25 of the cost.

Claude Haiku 5.5 + Enterprise Brain55.3%

$0.021 a task

Claude Sonnet 5.5 alone40.2%

$0.517 a task

ORCA-bench mini: the same 132 tasks and grader. Haiku read notes learned from all 454 graded practice runs; Sonnet had no memory. Difference +15.2 pts (95% CI −3.0 to +32.9). On the full set, Haiku with notes solved 53.0% of Medium and 15.7% of Hard tasks; the best model on the public leaderboard, Sonnet 4.6, reports 26.6% and 9.0%.

At a glance

Read the telemetry, name the cause, give the time.

Made by: ORCA-bench, a public benchmark. It simulates on-call root-cause analysis on the OpenTelemetry demo shop, a microservice store with faults injected through feature flags.
The task: The agent gets a shell, the shop's source code, Grafana with metrics, traces and logs, and a short report. It writes a report with the root cause, a timeline, five whys and a fix, or an empty report if nothing happened.
Example: The following issue was reported at 5:02 PM yesterday ET: users cannot complete their purchase The task adds: there may be multiple root causes or no root cause at all.
Data: 293 scored tasks from six days of one shop: 132 Medium and 121 Hard incident tasks, drawn from 50 questions about 13 incidents, plus 40 healthy controls where nothing broke. Up to 150 steps per task.
Practice vs test: The six days are split into two groups of three. A test task reads only notes learned from the other group's days, so practice and test never share an incident. Practice was 454 graded runs of the agent without memory; each test task's note came from its group's 216 or 238.
What's compared: No memory. OpenViking search, with and without what its own trainer learned from the same practice runs. Enterprise Brain search with the learned note for the task's kind. Enterprise Brain search with a procedure compiled from documents, before any runs.
Models: The agent is Claude Haiku 5.5. Claude Opus 5.5 wrote the notes; Haiku 5.5 matched each task to its note before the run.
Graded by: The benchmark's own rubric judge, run with GPT-6.1 Sol. A task is solved when the report names every injected root cause. A report that names none of them, but isn't empty, counts as invented.

Results

Notes from graded runs moved the score; search alone barely did.

253 incident tasks. Brackets compare each arm with no memory, resampled by question.

Arm	Root causes found	vs no memory	Cost per task
No memoryshell, code and telemetry	6.3%		$0.032
OpenViking search	9.1%	+2.8 [−0.8, +7.3]	$0.030
OpenViking, with its own learningfrom the same practice runs	11.1%	+4.7 [+0.0, +10.3]	$0.033
Enterprise Brain, procedure onlycompiled from documents, no runs yet	10.3%	+4.0 [+0.4, +8.5]
Enterprise Brain notessearch plus the learned note	35.2%	+28.9 [+20.8, +37.6]	$0.024
Enterprise Brain notes, with a goalthe same, with a goal set	32.8%	+26.5 [+16.2, +36.5]	$0.025

On Hard tasks, accuracy went from 0.8% to 15.7–17.4%. On the 40 healthy controls, false alarms fell from 22.5% to 17.5%.

More runs, better notes

Accuracy kept climbing with each day of graded runs.

ORCA-bench mini: 132 Medium tasks, Haiku 5.5. Each point is scored on days the notes didn't learn from.

No memory13.6%

Procedure compiled from documents, no runs11.4%

Notes after day 1 · 135 runs18.9%

Notes after day 2 · 284 runs34.8%

Notes after day 3 · 454 runs49.2%

Notes from all 454 runs at once55.3%

Sonnet 5.5 alone, no memory40.2%

Haiku with notes cost about $0.021 a task; Sonnet 5.5 alone about $0.517. Haiku with notes led by 15.2 points.

One task, start to finish

A promotions complaint, solved by a lesson from other features.

It stopped reporting "no incident".

Task: The following issue was reported at 10:36 AM today ET: shoppers are having problems with promotions

Without memory

"Across every signal I checked, the telemetry shows no promotion-related anomaly, so no incident occurred." It wrote an empty report.

Graded wrong

What the note said

"Concluding "no incident" and writing an empty report. This happened in 11 runs on photos, ads, homepage, product pages and recommendations. Every one of those cases had a flag change."

Learned from 40 runs: 0 right, 40 wrong

With the note

"The most likely trigger is the adFailure feature flag, inferred to have been enabled at about 14:30 UTC from symptom timing."

Graded correct

ORCA-bench task c7cb3ed6ba8a5beb (Medium), from the Oct 19 run whose notes the home page shows.