- Enterprise Brain
- Benchmarks
- τ²-bench banking
Benchmark · τ²-bench banking, public
Following 698 bank policies, learned from practice.
A support agent serves simulated bank customers and must act by the bank's policy documents. Notes learned from practice conversations tied an agent that reads every document as a file. Procedures compiled before any practice fell behind.
Tasks passed30.9% vs 27.8% with files+3.1 pts · 95% CI −3.4 to +10.0
Notes vs procedures alone+10.3 pts95% CI +4.1 to +17.2
Cost per conversation$0.069 vs $0.054per conversation
At a glance
Talk to the customer, check the policy, take the right action.
- Made by
- τ²-bench from Sierra Research, banking knowledge domain. Public.
- The task
- Talk with a simulated customer, answer from the bank's policy documents, and act through the bank's tools.
- Example
- A customer asks to change the email on their account in a way that signals an ownership dispute. The policies say to hand it to a person, so the expected action is a transfer to a human agent.
- Data
- 698 policy documents and 97 tasks, each run 3 times. On 23 hard tasks every arm scores 0–3%.
- Practice vs test
- Tasks are grouped by the documents they need (52 groups, 3 folds). Each fold's notes learn only from other folds' practice: 592 Haiku conversations on 74 tasks. A scan of 376 task-specific strings found none in memory.
- What's compared
- All 698 documents as files, read with a shell. OpenViking search, with and without its own learning. Enterprise Brain search with the learned note, with and without a goal. Enterprise Brain search with a procedure compiled before any practice.
- Models
- The agent is Claude Haiku 5.5; GPT-5.2 plays the customer. Claude Opus 5.5 compiled the procedures and wrote the notes.
- Graded by
- τ²'s programmatic checks on the bank's final database and the expected actions, averaged over 3 runs. Practice grades listed which expected actions weren't done, with names, ids and amounts removed.
Results
Notes caught up with reading every document.
97 tasks × 3 runs. Brackets compare each arm with the files arm.
| Arm | Passed | vs files | Cost per conversation |
|---|---|---|---|
| All documents as files | 27.8% | $0.054 | |
| OpenViking search | 20.3% | −7.6 [−13.7, −2.1] | |
| OpenViking, with its own learning | 21.6% | −6.2 | |
| Enterprise Brain, procedure onlycompiled before any practice | 21.0% | −6.9 | |
| Enterprise Brain notes, with a goal | 28.2% | +0.3 [−4.8, +5.8] | $0.070 |
| Enterprise Brain notes | 30.9% | +3.1 [−3.4, +10.0] | $0.069 |
Learned notes beat the compiled procedure they started from: +7.2 [+2.1, +12.7] with a goal and +10.3 [+4.1, +17.2] without.