- Enterprise Brain
- Benchmarks
- CRMArena-Pro
Benchmark · CRMArena-Pro, public
Answering CRM questions by learning how the data is kept.
A SQL agent answers business questions against a simulated Salesforce company. After reading notes learned from 660 graded practice questions, the same Haiku agent answered 76.7% of 330 new questions right, up from 57.6%, in fewer steps.
Questions answered right76.7% from 57.6%+19.1 pts · 95% CI +14.5 to +23.6
Questions won / lost68 / 5on the same 330 questions
Steps per question3.9 from 5.1cost for all 330: $2.18 → $1.10
At a glance
One question, one database, one SQL tool.
- Made by
- CRMArena-Pro, a public benchmark: a simulated Salesforce company with 27 record types and 29,221 records.
- The task
- Answer one business question by querying the CRM with SQL, while following a written policy.
- Example
During the winter of 2023, which agent took the longest average time to move opportunities from start to finish? Return only the Id of the agent.
- Data
- 22 kinds of question: 660 practice questions (30 per kind) and 330 scored questions (15 per kind).
- Practice vs test
- Within each kind, the first 30 questions are practice and the next 15 are scored. Memory saw each practice run, its grade, and the right answer when the run was wrong.
- What's compared
- The schema and one read-only SQL tool, up to 30 turns, with and without one
get_advicecall first that returns the learned note for that kind of question. - Models
- The agent is Claude Haiku 5.5. Claude Opus 5.5 wrote the notes.
- Graded by
- Automatically, with no model judge: an answer is right when its words overlap the right answer enough (token F1 of at least 0.5). Exact match shows the same gain.
Results
Right more often, in fewer steps, for half the cost.
330 scored questions.
| Arm | Right | Exact match | Cost, all 330 |
|---|---|---|---|
| SQL only | 57.6% | 51.8% | $2.18 |
| SQL + Enterprise Brain notes | 76.7%+19.1 [+14.5, +23.6] | 70.9% | $1.10 |