An assistant that finishes the job, after a few graded tries.

DolphinBench, the 480 held-out tests: Haiku with learned notes and the current memory search; Sonnet read the full history as files. Difference −0.2 pts (95% CI −4.5 to +4.1).

Made by

DolphinBench, a public benchmark from Mem0. An assistant acts for one person who has five years of dated messages.

The task

Carry out a dated request through simulated email, Slack, calendar and CRM tools.

Example

[2028-01-01] Create a January 9, 2028 9:30-10:30 AM Eastern cortado-and-crossword event at my usual Park Slope cafe, including enough location detail to identify it.

Data

3 people, about 500,000 tokens of history each, and 200 requests per person.

Practice vs test

For each person: 20 practice, 20 validation and 160 test requests, so 480 test requests in all. Requests that depend on the same source message stay on the same side.

What's compared

The history as monthly files, read with a shell. Enterprise Brain search, with source lookup and 8 prefetched facts. Search plus procedures compiled from the history. Search, procedures and the learned note for the request's kind, served before the first turn. Also Sonnet 5.5 reading the files.

Models

The agent is Claude Haiku 5.5. Haiku 5.5 extracted facts; Claude Opus 5.5 compiled procedures and wrote the notes.

Graded by

DolphinBench's own checks on the log of tool calls, with meaning-based checks judged by GPT-6 Sol. A request passes only if every check passes.