Fixing software issues with lessons from related ones.

An agent gets a GitHub issue and the repository, and must write a patch that passes the tests. With notes learned from 1,006 earlier tasks in the same repositories, Haiku resolved 30.6% of 217 new issues, up from 27.6%.

Issues resolved30.6% from 27.6%+3.0 pts · 95% CI +0.5 to +6.0

Issues won / lost18 / 7against no memory

Cost per run$0.023 from $0.020Haiku 5.5, two runs per issue

At a glance

Does experience from earlier fixes help on a related new issue?

Made by: SWE-ContextBench, a public benchmark built on real GitHub issues, where earlier tasks in the same repositories come with a record of past attempts.
The task: Write a patch for a GitHub issue, starting from the repository at its base commit, with no git history or web access, in 900 seconds.
Example: File extensions should be matched without case sensitivity. … I recently experienced this with a file name that had the extension SQL instead of sql and bat did not automatically apply the desired highlighting.
Data: 357 target issues across 40 repositories, and a history of 1,006 earlier tasks.
Practice vs test: Practice was one Haiku run on each of the 1,006 history tasks (739 resolved). Targets fixed by the same pull request as a history task were left out, leaving 220, of which 217 were scored, each run twice. A scan removed any note holding a target's id or code from its fix.
What's compared: No memory. OpenViking search, with and without its own learning. Enterprise Brain search with the learned note, with and without a goal. No compiled procedures.
Models: The agent is Claude Haiku 5.5. Claude Opus 5.5 wrote the notes.
Graded by: The benchmark's tests: an issue is resolved when the failing tests now pass and the passing tests still pass.

Results

Notes resolved more issues.

217 scored issues, two runs each.

Arm	Resolved	vs no memory	Cost per run
No memory	27.6%		$0.020
OpenViking search	29.7%	+2.1
Enterprise Brain notes, with a goal	30.0%	+2.3 [−0.5, +5.3]
Enterprise Brain notes	30.6%	+3.0 [+0.5, +6.0]	$0.023
Sonnet 5.5 alonea larger model, no memory	35.5%		$0.163