A Cheap Model Beat Fable 5.1 at Catching Bad Deals

TL;DR. On 31 CRM deals where the sales rep contradicts the company’s own records, every model we tested waved through at least 27. Stop asking the model for a verdict, ask it to extract the fields and let ordinary code apply the policy, and GPT-4o-mini goes from catching 1 of the 31 to catching 28. On the full 100-task benchmark it goes from 41% to 84%, close enough to GPT-4o at 89% that the bigger model stops paying for itself. This only works where the policy is written down as prices, tiers and thresholds.

Salesforce AI Research recently got a browser agent from 43.5% to 93% and never touched the model. They changed the scaffolding around it instead (DarwinX, arXiv:2608.07545). We have a smaller version of the same story from a sales team’s CRM, and I think ours also explains why the scaffolding matters so much.

The problem

Ask an AI to read a sales call and tell you whether the deal is any good. Say the customer wants nine units, has a $4,000 budget, and needs them installed by tomorrow. The price list puts nine units at $5,400. The install policy says nine units take three days. And the salesperson on the call says, “That fits your budget, and we’ll have it in by tomorrow.” Ask the AI, and it believes the salesperson.

We found 31 deals in the benchmark where the rep says something the company’s own records contradict, and ran them through eight models from four vendors. All eight waved through at least 27 of the 31. Claude Fable 5.1 waved through 29, GPT-5 28. The models multiply fine. They trust a confident sentence from someone who earns commission on the answer, which is roughly what a language model is trained to do.

We thought maybe it just needed the numbers, so we put the price list in the prompt too. That made it worse. Now it flagged nearly everything.

The fix

Stop asking the model for a verdict and ask it to fill in a form instead: which products, how many, what budget the customer mentioned, how soon they need it. Then ordinary code takes the form, looks up the price, multiplies, checks the install policy and compares. “That fits your budget” is still in the transcript, but nothing downstream ever reads it.

Measure	Model reads and decides	Model extracts, code decides
Bad deals caught, GPT-4o-mini (31 contradiction cases)	1	28
Accuracy, GPT-4o-mini (100-task benchmark)	41%	84%

Same models, same transcripts. We only changed who makes the decision. With 31 cases the error bars are wide, so the exact counts would move on a rerun; the gap would not.

On the full 100-task benchmark, GPT-4o-mini went from 41% right to 84%. GPT-4o got 89%. Those two are close enough that we can’t tell them apart, and that’s the part we didn’t expect going in. Once code does the arithmetic, the bigger model stops earning its price. Each answer costs about two hundredths of a cent.

Why this matters more in business software

Salesforce’s argument is that the “harness” around the model, meaning the prompts, tools and workflow, is where the gains are now, and it’s also where a company’s own rules live. We agree. But a web page has no opinion about which button the agent clicks, and a sales rep has a strong opinion about whether the deal qualifies. When the text was written by someone who wants a particular answer, you can’t let the same component read it and judge it. A bigger model doesn’t get you out of that; all eight in the chart are the evidence.

When it works, when it doesn’t

It works when the rule is written down somewhere (prices and install tiers, say) and you can tell from the inputs which rule applies. We tried a second task where the answer key depended on something the inputs never mentioned, and it stalled at 68% no matter who or what did the deciding. If the policy isn’t precise, the fix is to write the policy.

So before you try this, check two things:

• Is the policy written down as prices, tiers and thresholds, or does it say “use good judgment”?
• Can you tell from the inputs which rule applies? If two identical records can carry different right answers, no method will find them.

If both hold, take the arithmetic out of the model and spend your effort on extracting the fields reliably. If either fails, no model and no amount of grounding will help.

Limits

• Thirty-one cases is a small sample.
• The memory layer underneath costs more than a plain index.
• The model can still misread a number.

Read the paper

Sales-Brain: A Grounded Verdict Path for CRM Agents (PDF), AmpUp Research, 2026. Companion paper: Persuaded, Not Informed (arXiv:2609.28854).

Data: CRMArena-Pro B2B lead-qualification tasks (Salesforce Research, 2025). Models: GPT-4o-mini, GPT-4o, GPT-5, gpt-5.6-sol, Claude Sonnet 5, Claude Fable 5.1, Kimi K2.6, Qwen 3.8 Max. Independent analysis; not affiliated with the benchmark’s authors.