Training a Small Model on Your Agents' Advice

TL;DR

AmpUp Enterprise Brain already writes advice: for each kind of task, the steps that worked, the pitfalls that sank earlier runs, and how many runs taught it. Each session also closes with a transcript and an outcome. That is the same shape as the study notes and graded practice in Engram’s Learning in a World of Notes, where training a smaller model to use notes raised accuracy 7 to 20 points and cut the cost per query from $0.1677 to $0.0608. Training on a company’s own advice could move much of its repeated work to a small model, and the saving grows with volume.

Why does an assistant spend so much time catching up?

An assistant can spend much of its time catching up: finding the current policy, reading the history of a deal, or working out which document replaced another. When the same workspace supports thousands of questions, that repeated reading becomes expensive.

AmpUp Enterprise Brain keeps company memory in your own cloud data platform, and it learns advice from the work agents do there. That advice, with the records and outcomes behind it, could train a smaller assistant to do the repeated work for less.

Engram’s Learning in a World of Notes found that training a model to use notes improved both accuracy and cost. Those results come from Engram’s own workspaces and say nothing yet about Enterprise Brain; I cite them as evidence that the approach works. Enterprise Brain supplies the memory layer, and the training and evaluation pipeline described here would build on it.

What is advice, and why does it look like training data?

When an agent asks Enterprise Brain for advice, it gets the note for that kind of task: the steps that worked, the pitfalls that sank earlier runs, and how many runs taught it. When the session ends, it closes with a short transcript and its outcome, so the advice it served is judged, and a good outcome teaches it.

Engram’s training needs three things, and advice-style memory already holds each one:

What Engram’s training needs	What Enterprise Brain holds
Study notes the model reads before answering	Advice for each kind of task
Practice graded for being right and for cost	Sessions closed with a transcript and an outcome
The same library for every attempt	Pinned snapshots of memory in your own cloud

Advice is short and written for an assistant to read, so the assistant doesn’t have to work out the steps from the full history each time. Training teaches a model to use that advice well, and that is what lets a smaller, cheaper model take on the work.

What does training on memory look like? Think of a library and a student

Imagine a student with a school library, a study guide, and a pile of practice questions. The library holds the full story, and the guide helps the student find what’s useful quickly. Practice teaches the student when the guide is enough and when to open the original book.

In this picture, your cloud storage holds the library, and Enterprise Brain organizes the records and how they relate. The assistant is the student, and training improves its habits.

What does the stored data contribute?

Enterprise Brain stores facts in Apache Iceberg tables in your own catalog and keeps the source records those facts came from. On Google Cloud, setup puts those tables in your catalog and bucket, so the underlying storage is Cloud Storage. The object store holds the data; Enterprise Brain’s catalog, history and read rules make that data usable as memory.

Each part has a job in a training pipeline:

Stored material	How it helps the student
Original documents and conversations	The books to consult, and the evidence used to check an answer
Facts, procedures and learned advice	Ingredients for a concise study guide
Versions, timestamps and source references	A way to tell yesterday’s answer from today’s
Task sessions and outcomes, where recorded	Candidates for realistic practice questions, and examples of failure
A pinned workspace snapshot	The same library for every attempt in an experiment

A stored answer is only a candidate until someone checks it, and a record that advice was used doesn’t show the task succeeded. Training needs checked outcomes, so uncertain examples stay out of the reward signal.

How do you turn cloud memory into practice?

Start with a small, useful subject: one repeated kind of work, such as answering questions about customer approvals or helping engineers find current setup instructions. Select the records the assistant is allowed to learn from and freeze a snapshot. Serve the trained model only to people who could read that training scope, because model weights don’t inherit the store’s per-record access checks.

Then prepare the study materials. Advice is the study guide for each kind of task; add notes that summarize useful facts and point back to their sources, and keep the originals available. A good note makes evidence easier to find, but it can’t promise that nothing important was left out.

After that, write practice questions from real requests and from the records themselves. Sessions that closed with a recorded outcome are a natural place to start, because the advice served in them has already been judged. Each one gets a checked answer, supporting evidence and a clear grading rule, and only the grader sees the answer key. Hold back a separate set of questions (and their close variants) for the final test, and keep them out of note building.

Take a fictional customer whose discount was requested on Monday and approved on Thursday. “Was the discount approved?” has different answers on those two days, and the same cloud history can produce both exercises. If the note still says “pending” on Thursday, the student has to learn to check the approval record.

How does the assistant practice?

Give a trainable model the question, the notes, and tools for reading the source snapshot. It makes several attempts. Each complete attempt includes its searches, what it reads, its answer, and the resources it uses.

Grade each answer for correctness and account for its cost. An attempt that finds the right evidence with a focused lookup should beat an equally correct attempt that rereads the whole library. Never reward a short answer that’s wrong.

A reinforcement learning update uses those scores to make useful behavior more likely. A small adapter can hold the changes, so the original model weights stay frozen. During the experiment, keep the notes, source snapshot, tools and prompt fixed, so you can measure what training did.

After training, compare the adapted model with the original using the same notes, tools and test questions. Check accuracy, evidence quality, latency and cost. Also test what happens when a note is missing or a source changes, and deploy only when the trained version meets your quality bar.

The trained assistant can keep consulting Enterprise Brain as new records arrive. Updating the memory and retraining the model are separate decisions.

What did Engram’s study show?

In Learning in a World of Notes, Engram trained with reinforcement learning (GRPO) and LoRA adapters. A separate agent built the notes; the trained agent only read them. Across three workspaces, the authors report accuracy gains of 7 to 20 percentage points over the same agent equipped with notes before any training. On their law-firm workspace, a model trained once kept its advantage as the notes changed. (Study setup and results)

That gives us a concrete experiment to run: can an assistant learn to search narrowly, notice when its notes are incomplete, and check changing facts without rereading everything? The stored memory is the environment, and curated tasks with reliable grading supply the learning signal.

When is training worth doing?

The best candidate is a workspace many people ask about again and again, with tasks that stay useful long enough to repay the investment. A smaller trained model has to meet the quality bar, and its savings have to survive the cost of serving and maintaining it.

Start when you can name the recurring work, verify success, and measure enough query volume. If retrieval or note quality is the main source of error, fix that first. For occasional questions, poorly defined tasks or a short-lived project, a capable existing model with good memory may stay the better choice.

Changing facts alone don’t call for a new training run after every update. New tools, different kinds of work, or a measured drop in quality are better reasons to evaluate retraining.

How long does the training cost take to pay back?

Amortization means spreading one upfront expense over many later uses. If training saves ten cents on each question, every ten questions recover about a dollar of that expense.

The Harvey example in Engram’s post (a synthetic law-firm workspace) compares two models at about 78% accuracy: GPT-6 Sol at medium effort with notes, and a trained Qwen 3.8-27B with the same notes.

Item	Engram Harvey example
Extra training cost	$372.57
Comparison model with notes	$0.1677 per query
Trained model with notes	$0.0608 per query
Saving per query	$0.1069
Break-even volume	About 3,500 queries
At 400 queries per day	About 8.7 days

Both cost curves in Engram’s chart start from the same $1,894.49 of note building, so I treat that cost as common to both sides and leave it out of this incremental comparison. Engram estimates training cost from GPU price and utilization, so the payback above isn’t a full enterprise budget. (Engram cost analysis)

For your own deployment, add the extra data preparation, grading, deployment work and ongoing operating costs. Subtract only costs that truly apply to both alternatives. If your baseline has no notes, put the cost of building notes on the trained side and measure both alternatives again.

The calculator below starts from the Harvey numbers. Its extra project costs default to zero, matching the simplified comparison; those defaults aren’t an estimate for an Enterprise Brain deployment.

Payback days are the extra upfront cost divided by the net daily saving. The net daily saving is daily queries times the saving per query, minus any extra daily operating cost; if it’s zero or negative, training never pays back under those assumptions.

With the example rates, adding $5,000 of upfront work stretches the 400-queries-a-day payback to about 126 days. At only 40 queries a day, even the training-only example takes about 87 days.

What could a trained model save an enterprise in a year?

Payback is only the threshold. Past it, the saving on every question keeps adding up. At the Harvey rates, $0.1069 saved per question, this is the yearly saving before setup, serving and maintenance costs:

Repeated questions per day	Saving per year at Engram’s rates
400	About $15,600
2,000	About $78,000
10,000	About $390,000

This is my arithmetic on one synthetic workspace, not a forecast. It holds only for work the trained model handles at your quality bar; questions it can’t handle still go to the larger model. The best candidates are the tasks advice already covers: work that comes back often, with outcomes that show what a good answer looks like.

So train when the model meets your quality bar and the workload will stay useful for longer than the full payback period, counting likely refreshes and maintenance. The case is strongest when the same advice keeps being served for enough repeated work that the better habits pay for their own training.