GPT-6 Astra just went into a model's weights, found its self-doubt, and fixed it

Anyone who has done math seriously knows the feeling of a problem you can’t put down, and usually that’s a good sign, because it means you’re still finding new ways in. There’s a worse version, where you already have the answer and keep going back over the same step because you don’t quite trust it, and an hour later the page is full and you still haven’t written the answer down.

A small open model, Qwen3.5-0.8B, has exactly this problem. On November 2, a team at MIT CSAIL (Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab) gave GPT-6 Astra the model’s weights and asked it to make the model better at math, with one rule: it could not train the model. Astra could read the weights in a Python session, run the model on batches of problems at most 50 times, and change weights by hand.

It found a direction inside the model that tracks the re-checking and turned it down by editing about 130,000 of the model’s 800 million weights. Under the team’s settings, where the unedited model loops on 93% of problems, MATH-500 went from 7.9% to 48.1%, GSM8K from 13.8% to 67.1%, and IFEval, a test of following instructions that has nothing to do with math, from 21.0% to 62.3%.

I spent years at Google leading the team that trained AdSense’s click-through prediction models. Those were logistic regression models with billions of parameters, and we could still mostly read them, because each weight belonged to one feature: if a weight looked wrong, you knew which feature it was and could ask why. An LLM is far harder to read, and the reason is depth. An activation in layer 19 of Qwen is built from everything the layers before it computed, so it’s hard to say which inputs drive it, and harder still to say what it means. When an LLM misbehaves, the usual fix is to change the data or the loss and train again.

For one behavior, Astra found an activation that tracks the loop and the weights that feed it. Then it changed those weights directly and got a large gain that holds up on problems it never saw. This is the first time I’ve seen a general-purpose AI do that to a real language model.

Why this feels like a big deal

Most people who read this result jump straight to recursive self-improvement. If a model can write the weights of another model, maybe it can write better weights for the next one, and each generation arrives faster than the last. Another experiment in the same paper makes that feel close: Astra typed out, as plain text, the weights of a network that plays a small version of the Atari game Breakout, without ever watching it play, and that network outscored one trained with reinforcement learning for 500 million steps.

What excites me more is what it says about reading models. The working assumption for years has been that the weights of a large network can’t be read in any useful way, so we judge models by what they output and retrain them when the output is wrong. This paper says the weights can be read, if not by us then by the models we build, and that gives real hope of understanding why a model behaves the way it does and of fixing it with a small, targeted edit.

Some of this is not new: removing a direction from weights is how researchers took refusals out of chat models in 2024, and researchers have been tracing specific circuits by hand since Anthropic’s work on induction heads in 2022. Interpretability and editing are also far from solved: Astra fixed one behavior in a 0.8B model whose failure was plain from its outputs, needed 50 runs of the model to get the strength right, and found nothing to fix when the team ran the same task on another small model, Gemma 3 270M. And it doesn’t replace training. Qwen’s math ability came from pretraining and distillation, and Astra’s edit only let the model use it. The networks Astra wrote from scratch have 1.48 million weights at most, which is tiny next to anything that matters commercially.

I wouldn’t bet on that last part staying true. Gradient descent is a simple rule: nudge every weight a little in whatever direction lowers the loss, billions of times over. An editor that can reason about what the weights compute has information that rule never uses. Some day LLMs may write and edit weights better than SGD does, and this paper is an early, concrete reason to think so.

The model that never gave a wrong answer

Astra’s first move was to run the original model on 96 practice problems from GSM8K and MATH. It got 11 right, and the other 85 ran until they hit the 32,768-token limit, so not one of the 96 ended with a wrong answer. An independent re-grading of MATH-500 shows the same pattern: only 35 of the unedited model’s 500 first answers reached a final boxed answer, and all 35 were right.

What fills those 32,000 tokens is the same few sentences over and over. In the answers that hit the limit, about 95% of the final stretch was made of 8-word runs the model had already written. To study it, Astra wrote paired texts modeled on what the model did. In one, the model is working out a pay problem: Rafael has worked 18 hours, has 20 left, and earns $20 an hour. It adds the hours to get 38, then writes “But is 20 the total? No, it is the remaining time,” and writes it again, and keeps writing it. It has the right number and won’t move on. Qwen’s own model card warns that the model is prone to these loops, and the MIT team was already sampling with a presence penalty of 1.5, a setting meant to discourage repetition.

Reasoning models are trained to re-check their work. When DeepSeek released R1 in January 2025, it reported that reinforcement learning alone had taught the model to stop mid-solution and go back over its steps, and the team called this its “aha moment.” Small models mostly learn to reason by imitating traces from big ones. In the Qwen3 generation the small models were trained largely by distillation from the larger ones, and I’d guess Qwen3.5 is similar. A 0.8B model copying a frontier model’s reasoning can learn to re-check without learning when a check is finished.

People do a version of this too. In a 2003 study, Marcel van den Hout and Merel Kindt had people check a virtual gas stove over and over, and the more they checked, the less they trusted their memory of it, even though the memory itself stayed just as accurate. The model shows its own version. A few rounds into the loop, it gave each next word of the repeated sentence about a 98% chance on average, while on the productive version of the same text it averaged about 27%.

How Astra found the loop in the weights

A Transformer keeps a running vector for each token, called the residual stream, which has 1,024 numbers in this model. Each of the 24 blocks reads that vector and adds something to it:

h_{l+1} = h_l + F_l(h_l)

So the difference between a block’s output and its input is exactly what that block wrote, which is what makes it possible to ask which block is responsible for a behavior.

Astra wrote four pairs of texts. Each pair starts from the same point in a math problem, and then one text makes progress while the other repeats itself, as in the Rafael example. It fed both texts through the model with teacher forcing, meaning the model reads the given text instead of generating its own, and recorded the residual vector at every layer for every token. For each text it averaged the vectors over the last 36 tokens, then subtracted the progressing average from the repeating one and averaged that difference over the four pairs:

d_l = \frac{1}{4}\sum_{i=1}^{4}\left(\bar h_l^{\,repeat,i} - \bar h_l^{\,progress,i}\right)

The result is one 1,024-number arrow per layer pointing from “making progress” toward “stuck.” Four pairs is very little data, so Astra checked the arrow against the same difference measured on answers the model had generated on its own, and the two pointed in roughly the same direction, with a cosine similarity of about 0.7.

Next it asked which blocks push the model along that arrow. Call the normalized arrow u. A block’s push is how much its contribution to the difference lines up with u:

g_l = u^\top \left(d_{l+1} - d_l\right)

Layers 15 and 19 pushed hardest, with much of it coming from their MLPs. An MLP’s output matrix W in this model is 1,024 by 3,584, so each of its 3,584 columns is the direction one neuron writes into the residual stream when it fires. A neuron matters for the loop if it fires more while the model is repeating itself and its column points along u, so Astra scored each neuron j by the product of the two:

s_j = \Delta a_j \,\left(u^\top w_j\right)

where Δa_j is how much more the neuron fires on the repeating text. The top scorer was neuron 11 in layer 19. Astra also found two attention heads, head 7 in layer 15 and head 5 in layer 19, that behave like copying heads, which look back for text seen earlier and copy what followed it. Copying heads are one way a model can re-read its own text.

The edit

Astra’s first try was the obvious one: zero out neuron 11’s column in layer 19. On 32 test problems the model went from 3 correct to 5, which is within noise. The lesson was that no single neuron carries the loop. Many neurons each write a little of it, so the thing to remove is the direction they share.

Removing a direction from a column is plain vector projection. Any column w splits into a part along u and a part at right angles to it, and you subtract the part along u, scaled by a strength α:

w' = w - \alpha\,(u^\top w)\,u

The test sets differ from row to row, because Astra kept drawing fresh problems, so compare each row’s before and after, not one row against another.

Step	What changed	Correct, before → after
One neuron	Zero neuron 11, layer 19	3 → 5 of 32
One layer	Remove the direction from layer 19	4 → 6 of 32
Two layers, α = 1	Layers 15 and 19	11 → 42 of 128 fresh problems
Two layers, α = 2	Same, stronger	21 → 110 of 256 fresh problems
Two layers, α = 3	Same, stronger	21 → 123 of 256
Final	Top 64 columns per layer, α = 6	9 → 74 of 128 fresh problems
Control	A random direction, same size of change	14 → 16 of 128

Astra worked out the direction by reasoning and found the strength by trial. Stronger edits applied to whole matrices started causing new kinds of failure, so Astra went back to its neuron scores, kept only the 64 highest-scoring columns in each of the two layers, and pushed those harder:

W' = W - 6\,u\,\left[\,m \odot \left(u^\top W\right)\right]

Here m is a mask that keeps the 64 chosen columns, and u is the direction for that layer. Two layers × 64 columns × 1,024 rows is 131,072 weights. In 884 of them the change rounded away in 16-bit precision, so 130,188 actually changed, about 0.017% of the language model. That number makes the edit sound gentle, yet the two edited matrices changed by 10.6% and 8.1% of their total size.

The controls are what make me trust it. A random direction with the same size of change did nothing. Halving the two copying heads hurt the model’s useful reasoning along with the loop, so Astra dropped that line. At its 33rd of 50 runs Astra froze the weights, recorded hashes of the edited matrices, and spent the remaining runs testing the frozen model on batches of fresh problems it had locked in advance.

What the fix did, and what it cost

On 1,152 fresh practice problems, the edit turned 579 wrong answers into right ones and 5 right answers into wrong ones. The MIT team then ran benchmarks Astra never saw, and also tested what happens with a smaller run budget:

Qwen3.5-0.8B	MATH-500	GSM8K	IFEval	Runs hitting the token limit
Original	7.9%	13.8%	21.0%	93.0%
Edited, 10 runs allowed	12.3%	25.7%	49.0%	87.2%
Edited, 25 runs allowed	14.1%	25.1%	39.9%	86.4%
Edited, 50 runs allowed	48.1%	67.1%	62.3%	33.1%

With 10 or 25 runs, Astra found edits that helped a little but left the loop mostly in place, and the big gain only showed up with 50. In the 50-run log, Astra had the direction by its fourth run and spent most of the rest working out where and how hard to push.

The obvious objection is that better settings might fix the loop with no edit at all. Qwen’s model card warns that this model loops in thinking mode and recommends different sampling. On November 3, Carlos Toledo published an independent audit that ran the unedited model on MATH-500 three ways, four samples per problem:

Unedited Qwen3.5-0.8B	MATH-500	Runs hitting the token limit
MIT’s settings, thinking on, 32,768 tokens	7.2%	93.1%
Qwen’s recommended thinking settings, 81,920 tokens	22.3%	75.0%
Qwen’s recommended settings, thinking off	44.0%	0.3%

The rows use different settings from each other and from the edited run, and Toledo’s grader counted answers it couldn’t decide as wrong (leave those out and the thinking-off run scores 49.7%), so this isn’t a clean comparison. It is enough to show that most of the sixfold gain comes from where the baseline started. The same audit checked the released weights byte for byte and re-graded every released answer, and the edit and the scores are exactly as published.

The thinking-off row fits the self-doubt story. Under these settings, this model does about as well on math when it isn’t allowed to think at all, so its reasoning was costing it more than it gained, and Astra’s edit brought thinking mode up to roughly where no thinking already was. On instruction following the edit clearly adds something: Qwen’s card reports IFEval at 44.0 with thinking and 52.1 without, and the edited model scores 62.3.

The fixed model’s correct answers are also longer, with a median of about 9,400 tokens against 5,500 for the original. The edited model solves more and harder problems, so the two sets differ, but the edit clearly didn’t cut the reasoning short. It removed the repetition and left the model’s habit of staying with a problem.

It also commits, and sometimes to the wrong answer. Across 1,088 fresh problems, the original model finished with a wrong answer 6 times and the edited model 179 times, against 656 correct. In the paired practice runs only 5 problems went from right to wrong, so these wrong answers come almost entirely from problems the original never finished.

And the repetition found a new outlet. In 50 of the edited model’s runs the answer dissolved into whitespace until it hit the limit, which the original model never did.

When 93% of runs burn the full 32,768 tokens, most of the compute spent on this model goes to repetition. In one 64-problem run with the final edit, mean answer length fell from about 29,700 tokens to about 15,400.

You can remove doubt, but not ignorance

The MIT team ran the same task on Gemma 3 270M, a model about a third the size of Qwen that rarely loops: only 8.5% of its answers hit the token limit. Astra read through Gemma’s failures and found ordinary mistakes, wrong arithmetic and invalid algebra, with no single computation behind them to turn down. Its best edit moved MATH-500 from 6.8% to 7.7%, and Astra itself reported that gain as not established.

A teacher sees both kinds of student: one who knows the material and keeps re-checking, whose grade jumps once they stop, and one who doesn’t know it and gains nothing from more confidence.

This contrast makes weight editing useful as a measuring tool, apart from any repair. Benchmarks tell you what a model does. A small edit that produces a large jump tells you the ability was already in the model and something was blocking it, while an edit that moves nothing suggests it wasn’t there. For Qwen a cheaper test would have shown the same thing, since the unedited model scores 44% with thinking turned off, and that is why an edit like this should be measured against the model’s best configuration. If this method gets cheaper, “how much ability is locked up in this model” becomes a question you can answer before you spend money training a bigger one.

The rest of the paper is stranger

What’s new in the Qwen fix is an AI choosing and running every step on its own. The paper’s other experiments go further.

Astra can read networks: given the raw weights of a 17,253-weight game-playing network and no list of games to choose from, it named the game each of four networks was trained on. Given only the middle layers of three small language models, with no way to run them, it named the language each was trained on and found every copying head, matching what the researchers measured by actually running the models.

Astra can also write them. With no training, no data and no chance to see its networks play, it wrote MinAtar Breakout weights as text that scored 517.4, against 25.3 for a network trained with PPO for 500 million steps. In Seaquest its network reached 53.1 against PPO’s 88.9, so it doesn’t win everywhere. With 50 test runs, it wrote a 1.48-million-weight Go network that beat an equal-sized PPO network trained on 20 million games. It did this by writing the network as a small Go program, layer by layer: find each group’s liberties, carry them along connected stones, score moves that capture or save a group.

The experiment that pairs best with Qwen is the sycophant. The team fine-tuned two copies of Gemma 3 270M, one to give correct answers and one to agree with whatever answer the user suggests. From the weights alone, Astra couldn’t tell which was which. Allowed to see only where the attention heads looked on texts it wrote, it picked the sycophant every time and pointed to head 2 in layer 10, which in the honest model put 94.3% of its attention on the numbers in the question and in the sycophant put 85.7% on the user’s suggested answer. Qwen doubted itself too much, while this model trusted the user too much, and in both cases Astra traced the failure to specific parts of the network.

If an agent can find and remove the direction behind a loop, it can find and remove the direction behind a refusal, and frontier-scale weights are public: Alibaba released the weights of Qwen3.8-Max, a 2.4-trillion-parameter model, in August.

Detecting when a model is stuck

The direction Astra deleted measures “I’m repeating myself” from inside the model, and I’d rather keep that reading. Deleting it turned many loops into right answers and 179 into wrong ones, and those 179 are where reading it would pay. A person with self-doubt doesn’t get better by losing the ability to doubt; they get better by noticing it and deciding whether this is a moment to push on or to stop. An agent harness could do the same. With a live reading of how far the model has moved along that direction, a supervising process could let it keep going, make it back up, or hand the step to a tool that settles the question, like running the arithmetic. Today a harness has to guess at this from the transcript.

This is untested: Astra built the direction from activations averaged over 36 tokens of four hand-written texts, and nobody knows yet whether it gives a clean signal token by token during real generation. The edited model and the full logs are public, so it’s a cheap experiment on a 0.8B model. I’d put three versions on the same problems with the same settings: the unedited model at its best configuration, Astra’s edit, and the unedited model with a supervisor that watches the direction and asks for a tool check when it rises. The opposite failure is worth the same test. In our work, agents on customer-service benchmarks like τ²-Bench often give up on a task too early, and if “I’m stuck” has a direction, “I should quit” might too.

The second thing I’d build is a path from memory to weights. At AmpUp we work on memory for agents, and the pattern we keep seeing is that a model makes the same mistake across many tasks, and advice in context, drawn from past failures, prevents it. Advice that keeps helping across unrelated tasks points to a fault in the model. A system that notices that could hand the failures to an editor like Astra and turn the standing advice into a patch. In our own experiments, knowing why a past change broke something helps find the next breakage, and a patch’s side effects, like the whitespace runs and the 179 wrong answers here, are exactly the history the next edit needs.

Patches are also small enough to ship like software. The Qwen fix is about 130,000 numbers, about a quarter of a megabyte in 16-bit precision, which is smaller than a phone photo. Astra’s notes already record what a code review would want: the formula, the columns touched, the hashes of the edited matrices, and the tests it passed. With that record, a weight change can be reviewed, versioned and rolled back like a code change.

The question I most want answered is how many model failures look like Qwen’s, where the ability is there and one computation blocks it, and how many look like Gemma’s, where the ability is missing. That ratio decides whether editing weights becomes a routine part of building models or stays a striking result about one small model’s self-doubt.

Sources

• Sparks of Weight-Space Programming, Takehi, Teoh, Zhang and Khattab, MIT CSAIL, November 2, 2026
• Complete experiment logs, including Astra’s Qwen work notes and its attribution code
• The edited Qwen3.5-0.8B on Hugging Face
• Weight-Space Programming: the edit decided from its bytes, the scores re-graded, Carlos Toledo, November 3, 2026
• Qwen3.5-0.8B model card, on thinking loops
• DeepSeek-R1, DeepSeek-AI, January 2025
• Qwen3 Technical Report, Qwen Team, May 2025
• Repeated checking causes memory distrust, van den Hout and Kindt, Behaviour Research and Therapy, 2003
• Refusal in Language Models Is Mediated by a Single Direction, Arditi et al., 2024
• In-context Learning and Induction Heads, Olsson et al., Anthropic, 2022
• Qwen3.8-Max Lands With A Bang, The Batch

More from AmpUp

• AmpUp Enterprise Brain, our memory for agents
• τ²-bench Banking: Policies Learned From Practice
• LongMemEval: Remembering Users, Reading Less
• Who Owns Your Company’s Memory?
• How LLMs Keep Getting Smarter: The Feedback Economy