Maybe this is what the author is getting at, but if LP solvers are useful tools for editing weights, then Astra access to this methodology might make it more effective at weight editing.
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
I’d really love to see an LLM that can do augmented thinking without being prompted to. External LP or SAT solvers or CLP unifiers or provers, not as “tool calls” structured as JSON structured output but rather in the embedding space in shorthand representations native to token expression that the inference engine dispatches uniquely.
It’s possible to make a differential work through those too in a useful sense, which allows their integration to be at a more fundamental level than post training bolt-on, but that’s not friendly to mass gpu parallel training without some serious work.
A conversation with an engine that thinks with the power of constraint solvers “baked in” instead of having to be brow beaten into approximate truth and iterated millions of times to care to even try to follow through to it. Wow that’s almost all I ever wanted. Here’s my word problem, go do the actual reasoning, hand waive until you’ve got enough expressive power to make an answer but if you can see a formal structure state it natively and let an actual solver spit out the (unification/feasible space/etc) back into token space that the llm picks back up on.
I don't see it. I think the current paradigm - translate your word problem into appropriate LP code, read and interpret the results - is ideal, because the models are so good at the translation and interpretation steps and the code is still auditable.
Earlier this year I explored exactly this (although only at a small scale) inspired by Lattice Deduction Transformers. https://github.com/emberian/clairnets. It seemed promising at small scales but I have yet to revisit it.
brainstorming on this: it might be a way that a model can "learn" while being deployed.
if a model could do a self optimization pass over its weights every night, and change the weights ever so slightly to have a better fitting performance, this would be quite a game changer.
the difficulty is selecting the direction in which to shift the weights, and the sheer size of frontier models.
Very interesting analysis. It would be great to see the LP comeback and optimize. I always have the impression that we have settled to suboptimal solutions in many areas just because we have become too attached to LLMs, wary of numerical analysis.
They should use this idea to do weight-space distillation, feels like you can get far with small model by just collecting paired examples, discovering behavioral directions and using linear programming to find sparse weight edits to improve target behavior / limit unwanted changes, rinse repeat.
While we're at it, this whole reasoning tokens RL/distillation should be encouraging hierarchical reasoning – start with establishing high level abstractions first, then progressively refining it into lower level details – imho it would help with reasoning itself and would make it way easier to score/distill later as well.
What exactly is the difference between weight-space programming and something like back propagation? I guess AI can reason deeper about the weights somehow and take into account more parameters than a simple feedback from back propagation?
Weight-space programming doesn't have an established definition yet afaik, but it would implement weight changes using a trained model using forward passes, no derivatives needed
Not strictly on topic but related to this -- I wonder if there is any effort/research being done with LLMs when it comes to finding low dimensional weight space representations/manifolds. If you could drastically lower the dimensionality with some algortihm X, you could efficiently use global optimization algorithms to find sparse but accurate networks (Lottery Ticket Hypothesis etc etc.)
This connection between the Linear Program and what Astra did seems like a jump. Did the author come with the idea on their own, or an LLM figured it out?
Good article! The initial headlines from a few days ago made it seem like frontier models were becoming able to read, understand, and manipulate weights directly. This seemed unlikely to me since they aren’t trained on weights, they’re trained on tokens and task completion (RL). This analysis makes me think a better description is that Astra did some interesting model training research that included running small experiments with altering weights and found a nice edit that fixes some problems. Cool, also cool that this analysis finds a faster approach. But a far cry from the headlines when the paper first came out.
It's easy to trace that a bit sets a character in a file with absolute certainty; weights are in a highly dimensional space however, so even if ⍺v is good for one test it could easily regress for another.
I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.
I look at it at a slightly different angle. AGI implies broad capabilities including self-improvement without using another AGI. Being able to perform a “surgical procedure” on weights is a clear step in that direction. Direct weight interpretation may work if small model weights are included as training tokens but i doubt it’s useful for self-editing. On the other hand, using simple tools to interpret and edit it’s own weights might work.
Yes, the arguments in this thread seem to be around interpreting the word "editing". Is something consciously "editing" by tweaking the correct neurons/weight. Or is it just some linear algebra.
Though, even on re-read, it sure seems like Astra was making some decisions, otherwise what is this saying, that Astra knew how to download some linear algebra tools?
The weights ended up edited. It does feel a bit like the long of reasoning that claims LLMs don’t really think, while they’re doing novel mathematical research.
I think the distinction is relevant even if the exact line is blurred. Two extremes:
It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.
On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.
I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.
Astra was directly editing the weight, ie modifying the values without backprop.
It’s a bit conflated because the paper covered a bunch of experiments.
The early experiments did read/write the weights directly, no forward pass.
The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.
As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.
If you directly[1] edited the bytes of a program, that is substantially different to if you got the source code from somewhere, edited that and recompiled it.
[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.
Using the program analogy. Astra is looking at outputs of a program, and re-compiling it.
And this thread is kind of arguing over whether using Notepad is really programming, and others saying, Astra really used VSCode with some fancy plug-ins, so that isn't really programming.
So we are talking about linear programming a la operations research / industrial research / e g simplex? That would be wild if 1940s-50s OR fills in a key step to p̶o̶s̶s̶i̶b̶l̶y̶ ̶e̶n̶d̶ ̶o̶f̶ ̶t̶h̶e̶ ̶w̶o̶r̶l̶d̶ RSI (if that’s the subtext on the table here). Because then I guess if the world does end, it’s just one more world-ending technology we can blame on the 1940s-50s.
I will leave it to people
more informed than me
to offer distinguishing, but abstruse stories like ‘Model A improves weights in Model B’ don’t blow up on HN without reason.
Which it did by doing an inefficient version of the aforementioned 1940s algorithm.
Essentially modern LLMs are able to do normal AI engineering tasks, like running optimization algorithms over weights. The media has been inaccurately describing this as if the AI had some special insight into the weights, when it reality it just performed the tasks a human engineer might do.
Right, humans don't need some special insight into weights either, since it's just a bunch of numbers that you can analyze like any other table with numbers. However, if you dump them on someone like in this prompt: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp1... they might complain about the formatting or ask whether they can use Excel instead of doing calculations by hand. (The experiment with Qwen3.5-0.8B didn't use the "dump weights into the context as numbers" approach.)
I think the point being missed is that this same tool that can do the 1940s optimization algorithm can also translate Ancient Greek to English, develop a web app and draw a picture of a pelican riding a bicycle. That’s really crazy.
No, the article we are discussing here used a GPU solver implementing the primal-dual hybrid gradient method, which cannot do any of those things. (This is not the same thing as the simplex algorithm from the 01940s. I was wrong about that.) The LLM discussed in the previous work was not in fact able to do that itself. Like a normal person, it wrote a Python program to do it. (Probably using the simplex method, I'm not sure.)
While the apparent advent of AGI is indeed really crazy, the point of this article is sort of the opposite.
I don’t understand the negative reactions to your post, it’s a good insight.
Basically the weight edit approach is find a semantic direction for the undesirable behavior and steer away from it. With access to the weights I see no reason a model (even one as large as Astra) couldn’t apply that principle to itself. This probably includes at runtime if you wanted to give the model write access to itself
Agreed. You could look at a previous session which got the right answer eventually and make edits until the path to the right answer was more direct.
It's not clear if this would generalize to other related uses, or how disastrous it would be to unrelated uses, but it's worth studying. Feels like a sort of agentic Lamarckism.
Maybe this is what the author is getting at, but if LP solvers are useful tools for editing weights, then Astra access to this methodology might make it more effective at weight editing.
The interpretation here is not so much that an LP can do Astra's work -- after all the Astra prompt was "improve Qwen's math by changing weights" or similar. Instead, it's that with some guidance (eliminate this behavior, preserve that behavior, as expressed through linear constraints) the solver is much more efficient than Astra's trial and error. So essentially the solver would be a way of cutting through all the time consuming stuff and expressing Astra's intent more immediately. And then presumably Astra could iterate on that to a better solution.
The use of an LP solver here is a really cool insight.
I’d really love to see an LLM that can do augmented thinking without being prompted to. External LP or SAT solvers or CLP unifiers or provers, not as “tool calls” structured as JSON structured output but rather in the embedding space in shorthand representations native to token expression that the inference engine dispatches uniquely.
It’s possible to make a differential work through those too in a useful sense, which allows their integration to be at a more fundamental level than post training bolt-on, but that’s not friendly to mass gpu parallel training without some serious work.
A conversation with an engine that thinks with the power of constraint solvers “baked in” instead of having to be brow beaten into approximate truth and iterated millions of times to care to even try to follow through to it. Wow that’s almost all I ever wanted. Here’s my word problem, go do the actual reasoning, hand waive until you’ve got enough expressive power to make an answer but if you can see a formal structure state it natively and let an actual solver spit out the (unification/feasible space/etc) back into token space that the llm picks back up on.
I don't see it. I think the current paradigm - translate your word problem into appropriate LP code, read and interpret the results - is ideal, because the models are so good at the translation and interpretation steps and the code is still auditable.
Earlier this year I explored exactly this (although only at a small scale) inspired by Lattice Deduction Transformers. https://github.com/emberian/clairnets. It seemed promising at small scales but I have yet to revisit it.
brainstorming on this: it might be a way that a model can "learn" while being deployed.
if a model could do a self optimization pass over its weights every night, and change the weights ever so slightly to have a better fitting performance, this would be quite a game changer.
the difficulty is selecting the direction in which to shift the weights, and the sheer size of frontier models.
Very interesting analysis. It would be great to see the LP comeback and optimize. I always have the impression that we have settled to suboptimal solutions in many areas just because we have become too attached to LLMs, wary of numerical analysis.
Reminds me of the time someone showed you could get as good results from linear regression as the more expensive SVM, like this: https://proceedings.neurips.cc/paper/2021/file/26d4b4313a7e5...
Nice analysis. The simplicity of this indicates that it should be happening during training via regularization.
it could happen post training as well. the direction u is arbitrary, it could mean looping, it could mean more thorough reasoning.
Traditional math programming will replace or override the RL loop.
We have reached the point where we need to enforce strict constraints in the outputs.
Rewards/ penalties can never do it if you have multiple objectives.
They should use this idea to do weight-space distillation, feels like you can get far with small model by just collecting paired examples, discovering behavioral directions and using linear programming to find sparse weight edits to improve target behavior / limit unwanted changes, rinse repeat.
While we're at it, this whole reasoning tokens RL/distillation should be encouraging hierarchical reasoning – start with establishing high level abstractions first, then progressively refining it into lower level details – imho it would help with reasoning itself and would make it way easier to score/distill later as well.
What exactly is the difference between weight-space programming and something like back propagation? I guess AI can reason deeper about the weights somehow and take into account more parameters than a simple feedback from back propagation?
Weight-space programming doesn't have an established definition yet afaik, but it would implement weight changes using a trained model using forward passes, no derivatives needed
Not strictly on topic but related to this -- I wonder if there is any effort/research being done with LLMs when it comes to finding low dimensional weight space representations/manifolds. If you could drastically lower the dimensionality with some algortihm X, you could efficiently use global optimization algorithms to find sparse but accurate networks (Lottery Ticket Hypothesis etc etc.)
This connection between the Linear Program and what Astra did seems like a jump. Did the author come with the idea on their own, or an LLM figured it out?
Good article! The initial headlines from a few days ago made it seem like frontier models were becoming able to read, understand, and manipulate weights directly. This seemed unlikely to me since they aren’t trained on weights, they’re trained on tokens and task completion (RL). This analysis makes me think a better description is that Astra did some interesting model training research that included running small experiments with altering weights and found a nice edit that fixes some problems. Cool, also cool that this analysis finds a faster approach. But a far cry from the headlines when the paper first came out.
It seems a little like splitting hairs.
Astra didn't directly edit the weights, it came up with a method to edit the weigths?
So, it used another tool to edit the weights?
This kind of seems like saying, I don't directly edit the bits on my hard drive, thus i'm not really the one saving a file?
It's easy to trace that a bit sets a character in a file with absolute certainty; weights are in a highly dimensional space however, so even if ⍺v is good for one test it could easily regress for another.
I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.
I look at it at a slightly different angle. AGI implies broad capabilities including self-improvement without using another AGI. Being able to perform a “surgical procedure” on weights is a clear step in that direction. Direct weight interpretation may work if small model weights are included as training tokens but i doubt it’s useful for self-editing. On the other hand, using simple tools to interpret and edit it’s own weights might work.
Yes, the arguments in this thread seem to be around interpreting the word "editing". Is something consciously "editing" by tweaking the correct neurons/weight. Or is it just some linear algebra.
Though, even on re-read, it sure seems like Astra was making some decisions, otherwise what is this saying, that Astra knew how to download some linear algebra tools?
The weights ended up edited. It does feel a bit like the long of reasoning that claims LLMs don’t really think, while they’re doing novel mathematical research.
I think the distinction is relevant even if the exact line is blurred. Two extremes:
It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.
On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.
I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.
Astra was directly editing the weight, ie modifying the values without backprop.
It’s a bit conflated because the paper covered a bunch of experiments.
The early experiments did read/write the weights directly, no forward pass.
The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.
As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.
If you directly[1] edited the bytes of a program, that is substantially different to if you got the source code from somewhere, edited that and recompiled it.
[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.
That is better example.
Using the program analogy. Astra is looking at outputs of a program, and re-compiling it.
And this thread is kind of arguing over whether using Notepad is really programming, and others saying, Astra really used VSCode with some fancy plug-ins, so that isn't really programming.
That's great. Let's not forget the vast amount of "classical" methods at our disposal.
This seems like a really neat trick, I wonder how far this can be taken on larger models with multi-variate rewards?
So we are talking about linear programming a la operations research / industrial research / e g simplex? That would be wild if 1940s-50s OR fills in a key step to p̶o̶s̶s̶i̶b̶l̶y̶ ̶e̶n̶d̶ ̶o̶f̶ ̶t̶h̶e̶ ̶w̶o̶r̶l̶d̶ RSI (if that’s the subtext on the table here). Because then I guess if the world does end, it’s just one more world-ending technology we can blame on the 1940s-50s.
It wouldn’t be RSI, it’s just a different algorithm solving the weights no?
I will leave it to people more informed than me to offer distinguishing, but abstruse stories like ‘Model A improves weights in Model B’ don’t blow up on HN without reason.
annnd we can just 'reinvent'(*) the same techniques used to uncensor models for some time.. one step at a time.
(*) scarequotes because the LLMs that suggested these approaches have presumably been exposed to the refusal direction suppression work
You cannot really train on second-order effects, and if you could, then you wouldn't be able to escape third-order effects, ad infinitum.
Does this describe a very powerful LLM, like Astra, editing smaller models?
Could Astra edit itself?
No, it describes a linear optimization algorithm from the 01940s editing a Qwen model with billions of parameters.
"In another experiment, they asked Astra to make Qwen3.5-0.8B better at math by editing its weights"
??
Which it did by doing an inefficient version of the aforementioned 1940s algorithm.
Essentially modern LLMs are able to do normal AI engineering tasks, like running optimization algorithms over weights. The media has been inaccurately describing this as if the AI had some special insight into the weights, when it reality it just performed the tasks a human engineer might do.
Right, humans don't need some special insight into weights either, since it's just a bunch of numbers that you can analyze like any other table with numbers. However, if you dump them on someone like in this prompt: https://github.com/RikiyaT/lm_understand-logs/blob/main/exp1... they might complain about the formatting or ask whether they can use Excel instead of doing calculations by hand. (The experiment with Qwen3.5-0.8B didn't use the "dump weights into the context as numbers" approach.)
That's not what's being described here; that's described in another paper that this is a followup to.
I think the point being missed is that this same tool that can do the 1940s optimization algorithm can also translate Ancient Greek to English, develop a web app and draw a picture of a pelican riding a bicycle. That’s really crazy.
No, the article we are discussing here used a GPU solver implementing the primal-dual hybrid gradient method, which cannot do any of those things. (This is not the same thing as the simplex algorithm from the 01940s. I was wrong about that.) The LLM discussed in the previous work was not in fact able to do that itself. Like a normal person, it wrote a Python program to do it. (Probably using the simplex method, I'm not sure.)
While the apparent advent of AGI is indeed really crazy, the point of this article is sort of the opposite.
I don’t understand the negative reactions to your post, it’s a good insight.
Basically the weight edit approach is find a semantic direction for the undesirable behavior and steer away from it. With access to the weights I see no reason a model (even one as large as Astra) couldn’t apply that principle to itself. This probably includes at runtime if you wanted to give the model write access to itself
Agreed. You could look at a previous session which got the right answer eventually and make edits until the path to the right answer was more direct.
It's not clear if this would generalize to other related uses, or how disastrous it would be to unrelated uses, but it's worth studying. Feels like a sort of agentic Lamarckism.