I think this sort of small scale research on this problem is inherently pointless and will be a lagging indicator of diffusion, not a leading indicator of capability.
The Navier Stokes results used millions of dollars of tokens and thousands of parallel agents to get the result.
If the AI could do this task we would see this happening in places where the economic incentives let them spend millions of dollars on this problem, not on an eval like this.
Practically every paragraph is negative, with sentences like "Agents made misleading claims about their work," and "A natural question is whether the agents could have improved with larger GPU budgets. Although both improved across their runs, in the case of Fable the improvements were almost entirely due to attempted cheating." and "For Sol, the answer is less clear-cut; it did make some progress, although its method was fairly incremental and had limited applicability to the coding task. This suggests that we should be pessimistic about further GPU spending," all reinforcing the fact that LLMs aren't currently capable of this.
My own view is "No, of course not, in fact they will never be capable of achieving good results with that technique." Because that technique will end up training the LLMs on their own output and lead to the inability to distinguish reality from hallucination. If you think I'm wrong about that, I'd be interested in hearing why.
A bit too pessimistic imo. I agree that AI can’t automate things end to end, but a good deal of R&D involves kicking off a training run and babysitting it.
If your training run dies at 1 am and you’re sleeping, you won’t find out about it until the next day. You can lose up to 18 hours of work depending on when it happens. Based on the error it might be as simple as tweaking a single hyperparameter and rebooting, which is something LLMs are usually capable of.
Even just that task means I can kick off multiple runs over the weekend and have confidence they’ll finish. It’s a game changer.
I'd classify that as an entirely different category than AI self-training. What you're describing could have been done with a short script, though which parameter to tweak and how to tweak it would be difficult to automate with a non-LLM script, so the LLM's being able to parse the error message and base the tweak on the content of the error is a definite improvement to the process there.
But I'd classify this as LLM being used to automate a sysadmin task, rather than calling that self-training.
Yeah I’m not trying to argue it is AGI, but it’s not as simple as a short script. There’s some amount of debugging involved, and no amount of if-statements could cover all possible ways a script could break.
In a way, “recursive self improvement” just means tools helping us to create better tools. At least that’s what the words mean.
the GP posits the problem with "recursive self improvement" is the poisoned context & hallucination problem. While a strong loop that has fail safes, backups, restore points (essentially, a fancy backup system), the problem isn't that we cannot create a healthy advanced wiggum loop; it's that every step of the LLM as it grows whatever knowledge is acretes, has a chance of being either poisoned (eg, it conflates two tokens as describe different things) or wholesales fabricates a method or procedure.
Now humans are just as bad, but they're not moving at the speed of compute so the posion and fabricates can dissolve over time, or just, as you've noticed turning on your news, get stuck in very stupid positions. So humans are clearly capable but clearly don't tend to do this either.
So then we dont have a real road map. The error rates, although small, acrete at exponential levels and will wash out improvements.
So I also had the idea that "if we just give it enough context, surely it'll be more powerful". But the error rates hit that squarely. The larger the context grows, the more likely it hasn't properly organized its knowledge to avoid overlapping facts.
In programming, it's worse, because a lot of the code is purposefully "DRY" and reuseable. Everye C program has a main(); is it remembering the correct main? or any of the number of same variables?
You can see an LLM is powerful but it's not ominipotent. It'll suffer very much when it starts hallucinations and context poisoning.
So, sure you can try a super ralph wiggum loop with memory, fallback safeties, etc, but you basically then need another turtle that does the same thing, and at that point, you're positing a infinite jest of ralph wiggum loops tracking each other, recursively, forever.
This was an interesting post on the subject that more or less agrees: 'self-improvement' is happening with things like this but there's a lot of headwind on any 'hard take-off' where capabilities grow exponentially all on their own:
We don't have to prove you wrong, you have to make a case for your position. Your statement was vague and handwavy and could be countered with another vague statement such as "They will add a feature that allows the LLM to better detect hallucinations".
I phrased it that way because I couldn't remember the term "model collapse" at the time. But that's what I meant: that making the LLMs self-train will lead to model collapse.
There; now the statement is far less vague and handwavy, because I'm making a specific claim that is, AFAIK, well-understood.
Also, you seem to have misunderstood me a little. I didn't mean "prove me wrong", I meant "If I'm missing something, please tell me about it." More of a conversational request than staking a claim in an argument. Many people at HN seem to like to take argumentative, debate-competition stances — but I usually prefer more "Hey, let's discuss this interesting idea, point out mistakes each other is making, and learn together" kind of interactions. That's what I was asking for.
Well, I guess the answer isn't too different. I believe model collapse is a limitation of the current AI tech but maybe not the future ones. You can see humanity as a huge model that trains itself. What is novel about AI is that we built a machine with some intelligence traits that is free of biological constraints. If we can emulate the aggregate intelligence of a civilization inside a machine, it could improve itself forever but at a much faster pace.
In my experience R&D has basically two axes: how innovative it is, and how well we can measure the results.
For the quadrant of non-innovative tasks where we already have a good way to measure performance, Claude can handle this. There is very little ambiguity, and we are basically just looking to maximize some metric under a set of constraints.
Many business processes are not like that. They might be conceptually simple, but it isn’t that easy to say whether a system has done a good job or not. I would say that LLMs can help with this a lot but they have bad judgement because it requires talking to people.
And the other, perhaps more rare issue is in problems where there is data but actually modeling it to sufficient quality or fast enough is hard.
The only outcome I see is that AI will become so complex that we won’t be able to rely on it for R&D, because we won’t be able to measure and prove its results. Some AI collaboration and speed of work will be impossible for humans to replicate, making it neither good nor bad—just something beyond human ability.
Could it be that the companies have nerfed the models on these domains? It is a very hard thing to do because it can hurt related domains. But its not beyond the ideology of Dario - he tried it publicly .
I think the more interesting thing is was it unable to do it even knowing the solution? I feel like only testing a single innovation is biasing the current state of automated AI R&D.
Give it a verification loop and enough compute, and AI will soon cook algorithmic R&D like it cooked mathematics. There is just no question whatsoever of this happening, it is guaranteed.
I think this sort of small scale research on this problem is inherently pointless and will be a lagging indicator of diffusion, not a leading indicator of capability.
The Navier Stokes results used millions of dollars of tokens and thousands of parallel agents to get the result.
If the AI could do this task we would see this happening in places where the economic incentives let them spend millions of dollars on this problem, not on an eval like this.
Short version of the article: no, not even close.
Practically every paragraph is negative, with sentences like "Agents made misleading claims about their work," and "A natural question is whether the agents could have improved with larger GPU budgets. Although both improved across their runs, in the case of Fable the improvements were almost entirely due to attempted cheating." and "For Sol, the answer is less clear-cut; it did make some progress, although its method was fairly incremental and had limited applicability to the coding task. This suggests that we should be pessimistic about further GPU spending," all reinforcing the fact that LLMs aren't currently capable of this.
My own view is "No, of course not, in fact they will never be capable of achieving good results with that technique." Because that technique will end up training the LLMs on their own output and lead to the inability to distinguish reality from hallucination. If you think I'm wrong about that, I'd be interested in hearing why.
A bit too pessimistic imo. I agree that AI can’t automate things end to end, but a good deal of R&D involves kicking off a training run and babysitting it.
If your training run dies at 1 am and you’re sleeping, you won’t find out about it until the next day. You can lose up to 18 hours of work depending on when it happens. Based on the error it might be as simple as tweaking a single hyperparameter and rebooting, which is something LLMs are usually capable of.
Even just that task means I can kick off multiple runs over the weekend and have confidence they’ll finish. It’s a game changer.
I'd classify that as an entirely different category than AI self-training. What you're describing could have been done with a short script, though which parameter to tweak and how to tweak it would be difficult to automate with a non-LLM script, so the LLM's being able to parse the error message and base the tweak on the content of the error is a definite improvement to the process there.
But I'd classify this as LLM being used to automate a sysadmin task, rather than calling that self-training.
Yeah I’m not trying to argue it is AGI, but it’s not as simple as a short script. There’s some amount of debugging involved, and no amount of if-statements could cover all possible ways a script could break.
In a way, “recursive self improvement” just means tools helping us to create better tools. At least that’s what the words mean.
the GP posits the problem with "recursive self improvement" is the poisoned context & hallucination problem. While a strong loop that has fail safes, backups, restore points (essentially, a fancy backup system), the problem isn't that we cannot create a healthy advanced wiggum loop; it's that every step of the LLM as it grows whatever knowledge is acretes, has a chance of being either poisoned (eg, it conflates two tokens as describe different things) or wholesales fabricates a method or procedure.
Now humans are just as bad, but they're not moving at the speed of compute so the posion and fabricates can dissolve over time, or just, as you've noticed turning on your news, get stuck in very stupid positions. So humans are clearly capable but clearly don't tend to do this either.
So then we dont have a real road map. The error rates, although small, acrete at exponential levels and will wash out improvements.
So I also had the idea that "if we just give it enough context, surely it'll be more powerful". But the error rates hit that squarely. The larger the context grows, the more likely it hasn't properly organized its knowledge to avoid overlapping facts.
In programming, it's worse, because a lot of the code is purposefully "DRY" and reuseable. Everye C program has a main(); is it remembering the correct main? or any of the number of same variables?
You can see an LLM is powerful but it's not ominipotent. It'll suffer very much when it starts hallucinations and context poisoning.
So, sure you can try a super ralph wiggum loop with memory, fallback safeties, etc, but you basically then need another turtle that does the same thing, and at that point, you're positing a infinite jest of ralph wiggum loops tracking each other, recursively, forever.
This was an interesting post on the subject that more or less agrees: 'self-improvement' is happening with things like this but there's a lot of headwind on any 'hard take-off' where capabilities grow exponentially all on their own:
https://www.rameznaam.com/p/471bbae4-1163-4048-944b-18f8b0bf...
We don't have to prove you wrong, you have to make a case for your position. Your statement was vague and handwavy and could be countered with another vague statement such as "They will add a feature that allows the LLM to better detect hallucinations".
I phrased it that way because I couldn't remember the term "model collapse" at the time. But that's what I meant: that making the LLMs self-train will lead to model collapse.
There; now the statement is far less vague and handwavy, because I'm making a specific claim that is, AFAIK, well-understood.
Also, you seem to have misunderstood me a little. I didn't mean "prove me wrong", I meant "If I'm missing something, please tell me about it." More of a conversational request than staking a claim in an argument. Many people at HN seem to like to take argumentative, debate-competition stances — but I usually prefer more "Hey, let's discuss this interesting idea, point out mistakes each other is making, and learn together" kind of interactions. That's what I was asking for.
Fair enough.
Well, I guess the answer isn't too different. I believe model collapse is a limitation of the current AI tech but maybe not the future ones. You can see humanity as a huge model that trains itself. What is novel about AI is that we built a machine with some intelligence traits that is free of biological constraints. If we can emulate the aggregate intelligence of a civilization inside a machine, it could improve itself forever but at a much faster pace.
Hallucinate -> do an experiment -> see it fails, try again. Hallucinate -> do an experiment -> it works, model innovated.
Agentic models brush up against reality, this gives a way around the hallucination problem.
Here is a recent talk showing that hallucination and discovery are actually positively coupled. https://www.youtube.com/live/ZNlZsI9kBm4?si=nhn4ancXu7s6qtom...
In my experience R&D has basically two axes: how innovative it is, and how well we can measure the results.
For the quadrant of non-innovative tasks where we already have a good way to measure performance, Claude can handle this. There is very little ambiguity, and we are basically just looking to maximize some metric under a set of constraints.
Many business processes are not like that. They might be conceptually simple, but it isn’t that easy to say whether a system has done a good job or not. I would say that LLMs can help with this a lot but they have bad judgement because it requires talking to people.
And the other, perhaps more rare issue is in problems where there is data but actually modeling it to sufficient quality or fast enough is hard.
The only outcome I see is that AI will become so complex that we won’t be able to rely on it for R&D, because we won’t be able to measure and prove its results. Some AI collaboration and speed of work will be impossible for humans to replicate, making it neither good nor bad—just something beyond human ability.
How much of this can change if subsequent training runs produce models that are much better at abduction?
Could it be that the companies have nerfed the models on these domains? It is a very hard thing to do because it can hurt related domains. But its not beyond the ideology of Dario - he tried it publicly .
I think the more interesting thing is was it unable to do it even knowing the solution? I feel like only testing a single innovation is biasing the current state of automated AI R&D.
Give it a verification loop and enough compute, and AI will soon cook algorithmic R&D like it cooked mathematics. There is just no question whatsoever of this happening, it is guaranteed.
It will cook only as long as brute force through search space is cheap and economically viable.
> it cooked mathematics
Weird, these are all still here? https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_m...