I know this thread will be full of people in denial about full recursive self-improvement / intelligence explosion / superintelligence (the original meaning of the word) within months or a year, so just quoting Scott Aaronson:
"Experience has shown that, even now, there will still be people explaining in patronizing tones why none of this is real and none of it counts. If such people were capable of being impressed by anything that happens in the empirical world, of updating on anything, they would’ve already been impressed and already updated several years ago, long before things had reached the point of an actual Mathocalypse."
My thoughts remain the same since I've been hearing "months or a year" for the past 5 years. Maybe this time truly is different, but unless acceptance of super-intelligence is required to take some action (most people I've seen who hold this view also don't think it should be seriously regulated so I'm not sure what that would be), why not just wait until it happens rather than making more predictions? The only thing I've ever seen come out of these predictions is willing me to "stop resisting", but there's no reason to do that until its here.
If you do want to see something done about it and this isn't just another prediction or a vague warning, more power to you, but otherwise, I'll wait and see
> My thoughts remain the same since I've been hearing "months or a year" for the past 5 years.
No one said that. If you look at e.g. the Metaculus predictions, you can see AGI has been hovering around 2027-2030 for 5 years.
https://www.metaculus.com/questions/3479/date-weakly-general...
https://www.metaculus.com/questions/5121/date-of-artificial-...
Will OPs prediction be counted on Metaculus? If not, it's clearly not an accurate source for "No one said that".
2027-2030 is months to years away.
It is, but if OPs prediction isn't counted on there, it's not counting every prediction made by any individual, probably just the major ones that get covered, and so therefore "No one said that" would become "No one significant said that", which I'd imagine it probably true but, unless I'm mistaken, OPs prediction would sit in the same pool as all those other ones, they hold equal validity.
If you zoom out, you will see that the median for that estimate a few years ago was two thousand fifty. While certainly some people might have thought it happens the next day, the average person thought it's many years away.
No one said [AGI in months or years, five years ago] that. If you look at e.g. the Metaculus predictions, you can see AGI has been hovering around 2027-2030 for 5 years.
That statement just contracts itself - if the average prediction has been 2027-30, someone, a significant percentage of someones, said earlier since many have said later.
There are certainly a lot of other people making all kinds of predictions. Some of them will eventually be correct but which one?
Additionally even with all the breakthroughs I am still doubting RSI. Up till now all the advancements heavily rely on RL over Lean. But how do model formalize it's goal as in RSI and RL over it?
Pretty much, a pragmatic attitude at its finest
> My thoughts remain the same since I've been hearing "months or a year" for the past 5 years.
If you zoom into any curve it tends to look linear. I think that is what you keep doing. But if you look at the world as how it was 5 years ago compared to now, there is no denying there is a massive change.
Once again though, is me accepting this a pretext for something? If not, why not just wait and see? Unless there's something that needs to be done before it's too late, it seems an odd choice to push for belief when there's no reason we can't just wait a while and have actual proof
> we can't just wait a while and have actual proof
I am not pushing you to do anything. I am not asking you to "believe" or anything. But we are, in my and many other's estimations, in the midst of what was called the "singularity" in AI communities, although it isn't a true mathematical singularity, it is just a strong acceleration.
> I am not pushing you to do anything. I am not asking you to "believe" or anything
Fantastic, I'll remain at my previous position then, I'll believe it when I see it.
(As a very off-topic sidenote, the term singularity never made much sense to me. I believe it was originally used to describe the sci-fi concept of a human gestalt consciousness, but has shifted to being about super-intelligence which doesn't seem to have much to do with "singular" at all. But, language changes I suppose)
> I believe it was originally used to describe the sci-fi concept of a human gestalt consciousness,
Huh? The term itself was introduced by Vernor Vinge to refer to the accelerated rate of change as a result of the introduction of super-intelligence. He said it would make things harder to predict in the future. It is part of this article:
https://en.wikipedia.org/wiki/Technological_singularity
It was never about "consciousness".
Huh, looks like you're right, though I would argue with "it was never about consciousness", but yeh it certainly wasn't originally. I think I got my meaning from a sci-fi short story were a unified collective called themselves "The Singularity", and I guess that was my first exposure to the term as a kid.
The biggest risk of the singularity is incoherence, and it's actually looking like what is happening now.
Like AI video/audio of high quality makes you have to question and doubt any media you see.
AI generated text has become the vast majority of material that you see showing up on the internet.
Things are moving from centralized pieces of software used by lots of people to vibe coded apps.
Reality tunnels and almost fully close and you can live in an existence that seems 'normal' but is a completely simulated experience with no interaction with other humans.
> If not, why not just wait and see?
Curious what you need to see. Some physical impact to your way of life? Are you not able to extrapolate the obvious?
So you think it's already here?
There are many things we can do and many people are talking about it.
One such coalescion is: https://pauseai.info/
Absolutely, the thing is that much of the time when I hear these predictions it's not to encourage participation in the movement like this. Many people who I speak to about it are even against such movements, which is what leads me to question why I need to accept their predictions.
If someone were to come to me with something like this as their reasoning, I wouldn't feel the same way.
Heard. Then:
Exponential progress has been the repeated truth, mirroring predictions.
The result of exponential progress into 2027 is a complete de-mapping of what we know to be human, whatever that may look like exactly at this point is hard to say.
We need your help, and everyone's help, to put a pause on this such that we are able to ground and find a healthy way forward together for as many human beings as possible.
Please join us.
Gotta say I really appreciate this page, I'm happy to donate but providing tangible actions is a really solid step
https://pauseai.info/action
I'm still not sure I believe it, but in my view its a bit like global warming, the worst(or best) case scenario of it being wasted time and money is far outweighed by the inverse, so it only seems pragmatic to help
I don't know about others in the thread think there definitely needs to be something done before it's too late.
I understand that many people don't think it will happen, but I can't really understand the position that you think there is a real chance superintelligence happens, but aren't concerned about it and are happy to wait passively to find out.
If you care about humanity at all it should be concerning because superint (if real) could be heavily destructive to our society.
I agree and I'm happy to hear out this kind of thing, my "I'll wait and see" attitude comes when the expected response is that I accept that the AI has "won". Thats pretty vague because the expected response, if there even is one, is pretty vague, but it generally amounts to something like that.
> But if you look at the world as how it was 5 years ago compared to now, there is no denying there is a massive change.
Is there? Yes, LLMs are integrated into a bunch of software, but practically speaking life is pretty much the same as it was five years ago.
Tell that to the SWEs getting laid off left and right all over Silicon Valley!
> LLMs are integrated into a bunch of software
I should have restricted it to the fields that have been impacted and have seen significant adoption: art for commercial purposes, writing, software development and mathematics primarily, and those adjacent to those fields. It is true that most other fields have not really felt that many effects.
AI videos everywhere, software developers no longer writing any code and it being much harder to find a job, and people largely no longer using search and instead switched to chatbots is “same as it was in covid”?
People are really really bad at seeing incremental change as it occurs around them.
It's like watching your kids grow. Your there everyday and you typically miss a lot, but then your cousin shows up for a holiday after not seeing your kid for 4 months and they are like "holy shit they've grown".
If you made a SOTA LLM enabled computer show up 10 years ago people would be fucking panicked at the crap we're messing with a decade later. But because we mess with it a day at a time we're like, eh, whatever.
I'm not saying LLMs aren't an absolutely impressive technology, especially for software related tasks, but in practical terms, life remains the same as it was five years ago.
Who said "recursive self-improvement / intelligence explosion / superintelligence" would happen in "months or a year" before 2026?
I can't think of anyone except maybe Elon Musk.
It was in conversations just like this, so people you wouldn't know and I didn't know outside of that conversation. You might be tempted to dismiss those, but then you'd also have to dismiss OPs for the same reason.
Okay can you provide links to these HN comments?
There might be, but I'm curious about the volume (I suspect very low).
I think it was on Reddit more-so if memory serves. Maybe it was on HN too, but obviously it was years ago now, I don't have a strong memory of the specifics. I'm not going to go digging through years of comments though, if you don't want to believe me thats fine, it seems a bit absurd to think absolutely nobody believed that but it wasn't really my point to begin with
> it seems a bit absurd to think absolutely nobody believed that but it wasn't really my point to begin with
Okay. If your point isn't that people believed superintelligence was months to a year away years ago, what is your point?
All due respect, I wrote a paragraph and that was the first sentence.
My point was that I'd rather wait for proof than accept a prediction.
You're mis-remembering things. Nobody in 2021 claimed that superintelligence is likely to arrive in 2022. That just did not happen.
People were talking about superintelligence in 2021, but "by the end of the decade", not "next year".
As I said to someone else, this wasn't by major figures, this was in conversation with ordinary people. As I said to them, if you want to dismiss this you'd have to dismiss OPs prediction too, and probably cease making your own unless you can get a news-outlet to cover it
Why do you believe that "OP" is making a prediction on his own?
There are some well-known experts saying this, like Paul Christiano, Daniel Kokotajlo, etc. Both are ex-OpenAI and are advocating pause/slowdown.
That kind of feels like moving the goalposts. Nobody becomes Nobody of significance becomes Nobody of significance without others of significance making similar claims, all 3 of these falling under the banner of "Nobody".
But at this point its a matter of semantics, and wasn't really the point I was trying to make. And as I said to someone else, if the reason for making these predictions is to encourage a slowdown I'm much more sympathetic to them (since there a real reason to act of them)
I was _hoping that maybe_ 2050 and other software engineers were considering me a crazy. GPT-2 was out by then.
Not splitting hairs at all, but I recall a lot of mid-2030s estimates around that time period. The horizon has drawn in a lot in a few years.
Yeah, I remember that last decade the prediction was the 2040s, but then LLMs happened and it became 2030s.
Mid-2030s estimates aren't uncommon even this year.
And then there's this gem: https://www.dwarkesh.com/p/ege-tamay "AGI is still 30 years away — Ege Erdil & Tamay Besiroglu"
5 years ago, what concretely did you expect 5 years later? One comment from then https://news.ycombinator.com/item?id=29020401 falling way short of today, treated then as wild near-sci-fi.
Why try to predict the most important single event to happen in human history?
Surely you recognize how absurd it is you’ve gone to such lengths to make your priors palatable. Please Look Up.
> Why try to predict the most important single event to happen in human history?
You're free to try to do so, but why are you pushing it so heavily? To what end?
> Surely you recognize how absurd it is you’ve gone to such lengths to make your priors palatable
And there is the answer to my above question. I like to think it's not most people, even of those who are staunchly pro-AI, but to some previous doubts are at best a great shame (assuming prior is short for prior doubts) or at worst a strange kind of heresy (assuming prior isn't short for anything). Either one seems anti-intellectual though, and encourages just the kind of endless hype we've seen so many times before.
great demonstration of the mindset OP was talking about
Yet AI has changed the world incredibly fast.
I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away. In 2025 AI got a gold at the IMO for the first time, a competition for high school kids. Now it's likely solved the biggest problem in math. There's no reason to expect progress to stop here, especially on domains highly amenable to RL.
Level of intelligence doesn’t matter if it is not applied to truth searching and instead to reasoning backwards from personal wishes.
Isn't this post an example of the former?
> I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away
That’s what I am responding to.
Define truth
42
I actually asked a guy who got an IMO gold what he thought about it, and he's happy to concede that AI works. He used to be my go-to guy for those kinds of problems when I came across them. Now you can actually get it to solve those things on your standard plan.
"If you are smart, why can't you predict the future?"
It's not an ability to predict the future, but the reluctance to step back and look at the trendline rather than saying "But it still can't do X, Y or Z!".
Because they're _desperate_ to keep their social standing. That's literally all there is to it. The same occurred during Enlightenment and the industrial revolution. A small upper echelon of society (the church or aristocrats respectively) were deemed worthy enough to possess knowledge and thus to belong to an upper strata. The same story can be seen pretty much in every civilization btw (Greeks/India/China/Japan etc). When the masses started gaining access to this knowledge there was _massive_ pushback, even armed revolutions in certain cases.
What's happening today is that anyone and everyone can have access to superintelligence. Anyone with access to an LLM effectively is, simultaneously, a world-class diagnostician, engineer, architect, musician, mathematician... you get the point. This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.
> This will necessarily lead to a post-scarcity society
I wish I shared this optimism, unfortunately that upper echelon of society is the one which currently controls the superintelligence. Everyone may have access to it, but they own it.
I would share your concern if there wasn't such a thriving ecosystem of open weight models. And they're usually about a 6 months to a year behind.
Still, as of now, while I use AI for everything I do (mostly Claude but also ChatGPT), I don't use any open weight models (probably more of a mental block at this point), but I do see them as a necessary counterweight to this potential power imbalance.
Where are you going to run these open models? Natural resources are controlled by this upper echelon too.
Most households own a car which even if you go used has a price tag of around $20k if you want something average. It wouldn't be much of a stretch to see a future where every household owns a $20k AI computer. Or two. If you are fancy you own a $100k AI computer.
However, since there is an ongoing war on general computing it wouldn't surprise me if instead people get locked into subscriptions.
Shit people be telling ChatGPT isn't protected by law. You get into any trouble, your chat history gonna be gone over with a fine tooth comb. Some of the masses gonna realize that and gonna want a sever rack at home instead.
I bought a 2000 Honda Accord V6 from a friend for $3k a long time ago. I drove it for several years, adding probably another 90,000mi to it. It didn't need much maintenance during that. It's still kicking from what I understand. An incredible car, I'm a little sad I parted with it.
I still think of the economic benefit I got from that car. It drove me to university classes. It drove me to work. It took me to see a total solar eclipse. It brought me to see distant family regularly. I went on a lot of road trips on it. For roughly $3k of capital costs.
I look at most expensive things and size them up as "how many Accords is this?"
Almost 7 Accords is a lot.
For now. Open weight models are effectively only lagging ~6 months behind the frontier.
How would an LLM make me a world-class violin player? If scarcity is from needing real "stuff" (incl. time), then knowledge alone might not change that scarcity.
Same way as a human would? By watching you (upload video) and listening to you (upload audio) and talking to you about your technique (text to speech). It's obviously no substitute for the real thing, but for $20/month, it's a lot more accessible to more people than a human instructor would be.
Can anyone at any point in their life be turned into a world class violinist?
I don't get why academics get the shaft every time. These labs are worth trillions of dollars while your average professor has to deal with 18 year old shitheads and gets paid 50k a year. Like what?
AI boosters love to frame the trillion dollar corporations as the real victims. War is peace, freedom is slavery, scraping all known data and refusing the share the fruits ("distillation attacks") is democratization, people who are losing their jobs to megacorporations with lots of compute power are the real sheltered elites who are mad because they're hypocrites.
I only hear this part of non-reasoning from people who have never managed to accomplish anything and simply want to blame AI for their personal failures.
That professor is free and highly encouraged to navigate their web browser to Anthropic’s careers page.
I don't get what you are trying to say. We will be all working for Anthropic?
Anti-intellectualism. The median AI booster resents that there is a class of people that had to work to become proficient in something, and wants effort to become irrelevant so their lack of investment in it appears to no longer be a failing.
> Anyone with access to an LLM effectively is, simultaneously, a world-class diagnostician, engineer, architect, musician, mathematician... you get the point. This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.
It’s hard to take you AI-hype men seriously when you are this ravingly delusional.
We don’t have access to knowledge, we have access to information. Knowledge requires processing information and that just doesn’t happen any more.
A post-scarcity society is not possible without first wiping out society. If everyone has everything then everyone has nothing. And of course, natural resources are actually limited, and those power LLMs.
> This will necessarily lead to a post-scarcity society, since everyone knows everything and will be capable of producing anything.
A post-scarcity society requires MATERIAL abundance in the hands of everyone.
All the material is controlled by ~100 mentally ill people who would rather kill us all than share.
I'm sick and tired of seeing people go "AI skeptics are just afraid of not being important". That's complete bullshit. We are skeptical because we can see, with our eyes, that the claims of AI boosters continue to be false time and time again. The simple empirical fact is that LLMs can't actually do a good job at the things people claim they are good at, and the industry lies through its teeth about said capabilities. So we are skeptical.
Yep, argument by psychoanalysis is a convenient way to disregard any stated disagreements as really just rationalization by weak/scared/lazy/depressed/etc people.
Wild thread to be making an “LLMs can’t do what’s claimed!” last stand…
AI still can't do a lot of what's claimed. This thread is an example of it being able to do one admittedly-impressive thing, but it's still nowhere near living up to all the promises.
>I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away.
The tech industry cultivated that reputation though. People were just acting rationally by being more skeptical of even larger claims than they were making before.
> There's no reason to expect progress to stop here
Well, the amount of money and resources it took to get here is a reason. I don't think the resources exist for another order of magnitude increase, given the strain on our economy and society we're already under due to AI.
The frontier models are getting cheaper as they are getting more powerful [0]. It's really not going to crash. I wish it would, for the sake of humanity.
[0] https://epoch.ai/publications/the-plunging-price-of-thought
As does every technology, but websites are not a few million times faster than they were when they were invented, and your computer does not boot a few billion times faster (quite the opposite...).
Reality is extremely chaotic.
I was thinking more about the cost in time, power, hardware, and data to train the models, not the price charged to use the models. I don't think it's going to crash, either—but that doesn't mean that progress will increase forever.
We are well past the point where massive changes are guaranteed even if LLMs immediately and permanently hit a wall in capabilities.
My experience on teams is there are the high agency AI adopters. Management likes this group. Then there’s what management would characterize as those not adapting (or adapting enough). This is the group getting laid off / not advancing in the environment. And this group is particularly bitter about AI disrupting an otherwise stable career, with wishful thinking they’ll be able to go back.
Honestly, the second group kind of brought this upon themselves. You cannot survive in tech without having an open mind and moving fast.
I've seen the first group vibe up garbage and dump maintenance on the second group. It may be unfair to blame AI for that when management is ultimately responsible, but that's what a lot of people are doing. In their defense, management is being mislead by AI hype men who are saying AI is so intelligent it shouldn't be making dumb mistakes.
Agreed, its more complicated than GP describes. There is clearly lots of value, but there are also lots of poorly led organizations creating a mountain of problems for themselves. We are all still figuring out what AI really means just for software development, which is the most useful economic activity it can do by an enormous margin.
Human minds are still needed to understand many things related to developing correct software, especially in business domains. There is still no AI running a profitable business, because that is not a verifiable task that can be scaled in RL like math or coding. So extrapolating from a math proof that no one can understand to a economically useful agents taking over most jobs is quite suspicious.
What is changing now is RSI truly beginning to happen. Much of ML is a verifiable task that can be scaled, and a lot transfers from general coding as well. Leading researchers are predicting their own obsolescence as early as next year, and that is something I take seriously, though I cannot judge the likelihood myself.
As improvement accelerates, we may see generalization happen at a scale that means we don't need RLVR for every profession. We've been surprised by it before.
> because that is not a verifiable task that can be scaled in RL like math or coding
Is it not, or are the feedback loops just longer? I could imagine being able to build a pretty good simulator of business operation for various kinds of businesses. (Some decent ones already exist as games.) From there, if you can build a good enough simulation, I can't see why you couldn't RL it.
Of course it's easier said than done to generalize it to the real world, where the possibility space is large. But I don't necessarily see an impediment to getting there eventually.
The feedback loops are quarters and years, so no you cannot scale it like you can scale math or coding. This is the reason we are seeing the most progress in those domains. Even ML is harder to scale because test runs are expensive and consume significant calendar time, so research "taste" is still important.
A full simulation would be, but I suspect even a partial simulation of evolving a software project though multiple "checkpoints" over a quarter would improve model performance
Parts are already disappearing if not gone already. If not by lack of profit, it will be by liability, overuse and moderation. Filters are being put up, bans in place, services restricted.
Just the other day a model helped me root an old android tablet. Will that still work in 5 years? I don't really have great confidence.
Yes, as long as it’s legal for you to buy a GPU.
> I'm always shocked by the number of smart, technical people who seem to think that AI is going to crash and burn and go away.
I suspect that these "smart, technical people" you speak of have more nuanced views on AI than you realize. Myself, for example, a self-styled smartypants:
Like the dot-com bubble, I suspect that certain parts of the current AI landscape are unsustainable, and I suspect/hope that those specific features of AI will crash and burn, while other useful bits remain.
AI usage like we see in this post here--AI that assists with discoveries in math, medicine, genetics, astronomy--I think they're great, and I hope they will continue to advance human knowledge for the foreseeable future. They assist experts and professionals rather than replacing them.
In contrast, generative AI ("creative" AI writing, illustrating, music, etc) is a pox on civilization. It is industrialized low-effort plagiarism that harms artists' livelihoods, discourages new art, makes everything bland, and pollutes the world in multiple senses of the word. I hope it dies from a combination of negative sentiments, and too much real-world cost once token subsidies go away.
Then there are the AIs that do things like voice reproduction. They have some arguable positive uses (e.g., assistive technologies), but many negative ones (e.g., phone scams, replacing human narrators with copies of their own voices). I think we're better off without these apart from niche accessibility uses. I think it should be legal peril to reproduce a voice without permission.
Apart from helping with some cinema CGI, deepfakes and deepnudes seem like a net negative for civilization.
Knowledge-based AI like Google's AI search summaries might just have built-in expirations. I think people will burn out on being vigilant about hallucinations. And when the bulk of their training data is made by AIs, it'll be hard to prevent model collapse. The cost of human intervention might make it too expensive for the wide use we see today.
Coding AIs are a grayer area. I think it's highly problematic that they are basically open source plagiarism machines, but I begrudgingly believe that society will decide that the upsides outweigh the downsides in the long term. But I choose not to use AI in my own coding projects, and respect others who do the same.
> Like the dot-com bubble, I suspect that certain parts of the current AI landscape are unsustainable
It's possible to believe that _almost all_ AI companies will go bankrupt while still believing it is a massively important world changing technology.
They may have more nuanced views for sure, but nuance is famously lacking from most online discussions unfortunately. I would love to see more of it surface!
I like your framework of separating use cases, mostly to evaluate how it will change in those specific areas, but I think you can evaluate capabilities fairly simply: I like to separate AI that depends on giant data centers and huge power draw from the kind that doesn't (not always local)—basically expensive vs cheap.
The expensive stuff isn't sustainable. Not yet. It probably will be, who knows when. I suspect it will be eventually because there's good money in it.
Here are the claims I find highly suspect based on the above:
- Anyone making claims that require expensive AI and attaching it to a short-term timeframe
- Anyone saying highly capable AI is impossible in the long-term
- Anyone saying these specific companies will be the winners—there is absolutely no moat for these models
I personally think anyone claiming something like super-intelligence or something approaching what we might consider sentience even in the next 5 years is suspect as well. I would not take the bet myself, but I would also not bet against it. The main reason is power: there's literally only so much power we can generate, so much societal disruption due to creating the infrastructure we can absorb (not to mention the societal disruption due to the tech itself), so its capability to self-improve is going to be highly constrained.
So far, we see increases in capabilities also increase power usage, I believe linearly. So, we're not seeing increases in efficiency that will make this easier to absorb. There's a ceiling, it won't go up quickly, so I think actual sentient-like AI capabilities are quite farther out than promoters think. Not impossible! I think it's more like a decade+ of maturation of the software and hardware stack—which isn't really that long in the grand scheme of things but longer than most people are betting.
My instincts are very similar on all points.
To push back, or share the self-pushbacks I grapple with...
> In contrast, generative AI ("creative" AI writing, illustrating, music, etc) is a pox on civilization. It is industrialized low-effort plagiarism that harms artists' livelihoods, discourages new art, makes everything bland, and pollutes the world in multiple senses of the word. I hope it dies from a combination of negative sentiments, and too much real-world cost once token subsidies go away.
While this summarizes my feelings on the current versions (which, say, are roughly at or below the math olympiad stage in writing, video, etc), what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?
I really want to say that even then it won't be the same, because the human provenance is essential to the art, but if I truly couldn't tell the difference...
> While this summarizes my feelings on the current versions (which, say, are roughly at or below the math olympiad stage in writing, video, etc), what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?
But here we're dipping into science fiction again. This is not currently the case, and we seem a long way from it.
> and we seem a long way from it.
We're dipping into prediction. But we have the (seemingly relevant, at least) data point that we're in the same situation w.r.t. writing/art as we were w.r.t. to math a couple years ago...
To an extent, sure, but:
- There was progress on maths that led to this point, whereas it's been stalled on art for the same reasons since the beginning.
- Art doesn't lend itself as well to the 'add more agents/complexity' strategy; a 250,000 word limerick is a terrible limerick.
> what happens when they actually become good? Meaning, an AI show that was actually as good as The Wire? To the point you couldn't tell the difference without being told?
Perhaps this comment will make me look silly one day, but my guess is that we'll eventually get there, and a handful of AI-generated productions will truly shine. But "as good as" is an important phrase. Because current-generation AI is trained on existing art, it won't be pushing many envelopes, it'll mostly be remixing what we have, and that will grow stale.
After a few breakout hits with broad shared appeal, I think the novelty will wear off, and AI produced TV and cinema will be too resource-intensive to remain as a profitable going enterprise. There will likely be a short window of hyper-personalized media that will seem very exciting, until we grow weary of the isolation of it. For many people, half the fun of watching a thing is discussing it with other humans, and today's streaming media landscape already leaves many people unmoored from that shared experience.
There is also the question of intentionality. When AI creates that show as good as The Wire, is that because AI has something to say about the human condition? If not, what's the point? If so, should we be paying attention to what AI wants us to pay attention to?
So what if AI slop wasn't slop? If you care about the details and the values behind your work enough to make it actually good, you won't use gen AI or use it invisibly because it concedes control of the details.
If you believe that using the authenticity of your own experience in your art will become vanish from how people value art then yeah we're headed for a future where AI rules. I really hope that that isn't the case.
These are extremely jaw dropping results, but I feel even though it is easy to conflate them with RSI / intelligence explosion, another simpler explanation is that symbolic manipulation is something humans are notoriously bad at, hence we prizing individuals that can rise above and dedicate their lives to it, and call them intelligent.
For computers though, symbolic manipulation is something they are extremely good at, and something that clearly scales well will memory and compute. Therefore I would argue that calling a computer superintelligent would, in analogy to do what we do with humans, require computers to rise above things that are extremely difficult for them. It is of course hard to judge what would be hard for them not being them for the introspection, but it seems the ability to operate outside the realm of symbolic or numeric manipulation would be precisely that.
It stands to reason that if some proofs require chaining 250k pages together, no human would ever be able to put it together. That doesn't mean something that is capable of doing so is necessarily super-intelligent.
Still very impressive, and if true, a wonderful result of the symbiotic interplay between a human and a machine, each at what they are good at.
Agreed. The fact is, Claude didn't get here on its own. It took someone who has been working with the RH since they were a student in college; some 40 odd years of experience.
It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.
What if "superintelligence" is really a hybrid result of humans using and guiding these large pattern interpolators?
What if "anthrocentricity" is really humans being unwilling to acknowledge that they are made of the same vibrating condensed energy as everything else, such that other matter can just as easily be moved into an apparently self-determining pattern of behavior as we are.
What if we're not the specialist snowflake?
I’m on board with this, and have been since I first had a hallucinogen.
But the human brain for now still has 100x the parameter count as the largest models. And they don’t have any circuitry for all the “keep yourself alive” work our brains are doing all the time, which may be the majority.
how profound of you
This is incorrect because you can always record the human prompts guiding the model and train it on them. This is how alot of progress in software was done and the success there shows that this generalizes beyond the specific prompts. You could probably prompt Fable 7 "Solve <famous hard math problem>" and it could do so without an expert mathematician guiding it.
> It's unfortunate we're always focused on humans getting usurped by an "AI", instead of the reality that these models only succeed when we're there to show them what success looks like.
As with chess, the reign of the human-machine "centaur" as the greatest mathematician in the world will likely be brief. I'd guess within 5 years the humans won't be adding anything. I hope I'm wrong.
Can you point to a task which you think would be an indicator of RSI or an intelligence explosion?
If you ask a layman, the answer would probably be a model that zero-shots any problem in any reasonable domain. E.g. zero shot Riemann hypothesis. No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels. Simply provide an input, feed the input through the layers, answer out.
Yeah, you can achieve something similar if you scale the compute infinitely and intelligently search the exponentially large search space.
> No tricks like swarms of agents, no multiple agents correcting each other. No high, xhigh "thinking" levels.
Why would implementation matter? The thought experiment is that you have a black box that can do X. What is X that validates RSI/super-intelligence/etc? It's meant to clarify definitions and end the cycle of goal post moving.
I'm not sure I really buy into the way these concepts are currently defined. What progress to date was not RSI and did not build on top of earlier progress?
The main characteristic of our physiology is that of adaption, and we have fairly grounded research on progressive overload as a means to guide adaption to whichever direction we want. Should we then call humans RSI machines if they go work out or read a book, or study to get better at the activities they carry out?
Define an objective, choose an optimization algorithm, bring a loss down. If it keeps going down, do I call that RSI because the loss descended from the step that came before it?
Now, I see the nuance that the optimization algorithm changes over time, leading to interesting higher order effects the likes of which Hoefstadter describes in the GEB, but fundamentally, the idea is the same.
I also don't see what an intelligence explosion is supposed to mean. If we increase the models efficiency by 10x, and reduce costs by 10x, so we have 100x more software and math proofs being shipped by unit of time in say 2028, does that really solve all of our problems?
Say that gets distributed over the world. Do we get every human to be computer literate? Say that gets concentrated. Do we solve problems no one has the means to even care about?
I would consider an intelligence explosion when the models tell us whether our efforts are well or misdirected as looking back from the hindsight of 10 years ahead, and capable of explaining why in simple English.
People suggested that language models are incapable of intelligence because they don't do symbolic manipulation. I think it's fair to call a vector with ten thousand numbers some kind of symbol, but then what's the state in your brain?
The reliable symbolic manipulation done here is in the form of Lean proofs, an external tool used by the LLM. One has to wonder how far an LLM alone would have gotten if formal proof systems didn’t exist they can verify their reasoning with.
Yeah, I think it's not so much "math is symbolic manipulation so computers and by extension LLMs are going to be good at it". LLMs are not by construction so good at it because ML models are far more statistical approximators than formal logic systems. I think the main thing that math has going for it is sufficient formalisation through tools like Lean that it's very verifiable, and the latests hotness in improving models is RLVR which seems to really work very well when the verification is strong and relevant to the task (probably also why they're getting very good at writing code and especially finding exploits).
(I would also say that computers are not good at math in the sense that LLMs are proving to be good at it: reliable symbolic manipulation is important for correctness but it's no use for actual progress in mathematics without some very good informal thinking/creativity as well)
It is impressive because of how large the search space is not entirely because of the level of abstract reasoning required to solve it. The reasoning trace produced that lead to this result could be a part of training that teaches the next version of Claude how to navigate incredibly hard problems making it more efficient.
When Atari games or Go were solved around 10 years ago, nobody was saying that superintelligence was within months just because reinforcement learning worked on a domain that allowed learning by self-play. Now we have a domain that is solvable by reinforcement learning with verifiable rewards. It's very impressive that these models are capable of doing the things they do, but the techniques aren't transferable to the vast majority of other domains. ML research also isn't verifiable in the sense that math is. People who claim it's verifiable, because there's a target that can directly be optimized don't understand why RL works for math. It's that lean is able to massively shrink the search space. This simply cannot be done in other domains in the same way.
LLMs don't produce Lean proofs directly, they produce the proofs in natural language, and only translate them afterward. The fact that we have Lean has nothing to do with LLMs being good at theorem proving.
This is simply not true that they only invoke lean at the end. They invoke it multiple times in between to reduce the search space like GP said. See how e.g the prove2me harness (which anthropic used for FLT) works for more details.
(I think we mostly agree.) From a previos post about Navier-Stokes, the English version may have small holes and the AI make a few changes and fixups when translating to Lean.
What kind of a thing do you think AI won't be able to do in 2030? A specific thing please.
- Keep electricity costs down
- Stop lying
- Design personal quantum computers
- Save fanatics from the delusion that AI will solve their life
I mean something a human can do, but AI can't.
Lay brick? Install pipe? Wire cable? Plaster wall?
Here's a brick laying robot from 2017...
https://www.youtube.com/watch?v=6s17IAj-XpU
so yea....
Try telling it to stop or move to a different wall in natural language. I mean, you might as well call a train AI with definitions this generous.
Self-reproducing physical agents (robots with onboard AI and ability to change their surroundings/build physical things) with human or higher intelligence which don't need a global supply chain of microchips, electro motors, etc. but could live off the land. That's basically the human concept, right? Totally untenable for AI/robotics of the current generation. Would require something like 3D-printing new chips but also 3D-printing new printers. Probably never happening. AI is fundamentally different, reliant on data centers, which are immovable and more like organs than organisms. Robots only come out of factories, not out of other robots or eggs layed by robots. So as Jack of all trades in the game of survival on Earth, humans win for the foreseeable future. Also, AI currently can't exist without humans, whereas humans have thrived without AI. Whether a silicon based ecosystem can ever survive on its own is an interesting question in and of itself. Because no matter how smart data centers become, they degrade fast, and then what? A data center needs maintenance. But if robots maintain the data center, who will maintain the robots? Does this equation ever solve? Humans have bacteria, a complex microcosm of life maintaining them. I don't think such a dependency-hierarchy of "life forms" is feasible for AI/robots, and humans will always have to be in the loop. That's one answer to your question, and not just for the next 3 years.
What stands axiomatically are human-to-human connections that we're wired to crave. Not to mention human reproduction.
Other points: - Editing a live action movie that will get 8.0 or more on imdb (AI sucks at reading emotions and won't be able to tell apart good takes from bad ones) - Reliably making good jokes (many humans can do it, AI weirdly sucks at it) - Accountability (contradictory because of its undead nature) - Reliably not lying/cheating (with no accountability, incentives will always be misaligned)
Either AI won't be able to do this: - Make a version of itself 100x more efficient that runs on commodity hardware and undercuts athropic.
Or, alternatively, it will be able to do that. Either way there's no world where one company bottlenecks and holds the keys to AGI, and frankly that's good.
Love.
Throw a football at a wide receiver running in chaotic patterns while avoiding oncoming attackers.
Use chopsticks to pick up a single grain of rice.
Olympic level gymnastics via neural-control of muscle fibers in an extremely precise signaling orchestration.
You may say the above are robotics and not “the model can scale to this” domain but I’d counter an animal can do amazing things exactly because of its generalizable and plastic neural architecture and that is what’s in scope.
I’ll grant you love, and I’ll also grant you 2030 on these, especially the last one. I suspect the football and chopsticks ones are very achievable by 2030 if anyone cared enough to focus on them.
Shoot I can’t do those things either!
Have you seen the progress in robotics lately? They aren't yet at a level you described, but quite close - so I'd say it would be possible in 3 years even without intelligence explosion. (E.g. if all frontier AI research stops we'll still have improvements in robotics.)
There are already Unitree robots which are much closer to Olympic level than an ordinary dude: https://www.youtube.com/watch?v=Y3zb4o47yqA You might argue but it's kinda at a level you described - I think if Unitree wanted to make a football-thrower they could just do it now.
Fully autonomously run a business that generates $10+ million in ARR.
What about one that generates $1+ million ARR?
I find that unlikely as well. I should note though, that I incorrectly wrote "running a business", when I meant to say "scaling to $10+ million in ARR". I don't think there's necessarily that much difference in running a business of these two sizes, but scaling to $10m ARR is much harder than to $1m ARR.
Sure.
If $1MM is also unlikely, how about $100k? Do you think that's unlikely as well?
The point is: what is the smallest amount that, if surpassed, would surprise you?
Yeah, a lot of the RSI claims seem dependent on us having very specific knowledge about how much intelligence is required to build superintelligence that I haven't seen anyone claim to have. Maybe you have to be, I don't know, 100x smarter than humans before the recursive part of RSI becomes tractable. Or maybe there are fundamental limits you can do with the hardware that exists that we don't know about, which means building superintelligence will take more time. I'm not saying the 2027-2030 people are wrong, I just think it depends on a bunch of things we can't credibly claim to know at this point.
I'm not a fan of Dwarkesh, but even he laid out that machine learning research progress is very much verifiable. At least in the RSI sense, that our current approach can be drastically optimized.
I won't trust that source at all, it's all about hype and trust me bro considerations
The models are improving at certain fields, and one of those fields is creating new models, which are verified by applying them to fields where you can prove that they're better.
I agree they don't transfer to all domains, but we do not know how far the theoretical will overlap with practical science. It could be "not all that much" and we hit roadblocks quite soon, or it could be that as the models become better and better at reasoning and research within their theoretical domains, it turns out to translate really well to the real world.
We don't know, but my bet is that we will continue to see rapid progress and impacts on the real world, until we reach a point where sceptics will have to throw in the towel.
The nature of mathematical proofs requires us to stay vigilant and careful. OpenAI pulled some of the research results from their famous repo due to simple mistakes that where easily spotted by human experts. There have been infamous mass of incorrect Fermi's theorem proofs.
LLM-assisted research is proving to be useful, rewarding and joyful, but we should not treat LLMs as a perfect machine, incapable of making mistakes. Especially in such massive proofs as this one.
I won't say that it "doesn't count". I will simply ask for staying rational.
Open AI mixed some Lean verified proof with English-only proof. It's important to read the fine print of the anouncements.
> I know this thread will be full of people in denial about [my prediction]
You can only be in denial about the past or the present. The future has a chance of being different from your predictions.
Personally, regardless of whether you want to call it "denial" or something else, I'm fervently hoping the AI never achieves superintelligence, but I would settle for it not achieving it until we can do our best to ensure that the superintelligence will be benevolent towards us.
Call me selfish, call me petty, call me a human supremacist, call me anything you want, but I prefer humans being the pinnacle of evolution. I like being able to point to our civilization, flawed as it is, and say "No other species we know of has achieved the same."
What's so surprising about this? A lot of these problems can only be solved using a significant amount of capital. At home or work, using a few-hundred dollar token budget, it's mostly a slog and requires handholding for the majority of open-ended problems.
This technology won't be available to the broad public in the scale that it's available to a select group. If it can only be deployed to a limited amount, to most it remains a magical inaccessible box. And that majority should naturally be incentivized to look for cheaper alternatives, and to "keep moving the goalposts", or in simpler terms, looking where they can provide value, of which there is plenty!
> A lot of these problems can only be solved using a significant amount of capital. At home or work, using a few-hundred dollar token budget, it's mostly a slog and requires handholding for the majority of open-ended problems.
Consider just how much more inference capability one can buy per dollar spent today as compared to a year ago. I can get tasks done quickly for essentially no cost that, early last year, might have cost a frontier lab hundreds or thousands of dollars and would have been unavailable to the general public.
In my mind, the bigger unknown is whether actual profitable use cases will show up quickly enough to justify all the (enormous) investment made in the technology.
By now, many open math problems have been solved with a cheap subscription. Not of this magnitude, of course.
I'm one of those people who's skeptical of recursive self-improvement and superintelligence. My rationale for my skepticism is that the algorithms get too much credit for the results. The algorithms ultimately run on hardware built by humans and learn from data generated by humans. In the same vein, I think knowledge work gets too much credit for building things and generating high quality data. The output of knowledge work is only useful if it can be applied to the physical world e.g. a CAD file is useful when the actual thing it describes is made in the real world. Similarly, high quality data comes from several feedback cycles with the physical world. I think people generally underestimate how much of human capability is driven by our sensorimotor skills. I think the idea of recursive self-improvement / superintelligence is just an extension of this attitude.
Humanity as collective does not buy anything, or send the design to the plant to be made. As a collective, humanity transforms the natural world into things, for example, people take fibres from plants, turn them into yarn, weave yarn into fabric, and then turn fabric into clothes. Similarly, humanity as collective does not "buy" chips, people turn quartz, bauxite, copper ore, petroleum, etc into the components of computer hardware.
There's a common adage about how everyone should work as a service person to learn patience and empathy. I think everyone should also work in an assembly line and a mine to learn about where the things they use come from.
I don’t understand the technical basis for belief in continuous recursive self-improvement. Frontier models are increasingly impressive at complex math and coding, but the math and code underlying the transformer architecture are not complex. Major advancements in capability have come from scale: more data and more processing power. How does solving the Riemann Hypothesis (or similar achievements) further increase those? Aside from supplying fodder for investment-seeking hype.
Below is just a hand-wavy argument.
Neural networks are function approximators. They're great because they can be trained to approximate almost any function.
What if, somewhere out there in platonic space, there's a complicated function that represents what a philosopher might call true intelligence. Perhaps it's structured totally differently than a neural network and shares no common architecture with a transformer.
Transformer-based LLMs are likely completely different in their fundamental structure. But perhaps the neural networks we have today are succeeding in intellectual tasks because they are beginning to approximate it. Like a bitmap approximates a vector image, it's not the same; the neural network is merely made of a bunch of repetitive simple elements, tuned to imitate the real thing. A higher resolution bitmap image will be a better approximation but still never become a vector image no matter the pixel count.
But at some point, the neural network might be good enough at intellectual tasks that when tasked with searching function space for an algorithm that does the job its neural network does but better, it finds the real thing it's been approximating all along.
There are many big technical question-marks on if RSI is even possible. However, the advances in LLMs that we've seen is definitely not just limited to data and scale.
First: modern LLM architectures are quite far from the GPT-2 era (delta/linear attention, GQA, sparse attention methods, looped transformers, mixture of experts, engrams, etc).
Second: RL environments are largely autonomously generated at this point, as models get smarter they can generate even more complex environments to train in. While this does go into the "data" category, it's not human-generated or even gathered data. This gets particularly spooky when you think about things we cannot yet simulate, that models can develop mathematical methods & simulators for. For example: better/faster physics simulations for robotics RL, better biological simulation environments etc.
My understanding is that LLMs can be used to make small improvements on top the transformer architecture, to make LLMs that are even more intelligent and so are more capable of making more improvements. At some point we might reach the practical limit of intelligence using transformers, and if the LLMs of that time aren't able to make the leap to discover a new architecture, then the improvement stalls. I don't think it will be possible
I don't think you need to be in the football team mindset of us vs them, we're all seeing the progress but most of it is still built on top of what humans have put together over decades and not completely new inventions, which is probably what would convince most that we've achieved "superintelligence" or whatever you want to call it.
LLMs are jagged in their capabilities. Opus 4.8 was already an excellent programmer, but it played Pokémon Red far worse than a human child, even if it did, eventually, manage to beat it.
I note that the dazzling events that have happened have all been in coding, cybersec, and math, that is, domains where the correctness of the work can be verified entirely inside the computer. It is unclear whether further progress along these lines leads to crazy scientific breakthroughs too, which is something people are taking as a given when it comes to superintelligence.
The pace of progress is incredible, but the end result is hardly unexpected. We're talking about a theoretical domain that's easily verifiable and has no side effects.
You have the input (hypothesis), the output (true/false) and the lean kernel. There exists some path from the input to one of the outputs, you "just" need to find it. You use AI to search this exponential space, occasionally prunning ideas that are dead ends, and throw as many agents as possible to scale it up.
In order to keep my mind safe from hype waves, I try to set my expectations periodically and see which of them happened in retrospect, updating my beliefs accordingly. At 2026-09-22 I made a personal 10 statements "prediction" to happen by 2027-03-22, which one of the statements was:
AI will solve exactly one or two millennium problems.
So I will become impressed by something, but only this is not enough. If AI solve three or more problems, I will become impressed and believe that some serious RSI stuff might be happening. If AI couldn't solve any problem (which it seems it did) I would become impressed in the other direction, and believe AI got stuck in a way that I didn't imagined (together with other 9 statements).
There may exist people (like me) who don't believe superintelligence is happening with current set of evidences, but believe it with some stronger evidences.
I cannot really comment on RSI, but I think it is important to make the point that an LLM doing large, even crucial parts in a proof of RH, it cannot be viewed in an anthropocentric light.
"If a human proved RH, they would clearly be extremely smart, so imagine how smart the LLM is!"
It is not surprising to me that LLMs show extreme aptitude for math while still having clear deficits in more real world applications. Handling stuff like super difficult math is precisely where AI has its greatest advantage over humans. Perfect recall, never tiring, never lose focus, attention to detail, etc.
If there is one area where AI can be pushed extremely hard ut is math. So the most amazing progress should be found there.
Whether AI with it's math achievements is extremely cool, whether it's intelligent in a philosophical sense and whether it's capable of RSI are 3 distinct topics.
The terms "super intelligence" or "intelligence explosion" are imprecise to the point of being useless. We have "super speed" in the form of automobiles. We have "super strength" in the form of excavators or bull dozers. We have recursive self improvement of tools by having better automation of machining. Intelligence != human ability. The intelligence demonstrated in the proof of RH is breathtaking, but narrow and brittle. It doesn't generalize to having grocery shopping bots, or automated psychotherapists.
These disagreements are natural and expected if most people are not properly defining their terms. What do you mean by artificial / super intelligence (explosion)?
I think it's a perfectly sane position to acknowledge amazing progress with very large neural networks, expect further progress, and also not buy into the specific mythology around the singularity or extropianism.
These things come bundled together quite often, and that's what I see people (reasonably, imo) resisting.
Doesn't mean there isn't unfair dismissal too, but I feel that's gotten much more rare.
> self-improvement / intelligence explosion / superintelligence
These terms mean different things to different people. What are the particular implications of this that you're suggesting are imminent?
I think this is all real and counts. Where I struggle is explaining to non-technical people how their everyday lives are going to change directly or indirectly. I hope these discoveries are the first in a sequence that will culminate in or surpass a cure for cancer and diabetes, cheaper and cleaner energy and water, and other tangible everyday life improvements that influence millions of people.
A cure for cancer and disease will only be available to the oligarchy, we the peasants will be killed off since we're no longer needed.
Yes, and that were the limits of "AI", "SuperIntellingence", etc become clear.
The world already produces enough resources to feed and cloth and house every single person. Many pf mankind's fundamental problems are no due to lack of technology.
Heck, why can't I have a proper OS, if AI is working wonders?
This is conspiracy theory level thinking. "The oligarchy" consists of humans just like you, who also want cancer and disease to be cured for everyone just as much as you do.
Human history is full examples where the significantly more powerful and wealthy relish in stamping out the rodents
What is the action item exactly for non-technical people? How do you prepare for “things will be different in unpredictable ways at an unpredictable time”? Even if that time moves sooner, still feels like very little info to truly act on.
If you watch movies or read fiction, we’ve expected for years that life in the future will be very different.
Support building more datacenters, promote and integrate SI at your workplace and in your personal life as much as possible, stop resisting, stop criticizing Anthropic and OpenAI, purchase their subscriptions and stock, or be left behind to dig around in the dust
> SI
lmfao, buddy
How does one fail to see sarcasm this obvious?
Oh that’s easy, you scroll HN seconds after waking up and still in bed, that’s my bad hahah
Fair enough lol, I just thought it might have even been too direct at first
Seen enough freaks on twitter use SI unironically that my eyes glazed over the instant I saw it here and my brain turned off
I think there's a key distinction missing in a lot of these techno-optimist takes.
Abundance of productive capacity does not automatically equal abundant access to its products.
If all of this happens in a way where a few people control these new, automated means of production, all of these wonderful things do not automatically follow.
Why would they share? And worse yet, if they have this automated production capacity without reliance on labor, why would they interact positively with society at all? What's their incentive? And no, you can't say markets, because once you have a machine that builds you whatever you want, you don't need to trade anymore, money and markets become meaningless.
You're basically reduced to hoping for charity from a group of people that have repeatedly proven their lack thereof.
RL is not scalable to super intelligence. Super intelligence should be able to learn continuosly without verifiable rewards.
You don't need RL all the way to ASI. You just need RL far enough that the model becomes good enough at AI research that it discovers something better than RL.
> becomes good enough at AI research that it discovers something better than RL
That might require continuous learning without verifiable rewards. RL models might help humans join the dots faster, but we don't know how many years that will take.
If its not possible with neural nets as we know them now, and requires a number of completely novel breakthroughs, there could be far from exponential progress even with the best RL models.
Yep, I agree. Improving AI might get harder, faster than the AI gets better.
But it also might not, which is the scary part.
I am not impressed because I understand how it works. Not because what things it can or cannot do..
We can start talking when we can train these things without having to beat on its head a million times in a million different ways to teach it one little thing...
How do you think our species was trained?
our species was not “trained”
This is such a substanceless argument, it’s pure vibes. It just says “Anyone who doesn’t believe in imminent intelligence explosion is an annoying curmudgeon.”
There are very solid arguments about why excellence at proving mathematical theorems does not necessarily imply general intelligence of the sort that people associate with ASI.
We’re not arguing with you, we’re pointing to the giant fucking meteor above our heads and laughing/lamenting that you still refuse to see it. It’s not an exchange between balanced discursive peers, not really. It’s an astonished exchange between those of us who have Looked Up about those of us who have yet to, and maybe never will, not really.
I encourage you to try, tho. For one thing, we need good people like you with us in reality. For another, you might die in some horrific catastrophe before you get a chance to, if you wait much longer.
what does “look up” mean?
https://en.wikipedia.org/wiki/Don%27t_Look_Up
I mean, you're barely arguing, you just sound a tad psychotic.
If people kept applying the same progress expectations to cpu frequency since 20001 we'd have 400ghz cpus by now
It's OK to admit we just don't know when or where it'll srop
Also virtually no one argues llms will go away, but a lot of people argue the current companies might, it's completely different.
There are a ton of people who think LLMs are completely useless, the whole thing is some combination of scam and delusion, and it will all come crumbling down once the music stops.
There is also a ton of people who think llms are Gods and will cure every disease including aging
You need retards on both sides of the spectrum for a balanced society
> superintelligence within months or a year
Didn’t we already achieve super intelligence?
Yes, 10 times already. You're supposed to drink a shot of vodka each time you read the term in an article or comment.
I love hn
No, we achieved artificial general intelligence. The fact that you mixed up these two wildly distinct terms should be a sign that you should not trust your intuitions about those topic.
Neither of us know what the growth curve looks like, so focusing on just calling one side naive is ignoring the naivety of the other side as well. If you want to extrapolate, go ahead, but that doesn't make you some oracle, nor does it make what you're suggesting factual. There are many potential scenarios and we should consider them, sure.
You're listing poorly defined words that mean different things to different people and wonder why people disagree with you...? My guess is that a lot of people you perceive this "denial" from are simply realists who are tired of seeing blind hype.
Note that every technology exhibits extreme growth until it... Doesn't. You can't predict the future. Whether any kind of RSI will happen depends on how hard of a problem intelligence is, and the precise point is that no person on earth knows, otherwise we wouldn't need to dedicate an entire research field to it.
Anyone who is sure that X will happen or not happen within Y years is just lying to themselves. For example, if you're so sure RSI will happen, why not invest all the money you have that isn't strictly necessary for your survival into this? Surely it will pay off in Y few years...
And note that I'm not saying it won't happen! I'm saying it's stupid to make a prediction of either case.
Your exact argument applies in full to climate change. No one can prove that the physical processes we know to be underway today will continue.
Yes, we're kinda guessing there too and I would expect every scientist to have a reasonable view to account for these uncertainties. I'm not sure if you meant to disagree with me, because I agree with you ^^
I also expect climate change to still be a process that is easier to research than intelligence. With climate change you can at least kinda look and measure and apply general principles of geology and physics. With intelligence, we've been at it for ~60 years and don't even have a definition yet, don't know what causes it, have no instruments, next to no introspection capabilities that neurology provides, etc.
Both the deniers and claqueurs are from the same ilk: Silly adults who don't have the intellectual capacity to create or even fully understand the subject, yet possess an almost vulgar self-confidence to constantly opine about its future.
Good observation. Perhaps the noble path is to chop wood and carry water, after all.
A slight nitpick: the hacker news equivalent of chop wood, carry water would be something like “charge the car, write a CRUD app.” Ordinary, not mystical powers doing magic. not high faulting thoughts but the basic simplicity of living in each moment and letting the necessary actions happen freely.
Seems the CRUD app may not need writing soon, and the car may charge itself.
I fear this is the result of a massive disconnect based on what you were told to expect. I read Kurzweil when that first came out, and maybe I'm remembering wrong, but I remember the singularity being the rate of technological progress in all domains will be so fast as to be asymptotic to an infinite slope and anything that will ever be possible will happen more or less instantly. Scott Alexander told me in a comments argument on his blog ten years ago that encryption is pointless because an AI will turn Jupiter into computronium and break it. Zvi was bandying about with the "take over its light cone" and "erase all value from the universe" phrases.
If that's the standard, then sorry, I'm not impressed by proving the Riemann hypothesis. If the standard is rapidly improving software for generating other software (which may or may not itself be any better than existing software) and math proofs (that may or may not have any meaningful real-world impact), then sure, I'm impressed. It doesn't mean I expect the daily lived experience of more than a few hundred thousand people globally to change much in the next decade, and most of the ones whose lives will change are software developers and mathematicians.
On that note, I have to imagine you have something different in mind when you talk about the "original meaning" of superintelligence, as you seem to not be talking about the Kurzweil/Bostrom/Yukowsky type of being effectively indistinguishable from God or at least the ancient alien civilizations from the Space Odyssey or Expanse novels, which again, I'm not seeing much of a hint toward because we have very good automated theorem provers. Proving Riemann doesn't mean we can now figure out zero point energy and turn planets into stars.
But it's not clear how well superhuman ability in math generalizes to other domains.
people always under emphasize too that the gains of tomorrow aren’t made with the technology/tools of today (I think Kurzweil is where I heard this first). RSI, if landed, seems like that on steroids
> people in denial about superintelligence
As long as I get to keep my house...
I won't believe it until I get a robot to put my dishes into the dish washer, put copper meshes and squeeze spray foam under my siding panels.
> Experience has shown that, even now, there will still be people explaining in patronizing tones why [...]
This is true no matter which side of the debate you're on. See [1] for an example.
[1] https://news.ycombinator.com/item?id=50575797
No offense, but have your heard about the infinite monkey theorem?
I would say, seriously, that this is bounded superintelligence. That superintelligence needed top human talent to achieve the goal. It was not a prompt like: "solve the Riemann hypothesis".
Coincidentally, yesterday I did a small presentation on cybersecurity and AI where I made a similar point: scarce top talent in cybersecurity (e.g. Calif) gets superpowers with AI, while AI replaces a lot of the skills of average cybersecurity professionals using simple prompts.
I think the breaking point is when AI can solve complex problems on its own that even top talent can't achieve. IMHO, I've seen some of this in the latest Google models (especially Carbon, and then Argon), where they can do more research independently.
"Human-assisted superintelligence" (HASI). ;)
I think you can be extraordinarily impressed and believe (or expect) that there significant limits to this technology.
I mean, AI could make very impressive 20 second videos two years ago. Progress has been slow and incremental since; Ten minute video still look far away.
I'd acknowledge is making quick progress on things you can find massive and reliable data on but even in it's good mode it seems to have palpable weaknesses. Someone was claiming Claude has become more and more persuasive over time but all the generators "still sound like slop" to me and many people.
Still, I have hope.
It’s not too late to Look Up for you. Yes, I’m talking to you — the person reading this with a vague grimace of disbelief and discomfort and disdain. It’s not too late for you to swallow some pride and reassess. Everyone has to eventually, especially me; why not now, for you?
So please. I’m begging you. LOOK. UP.
I looked up, it said 'naive'
:/
It’s a bait and switch to use a harvard fellow of mathematics solving a problem using AI with “The AI is super intelligent”. People are getting really tired of the constant bait and switches and subterfuge in press releases so unsurprisingly they just start rejecting it by default.
I don’t think like that, the most important question is: could he have solved the problem without AI assistance, and in the same time frame? I’d guess no.
I don't see anybody claiming that solving the Riemann hypothesis would be proof that current models are superintelligent. Reminder that the definition of the term is "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest". I do sometimes see people arguing that current models are AGI, but I don't think even this is true - current models are slightly superhuman in some narrow domains like math, human-level at a bunch of them, and still pretty stupid in some ways, most notably agency.
The reason people worry about superintelligence is that one of the "domains of interest" is AI research, and once you have an AI that's as good at AI research as your human researchers, your AI progress shoots up, probably exponentially. So the delay between automated AI research and superintelligence might be tiny.
Stochastic parrot truthers in shambles.
We're really using a term invented by Trump yesterday for something we've been working on for 80 years?
Superintelligence is an old term with an established meaning. What Trump does is to use it for AI in general.
The term was not invented by Trump. For example, the (in)famous book Superintelligence by Nick Bostrom was published in 2014[1]. It has a long been used to refer to a theoretical general AI that far surpasses humans.
The Trump thing is a insane newspeak attempt to forcibly replace the use of "artificial intelligence" with "super intelligence" in a general sense. It is ridiculous, obviously, but the best thing we can do is to continue using "superintelligence" in its original, narrower sense.
[1] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...
Actually, I think the best thing we can do is take Trump round the back of the woodshed.