If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
given the state of our weapons industry I think it's more like
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
And search space in software can be absolutely staggeringly large. Just a little bit of directing and you can shrink the problem from all the entropy in the universe to a few hours of compute needed to find an exploit.
Personally I think it's a good thing - many vulnerabilities are being found and fixed now, people are wide awake and paying attention to them being found so updating is quicker and easier, and since code is finite, so are vulnerabilities so in, say, a year or so there's going to be the status quo of LLM vulnerability scanners run by dozens of companies running 24/7 on every software ever... unable to find anything.
So yeah it sucks right now, there's outages, probable PII theft left right and center (but that's nothing new at this point), millions / billions in damages and lost productivity from Office365 and co being offline for a day or two... but it's still manageable, the world isn't on fire, and it's better to get all that out of the way now than in an actual crisis.
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
The observation was "if one person found the bug with LLMs, other people are also going to find the bug". Whether it was due to communication between instances of the agent or just due to the predictability of an individual agent is not clear and there is not much evidence for the former.
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
No (well maybe no), because they are lobotomizing the public models on cyber tasks.
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
It’s a reflection of the times.
If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
It's never clear quite where you are in the technologically cycle.
But it's like inventing a hammer and finding you can smash whatever you want with it.
Soon you'll be building houses with it.
Maybe that's being glib but it's just a tool being applied to a world that is still largely prestine from its impact (pun unintended).
given the state of our weapons industry I think it's more like
.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
> It's never clear quite where you are in the technologically cycle.
The problem is that everyone is at a different place in the cycle. As Gibson put it, "The future is already here - it's just not evenly distributed."
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
The “hint” is for the modern equivalent of script kiddies.
i.e fast followers of the researchers who don’t have the resources to burn through checking all of a tool.
Cost scales with the size of your search space. Smarter questions get answers for less money & time.
And search space in software can be absolutely staggeringly large. Just a little bit of directing and you can shrink the problem from all the entropy in the universe to a few hours of compute needed to find an exploit.
Personally I think it's a good thing - many vulnerabilities are being found and fixed now, people are wide awake and paying attention to them being found so updating is quicker and easier, and since code is finite, so are vulnerabilities so in, say, a year or so there's going to be the status quo of LLM vulnerability scanners run by dozens of companies running 24/7 on every software ever... unable to find anything.
So yeah it sucks right now, there's outages, probable PII theft left right and center (but that's nothing new at this point), millions / billions in damages and lost productivity from Office365 and co being offline for a day or two... but it's still manageable, the world isn't on fire, and it's better to get all that out of the way now than in an actual crisis.
Can I take this to mean a recent model release is able to find this bug, and these groups have access to that model?
What are the odds several humans independently uncovered a complex vulnerability like this simultaneously?
or maybe the model leaked the exploit from one to the other...
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
This was one of the takeaways from Greg Kroah-Hartman‘s recent Linux in the land of LLM keynote.
The observation was "if one person found the bug with LLMs, other people are also going to find the bug". Whether it was due to communication between instances of the agent or just due to the predictability of an individual agent is not clear and there is not much evidence for the former.
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
That seems likely, but maybe it was being actively exploited so several people started investigating.
No (well maybe no), because they are lobotomizing the public models on cyber tasks.
It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.
We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
When everyone is using the same tools for the same purpose...