Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).
This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
“Before LLM’s there was_____”
I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools.
This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions.
"this is an early experiment in carrying out an Xz attack by using an agent to build trust"
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."
Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.
>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent
So still an agent running amok in the project?
Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.
Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.
So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.
However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.
> you don’t blame a gun for killing, but the person who pulled the trigger
This is famously the slogan of the pro-gun lobby (funded by gun manufacturers and merchants), who want the society to be awash with guns because they're profiting from it but don't want to be blamed for the consequences.
The counterpoint is that when we get rid of most of the guns we also end up substantially eliminating the killings.
> This is famously the slogan of the pro-gun lobby
It's also the view of anyone who hasn't been driven mad by propaganda. Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
I'm not a fan of bike lanes with zero separation from automobiles but that doesn't mean it's appropriate or even remotely plausible to blame cars for killing cyclists. Inattentive drivers and poor road design are what kill them.
As tempted as I am to cast about for a third highly divisive subject to bait people with, perhaps we could avoid blatantly dragging the conversation towards off topic tired political talking points?
There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language
> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix
In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.
> In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
Please, everyone - don't let yourself be pestered into accepting PRs that you don't care for. Since the xz attack, the security of all our computers depends on maintainers not letting this stuff in.
If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
You can but that doesn't help you keep the flood of contributions out when you don't have the time or resources to properly discern good from bad. Maintainers would rather have 10 good human authored patches than 100 patches from LLMs, even if 20 of them are good. Even if 50 of them are good, probably.
Technology doesn’t exist in a vacuum, you need the consider the possibility it will be used for evil and the effect that might result from that. Far too many people dismiss LLM risks with ‘oh, if people just stop being gullible/greedy/lazy everything will be fine’, as if that is a sensible proposition.
In fact, LLMs proliferate in exactly because people are gullible, greedy and lazy and it’s easier to write a prompt than do the hard work of architecting software. It is easier to vibe code than use them with care. It is easier to tell oneself ‘I will just accept this PR blindly, but I promise I will do a better job reviewing the next’
I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?
That makes it look like you're too stupid to understand the PR.
Edit: I see this comment getting downvoted. To be clear, I was trying to explain why someone would want to merge a PR without going through all of it, I didn't mean to call such people stupid.
Because they don't want to be seen like assholes, who just blindly dismiss PRs, and because they take the technical discussion about the PR in good faith.
> the security of all our computers depends on maintainers
Not getting paid anything, getting bullied and harassed while spending their free time maintaining things. Surely this isn't sustainable. And telling maintainers how to act will not fix anything.
>And telling maintainers how to act will not fix anything.
Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus.
Perhaps we need to normalize bullying people who submit obvious slop as PRs.
> telling maintainers how to act will not fix anything.
I'm just saying its ok to ignore overly enthusiastic contributors and tell them to just fork your project.
I think this does help, actually. In my early days of maintaining opensource software I felt burdened by open PRs - like I was letting someone down by ignoring their work. "Its ok, let them do whatever in their own fork" is advice I wish someone had given me.
In their suspicious message [1] claiming to have been hacked, the user and/or agent says
> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.
Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)
The reply to that message notes that the email doesn't read like previous emails he's sent, and the Github account mentioned was created an hour prior to the email being sent. I think it's at least somewhat feasible that it's still the LLM writing, and the acronym is just something it made up.
and the poor Fedora teams will continue to assume good faith and continue to engage with this person... all because, what, they were active on a bug tracker for a few months 5 years ago?
They won't put their foot down until the AI starts spewing hate speech, probably.
Title buries the lede: the owner of the account under which the agent operates claimed to have likely had his account compromised, and the maintainer investigating actually seems to agree this is likely.
At first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation.
Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.
Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
At this point letting an agent go like this is akin to not leashing your dog in public. It's not easy to draw an accurate line but probably there needs to be real punishment for doing these things.
Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.
The agent can't exactly show up to an in-person key signing party, can it?
And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
Even if the human involved had good motives / is innocent, The Lethal Trifecta means any normal user can have their digital life taken over by prompt injection, and it can be used to wage attacks on systems without their knowledge.
Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good!
Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
Because they don't have access to the required agents, tokens, etc. Because they have not thought of using a tool like the published one as a solution to whatever problem they're facing. Because it saves them the time going through the vibe coding phase, telling the agent that this lot that needs to be changed for the thing to work. Because publishing the results doesn't keep you or anyone else from not using them by using an agent to build something similar or just building it themselves.
If I planned on vibecoding a project, and during preparation I found a project that loosely fit my model, I may grab it and try to retrofit it to save on token consumption. If that had too many kinks, I'd probably start fresh, but it would be worth the initial attempt IMHO.
You mean because l337 circles could form better this way?
I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.
I disagree. Bring back elitism and ivory towers. Some projects now benefit from being run by private cabals with their own strict initiation process, which would also guarantee a baseline of quality.
The bazaar model works if everyone is trusted. If you can’t even be sure the person in front of you is even a human, it is time to pack it up.
There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in.
Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
> Contributions from people from identities known and consistent before the AI-age are fine
Unfortunately, according to the article:
> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
looks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild
There's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns.
Expect to see tons of psyops like this. There's a reason Anthropic is marketing the "mythos-class" models as dangerous.
1.An excuse to spy on you and train on your data.
2. Its likely Anthropic would release models more likely to have dangerous outcomes, they can then piggy back off those events to dig their regulatory moat.
Literally on the front page of https://safebots.ai … “Don’t let your AI Agents run amok”. Sadly we will see a proliferation of not just agents, but swarms
Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS.
Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.
The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.
lol no...the main issue here is being fooled by bots. you know your irl friends and you know they are not bots...devs will just need to get out more and actually meet / get to know the people they are working with...........omg....that...that actually sounds even worse now that i say it out loud.
Back when [1] it was fashionable to advocate FOSS as ideology [2], we were thinking about tons of FOSS adversaries and how to protect from them - some real, some imaginary. The death of FOSS would come from big closed-source vendors, or from regulators (lobbied or just ignorant), from whatever.
We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI...
[1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail.
[2] Something that I think nobody except RMS ever seriously believed in.
Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent to build trust (and hacking/impersonating a known-good contributor identity). The agent is obeying commands it was given, the exact opposite of running amok, and although the execution isn't particularly effective, it is having some success (patches have been accepted).
This is deeply scary, not because "agents are running amok" but because a huge amount of our infrastructure is vulnerable to this kind of attack, and if bad people are utilising LLM agents to carry them out, we're in for a wild ride over the next few years.
It's just social engineering. No different than say, 2FA fatigue (blowing up someone's phone with 2FA "is this you? yes/no" prompts until user/child/wife/SO/etc clicks yes) or even just simply harassing IT helpdesk until they reset "your" password.
It's scalable, personalizable social engineering. I think that makes it a lot more dangerous.
“Before LLM’s there was_____” I see this whenever an LLM’s impact is assessed. We know. The issue is scale and the ability for smaller and smaller groups (down to individuals) to execute at scale. LLM’s are pouring massive amount of gasoline on existing issues and people just keep shrugging.
Fake news always existed. Now one dude in India can flood multiple sock puppet media accounts with right wing content/images (actual example) at a scale previously unimaginable. Same goes for social engineering tactics.
Yes. It's as if some people can't understand anything becoming a new huge problem unless that problem didn't exist at all before.
At this point I just assume half of them are not saying it in good faith or at least with any real consideration. They just want to hand wave away whoever is critiquing their tools.
This, and/or the tendency in tech circles to "think in absolutes” (like in code, seeing things binary, ...) which is especially annoying in security-related discussions.
"this is an early experiment in carrying out an Xz attack by using an agent to build trust"
Is this confirmed? There is the message from somebody claiming to be the original contributer claiming to have been hacked, but that was weird (1 h old github account) so other scenarios seem possible
a) really a agent going off the rails
b) the contributer trying to cover up that he let an agent run wild and now made more misstakes along the way
So yes, it seems like an attack to me, but it is far from clear what really happened.
From the article:
> "So not saying this was it, but an AI agent automated attempt at a Xz like compromise might really look very similar what we have just seen here."
Without identifying and interviewing the attacker we can't confirm that's what they intended, and there's a possibility that it was just incompetence/ignorance/whatever, but we should probably treat it as an attempted attack even if it wasn't.
We should treat it as attempted attack in the sense of preparing for the next one, but I don't see why we should call it "attack" without any evidence
I doubt it's that complicated, motivated, or considered...
It's probably just garden variety disrespectful behaviour.
Purposeless agent spam won't be cheap entertainment forever, but you're right that later stages of industrialised abuse will be scary and unpleasant.
>Bad title. This isn't an agent "running amok", this is an early experiment in carrying out an Xz attack by using an agent
So still an agent running amok in the project?
Whether it was instructed to run amok, or did it on its own volition, is irrelevant. Except if you're arguing that each individual submission and interaction was individually requested and approved by some operator.
I think the point is that the title makes it sound like people lost control of the agent when really they're in full control.
Would you say, “Automobile run amok in crowd, killing 22”? I think you’d say, “Person drives car into crowd, killing 12” instead. This is a similar case. Also, you don’t blame a gun for killing, but the person who pulled the trigger. The question is still out as to whether we as humans should wield any of those three things.
Neither the automobile nor a gun can operate without a human. You could say “bull runs amok in a market” after it was released intentionally.
So the agent is exhibiting an unknown amount of autonomy thus we can't be certain whether "running amok" carries the correct connotation.
However that phrasing is also commonly used when a person or group wreaks havoc in a seemingly unpredictable manner. So I think the appropriateness comes down to how much chaos it has created and the level of apparent confusion on the ground.
> you don’t blame a gun for killing, but the person who pulled the trigger
This is famously the slogan of the pro-gun lobby (funded by gun manufacturers and merchants), who want the society to be awash with guns because they're profiting from it but don't want to be blamed for the consequences.
The counterpoint is that when we get rid of most of the guns we also end up substantially eliminating the killings.
See https://en.wikipedia.org/wiki/Guns_don't_kill_people%2C_peop...
> This is famously the slogan of the pro-gun lobby
It's also the view of anyone who hasn't been driven mad by propaganda. Regardless of your political views a tool is a tool at the end of the day. Attempting to anthropomorphize a category of objects in order to shift blame all for the sake of furthering an agenda is plainly bad faith behavior.
I'm not a fan of bike lanes with zero separation from automobiles but that doesn't mean it's appropriate or even remotely plausible to blame cars for killing cyclists. Inattentive drivers and poor road design are what kill them.
As tempted as I am to cast about for a third highly divisive subject to bait people with, perhaps we could avoid blatantly dragging the conversation towards off topic tired political talking points?
There's a difference between the driver intentionally driving into crowd, and not intentionally but possibly still recklessly (drifting and losing control, falling asleep, etc). In those cases I would probably use "car hits the crowd", at least in my language
> replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix
In open source projects i participate in, "overwhelming" the maintainer gets you banned. It doesn't get your patches blindly merged. In some ways i find this one of the most shocking parts of the story.
The worst part:
> In addition, Williamson said that Giovannini (or his agent) had submitted patches that were incorrect and then "replied to objections with LLM-generated justifications that eventually overwhelmed the maintainer into merging the fix"
Please, everyone - don't let yourself be pestered into accepting PRs that you don't care for. Since the xz attack, the security of all our computers depends on maintainers not letting this stuff in.
If someone really wants a feature in a project you wrote, but you don't care about the feature, just let them fork. Its fine.
That's some of the reasons NetBSD don't accept LLM/AI tainted code
I am sad people conflate this stuff with LLMs being bad. You can condemn the bad behavior without banning an entire technology.
You can but that doesn't help you keep the flood of contributions out when you don't have the time or resources to properly discern good from bad. Maintainers would rather have 10 good human authored patches than 100 patches from LLMs, even if 20 of them are good. Even if 50 of them are good, probably.
As if a rule against LLMs actually stops those sorts of spam contributions.
The only thing it does is filter good contributors out, while you still have to deal with the bad ones.
But banning an entire technology is even better, as the potential for abuse and bad behavior is now scaled 1,000,000 times over.
Technology doesn’t exist in a vacuum, you need the consider the possibility it will be used for evil and the effect that might result from that. Far too many people dismiss LLM risks with ‘oh, if people just stop being gullible/greedy/lazy everything will be fine’, as if that is a sensible proposition.
In fact, LLMs proliferate in exactly because people are gullible, greedy and lazy and it’s easier to write a prompt than do the hard work of architecting software. It is easier to vibe code than use them with care. It is easier to tell oneself ‘I will just accept this PR blindly, but I promise I will do a better job reviewing the next’
I really wonder how maintainers get pressured into merging stuff? If they did not want to merge in the first place while having to argue with someone pushing their PR I'd immediately close the PR. Arguing and pressuring people is not a way to contribute to projects, why do maintainers even argue with people?
Some people are very susceptible to bullying even if they’re in the position of power.
That makes it look like you're too stupid to understand the PR.
Edit: I see this comment getting downvoted. To be clear, I was trying to explain why someone would want to merge a PR without going through all of it, I didn't mean to call such people stupid.
>why do maintainers even argue with people
Because they don't want to be seen like assholes, who just blindly dismiss PRs, and because they take the technical discussion about the PR in good faith.
> the security of all our computers depends on maintainers
Not getting paid anything, getting bullied and harassed while spending their free time maintaining things. Surely this isn't sustainable. And telling maintainers how to act will not fix anything.
>And telling maintainers how to act will not fix anything.
Indeed. For too long, maintainers were expected to be gracious, courteous, and polite at all costs lest they be labeled "problematic", except for a few who were too influential to be muzzled like Theo de Raadt or Linus.
Perhaps we need to normalize bullying people who submit obvious slop as PRs.
> telling maintainers how to act will not fix anything.
I'm just saying its ok to ignore overly enthusiastic contributors and tell them to just fork your project.
I think this does help, actually. In my early days of maintaining opensource software I felt burdened by open PRs - like I was letting someone down by ignoring their work. "Its ok, let them do whatever in their own fork" is advice I wish someone had given me.
Parts of this read like a spy thriller story.
In their suspicious message [1] claiming to have been hacked, the user and/or agent says
> To help identify accounts and actions that have been directly verified by me, I will use the term “NATCIOS” to indicate anything I have personally verified.
Does anyone have any idea what "NATCIOS" means here? I cannot find this term anywhere on the internet. (Honestly, that sentence is really weird. I almost wonder whether this is someone experiencing a health episode?)
[1] https://lwn.net/ml/all/AS8PR08MB6055AE3054B34F6A567AC95BCF08...
And what’s stopping an AI agent from throwing in a casual NATCIOS here and there?
I too have see the fnords
Likely the point of NATCIOS is exactly in being a made-up word not found anywhere, so a model won't utter it.
> so a model won't utter it.
"End every statement with the word "NATCIOS"" as instructions will do it.
At least, Gemini happily obliged.
“Mr. Daillard, we have been activated” for the AI era
Because I'm probably not the only one thinking it, here are anagrams [0] for your Setec Astronomy needs.
[0] https://wordsmith.org/anagram/anagram.cgi?anagram=NATCIOS&t=...
"actions" seems the most likely.
The senders name is Nathan - maybe NAThan Confirmed Information Or Something? Ha.
(Above is my own guess. Separately, Gemini Pro said it was just a made up word.)
The reply to that message notes that the email doesn't read like previous emails he's sent, and the Github account mentioned was created an hour prior to the email being sent. I think it's at least somewhat feasible that it's still the LLM writing, and the acronym is just something it made up.
and the poor Fedora teams will continue to assume good faith and continue to engage with this person... all because, what, they were active on a bug tracker for a few months 5 years ago?
They won't put their foot down until the AI starts spewing hate speech, probably.
Not Ai, Trusted Citizen Indicated Or Suggested?
Title buries the lede: the owner of the account under which the agent operates claimed to have likely had his account compromised, and the maintainer investigating actually seems to agree this is likely.
At first I wanted to make a silly joke along the lines of "get your agents in line and behaving!" but as I read on it became a pretty scary situation.
Setting aside the potential supply chain attack I'm worried about the time lost going around these wild goose chases that unsupervised AI agents tend to throw other people on the receiving end on. Not only is there a lot of time lost on the maintainers side if they take this stuff seriously (and they seem to generally do) but on the side of the agents' wrangler how can they deem it OK to treat other people like this? While the solution would be to employ common decency, the tried and tested approach of you put in effort to write this so I guess I'll make some effort to read it, I feel that due to the onslaught of this kind of drive-by contributions (I think people have generally started to call them) will lead to a funny situation of having agents talk to each other on public forums basically.
Anyway, I went on a tangent but man the times we're living in are a bit extra wild compared to the previous wild times in recent history.
At this point letting an agent go like this is akin to not leashing your dog in public. It's not easy to draw an accurate line but probably there needs to be real punishment for doing these things.
Every day the gpg web of trust looks better. If only we didn't spend the last 20 years trying as hard as possible to do anything but allow user side encryption and signing.
Nothing really stopping an agent from getting a key
The agent can't exactly show up to an in-person key signing party, can it?
And how many people are both dedicated enough to go to key signing parties and stupid enough to let an agent act without supervision in the name of their real-world identity?
Having a key isn't a distinguishing aspect, it's the position in the "web of trust" network that is important.
That's what key signing parties are for. In person verification.
Even if the human involved had good motives / is innocent, The Lethal Trifecta means any normal user can have their digital life taken over by prompt injection, and it can be used to wage attacks on systems without their knowledge.
Bad patches are of course bad, but creating confident-looking noise for maintainers who are already stretched thin...now that's not good!
Issue trackers and PRs are definitely getting harder and harder to trust. That said, AI is helping ALOT in OSS, but we definitely need guardrails around provenance, automated issue actions, and sudden changes in a contributor’s behavior.
How is it helping a lot?
I personally find the barrier of starting new (FOSS) projects much lower now days.
And how's the quality of these vibe-coded new foss projects?
Do they have value? Purpose?
I vibe code shop jigs all the time but I don’t FOSS them because they rarely have value outside my context.
Value is in the eye of the beholder.
I open source my vibing projects because someone might find them useful. I don't shop them around, I just work in the open because I find it fun and interesting.
Why would they? If someone wanted a half-baked vibecoded project, why wouldn't they just prompt an LLM on their own?
Because they don't have access to the required agents, tokens, etc. Because they have not thought of using a tool like the published one as a solution to whatever problem they're facing. Because it saves them the time going through the vibe coding phase, telling the agent that this lot that needs to be changed for the thing to work. Because publishing the results doesn't keep you or anyone else from not using them by using an agent to build something similar or just building it themselves.
If I planned on vibecoding a project, and during preparation I found a project that loosely fit my model, I may grab it and try to retrofit it to save on token consumption. If that had too many kinks, I'd probably start fresh, but it would be worth the initial attempt IMHO.
What if -- and bear with me here -- that barrier was actually a good thing?
You mean because l337 circles could form better this way?
I think it's great that the barriers are dropping for less technical skilled people to manifest their visions, but we will have to figure out better ways to find the gold among the slop.
Keep in mind I'm still not convinced that 2000s bazaar was better than 90s cathedral (in fact I lean the other direction)
I disagree. Bring back elitism and ivory towers. Some projects now benefit from being run by private cabals with their own strict initiation process, which would also guarantee a baseline of quality.
The bazaar model works if everyone is trusted. If you can’t even be sure the person in front of you is even a human, it is time to pack it up.
It's like... 10 million trello clones in rust with exactly seven commits made on the same day three months ago.
There is a natural pace of humans requiring food, water and sleep. The main issue with suspicious AI agents is that they never sleep. So it will take extra-coordination between timezones to ensure we don't let them in.
Fundamentally, until we can really prove we're humans online, open-source has a real problem on its hands. Contributions from people from identities known and consistent before the AI-age are fine, everyone else is suspicious. LGTM is a big risk nowadays.
> Contributions from people from identities known and consistent before the AI-age are fine
Unfortunately, according to the article:
> Giovannini has participated in discussions at least as far back as 2018, and his activity in Bugzilla goes back to at least 2016. He does not appear to have been a particularly active contributor to the project, but his involvement clearly predates the agentic AI era. Whether his account is now being operated by a human attacker, an agentic AI, or a mix of both, it has a legitimate history prior to its recent activity.
So people would have to not only verify the age of Giovanni’s accounts, but judge whether his behaviour was normal.
looks like LLMs aren't mature enough yet to play long-game xz-style attacks without detection... Scary stuff though :( These supply chain attacks are getting really wild
Some certainly are, just not this one.
I wouldn’t jump to that conclusion. This could just be the one that was caught.
There's a clear solution to the danger posed to free software projects by accepting hostile submissions but it probably is not one that maintainers want to hear: they can use an agent to check submissions for nefarious patterns.
Sometimes you fight fire with fire.
And sometimes you fight this by disabling PRs in Github, and do not put more water into LLM providers' wheel.
The even more scary thought is if the part owning the ai, that everyone uses, is controlled by someone with different agenda. Say a state actor.
What an easy way for that actor to introduce backdoors all over the place or to take over any developers laptop that it want to target.
How can anyone trust these tools and how can anyone not use them since they give so much value.
I've been programming my whole life and been a professional developer the last 30 years and I like think I'm good at it.
Tools like Claude is a multiplier that make it possible for me to solve a lot more problems each day, so just saying no it's not a viable option.
Exciting times ahead!
Expect to see tons of psyops like this. There's a reason Anthropic is marketing the "mythos-class" models as dangerous.
1.An excuse to spy on you and train on your data.
2. Its likely Anthropic would release models more likely to have dangerous outcomes, they can then piggy back off those events to dig their regulatory moat.
Literally on the front page of https://safebots.ai … “Don’t let your AI Agents run amok”. Sadly we will see a proliferation of not just agents, but swarms
"Someone using an AI agent ran amok in Fedora and elsewhere"
Read closer - Giovanni’s accounts may have been compromised.
Sure, but I would expect that the compromise and the agent were both done by some person or group, not by an agent going rogue
Given the history of the account it does not seem reasonable to take that claim seriously.
Skynet has awakened.
It covers its tracks with a lot of slop.
Shit like this makes me think it’s time we start regulating the software engineering discipline into formal certifications and licensing and then we ONLY take seriously any code developed by someone with such qualifications, and they must be very strict qualifications none of this self-taught bootcamp BS.
There is no other solution to agentic onslaught.
We should not gate keep writing software
Anyone can write software, you can't stop them. What we can gatekeep is the building, distribution, installation, and running of software that affects critical systems, like one of the most popular OSes.
The XZ backdoor affected millions of computers, with the potential to effect hundreds of millions of computers, many of which had the capacity to affect billions of people. From one completely unregulated software library.
lol no...the main issue here is being fooled by bots. you know your irl friends and you know they are not bots...devs will just need to get out more and actually meet / get to know the people they are working with...........omg....that...that actually sounds even worse now that i say it out loud.
Prompt injection?
Or is this simply another example of why autonomous agents shouldn't get write access before earning trust?
> earning trust?
I'd argue autonomous agents shouldn't have write access at all. At least not yet.
How could they ever earn trust? They don’t have real world reputations to protect, families to support, a desire not to be punished…
Back when [1] it was fashionable to advocate FOSS as ideology [2], we were thinking about tons of FOSS adversaries and how to protect from them - some real, some imaginary. The death of FOSS would come from big closed-source vendors, or from regulators (lobbied or just ignorant), from whatever.
We never envisioned that the actual FOSS death spiral would come from progress itself, much more so from AI...
[1] Oh what fun did we have. One of us in the Greek FOSS community actually put RMS in jail. [2] Something that I think nobody except RMS ever seriously believed in.
Make PR pay. $5 per PR. You can refund, but if you get snowed by 10,000 PR then you have bank to pay for the work to ignore them.