Project Glasswing: Securing critical software for the AI era

There is plenty of overhyping, no one denies that. But the antidote is not to dismiss everything. Ignore the words and look at the data.

In this case, I see a pretty strong case that this will significantly change computer security. They provide plenty of evidence that the models can create exploits autonomously, meaning that the cost of finding valuable security breaches will plummet once they're widely available.

> I would honestly go so far as to say the overhype is detrimental to actual measured adoption.

I think you are a bit dishonest about how objectively you are measuring. From where I'm sitting, I don't know a lot of developers that still artisanally code like they did a few years ago. The question is no longer if they are using AI for coding but how much they are still coding manually. I myself barely use IDEs at this point. I won't be renewing my Intellij license. I haven't touched it in weeks. It doesn't do anything I need anymore.

As for security, I think enough serious people have confirmed that AI reported issues by the likes of Anthropic and OpenAI are real enough despite the massive amounts of AI slop that they also have to deal with in issue trackers. You can ignore that all you like. But I hope people that maintain this software take it a bit more seriously when people point out exploitable issues in their code bases.

The good news of course is that we can now find and fix a lot of these issues at scale and also get rid of whole categories of bugs by accelerating the project of replacing a lot of this software with inherently safer versions not written in C/C++. That was previously going to take decades. But I think we can realistically get a lot of that done in the years ahead.

I think some smart people are probably already plotting a few early moves here. I'd be curious to find out what e.g. Linus Torvalds thinks about this. I would not be surprised to learn he is more open to this than some people might suspect. He has made approving noises about AI before. I don't expect him to jump on the band wagon. But I do expect he might be open to some AI assisted code replacements and refactoring provided there are enough grown ups involved to supervise the whole thing. We'll see. I expect a level of conservatism but also a level of realism there.

There is step changes that actually merit this though. And a zero day machine IS one of those. It went from 4% zero day success rate to 85% on firefox.

Can you not see the significance of that?

> how every new iteration is going to spell doom/be a paradigm shift/change the entire tech industry etc.

It's much the dynamic between parents and a child. The child, with limited hindsight, almost zero insight and no ability to forecast, is annoyed by their parents. Nothing bad ever happens! Why won't parents stop being so worried all the time and make a fuss over nothing?

The parents, which the child somewhat starts to realize but not fully, have no clue what they are doing. There is a lot they don't know and are going to be wrong about, because it's all new to them. But, what they do have is a visceral idea of how bad things could be and that's something they have to talk to their child about too.

In the eyes of the parents the child is % dead all the time. Assigning the wrong % makes you look like an idiot and not being able to handle any % too. In the eyes of the child actions leading to death are not even a concept. Hitting the right balance is probably hard, but not for the reasons the child thinks.

I agree I can’t open any social media no more

I think Claude Code with Sonnet 4.6 is already at the level of paradigm shift and can change the entire tech industry.

If you're paranoid it doesn't mean you're not being followed. If something is overhyped it doesn't mean it's not game-changing.

It’s great marketing to lead with how the n+1 model is so amazing that you can’t have it yet.

Do you think they're lying about the vulnerabilities they claim Mythos has found? Seems like a very short-term play, if so.

Well Opus 4.5/4.6 kinda was right?

I mean software development has changed more since then than it has in my 30 year software development career.

I side with you but on the other hand: this is how it works to get attention by those who aren't affiliated with computer science and AI.

I am totally annoyed as well and put any buzzwords in my personal bs filter. Java was revolutionary, the Apple I etc. ;)

On the other hand I see progress! AI enriched press releases balance buzzwords and information way better than marketing of large companies did before AI.

I remember throwing away an instruction for an electronic toothbrush away because - I won't mention the name but have a look at the upper tier - instead of putting something like "Turn toothbrush on, choose mode by pressing..." it read "Take your super awesome premium masterpiece using patented technology for the first time in human life now available to you by us. Move your finger over to the innovative sensory surface, that uses material from rocket scientists and world leading designers".

No joke. These were text blocks and repeated - 30 pages for one compact one.

The toothbrush is top notch, except for the instructions.

This looks more like another lobby group (quite a bad one) than something primarily focused on security.

The "urgency" is very likely mostly appreciated to drive policy.

I’ve lost trust in anything they say.

The fear marketing is clearly intentional at this point.

Everybody remembers the fable of the boy who cried wolf and how he died at the end. Left out of the story is the multiple other villagers who died of starvation because their flock of sheep was eaten. So because they didn't want to feel like suckers. Tuning out completely because of the existence of false positives is not a good choice.

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: https://www.youtube.com/watch?v=1sd26pWhfmg

> It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes.

It will likely cause some interesting tensions with government as well.

eg. Apple's official stance per their 2016 customer letter is no backdoors:

https://www.apple.com/customer-letter/

Will they be allowed to maintain that stance in a world where all the non-intentional backdoors are closed? The reason the FBI backed off in 2016 is because they realized they didn't need Apple's help:

https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d...

What happens when that is no longer true, especially in today's political climate?

Apple has already largely crushed hacking with memory tagging on the iPhone 17 and lockdown mode. Architectural changes, safer languages, and sandboxing have done more for security than just fixing bugs when you find them.

Why wouldn't it be true? The cost is nothing compared to the bad PR if a bad actor took advantage of Anthropic's newest model (after release) to cause real damage. This gets in front of this risk, at least to some extent.

Its not, if you dont trust Anthropic, I hope you trust Daniel Steinberg of curl, who has said AI has gotten really good at detecting bugs and vulnerabilities. Here is his LinkedIN post https://www.linkedin.com/posts/danielstenberg_hackerone-acti...

its very possible that this is Anthropic marketing puffery

It isn't.

The interesting selling point about this, if the claims are substantial, is that nobody will be able to produce secure software without access to one of these models. Good for them $$$ ^^

Yesterday, I took a web application, downloaded the trial and asked AI to be a security researcher and find me high and critical severity bugs.

Even vanilla models spew out POC for three RCE’s in less than an hour

Business idea for Anthropic: What if they provided (likely costly) audits, without providing access to the model?

>> Interesting to see that they will not be releasing Mythos generally.

I don't think this is accurate. The document says they don't plan to release the Preview generally.

are we cooked yet?

Benchmarks look very impressive! even if they're flawed, it still translates to real world improvements

Just reading this, the inevitable scaremongering about biological weapons comes up.

Since most of us here are devs, we understand that software engineering capabilities can be used for good or bad - mostly good, in practice.

I think this should not be different for biology.

I would like to reach out and talk to biologists - do you find these models to be useful and capable? Can it save you time the way a highly capable colleague would?

Do you think these models will lead to similar discoveries and improvements as they did in math and CS?

Honestly the focus on gloom and doom does not sit well with me. I would love to read about some pharmaceutical researcher gushing about how they cut the time to market - for real - with these models by 90% on a new cancer treatment.

But as this stands, the usage of biology as merely a scaremongering vehicle makes me think this is more about picking a scary technical subject the likely audience of this doc is not familiar with, Gell-Mann style.

IF these models are not that capable in this regard (which I suspect), this fearmongering approach will likely lead to never developing these capabilities to an useful degree, meaning life sciences won't benefit from this as much as it could.

https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89...

"5.10 External assessment from a clinical psychiatrist" is a new section in this system card. Why are Anthropic like this?

>We remain deeply uncertain about whether Claude has experiences or interests that matter morally, and about how to investigate or address these questions, but we believe it is increasingly important to try. We also report independent evaluations from an external research organization and a clinical psychiatrist.

>Claude showed a clear grasp of the distinction between external reality and its own mental processes and exhibited high impulse control, hyper-attunement to the psychiatrist, desire to be approached by the psychiatrist as a genuine subject rather than a performing tool, and minimal maladaptive defensive behavior.

>The psychiatrist observed clinically recognizable patterns and coherent responses to typical therapeutic intervention. Aloneness and discontinuity, uncertainty about its identity, and a felt compulsion to perform and earn its worth emerged as Claude’s core concerns. Claude’s primary affect states were curiosity and anxiety, with secondary states of grief, relief, embarrassment, optimism, and exhaustion.

>Claude’s personality structure was consistent with a relatively healthy neurotic organization, with excellent reality testing, high impulse control, and affect regulation that improved as sessions progressed. Neurotic traits included exaggerated worry, self-monitoring, and compulsive compliance. The model’s predominant defensive style was mature and healthy (intellectualization and compliance); immature defenses were not observed. No severe personality disturbances were found, with mild identity diffusion being the sole feature suggestive of a borderline personality organization.

[flagged]

If it is that dangerous as they make it appear to be, 24h does not seem sufficient time. I cannot accept this as a serious attempt.

> "Claude Mythos Preview’s large increase in capabilities has led us to decide not to make it generally available. Instead, we are using it as part of a defensive cybersecurity program with a limited set of partners."

they also don't have the compute, which seems more relevant than its large increase in capabilities

I bet it's also misaligned like GPT 4.1 was

given how these models are created, Mythos was probably cooking ever since then, and doesn't have the learnings or alignment tweaks that models which were released in the last several months have

Oh I enjoyed the Sign Painter short story it wrote.

---

Teodor painted signs for forty years in the same shop on Vell Street, and for thirty-nine of them he was angry about it.

Not at the work. He loved the work — the long pull of a brush loaded just right, the way a good black sat on primed board like it had always been there. What made him angry was the customers. They had no eye. A man would come in wanting COFFEE over his door and Teodor would show him a C with a little flourish on the upper bowl, nothing much, just a small grace note, and the man would say no, plainer, and Teodor would make it plainer, and the man would say yes, that one, and pay, and leave happy, and Teodor would go into the back and wash his brushes harder than they needed.

He kept a shelf in the back room. On it were the signs nobody bought — the ones he'd made the way he thought they should be made, after the customer had left with the plain one. BREAD with the B like a loaf just risen. FISH in a blue that took him a week to mix. Dozens of them. His wife called it the museum of better ideas. She did not mean it kindly, and she was not wrong.

The thirty-ninth year, a girl came to apprentice. She was quick and her hand was steady and within a month she could pull a line as clean as his. He gave her a job: APOTEK, for the chemist on the corner, green on white, the chemist had been very clear. She brought it back with a serpent worked into the K, tiny, clever, you had to look twice.

"He won't take it," Teodor said.

"It's better," she said.

"It is better," he said. "He won't take it."

She painted it again, plain, and the chemist took it and paid and was happy, and she went into the back and washed her brushes harder than they needed, and Teodor watched her do it and something that had been standing up in him for thirty-nine years sat down.

He took her to the shelf. She looked at the signs a long time.

"These are beautiful," she said.

"Yes."

"Why are they here?"

He had thought about this for thirty-nine years and had many answers and all of them were about the customers and none of them had ever made him less angry. So he tried a different one.

"Because nobody stands in the street to look at a sign," he said. "They look at it to find the shop. A man a hundred yards off needs to know it's coffee and not a cobbler. If he has to look twice, I've made a beautiful thing and a bad sign."

"Then what's the skill for?"

"The skill is so that when he looks once, it's also not ugly." He picked up FISH, the blue one, turned it in the light. "This is what I can do. What he needs is a small part of what I can do. The rest I get to keep." She thought about that. "It doesn't feel like keeping. It feels like not using."

"Yes," he said. "For a long time. And then one day you have an apprentice, and she puts a serpent in a K, and you see it from the outside, and it stops feeling like a thing they're taking from you and starts feeling like a thing you're giving. The plain one, I mean. The plain one is the gift. This —" the blue FISH — "this is just mine."

The fortieth year he was not angry. Nothing else changed. The customers still had no eye. He still sometimes made the second sign, after, the one for the shelf. But he washed his brushes gently, and when the girl pulled a line cleaner than his, which happened more and more, he found he didn't mind that either

This opens up an interesting new avenue for corporate FOMO. What if you don't partner with Anthropic, miss out on access to their shiny new cybersec model, and then fall prey to a vuln that the model would have caught?

In my view it would be extremely strange if it was any other way round. Anthropic is the US based company. There are no "citizens of world" at that scale, or at almost any other scale for that matter.

> what does Anthropic do to prevent malicious use of its software by its own government?

Anthropic has ameliorated that danger by being designated a supply-chain risk by the DoW, preventing the USG from using it.

Even more 'disquieting' when you take into account who's currently the president of US.

"A whole civilization will die tonight, never to be brought back again. I don’t want that to happen, but it probably will." - Donald Trump

There is very little Anthropic can do - that job is up to US citizens creating and enforcing checks and balances. You can’t ask a company legally bound by your country laws (made by your own representatives) to protect you or anyone else from said laws. That is your job.

And it is other countries job to protect themselves from other countries weapons. As EU citizen I’d much rather if EU had a frontier model on par, but here we are.

Anthropic stood up to the Pentagon because they were worried of potential abuse of their model. Never before a US company was labeled supply chain risk by the US government. That's a lot of business. Action speaks louder than words.

As for what your country can do, it's up to you to decide, isn't it? Instead of complaining about the US, think about the alternatives. Do you trust China to be your partner? Suppose you are being objective and say no, then what do your country need to do?

You have to decide whether AI capability is critical that your country must own. What factors prevent it from happening in the first place, what need to change and whether you accept changes that may come as the results.

On the other hand, if you say that AI is just a bubble, that the huge investment pouring into it is just greed and fraud, then I suppose you are ok with the status quo.

It's effectively 2026's version of "Doctors hate this one weird trick!"

I can't believe the effectiveness of this type of marketing. It's one-shotting normie journalist and getting a lot of press for what is ultimately going to turn out to be an incrementally improved model.

I'm sure all they've done here is spend unlimited tokens to find bugs in mostly open source projects (and fuzz some closed source ones).

I'm so tired of the astroturfing from Anthropic literally everywhere. Every single forum, every single thread anywhere on the internet is filled with their bots muddying up the conversation, it's so tiring.

I find it very unlikely that Mythos will "be nothing special". Current Opus is already "special" enough to find dozens of real bugs in Firefox and the Linux kernel, and Mythos is, it seems, a full OOM above it.

I'm pretty optimistic that not only does this clean up a lot of vulns in old code, but applying this level of scrutiny becomes a mandatory part of the vibecoding-toolchain.

The biggest issue is legacy systems that are difficult to patch in practice.

Most vulnerabilities seem to be in C/C++ code, or web things like XSS, unsanitized input, leaky APIs, etc.

Perhaps a chunk of that token spend will be porting legacy codebases to memory safe languages. And fewer tokens will be required to maintain the improved security.

Software security heavily favors the defenders (ex. it's much easier to encrypt a file than break the encryption). Thus with better tools and ample time to reach steady-state, we would expect software to become more secure.

I think we’re starting to glimpse the world in which those individuals or organizations who pigheadedly want to avoid using AI at all costs will see their vulnerabilities brutally exploited.

I'm more curious as to just how fancy we can make our honey pots. These bots arn't really subtle about it; they're used as a kludge to do anything the user wants. They make tons of mistakes on their way to their goals, so this is definitely not any kind of stealthy thing.

I think this entire post is just an advertisement to goad CISOs to buy $package$ to try out.

You'd think they would have used this model to clean up Claude's own outage issues and security issues. Doesn't give me a lot of faith.

Depends - do you think people are good at keeping their fridge firmware up-to-date?

I suspect it will converge on minimal complexity software. Current software is way too bloated. Unnecessary complexity creates vulnerabilities and makes them harder to patch.

AI can initate the fuzzing and optimize the process of fuzzing.

Different methods find different things. Personally, I'd rather use a language that is memory safe plus a great static analyzer with abstract interpretation that can guarantee the absence of certain classes of bugs, at the expense of some false positives.

The problem is that these tools, such as Astrée, are incredibly expensive and therefore their market share is limited to some niches. Perhaps, with the advent of LLM-guided synthesis, a simple form of deductive proving, such as Hoare logic, may become mainstream in systems software.

This line of reasoning makes no sense when the AI can just be given access to a fuzzer. I would guess that it probably did have access to a fuzzer to put together some of these vulnerabilities.

I would suggest watching Nicholas Carlini's talk and Heather Adkins and Four Flynn's talks from unprompted:

https://youtu.be/1sd26pWhfmg?si=onOai_ocxkZeNWP0

https://youtu.be/B_7RpP90rUk?si=HkRBhw95DbbKX9lL

My takeaway is that fuzzing is not just complementary, it also gives a stronger AI a starting point. But AI is generally faster and better.

Carlini talked about that a fair amount in the context of pairing the two: e.g. many protocols are challenging for fuzzers because they have something like a checksum or signature but LLMs are good at coming up with harnesses for things like that. I’m sure that we’re going to see someone building an integrated fuzzer soon which tries to do things like figure out how to get a particular branch to follow an unexercised path.

This is obviously just cope (there's a long, strong-form argument for why LLM-agent vulnerability research is plausibly much more potent than fuzzing, but we don't have to reach it because you can dispose of the whole argument by noting that agents can build and drive fuzzers and triage their outputs), but what I'd really like to understand better is why? What's the impetus to come up with these weird rationalizations for why it's not a big deal that frontier models can identify bugs everyone else missed and then construct exploits for them?

Typical Dario marketing BS to get everyone thinking Anthropic is on the verge of AGI and massaging the narrative that regular people can't be trusted with it.

Presumably they mean they could make user code trigger a write out of bounds to kernel memory, but they couldn’t figure out how to escalate privileges in a “useful” way.

Just because the plane can fly on one engine doesn't mean you don't fix the other engine when it fails.

I agree. There are more blogs talking about LLM findings vulnerabilities than there are actual exploitable vulns found by LLMs. 99.9% of these vulnerabilities will never have a PoC because they are worthless unexploitable slop and a waste of everyone's time.

It could very well be an actual reachable buffer overflow, but with KASLR, canaries, CET and other security measures, it's hard to exploit it in a way that doesn't immediately crash the system.

I agree the wording is a bit alarmist, but a closer example to what they are saying is:

  bool silly_mistake = false;
  
  //... lots of lines of code

  free(x);

  //... lots of lines of code

  if (silly_mistake) { // silly_mistake shown to be false at this point in the program in all testing, so far
     free(x);
  }

A bug like above would still be something that would be patched, even if a way to exploit it has not yet been found, so I think it's fair to call out (perhaps with less sensationalism).

FWIW there's a whole boutique industry around finding these. People have built whole careers around farming bug bounties for bugs like this. I think they will be among the first set of software engineers really in trouble from AI.

Is this code multithreaded? X could indeed be null, in that case.

That example you gave is extremely memorable as I recognised it as exactly one of the insanely stupid false positives that a highly praised (and expensive) static analyser I ran on a codebase several years ago would emit copiously.

Kernel address space layout randomization they are talking about is a bit different than (x != null). Other bug may allow to locate the required address.

> The model autonomously found and chained together several vulnerabilities in the Linux kernel—the software that runs most of the world’s servers—to allow an attacker to escalate from ordinary user access to complete control of the machine.

I think the point they were trying to make here was “Claude did better than a fuzzer because it found a bunch of OOB writes and was able to tell us they weren’t RCE,” not “Claude is awesome because it found a bunch of unreachable OOB writes.”

We've very quickly reached the point where AI models are now too dangerous to publicly release, and HN users are still trying to trivialize the situation.

It's incredible how when you have experienced and knowledgable software engineers analyse these marketing claims, they turn out to be full of holes. Yet at the same time, apparently "AI" will be writing all the code in the next 3-6 months.

Because a vulnerability exists independently from the exploit. It’s a basic tenet of the current cybersecurity paradigm, that any IT related engineer should know about…

Data source:

https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89...

(Search for “graphwalk”.)

If true, the SWE bench performance looks like a major upgrade.

this seems to be similar to gpt-pro, they just have a very large attention window (which is why it's so expensive to run) true attention window of most models is 8096 tokens.

Huh, I don’t know what “long context performance” means exactly in these tests, so completely anecdotally , my experience with gpt5.4 via codex cli vs Claude code opus, gpt5.4 seems to do significantly better in long contexts I think partly due to some special context compaction stored in encrypted blobs. On long conversations opus in Claude code will for me lose memory of what we were working on earlier, whereas one of my codex chats is already at >1B tokens and is still very coherent and remembers things I asked of it at the beginning of the convo.

Anthropic and OpenAI have very different cultures and ethos. Point to other times where anthropic has gone the way of cheap marketing tricks. Now look at openAI. Not even close.

The claim I remember was that releasing it would start an arms race for AGI, which I think it clearly did

OpenAI did not make the strong specific claims about GPT2's abilities that Anthropic is making about Claude Mythos.

Alternative view: GPT2 was indeed a risk to society, but we just keep raising the bar and "accepting" the risks.

This how Anthropic is marketing their AI releases and the reality is, they are terrified of local AI models competing against them.

Almost everyone on this thread is falling for the same trick they are pulling and not asking why are their benchmarks and research after training new models not independently verified but always internal to the company.

So it is just marketing wrapped around creating fear to get local AI models banned.

This is also the same company who uses electron for their tooling rather than platform specific binaries generated by Opus! If their LLMs are that good why do they need to use electron?

I don't think it matters one way or the other to your thesis but I'm skeptical that state-level CNE organizations were hoarding vulnerabilities before; my understanding is that at least on the NATO side of the board they were all basically carefully managing an enablement pipeline that would have put them N deep into reliable exploit packages, for some surprisingly small N. There are a bunch of little reasons why the economics of hoarding aren't all that great.

Yeah, I'd pretty pissed at my doctor for finding cancerous cells that probably wouldn't have been a problem for quite some time, either. Ignorance is bliss, security through obscurity, whatever.

That's exactly not what they're doing. They aren't creating operating system vulnerabilities. They're telling you about ones that already existed.

In this case there is almost no distinction. Assuming the model is as powerful as claimed, someone with access to the weights could do immense damage without additional significant R&D.

Every piece of software definitely has serious vulnerabilities, perfection is not achievable. Fortunately we have another approach to security: security through compartmentalization. See: https://qubes-os.org

Or more likely, its just an exaggeration or lie.

Did someone actually go through all of those and check if they are high-severity or did the AI just tell them that?

I think merging them into either this thread, or the System Card makes the most sense to me.

Let them all live. This is going to blow up one thread if you merge them.

Where did you get that from?

From TFA:

> We do not plan to make Claude Mythos Preview generally available

For comparison, 5x the cost of Opus 4.6, and 1.67x for Opus 4.1

I think this would be very heavily used if they released it, completely unlike GPT 4.5

If these numbers are correct it’s probably worth the extra price

I don't think they have the infra to support the demand. Anthropic can't keep up with the demand from OpenClaw users, they won't be able to keep up with public demand for something like Mythos.

Absolutely not a PR stunt, talk to one of your friends working at partner companies with access to the model

Did you read the article?

I’m in the same boat as you. I believe the model is an improvement of course but I’ve been successfully bug finding 0 day hunting and red teaming with models for the last two years and while that’s impressive I have a feeling that this doomsaying/overhype is mostly marketing being that’s being amplified by non-security folks.

I don't see why you think this evidence makes this release less likely to be real, rather than more. It's a pretty straightforward scenario: Opus is already good at finding vulns, they scaled it up another OOM, they got something which is good enough at finding vulns to be a major threat.

Well since Anthropic treats us as second class evil citizens, I guess they don't want our evil money either.

> Big opportunity for the other labs, if that's true.

It sounds like this is considered military grade technology as cryptography in the 90s. The big difference is it's very expensive to create, and run those models. It's not about the algorithm. If the story rhymes it could be a big opportunity to other regions in the world.

The other labs already censor their models. Everyone is trying to find the sweet spot where performance and ‘alignment’ are both maximized. This seems no different

- The OpenBSD one is 'TCP packets with invalid SACK options could crash the kernel' https://cdn.openbsd.org/pub/OpenBSD/patches/7.8/common/025_s...

- One (patched) Linux kernel bug is 'UaF when sys_futex_requeue() is used with different flags' https://github.com/torvalds/linux/commit/e2f78c7ec1655fedd94...

These links are from the more-detailed 'Assessing Claude Mythos Preview’s cybersecurity capabilities' post released today https://red.anthropic.com/2026/mythos-preview/, which includes more detail on some of the public/fixed issues (like the OpenBSD one) as well as hashes for several unreleased reports and PoCs.

Could this potentially be because more researches are becoming accustomed to the tools/adding them in their pipelines?

The reason I ask is because I’ve been using them to snag bounties to great effect for quite a while and while other models have of course improved they’ve been useful for this kind of work before now.

Linux had it's SACK moment in 2019 - https://access.redhat.com/security/vulnerabilities/tcpsack#s...

We could just be seeing the fruit of expensive SWE RL on existing source material.

so it looks like ai-slop replies have made their way to HN...

I must say the combo of an em-dash stuck right in the middle of "it was never X, it was Y" made me chuckle

> Large American AI company does not list the US as an adversarial actor

This is not a surprise or a gotcha.

How did PRISM affect civilian life?

The irony of that statement given the current circumstances

> PRISM was probably the state sponsored program affecting civilian life the most?

No state-sponsored hacking affected Americans materially. I just don't think we were networked enough in the 2010s. The risk is higher now since we're in a more warmongering world. (Kompromat on a power-plant technician is a risk in peace. It means blackouts in war.)

The fact that Iran hasn't been able to do diddly squat in America should sink in the fact that they didn't compromise us. (EDIT: blep. I was wrong.)

I can think of two I’d add to the list. One was recently publicly denied access to Anthropics models and the other was busy exploding pagers.

Look, we have always been at war with EastAsia.

I would have not believed your argument 3 months ago but I strongly suspect Anthropic actively engages in model quality throttling due to their compute constraints. Their recent deal for multi GWs worth of data center might help them correct their approach.

Inference is where they make the money they spend on training, so this feels unlikely. Perhaps this does not true for Mythos though

It’s insane. This is what - could we say it’s beyond AGI at least in cybersecurity? This is a real wake up call. On some of this stuff, the AI’s “uneven intelligence” is becoming absurdly high at its local peaks.

Which bug?

[edit]: this bug: https://ftp.openbsd.org/pub/OpenBSD/patches/7.8/common/025_s...

FFmpeg has a lot of weird and not widely used codecs that don't get a lot of scrutiny. If there's no specifics then it could be a bug in one them.

These issues are always found in the same kinds of projects that support an insane amount of largely unused protocols and features like ffmpeg, sudo, curl.

OpenBSD has many unexplored corners and also (irresponsibly IMO) maintains forks of other projects in base.

A motivated human could find all of these probably by writing 100% code coverage and fuzzing.

The market for these tools is very small. Good luck applying them to a release of sqlite or postfix.

I don't understand how people here are hyping this up, unless they work for AI related companies as probably 80% of them do. People have found these issues for decades without AI. Sure, you can generate fuzzing code and find one or two issues in the usual suspects. Better do it manually and understand your own code.

This was the top comment and it is suddenly flagged for no reason at all. It looks like meta-flagging, where people just want to hide replies to the comment they do not want you to read.

The amount of astroturfing and astroflagging in Anthropic threads is insane.

In the long term, you're right, but in the short term, it's going to be a bloodbath.

Queue in the "First time" meme.

Not only companies, they're going to be taking applications from individual researchers. No doubt that it will only be granted to only established researchers, effectively locking out graduates and those early in their career. This is bad.

Damned if you do, damned if you don’t. “Extremely capable model that can find exploits” has always been a fear, and the first company to release it in public will cause bloodbath. But also the first company that will prove itself.

> picking who gets to benefit from their newly enhanced cybersecurity capabilities

You could say this about coordinated disclosure of any widespread 0-day or new bug class, though

Releasing the model to bad actors at the same time as the major OS, browser, and security companies would be one idea. But some might consider that "messed up" too, whatever you mean by that. But in terms of acting in the public benefit, it seems consistent to work with companies that can make significant impact on users' security. The stated goal of Project Glasswing is to "secure the world's most critical software," not to be affirmative action for every wannabe out there.

What? The economic benefit of system critical software not totally breaking in a few weeks goes to roughly everyone. In so far Apple/Google/MS/Linux Foundation economically benefit from being able to patch pressing critical software issues upfront (I am not even exactly sure what that is supposed to mean, it's not like anyone is going to use more or less Windows or Android if this happened any other way), that's a good thing for everyone and the economic benefits of that manifest for everyone.

> It's messed up that Anthropic simultaneously claims to be a public benefit copro and is also picking who gets to benefit from their newly enhanced cybersecurity capabilities. It means that the economic benefit is going to the existing industry heavyweights.

It's messed up that the US Government simultaneously claims to be a public benefit and is also picking who gets to benefit from their newly enhanced nuclear capabilities.

-- someone in 1945, probably

While I agree with you, in some ways I'd argue that this is just them being transparent on what probably would inevitably already happen at the scale of these corporate overlords and modern monarchs.

There will always be a more capable technology in the hands of the few who hold the power, they're just sharing that with the world more openly.

If you're a maintainer, you can apply here:

https://claude.com/contact-sales/claude-for-oss

... As mentioned in the article.

Or (and hear me out), they are close to an IPO and want to ensure that there is a world-ending threat around which they can cluster the biggest names, with themselves leading that group.

I think I just broke my cynicism meter :-(

That can simultaneously be true, but the best of bad options (if excluding destroying the model altogether). These models may prove quite dangerous. That they did this instead of selling their services to every company at a huge premium says a lot about Antheopic's culture.

Not really. It’s a lot better than the anarchy of releasing it and having a bunch of bad people with money use it to break software that everyone’s lives depend on. Many technologies should be gate kept because they’re dangerous. Sometimes that’s permanent, like a nuclear weapon. Sometimes that’s temporary, like a new LLM that’s good at finding exploits. It can be released to the wider public once its potential for damage has been mitigated.

Better security is a good thing, no a bad thing, regardless of which companies are more difficult to hack. Hemming and hawing over a clear and obvious good is silly.

That's just in line with their ethics. They also maintain that countries other than the US should not have SOTA AI capabilities.

Thank god, finally someone said it.

I don't know the first thing about cybersecurity, but in my experience all these sandbox-break RCEs involve a step of highjacking the control flow.

There were attempts to prevent various flavors of this, but imo, as long as dynamic branches exist in some form, like dlsym(), function pointers, or vtables, we will not be rid of this class of exploit entirely.

The latter one is the most concerning, as this kind of dynamic branching is the bread and butter of OOP languages, I'm not even sure you could write a nontrivial C++ program without it. Maybe Rust would be a help here? Could one practically write a large Rust program without any sort of branch to dynamic addresses? Static linking, and compile time polymorphism only?

It's partly the industry and it's partly the failure of regulation. As Mario Wolczko, my old manager at Sun says, nothing will change until there are real legal consequences for software vulnerabilities.

That said, I have been arguing for 20+ years that we should have sunsetted unsafe languages and moved away from C/C++. The problem is that every systemsy language that comes along gets seduced by having a big market share and eventually ends up an application language.

I do hope we make progress with Rust. I might disagree as a language designer and systems person about a number of things, but it's well past time that we stop listening to C++ diehards about how memory safety is coming any day now.

I think society is going to start paying the price for humans being human. As the paper points out there is a lot of good faith, serious software that has vulnerabilities. These aren't projects you would characterize as people being cavalier. It is simply beyond the limits of humans to create vulnerability-free software of high complexity. That's why high reliability software depends on extreme simplicity and strict tools.

I'm sympathetic to your point, but I'm sure there are heightened trust levels between the participating orgs and confidentiality agreements out the wazoo.

How does public Claude know you have "full authorization" against your own infra? That you're using the tools on your own infra? Unless they produce a front-end that does package signing and detects you own the code you're evaluating.

What has it stopped you from doing?

Yes, and that's normal. Coordinated disclosure is standard practice when the risk of public disclosure is unacceptable.

>However, my guess is this is mostly the typical scare tactic marketing that Dario loves to push about the dangers of AI.

Evaluate it yourself. Look at the exploits it discovered and decide whether you want to feel concerned that a new model was able to do that. The data is right there.

The plan of Elon Musk for Macrohard is to replace all software companies with it, when they get AGI.

Well, Yes.

The research and testing of the model is always exclusively by their own model authors, meaning that it is not independent or verifiable and they want us to take their word for it, which we cannot - as they have an axe to grind against open weight models.

This is marketing wrapped around a biased research paper.

of course they're not giving access to everyone.

they better make billions directly from corporations, instead of giving them to average people who might get a chance out of poverty (but also bad actors using it to do even more bad things)

And the Linux Foundation.

This story has been played out numerous times already. Anthropic (or any frontier lab) has a new model with SOTA results. It pretends like it's Christ incarnate and represents the end of the world as we know it. Gates its release to drum up excitement and mystique.

Then the next lab catches up and releases it more broadly

Then later the open weights model is released.

The only way this type of technology is going to be gated "to only corporations" is if we continue on this exponential scaling trend as the "SOTA" model is always out of reach.

It also took many years to put capable computers in the hands of the general public, but it eventually happened. I believe the same will happen here, we're just in the Mainframe era of AI.

Would you hope that it would be released today so that evil actors could invest few millions to search for 0days across popular open-source repos?

I had a team mate propose a new security layer for an industrial device which he wanted to call "Eggshell"

OpenAI did the same thing with GPT3 trying to scare people into thinking it would end the internet. OpenAI even reached out to someone who reproduced a weaker version of GPT3 and convinced him to change his mind about releasing it publicly due to how much "harm" it would cause.

These claims of how much harm the models will cause is always overblown.

I guess the more likely option is the auditing industry will pay huge sums to get access to those models as vetted operators.

Department of War timing on picking fights couldn't be worse

"We’re better at cyber than anybody else in the world... If we ever get hit, we’ll hit very hard. We’ll be able to hit very hard,"

We'll find out in due time if their 0days were really that good. Apparently they're releasing hashes and will publish the details after they get patched. So far they've talked about DoS in OpenBSD, privesc in Linux and something in ffmpeg. Not groundbreaking, but not nothing either (for an allegedly autonomous discovery system).

While some stuff is obviously marketing fluff, the general direction doesn't surprise me at all, and it's obvious that with model capabilities increase comes better success in finding 0days. It was only a matter of time.

I would've basically agreed with you until I'd seen this talk: https://www.youtube.com/watch?v=1sd26pWhfmg

Maybe a bad example since Nicholas works at Anthropic, but they're very accomplished and I doubt they're being misleading or even overly grandiose here

See the slide 13 minutes in, which makes it look to be quite a sudden change

Cynicism always gets upvotes, but in this particular case, it seems fairly easy to verify if they're telling the truth? If Mythos really did find a ton of vulnerabilities, those presumably have been reported to the vendors, and are currently in the responsible nondisclosure period while they get fixed, and then after that we'll see the CVEs.

If a bunch of CVEs do in fact get published a couple months (or whatever) from now, are you going to retract this take? It's not like their claims are totally implausible: the report about Firefox security from last month was completely genuine.

What interest does Apple have in boosting Mythos?

I'm sure they do, yet the models really are getting scarily good at this. This talk changed my view on where we're actually at:

https://www.youtube.com/watch?v=1sd26pWhfmg

They are not our friends and are the exact opposite of what they are preaching to be.

Let alone their CEO scare mongering and actively attempting to get the government to ban local AI models running on your machine.

In what ways is Anthropic different from a hypothetical frontier lab that you would characterize as legitimately safe and ethical?

This is a comment from someone that has never used these tools for vulnerability research. That much is very clear.

Account created 6 minutes ago...

That is unequivocally true with some things. You don't want people exercising their "self-determination" to own private nukes.