Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
musl only, the quick fix:
Which is a good advice in general.
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
> You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
1Password subscription and the memorized key. 1Password is the only place my ssh private keys are stored.
The somewhere for password authentication is a place you have physical access. Not ssh.
I log in via console, either physical or virtual (the latter requires MFA though)
A scenario where all the backups are gone and the server is somehow the only thing that survived is not a very plausible one.
Does this happen to you a lot? You may have other issues
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
imo this is a good advice only if the devices that connect to the server have their home encrypted and ssh private key protected.
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
Who does PasswordAuthentication=yes in this day and age anyway.
People who use AuthenticationMethods "publickey,password"
You must present a key first after which you must enter a password.
Just do no password at all.
Putting the one in 1fa? No thanks.
As this is apparently affecting openwrt, probably a fair number of openwrt users, if I had to guess
If you run a cockpit service on your server you will need password authentication enabled, unfortunately.
it's the default for most systems. I can't remember a time I had to not turn it off.
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
I would say that PasswordAuthentication=yes is by itself a vulnerability.
Since OpenSSH is included in the base install, and enabled on OpenBSD - is it correct that they will need to update the famous:
“Only two remote holes in the default install, in a heck of a long time!”
https://www.openbsd.org/
This vulnerability doesn’t exist on non-Linux systems, so not yet!
OpenBSD doesn't provide musl-linked OpenSSH by default, so not yet[x2]
No, the bug is in musl but manifests in OpenSSH when compiled against it.
This looks terrible for musl, what a collection of terrible decisions on their part. I guess I'm staying away from it.
To err is human. There’s no escaping it, I’m afraid.
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
I'm sure some people running lightweight containers just had a heart attack.
I would imagine the vast majority of lightweight containers don't contain an ssh server?
You'd be surprised by the amount of people using containers that don't know/care what is inside.
>We consider the root cause of this bug to be outside OpenSSH itself, but we're in a good position to deliver a mitigation quickly.
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
I presume Dropbear, thus musl-based OpenWrt routers are also not affected? Not that Dropbear has been an example in that regard.
Since after you posted, various people in the thread have reported dropbear is not affected
There is this write-up/fancy page from the person who found the vulnerability[0] (HN discussion[1], how I found this thread).
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
[0] https://sshhole.com/
[1] https://news.ycombinator.com/item?id=50378228
Workaround? Why not a fix?
AFAIU because the bug is in musl and not in OpenSSH
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.