Oh boy this was a major problem at our budding fintech. Here's what DIDN't work:
1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.
2. Phone number 2FA. Significantly slowed legitimate user access but still didn't fully stop credential stuffers.
What did work:
3. rate limits and carefully tailored scripts that detected usage patterns and autobanned. Eventually they gave up on us guess wasn't worth the trouble. However I'm sure we lost a few legitimate users too in the process.
What I would try in the future:
- Passkeys as 2fa. Most browser automation platforms can't handle passkey auth inside a VM.
Not sure why financial institutions still bother with passwords - every time i try to login to wise or something it requires email code/link. At that point just use the email auth.
I spent a year doing security for a highly targeted fintech-adjacent where credential stuffing was the primary security threat, and all non-phishing-resistant MFA was table stakes: all the real work was in combatting cred-stuffing attacks that had already defeated (usually through elaborate phishing) the MFA.
> 1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.
Don't you typically use that for valid users? As-in, you allow access when the fingerpint matches their existing fingerprint and when it doesn't you require additional information to be presented (i.e. security code).
So if somebody shifts their ip around they end up needing more information than just user+pass to login but somebody that doesn't (i.e. a normal person at home) does have the easy way to login.
There are a lot of dedicated anti-detect browsers, you can search for that term or fingerprint switcher, multi-accounting browsers, etc. Many of them are based on Chromium.
In my experience they're generally detectable by mismatches in various attributes compared to the "real" browser whose user agent they are spoofing (though of course, the ground truth of adversarial detection is always hard to know for sure).
Some years ago I researched the whole credential stuffing ecosystem for a course paper at uni.
Credential Stuffing is (or at least was) a gigantic market, and it is one of the biggest headaches for the biggest pay-walled services, like Netflix, HBO, Prime, etc.
The people that made a living out of it were stuffing millions or billions of credentials (sourced from database leaks) in the most popular services, hoping to then sell the accounts for small amounts of money, like a dollar for a Netflix account with a 10-day warranty. It's a numbers game at heart with a substantial technical aspect, where you need to optimize your checker code to essentially send properly formatted requests that can't be intercepted and don't arouse suspicion, and then you had an ecosystem of "methods" that are certain request-response chains that make your login request look like it's from a real person. People needed to figure out advanced methods to not invoke a CAPTCHA check, which is cost-prohibitive, but not impossible to solve automatically (AI wasn't a thing back then). You then have to buy millions of proxies that are able to route the requests from different IPs so that you're not sending millions of requests from a single IP. Checkers had reached a point where, depending on your proxies, were performing 10,000 or even 20,000 checks per minute. Multithreading was the cornerstone of these technologies, as a simple 2vCPU VM was already bottlenecked by proxy speeds.
Back when I looked into it, it was the wild west, as SSO and other technologies just weren't a thing yet. Companies would become fads of this credential stuffing scene, and it would take a dev team an entire sprint just for them to make a login page that was able to at least force a CAPTCHA check for each single request, and that's IF they had the proper monitoring tools to notice the gigantic spike in login requests. Having a valid account to a service like Ebay where you can then order whatever you want with the linked credit-card, you can understand how big of a security issue this is.
I haven't looked at it recently, but I assume that this has become vastly more difficult for the common-place services like streaming providers and digital goods marketplaces. SSO, IAM platforms like Keycloak, and advanced request scanning techniques have evolved. I'm guessing things have become substantially better, but it's always going to be a big issue for those smaller websites without a dedicated dev team or without at least someone maintaining them.
Password managers and 2FA mostly solve the problem of credential stuffing but unfortunately millions of people still don't use it. 2FA is actually pretty good unless your account is a high profile account then bad actors might try to SIM swap[0] you and hijack your mobile phone number and essentially bypass 2FA. But there are others ways of 2FA too.
I am so happy I am no longer responsible for these. We had a solid monitor and an analysis script that was quite good at dealing with the attacks.
Then the fun thing was that some lawyers concluded this is still a breach on success and that we should be responsible and report/mitigate these.
How? How do you stop your users from making dumb decisions? The only solution seems to be to "give up" and go passwordless, putting the credentials to the big boys in town.
For us, introducing a simple device and location validation system (track which users log in with which devices and from where), combined with breached password detection from HIBP, which both can trigger an email validation code flow, practically solved the credential stuffing issues we had immediately.
For the user it's kind of a a soft MFA via email where they don't have to enable it, but also don't always get the challenge.
Astonishingly, we had barely any complaints about the system via customer care and also didn't notice a drop in (valid) logins or conversion rates.
To me, that seems like a pretty reasonable approach... adding a password change at the end would probably be a good last add.
I tend to generate my passphrases for sites now, my only complaint is a password field should accept at least 100 characters. Assuming it's salted+hashed anyway, it's almost irresponsible to limit to under 20 characters. I'd rather see a minimum of 15 chars and a suggestion to use a "phrase or short sentence" in the hint/tip.
I wrote an auth system and integrated the zxcvbn strength check and HIBP as default enabled options. The password entry allowed for up to 1kb input, mostly as a practical limit. I also tend to prefer having auth separated from the apps, in that if auth fails via DDoS, etc, then already authenticated users aren't interrupted.
Sure, and users are annoyed they don't get to use their go-to "letmein" password as well.
But this is worth it for certain platforms, especially where your users have something real to lose when they get pwned by cred stuffing (e.g. bitcoin casino).
Personally, I'd go a step further and say it's also worth it anywhere an account takeover can impact other people like Reddit and Twitter where cred stuffing lets you take over high profile accounts and then post a crypto scam or something more targeted, for example.
Mild inconvenience on a /register form doesn't sound like a serious counterbalance to the large cost of trivial account takeovers.
Solutions exist that can completely block credential stuffing attacks most people just don't know about them, for example: https://layer3intel.com/tripwire
The author, Dan, is at FusionAuth, so that might be a good place to start.
I work for Stytch (another CIAM provider) on the fraud and security side and we do these too. I'd say you see credential stuffing defenses integrated into the auth provider rather than standalone rate limiting because so much of the relevant context is tied up in the auth side.
And, all the error messages end up being bad, as is the case for many security things. For our own features like Intelligent Rate Limiting https://stytch.com/docs/fraud/guides/device-fingerprinting/d... it's usually a bad idea to tell a user "You hit the limit, come back in an hour or contact support" because it gives an attacker information on how to improve. And we regularly see probing behavior where an attacker is trying to find the edges of a defense before starting a full-scale attack.
On the side topic of error messages - if you've ever seen "If your account exists, the password has been reset" that's another useless error message because "No account exists with that email" enables account enumeration.
Thanks for the feedback, appreciate it. I wanted a general overview of the problem and solutions first. Maybe I'll write a follow on article about services or libraries.
Love how the HN community is sharing rate limiting and other credential stuffing "war stories" here too.
at my first job I was able to identify a remote username enumeration vulnerability which, when combined with cred stuffing based on some publicly known cred dumps, netted me a couple dozen good user accounts on our web app. we ended up rate limiting login attempts and adding random delay to failed logins to mitigate the username enumeration vuln (tldr - if you sent creds w a nonexistent username the system immediately failed the login attempt, but if you sent w a good username and any password the system had to take time to hash and salt your password then compare it to the stored hashed passwords, so attempts w a good username have a much higher latency than w a bad one)
Yeah.. I started adding a random 1-4 second delay returning from failed logins regardless of the reason when creating auth systems. I really wish I would have been able to open-source the auth platform I wrote at a previous job.
It wasn't that complex, but pretty nice in that it had pretty typical features for auth, was simple to setup/configure and would simply generate a signed jwt passed to your app/url. I had db adapters to be able to use SQLite, MS-SQL, PostgreSQL and DynamoDB. It also had integrations for AD, Okta and Azure Entra.
tl;dr: "Credential stuffing is an attack where malicious actors use stolen username-password pairs from one service to attempt unauthorized access to other services. Unlike brute force attacks that try many passwords against one account, credential stuffing exploits the widespread problem of password reuse across multiple platforms. Defending against credential stuffing requires multiple strategies. Unfortunately no single defense provides complete protection."
I first noticed this weakness when I was caught in a "password breach", which was really a fishing site for passwords.
I checked the list of passwords, and a significant amount had values like "uRFcWEBSITENAME".
I reasoned first they would try USERNAME, PASSWORD, and then try USERNAME with passwords updated for the other websites they were hacking.
Even the section of passwords beginning with just the first letter of WEBSITENAME was unusually large. Then I scanned for passwords containing the first few letters of WEBSITENAME... yup, a conspicuous lot of them. Those people at least tried to be smart about their password uniqueness, but it didn't really work.
I've heard of users using a similar trick on email addresses for the purpose of finding out which site leaked their information in the future. At least now you know which sites have bad security with passwords.
I'd argue that the really old custom of providing the user a fixed username (e.g. user ID) upon registration, and requiring that for login, provides complete protection vs credential stuffing. It has other drawbacks of course, which is why you rarely see it in use anymore...
> Do you mean a username that is distinct from email or phone number?
No, he means a unique user id, generated by the server when you sign up for the service. Then for every login attempt, you provide the username/email + user id + password.
Because credential stuffing relies on the user reusing a username + password from another site. If you provide the user with a username they don't select, it won't be reused.
But then they have to remember the username AND the password? This doesn't help with users already having the password re-use problem. This would only work for those with a password manager, but then they are also less likely to re-use a password.
Also, wouldn't this prevent lost password recovery? if you can't identify a user by their email?
> But then they have to remember the username AND the password?
The commenter already acknowledged that the solution has drawbacks. The only claim made was that it solves credential stuffing, not that it doesn't inconvenience the user.
> This would only work for those with a password manager
It would also work for those without a password manager, because they'd have no choice.
> Also, wouldn't this prevent lost password recovery? if you can't identify a user by their email?
They're not mutually exclusive. You can have both. A compulsory unique user ID to login, and an email based password recovery mechanism.
Yes, there are clear ergonomic reasons why we don't do this "assign a username" thing. But it would stop password stuffing.
You'd presumably do username recovery the same way you do password recovery, so it would only be accessible to an attacker who compromised the user's email.
Oh boy this was a major problem at our budding fintech. Here's what DIDN't work:
1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.
2. Phone number 2FA. Significantly slowed legitimate user access but still didn't fully stop credential stuffers.
What did work:
3. rate limits and carefully tailored scripts that detected usage patterns and autobanned. Eventually they gave up on us guess wasn't worth the trouble. However I'm sure we lost a few legitimate users too in the process.
What I would try in the future:
- Passkeys as 2fa. Most browser automation platforms can't handle passkey auth inside a VM.
Not sure why financial institutions still bother with passwords - every time i try to login to wise or something it requires email code/link. At that point just use the email auth.
I spent a year doing security for a highly targeted fintech-adjacent where credential stuffing was the primary security threat, and all non-phishing-resistant MFA was table stakes: all the real work was in combatting cred-stuffing attacks that had already defeated (usually through elaborate phishing) the MFA.
> 1. Browser fingerprinting or ip bans. They used advanced fingerprint-shifting browsers and residential proxy ips.
Don't you typically use that for valid users? As-in, you allow access when the fingerpint matches their existing fingerprint and when it doesn't you require additional information to be presented (i.e. security code).
So if somebody shifts their ip around they end up needing more information than just user+pass to login but somebody that doesn't (i.e. a normal person at home) does have the easy way to login.
>They used advanced fingerprint-shifting browsers
I'm guessing this would be Firefox, possibly using in house extensions or userscripts designed to help further avoid fingerprinting?
There are a lot of dedicated anti-detect browsers, you can search for that term or fingerprint switcher, multi-accounting browsers, etc. Many of them are based on Chromium.
In my experience they're generally detectable by mismatches in various attributes compared to the "real" browser whose user agent they are spoofing (though of course, the ground truth of adversarial detection is always hard to know for sure).
Some years ago I researched the whole credential stuffing ecosystem for a course paper at uni.
Credential Stuffing is (or at least was) a gigantic market, and it is one of the biggest headaches for the biggest pay-walled services, like Netflix, HBO, Prime, etc.
The people that made a living out of it were stuffing millions or billions of credentials (sourced from database leaks) in the most popular services, hoping to then sell the accounts for small amounts of money, like a dollar for a Netflix account with a 10-day warranty. It's a numbers game at heart with a substantial technical aspect, where you need to optimize your checker code to essentially send properly formatted requests that can't be intercepted and don't arouse suspicion, and then you had an ecosystem of "methods" that are certain request-response chains that make your login request look like it's from a real person. People needed to figure out advanced methods to not invoke a CAPTCHA check, which is cost-prohibitive, but not impossible to solve automatically (AI wasn't a thing back then). You then have to buy millions of proxies that are able to route the requests from different IPs so that you're not sending millions of requests from a single IP. Checkers had reached a point where, depending on your proxies, were performing 10,000 or even 20,000 checks per minute. Multithreading was the cornerstone of these technologies, as a simple 2vCPU VM was already bottlenecked by proxy speeds.
Back when I looked into it, it was the wild west, as SSO and other technologies just weren't a thing yet. Companies would become fads of this credential stuffing scene, and it would take a dev team an entire sprint just for them to make a login page that was able to at least force a CAPTCHA check for each single request, and that's IF they had the proper monitoring tools to notice the gigantic spike in login requests. Having a valid account to a service like Ebay where you can then order whatever you want with the linked credit-card, you can understand how big of a security issue this is.
I haven't looked at it recently, but I assume that this has become vastly more difficult for the common-place services like streaming providers and digital goods marketplaces. SSO, IAM platforms like Keycloak, and advanced request scanning techniques have evolved. I'm guessing things have become substantially better, but it's always going to be a big issue for those smaller websites without a dedicated dev team or without at least someone maintaining them.
Password managers and 2FA mostly solve the problem of credential stuffing but unfortunately millions of people still don't use it. 2FA is actually pretty good unless your account is a high profile account then bad actors might try to SIM swap[0] you and hijack your mobile phone number and essentially bypass 2FA. But there are others ways of 2FA too.
[0] https://en.wikipedia.org/wiki/SIM_swap_scam
Is the paper public? Would love to review/reference it for the newsletter.
No unfortunately, and it's pretty old. It was a paper/report for a course during my undergrad, so not polished by any means.
I am so happy I am no longer responsible for these. We had a solid monitor and an analysis script that was quite good at dealing with the attacks.
Then the fun thing was that some lawyers concluded this is still a breach on success and that we should be responsible and report/mitigate these.
How? How do you stop your users from making dumb decisions? The only solution seems to be to "give up" and go passwordless, putting the credentials to the big boys in town.
For us, introducing a simple device and location validation system (track which users log in with which devices and from where), combined with breached password detection from HIBP, which both can trigger an email validation code flow, practically solved the credential stuffing issues we had immediately.
For the user it's kind of a a soft MFA via email where they don't have to enable it, but also don't always get the challenge.
Astonishingly, we had barely any complaints about the system via customer care and also didn't notice a drop in (valid) logins or conversion rates.
To me, that seems like a pretty reasonable approach... adding a password change at the end would probably be a good last add.
I tend to generate my passphrases for sites now, my only complaint is a password field should accept at least 100 characters. Assuming it's salted+hashed anyway, it's almost irresponsible to limit to under 20 characters. I'd rather see a minimum of 15 chars and a suggestion to use a "phrase or short sentence" in the hint/tip.
I wrote an auth system and integrated the zxcvbn strength check and HIBP as default enabled options. The password entry allowed for up to 1kb input, mostly as a practical limit. I also tend to prefer having auth separated from the apps, in that if auth fails via DDoS, etc, then already authenticated users aren't interrupted.
One solution for credential stuffing is to generate passwords for users.
I guess ChatGPT doesn't know that one, yet.
> generate passwords for users
That would rub me the wrong way for sure. Maybe not to the point of abandoning your platform, but I'd still be irked.
Sure, and users are annoyed they don't get to use their go-to "letmein" password as well.
But this is worth it for certain platforms, especially where your users have something real to lose when they get pwned by cred stuffing (e.g. bitcoin casino).
Personally, I'd go a step further and say it's also worth it anywhere an account takeover can impact other people like Reddit and Twitter where cred stuffing lets you take over high profile accounts and then post a crypto scam or something more targeted, for example.
Mild inconvenience on a /register form doesn't sound like a serious counterbalance to the large cost of trivial account takeovers.
Solutions exist that can completely block credential stuffing attacks most people just don't know about them, for example: https://layer3intel.com/tripwire
As a person with two useless undergraduate and one equally useless graduate degrees I was scared to click the article.
More details on existing services or libraries for rate limiting would be nice. Still an informative article.
The author, Dan, is at FusionAuth, so that might be a good place to start.
I work for Stytch (another CIAM provider) on the fraud and security side and we do these too. I'd say you see credential stuffing defenses integrated into the auth provider rather than standalone rate limiting because so much of the relevant context is tied up in the auth side.
And, all the error messages end up being bad, as is the case for many security things. For our own features like Intelligent Rate Limiting https://stytch.com/docs/fraud/guides/device-fingerprinting/d... it's usually a bad idea to tell a user "You hit the limit, come back in an hour or contact support" because it gives an attacker information on how to improve. And we regularly see probing behavior where an attacker is trying to find the edges of a defense before starting a full-scale attack.
On the side topic of error messages - if you've ever seen "If your account exists, the password has been reset" that's another useless error message because "No account exists with that email" enables account enumeration.
Thanks for the feedback, appreciate it. I wanted a general overview of the problem and solutions first. Maybe I'll write a follow on article about services or libraries.
Love how the HN community is sharing rate limiting and other credential stuffing "war stories" here too.
Good Article
at my first job I was able to identify a remote username enumeration vulnerability which, when combined with cred stuffing based on some publicly known cred dumps, netted me a couple dozen good user accounts on our web app. we ended up rate limiting login attempts and adding random delay to failed logins to mitigate the username enumeration vuln (tldr - if you sent creds w a nonexistent username the system immediately failed the login attempt, but if you sent w a good username and any password the system had to take time to hash and salt your password then compare it to the stored hashed passwords, so attempts w a good username have a much higher latency than w a bad one)
Yeah.. I started adding a random 1-4 second delay returning from failed logins regardless of the reason when creating auth systems. I really wish I would have been able to open-source the auth platform I wrote at a previous job.
It wasn't that complex, but pretty nice in that it had pretty typical features for auth, was simple to setup/configure and would simply generate a signed jwt passed to your app/url. I had db adapters to be able to use SQLite, MS-SQL, PostgreSQL and DynamoDB. It also had integrations for AD, Okta and Azure Entra.
[flagged]
tl;dr: "Credential stuffing is an attack where malicious actors use stolen username-password pairs from one service to attempt unauthorized access to other services. Unlike brute force attacks that try many passwords against one account, credential stuffing exploits the widespread problem of password reuse across multiple platforms. Defending against credential stuffing requires multiple strategies. Unfortunately no single defense provides complete protection."
"Unfortunately no single defense provides complete protection"
For the server, maybe. For the user, a password manager (used properly) is a solid single defense solution.
I first noticed this weakness when I was caught in a "password breach", which was really a fishing site for passwords.
I checked the list of passwords, and a significant amount had values like "uRFcWEBSITENAME".
I reasoned first they would try USERNAME, PASSWORD, and then try USERNAME with passwords updated for the other websites they were hacking.
Even the section of passwords beginning with just the first letter of WEBSITENAME was unusually large. Then I scanned for passwords containing the first few letters of WEBSITENAME... yup, a conspicuous lot of them. Those people at least tried to be smart about their password uniqueness, but it didn't really work.
I've heard of users using a similar trick on email addresses for the purpose of finding out which site leaked their information in the future. At least now you know which sites have bad security with passwords.
I'd argue that the really old custom of providing the user a fixed username (e.g. user ID) upon registration, and requiring that for login, provides complete protection vs credential stuffing. It has other drawbacks of course, which is why you rarely see it in use anymore...
Do you mean a username that is distinct from email or phone number?
If so, why does that provide protection against credential stuffing? A username can be reused across different applications.
What am I missing?
> Do you mean a username that is distinct from email or phone number?
No, he means a unique user id, generated by the server when you sign up for the service. Then for every login attempt, you provide the username/email + user id + password.
How does providing a predictable username protect against anything at all?
Because credential stuffing relies on the user reusing a username + password from another site. If you provide the user with a username they don't select, it won't be reused.
But then they have to remember the username AND the password? This doesn't help with users already having the password re-use problem. This would only work for those with a password manager, but then they are also less likely to re-use a password.
Also, wouldn't this prevent lost password recovery? if you can't identify a user by their email?
> But then they have to remember the username AND the password?
The commenter already acknowledged that the solution has drawbacks. The only claim made was that it solves credential stuffing, not that it doesn't inconvenience the user.
> This would only work for those with a password manager
It would also work for those without a password manager, because they'd have no choice.
> Also, wouldn't this prevent lost password recovery? if you can't identify a user by their email?
They're not mutually exclusive. You can have both. A compulsory unique user ID to login, and an email based password recovery mechanism.
Yes, there are clear ergonomic reasons why we don't do this "assign a username" thing. But it would stop password stuffing.
You'd presumably do username recovery the same way you do password recovery, so it would only be accessible to an attacker who compromised the user's email.