If you’re wondering if GitHub has been down more since the Microsoft acquisition, I actually just recently did some analysis on that using their status page data (the answer is yes):
Or it could be some of the older employees decided to cash out after the acquisition, and they've lost some institutional knowledge about how to keep things running smoothly?
The addition of interns and new members typically cause more outages, as they make a lot of changes and discover how things work or not.
Retaining older employees has little effect to prevent that, because they typically don't/can't review much of what's happening from a new team, sometimes a whole department, who joined in on the project and has its own agenda.
I was just thinking that Github stayed fairly consistent and Bitbucket didn't really challenge it on features, and then Gitlab came in, shipped loads of popular features and lots of CI functionality, and Github has been delivering a lot of new features a lot faster ever since really, so I think Microsoft acquisition is quite possibly not a root cause of downtime changes.
It's also worth noting that downtime and issues caused during updates in self-hosted Github are significant. I have experienced them, and I guess the main difference is actually that you can schedule the risk, rather than accept it arbitrarily which has positives but comes at significant cost too.
If you’re wondering if this^ website is hugged to death right now, I checked it on downforeveryoneorjustme.com at 11:08 UTC/04:08 PDT/07:08 EDT(the answer is also yes)
I set up a self hosted Gitlab where on-prem was a requirement and it's surprisingly easy. Obviously another service that needs to be maintained and updated but as code repositories and connected services are a crucial part of infrastructure it makes sense to have control over that.
Not the biggest fan of Gitlabs UI but I got used to it quickly.
How much downtime could it get? They might redesign the homepage a few times and break it accidentally, then it's done. They're not gonna redesign it again and again every week forever.
You under-appreciate how enterprise software development tends to go.. there's always new "features" to be had, KPIs to measure and no time to tackle tech debt...
Hosting your own does not guarantee you 100% uptime. It just means someone less expert than the GitHub folks will be responsible for bringing things back online. It also costs you time and effort, which is a problem if you're on a deadline.
Same goes for the question of where to host it physically. It seems unlikely your physical server will have better uptime than a virtual server in the cloud.
> Why pay someone to mess it up, when I can mess it up for free?
Because it isn't free. Your time is a huge cost.
A senior dev who spends even 10 hrs on standing up a git server has blown through years' worth of GitHub costs, and that's assuming you're even actually using the paid service.
Factor in the extreme security requirements of a code server, including needing to update dependencies daily, and you're spending far more time self-hosting with riskier results.
To add to this, and at the risk of restating my earlier points: even if you get it working, it still probably won't be as reliable as a provider like GitHub.
Keeping it secure is no small thing, especially if you want to permit access from arbitrary IPs on the Internet (rather than using a VPN, say). GitHub does this, and presumably they have solutions in place for everything from intrusion-detection to DDoS protection.
GitHub employs people to take care of server failover and data backups. You could spend your own time building your own solutions here, but they're unlikely to be as good as GitHub's. Your solution is guaranteed to be less well tested.
And that's assuming you even have a server room in the first place. You could run your own Git in the cloud, of course, but you're not really 'running your own' if you do that. GitHub take care of the server question (apparently they use a physical-server provider called Carpathia [0]), and because git always needs to be available but is only used rarely, the amount they charge you is probably less than the cost of running a dedicated server for the purpose.
And all that is assuming that a self-hosted GitLab is just as good as GitHub from the developer's point of view. It may or may not really matter, but GitHub is probably the more polished and feature-rich service.
Building a competitor to GitHub is possible, but not trivial, see SourceHut. (We've been talking about GitHub, but of course they're not the only Git provider.)
I can see only a few situations where it makes good sense to run your own Git/GitLab:
1. Your Internet connection is slow and/or unreliable
2. There are extraordinary safety/security concerns associated with your source-code (military avionics code, say) so you want to run Git in an isolated network (no Internet connectivity at all)
3. Related to point 2: You don't want your organisation's data to reside in the USA. (To my knowledge GitHub don't offer any choice about this, but I could be mistaken.)
For the average developer though, I don't see much upside. Having more control isn't a compelling advantage, it's another way of saying you have more obligations.
I find it a bit surprising that it is still a headline, sure it is a huge service used by many. Though as you write its not the first time this year. But apparently still people use a single service as their 'origin'.
So many services rely on github that it has become a single point of failure for online infrastructure as we’ve come to know it. I remember seeing this shared many years ago [1] and not much seems to have changed in the interim. If anything the MS acquisition has only exacerbated matters.
If you let Github become a single point of failure, maybe you are using git wrong? It's named "Distributed version-control" for a reason. I can really recommend this talk by Linus Torvalds about how git is more of a way of working than a piece of software: https://www.youtube.com/watch?v=4XpnKHJAok8
I may use git wrong, but didn't use GitHub wrong. First, Github provides more functions than a git repo. Second, I haven't seen anyone or any company using Git in the way as Linus said in the video. I believe most companies are not either.
I'm sure its 'less bad' now -- but here's an example of an absolutely horrendous security-bug-as-a consequence-of-design-deeply-baked-into-rails ... https://github.com/rails/rails/issues/13420
Interestingly -- the rails developers decided to put in a really horrendous hack to mitigate the common paths through which this design-flaw might lead to unexpected security outcomes ...
In a way, one could argue that the willingness to put in a horrendous hack to 'mitigate' a security flaw provides an example which demonstrates some amount of 'security reasonableness' in rails ...
In reality tho -- I think that this example serves as evidence more for the fact that rails is deeply flawed and very unlikely to be secure in practice -- for reasons of design complexity alone.
There were multiple high profile rails vulnerabilities in the past, such as the infamous yaml vulnerability in 2013 which allows executing arbitrary code in all rails app.
Kallithea[0] has support for both. You might want to test it out and see if it's a good fit for your setup. I enjoyed working with it when when I used it.
99.95% SLA means less than 66 minutes of downtime is acceptable per quarter. I guess they will be offering credits to enterprise customers this time for the first time.
Edit: Correct a mistake but the time calculated is correct.
After Facebook bought Whatsapp it also started having outages at a scale never seen before. Whatsapp used to be a service that was practically never down, after the acquisitions it sometimes had outages of up to a day. Rumours were that it was related to features that FB wanted to add (e.g. video chat). Maybe there's something similar at play here, that Github integrates MSFT services without proper testing?
As few people seem to use Git without those features it would probably be a good idea. Git isn't really decentralized if large parts of the workflow are vendor specific and not cloned.
I'm not sure how active Linus still is in git development, but at least historically, the issue there is that the Linux kernel dev workflow doesn't use any of those features.
That's not really true. Kernel development users PRs, but they're implemented as mailing list threads. They also do code reviews which are implemented as... mailing list threads. It's the most decentralised and vendor-agnostic implementation we can have right now.
So an ideal solution would use mailing list threads as the protocol with a custom UI on top. Same as we use git as the protocol with GitHub eye candy on top. Imagine github as we have now, but all data for issues and PRs are saved as e-mail. Of course, that doesn't benefit GitHub in any way, so we'll never have that.
Browsing through the status page makes me wonder how they manage to have issues like this so frequently. Are there publicly available post-mortems for previous incidents? From a first glance I can't find any on the status page.
Having an electronics engineering background, my personal pet theory is that the convoluted layers upon layers of automagic container management, load balancing and scaling mechanisms act like nested control loops in respect to each other and sudden load increases (e.g. Monday morning load spike) cause the system to essentially produce a step response and it starts overshooting/oscillating. Just a thought tough.
Earlier this year, they were blaming several of the outages on "the database". Not sure if that's still their scapegoat or if they've come up with a new one.
It feels like this is happening a lot more(since quarantine) . It's really weird for how big this company is, how important, and have microsoft's backing. It is a little concerning given how much my work depends on its stability.
It drives me insane that they don’t localize the times on their status page, is there a real reason they are displayed as UTC? If you are looking at a status page chances are anxiety is already high, to throw time zone conversion into the mix too is just obnoxious.
Since turning on resist fingerprinting in Firefox, I am totally used to this problem. I have now memorized and automatically do the timezone math in my head.
Sadly I don't have a real solution for you but at least I have an option that means it's all consistent! :-D
First time I've ever seen it down, I guess there's a first for everything. Now I understand why many people are very paranoid when it comes to taking backups.
Is it worth backing up my GitHub repos somewhere else? What do other people use as an alternative source of truth for their code?
I think this implied that it never was down for him before when it was reported to be down. Which is exactly my experience, too. I've seen a lot of GH down posts here - it never was for me. Now it's the same.
https://downdetector.com/status/github/archive/2020/06, This verifies your data as well, but if you look at the data from Jan '20 to July '19(massive issues in july), there have been more outages. Keep in mind that the stats don't include small outages that last upwards of 2 mins or are region specific. My team and I have been encountering a lot of two min or less kind(usually fixes on a refresh 10 seconds latter). It gets annoying after a while. Its weird given that github hasn't adjusted to quarantine effect.
You can always set up a «git fetch» cronjon on your dev laptop/rpi/vps/toaster. I had one on all repos in my ~/git directory to ensure I have up-to-date repos without manual intervention.
The code is not the problem; you and other collaborators have a copy of the code locally. It's the fact that I can't submit or update PRs anymore and collaborate with my colleagues that's annoying, and no backup can fix that.
If you’re wondering if GitHub has been down more since the Microsoft acquisition, I actually just recently did some analysis on that using their status page data (the answer is yes):
https://nimbleindustries.io/2020/06/04/has-github-been-down-...
It's not surprising. I don't have any data to prove it but it feels like they're shipping improvements and new features at a much faster rate.
Or it could be some of the older employees decided to cash out after the acquisition, and they've lost some institutional knowledge about how to keep things running smoothly?
The addition of interns and new members typically cause more outages, as they make a lot of changes and discover how things work or not.
Retaining older employees has little effect to prevent that, because they typically don't/can't review much of what's happening from a new team, sometimes a whole department, who joined in on the project and has its own agenda.
I was just thinking that Github stayed fairly consistent and Bitbucket didn't really challenge it on features, and then Gitlab came in, shipped loads of popular features and lots of CI functionality, and Github has been delivering a lot of new features a lot faster ever since really, so I think Microsoft acquisition is quite possibly not a root cause of downtime changes.
It's also worth noting that downtime and issues caused during updates in self-hosted Github are significant. I have experienced them, and I guess the main difference is actually that you can schedule the risk, rather than accept it arbitrarily which has positives but comes at significant cost too.
And not for the better, imo the new UI sucks, mostly my eyes have to travel much more than before for the same amount of info.
that's called changing your habits. In 6 months when your eyes remember where are the pertinent information, you will say the same about the old UI.
More information is displayed in a more concise way since the update.
Where are they hosted? Could they be moving infrastructure?
Some things are in Azure, but IIRC not everything (not the git part of github anyway).
Ironically, it seems your site is down? Hug of death?
Site is back. Thank you, Cloudflare.
If you’re wondering if this^ website is hugged to death right now, I checked it on downforeveryoneorjustme.com at 11:08 UTC/04:08 PDT/07:08 EDT(the answer is also yes)
Yep, sorry about that. Here's a cache link:
https://web.archive.org/web/20200629103627/https://nimbleind...
Again? This is like the fourth time GitHub went down. Last time this happened was almost a week ago. [0]
Seriously, just consider self-hosting [1] rather than 'centralizing everything' [2]
[0] https://news.ycombinator.com/item?id=23604944
[1] https://news.ycombinator.com/item?id=23572532
[2] https://news.ycombinator.com/item?id=22867803
I set up a self hosted Gitlab where on-prem was a requirement and it's surprisingly easy. Obviously another service that needs to be maintained and updated but as code repositories and connected services are a crucial part of infrastructure it makes sense to have control over that.
Not the biggest fan of Gitlabs UI but I got used to it quickly.
> Not the biggest fan of Gitlabs UI but I got used to it quickly.
It gets more attractive now that GitHub starts changing theirs. Might as well adapt to GitLab now.
You can self host github or bitbucket as well, but it does get expensive.
There's always https://gitea.io, the self-hosted Git server project that powers https://codeberg.org/
Or live with it and list GitHub downtimes as one of your business risks.
Absolutely, but if downtimes go up, then it starts making sense distrusting that provider and thinking about alternatives.
How much downtime could it get? They might redesign the homepage a few times and break it accidentally, then it's done. They're not gonna redesign it again and again every week forever.
Are you sure about that?
You under-appreciate how enterprise software development tends to go.. there's always new "features" to be had, KPIs to measure and no time to tackle tech debt...
> "Or live with it..."
Try working within a deadline, no engineer worth their salt would ever take this advice.
self hosting is definitely a viable solution.
An engineer can surely plan enough leeway into their deadlines that few hours of github outage doesn't scupper their project
Yes, the question is probably will management allow that?
Hosting your own does not guarantee you 100% uptime. It just means someone less expert than the GitHub folks will be responsible for bringing things back online. It also costs you time and effort, which is a problem if you're on a deadline.
Same goes for the question of where to host it physically. It seems unlikely your physical server will have better uptime than a virtual server in the cloud.
It's about control, not anything else.
If all I need is a git repo, with some tools. Why pay someone to mess it up, when I can mess it up for free?
The moment you place it in someone else's work queue you are tied to them... and they might not care about your projects deadlines. Just like github.
> Why pay someone to mess it up, when I can mess it up for free?
Because it isn't free. Your time is a huge cost.
A senior dev who spends even 10 hrs on standing up a git server has blown through years' worth of GitHub costs, and that's assuming you're even actually using the paid service.
Factor in the extreme security requirements of a code server, including needing to update dependencies daily, and you're spending far more time self-hosting with riskier results.
To add to this, and at the risk of restating my earlier points: even if you get it working, it still probably won't be as reliable as a provider like GitHub.
Keeping it secure is no small thing, especially if you want to permit access from arbitrary IPs on the Internet (rather than using a VPN, say). GitHub does this, and presumably they have solutions in place for everything from intrusion-detection to DDoS protection.
GitHub employs people to take care of server failover and data backups. You could spend your own time building your own solutions here, but they're unlikely to be as good as GitHub's. Your solution is guaranteed to be less well tested.
And that's assuming you even have a server room in the first place. You could run your own Git in the cloud, of course, but you're not really 'running your own' if you do that. GitHub take care of the server question (apparently they use a physical-server provider called Carpathia [0]), and because git always needs to be available but is only used rarely, the amount they charge you is probably less than the cost of running a dedicated server for the purpose.
And all that is assuming that a self-hosted GitLab is just as good as GitHub from the developer's point of view. It may or may not really matter, but GitHub is probably the more polished and feature-rich service.
Building a competitor to GitHub is possible, but not trivial, see SourceHut. (We've been talking about GitHub, but of course they're not the only Git provider.)
I can see only a few situations where it makes good sense to run your own Git/GitLab:
1. Your Internet connection is slow and/or unreliable
2. There are extraordinary safety/security concerns associated with your source-code (military avionics code, say) so you want to run Git in an isolated network (no Internet connectivity at all)
3. Related to point 2: You don't want your organisation's data to reside in the USA. (To my knowledge GitHub don't offer any choice about this, but I could be mistaken.)
For the average developer though, I don't see much upside. Having more control isn't a compelling advantage, it's another way of saying you have more obligations.
[0] https://github.com/holman/ama/issues/553
What about the business risk of someone tripping on your git server power cable?
You can plug it back in yourself.
I find it a bit surprising that it is still a headline, sure it is a huge service used by many. Though as you write its not the first time this year. But apparently still people use a single service as their 'origin'.
So many services rely on github that it has become a single point of failure for online infrastructure as we’ve come to know it. I remember seeing this shared many years ago [1] and not much seems to have changed in the interim. If anything the MS acquisition has only exacerbated matters.
[1] https://www.brandonsavage.net/github-your-single-point-of-fa...
If you let Github become a single point of failure, maybe you are using git wrong? It's named "Distributed version-control" for a reason. I can really recommend this talk by Linus Torvalds about how git is more of a way of working than a piece of software: https://www.youtube.com/watch?v=4XpnKHJAok8
The git aspects of github are not the things people build SPOF on.
It’s the code review tooling, the artifact storage & the deployment pipelines.
A distributed version of those would be awesome...
I may use git wrong, but didn't use GitHub wrong. First, Github provides more functions than a git repo. Second, I haven't seen anyone or any company using Git in the way as Linus said in the video. I believe most companies are not either.
I'm using git bug, so I don't care at all about some delays to sync my issues.
I'm not using GH actions, only Cirrus, Travis and Appveyor, which can be triggered manually if the API service is down also.
https://github.com/MichaelMure/git-bug
> Gitlab is written on Ruby on Rails; you should understand the security implications of Ruby on Rails before using Gitlab
Can someone ELI5 the security implications of Ruby on Rails?
I'm sure its 'less bad' now -- but here's an example of an absolutely horrendous security-bug-as-a consequence-of-design-deeply-baked-into-rails ... https://github.com/rails/rails/issues/13420
Interestingly -- the rails developers decided to put in a really horrendous hack to mitigate the common paths through which this design-flaw might lead to unexpected security outcomes ...
In a way, one could argue that the willingness to put in a horrendous hack to 'mitigate' a security flaw provides an example which demonstrates some amount of 'security reasonableness' in rails ...
In reality tho -- I think that this example serves as evidence more for the fact that rails is deeply flawed and very unlikely to be secure in practice -- for reasons of design complexity alone.
Thank you. I agree with you, even though most of the tech world is run on bodges, that does not mean it is the right thing to do.
There were multiple high profile rails vulnerabilities in the past, such as the infamous yaml vulnerability in 2013 which allows executing arbitrary code in all rails app.
Thank you for explaining
Funny. Unless something changed, GitHub is also extensively using RoR.
Has anyone here tried self hosting Phabricator?
Some of our private repos are Mercurial so it would be nice to have both git and hg repos on a single platform.
Years ago, it was too complex for me. I recently switched to Gogs and I'm satisfied with that. But that's Git only.
Thanks. I need something that can handle our hg repos.
Sourcehut (hg.sr.ht) is supports Mercurial and can also be self-hosted. Not sure how complicated it is to do so.
Kallithea[0] has support for both. You might want to test it out and see if it's a good fit for your setup. I enjoyed working with it when when I used it.
[0]https://kallithea-scm.org/
Well, at least Microsoft hasn't had a Sidekick like event happen to GitHub. https://en.m.wikipedia.org/wiki/2009_Sidekick_data_loss
99.95% SLA means less than 66 minutes of downtime is acceptable per quarter. I guess they will be offering credits to enterprise customers this time for the first time.
Edit: Correct a mistake but the time calculated is correct.
99.95% that is.
Source: https://help.github.com/en/github/site-policy/github-enterpr...
Nice. I hope GitHub is down 67 minutes every quarter, just so that Microsoft loses some money.
You must have a fulfilling life with lots of friends
C'mon microsoft GH is down it's becoming a weekly trend here on HN..
inb4 PR strategy
After Facebook bought Whatsapp it also started having outages at a scale never seen before. Whatsapp used to be a service that was practically never down, after the acquisitions it sometimes had outages of up to a day. Rumours were that it was related to features that FB wanted to add (e.g. video chat). Maybe there's something similar at play here, that Github integrates MSFT services without proper testing?
Sounds like a good time to run something like this https://github.com/clockfort/GitHub-Backup
Git itself is not the issue - the problem is Review Comments, Pull requests and issues.
Are you saying that Git should implement those features?
That would be pretty neat.
Having data outside of the repo is pretty strange in itself, that breaks the federated approach of git.
As few people seem to use Git without those features it would probably be a good idea. Git isn't really decentralized if large parts of the workflow are vendor specific and not cloned.
I'm not sure how active Linus still is in git development, but at least historically, the issue there is that the Linux kernel dev workflow doesn't use any of those features.
That's not really true. Kernel development users PRs, but they're implemented as mailing list threads. They also do code reviews which are implemented as... mailing list threads. It's the most decentralised and vendor-agnostic implementation we can have right now.
Yes, but if you're happy to do everything by mailing lists, you don't have to worry about github going down.
So an ideal solution would use mailing list threads as the protocol with a custom UI on top. Same as we use git as the protocol with GitHub eye candy on top. Imagine github as we have now, but all data for issues and PRs are saved as e-mail. Of course, that doesn't benefit GitHub in any way, so we'll never have that.
Fossil [1] has those integrated, along with a built-in web interface (which git kinda has)
[1] https://www.fossil-scm.org/
I wish fossil had become more mainstream. It seemed such a cool tool when I tried it 8 to 9 years ago.
Shameless plug: https://github.com/MichaelMure/git-bug
Yes I realize the irony of pushing a project hosted on github when it's down.
I’m getting errors on pushes too
Browsing through the status page makes me wonder how they manage to have issues like this so frequently. Are there publicly available post-mortems for previous incidents? From a first glance I can't find any on the status page.
Having an electronics engineering background, my personal pet theory is that the convoluted layers upon layers of automagic container management, load balancing and scaling mechanisms act like nested control loops in respect to each other and sudden load increases (e.g. Monday morning load spike) cause the system to essentially produce a step response and it starts overshooting/oscillating. Just a thought tough.
Earlier this year, they were blaming several of the outages on "the database". Not sure if that's still their scapegoat or if they've come up with a new one.
GitHub Pages seems to be up though. My website is still online.
I'm guessing the core functionality of Pages is quite isolated from the rest of the site since it's just serving static sites over a CDN.
In the OP you can see that GH pages are currently the only service that is not broken
It feels like this is happening a lot more(since quarantine) . It's really weird for how big this company is, how important, and have microsoft's backing. It is a little concerning given how much my work depends on its stability.
Its back up for me
Monday blues...
Unicorn!
I get the falling off a cliff.
No, you fall of a cliff in that very specific manner only when ACME fails you.
It's back up for me now
Ag nee ouens !
+1 down in India
Looks like something went wrong!
Can we make this thread a sticky?
I hate Microsoft. I've seen a lot of unicorn lately.
I love downvotes.
The prophecy, it's starting...
We had issues producing the response to your request.
Sorry about that. Please try refreshing and contact us if the problem persists.
Contact Support — GitHub Status — @githubstatus
It drives me insane that they don’t localize the times on their status page, is there a real reason they are displayed as UTC? If you are looking at a status page chances are anxiety is already high, to throw time zone conversion into the mix too is just obnoxious.
Since turning on resist fingerprinting in Firefox, I am totally used to this problem. I have now memorized and automatically do the timezone math in my head.
Sadly I don't have a real solution for you but at least I have an option that means it's all consistent! :-D
I'm the exact opposite.
I hate it when sites use their local time, instead of just using UTC everywhere. I got accustomed to using UTC ~25 years ago.
At worst, it'll take you two seconds to open up a terminal and run
if you can't handle the conversion "internally".
works for me... loaded this file no problem: https://github.com/washingtonpost/data-police-shootings/blob...
First time I've ever seen it down, I guess there's a first for everything. Now I understand why many people are very paranoid when it comes to taking backups.
Is it worth backing up my GitHub repos somewhere else? What do other people use as an alternative source of truth for their code?
Not sure if you were being sarcastic but Github goes down quite often :)
https://hn.algolia.com/?q=github+down
I think this implied that it never was down for him before when it was reported to be down. Which is exactly my experience, too. I've seen a lot of GH down posts here - it never was for me. Now it's the same.
Possibly region/cluster specific? Not loading for me from Denmark.
https://downdetector.com/status/github/archive/2020/06, This verifies your data as well, but if you look at the data from Jan '20 to July '19(massive issues in july), there have been more outages. Keep in mind that the stats don't include small outages that last upwards of 2 mins or are region specific. My team and I have been encountering a lot of two min or less kind(usually fixes on a refresh 10 seconds latter). It gets annoying after a while. Its weird given that github hasn't adjusted to quarantine effect.
GitHub is down all the god damn time.
At this point, I consider it the least reliable hosting service available. But, to be fair, it's also the one with the most traffic.
We moved to github from Bitbucket because the latter was not up to scratch at all. Even with this issue I'm not planning to move back.
Okay, true, Atlassian cloud products are way worse, even when they work. I wouldn't consider them a serious SaaS player in the slightest.
Atlassian does SaaS like Java does terseness.
You can always set up a «git fetch» cronjon on your dev laptop/rpi/vps/toaster. I had one on all repos in my ~/git directory to ensure I have up-to-date repos without manual intervention.
The code is not the problem; you and other collaborators have a copy of the code locally. It's the fact that I can't submit or update PRs anymore and collaborate with my colleagues that's annoying, and no backup can fix that.
I'd consider mirroring to another provider like gitlab.
https://docs.gitlab.com/ee/user/project/repository/repositor...