It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today:
$ curl -i -L https://github.com/kyleconroy/sqlc/releases/download/v1.17.0/sqlc_1.17.0_linux_amd64.tar.gz
HTTP/2 302
server: GitHub.com
date: Fri, 24 Mar 2023 20:51:56 GMT
content-type: text/html; charset=utf-8
location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/193160679/09048595-c7f4-45b5-858a-7f55baa2fd7d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230324T205156Z&X-Amz-Expires=300&X-Amz-Signature=772d0aa8c5c19b0a5ef84d718d2faf0d81f24b224a4ef634d2410787e8f50bad&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=193160679&response-content-disposition=attachment%3B%20filename%3Dsqlc_1.17.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
> What are the odds this happens the same day they rotate their SSH keys?
Definitely a bad for them. When it rains, it pours.
Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?
I know someone who joined a company and found a dead-man's switch in the server.
He could have taken it out, but instead he just resets it every three months, just like the guy before him.
If the company ever gets rid of him and doesn't hire someone equally skilled and thorough, the production server will eat itself right about the time his unemployment benefits run out.
Or the team is on a new project and after ten attempts to get new owners have an outlook rule to delete any mails about the old project.
The only way to do cert renewal at an org level is one well organized team of not creative software types. yeah yeah the team will automate but in the meantime someone has to check all the dates carefully. And usually good public certs can't be fully automated, at least in the deploy bit.
I heard about a new cert once with a longer private key that cauaed all the terminating F5s to fall over due to out of CPU
Could be a good chance. I'd venture to guess they failed to update the known_hosts file for one of their systems that handles certificate management. Strictly me taking a stab at the answer though.
They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ What is going on over there?
npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID
npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID
npm ERR! request to https://pkg-
npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason:
Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com
Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.
Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)".
If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops.
They "fixed" it and promptly had another related outage the very next day.
Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/
The cert for objects.githubusercontent.com has also expired:
What are the odds this happens the same day they rotate their SSH keys?
It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today:
> What are the odds this happens the same day they rotate their SSH keys?
Definitely a bad for them. When it rains, it pours.
They laid off the wrong person.
This is so comical because it's so relevant.
Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?
Failure of forethought with the wrong deadlines in place, soup to nuts.
Or maybe just laziness.
I know someone who joined a company and found a dead-man's switch in the server.
He could have taken it out, but instead he just resets it every three months, just like the guy before him.
If the company ever gets rid of him and doesn't hire someone equally skilled and thorough, the production server will eat itself right about the time his unemployment benefits run out.
I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”
I guess copilot can't write monitoring rules.
> I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”
I'd almost guarantee you're right on the money with that line of thinking…
Or the team is on a new project and after ten attempts to get new owners have an outlook rule to delete any mails about the old project.
The only way to do cert renewal at an org level is one well organized team of not creative software types. yeah yeah the team will automate but in the meantime someone has to check all the dates carefully. And usually good public certs can't be fully automated, at least in the deploy bit.
I heard about a new cert once with a longer private key that cauaed all the terminating F5s to fall over due to out of CPU
cert-manager though amiright?
Like what's going on there?
This seems to now be fixed.
Could be a good chance. I'd venture to guess they failed to update the known_hosts file for one of their systems that handles certificate management. Strictly me taking a stab at the answer though.
Now they are owned by Microsoft, its a celebratory 10 year tradition. Cause a worldwide outage, by letting your certificate expire...
"Windows Azure Service Disruption from Expired Certificate" (2013) - https://azure.microsoft.com/en-us/blog/windows-azure-service...
They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ What is going on over there?
Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz).
Here is the report for this incident: https://www.githubstatus.com/incidents/x7njwb481j9b
EDIT: this specific issue is resolved
Failing for us in GitHub Actions
For SEO purposes:
This also applies to their avatars subdomain, causing them not to load anymore.
Are you sure? avatars.githubusercontent.com works fine for me.
> Are you sure? avatars.githubusercontent.com works fine for me.
That’s because they already resolved the issue → https://www.githubstatus.com/incidents/x7njwb481j9b
This is what I saw an hour ago:
I wonder if this has anything to do with the recent SNAFU from a Senior Security Engineer* there?
https://twitter.com/viibeeng/status/1639374358287118336
(*yeah we can all make mistakes, but it's 2023, if you've not build controls into your workflows by now you don't deserve to be a Senior anything)
I built a free monitoring service some years ago if anyone doesn't want to be the victim of this...
https://ismycertexpired.com/check?domain=objects.githubuserc...
It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/
… well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert.
The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…
Also serving wrong certificates for a lot of content domains.
https://news.ycombinator.com/item?id=35295191
Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.
Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)".
If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops.
They "fixed" it and promptly had another related outage the very next day.
That was one of them I recall.
And today of all days I have a moment to upgrade homebrew stuff.
got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday
Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.
For us dumb dumbs what does this mean?
Seems to be resolved now. My `brew update` works again.
ChatGPT, rotate my certs
It needs a plugin for that.
Did we learn nothing from Son of Anton?
Why? Don't most cloud providers have auto-renewing certs now?
Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/
It's back now!
didn’t they announce a bunch of layoffs recently?
Back up now, it looks like.
Not Before Fri, 18 Mar 2022 00:00:00 GMT
Not After Tue, 21 Mar 2023 23:59:59 GMT
3-day certs.
There's a whole year between those dates.
3 day plus 365 days ;) you're missing a full year
the year is different
Short lived certs are a thing, also