Aldipower 8 minutes ago

Apparently the DENIC team was on a party this evening! Party hard, but not too hard. https://bsky.app/profile/denic.de/post/3ml4r2lvcjg2h

  • FinnKuhn 6 minutes ago

    A real party killer if I have ever seen one.

    • SOLAR_FIELDS 4 minutes ago

      At least all of the appropriate people were in a room together when the outage happened

  • walrus01 3 minutes ago

    Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.

krystofbe 1 hour ago

Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE:

RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer.

Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover.

  • qazwsxedchac 24 minutes ago

    So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat.

    Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Sould make for an interesting post mortem.

    • walrus01 13 minutes ago

      It looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed.

  • dlopes7 13 minutes ago

    I love how I work with IT for 20 years and don't understand a single acronym here other than DNSSEC

    • walrus01 12 minutes ago

      To be fair, advanced real world knowledge of public/private key PKIs (x.509 or other), things like root CAs, are a fairly esoteric and very specialized field of study. There's people whose regular day jobs are nothing but doing stuff with PKI infrastructure and their depth of knowledge on many other non-PKI subjects is probably surface level only.

      • hannob 6 minutes ago

        I know quite a bit about PKI and X.509, and I can tell you that much: the overlap with how DNSSEC works is limited.

      • hathawsh 4 minutes ago

        Is that actually true, though? Even though it's not really my job, I find myself debugging certificates and keys at least once a month, and that's after automating as much as possible with certbot and cloud certificates. PKI always seems to demand attention.

      • mschuster91 2 minutes ago

        It's not made easier by the fact that a lot of cryptography is either very old and arcane or it's one hell of a mess of code that doesn't make sense without reading standards.

        I had the misfortune of having to dig deep into constructing ASN.1 payloads by hand [1] because that's the only thing Java speaks, and oh holy hell is this A MESS because OF COURSE there's two ways to encode a bunch of bytes (BIT STRING vs OCTET STRING) and encoding ed25519 keys uses BOTH [2].

        And ed25519 is a mess in itself. The more-or-less standard implementation by orlp [3] is almost completely lacking any comments explaining what is going on where and reading the relevant RFCs alone doesn't help, it's probably only understandable by reading a 500 pages math paper.

        End of rant.

        [1] https://github.com/msmuenchen/meshcore-packets-java/blob/mai...

        [2] https://datatracker.ietf.org/doc/html/rfc8410#appendix-A

        [3] https://github.com/orlp/ed25519/tree/master

tom1337 20 minutes ago

I have never used DNSSEC and never really bothered implementing it, but do I understand it correctly that we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it which now breaks because the central organisation managing this certificate has an outage taking basically all domains with them?

  • Medowar 14 minutes ago

    What you see here is decentralisation working. The issue is with the operator of the de TLD, and as such only that TLD is affected. DNS is not decentralised in such a way, that multiple organisations run the infrastructure of a TLD, those are always run by a single entity.(.com and .net are operated by Verisign)

    So what the issue is, that the operator has, does not change the impact.

siva7 45 minutes ago

Crazy. I can't remember an incident like this ever happened before and it's still not fixed? .de is probably the most important unrestricted domain after .com from an economical perspective. Millions of businesses are "down".

  • lschueller 41 minutes ago

    It's Germany, pessimistic time estimation + 1/3 and you are in a realistic time frame for the issue being resolved.

    • warpspin 38 minutes ago

      It's night. Somebody has to fill a form to approve night work first.

      • greyhound 26 minutes ago

        And send it by post for approval, which will take 5-30 business days.

        • 9dev 23 minutes ago

          Oh come on, that’s not true. You could also fax it. That might come with an additional processing fee though.

        • rasz 9 minutes ago

          Dont be ridiculous, thats what FAX is for.

    • Cockbrand 28 minutes ago

      In addition: it's Germany, pessimistic cost estimation + 2000%, and you are in a realistic budget for the issue being resolved.

  • rwmj 36 minutes ago

    I remember when .com went down, in July 1997.

    https://archive.nytimes.com/www.nytimes.com/library/cyber/we...

    • ctippett 15 minutes ago

      > For instance, the name "www.nytimes.com" corresponds to nine different computers that answer requests for The New York Times on the Web, one of which is 199.181.172.242

        $ dig -x 199.181.172.242 +short
        www2.nytimes.com.
      

      Neat.

pocksuppet 56 minutes ago

I must be early. There's not a single tptacek DNSSEC rant in this thread yet.

  • aberoham 53 minutes ago

    He’s busy with MathAcademy earning XP-SEC

  • 0123456789ABCDE 29 minutes ago

    doesn't this event speak for itself though?

chromehearts 1 hour ago

I was STRESSING tf out because I wasn't able to connect to my services & apps through my domains like at all .. they only work when using my phone data ? .. thank god it's not my fault this time

  • Locke80 1 hour ago

    But we're Germans, and we need someone to blame.

    • AndroTux 55 minutes ago

      I'm blaming chromehearts anyways

    • lschueller 38 minutes ago

      Thank god for the german chain of blame: 1. The system 2. The neighbor 3. China

      • warpspin 28 minutes ago

        You definitely forgot Merkel and Habeck.

        • Cockbrand 27 minutes ago

          Danke Merkel!!1!11!!

kuerbel 1 hour ago

I just spent the better half of an hour to debug unbound and the pihole because I thought it's a me problem...

Good news though, if you add domain-insecure: "de" to your unbound config everything works fine

  • Bender 1 hour ago

    I don't even enable DNSSEC in Unbound. There just isn't enough adoption yet for me to feel like I am missing out on something, yet.

    "Cloudflare Radar data shows 8.11% of domains are signed with DNSSEC, but only 0.47% of queries are validated end-to-end." [1]

    Zones I may care about:

    - Amazon.com: unsigned

    - My banks: unsigned

    - Hacker News: unsigned

    - Email that I do not host: unsigned

    - My power companies billing: unsigned

    - I found some! id.me and irs.gov are signed.

    [1] - https://technologychecker.io/blog/dnssec-adoption

  • V__ 3 minutes ago

    Just before the outage happened I updated multiple client servers. That was a very stressfull hour trying to figure out why nothing works.

sundiver 1 hour ago

Yes, all .de domains down because of DNSSEC failure at Denic https://dnsviz.net/d/de/dnssec/

sunaookami 1 hour ago

https://status.denic.de/ says "Partial Service Disruption" for DNS Nameservice now.

EDIT: it says "Service Disruption" now

  • MASNeo 1 hour ago

    At least they have some humor left.

    Edit: Now even the humor is gone.

    • sunaookami 43 minutes ago

      Can only be topped when the status page is not reachable anymore :D

      • lschueller 31 minutes ago

        Or only accessible through a german dns server

  • niklasrde 26 minutes ago

    It says "Server Not Found" now

__michaelg 1 hour ago

Finally establishing the concept of Feiertag on the internet. Come back tomorrow.

  • 9753268996433 58 minutes ago

    Using this newfangled thingamabob on a silent holiday will result in the police kicking in your door the next morning.

  • throw1234567891 50 minutes ago

    Internetfreie Dienstage, 21st century variant of Autofreie Sonntage.

iknowstuff 1 hour ago

Kurzgesagt predicted this, Germany is OVER

0x80h 15 minutes ago

Am I reading this correctly? All .de domains are down? Looking forward to reading the postmortem.

dwedge 51 minutes ago

On a slightly unrelated note, I was setting nameservers for two .de domains a few weeks ago and thought my provider was being crazily strict because they kept getting rejected. Turns out you can't point to a nameserver until that nameserver has a zone for the domain, and you can't use nameservers from two providers unless those two providers are both in the NS records at both ends

  • whalesalad 48 minutes ago

    Common paint point with DNSSEC. It’s brutal in the domain industry because when you buy a name with DNSSEC enabled it oftentimes can’t be setup to resolve due to these sorts of issues. Typically seller needs to deactivate first.

kangalioo 1 hour ago

So glad I found someone mention this. Amazon.de, SPIEGEL.de is down. Highly prominent sites unreachable. I wonder how long this will last and how big of a thing this ends up being once people talk about it :o Feels big to me

  • moltar 1 hour ago

    Both examples open for me

    • irundebian 1 hour ago

      Some domains work, some not. I assume that working domains are cached.

  • balou23 1 hour ago

    amazon.de, spiegel.de are down for me, too. heise.de works, but that might've been cached somewhere on my side.

    • yk 1 hour ago

      dig manages to dig out ips for heise.de and tagesschau.de but not spiegel.de amazon.de and google.de However, dig @8.8.8.8 has still amazon.de cached, unlike 1.1.1.1 so perhaps Google to the rescue?

      [Edit] After playing around with it, google seems to have at least some pages cached. After setting dns to 8.8.8.8 amazon.de and spiegel.de work again, my blog does not.

  • theanonymousone 55 minutes ago

    idealo.de, ebay.de, and spiegel.de are down, but amazon.de opens for me.

elevation 55 minutes ago

I've considered hard-coding some addresses into firmware as a fallback for a DNS outtage (which is more likely than not just misconfigured local DNS.) Events like this help justify this approach to the unconcerned.

  • whalesalad 46 minutes ago

    The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.

g4cg54g54 23 minutes ago

funfact: enabling DNS sec NOW will fix your domain instantly if dnssec was disabled before

-> no idea if that also "heals" anyone who had dnssec on before.

-> no idea if maybe they need to roll back something and then rebreak the new dnssec i made a minute later lol...

yosamino 1 hour ago

The last time .de I remember .de had a major outage like this was 2010. I would cite some sources but... you know. That was a fun afternoon, though.

I am very happy that it doesn't happen more often.

kaltsturm 1 hour ago

even their own status page is not reachable: https://status.denic.de/

As fallback they should use their X account: https://x.com/denic_de

  • dgellow 38 minutes ago

    Seems to be up now?

    May 5, 2026 23:28 CEST

    May 5, 2026 21:28 UTC

    INVESTIGATING

    Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible. Based on current information, users and operators of .de domains may experience impairments in domain resolution. Further updates will be provided as soon as reliable findings on the cause and recovery are available. DENIC asks all affected parties for their understanding. For further enquiries, DENIC can be contacted via the usual channels.

    • elch 34 minutes ago

      All .de domains are down for me.

    • kaltsturm 33 minutes ago

      with firefox: KO with chrome: OK

victorbjorklund 1 hour ago

I was just wondering what was up with our .de site.

warpspin 1 hour ago

Whole .de TLD seems to go offline right now due to dnssec or missing nic.de nameservers?

  • fweimer 1 hour ago

    This works:

        $ unbound-host -t A www.denic.de
        www.denic.de has address 81.91.170.12
    

    This does not:

        $ unbound-host -D -t A www.denic.de
        www.denic.de has address 81.91.170.12
        validation failure <www.denic.de. A IN>: signature crypto failed from 194.246.96.1 for DS denic.de. while building chain of trust
    

    So it does seem DNSSEC-related.

    EDIT My explanation was wrong, this is not how keytags work. The published keytag data is consistent:

        de. 3600 IN DNSKEY 256 3 8 AwEAAfRLmzuIXVf7x5A0+U7hke0dS+GEJG0EdPhnOthCCLhy0t0WqLyoXJOhnfsTJ8vQX5fd9qOJc9gyr3SWJZkXAhPm3yPSC7FWWHF70WZTKKM9CekmKdqwMwq6ZCjMSUcecCuSF4Sbt1MRszV7rFmfGVklA1l5UzNbqwD+Dr5vfcLn ;{id = 33834 (zsk), size = 1024b}
        de. 3600 IN DNSKEY 257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b}
        de. 3600 IN DNSKEY 256 3 8 AwEAAbTe1PJi8EgIudNGb+KRTxBL2aCu5rXkZ+aIe/TC88pwRdrXYeXODp1ihZWFop5CrbWRBLrk/YUPBE8aBc6oJP+58dSkdMLYkjSkmvdvYx+zXnRLWlF2bapxvZxshATJDfGjGbCiWxKEOoyRx3UhICtHC+cUSddsEvzfacUcBb6n ;{id = 32911 (zsk), size = 1024b}
        de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519030655 20260505013655 26755 de. ke56T5GZt/X6zMBAF+ouyCTnAd7RY7MsnDcfa9jyyOwSouRXhvzim/V13JDTMBAnpAHxWQXoruXrAZ6A6re5N+8Pp2utVkAEKTWs0r4UOLNKoZ2+zMwNplKjNNnY5PJIbHfa5myyziLiIsi//qDIgQEACFk+pZcHXrRdqRoXPCL3UtfaXjk3+duDQdlPnYsJys5UshjVpkALSMChW7J0anzr0sG+f9ytstBneymMwFYOUC3NqbejbLPZsXGPZBQKPAoVJuV5q3znopbcqrDFfjI7bmX3QPYNvOaiT1ElBfi2piJVpDzMaMAmm2jCmvrf5VeTOBccMroh8sBtDPsaEg== ;{id = 26755}
    

    The signature on the SOA record still does not verify:

        de. 86400 IN SOA f.nic.de. dns-operations.denic.de. 1778014672 7200 7200 3600000 7200
        de. 86400 IN RRSIG SOA 8 1 86400 20260519205754 20260505192754 33834 de. aZoiAJ+PaHUDVSHNXfV/R26ZK3GpFB7ek2Z46VnZdmPEDaTww+a7PkiQ98W83xohUunXYSvQCMeGYfUre5UT76eBKThdxW2a6ImX9/x/oEzQ9x/69Y/NSeTckOv9m3HCLBOug01op1koiHOIAVEvonOmXEHHqo1P4sR/fNbcVg4= ;{id = 33834}
  • kaltsturm 1 hour ago

    not all: https://www.heise.de/ works

    • warpspin 59 minutes ago

      Probably just a high TTL.

      • 0123456789ABCDE 34 minutes ago

        can confirm, at least another 54k seconds from where i sit

    • edb_123 27 minutes ago

      Doesn't work here, at least not anymore. Every single .de domain I have tried doesn't resolve.

lxgr 1 hour ago

Wow, I thought I was somehow unaffected but my resolver must just have cached the sites I'd tried.

tarruda 57 minutes ago

Mailbox.org (also from Germany) seems to be experiencing issues too.

binghatch 1 hour ago

Wow… it’s definitely not all .de TLDs, but a lot of prominent ones definitely.

  • phit_ 1 hour ago

    its gonna be all .de domains once caches dry out, anything that still works right now is bound to eventually fail until the underlying issue is resolved

    • fossdd 1 hour ago

      Any .de domain with DNSSEC

      • meineerde 1 hour ago

        Any .de domain is affected, regardless of the domain's dnssec deployment status, as long as you use a resolver which validates dnssec.

      • mrngm 44 minutes ago

        Unfortunately, even domains that did not have DNSSEC enabled earlier today are affected.

        We observed issues on a non-DNSSEC .de domain at 19:45Z and confirmed around 20:12Z it wasn't just us, but also more high profile domain names.

  • eliaskg 50 minutes ago

    Amazon is completely down in Germany. Not only on amazon.de, even in the app.

dark-star 1 hour ago

How come I have zero problems with any .de domain I tried accessing in the last half hour?

  • AndroTux 58 minutes ago

    maybe your upstream doesn't validate DNSSEC?

    • dark-star 51 minutes ago

      maybe? I'm using PiHole and 8.8.8.8/1.1.1.1 as upstream, and both options show "DNSSEC" next to their options in settings, so I assumed DNSSEC was enabled (unless I have to enable this somewhere else as well?)

      • warpspin 45 minutes ago

        That's weird cause 8.8.8.8/1.1.1.1 will already answer with SERVFAIL right now, unless the domain is still in the cache.

jiggawatts 1 hour ago

I work with a few people specialised in IT security, and some of them take their jobs too seriously and will "lock down" everything to the point that it becomes a very real risk that they lock out everyone including themselves.

Fundamentally, security is a solution to an availability problem: The desire of the users is for a system to remain available despite external attack.

Systems that become unavailable to everyone fail this requirement.

A door with its keyhole welded shut is not "secure", it's broken.

  • QuantumNomad_ 1 hour ago

    Security is not just a solution to availability. It is also to keep sensitive data (PII, or business secrets, or passwords, or cryptographic private keys, and so on) away from the hands of bad actors.

    If I’m unable to use Amazon for 24 hours it doesn’t really matter. If a photo copy of my passport is leaked that’s worries and potential troubles for years.

  • senkora 56 minutes ago

    Security = Confidentiality + Integrity + Availability

    or alternatively,

    Security = (exclude unauth'd reads) + (exclude unauth'd writes) + (include auth'd reads and auth'd writes)

    Gotta satisfy all parts in order to have security.

    • jiggawatts 45 minutes ago

      If you squint at it, you can convert all three to just availability.

          Confidentiality = available to us, but nobody else.
      
          Integrity = available to us in a pristine condition.
      

      It's a bit reductive, I'll admit, but it can be a useful exercise in the same way that everything in an economy can be reduce to units of either: "human time", "money" or "energy". Roughly speaking they're interchangeable.

      E.g.: What's the benefit to you if your data is so confidential that you can't read it either? This is a real problem with some health information systems, where I can't access my own health records! Ditto with many government bureaucracies that keep my records safe and secure from me.

siginator 55 minutes ago

how is that possible?

  • aweiher 15 minutes ago

    Solar Flares

    • dnnddidiej 3 minutes ago

      Took more than cloud flares?

pogii123 1 hour ago

For me bmw.de works but www.bmw.de not

  • benny_s 1 hour ago

    bmw.de is down for me too

    • MikeNotThePope 1 hour ago

      Both domains page load for me from Amsterdam. I wonder if there's communication disruption. Undersea cable severed?

      • pogii123 1 hour ago

        $ nslookup bmw.de ~ Server: 8.8.8.8 Address: 8.8.8.8#53

        Non-authoritative answer: Name: bmw.de Address: 160.46.226.165

        $ nslookup www.bmw.de ~ ;; Got SERVFAIL reply from 8.8.8.8, trying next server Server: 8.8.4.4 Address: 8.8.4.4#53

        * server can't find www.bmw.de: SERVFAIL

      • dark-star 1 hour ago

        You mean the big undersea cable between the Netherlands and Germany? ;-)

    • dark-star 1 hour ago

      both work for me from inside Germany