Some might call this pedantic, but I think physical reality provided us the earliest idempotency keys, which were natural keys in specific systems. Some effects are idempotent because they address physical reality in a way that converges or provides mutual exclusion.
The pigeon-hole principle isn't an instruction guide for birds, it just describes the reality of occupancy. An egg-tooth keeps pecking, escaping at most one egg. A dog may circle, but occupies at most one bed.
The real value of idempotence is being truly end-to-end. It is about the final effect being controlled, rather than about some message hop in the middle. In my opinion, the impulse to isolate it to a messaging layer is often misguided. It presupposes that the other layers don't also have similar failure modes.
It makes for a more robust system when you can elicit the desired idempotence from natural keys and occupancy rules. For example, the way REST can make creation idempotent when the name can be established ahead of the creation request, or deletion is idempotent when names are not reused.
Natural idempotence requires deep understanding of the domain, which developers (and coding agents) do not always have. Also, it's still a limited solution, which doesn't solve the generic case. So it's each much easier to use technical idempotency solutions like:
- stamp every initial request / domain event with the UUID (idempotency key)
- use client-side IDs instead of using server-generated ones with RETURNING
- use PUT instead of POST
- use UPSERTs
- etc.
IMO, you migh as well use CQRS/ES which is a natural evolution of this idea of modeling the physical reality rather than trying to reduce it to CRUD/L
Here, we can draw an analogy between natural hedging and financial hedging (derivatives). Natural hedging requires planning, preparation, and hard work, and even that doesn't cover every scenario, but it's cheap. With derivatives, you pay someone else to take care of it. The other party may not even understand the underlying asset, but they have quant models (a technical solution)
RE: the question about the first mentions of idempotency keys, IMO cryptographic nonces (numbers used only once) are somewhat related. Idempotency keys are used to facilitate message replay, while nonces are used to prevent replay. But the basic idea is similar
The biggest misconception here is that the client or requester is responsible for the keys. They aren’t. They can be but ideally a signed response from the service with a key is ideal. Still, a client header key is better than no key.
In situations where it’s fire and forget, like payments, it’s fine to generate a deterministic key from the requesters side. In situations where transactions or escrow is in play, it’s best to request from the service so the service can orchestrate.
But I’m not a CTO, just an engineer. I can’t spin bad decisions into “features” like they can. Or small novel fixes as profound advances to the industry.
Spoiler: the best free restaurant bread in America is cranberry walnut bread served at Le Diplomate in Washington, D.C., and Parc in Philadelphia. Second place, surprisingly, is Cheddar Bay biscuits at Red Lobster, and third is the dark wheat at Cheesecake Factory (which I usually like but, coincidentally, earlier this week I found lacking).
I don’t have access to the Atlantic, but I’d probably throw Cracker Barrel’s biscuits and cornbread in there, but I suppose those aren’t free.
The Birrell/Nelson call identifier find is great. One thing that still surprises me is how inconsistent idempotency key semantics are across modern APIs, forty years later. Stripe stores the first response and replays it for 24h, even if the first attempt was a 500. Some other payment APIs only dedupe successful requests, so a retry after a failure creates a brand new attempt. Same header name, very different contracts. The scary part is these details usually live in one paragraph of prose in the docs, not in the OpenAPI spec, so client generators and retry middleware have no way to know about them. I have been bitten by a provider quietly changing their key retention window in a changelog nobody read.
Some might call this pedantic, but I think physical reality provided us the earliest idempotency keys, which were natural keys in specific systems. Some effects are idempotent because they address physical reality in a way that converges or provides mutual exclusion.
The pigeon-hole principle isn't an instruction guide for birds, it just describes the reality of occupancy. An egg-tooth keeps pecking, escaping at most one egg. A dog may circle, but occupies at most one bed.
The real value of idempotence is being truly end-to-end. It is about the final effect being controlled, rather than about some message hop in the middle. In my opinion, the impulse to isolate it to a messaging layer is often misguided. It presupposes that the other layers don't also have similar failure modes.
It makes for a more robust system when you can elicit the desired idempotence from natural keys and occupancy rules. For example, the way REST can make creation idempotent when the name can be established ahead of the creation request, or deletion is idempotent when names are not reused.
Natural idempotence requires deep understanding of the domain, which developers (and coding agents) do not always have. Also, it's still a limited solution, which doesn't solve the generic case. So it's each much easier to use technical idempotency solutions like:
- stamp every initial request / domain event with the UUID (idempotency key)
- use client-side IDs instead of using server-generated ones with RETURNING
- use PUT instead of POST
- use UPSERTs
- etc.
IMO, you migh as well use CQRS/ES which is a natural evolution of this idea of modeling the physical reality rather than trying to reduce it to CRUD/L
Here, we can draw an analogy between natural hedging and financial hedging (derivatives). Natural hedging requires planning, preparation, and hard work, and even that doesn't cover every scenario, but it's cheap. With derivatives, you pay someone else to take care of it. The other party may not even understand the underlying asset, but they have quant models (a technical solution)
RE: the question about the first mentions of idempotency keys, IMO cryptographic nonces (numbers used only once) are somewhat related. Idempotency keys are used to facilitate message replay, while nonces are used to prevent replay. But the basic idea is similar
The biggest misconception here is that the client or requester is responsible for the keys. They aren’t. They can be but ideally a signed response from the service with a key is ideal. Still, a client header key is better than no key.
In situations where it’s fire and forget, like payments, it’s fine to generate a deterministic key from the requesters side. In situations where transactions or escrow is in play, it’s best to request from the service so the service can orchestrate.
But I’m not a CTO, just an engineer. I can’t spin bad decisions into “features” like they can. Or small novel fixes as profound advances to the industry.
Spoiler: the best free restaurant bread in America is cranberry walnut bread served at Le Diplomate in Washington, D.C., and Parc in Philadelphia. Second place, surprisingly, is Cheddar Bay biscuits at Red Lobster, and third is the dark wheat at Cheesecake Factory (which I usually like but, coincidentally, earlier this week I found lacking).
I don’t have access to the Atlantic, but I’d probably throw Cracker Barrel’s biscuits and cornbread in there, but I suppose those aren’t free.
Texas Roadhouse yeast rolls
Ringo is adorable, the crafty b@stard!
The Birrell/Nelson call identifier find is great. One thing that still surprises me is how inconsistent idempotency key semantics are across modern APIs, forty years later. Stripe stores the first response and replays it for 24h, even if the first attempt was a 500. Some other payment APIs only dedupe successful requests, so a retry after a failure creates a brand new attempt. Same header name, very different contracts. The scary part is these details usually live in one paragraph of prose in the docs, not in the OpenAPI spec, so client generators and retry middleware have no way to know about them. I have been bitten by a provider quietly changing their key retention window in a changelog nobody read.