> A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding
Did you max out the capacity of the best server you can buy?
Such a database can serve millions of customers (the numbers given).
You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer inspect the essential data of your system is the day your company better be listed on the NASDAQ and ready to pay a few hundred engineers 300k salaries.
What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for sequences?
What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions?
On cross-shard reads: how do you do sorting? And cross-shard joins?
I’d love to be proven wrong, but I suspect the 768 servers look like 1 only on the very surface, and you’ll get wildly different characteristics from cross-shard and single-shard queries.
I personally would prefer if they _didn’t_ look like 1 if they can’t behave like 1.
Of course 768 servers NEVER behave as 1. This is physically impossible.
Global services using relational dbs typically severely restrict queries that run against the cluster. So no joins, no intervals, no grouping, etc.
Transactional queries are usually limited to something like "get a single record, preferably from cache". For many typical web services this can go VERY FAR. Only a handful of global services needs more than a few dozen database servers and a caching cluster. In fact, i have seen major businesses running off a pair of very big postgres instances.
Analytical stuff is extracted into dedicated storages optimized for throughput, like Snowflake or Redshift or BigQuery.
Looks like the GIF is fully built out in code. It's really nice to look at, well made, and easy to understand too.
I wonder what program or code they used. I'd love to know.
p.sI thought it was a GIF, but it's an iframe. That was a nice little surprise.
Yeah, I'm looking at it in developer mode. It's using a GSAP timeline approach to update SVG properties. I'm curious how they handle security and caching for something like this. It looks like they're using Tailwind, at least. but this approach is really clean and nice.
It really feels like the best way to learn is by studying other people's code.
I disagree with the opening premise:
> A single database server cannot handle such demand, so we must spread the queries and data out across many servers with database sharding
Did you max out the capacity of the best server you can buy?
Such a database can serve millions of customers (the numbers given).
You always want to scale up the other parts first, request handlers, caching, etc. The day you can no longer inspect the essential data of your system is the day your company better be listed on the NASDAQ and ready to pay a few hundred engineers 300k salaries.
Well, they are selling this thing so they don't want you to buy a big server (with a read replica) as that's much cheaper.
What about sequences? The example shows an auto-incrementing user ID. How’s that possible without contention between all shards? Is the proxy responsible for sequences?
What about foreign keys? Do they all have to live on the same shard? How do you do distributed transactions?
On cross-shard reads: how do you do sorting? And cross-shard joins?
I’d love to be proven wrong, but I suspect the 768 servers look like 1 only on the very surface, and you’ll get wildly different characteristics from cross-shard and single-shard queries.
I personally would prefer if they _didn’t_ look like 1 if they can’t behave like 1.
A 767 servers KV store should be enough for everyone
Of course 768 servers NEVER behave as 1. This is physically impossible.
Global services using relational dbs typically severely restrict queries that run against the cluster. So no joins, no intervals, no grouping, etc.
Transactional queries are usually limited to something like "get a single record, preferably from cache". For many typical web services this can go VERY FAR. Only a handful of global services needs more than a few dozen database servers and a caching cluster. In fact, i have seen major businesses running off a pair of very big postgres instances.
Analytical stuff is extracted into dedicated storages optimized for throughput, like Snowflake or Redshift or BigQuery.
Sibling post has author answering questions in comments: https://news.ycombinator.com/item?id=48925420
Looks like the GIF is fully built out in code. It's really nice to look at, well made, and easy to understand too. I wonder what program or code they used. I'd love to know.
p.sI thought it was a GIF, but it's an iframe. That was a nice little surprise.
Specifically, it's an JS-controlled/animated SVG embedded in an iframe.
Yeah, I'm looking at it in developer mode. It's using a GSAP timeline approach to update SVG properties. I'm curious how they handle security and caching for something like this. It looks like they're using Tailwind, at least. but this approach is really clean and nice.
It really feels like the best way to learn is by studying other people's code.
What kind of security and caching concerns do they need to handle to animate an SVG?
Load balancers, microservices and horizontal scaling?
Previously: https://news.ycombinator.com/item?id=48925420