The conversation

samsquireHNSource author
Lessons learnedExperience excerpt · English

Thanks for this article planetscale

Thanks for this article planetscale. I've been enamoured with sharding recently but more for multithreaded performance. I want multimaster postgres. I started trying to write a postgres synchronizer, by sorting every data by row and column and hashing the data of every column and row, then doing a rolling hash of the data. In theory, two databases can synchronize by sending the final hash of their data and then doing a binary search backwards until the hashes match. This way you can work out which parts of the databases differ and need to be transmitted. If the databases are identical, very little data gets transferred. One problem I've not worked out is how to decide which database is the winning database without having to change application queries. If you synchronize two multimaster Postgres databases that have had independent writes to different sections, how do you identify which database is the source of truth for a column/row combination?

Original discussionHow does database sharding work?107 source comments · at import
3

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 3

Direct replies available in this import. Original-language excerpts.

PR
preseingerHN

"a binary search backwards" is not well defined and doesn't guarantee any upper bound on consistency sharding data based on individual rows within tables is tricky, you won't get reliable guarantees for queries in this way "which database is the winning database" is a function of individual rows, a query that reads data from N different row-owners needs to query N different instances, or else accept that it will work against stale data

Full original
AD
AdieuToLogicHN

> One problem I've not worked out is how to decide which database is the winning database without having to change application queries. I believe this is a case of the "Two Generals' Problem"[0], which implies that there is no provably correct solution to achieve this. > If you synchronize two multimaster Postgres databases that have had independent writes to different sections, how do you identify which database is the source of truth for a column/row combination? You can't without a quorum[1] and even that does not guarantee success. 0 - https://en.wikipedia.org/wiki/Two_Generals%27_Problem 1 - https://en.wikipedia.org/wiki/Split-brain_(computing)

Full original
MY
myzieHN

Spock may be of interest to you: https://pgedge.github.io/spock/

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.