The conversation

Lessons learnedExperience excerpt · English

I once was the CTO of a company that serviced about 100k requests per day across all our services

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to understand and we did need consistency guarantees for this workflow and this would remove the need for yet another place data got stored and transfered. In practice, this eventually ended up being very awkward because extending the functionality (since we had "built" it) and had to work around internal pg semantics (we should have just moved off much sooner). It also did not scale well. We ended up getting a ton of disk contention on our RDS instance in non-obvious ways, and the vacuum runs on that table was a nightmare. …

Original discussionPostgres LISTEN/NOTIFY actually scales84 source comments · at import
3

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 3

Direct replies available in this import. Original-language excerpts.

ZM

It is so rare that a new moving part beats other factors. But the one thing I will say is that there's tremendous advantages to of all things, your queuing system being independent of other architectural pieces : it's the one component built to natively store and forward, so if you can keep its lifecycle separate then you can harness that to decouple other systems from each other during updates, troubleshooting, patch windows etc.

Full original
HM

Not saying this in a mean way: It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to. As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.

Full original
EU

I feel this is a management problem, and I disagree with the takeaway about adding a piece to the infra stack. "My devs don't want to work on part of our stack" is not a valid reason to add a new network node imo. Just make them work on it.

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.