The conversation

smittywerbenHNSource author
Lessons learnedExperience excerpt · English

The DB people are probably trying too hard to cater to the queues

… The DB people are probably trying too hard to cater to the queues. Ideally I'd have normalized the data and modeled the relations such transactions don't lock up the whole table. Then I started questioning why I needed a queue at all when databases (sans SQLite which is fast enough as is) are made for pooling access to a database. Kafka supports pipelining to a relational database but this part is where you kind of have to be experienced to not footgun and I'm not at that level. I think using it as a queue in that you're short-circuiting it from the relational database pipeline is non-standard for Kafka. I suspect that's where a lot of the Kafka hate is from. I could understand if the distributed transactions part is hell but at that point it's like why'd you skip the database then? Trying to get that free lunch I assume. I have an alternative. Try inserting everything into a SQLite file. Running into concurrency issues? Use a second SQLite file. Two computers? send it over the network. More issues? Since it's SQL just switch to a real database that will pool the clients. …

Original discussionAsk HN: What's your go-to message queue in 2025?97 source comments · at import
2

What was this responding to?

Parent context · original excerpt

EN
enetherHN

The space is confusing to say the least. Message queues are usually a core part of any distributed architecture, and the options are endless: Kafka, RabbitMQ, NATS, Redis Streams, SQS, ZeroMQ... and then there's the “just use Postgres” camp for simpler use cases. I’m trying to make sense of the tradeoffs between: - async fire-and-forget pub/sub vs. sync RPC-like point to point communication - simple FIFO vs. priority queues and delay queues - intelligent brokers (e.g. RabbitMQ, NATS with filters) vs. minimal brokers (e.g. Kafka’s client-driven model) There's also a fair amount of ideology/emotional attachment - some folks root for underdogs written in their favorite programming language, others reflexively dismiss anything that's not "enterprise-grade". And of course, vendors are always in the mix trying to steer the conversation toward their own solution. If you’ve built a production sy…

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 2

Direct replies available in this import. Original-language excerpts.

J4
j45HN

It’s less about DB people or not. Many of us have used both sides and settled on one area to start. Kafka, et al are amazing. Also almost always overkill in the first x months or years. It’s not too much of a stretch to model your queue first in something like Postgres, which oddly offers things a little beyond a traditional rbdms, and when the model implementation in the domain reveals itself… it can shine a nice light in the direction of a Kafka, etc.

Full original
SM
smittywerbenHN

I already spent time proofreading it, but I forgot that I couldn't edit it after so much time. Unless the merciful admin lets me roll the 12-sided dice to replace the above cringe with the below piece I carved out of soap. --- I always check for maintained libraries for my programming languages for any messaging library. Bindings in many languages are consistent across Kafka, ZeroMQ, and NATS. Kafka is a write-ahead log, not a queue per se. It handles transactions to the disk. The networking is a simple broadcast, not a shared queue. You also can't (canonically, at least) pop/insert/delete rows. It's append-only. It can do basic seeking, like replaying from the start. ZeroMQ is a good choice for learning from its excellent documentation, and programmers interested in C programming. Probably a good lead into Beej's networking guide. ZeroMQ is the odd one as it has no central broker ("Zero…

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.