The conversation

Business & incomeExperience excerpt · English

If you're designing a proto schema around these problems, you can seriously compromise readability …

… If you're designing a proto schema around these problems, you can seriously compromise readability and maintainability just to work around poor implementation decisions in Go protobuf itself. I'm guilty of that kind of optimization, but when my team saw the benchmarks numbers they agreed it was worth it. This is not the kind of decision you want to have to make in a technical project, but I emphasize that it can still be the right choice in many circumstances. The prost crate for Rust is not official but already gives you more control over your schema. You can technically use your own Rust code instead of the generated code, though I don't see anyone actually do this and it doesn't seem to be encouraged. In any case, my biggest issue with prost is that it makes it difficult (or perhaps currently impossible) to share sub-messages with Arc [2], which on the other hand is trivial with Go just using *. While prost avoids allocations in more cases than Go protobuf, my experience has been that it avoids copies less, and some of those copies require allocations anyway. …

Original discussionDemystifying the protobuf wire format71 source comments · at import
2

What was this responding to?

Parent context · original excerpt

CM

I find it interesting that the folks running away screaming from protobuf are using it in conjunction with gRPC. Is the problem really with the wire format or is it a problem with all of the stuff above? I've been using protobuf for a (non-web) hobbyist project for some time now and find it fairly straightforward to use, especially when working across multiple implementation languages. For me, it seems to be a nice middle-ground between the ease of JSON and the efficiency of a hand-rolled serialization format.

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 2

Direct replies available in this import. Original-language excerpts.

NE

On overhead, being modeled after Go also unfortunately hurts otherwise excellent .NET gRPC tooling. A lot of types could have been there based on top of standard library types rather than protobuf's - RepeatedField<T> is just List<T> except forces buffer reallocation. Could have been T[] too and just require the users to procure it themselves. Or a pooled type underneath. For short-lived RepeatedFields this is non-issue - they die in Gen0 heap and GC just shrugs them off, but the cost is definitely there and is felt with all these new vector DB libraries passing 1536-long buffers of F32s.

Full original
LO

The sizes are computed by traversing the object tree before writing it. If you wanted a super fast implementation you could reserve space for a, say, three byte VarInt and then write it as three bytes no matter what. A VarInt that is four bytes can still validly represent the number 5, even though that would normally take one byte. The thing that pisses me off about protobuf is that the wire format doesn’t distinguish between different types of binary data: if it said “this is an object binary” then we could decompose it, even if we didn’t have the protocol definition. As it is, could be a string or an array or an object.

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.