Because the serialization for the RPC and storage have different priorities, it might be beneficial to use a different library for each.
Important factors:
- security, e.g.:
- handling of malicious input (buffers should not be trusted)
- secure RPC, if included (e.g. DoS or memory exhaustion vulnerabilities)
- native and cross-language adoption for easy interop
- ease of use
- reasonable performance
The considered libraries:
- protobuf
- cap'n'proto
- flatbuffers
- serde
The current preference is for protobuf using the prost library.
Important factors:
- consistent binary representation for hashing
- preserve ordering (for DB keys)
- ease of use
- reasonable performance
The considered libraries:
- bincode
- borsh
The most mature and widely adopted option. Usually combined with gRPC framework. The Tendermint Rust ABCI provides protobuf definitions.
Implementations:
- https://github.com/danburkert/prost - Rust native
- https://github.com/stepancheg/rust-protobuf - Rust native
- https://github.com/tafia/quick-protobuf - missing features
A comparison of the two main competing Rust implementations seems to favor Prost. Prost reportedly generates cleaner (more idiomatic) Rust code (https://hacks.mozilla.org/2019/04/crossing-the-rust-ffi-frontier-with-protocol-buffers/#comment-24671). Prost also has better performance (https://github.com/danburkert/prost/issues/398#issuecomment-751600653). It is possible to also add serde derive attributes for e.g. JSON support. JSON can be useful for development, requests inspection and web integration. However, to reduce attack surface, we might want to disallow JSON for write requests on mainnet by default.
gRPC implementations:
- https://github.com/hyperium/tonic - Rust native, using Prost and Tokio
- https://github.com/tikv/grpc-rs - build on C core library
- https://github.com/stepancheg/grpc-rust - not production ready
It avoids serialization altogether, you use the data natively in a representation that is efficient for interchange ("zero-copy"). The other cool feature is its "time-traveling RPC". On the other hand concern for this lib is a much lower adoption rate, especially the Rust port which is not as complete. The format is designed to be safe against malicious input (on the both sides of a communication channel), but according to FAQ the reference impl (C++) has not yet undergone security review.
Implementations:
Similar to protobuf, but zero-copy like Cap'n'proto, hence a lot faster.
Unfortunately, the Rust implementation is lacking buffer verifiers, which is crucial for handling malicious requests gracefully. There is only draft implementation google/flatbuffers#6269. This most likely rules out this option.
Implementations:
Serde is Rust native framework with great ergonomics. It supports many different formats implemented as libraries. It's used in some DBs too. Serde itself gives no security guarantees, handling of malicious input depends heavily on the used format. Serde can be used in combination with many other formats, like protobuf.
https://github.com/servo/bincode
Built on top of serde. Easy to use.
https://github.com/near/borsh-rs
Used in the Near protocol, it guarantees consistent representations and has a specification. It is also faster than bincode and is being implemented in other languages.