Allow large byte arrays in received messages to be allocated from ArrayPool

[bill-poole]

I have a gRPC service that streams data from clients in a request stream, where each message is a chunk of data up to 1 MB. Currently, the gRPC library allocates a new array for each chunk, creating a lot of GC pressure, especially because the arrays are all allocated on the LOH. It would be great if we could configure the gRPC endpoint in ASP.NET to allocate those arrays from ArrayPool<byte>.Shared, rather than allocating new arrays. My endpoint logic would then be responsible for returning those arrays to the array pool when processing is complete.

[vladislav-prishchepa]

Hello, @bill-poole, it could be quite challenging to achieve that.

First of all, you need to change array allocators, so you need to customize message parsers which are emitted by protoc, and they are not partial members. It can be achieved by writing protoc plug-ins, but I'm afraid it's already out of scope of grpc-dotnet. Another important thing of general-purpose API is that it should be easy/safe to use, I mean, like implementing IDisposable for messages (which is a breaking change actually), message instances could contain another messages with IDisposable members and handling this could became too complicated really quickly.

Some time ago I had, I guess, similar case: a service which acts like a smart proxy which receives a message with large opaque binary payload and sends this payload wrapped with another proto message to another service, in duplex mode. The only difference is than it was protobuf-over-websocket rather than gRPC. With large payloads (~1MB), of course, I faced the same issues with intense LOH allocations. I manage to write a PoC powered by protobuf-net (with code-first approach it's possible to customize allocations, not easy at all actually, but still possible) which could serve such wrapped messages without any LOH allocations. But I realized that it's not over yet: both message recipients must done the same thing if they are not C++ apps (they have out-if-box protoc arena allocation feature that makes it much easier). Finally, it was decided that protobuf doesn't fit well for large binary payload wrapping and it's easier to rewrite all involved services (transport level only, it was really great that only internal services used this were involved). New transport protocol had messages with trivial layout header_length + header + payload (header was protobuf-encoded) which was easy to use in both serialization and deserialization scenarios regardless of platform.

Returning to your scenario, the best solution depends on your protocol (I think key point are if any original message transform occurs and if message contain single or multiple large binary payloads), but I suppose it's out of drpc-dotnet and maybe even protoc scope anyway.

[bill-poole]

Thanks @vladislav-prishchepa for the very detailed response and explanation. There doesn't seem to be a lot of good, easy options for streaming large amounts of data to a service in .NET.

I could do a large HTTP POST and read the request stream directly in ASP.NET Core, but the default Kestrel maximum request body size is 30 MB, and it's not probably a good idea to increase it significantly because it is a global setting.

I could use a multipart/form-data request, but that's really more for uploading files from a browser, rather than for service-to-service communication.

So, that leaves us with streaming chunks of data over a gRPC request stream. I can change the service contract so that each chunk is sent as a repeated bytes, breaking each chunk down into a List<byte[]> of 80 KB byte arrays to stay under the 85 KB limit to prevent LOH allocations. The byte arrays should all be very transient, so most should be collected in Gen 0. But it still creates a lot of GC pressure, causing more frequent collections.

It seems like there should be a simpler, more "out of the box" way of streaming data to a service in .NET without creating huge amounts of GC pressure. @JamesNK, should I open an issue for that somewhere, or do we just file it under "too hard"?