Enhance Kafka exporter to respect max message size

[yurishkuro]

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

The exporter has a config option MaxMessageBytes but it itself does not respect it and attempts to send a full serialized payload to the Kafka driver which may reject it based on this setting.

Describe the solution you'd like

Most payloads can be safely split into chunks of "safe" size that will be accepted by the Kafka driver. For example, in Jaeger integration tests there is a test that writes a trace with 10k spans, which is 3Mb in size when serialized as JSON. The trace can be trivially split into multiple messages that would fit in the default 1Mb size limit.

Describe alternatives you've considered

No response

Additional context

jaegertracing/jaeger#6437 (comment)

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[RenuBhati]

Hey @yurishkuro, I would like to contribute to this issue. Can you please assign me this?

[RenuBhati]

Hey @yurishkuro, I’m tied up with another project and it’s taking longer than expected. I won't be able to pick this up. Thanks for your understanding.

[LZiHaN]

hi @yurishkuro , is this task still available? I’d like to give it a try, assign please

[chahatsagarmain]

@LZiHaN are you working on this ?

[LZiHaN]

@LZiHaN are you working on this ?

Yes, I'm working on it.

[LZiHaN]

Hi @yurishkuro ,

I’m working on implementing this feature and wanted to confirm if the approach I’m considering for message splitting and reassembling is feasible.
My plan is as follows:

1. Splitting the Message: When a message exceeds MaxMessageBytes, I will split it into multiple chunks. Each chunk will include the following information in the message headers:

• message_id (to identify the original message),
• chunk_index (the position of this chunk in the message),
• total_chunks (the total number of chunks for the message).

2. Reassembling the Message: On the consumer side, I will use the message_id in the header to group the chunks. Each chunk will have its chunk_index and total_chunks in the header to help with ordering. Once all chunks are received, I will concatenate the Value fields from each chunk to reassemble the original message.

Is this approach viable, and would it work seamlessly with the Kafka producer/consumer setup? Or are there any potential issues with storing this information in the headers and reassembling the message on the consumer side? Looking forward to your feedback.

[yurishkuro]

@LZiHaN this is a possible approach but it would be a breaking change since a consumer that does not understand this chunking may not be able to reassemble the message. My idea was that we instead split the spans from the payload into multiple payloads, such that each payload fits in the MaxMessageSize when serialized. It's not quite simple to implement because it's possible the payload has one huge spans, but if we can split it this way then it's a fully backwards compatible solution.

[shivanshuraj1333]

@yurishkuro this seems to introduce some performance tradeoffs:

So, this marshaller, creates a message, which is then exported using sarama client to kafka. Now irrespective of what kafka configuration is, this export would fail for sizes which exceeds the client configuration (exporter).

Now, one solution could be to implement chunking traces received here based on the configured size. (as you suggested)

However, it looks like we need to calculate the size and then do the chunking based on the resulting size.
The tradeoff I see with this approach:

1. We need to calculate the size multiple times
   • Receive the trace
   • Calculate the size
     • If the size is greater than the configuration, chunk and try again with the smaller chunks
2. How to chunk?
   • Any preferred algorithm? (maybe binary search the spans, based on number of spans, and split accordingly)

Another approach could be appending spans in a given trace, while calculating size after each append, once the limit is received, again build the resource packet and start appending to it. For this a new size aware SplitTraces can be created.

Or maybe some other approach, WDYT would be a better implementation to it?

The size check is done here in the sarama client, and this ByteSize function can be used in kafka exporter to calculate the size.

[yurishkuro]

To avoid perf issues the marshaling can be optimistic: try to marshal the whole thing first and only if the result is larger than max message then try to chunk the spans. As for the chunking algorithm, you don't need to be stuck in analysis paralysis, just write something that works correctly. However you implement it will be an improvement over the current state since now the message just gets dropped.

[yurishkuro]

Btw binary search sounds like a reasonable approach - keep dividing spans in half until acceptable size is produced. Other methods would be hard since marshalers do not allow concatenation of serialized parts.