Propose using a different schema to represent Events in a span

[awangc]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

When storing Span Events in elasticsearch, the event name becomes the key, under which different attributes are stored, e.g. if we have events with name "my-event-1", "my-event-2", then in Elasticsearch we'll have Events.my-event-1.time, Events.my-event-2.time, etc. This does not seem to follow the data format for events for a Span from opentelemetry collector, which are modeled as an array of Span_Event, in which a Span_Event will contain fields like time, name and array of attributes.

The issue I see with this approach is that if name is given arbitrary values (e.g. random UUIDs), then we could see an arbitrary increase in the number of keys.

Describe the solution you'd like

Store span events as an array in elasticsearch, in which each element is an object with fields with time, name and array of attribute (with dropped attribute counts as another possible field - like the Span_Event class)

Admittedly this format may require nested objects which may bring its own performance issues, but it resembles more the data layout from opentelemetry pdata.

Describe alternatives you've considered

No response

Additional context

The schema proposed above would follow the same format for spans, e.g., we have Span.Name and Span.Attributes, and we'd have Event.Name and Event.Attributes, and more closely represents the Event as defined in opentelemetry

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

I believe this needs to be addressed in https://github.com/open-telemetry/opentelemetry-proto

[awangc]

@atoulme This issue is related to elasticsearchexporter's chosen way to encode Events, i.e., the elasticsearchexporter is choosing to use Event name as key when storing Span Events in Default mapping mode (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/internal/objmodel/objmodel.go#L202), this can create field mapping explosion in Elasticsearch if Event name grows indiscriminately (e.g. using random UUIDs). I do not think this is pertinent to opentelemetry-proto?

[dmathieu]

Should this issue be in the core repo? It's about an exporter that's hosted in contrib.

[awangc]

@dmathieu oh, you're right, this issue should be in the contrib repository, my mistake. Let me close and open over there. Thanks!

[dmathieu]

For the record, the new issue: open-telemetry/opentelemetry-collector-contrib#37028