[processor/logdedup] Use a specific field as deduplication key

[mauri870]

Component(s)

processor/logdedup

Is your feature request related to a problem? Please describe.

Currently the processor uses the entire log message as the deduplication key, allowing the user to remove certain fields from being considered during the hashing part via the exclude_fields option.

One scenario that is not covered is that the deduplication key could be already present as a body or attribute in the log message. You could exclude the other body and attributes via exclude_fields, but this negative match is very error prone and does not scale well.

Describe the solution you'd like

I propose we add a new option to use a certain LogRecord field as deduplication key. This option accepts ottl style path variables:

processors:
    logdedup:
        key:  body["my_id_field"]

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/logdedup: @MikeGoldsmith @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Makes sense to me.

[djaglowski]

I suggest we use ottl style references to fields: body["foo"] instead of body.foo

[mauri870]

@djaglowski Do you have any resources on how to evaluate the OTTL string from the field? I'm struggling to follow up on this part. For example, I have a LogRecord and an OTTL string like body["my_field"] and I need to extract the actual value from the LogRecord. I’d appreciate any help!