Investigate further optimizations of process definition importing

[chillleader]

What should we do?

We are currently polling Operate every N (configurable) seconds to update the process definition registry in the Connector runtime. This causes high data usage between Operate and ElasticSearch as Operate needs to fetch all process definitions for every polling request.

We can optimize this process by storing the pagination index in memory inside the Connector runtime and only poll for the next page, instead of starting from scratch every time.

Besides, we should investigate if it is possible to further optimize how the connector runtime imports and analyzes process definitions. Some ideas:

Can we add a quick check for the BPMN XML to determine if it contains any connectors before parsing the process model?

Why should we do it?

1. We received customer complaints about excessive data usage between Operate and Elastic caused by this polling.

2. From time to time, we are encountering issues in the clusters that have an abnormal amount of process definitions deployed, which causes delayed startup time and OOM errors.

https://jira.camunda.com/browse/SUPPORT-24457

[sbuettner]

As discussed we cant "just" use pagination due to the way pagination works in elastic right? Could you raise this with the REST API folks?

[sbuettner]

Related product hub epic: https://github.com/camunda/product-hub/issues/2572

[sbuettner]

Re-adding this to discuss it in our grooming. cc @mattli2024

[crobbins215]

@sbuettner and @mattli2024, I see this is going to be readded to grooming to discuss, this looks like its causing a major issue for the customer, anything we can do to work around it in another way? Even if that includes us contributing to core? We must have some mitigating options?

Core is all booked up for 8.7 & 8.8

[sbuettner]

@crobbins215 As written in the linked support ticket: https://jira.camunda.com/browse/SUPPORT-24457?focusedCommentId=385314&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-385314 we will have to align with other components to make it happen.

[crobbins215]

@crobbins215 As written in the linked support ticket: https://jira.camunda.com/browse/SUPPORT-24457?focusedCommentId=385314&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-385314 we will have to align with other components to make it happen.

Hey, I've read the latest update. The core team can't pick this up currently, and it is affecting connectors. Can you and @mattli2024 discuss what options we have?

[sbuettner]

@crobbins215 Sure, but I am afraid we cant do much either right now as we depend on the REST API to provide the data.