fix performance regression with MongoDB 6 regarding snapshot aggregat…

…ions

* next round: a total of 3 additional indexes need to be in place to have MongoDB 6 aggregation queries perform the same as with MongoDB 5
* documented in "MongoDB tuning" section in documentation
* removed/replaced index which is not needed
* fixed Helm configuration which caused empty config values

Signed-off-by: Thomas Jäckle <[email protected]>

connectivity/service/src/main/resources/connectivity.conf

@@ -6,11 +6,20 @@ ditto {
     database = ${?MONGO_DB_DATABASE}
 
     read-journal {
+      # additional index which speeds up background (e.g. cleanup) aggregation queries
+      # seems to be required for MongoDB version >=6, otherwise a lot of read Disk IOPS are needed
       should-create-additional-snapshot-aggregation-index-pid-id = false
       should-create-additional-snapshot-aggregation-index-pid-id = ${?MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID}
 
-      should-create-additional-snapshot-aggregation-index-pid = false
-      should-create-additional-snapshot-aggregation-index-pid = ${?MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID}
+      # additional index which speeds up background (e.g. cleanup) aggregation queries
+      # seems to be required for MongoDB version >=6, otherwise a lot of read Disk IOPS are needed
+      should-create-additional-snapshot-aggregation-index-pid-sn = false
+      should-create-additional-snapshot-aggregation-index-pid-sn = ${?MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN}
+
+      # additional index which speeds up background (e.g. cleanup) aggregation queries
+      # seems to be required for MongoDB version >=6, otherwise a lot of read Disk IOPS are needed
+      should-create-additional-snapshot-aggregation-index-pid-sn-id = false
+      should-create-additional-snapshot-aggregation-index-pid-sn-id = ${?MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN_ID}
 
       hint-name-filterPidsThatDoesntContainTagInNewestEntry = null
       hint-name-filterPidsThatDoesntContainTagInNewestEntry = ${?MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY}

deployment/helm/ditto/Chart.yaml

@@ -16,7 +16,7 @@ description: |
   A digital twin is a virtual, cloud based, representation of his real world counterpart
   (real world “Things”, e.g. devices like sensors, smart heating, connected cars, smart grids, EV charging stations etc).
 type: application
-version: 3.5.9  # chart version is effectively set by release-job
+version: 3.5.10  # chart version is effectively set by release-job
 appVersion: 3.5.8
 keywords:
   - iot-chart

deployment/helm/ditto/templates/connectivity-deployment.yaml

@@ -225,18 +225,20 @@ spec:
               value: "{{ .Values.connectivity.config.cleanup.deleteFinalDeletedSnapshot }}"
             - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
               value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
-            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
-              value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN
+              value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPidSn }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN_ID
+              value: "{{ .Values.connectivity.config.readJournal.indexes.createSnapshotAggregationIndexPidSnId }}"
             - name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
-              value: "{{ .Values.connectivity.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
+              value: {{ .Values.connectivity.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
-              value: "{{ .Values.connectivity.config.readJournal.hints.listLatestJournalEntries }}"
+              value: {{ .Values.connectivity.config.readJournal.hints.listLatestJournalEntries | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
-              value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
+              value: {{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
-              value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
+              value: {{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
-              value: "{{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
+              value: {{ .Values.connectivity.config.readJournal.hints.listNewestActiveSnapshotsByBatchId | default "null" | quote }}
             - name: CONNECTION_SNAPSHOT_INTERVAL
               value: "{{ .Values.connectivity.config.persistence.snapshots.interval }}"
             - name: CONNECTION_SNAPSHOT_THRESHOLD

deployment/helm/ditto/templates/policies-deployment.yaml

@@ -243,18 +243,20 @@ spec:
               value: "{{ .Values.policies.config.cleanup.deleteFinalDeletedSnapshot }}"
             - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
               value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
-            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
-              value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN
+              value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPidSn }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN_ID
+              value: "{{ .Values.policies.config.readJournal.indexes.createSnapshotAggregationIndexPidSnId }}"
             - name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
-              value: "{{ .Values.policies.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
+              value: {{ .Values.policies.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
-              value: "{{ .Values.policies.config.readJournal.hints.listLatestJournalEntries }}"
+              value: {{ .Values.policies.config.readJournal.hints.listLatestJournalEntries | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
-              value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
+              value: {{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
-              value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
+              value: {{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
-              value: "{{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
+              value: {{ .Values.policies.config.readJournal.hints.listNewestActiveSnapshotsByBatchId | default "null" | quote }}
             - name: POLICIES_PERSISTENCE_PING_RATE_FREQUENCY
               value: "{{ .Values.policies.config.persistence.pingRate.frequency }}"
             - name: POLICIES_PERSISTENCE_PING_RATE_ENTITIES

deployment/helm/ditto/templates/things-deployment.yaml

@@ -244,18 +244,20 @@ spec:
               value: "{{ .Values.things.config.cleanup.deleteFinalDeletedSnapshot }}"
             - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID
               value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPidId }}"
-            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID
-              value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPid }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN
+              value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPidSn }}"
+            - name: MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN_ID
+              value: "{{ .Values.things.config.readJournal.indexes.createSnapshotAggregationIndexPidSnId }}"
             - name: MONGODB_READ_JOURNAL_HINT_NAME_FILTER_PIDS_THAT_DOESNT_CONTAIN_TAG_IN_NEWEST_ENTRY
-              value: "{{ .Values.things.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry }}"
+              value: {{ .Values.things.config.readJournal.hints.filterPidsThatDoesntContainTagInNewestEntry | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_LATEST_JOURNAL_ENTRIES
-              value: "{{ .Values.things.config.readJournal.hints.listLatestJournalEntries }}"
+              value: {{ .Values.things.config.readJournal.hints.listLatestJournalEntries | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID_ID
-              value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId }}"
+              value: {{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPidId | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_PID
-              value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid }}"
+              value: {{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchPid | default "null" | quote }}
             - name: MONGODB_READ_JOURNAL_HINT_NAME_LIST_NEWEST_ACTIVE_SNAPSHOT_BY_BATCH_ID
-              value: "{{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchId }}"
+              value: {{ .Values.things.config.readJournal.hints.listNewestActiveSnapshotsByBatchId | default "null" | quote }}
             - name: THING_SNAPSHOT_INTERVAL
               value: "{{ .Values.things.config.persistence.snapshots.interval }}"
             - name: THING_SNAPSHOT_THRESHOLD

deployment/helm/ditto/values.yaml

@@ -656,8 +656,10 @@ policies:
       indexes:
         # createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
         createSnapshotAggregationIndexPidId: false
-        # createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
-        createSnapshotAggregationIndexPid: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSn: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn"+"_id" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSnId: false
       # hints contains hint names to configure for different aggregation calls done in MongoReadJournal
       hints:
         # filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
@@ -885,8 +887,10 @@ things:
       indexes:
         # createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
         createSnapshotAggregationIndexPidId: false
-        # createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
-        createSnapshotAggregationIndexPid: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSn: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn"+"_id" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSnId: false
       # hints contains hint names to configure for different aggregation calls done in MongoReadJournal
       hints:
         # filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
@@ -1363,8 +1367,10 @@ connectivity:
       indexes:
         # createSnapshotAggregationIndexPidId whether to create the "pid"+"_id" compound index on the snapshot collection
         createSnapshotAggregationIndexPidId: false
-        # createSnapshotAggregationIndexPid whether to create the "pid" compound index on the snapshot collection
-        createSnapshotAggregationIndexPid: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSn: false
+        # createSnapshotAggregationIndexPidSn whether to create the "pid"+"sn"+"_id" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSnId: false
       # hints contains hint names to configure for different aggregation calls done in MongoReadJournal
       hints:
         # filterPidsThatDoesntContainTagInNewestEntry contains the hint name to use in the aggregation query
@@ -1691,7 +1697,7 @@ nginx:
     # repository for the nginx docker image
     repository: docker.io/nginx
     # tag for the nginx docker image
-    tag: 1.25
+    tag: 1.26
     # pullPolicy for the nginx docker image
     pullPolicy: IfNotPresent
   # extraEnv to add arbitrary environment variables to nginx container
@@ -1827,7 +1833,7 @@ swaggerui:
     # repository for the swagger ui docker image
     repository: docker.io/swaggerapi/swagger-ui
     # tag for the swagger ui docker image
-    tag: v5.9.1
+    tag: v5.17.14
     # pullPolicy for the swagger ui docker image
     pullPolicy: IfNotPresent
   # extraEnv to add arbitrary environment variable to swagger ui container

documentation/src/main/resources/pages/ditto/installation-operating.md

@@ -29,6 +29,51 @@ the following environment variables in order to configure the connection to the
 * `PEKKO_PERSISTENCE_MONGO_JOURNAL_WRITE_CONCERN`: Configure Pekko Persistence MongoDB journal write concern
 * `PEKKO_PERSISTENCE_MONGO_SNAPS_WRITE_CONCERN`: Configure Pekko Persistence MongoDB snapshot write concern
 
+#### MongoDB tuning
+
+This section contains known aspects when/how to tune Ditto working with MongoDB.
+
+##### Background aggregation queries
+
+Ditto runs several "background" queries against MongoDB (e.g. in order to clean up data in the background, 
+depending on the current load to the DB) using `aggregate`.  
+This mainly is done on the "snapshot" collections, e.g. `things_snaps`, `policies_snaps`, `connections_snaps`.  
+
+###### MongoDB 5
+
+In MongoDB 5, the default settings of Ditto work just fine - the MongoDB `aggregate` queries run quick enough and do
+not cause much disk read operations.
+
+###### MongoDB 6
+
+In MongoDB 6 however, the `aggregate` queries Ditto does to the snapshot collections drastically slowed down with much
+data in the snapshot stores.  
+If you encounter poor query performance and e.g. a lot of additional "disk read IOPS", you are also affected by this
+issue. In that case, there are options in Ditto to enable creation of additional indexes which speed up the aggregation 
+queries to a comparable level than with MongoDB 5.
+
+The following environment variables can be enabled to create those indexes:
+
+* `MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_ID`: `true`
+* `MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN`: `true`
+* `MONGODB_READ_JOURNAL_SHOULD_CREATE_ADDITIONAL_SNAPSHOT_AGGREGATION_INDEX_PID_SN_ID`: `true`
+
+They are also configurable via Helm values, e.g. for the `things` service:
+```yaml
+things:
+  config:
+    # holds configuration regarding the MongoReadJournal and e.g. the aggregation queries which are performed in it
+    readJournal:
+      # indexes contains configuration about additional indexes to create
+      indexes:
+        # whether to create the "pid"+"_id" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidId: true
+        # whether to create the "pid"+"sn" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSn: true
+        # whether to create the "pid"+"sn"+"_id" compound index on the snapshot collection
+        createSnapshotAggregationIndexPidSnId: true
+```
+
 ### Ditto configuration
 
 Each of Ditto's microservice has many options for configuration, e.g. timeouts, cache sizes, etc.