Multi-node Concurrency Problems

[stevendanna]

The following is a list of the issues we know we need to solve with respect to allowing multiple automate installations to run against a single database:

• nodemanager/compliance job scheduler should only run scheduled jobs once
• authz OPA state needs to be refreshed across all nodes on update - check for caching sync as well
• multiple services: elasticsearch migrations - we can on hold (BE Upgrades, Back up & Restore, Manual migration)
• multiple services: postgresql schema changes - test it out (BE Upgrades, Back up & Restore, Manual migration)
• compliance, ingest, authz: RBAC Project update process: scripts can only run one at a time, currently controlled by in-process state machine.
• pg-sidecar-service: Is database initialization concurrency safe? talks to HA proxy or not? where it is used -- backup, retention?
• deployment-service: backup/restore
• events-service: NATS is storing data on-disk -- on hold?
• How do we route data to gnats if it isn't HTTP -- ??
• secrets-service: secrets-key is stored on disk -- Config (bashing host can solve it), secret keys need to moved to shared storage. Need to analyze what all data is getting stored in file?
• ingest-service: node-state index holds "latest" state but this could be incorrect for retried requests in concurrent setting. -??
• ingest-service: scheduled jobs should only run on one node

Aha! Link: https://chef.aha.io/epics/SH-E-770

[irvingpop]

For Elasticsearch, there's no simple equivalent to Postgres of a "schema change", once an index is created you have very little ability to change its mappings. two approaches I've seen are:

1. dump/restore indices with new mappings
2. Upon "schema" change, start creating new indices with new mappings. App must be aware of multiple versions of the schema and which indices are which version.

For nats, I'm wondering if we should just move this to the backend for safe-keeping.

[stevendanna]

For Elasticsearch, there's no simple equivalent to Postgres of a "schema change", once an index is created you have very little ability to change its mappings. two approaches I've seen are:

1. dump/restore indices with new mappings
2. Upon "schema" change, start creating new indices with new mappings. App must be aware of multiple versions of the schema and which indices are which version.

Right now most of the migrations we've done we have reindexed data into a new index. There a a number of issues wit this that we'll have to sort out around how we handle upgrades. During which we might have two versions of the software running.