Redis is our only component that cannot be scaled

[utkonos]

I noticed this bullet point in the High-Throughput Optimizations documentation:

https://cybercentrecanada.github.io/assemblyline4_docs/installation/cluster/optimizations/#:~:text=Redis%20is%20our%20only%20component%20that%20cannot%20be%20scaled

Let me know if this has already been discussed, but what if this component were replaced with Apache Kafka? The queuing etc could then scale as Kafka can run as a cluster.

[cccs-rs]

We are looking at reducing our Redis usage especially as a communication pipe between containers/services. One way is to move some components (ie. Dispatcher, Ingester, Service Server) over to Rust (to maintain performance) and expose an API server for direct interaction instead of using Redis to pass messages back and forth.

We use Redis for other things as well such as storing certain configurations like custom File Identification so we probably won't forgo Redis completely but we are looking at ways to make it less of a bottleneck as possible since clustering Redis has posed some issues in the past.

[utkonos]

Sounds good! Thanks!