State of Polystore components

[MarieSaphira]

Hi,
to create deployment scrips for Kubernetes I need to know if the polystore components are stateless (except for the databases).

Can you confirm this? @DavyLandman @tvdstorm @pinoval @OrfenCLMS @zolotas4 @kolovos @Danny2097 @malwash

[DavyLandman]

Yup, looks about right.

Just for clarity, currently QL server is not stateless, but after fixing typhon-project/typhonql#68 we can be stateless.

[MarieSaphira]

Analytics and NLAE are not stateless.
How about the API @Stratidakos ?

[Stratidakos]

The API is stateless.

[DavyLandman]

Hi @Stratidakos last time we talked it was statefull, since it contains keeps around the current model in memory? (as in, it's not re-checking the Polystore DB for updated models)

This is primarily a thing for when you run the API behind a load balancer. Then the update model call will only reach one of the instances.

If this is solved, cool 👍

[Stratidakos]

No, thank you for reminding me. The fact, that we keep the models in a MongoDB, makes the API stateful.
We will change a few things to make it stateless.

[theofilis]

@DavyLandman I have a question just to clarify a thing on the @MarieSaphira's diagram?

The PolystoreDB is referring to storage where API stores its configurations or I misunderstood it?

[MarieSaphira]

yes, it's polystore-mongo

[theofilis]

Ok, this is great because. We found that the service is storing the model there. Give us some time to verify it.

If this is true, we can assume that the Polystore API is stateless because it stores its state to the polystore-mongo database.

Otherwise, we will modify it to store the model their.

How is it sound?

[MarieSaphira]

The DL and ML models are uploaded to the polystore-mongo at startup. If the API now only uses the models from the database and does not keep them in memory as it was before (see @DavyLandman post above) you are fine.

[DavyLandman]

Does that mean that everytime someone sends a query to the polystore api, it retrieves the models fresh from the mongo db?

[theofilis]

@DavyLandman good point, I'll check for this.

I supposed to make the API stateless, it should every time request the model from the Database [least effort], of course, this has a negative impact on query performance.

I am thinking, two possible solutions to minimize the impact, which are:

1. To set-up the API to keep the model in memory and periodically request the version from the database.
2. [most difficult] Or to set up a mechanism to update each instance of the Polystore API cached model.

[DavyLandman]

A good thing to consider is that the evolution tool will send model updates and queries for the new model quickly afterwards, so any caching has to take that into consideration.

Is there maybe a kind of mongo event you can subscribe to?

[theofilis]

@DavyLandman Ok I have never used MongoDB events. I have to study them a little.

[DavyLandman]

There appears to be a change stream: https://docs.mongodb.com/manual/changeStreams/ I've never used it, but it might be nice for this purpose?