[BUG] After restarting the typhonql-server it never gets initialized if the polystore isn't also restarted

[DavyLandman]

Describe the bug

When the typhonql-server restarts, it needs to get the ML & DL model from the polystore-api. Currently it waits for a call form the polystore-api to set this up, but the API doesn't know when the typhonql-server is restarted, so it never resends it.

Expected behavior

When the typhonql-server restarts (like for example after a container update), it should be ready to handle incoming requests.

REST API

typhonql-server             | 06:07:46.708 [qtp1092572064-13] ERROR engineering.swat.typhonql.server.QLRestServer - Failed to handle response
typhonql-server             | java.io.IOException: Backend is not initialized yet
typhonql-server             |   at engineering.swat.typhonql.server.QueryEngine.getBackend(QueryEngine.java:23) ~[classes/:?]
typhonql-server             |   at engineering.swat.typhonql.server.QueryEngine.resetDatabase(QueryEngine.java:47) ~[classes/:?]

Work-around
Follow a docker-compose up -d with a docker-compose restart typhon-polystore-api

Possible solution
Will have to discuss with @OrfenCLMS about which strategy is preferred:

1. Change the REST interface of the QL server that it always receives the ML & DL information on every call.
2. On startup of the server, we send a message to the polystore-api to register a fresh typhonql-server (not sure if we can always know where the polystore-api server is running, especially in relation to k8s, see: How should we do service discovery in a container runtime agnostic way? typhondl#24)
3. Implement a ping/healthcheck feature from the polystore-api, so that it knows when a service has restarted.

[OrfenCLMS]

This will need to have a more in depth discussion, but regarding potential solution 1, this would make all requests a bit bloated without needing the full ML & DL models apart from edge cases (QL server does not restart that often i assume). Additionally, the ML & DL models can potentially be really big (probably not, but still). Still, this could be used as a temporary workaround until/unless we think of a better solution.

[DavyLandman]

I agree, option 1 is a work around, but it does keep it more stateless, and avoids cyclic dependencies between the API & QLServer.

[DavyLandman]

happy to say this problem has been solved (h/t @OrfenCLMS). We decided to always send along the ML & DL model over every call to the rest server. This allowed the whole QLServer to become stateless, which also helps for the plans for k8s (see typhon-project/typhon-polystore-api#14)