Standardize AI Manager-Client-Model System

[parsehex]

Taking LLM as an example.

• (py_api/client/llm/) There are several Clients for using a given LLM model (TODO: Allow loading model with any supported client).
• Models come from a certain Source.
  • There are Local models which can be loaded by different Clients depending on the model format. Example: GGUF models can be loaded by LlamaCpp or Transformers.
  • External models are typically from a 3rd-party API, likely with a Client made for that API service. Each service would be a separate Source, with models that are loaded only with that Client. Example: OpenAI has gpt-* models which are only used by the OpenAI LLM Client.
  • TODO: Currently, even External models are "loaded" in the API, or at least can be, to choose which model to use when completing. Need to rethink models and sources, etc.
• (py_api/client/llm_client_manager.py) There is a Manager for handling the usage of the LLM Clients. The Manager generally just provides a common interface to interact with any LLM model that is supported. It might also do some things to help maintain compatibility between the different Clients.
• (py_api/api/llm_api.py) An API (actually 2: REST and WS) is exposed for LLMs. The API routes are able to interact with the Manager to get their results.
• (py_api/models/llm/) Pydantic models for the API and Clients are used (work in progress).

Other AIs [TTS, STT, Img] (attempt to) use this general pattern.

The goal here is to improve upon this structure and consistently apply it to all AI types that we support.

[parsehex]

We're now using a Base class for each Manager to extend from.

I'm thinking it might be good to make a Base class that each AI Type's Base Client extends from, since all clients have some shared functionality.

[parsehex]

The Base Client class would have methods for loading and unloading models and props for supporting those. I think it should also have a list_models method which would only be implemented by clients like OpenAI that use non-local models.

Further, the Base TTS Client class could have a list_voices which is similarly implemented by clients that can do voice cloning.

Diffusers Img Client could have list_loras and other related methods.

[parsehex]

Making more progress.

I've created an ImgModelStatus component, which should probably be made generic (I made minimal changes to get it to work for Img models).

Further, we need to consistently implement WebSockets, on the server and client.

• Client should auto-reconnect
• Sending messages from the server should be done consistently
• TS types need to be refactored (still have old code from using Oobabooga)

[parsehex]

Still need to get a handle on Pydantic errors. Maybe we should mostly/only return dictionaries and only validate in the manager and API routes? That way we can catch the error and log more useful info than a mile-long stack trace that mostly isn't even my code (any way to improve fastapi traces?), e.g. log which model we were using that returned the wrong shape, anything else?

[parsehex]

I think we're close to having "standardized" these AIs. After squaring away the models, I think we could do some extra to make options and names a bit more consistent across AI types and clients, and we need to more completely document/validate the API.