Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: hot model load and reload #1892

Open
ramonpzg opened this issue Jan 28, 2025 · 0 comments
Open

feature: hot model load and reload #1892

ramonpzg opened this issue Jan 28, 2025 · 0 comments
Assignees
Labels
P2: enhancement low impact on functionality
Milestone

Comments

@ramonpzg
Copy link
Contributor

ramonpzg commented Jan 28, 2025

This enhancement proposal addresses two important poits:

  1. Automatic (or Hot) Model Reload: At the moment, updating a model.yaml while the model is loaded is not possible, therefore it would be worth it to investigate a "hot model reload" approach. This would be useful when cortex is operating inside a robot or when a user of jan is changing the prompt on the fly within a conversation. Not changing the parameters of a model and not having to reload it for these to take effect can enhance different workflows while using cortex.
  2. Hot model load: When no engine or model has been loaded into cortex and a request gets sent, it would be useful to use the information from the model requested and its engine and load it on the fly to serve the request. In addition, we could provide an extra parameter so that the developer can say, for example, "keep-alive": true or "sleep-after": "2m".
@ramonpzg ramonpzg added the P2: enhancement low impact on functionality label Jan 28, 2025
@ramonpzg ramonpzg added this to the v1.0.9 milestone Jan 28, 2025
@ramonpzg ramonpzg self-assigned this Jan 28, 2025
@ramonpzg ramonpzg moved this to Investigating in Menlo Jan 28, 2025
@ramonpzg ramonpzg added this to Menlo Jan 28, 2025
@ramonpzg ramonpzg changed the title enhancement: hot model reload when model.yml gets updated and model is loaded enhancement: hot model load and reload Jan 29, 2025
@ramonpzg ramonpzg changed the title enhancement: hot model load and reload feature: hot model load and reload Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2: enhancement low impact on functionality
Projects
Status: Investigating
Development

No branches or pull requests

1 participant