You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This enhancement proposal addresses two important poits:
Automatic (or Hot) Model Reload: At the moment, updating a model.yaml while the model is loaded is not possible, therefore it would be worth it to investigate a "hot model reload" approach. This would be useful when cortex is operating inside a robot or when a user of jan is changing the prompt on the fly within a conversation. Not changing the parameters of a model and not having to reload it for these to take effect can enhance different workflows while using cortex.
Hot model load: When no engine or model has been loaded into cortex and a request gets sent, it would be useful to use the information from the model requested and its engine and load it on the fly to serve the request. In addition, we could provide an extra parameter so that the developer can say, for example, "keep-alive": true or "sleep-after": "2m".
The text was updated successfully, but these errors were encountered:
ramonpzg
changed the title
enhancement: hot model reload when model.yml gets updated and model is loaded
enhancement: hot model load and reload
Jan 29, 2025
ramonpzg
changed the title
enhancement: hot model load and reload
feature: hot model load and reload
Jan 30, 2025
This enhancement proposal addresses two important poits:
model.yaml
while the model is loaded is not possible, therefore it would be worth it to investigate a "hot model reload" approach. This would be useful when cortex is operating inside a robot or when a user of jan is changing the prompt on the fly within a conversation. Not changing the parameters of a model and not having to reload it for these to take effect can enhance different workflows while using cortex."keep-alive": true
or"sleep-after": "2m"
.The text was updated successfully, but these errors were encountered: