Rename autoscaler parameters and add .update_autoscaler method #2885
Conversation
mwaskom
changed the title
| @@ -586,17 +592,14 @@ def function( | |||
| # Or, pass (request, limit) to additionally specify a hard limit in MiB. | |||
| memory: Optional[Union[int, tuple[int, int]]] = None, | |||
| ephemeral_disk: Optional[int] = None, # Specify, in MiB, the ephemeral disk size for the Function. | |||
| min_containers: Optional[int] = None, # Minimum number of containers to keep warm, even when Function is idle. | |||
There was a problem hiding this comment.
In a way it could be nice if these shared a common prefix (instead of suffix) for in-editor discoverability when typing, but containers_min sounds so much worse that I don't know what's best...
There was a problem hiding this comment.
Yeah container_floor / container_ceiling is one way you could spell that but it would be sort of bizarre.
Most users won't be typing though!
| min_containers: Optional[int] = None, # Minimum number of containers to keep warm, even when Function is idle. | ||
| max_containers: Optional[int] = None, # Limit on the number of containers that can be spun up for this Function | ||
| buffer_containers: Optional[int] = None, # Number of additional idle containers to maintain under active load | ||
| scaledown_window: Optional[int] = None, # Max amount of time a container can remain idle before scaling down. |
There was a problem hiding this comment.
Not sure about this name, but the might have already been a discussion on that elsewhere?
"scaledown" feels like something more global imo, something that affects the whole pool of containers, which is only indirectly/partially true in this case.
Related - what happens if you adjust the scaledown window using update_autoscaler on a function with live containers - will the existing running containers have their scaledown_window updated or will they still respect the value from when they started up? (don't remember how this is implemented)
There was a problem hiding this comment.
Yeah we discussed extensively, it's probably my least favorite of the names here, but at this point we've made all the backend changes and I think we should value consistent frontend / backend terminology in the absence of a clear alternative.
There was a problem hiding this comment.
FWIW the primary motivation here was to avoid confusion arising from the fact that we don't always honor the full window as a "timeout"; containers aren't guaranteed to stay up for x seconds after becoming idle, rather, they are guaranteed not to be idle for more than x seconds. It's right that the pool of containers is what's scaling up / scaling down, and individual containers can be affected by that at any point within their scaledown_window.
| scaledown_window: Optional[int] = None, | ||
| buffer_containers: Optional[int] = None, | ||
| ) -> None: | ||
| """Override the current autoscaler behavior for this Cls instance. |
There was a problem hiding this comment.
Should we start calling a "Cls instance" a Service soon? So much easier to pronounce
There was a problem hiding this comment.
Maybe? I still don't think I understand what the plan is for changes to the client object model.
There was a problem hiding this comment.
Following up on this on Slack :)
Describe your changes
Introduces the new autoscaler parameters and the new
.update_autoscalermethods and updates all internal usage. Adds a deprecation warning on usage of the old names (currentlypending=Trueso that we can update docs and integration tests).Changelog
App.functionandApp.clsparameters that configure the behavior of Modal's autoscaler:concurrency_limitis nowmax_containerskeep_warmis nowmin_containerscontainer_idle_timeoutis nowscaledown_windowbuffer_containers(previously available as_experimental_buffer_containers). When your Function is actively handling inputs, the autoscaler will spin up additionalbuffer_containersso that subsequent inputs will not be blocked on cold starts. When the Function is idle, it will still scale down to the value given bymin_containers..keep_warmmethod and replacing it with a new more general method,.update_autoscaler, which can be used to override any of these parameters without redeploying your App.