Clear and Restore In-Memory Footprint for Local LLM Platform

[philippzagar]

Problem

SpeziLLM provides the LocalPlatform for on-device LLM execution. However, local LLM inference is highly resource-intensive, particularly in terms of main memory consumption. On iOS, this often results in the operating system conservatively terminating the resource-heavy application when it is moved to the background. Currently, SpeziLLM offers no mechanism to mitigate this issue, forcing users to restart the app entirely after it is terminated.

Solution

Introduce two simple functions (likely as part of LLMRunner to ensure cross-platform applicability, but primarily affecting the local platform) to give developers more control over the memory footprint of the app using SpeziLLM.
This function would:

1. Offloading resources: Pause LLM execution and remove in-flight state from main memory, e.g. when the app moves to the background.

   • This could be done by either:
     • Moving the execution state to disk, or
     • Completely clearing the state from memory.

2. Loading resources again: Provide a complementary function to resume LLM execution, e.g. when the app is resumed and moves into the foreground.

   • This function would:
     • Restore the state from disk back into memory, or
     • Reinitialize the platform state as necessary (refer to MLX documentation for guidance).

This approach would enable resource-efficient handling of LLM execution during app backgrounding and prevent unnecessary terminations and restarts.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines

[philippzagar]

Issue was raised in https://github.com/orgs/StanfordSpezi/discussions/69, I formalized it as a feature request in the SpeziLLM repo.

[philippzagar]

Tagging @LeonNissen as he currently leads the developments of the local LLM execution via SpeziLLM! 🚀

[bryan1anderson]

I definitely feel the most crucial part of this is a simple panic button. Restoring would be nice.. but if we cannot clean up resources at all.. very problematic

[bryan1anderson]

Seems like

• Cancel tasks
• Clear MLX cache
• Empty session arrays

Would be a good start?

[LeonNissen]

@bryan1anderson great idea! 🚀

I think we could start by removing the reference of the ModelContainer in https://github.com/StanfordSpezi/SpeziLLM/blob/main/Sources/SpeziLLMLocal/LLMLocalSchema.swift

I would try that one first.

[philippzagar]

@bryan1anderson great idea! 🚀

I think we could start by removing the reference of the ModelContainer in https://github.com/StanfordSpezi/SpeziLLM/blob/main/Sources/SpeziLLMLocal/LLMLocalSchema.swift

I would try that one first.

@LeonNissen I guess you're rather referring to the LLMLocalSession (representing the mutable LLM state and its resources in-flight), correct?
We should check how MLX handles these cleanup scenarios, there should definitely be some code within MLX responsible for freeing up resources. Maybe we can then build on top of that existing functionality, also in order to make the cleanup a bit more flexible (instead of just removing the ModelContainer reference)