Function Calling for Local and Fog layers

[philippzagar]

Problem

SpeziLLM currently supports function calling exclusively through the OpenAI APIs. Developers can leverage a declarative DSL to describe function schemas, offering a type-safe and versatile alternative to verbose JSON schema definitions. However, this functionality is not yet available for the local and fog inference layers, limiting the scope of SpeziLLM’s function calling capabilities to cloud-based inference only.
Native support for function calling on the local and fog layers is needed to achieve parity across all inference layers.

Solution

SpeziLLM’s function calling capabilities should be extended to include the local and fog inference layers, as function calling offers significant use cases for applications.

• Fog: The default fog node in SpeziLLM utilizes Ollama for LLM inference. Ollama supports function calling as per the OpenAI API standard, enabling seamless integration with the existing function calling logic in the SpeziLLM OpenAI layer. This compatibility should be leveraged to implement function calling support for the fog layer.
• Local: SpeziLLM uses the mlx-swift library for local inference on devices. Currently, mlx-swift on device does not natively support function calling. However, community projects such as llm-structured-output and mlx-omni-server have extended MLX to enable function calling. Also, the MLX server seems to enable function calling for some models. SpeziLLM could adapt these features to introduce function calling capabilities for local inference on devices.

Additional context

This Huggingface article describes how to implement function calling (tool use) for arbitrary LLMs. This might be useful for the MLX. local layer

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines