SpeziLLM Remote OpenAI integration (#41)

# SpeziLLM Remote OpenAI integration

## ♻️ Current situation & Problem
Currently, the module provides basic OpenAI integration, however, not in
the SpeziLLM ecosystem.


## ⚙️ Release Notes 
- Add the `SpeziLLMOpenAI` target that provides an OpenAI integration on
the basis of the SpeziLLM ecosystem.


## 📚 Documentation
Added in-line docs + DocC articles + README


## ✅ Testing
Wrote basic UI test cases, manual testing


## 📝 Code of Conduct & Contributing Guidelines 

By submitting creating this pull request, you agree to follow our [Code
of
Conduct](https://github.com/StanfordSpezi/.github/blob/main/CODE_OF_CONDUCT.md)
and [Contributing
Guidelines](https://github.com/StanfordSpezi/.github/blob/main/CONTRIBUTING.md):
- [X] I agree to follow the [Code of
Conduct](https://github.com/StanfordSpezi/.github/blob/main/CODE_OF_CONDUCT.md)
and [Contributing
Guidelines](https://github.com/StanfordSpezi/.github/blob/main/CONTRIBUTING.md).

CITATION.cff

@@ -12,9 +12,9 @@ authors:
 - family-names: "Schmiedmayer"
   given-names: "Paul"
   orcid: "https://orcid.org/0000-0002-8607-9148"
-- family-names: "Ravi"
-  given-names: "Vishnu"
-  orcid: "https://orcid.org/0000-0003-0359-1275"
+- family-names: "Zagar"
+  given-names: "Philipp"
+  orcid: "https://orcid.org/0009-0001-5934-2078"
 title: "SpeziLLM"
 doi: 10.5281/zenodo.7538165
 url: "https://github.com/StanfordSpezi/SpeziLLM"

Package.swift

@@ -26,11 +26,11 @@ let package = Package(
     dependencies: [
         .package(url: "https://github.com/MacPaw/OpenAI", .upToNextMinor(from: "0.2.4")),
         .package(url: "https://github.com/StanfordBDHG/llama.cpp", .upToNextMinor(from: "0.1.6")),
-        .package(url: "https://github.com/StanfordSpezi/Spezi", .upToNextMinor(from: "0.8.0")),
+        .package(url: "https://github.com/StanfordSpezi/Spezi", .upToNextMinor(from: "0.8.2")),
         .package(url: "https://github.com/StanfordSpezi/SpeziStorage", .upToNextMinor(from: "0.5.0")),
         .package(url: "https://github.com/StanfordSpezi/SpeziOnboarding", .upToNextMinor(from: "0.7.0")),
         .package(url: "https://github.com/StanfordSpezi/SpeziSpeech", .upToNextMinor(from: "0.1.1")),
-        .package(url: "https://github.com/StanfordSpezi/SpeziChat", .upToNextMinor(from: "0.1.1")),
+        .package(url: "https://github.com/StanfordSpezi/SpeziChat", .upToNextMinor(from: "0.1.2")),
         .package(url: "https://github.com/StanfordSpezi/SpeziViews", .upToNextMinor(from: "0.6.3"))
     ],
     targets: [
@@ -63,10 +63,10 @@ let package = Package(
         .target(
             name: "SpeziLLMOpenAI",
             dependencies: [
+                .target(name: "SpeziLLM"),
                 .product(name: "OpenAI", package: "OpenAI"),
                 .product(name: "Spezi", package: "Spezi"),
                 .product(name: "SpeziChat", package: "SpeziChat"),
-                .product(name: "SpeziLocalStorage", package: "SpeziStorage"),
                 .product(name: "SpeziSecureStorage", package: "SpeziStorage"),
                 .product(name: "SpeziSpeechRecognizer", package: "SpeziSpeech"),
                 .product(name: "SpeziOnboarding", package: "SpeziOnboarding")

README.md

@@ -56,7 +56,7 @@ The section below highlights the setup and basic use of the [SpeziLLMLocal](http
 
 ### Spezi LLM Local
 
-The target enables developers to easily execute medium-size Language Models (LLMs) locally on-device via the [llama.cpp framework](https://github.com/ggerganov/llama.cpp). The module allows you to interact with the locally run LLM via purely Swift-based APIs, no interaction with low-level C or C++ code is necessary.
+The target enables developers to easily execute medium-size Language Models (LLMs) locally on-device via the [llama.cpp framework](https://github.com/ggerganov/llama.cpp). The module allows you to interact with the locally run LLM via purely Swift-based APIs, no interaction with low-level C or C++ code is necessary, building on top of the infrastructure of the [SpeziLLM target](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm).
 
 #### Setup
 
@@ -80,25 +80,29 @@ class TestAppDelegate: SpeziAppDelegate {
 }
 ```
 
-Spezi will then automatically inject the `LLMRunner` in the SwiftUI environment to make it accessible throughout your application. 
-The example below also showcases how to use the `LLMRunner` to execute a SpeziLLM-based [`LLM`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llm).
+#### Usage
 
-```swift
-class ExampleView: View {
-    @Environment(LLMRunner.self) var runner
-    @State var model: LLM = LLMLlama(
-        modelPath: URL(string: "...") // The locally stored Language Model File in the ".gguf" format
-    )
+The code example below showcases the interaction with the `LLMLlama` through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above..
+Based on a `String` prompt, the `LLMGenerationTask/generate(prompt:)` method returns an `AsyncThrowingStream` which yields the inferred characters until the generation has completed.
 
-    var body: some View {
-        EmptyView()
-            .task {
-                // Returns an `AsyncThrowingStream` which yields the produced output of the LLM.
-                let stream = try await runner(with: model).generate(prompt: "Some example prompt")
-
-                // ...
-            }
-    }
+```swift
+struct LocalLLMChatView: View {
+   @Environment(LLMRunner.self) var runner: LLMRunner
+
+   // The locally executed LLM
+   @State var model: LLMLlama = .init(
+        modelPath: ...
+   )
+   @State var responseText: String
+
+   func executePrompt(prompt: String) {
+        // Execute the query on the runner, returning a stream of outputs
+        let stream = try await runner(with: model).generate(prompt: "Hello LLM!")
+
+        for try await token in stream {
+            responseText.append(token)
+       }
+   }
 }
 ```
 
@@ -107,43 +111,51 @@ class ExampleView: View {
 
 ### Spezi LLM Open AI
 
-A module that allows you to interact with GPT-based large language models (LLMs) from OpenAI within your Spezi application.
+A module that allows you to interact with GPT-based Large Language Models (LLMs) from OpenAI within your Spezi application.
+`SpeziLLMOpenAI` provides a pure Swift-based API for interacting with the OpenAI GPT API, building on top of the infrastructure of the [SpeziLLM target](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm).
 
 #### Setup
 
-You can configure the `OpenAIModule` in the `SpeziAppDelegate` as follows.
-In the example, we configure the `OpenAIModule` to use the GPT-4 model with a default API key.
+In order to use `LLMOpenAI`, the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner) needs to be initialized in the Spezi `Configuration`. Only after, the `LLMRunner` can be used to execute the ``LLMOpenAI``.
+See the [SpeziLLM documentation](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) for more details.
 
 ```swift
-import Spezi
-import SpeziLLMOpenAI
-
-class ExampleDelegate: SpeziAppDelegate {
+class LLMOpenAIAppDelegate: SpeziAppDelegate {
     override var configuration: Configuration {
-        Configuration {
-            OpenAIModule(apiToken: "API_KEY", openAIModel: .gpt4)
+         Configuration {
+             LLMRunner {
+                LLMOpenAIRunnerSetupTask()
+            }
         }
     }
 }
 ```
 
-The OpenAIModule injects an `OpenAIModel` in the SwiftUI environment to make it accessible throughout your application. The model is queried via an instance of [`Chat` from the SpeziChat package](https://swiftpackageindex.com/stanfordspezi/spezichat/documentation/spezichat/chat).
+#### Usage
+
+The code example below showcases the interaction with the `LLMOpenAI` through the the [SpeziLLM](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm) [`LLMRunner`](https://swiftpackageindex.com/stanfordspezi/spezillm/documentation/spezillm/llmrunner), which is injected into the SwiftUI `Environment` via the `Configuration` shown above.
+Based on a `String` prompt, the `LLMGenerationTask/generate(prompt:)` method returns an `AsyncThrowingStream` which yields the inferred characters until the generation has completed.
 
 ```swift
-class ExampleView: View {
-    @Environment(OpenAIModel.self) var model
-    let chat: Chat = [
-        .init(role: .user, content: "Example prompt!"),
-    ]
-
-    var body: some View {
-        EmptyView()
-            .task {
-                // Returns an `AsyncThrowingStream` which yields the produced output of the LLM.
-                let stream = try model.queryAPI(withChat: chat)
-
-                // ...
-            }
+struct LLMOpenAIChatView: View {
+    @Environment(LLMRunner.self) var runner: LLMRunner
+
+    @State var model: LLMOpenAI = .init(
+        parameters: .init(
+            modelType: .gpt3_5Turbo,
+            systemPrompt: "You're a helpful assistant that answers questions from users.",
+            overwritingToken: "abc123"
+        )
+    )
+    @State var responseText: String
+
+    func executePrompt(prompt: String) {
+        // Execute the query on the runner, returning a stream of outputs
+        let stream = try await runner(with: model).generate(prompt: "Hello LLM!")
+
+        for try await token in stream {
+            responseText.append(token)
+        }
     }
 }
 ```

Sources/SpeziLLM/Helpers/Chat+Append.swift

@@ -0,0 +1,47 @@
+//
+// This source file is part of the Stanford Spezi open-source project
+//
+// SPDX-FileCopyrightText: 2022 Stanford University and the project authors (see CONTRIBUTORS.md)
+//
+// SPDX-License-Identifier: MIT
+//
+
+import SpeziChat
+
+
+extension Chat {
+    /// Append an `ChatEntity/Role/assistant` output to the `Chat`.
+    /// Automatically overwrites the last `ChatEntity/Role/assistant` message if there is one, otherwise create a new one.
+    ///
+    /// - Parameters:
+    ///     - output: The `ChatEntity/Role/assistant` output `String` (part) that should be appended.
+    @MainActor
+    public mutating func append(assistantOutput output: String) {
+        if self.last?.role == .assistant {
+            self[self.count - 1] = .init(
+                role: self.last?.role ?? .assistant,
+                content: (self.last?.content ?? "") + output
+            )
+        } else {
+            self.append(.init(role: .assistant, content: output))
+        }
+    }
+
+    /// Append an `ChatEntity/Role/user` input to the `Chat`.
+    ///
+    /// - Parameters:
+    ///     - input: The `ChatEntity/Role/user` input that should be appended.
+    @MainActor
+    public mutating func append(userInput input: String) {
+        self.append(.init(role: .user, content: input))
+    }
+
+    /// Append an `ChatEntity/Role/system` prompt to the `Chat` at the first position.
+    ///
+    /// - Parameters:
+    ///     - systemPrompt: The `ChatEntity/Role/system` prompt of the `Chat`, inserted at the very beginning.
+    @MainActor
+    public mutating func append(systemMessage systemPrompt: String) {
+        self.insert(.init(role: .system, content: systemPrompt), at: 0)
+    }
+}

Sources/SpeziLLM/LLM.swift

@@ -7,36 +7,43 @@
 //
 
 import Foundation
+import SpeziChat
 
 
 /// The ``LLM`` protocol provides an abstraction layer for the usage of Large Language Models within the Spezi ecosystem,
 /// regardless of the execution locality (local or remote) or the specific model type.
 /// Developers can use the ``LLM`` protocol to conform their LLM interface implementations to a standard which is consistent throughout the Spezi ecosystem.
 ///
-/// It is recommended that ``LLM`` should be used in conjunction with the [Swift Actor concept](https://developer.apple.com/documentation/swift/actor), meaning one should use the `actor` keyword (not `class`) for the implementation of the model component. The Actor concept provides guarantees regarding concurrent access to shared instances from multiple threads.
+/// The ``LLM`` contains the ``LLM/context`` property which holds the entire history of the model interactions.
+/// This includes the system prompt, user input, but also assistant responses.
+/// Ensure the property always contains all necessary information, as the ``LLM/generate(continuation:)`` function executes the inference based on the ``LLM/context``.
 ///
 /// - Important: An ``LLM`` shouldn't be executed on it's own but always used together with the ``LLMRunner``.
 /// Please refer to the ``LLMRunner`` documentation for a complete code example.
 ///
 /// ### Usage
 ///
 /// An example conformance of the ``LLM`` looks like the code sample below (lots of details were omitted for simplicity).
-/// The key point is the need to implement the ``LLM/setup(runnerConfig:)`` as well as the ``LLM/generate(prompt:continuation:)`` functions, whereas the ``LLM/setup(runnerConfig:)`` has an empty default implementation as not every ``LLMHostingType`` requires the need for a setup closure.
+/// The key point is the need to implement the ``LLM/setup(runnerConfig:)`` as well as the ``LLM/generate(continuation:)`` functions, whereas the ``LLM/setup(runnerConfig:)`` has an empty default implementation as not every ``LLMHostingType`` requires the need for a setup closure.
 ///
 /// ```swift
-/// actor LLMTest: LLM {
-///     var type: LLMHostingType = .local
-///     var state: LLMState = .uninitialized
+/// @Observable
+/// public class LLMTest: LLM {
+///     public let type: LLMHostingType = .local
+///     @MainActor public var state: LLMState = .uninitialized
+///     @MainActor public var context: Chat = []
 ///
-///     func setup(/* */) async {}
-///     func generate(/* */) async {}
+///     public func setup(/* */) async throws {}
+///     public func generate(/* */) async {}
 /// }
 /// ```
-public protocol LLM {
+public protocol LLM: AnyObject {
     /// The type of the ``LLM`` as represented by the ``LLMHostingType``.
-    var type: LLMHostingType { get async }
+    var type: LLMHostingType { get }
     /// The state of the ``LLM`` indicated by the ``LLMState``.
-    @MainActor var state: LLMState { get }
+    @MainActor var state: LLMState { get set }
+    /// The current context state of the ``LLM``, includes the entire prompt history including system prompts, user input, and model responses.
+    @MainActor var context: Chat { get set }
 
 
     /// Performs any setup-related actions for the ``LLM``.
@@ -46,11 +53,25 @@ public protocol LLM {
     ///   - runnerConfig: The runner configuration as a ``LLMRunnerConfiguration``.
     func setup(runnerConfig: LLMRunnerConfiguration) async throws
 
-    /// Performs the actual text generation functionality of the ``LLM`` based on an input prompt `String`.
+    /// Performs the actual text generation functionality of the ``LLM`` based on the ``LLM/context``.
     /// The result of the text generation is streamed via a Swift `AsyncThrowingStream` that is passed as a parameter.
     ///
     /// - Parameters:
-    ///   - prompt: The input prompt `String` used for the text generation.
     ///   - continuation: A Swift `AsyncThrowingStream` enabling the streaming of the text generation.
-    func generate(prompt: String, continuation: AsyncThrowingStream<String, Error>.Continuation) async
+    func generate(continuation: AsyncThrowingStream<String, Error>.Continuation) async
+}
+
+
+extension LLM {
+    /// Finishes the continuation with an error and sets the ``LLM/state`` to the respective error (on the main actor).
+    ///
+    /// - Parameters:
+    ///   - error: The error that occurred.
+    ///   - continuation: The `AsyncThrowingStream` that streams the generated output.
+    public func finishGenerationWithError<E: LLMError>(_ error: E, on continuation: AsyncThrowingStream<String, Error>.Continuation) async {
+        continuation.finish(throwing: error)
+        await MainActor.run {
+            self.state = .error(error: error)
+        }
+    }
 }

Sources/SpeziLLM/LLMError.swift

@@ -9,46 +9,34 @@
 import Foundation
 
 
-/// The ``LLMError`` describes possible errors that occur during the execution of the ``LLM`` via the ``LLMRunner``.
-public enum LLMError: LocalizedError {
-    /// Indicates that the local model file is not found.
-    case modelNotFound
-    /// Indicates that the ``LLM`` is not yet ready, e.g., not initialized.
-    case modelNotReadyYet
-    /// Indicates that during generation an error occurred.
-    case generationError
+/// Defines errors that may occur during setting up the runner environment for ``LLM`` generation jobs.
+public enum LLMRunnerError: LLMError {
+    /// Indicates an error occurred during setup of the LLM generation.
+    case setupError
 
 
     public var errorDescription: String? {
         switch self {
-        case .modelNotFound:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_FOUND_ERROR_DESCRIPTION", bundle: .atURL(from: .module)))
-        case .modelNotReadyYet:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_READY_ERROR_DESCRIPTION", bundle: .atURL(from: .module)))
-        case .generationError:
-            String(localized: LocalizedStringResource("LLM_GENERATION_ERROR_DESCRIPTION", bundle: .atURL(from: .module)))
+        case .setupError:
+            String(localized: LocalizedStringResource("LLM_SETUP_ERROR_DESCRIPTION", bundle: .atURL(from: .module)))
         }
     }
 
     public var recoverySuggestion: String? {
         switch self {
-        case .modelNotFound:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_FOUND_RECOVERY_SUGGESTION", bundle: .atURL(from: .module)))
-        case .modelNotReadyYet:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_READY_RECOVERY_SUGGESTION", bundle: .atURL(from: .module)))
-        case .generationError:
-            String(localized: LocalizedStringResource("LLM_GENERATION_ERROR_RECOVERY_SUGGESTION", bundle: .atURL(from: .module)))
+        case .setupError:
+            String(localized: LocalizedStringResource("LLM_SETUP_ERROR_RECOVERY_SUGGESTION", bundle: .atURL(from: .module)))
         }
     }
 
     public var failureReason: String? {
         switch self {
-        case .modelNotFound:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_FOUND_FAILURE_REASON", bundle: .atURL(from: .module)))
-        case .modelNotReadyYet:
-            String(localized: LocalizedStringResource("LLM_MODEL_NOT_READY_FAILURE_REASON", bundle: .atURL(from: .module)))
-        case .generationError:
-            String(localized: LocalizedStringResource("LLM_GENERATION_ERROR_FAILURE_REASON", bundle: .atURL(from: .module)))
+        case .setupError:
+            String(localized: LocalizedStringResource("LLM_SETUP_ERROR_FAILURE_REASON", bundle: .atURL(from: .module)))
         }
     }
 }
+
+
+/// The ``LLMError`` defines a common error protocol which should be used for defining errors within the SpeziLLM ecosystem.
+public protocol LLMError: LocalizedError, Equatable {}

Sources/SpeziLLM/LLMHostingType.swift

@@ -14,4 +14,6 @@ public enum LLMHostingType: String, CaseIterable {
     case fog
     /// Remote, cloud-based execution of the ``LLM``.
     case cloud
+    /// Mock execution
+    case mock
 }