Skip to content

Commit

Permalink
Updated readme
Browse files Browse the repository at this point in the history
  • Loading branch information
srgtuszy committed Oct 14, 2024
1 parent 1b21b4e commit f1baad1
Showing 1 changed file with 17 additions and 7 deletions.
24 changes: 17 additions & 7 deletions README.mdown
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,17 @@

Swift bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) thanks to which you'll be able to run compatible LLM models directly on your device.

## Features

- Lightweight and easy to use
- Works on macOS and Linux
- Supports streaming via structured concurrency
- Swift 6 ready!

## TODO

- [ ] Unit tests

- [ ] Support streaming mode with `AsyncStream`

## How to install

Use swift package manager:
Expand All @@ -18,11 +23,16 @@ Use swift package manager:

## How to use

Currently, the library supports non-streaming inference. It's as simple as initializing with a path to model and passing a prompt:
Here's a quick example on how to use it. For more, please refer to an example app in `example/` folder.

```swift
let llama = try LLama(modelPath: "<path to model in gguf>")
let prompt = "Identify yourself, large language model!"
let result = try await llama.infer(prompt: prompt, maxTokens: 1024)
print(result)
// Initialize model
let model = try Model(modelPath: "<model path>")
let llama = try LLama(modelLoader: model)

// Results are delivered through an `AsyncStream`
let prompt = "what is the meaning of life?"
for try await token in await llama.infer(prompt: prompt, maxTokens: 1024) {
print(token, terminator: "")
}
```

0 comments on commit f1baad1

Please sign in to comment.