Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
tybalex authored Jun 27, 2024
1 parent 010f4d2 commit 7df7b7f
Showing 1 changed file with 6 additions and 5 deletions.
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,23 @@

### tools.cpp quickstart
1. build from source:
Mac user

- Mac user
```
make
```

Nvidia-Cuda user:
- Nvidia-Cuda user:
```
make LLAMA_CUDA=1
```

2. Install helper package:
2. Install a helper package that fixes some rare edgecases:
```
npm install jsonrepair
```

3. Download a compatible gguf model:
3. Download a compatible Rubra's gguf model:
For example:
```
wget https://huggingface.co/sanjay920/Llama-3-8b-function-calling-alpha-v1.gguf/resolve/main/Llama-3-8b-function-calling-alpha-v1.gguf
Expand All @@ -28,7 +29,7 @@ wget https://huggingface.co/sanjay920/Llama-3-8b-function-calling-alpha-v1.gguf/
./llama-server -ngl 35 -m Llama-3-8b-function-calling-alpha-v1.gguf --port 1234 --host 0.0.0.0 -c 16000 --chat-template llama3
```

5. That's it! Make sure you turn `stream` off when making api calls to the server, as streaming feature is not supported yet.
5. That's it! MAKE SURE you turn `stream` OFF when making api calls to the server, as the streaming feature is not supported yet. And we will support streaming too soon.

### Recent API changes

Expand Down

0 comments on commit 7df7b7f

Please sign in to comment.