Chroma db

[neph1]

Hi.
I'm working on adding chroma db to koboldcpp. I have some basic functionality up and running here: https://github.com/neph1/koboldcpp/tree/chromadb
while I try to understand how to use it better.
I don't think it's ready for a PR, yet, and I'm not sure it's within the scope of the project (keeping requirements to a minimum)?

• Currently it will load all .txt files in a subfolder called "dbdata" into a collection, which it will query from during inference.
• It will either use the segment after the last stop_sequence, or the whole query if none can be found.
• It uses 512 tokens from the results, weighed against max tokens. (I will make this a percentage)
• It puts the result at the top of the prompt.
• No persistence, yet.

Ideas are welcome!

[LostRuins]

To be honest, I am very skeptical about the quality of chromadb results. I have yet to see evidence of it fetching contextual information in a coherent manner, it's more like copy pasting random parts of the history at the top. Now that real long context is possible, I'd say that would provide far better results. What do you think?

[neph1]

I have just started, but my take is that it does help in some cases. Like story writing. It can fetch characters and bits of information on the go.
What I would really like is a matching threshold, so that if the result is below it, it would rather not fetch it. Now it just takes the most likely bits regardless of how well they match. So, sometimes it really is pasting random parts at the top.

The query is important. Since (right now) it only picks the latest prompt, you need to be specific, and possibly repeat yourself for it to stay on topic.

My hope is that it will be possible to provide long descriptions of things, places and characters and not have to worry about the context building up. No matter how long it becomes, it will never be enough.

[neph1]

To give a concrete example of how it can be used for world building, I created this text and placed it for chromadb to find:

Heaven's View Inn. Perched on the east side of the Assamyrian Gorge, the inn has a breathtaking view of the valley below, with the Mortan river crashing through it. Being situated on a somewhat strategic location between the states of Assamyra and Goldoth, it usually hosts a mix of military and shady borderland characters. One thing is certain about the inn, one will never have an uneventful time, there.

I then asked the ai to tell me about it:

You
Hi, can you describe Heaven's View Inn for me?

KoboldAI
Certainly, Heaven's View Inn is a 5 star luxury resort in bali

You
Hi, can you describe Heaven's View Inn, on the Assamyrian Gorge for me?

KoboldAI
Certainly, Heaven's View Inn is a remote and exotic inn, perched on the east side of the Assamyrian Gorge in a rugged and mountainous region. The inn has a breathtaking view of the gorge and the Mortan River that flows through it. It is a place of natural beauty and serenity, but also a place of danger and adventure.

KoboldAI
The Assamyrian Gorge is a wild and untamed place, and the inn is nestled into the rocks, providing a sanctuary for travelers and explorers who dare to venture into this unforgiving terrain. The inn is accessible only by a treacherous path that winds its way through the mountains, and only the bravest or most determined travelers daring to make the journey.

So the use case here is to be able to provide a rich world, without having to keep everything in the context all the time.

[Alihkhawaher]

Thank you for your efforts. I think it would be great to have on the fly db that stores the context.

Sorry, I do not how this works but if it is me, I would find a way to make the AI model summarize and compress past context to the db instead of memorizing everything. Is that possible? Like taking notes of the important parts...

[LostRuins]

What you should compare instead is the verbatim data that chromadb is injecting to the context - not the response of the AI.

[neph1]

Thank you for your efforts. I think it would be great to have on the fly db that stores the co

This doesn't store or compress the context. It reads a set of predefined texts on startup, containing information it can use to craft a response.

[neph1]

What you should compare instead is the verbatim data that chromadb is injecting to the context - not the response of the AI.

Compare with what? In this instance the injected data is exactly what I wrote in the text. I've seen it quote pretty unrelated data, based on greeting phrases, etc, when using chat logs, if that's what you mean.

[neph1]

I've refactored this somewhat, and made a generic 'extension' interface and a class that implements it. I don't know if this has any resemblance to other guis' 'extensions'.
The main benefit compared to previously is that it has no import requirements (apart from 'extension.py'). Extensions are dynamically loaded based on the command line and are called upon during inference.
To prove the functionality beyond chroma, I created a small class that lets the AI check the system time. (Getting it to use it is a different matter.)

This is mostly for my own amusement as I explore LLMs, but I'd be happy to make a PR if it is of interest for the main project.

[LostRuins]

I think something like chromadb would work best as a downstream tool that calls the api rather than an integration into the main server. Since it does not need to work with the model itself but rather just manipulating the context via the rest api. In fact I believe stuff like Tavern already has support for it, as do some intermediate proxies.

[lilblam]

I found chroma not very reliable in my use case. I wanted to add a description of something and be able to ask questions about it. It failed to do it right more often than not. But if I paste the entire description in the prompt and include a question, it works really well. I tend to think that as contexts grow, stuff like this would become increasingly unnecessary too. There are so many new papers that expand context in various ways, that a few months from now I wouldn't be surprised to see absurdly large contexts for local models.

[neph1]

Yes, it seems to be a bit hit and miss. Even worse, it can also 'pollute' the context with wrong information, making the model mix up characters, etc, if using it for story-telling.