You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have looked for existing issues (including closed) about this
Feature Request
We currently have limited streaming support for Anthropic and much of the code and traits are duplicated to support chunked messages even though the processing and majority of the internal code can be shared.
There is likely a natural way to promote a streaming-styled set of traits that can reuse partial components from the normal generation traits that can be implement across the rest of the providers.
Motivation
Streaming is the most natural way to incorporate LLM responses in client applications. While it's application in RAG can be tricky, not providing ubiquitous streaming support throughout our ecosystem leaves us missing an entire category of LLM projects!
Proposal
Study streaming support across the ecosystem and refactor the completion modules to reuse aspects of Message etc.
This includes being able to produce Message types from streaming outputs so that it can be easily piped into message histories, etc.
Also includes generalizing aspects of completion into other methods so that streaming_completion can reuse`.
Implement streaming across all of our providers (openai, gemini, etc.)
Feature Request
We currently have limited streaming support for Anthropic and much of the code and traits are duplicated to support chunked messages even though the processing and majority of the internal code can be shared.
There is likely a natural way to promote a streaming-styled set of traits that can reuse partial components from the normal generation traits that can be implement across the rest of the providers.
Motivation
Streaming is the most natural way to incorporate LLM responses in client applications. While it's application in RAG can be tricky, not providing ubiquitous streaming support throughout our ecosystem leaves us missing an entire category of LLM projects!
Proposal
completion
modules to reuse aspects ofMessage
etc.Message
types from streaming outputs so that it can be easily piped into message histories, etc.completion
into other methods so thatstreaming_completion
can reuse`.rig-providers
crate #312 issue is resolved).Alternatives
Continue with how Anthropic has it implemented with silo'd support.
The text was updated successfully, but these errors were encountered: