Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: standardized streaming support #320

Open
1 task done
0xMochan opened this issue Feb 24, 2025 · 0 comments
Open
1 task done

feat: standardized streaming support #320

0xMochan opened this issue Feb 24, 2025 · 0 comments

Comments

@0xMochan
Copy link
Contributor

  • I have looked for existing issues (including closed) about this

Feature Request

We currently have limited streaming support for Anthropic and much of the code and traits are duplicated to support chunked messages even though the processing and majority of the internal code can be shared.

There is likely a natural way to promote a streaming-styled set of traits that can reuse partial components from the normal generation traits that can be implement across the rest of the providers.

Motivation

Streaming is the most natural way to incorporate LLM responses in client applications. While it's application in RAG can be tricky, not providing ubiquitous streaming support throughout our ecosystem leaves us missing an entire category of LLM projects!

Proposal

  • Study streaming support across the ecosystem and refactor the completion modules to reuse aspects of Message etc.
    • This includes being able to produce Message types from streaming outputs so that it can be easily piped into message histories, etc.
    • Also includes generalizing aspects of completion into other methods so that streaming_completion can reuse`.
  • Implement streaming across all of our providers (openai, gemini, etc.)

Alternatives

Continue with how Anthropic has it implemented with silo'd support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants