Chat with Video #23
Merged
Chat with Video #23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
evanderiel
approved these changes
Dec 14, 2023
evanderiel
reviewed
Dec 14, 2023
evanderiel
reviewed
Dec 14, 2023
evanderiel
reviewed
Dec 14, 2023
evanderiel
reviewed
Dec 14, 2023
evanderiel
reviewed
Dec 14, 2023
evanderiel
approved these changes
Dec 14, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
This Pull Request delivers a comprehensive upgrade for the chat with video use case, incorporating new features like streaming video indexing, chat support in vLLM deployment, and chat with video endpoint.
Key Changes:
Full Pipeline for Chat with Video Usecase: A complete pipeline has been implemented to support chat with video use case.
Streaming Video Indexing: A new endpoint has been added for indexing videos, capable of extracting and streaming transcripts and captions. This feature ensures real-time processing and presentation of video content.
File Storage for Transcripts and Captions: Initially, generated transcripts, captions, and combined timelines are stored as files. Plans are underway to migrate these to a database for better management and retrieval.
Chat Endpoints in vLLM Deployment: Integration of chat endpoints into the vLLM deployment, supporting dialog (a list of messages) as input.
Customizable Chat Templates: Added support for chat templates with the flexibility to provide custom templates when the LLM's tokenizer lacks a chat template.
LLM-based Chat for Video Content: Introducing an endpoint that leverages LLM to facilitate chat based on timelines generated from video transcripts and captions.
Example Notebook for Video Chat Usecase: A notebook has been included to demonstrate the usage and capabilities of the new chat with video feature.
Testing for Chat Templates and Methods: Tests have been added for the newly implemented chat templates and chat functionalities in the vLLM deployment.