Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google Gemini Plugin #267

Merged
merged 6 commits into from
Dec 19, 2023
Merged

Google Gemini Plugin #267

merged 6 commits into from
Dec 19, 2023

Conversation

gogwilt
Copy link
Contributor

@gogwilt gogwilt commented Dec 14, 2023

Adds a plugin for Google Vertex AI (specifically Gemini).

Screenshot 2023-12-14 at 12 53 22 PM

@gogwilt gogwilt requested a review from abrenneke December 14, 2023 21:40
@gogwilt
Copy link
Contributor Author

gogwilt commented Dec 15, 2023

Google API oddities:

  • At least with inlineData, you can only interact with Gemini Pro Vision for a single "turn." If you have a chat conversation of length > 1, and any of those messages have an image, you get a 400 error.
  • The API spec says that you can only use inlineData for a single image. Otherwise, you have to upload to GCS. I'm guessing that this restriction is even tighter, based on the bullet point above.

For reference, here's a graph that causes Gemini Pro Vision to fail.
Screenshot 2023-12-15 at 8 13 38 AM

@gogwilt
Copy link
Contributor Author

gogwilt commented Dec 17, 2023

Remote debugging seems to work. I couldn't get local npm modules to work, so the test was pretty janky. Will test once this is published.

Google auth seems to work. At a minimum, setting the app default credentials json file to a bogus file made it stop working... Probably worth asking people whether they're able to get it working once we publish this.

GCS storage is still not done, but I think it would be best to add this in a subsequent PR. For now, Gemini is fully usable in text-only mode, and vision is usable with some restrictions.

@gogwilt
Copy link
Contributor Author

gogwilt commented Dec 17, 2023

@abrenneke - I'd love to get your review on this! I think it's worth getting a release out for it.

Copy link
Collaborator

@abrenneke abrenneke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks!

@abrenneke abrenneke merged commit 7e03c9c into main Dec 19, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants