Streaming with Google-Vertex Provider Not Working

[jon2718]

Dear Vercel,

I am building a chatbot to interact with the Google Gemini models including both fine tuned models that I've deployed in VertexAI as well as the base Gemini models. I am using the google-vertex provider.

I am using streamText. When responses arrivve they are are output in chunks rather than as streaming (character bv character). Does the google-vertex provider work with streaming? I am trying to see if doStream is being called in the provider but am not sure.

I am not clear whether streaming requires some special settings on the deployed model in VertexAI and have not been able to find any information about this re: Google Docs.

Here is some simple sample code (although I see the same lack of streamining when I combine streamText with the UI):

import { streamText } from 'ai';
import { vertex } from '@ai-sdk/google-vertex';
import { createVertex } from '@ai-sdk/google-vertex';

const euVertex = createVertex({
//project: 'edubot-439916'
project: PROJECT_ID,
location: 'us-central1', // optional
baseURL: 'https://us-central1-aiplatform.googleapis.com/v1/',
googleAuthOptions: {
//apiKey: API_KEY
keyFilename:'edubot-439916-0950981fb9bc.json',
},
});

const model = euVertex('projects/' + PROJECT_ID + '/locations/us-central1/endpoints/' + ENDPOINT)

const result = streamText({
model: model,
prompt: 'Generate a lecture on stationary states in quantum mechanics',
});

//example: use textStream as an async iterable
for await (const textPart of result.textStream) {
console.log(textPart);
}

[lgrammel]

When you say streaming is not working, do you mean all text arrives in one chunk, or that the chunks are too large and it does not visually seem to stream?

If it's the latter, then that is the default way how vertex returns generated responses. That being said, you can rechunk and create a smoother stream with stream transformations: https://sdk.vercel.ai/docs/ai-sdk-core/generating-text#smoothing-streams

[jon2718]

Thanks so much. This smoothStream helps a lot to smooth things in the UI. It's still a little choppy though

When you say to distinguish between all text arriving in one chunk or that chunks are too large and it does not visually seem to stream, I'm seeing that text is arriving in chunks but before I put the smoothStream they display in a choppy manner. The smoothStream helps smooth things out. Is there a way I can get it to be even smoother? Maybe I need to be running on a GPU in googleVertex or on a dedicated endpoint.

Your thoughts are greatly appreciated and thanks so much for getting back to me so quickly...

[lgrammel]

You can use a custom regexp to stream each character; keep in mind though that this can cause a large number of rerenders and performance issues on the client.