"useSearchGrounding" with Google Gen AI breaks tool calling

[acron0]

Description

Use a basic tool, such as

{
toolTest: {
        description: 'Test tool - use this if the user asks to test the tools',
        parameters: z.object({}),
        execute: async () => {
            console.log('Test tool executed')
            return { message: 'Test tool executed' }
        },
    },
}

Then use a prompt such as "test the tools" and observe the console log and the model will respond acknowledging the test.

• WORKS when using an OpenAI model such as openai('gpt-4o')
• WORKS when using a Google Gen AI model such as google('gemini-1.5-pro-latest')
• DOES NOT WORK when using a Google Gen AI model AND { useSearchGrounding: true } option

Code example

const result = await ai({
    model: google('gemini-1.5-pro-latest', {
        useSearchGrounding: true,
    }),
    //model: openai('gpt-4o'),
    prompt: "test the tools" 
    tools: {
        toolTest: {
                description: 'Test tool - use this if the user asks to test the tools',
                parameters: z.object({}),
                execute: async () => {
                    console.log('Test tool executed')
                    return { message: 'Test tool executed' }
                },
            },
        }),
    maxToolRoundtrips: 3,
    temperature: 0.85,
    maxSteps: 3,
    })

AI provider

google('gemini-1.5-pro-latest')

Additional context

No response

[kyleledbetter]

Can confirm, if I have this for gemini-2.0-flash-exp:

export const customModel = (apiIdentifier: string) => {
  if (apiIdentifier.startsWith('claude')) {
    return wrapLanguageModel({
      model: anthropicClient(apiIdentifier),
      middleware: customMiddleware,
    });
  } else if (apiIdentifier.startsWith('gemini')) {
    return wrapLanguageModel({
      model: googleClient(apiIdentifier, { useSearchGrounding: true }),
      middleware: customMiddleware,
    });
  } else {
    return wrapLanguageModel({
      model: openaiClient(apiIdentifier),
      middleware: customMiddleware,
    });
  }
};

Google can get current information but can't call tools like getWeather (or custom ones i created)

but if i set useSearchGrounding: false

[shaper]

I believe this is a known Google-side API limitation -- enabling search grounding disallowing the use of other tools. I recall seeing it in API docs and observing/documenting it during implementation, but I am not finding it now when looking around. I'll keep this open, please let me know if you either find docs around this or see cases where folks are able to do function/tool-calling with grounding enabled with Google Generative AI via some other route.

[BrianHung]

https://discuss.ai.google.dev/t/codeexecution-and-function-calling/58663

Think we should have a console.warn if you enable searchGrounding and also pass in tools. Workaround could be use to generateText with searchGrounding as a tool call.

[kyleledbetter]

Ya my workaround was to have two entries for now (not ideal)

  {
    id: 'gemini-2.0-flash-exp',
    label: 'Gemini 2.0 Flash',
    apiIdentifier: 'gemini-2.0-flash-exp',
    description: 'Gemini 2.0 Flash without search grounding',
    provider: 'google',
    superUser: false,
    searchGrounding: false,
  },
  {
    id: 'gemini-2.0-flash-exp-search',
    label: 'Gemini 2.0 Flash with Search',
    apiIdentifier: 'gemini-2.0-flash-exp',
    description: 'Gemini 2.0 Flash with search grounding',
    provider: 'google',
    superUser: false,
    searchGrounding: true,
  },

with

return wrapLanguageModel({
      model: googleClient(apiIdentifier, { useSearchGrounding: searchGrounding }),
      middleware: customMiddleware,
    });

Absolutely not ideal

And I'm actually perplexity for searching the internet, but like you say you can do something like this for a tool:

          useSearchGrounding: {
            description: 'Use search grounding to search the web for relevant information',
            parameters: z.object({
              query: z.string().describe('search query'),
            }),
            execute: async ({ query }) => {
              const { fullStream } = streamText({
                model: customModel(model.apiIdentifier, model.useSearchGrounding),
                system:
                  'Write about the given topic. Markdown is supported. Use headings wherever appropriate.',
                prompt: query,
                experimental_transform: smoothStream({
                  delayInMs: 10, // optional: defaults to 10ms
                  chunking: 'line', 
                }),
              });

              return {
                success: true,
                content: fullStream,
              };
            },
          },