Structured output with fileData of a large file (>20mb) fails with error

[yoeven]

Description of the bug:

I'm getting an unknown error when I use structured output with fileData of a large file (>20mb).

Error: GoogleGenerativeAIFetchError: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:generateContent: [400 Bad Request] Request contains an invalid argument

What I have tested, works when files are smaller or when I remove structured output but files are larger.

Actual vs expected behavior:

Reproducible snippet:

const uploadResult = await geminiFileManager.uploadFile(`/tmp/${localFileName}`, {
      mimeType: _mimeType,
      displayName: localFileName,
 });

let file = await geminiFileManager.getFile(uploadResult.file.name);
while (file.state === FileState.PROCESSING) {
  // Sleep for 10 seconds
  await new Promise((resolve) => setTimeout(resolve, 10_000));
  // Fetch the file from the API again
  file = await geminiFileManager.getFile(uploadResult.file.name);
}

if (file.state === FileState.FAILED) {
  throw new Error("Audio processing failed.");
}

const zodSchema = z
  .array(z.number().describe("The speaker label number starts from 0"))
  .min(chunks?.length)
  .max(chunks?.length)
  .describe("Array of speaker labels mapped to each chunk");

const jsonSchema = zodToJsonSchema(zodSchema);
delete jsonSchema.$schema;

const model = gemini.getGenerativeModel({
    model: "gemini-1.5-flash",
    safetySettings: safetySettings,
    generationConfig: {
      responseMimeType: "application/json",
      responseSchema: jsonSchema as any,
      temperature: 0,
    },
});

const diarizationResp = await model.generateContent([
 {
fileData:{
          fileUri: file.uri,
          mimeType: file.mimeType,
        }
},
  `You are a speech diarization labeler system. Label each transcribed audio chunk with a speaker label number. The transcribed audio chunks are in the following JSON array:\n${JSON.stringify(chunks, null, 2)}\nSPEAKER LABELS:`,
]);

What responseSchema looks like:

{"type":"array","items":{"type":"number","description":"The speaker label number starts from 0"},"minItems":66,"maxItems":66,"description":"Array of speaker labels mapped to each chunk"

Reference filed use to test: https://uuvhpoxkzjnrvvajhnyb.supabase.co/storage/v1/object/sign/default/58%20MIN%20Creepy%20Shocking%20Videos%20You%20Must%20See%20Before%20They%20Disappear%20(128kbit_AAC)333.mp3?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cmwiOiJkZWZhdWx0LzU4IE1JTiBDcmVlcHkgU2hvY2tpbmcgVmlkZW9zIFlvdSBNdXN0IFNlZSBCZWZvcmUgVGhleSBEaXNhcHBlYXIgKDEyOGtiaXRfQUFDKTMzMy5tcDMiLCJpYXQiOjE3Mzc4NjE2ODIsImV4cCI6MTc0MDQ1MzY4Mn0.mbYVrBvon4EAU8XNvf3zjPlDiyNQfqW7eFCXDmtEWsU&t=2025-01-26T03%3A21%3A22.281Z

Any other information you'd like to share?

No response