Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How i can applay a json schema #3

Open
MAUROFAILO opened this issue Dec 10, 2024 · 5 comments
Open

How i can applay a json schema #3

MAUROFAILO opened this issue Dec 10, 2024 · 5 comments
Labels
documentation Improvements or additions to documentation

Comments

@MAUROFAILO
Copy link

I need to process a PDF file containing invoice data.
I want the request to return a json file according to some json schema. The json schema is saved in a string variable. I don't understand how do I apply the json schema to the request? If anyone can help me I'd be grateful. Thank you.

@MaxiDonkey
Copy link
Owner

Hi,

Does your issue match what's explained in this section of the online help?

https://ai.google.dev/gemini-api/docs/structured-output?lang=rest

@MAUROFAILO
Copy link
Author

Thank you. But I'm not an expert as you.

In my case I run this code:

The code is this:

Chat:=Gemini.Chat.Create('models/gemini-1.5-flash-latest',  
                          Procedure (Params: TChatParams)
                          Begin
                            Params.Contents([TPayload.Add('Extract all the main data of this invoice in pdf format according to the 
                                                                                 attached scheme ', [FileUri,MySchemaUri])]);
                            End);

( Where "MySchemaUri" is a string like this:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "fattura": {
      "type": "object",
      "properties": 
  {

  "type": "object",
  "properties": {
    "Cedente": {
      "type": "object",
      "properties": 
      {
        "cedente_denominazione":             {"type": "string"},
        "cedente_nome":                      {"type": ["string", "null"]},
        "cedente_cognome":                   {"type": ["string", "null"]},
        "cedente_indirizzo":                 {"type": ["string", "null"]},
        "cedente_citta":                     {"type": ["string", "null"]},
        "cedente_stato":                     {"type": ["string", "null"]},
        "cedente_telefono":                  {"type": ["string", "null"]},
etc .......

How i can i do this with a code like this?

    Chat:=Gemini.Chat.Create('models/gemini-1.5-flash-latest',  
                              Procedure (Params: TChatParams)
                              Begin
                                Params.Contents([TPayload.Add('Extract all the main data of this invoice', [FileUri])]);
                                Params.GenerationConfig(??????????????????????????????????)
                                begin
                                  ????????????????????????????????   MySchemaUri
                                end
                              End);

Could you tell me what to insert instead of ??????

I would be very grateful and thank you.

@MaxiDonkey
Copy link
Owner

MaxiDonkey commented Dec 13, 2024

Hello Maurofailo,

You have two options to solve your problem: either by using an instance of TSchema or, more simply, by providing a TJSONObject object.

  1. Using TSchema

In the sources, the Gemini.Schema unit allows you to create instances of TSchema. In fact, I have provided an example of its usage in the Gemini.Functions.Example unit.

function TWeatherReportFunction.GetInputSchema: string;
begin
//  Result :=
//    '{'+
//    '"type": "object",'+
//    '"properties": {'+
//         '"location": {'+
//             '"type": "string",'+
//             '"description": "The city and state, e.g. San Francisco, CA"'+
//           '},'+
//         '"unit": {'+
//             '"type": "string",'+
//             '"enum": ["celsius", "fahrenheit"]'+
//           '}'+
//     '},'+
//     '"required": ["location"]'+
//    '}';

  {--- If we use the TSchemaParams class defined in the Gemini.Schema unit }
  var Schema := TSchemaParams.New(
    procedure (var Params: TSchemaParams)
    begin
      Params.&Type(stOBJECT);
      Params.Properties('properties',
        procedure (var Params: TSchemaParams)
        begin
          Params.Properties('location',
            procedure (var Params: TSchemaParams)
            begin
              Params.&Type(stSTRING);
              Params.Description('The city and state, e.g. San Francisco, CA');
            end);
          Params.Properties('unit',
            procedure (var Params: TSchemaParams)
            begin
              Params.&Type(stSTRING);
              Params.Enum(['celsius', 'fahrenheit']);
            end);
        end);
      Params.Required(['location', 'unit']);
    end);
  Result := Schema.ToJsonString(True);
end;

Since you did not provide the complete schema, I cannot write the exact mapping you need to produce to address this issue. However, the provided conversion example should allow you to do this easily.

In the end, your code should look like this:

//uses Gemini.Schema;

  var Schema := TSchemaParams.New(
    procedure (var Params: TSchemaParams)
    begin
      Params.&Type(stOBJECT);
      Params.Properties('properties',
        procedure (var Params: TSchemaParams)
        begin
          Params.Properties('fattura',
            procedure (var Params: TSchemaParams)
            begin
              Params.&Type(stobject);
      ...
    end);

  Chat:=Gemini.Chat.Create('models/gemini-1.5-flash-latest',  
      Procedure (Params: TChatParams)
      Begin
         Params.Contents([TPayload.Add('Extract all the main data of this invoice', [FileUri])]);
         Params.GenerationConfig(
           procedure (var Params: TGenerationConfig)
           begin
             Params.ResponseSchema(Schema);
           end);
      End);
  1. Using TJSONObject

A second solution consists of keeping your schema script as a string, then converting it into a TJSONObject as follows:

  TJSONObject.ParseJSONValue(StringSchema);

And thus,

Chat:=Gemini.Chat.Create('models/gemini-1.5-flash-latest',  
      Procedure (Params: TChatParams)
      Begin
         Params.Contents([TPayload.Add('Extract all the main data of this invoice', [FileUri])]);
         Params.GenerationConfig(
           procedure (var Params: TGenerationConfig)
           begin
             //Params.ResponseMimeType('application/json'); Perhaps this is necessary if the desired output is exclusively in JSON.
             Params.ResponseSchema(TJSONObject.ParseJSONValue(StringSchema) as TJSONObject);
           end);
      End);

This second method appears simpler and does not require invoking Gemini.Schema.

@MaxiDonkey MaxiDonkey pinned this issue Dec 13, 2024
@MaxiDonkey MaxiDonkey unpinned this issue Dec 13, 2024
@MaxiDonkey MaxiDonkey added the documentation Improvements or additions to documentation label Dec 13, 2024
@MAUROFAILO
Copy link
Author

Sorry again.
I hope this is the last time I bother you!!!!
The problem is that converting this schema saved into a tmemo into a jsonobject does not work.
The following json shema seems correct to me and on internet.
I test the json schema in https://www.jsonschemavalidator.net/
This function ...
TJSONObject.ParseJSONValue(memo1.lines.text) as TJSONObject) .... return Nil !!!!!!!
or as in your example
Params.ResponseSchema(TJSONObject.ParseJSONValue(memo1.lines.text) as TJSONObject); .... return Nil !!!!!!!

The convertion does not work.
Have you a idea ???

This is the schema i used and saved in a tmemo.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "fattura": {
      "type": "object",
      "properties": 
  {

  "type": "object",
  "properties": {
    "Cedente": {
      "type": "object",
      "properties": 
      {
        "cedente_denominazione":             {"type": "string"},
        "cedente_nome":                      {"type": ["string", "null"]},
        "cedente_cognome":                   {"type": ["string", "null"]},
        "cedente_indirizzo":                 {"type": ["string", "null"]},
        "cedente_citta":                     {"type": ["string", "null"]},
        "cedente_stato":                     {"type": ["string", "null"]},
        "cedente_telefono":                  {"type": ["string", "null"]},
        "cedente_fax":                       {"type": ["string", "null"]},
        "cedente_partita_iva":               {"type": "string"}
        "cedente_iban":                      {"type": ["string", "null"]},
        "cedente_Ust_ID_Nr_1":               {"type": ["string", "null"]},
        "cedente_Ust_ID_Nr_2":               {"type": ["string", "null"]},
      },
      "required": [
        "cedente_partita_iva",
        "cedente_denominazione"
        "cedente_nome",
        "cedente_cognome",
        "cedente_indirizzo"
      ]
    },
    "Cessionario": {
      "type": "object",
      "properties": 
      {
        "cessionario_denominazione":       {"type": "string"},
        "cessionario_nome":                {"type": ["string", "null"]},
        "cessionario_cognome":             {"type": ["string", "null"]},
        "cessionario_indirizzo":           {"type": ["string", "null"]},
        "cessionario_citta":               {"type": ["string", "null"]},
        "cessionario_stato":               {"type": ["string", "null"]},
        "cessionario_codice_cliente":      {"type": ["string", "null"]},
        "cessionario_vat_number":          {"type": ["string", "null"]},
        "cessionario_partita_iva":         {"type": "string"}
        "cessionario_iban":                 {"type": ["string", "null"]},
        "cessionario_Ust_ID_Nr_1":          {"type": ["string", "null"]},
        "cessionario_Ust_ID_Nr_2":          {"type": ["string", "null"]},
      },
      "required": [
        "cessionario_partita_iva",
        "cessionario_denominazione"
        "cessionario_nome",
        "cessionario_cognome",
        "cessionario_indirizzo"
      ]
    },
    "Dati_Fattura": {
      "type": "object",
      "properties": 
      { 
        "codice_tipo_documento":    {"type": ["string", "null"]},, 
        "tipo_documento":    {"type": ["string", "null"]},
        "codice_destinatario_sdi": {"type": ["string", "null"]},
        "numero_fattura":    {"type": "string"},
        "data_fattura":      {"type": "string","format": "dd/mm/yyyy"},
        "data_ordine":       {"type": "string","format": "dd/mm/yyyy"},
        "data_spedizione":   {"type": "string","format": "dd/mm/yyyy"},
        "numero_ordine":     {"type": ["string", "null"]},
        "note_spedizione":   {"type": ["string", "null"]},
        "note_trasporto":    {"type": ["string", "null"]},
        "agente":            {"type": ["string", "null"]},
        "termini_pagamento": {"type": ["string", "null"]},
      },
      "required": [
        "numero_fattura",
        "data_fattura"
      ]
    },
    "Spedizione": {
      "type": "object",
      "properties": 
      {
        "indirizzo_spedizione": {"type": ["string", "null"]},
      }
    },
    "Causale": {
      "type": "object",
      "properties": 
      {
        "causale_emissione": {"type": ["string", "null"]},
      }
    },
    "Righe": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": 
        {
          "codice_articolo": {"type": ["string", "null"]},
          "descrizione":     {"type": "string"},
          "descrizione_in_Italiano":     {"type": "string"},
          "quantita":        {"type": ["number", "null"]},
          "unita_misura":    {"type": ["string", "null"]},
          "prezzo_unitario": {"type": ["number", "null"]},
          "sconto":          {"type": ["number", "null"]},
          "importo":         {"type": ["number", "null"]},
          "iva":             {"type": ["number", "null"]}
        },
        "required": [
          "codice_articolo",
          "descrizione",
          "quantita",
          "prezzo_unitario",
          "importo",
          "aliquota_iva"
        ]
      }
    },
    "Cassa_Previdenza": {
      "type": "object",
      "properties": 
      {
        "cassa_previdenza_codice":     {"type": ["string", "null"]},
        "cassa_previdenza_aliquota":   {"type": ["number", "null"]},
        "cassa_previdenza_imponibile": {"type": ["number", "null"]},
        "cassa_previdenza_ritenuta":   {"type": ["number", "null"]},
      }
    },
    "Ritenute": {
      "type": "object",
      "properties": 
      {
        "ritenuta_codice":     {"type": ["string", "null"]},
        "ritenuta_aliquota":   {"type": ["number", "null"]},
        "ritenuta_imponibile": {"type": ["number", "null"]},
        "ritenuta_importo":    {"type": ["number", "null"]},
      }
    },
    "Riepilogo_Iva": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": 
        {
          "imponibile":           {"type": ["number", "null"]},
          "codice_iva":           {"type": ["number", "null"]},
          "aliquota_iva":         {"type": ["number", "null"]},
          "descrizione_aliquota": {"type": ["string", "null"]},
          "importo_iva":          {"type": ["number", "null"]}
        },
        "required": [
          "imponibile",
          "aliquota_iva",
          "importo_iva"
        ]
      }
    },
    "Operazioni_Non_Soggette_Iva": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": 
        {
          "imponibile":           {"type": ["number", "null"]},
          "codice_iva":           {"type": ["number", "null"]},
          "aliquota_iva":         {"type": ["number", "null"]},
          "descrizione_aliquota": {"type": ["string", "null"]},
          "iva":                  {"type": ["number", "null"]}
        },
        "required": [
          "imponibile",
          "codice_iva",
          "aliquota_iva",
          "iva"
        ]
      }
    },
    "Totale_Generale": {
      "type": "object",
      "properties": 
      {
        "totale": {"type": ["number", "null"]},
      },
      "required": [
        "totale"
      ]
    },
    "Totale_Da_Pagare": {
      "type": "object",
      "properties": 
      {
        "totale_da_pagare": {"type": ["number", "null"]},
      },
      "required": [
        "totale"
      ]
    }
  },
  "required": [
    "Cedente",
    "Cessionario",
    "Dati_Fattura",
    "Righe",
    "Cassa_Previdenza",
    "Ritenute",
    "Riepilogo_Iva",
    "Operazioni_Non_Soggette_Iva"
    "Totale_Generale",
    "Totale_Da_Pagare"
  ]
}

}

@MaxiDonkey
Copy link
Owner

MaxiDonkey commented Dec 13, 2024

Hello MAUROFAILO,

You’re not bothering me; this space is precisely meant for such discussions, so no need to worry.
I’ve reviewed your issue, and it seems to stem from the invalidity of the scheme. Indeed, there are missing commas in various places, or extra commas have been added. The "required" sections are not perfectly correct.

Here is a corrected version, but you need to review it to ensure it is valid.

Note

First of all, you need to remove the first line; otherwise, the schema will be invalid.
"$schema": "http://json-schema.org/draft-04/schema#",

{
  "type": "object",
  "properties": {
    "fattura": {
      "type": "object",
      "properties": {
        "Cedente": {
          "type": "object",
          "properties": {
            "cedente_denominazione": {"type": "string"},
            "cedente_nome": {"type": ["string", "null"]},
            "cedente_cognome": {"type": ["string", "null"]},
            "cedente_indirizzo": {"type": ["string", "null"]},
            "cedente_citta": {"type": ["string", "null"]},
            "cedente_stato": {"type": ["string", "null"]},
            "cedente_telefono": {"type": ["string", "null"]},
            "cedente_fax": {"type": ["string", "null"]},
            "cedente_partita_iva": {"type": "string"},
            "cedente_iban": {"type": ["string", "null"]},
            "cedente_Ust_ID_Nr_1": {"type": ["string", "null"]},
            "cedente_Ust_ID_Nr_2": {"type": ["string", "null"]}
          },
          "required": [
            "cedente_partita_iva",
            "cedente_denominazione",
            "cedente_indirizzo"
          ]
        },
        "Cessionario": {
          "type": "object",
          "properties": {
            "cessionario_denominazione": {"type": "string"},
            "cessionario_nome": {"type": ["string", "null"]},
            "cessionario_cognome": {"type": ["string", "null"]},
            "cessionario_indirizzo": {"type": ["string", "null"]},
            "cessionario_citta": {"type": ["string", "null"]},
            "cessionario_stato": {"type": ["string", "null"]},
            "cessionario_codice_cliente": {"type": ["string", "null"]},
            "cessionario_vat_number": {"type": ["string", "null"]},
            "cessionario_partita_iva": {"type": "string"},
            "cessionario_iban": {"type": ["string", "null"]},
            "cessionario_Ust_ID_Nr_1": {"type": ["string", "null"]},
            "cessionario_Ust_ID_Nr_2": {"type": ["string", "null"]}
          },
          "required": [
            "cessionario_partita_iva",
            "cessionario_denominazione",
            "cessionario_indirizzo"
          ]
        },
        "Dati_Fattura": {
          "type": "object",
          "properties": {
            "codice_tipo_documento": {"type": ["string", "null"]},
            "tipo_documento": {"type": ["string", "null"]},
            "codice_destinatario_sdi": {"type": ["string", "null"]},
            "numero_fattura": {"type": "string"},
            "data_fattura": {"type": "string", "format": "dd/mm/yyyy"},
            "data_ordine": {"type": ["string", "null"], "format": "dd/mm/yyyy"},
            "data_spedizione": {"type": ["string", "null"], "format": "dd/mm/yyyy"},
            "numero_ordine": {"type": ["string", "null"]},
            "note_spedizione": {"type": ["string", "null"]},
            "note_trasporto": {"type": ["string", "null"]},
            "agente": {"type": ["string", "null"]},
            "termini_pagamento": {"type": ["string", "null"]}
          },
          "required": ["numero_fattura", "data_fattura"]
        },
        "Spedizione": {
          "type": "object",
          "properties": {
            "indirizzo_spedizione": {"type": ["string", "null"]}
          }
        },
        "Causale": {
          "type": "object",
          "properties": {
            "causale_emissione": {"type": ["string", "null"]}
          }
        },
        "Righe": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "codice_articolo": {"type": ["string", "null"]},
              "descrizione": {"type": "string"},
              "descrizione_in_Italiano": {"type": ["string", "null"]},
              "quantita": {"type": ["number", "null"]},
              "unita_misura": {"type": ["string", "null"]},
              "prezzo_unitario": {"type": ["number", "null"]},
              "sconto": {"type": ["number", "null"]},
              "importo": {"type": ["number", "null"]},
              "aliquota_iva": {"type": ["number", "null"]}
            },
            "required": [
              "descrizione",
              "quantita",
              "prezzo_unitario",
              "importo"
            ]
          }
        },
        "Cassa_Previdenza": {
          "type": "object",
          "properties": {
            "cassa_previdenza_codice": {"type": ["string", "null"]},
            "cassa_previdenza_aliquota": {"type": ["number", "null"]},
            "cassa_previdenza_imponibile": {"type": ["number", "null"]},
            "cassa_previdenza_ritenuta": {"type": ["number", "null"]}
          }
        },
        "Ritenute": {
          "type": "object",
          "properties": {
            "ritenuta_codice": {"type": ["string", "null"]},
            "ritenuta_aliquota": {"type": ["number", "null"]},
            "ritenuta_imponibile": {"type": ["number", "null"]},
            "ritenuta_importo": {"type": ["number", "null"]}
          }
        },
        "Riepilogo_Iva": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "imponibile": {"type": ["number", "null"]},
              "codice_iva": {"type": ["number", "null"]},
              "aliquota_iva": {"type": ["number", "null"]},
              "descrizione_aliquota": {"type": ["string", "null"]},
              "importo_iva": {"type": ["number", "null"]}
            },
            "required": ["imponibile", "aliquota_iva", "importo_iva"]
          }
        },
        "Operazioni_Non_Soggette_Iva": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "imponibile": {"type": ["number", "null"]},
              "codice_iva": {"type": ["number", "null"]},
              "aliquota_iva": {"type": ["number", "null"]},
              "descrizione_aliquota": {"type": ["string", "null"]},
              "iva": {"type": ["number", "null"]}
            },
            "required": ["imponibile", "codice_iva", "aliquota_iva", "iva"]
          }
        },
        "Totale_Generale": {
          "type": "object",
          "properties": {
            "totale": {"type": ["number", "null"]}
          },
          "required": ["totale"]
        },
        "Totale_Da_Pagare": {
          "type": "object",
          "properties": {
            "totale_da_pagare": {"type": ["number", "null"]}
          },
          "required": ["totale_da_pagare"]
        }
      },
      "required": [
        "Cedente",
        "Cessionario",
        "Dati_Fattura",
        "Righe",
        "Cassa_Previdenza",
        "Ritenute",
        "Riepilogo_Iva",
        "Operazioni_Non_Soggette_Iva",
        "Totale_Generale",
        "Totale_Da_Pagare"
      ]
    }
  }
}

Additionally, one way to check if the format is valid is as follows: I saved the JSON into a text file and then validated it with this code:

var
  JSON: TJSONObject;
  StringSchema: String;
begin
  with TStringList.Create do
  try
    LoadFromFile('Schema.txt');
    StringSchema := Text;
  finally
    Free;
  end;  
  StringSchema := StringSchema.Replace(#10, '').Replace(#13, ''); //Remove the line breaks for optimal functioning of ParseJSONValue.
  JSON := TJSONObject.ParseJSONValue(StringSchema) as TJSONObject;
  Memo1.Text := JSON.Format();
  ...

However, without the PDF file, I cannot guarantee the final result will work. That said, I believe it shouldn’t be an issue. If you encounter 400 errors, it means one of the required fields in the schema is missing from the PDF.

With the corrected JSON, test the following code:

var Chat:=Gemini.Chat.Create('models/gemini-1.5-flash-latest',
      Procedure (Params: TChatParams)
      Begin
        Params.Contents([TPayload.Add('Extract all the main data of this invoice', [FileUri])]);
        Params.GenerationConfig(
          procedure (var Params: TGenerationConfig)
          begin
            Params.ResponseMimeType('application/json');
            Params.ResponseSchema(JSON);
          end
        );
      end);  

Let me know if further clarification is needed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants