Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JSON Schema definition of the submitted data to the JSON registration backend plugin #4980

Open
Tracked by #4908
viktorvanwijk opened this issue Jan 2, 2025 · 0 comments · May be fixed by #5007
Open
Tracked by #4908

Comments

@viktorvanwijk
Copy link
Contributor

viktorvanwijk commented Jan 2, 2025

Sub task of #4908

Example

An example form with the following configuration:

  • Authentication via DigiD, capturing a BSN
  • First form step with text fields firstName and lastName
  • Second form step with a user upload attachment

Should lead to a JSON object being sent with the shape:

{
    "values":
    {
        "auth_bsn": "123456782",
        "firstName": "Donnie",
        "lastName": "Darko",
        "attachment": "<base64 encoded data>"
    },
    "schema":
    {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "type": "object",
        "properties":
        {
            "auth_bsn":
            {
                "type": "string",
                "pattern": "^\\d{9}$"
            },
            "firstName":
            {
                "type": "string"
            },
            "lastName":
            {
                "type": "string"
            },
            "attachment":
            {
                "type": "string",
                "contentEncoding": "base64"
            }
        },
        "required":
        [
            "auth_bsn",
            "firstName",
            "lastName"
        ],
        "additionalProperties": false
    }
}
@viktorvanwijk viktorvanwijk self-assigned this Jan 2, 2025
@viktorvanwijk viktorvanwijk added enhancement topic: registration waiting for approval An estimate was made but the stakeholder still needs to approve it. owner: venlo labels Jan 2, 2025
@viktorvanwijk viktorvanwijk changed the title Optionally, a JSON Schema definition of the submitted data is provided in the schema key Add JSON Schema definition of the submitted data to the JSON registration backend Jan 2, 2025
@viktorvanwijk viktorvanwijk changed the title Add JSON Schema definition of the submitted data to the JSON registration backend Add JSON Schema definition of the submitted data to the JSON registration backend plugin Jan 2, 2025
viktorvanwijk added a commit that referenced this issue Jan 2, 2025
…lugin

Just as an example of what will be implemented in #4980
@viktorvanwijk viktorvanwijk added this to the Release 3.1.0 milestone Jan 6, 2025
@viktorvanwijk viktorvanwijk moved this from Todo to In Progress in Development Jan 6, 2025
viktorvanwijk added a commit that referenced this issue Jan 8, 2025
The schema for the registration variables are left out for now, as they can't be selected in the JSON dump plugin options
viktorvanwijk added a commit that referenced this issue Jan 8, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 8, 2025
- Add test for JSON schemas of formio components
- Add test for form to JSON schema
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
The schema for the registration variables are left out for now, as they can't be selected in the JSON dump plugin options
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
- Add test for JSON schemas of formio components
- Add test for form to JSON schema
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
- Add test for JSON schemas of formio components
- Add test for form to JSON schema
viktorvanwijk added a commit that referenced this issue Jan 9, 2025
There was a bug with the previous attachment processing, where attachments would get overwritten if multiple were uploaded
viktorvanwijk added a commit that referenced this issue Jan 10, 2025
The schema for the registration variables are left out for now, as they can't be selected in the JSON dump plugin options
viktorvanwijk added a commit that referenced this issue Jan 10, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
- Add test for JSON schemas of formio components
- Add test for form to JSON schema
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
Also updated form_variables_to_json_schema
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
…led field

Non-required fields can be empty upon submission, so need to include an empty string.
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
…t if no value was submitted

If the component is not required, the value is an empty dict, which means there are no required properties.

The `component is None` check is also not necessary anymore, as the user defined variables are excluded earlier already
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
…omponents

Only add enum/properties to the schema when the data source is not another form variable. If the data source is a form variable, the list of options is not available here, so it doesn't make sense to include the enum/properties in the JSON schema
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
When the data source for these components is another form variable, the options are not available in the component.as_json_schema methods based on the form alone, because all user defined variables and calculated values are saved in the DB at the moment of submission. So just the form is not enough to get this data, and we need to do some post-processing in the plugin where we have the submission available
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
When the data source for these components is another form variable, the options are not available in the component.as_json_schema methods based on the form alone, because all user defined variables and calculated values are saved in the DB at the moment of submission. So just the form is not enough to get this data, and we need to do some post-processing in the plugin where we have the submission available
viktorvanwijk added a commit that referenced this issue Jan 22, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
@viktorvanwijk viktorvanwijk moved this from In Progress to Implemented in Development Jan 22, 2025
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
…f variables to include

Also updated the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
- Add test for JSON schemas of formio components
- Add test for form to JSON schema
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
Also updated form_variables_to_json_schema
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
…led field

Non-required fields can be empty upon submission, so need to include an empty string.
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
…t if no value was submitted

If the component is not required, the value is an empty dict, which means there are no required properties.

The `component is None` check is also not necessary anymore, as the user defined variables are excluded earlier already
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
…omponents

Only add enum/properties to the schema when the data source is not another form variable. If the data source is a form variable, the list of options is not available here, so it doesn't make sense to include the enum/properties in the JSON schema
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
When the data source for these components is another form variable, the options are not available in the component.as_json_schema methods based on the form alone, because all user defined variables and calculated values are saved in the DB at the moment of submission. So just the form is not enough to get this data, and we need to do some post-processing in the plugin where we have the submission available
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
In the JSON dump docker app, explicitly load the received data before sending again. This prevents the data from being interpreted as a string instead of a JSON object.
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
to_multiple, rewrite_formio_components, and DataContainer need to be used in the JSON dump plugin
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
* Rewrite fomio components only once before the processing, as it can be called 10-100 times if done for each component separately.
* Perform component matching based on the dict, as it makes the structure flatter and easier to follow
* Make `post_process` a plain function, as there is not much reason for it to be part of the plugin class. Also, if the base plugin ever gains a `post_process` hook, this would be problematic
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
* Removed the user defined part, as it is valid for any type of form variable, albeit very simplistic
* Convert to a constant
* Use it as a base schema for all static variables
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
openforms.forms.utils is import/export stuff, so created a dedicated module json_schema

The idea is to fully define the schema generation in terms of `FormVariable`, as we have three possible options:
* Formio component -> `FormVariable` with source set to component (we can get more info from the component
* User-defined variable -> `FormVariable` with source set to user_defined (we can't get more info at all)
* Static variable -> `FormVariable` that only exists in-memory and not in the DB, but the implementation details are outside of it

Implementation-wise:
* Formio component -> schema will be generated inside `FormVariable`. If it fails, fall back to basic schema based on data type.
* User-defined variable -> use basic schema based on data type
* Static variable -> generate schema separately, and assign it manually the `json_schema` property of a `FormVariable` instance
viktorvanwijk added a commit that referenced this issue Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Implemented
Development

Successfully merging a pull request may close this issue.

2 participants