Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the schema generator self contained as a Python package #127

Open
acmiyaguchi opened this issue May 8, 2020 · 0 comments
Open

Make the schema generator self contained as a Python package #127

acmiyaguchi opened this issue May 8, 2020 · 0 comments

Comments

@acmiyaguchi
Copy link
Contributor

Currently, the schema generator requires a full working environment within the docker environment in order to work. This means to test new changes to the schema generator, the entire container must be rebuilt at each specific revision.

If the schema generation logic in bin were moved into the main package, with only the git and jsonschema-transpiler dependencies required in the environment, then it would be much easier to test generated schemas.

The ideal workflow would involve calling the application within a schema repository. For example:

cd mozilla-pipeline-schemas
mozilla-schema-generator generate \
    --input schemas/ \
    --interim generated-schemas/ \
    --output-branch test-generated-schemas

This would allow us to move away from coreutils and bash for schema generating logic. It is difficult (but not impossible) to test changes to the schema generator. However, it would be easier to test things when the script can be broken into smaller chunks.

We should make pushing to git an optional. Sometimes, what we want to do is to generate a diff between two versions of the schema repository. For this, we can specify an interim folder where generated schemas can be dumped into.

Some of the utilities and testing from https://github.com/mozilla-services/mozilla-pipeline-schemas/blob/master/scripts/bigquery_schema_diff.py may be useful here. In particular, the tmp_git fixture would be useful for testing pushes to generated-schemas, without requiring an SSH key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant