Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add staging to selftest #127

Merged
merged 6 commits into from
Feb 23, 2024
Merged

Add staging to selftest #127

merged 6 commits into from
Feb 23, 2024

Conversation

jku
Copy link
Member

@jku jku commented Feb 23, 2024 •
edited
Loading

This might be enough to close #121.

Summary

Do several small changes required to support staging tests in self-test

  • the enable-staging action input defaults to false: we may want to reconsider before releasing -- maybe opt-out is better?
  • only a few tests are marked as compatible with staging at this point: I may have missed some but since there are loads of false negatives in the remaining tests (tests that seemingly have the correct result but only because they fail for the wrong reasons on staging) I didn't want to extend without research

TODO?

  • --staging flag is sort of documented in cli_protocol.md but it could probably be improved

Test run https://github.com/sigstore/sigstore-conformance/actions/runs/8017481217

jku added 5 commits February 23, 2024 11:15
@jku
test: Add marker "staging"
1ab43e4 
Only run the marked tests when "--staging" is used. This allows us
* enable only the tests we know work on staging
* slowly add support to more tests (likely by adding staging test assets)
* at some point switch the marker use so that default is staging support
  and the remaining tests are marked with skipif()

I've only enabled three tests: they seem like they should work on
staging, I am not sure about any others.

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
action.py: Support GHA_SIGSTORE_CONFORMANCE_ENABLE_STAGING
fa4ec08 
This variable results in the test suite being run twice:
* first without --staging
* then with --staging

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
action: Add new input "enable-staging"
5fe520a 
Input defaults to false for now: We may want to consider enabling
by default when we release (or requiring a value).

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
self-test: Enable staging test
4ea10b7 
Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
Action: Tweak output
7965696 
* Single summary output line
* Add line in log with the "staging"/"production": this is useful to
  tell apart the two pytest result blocks

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku
Copy link
Member Author

jku commented Feb 23, 2024 •
edited
Loading

cc @javanlacerda please have a look:

  • did I miss anything?
  • does the approach of doing two runs of the test suite in a single GH action call make sense?

@jku
README: Update the action usage section
e671aeb 
The example had bit-rotted a bit so I rewrote it and added
enable-staging and xfail to README.

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku jku marked this pull request as ready for review February 23, 2024 15:09
@javanlacerda
Copy link
Contributor

cc @javanlacerda please have a look:

  • did I miss anything?
  • does the approach of doing two runs of the test suite in a single GH action call make sense?

Hey @jku! It seems awesome!

I am just wondering if we could have staging as default and production as optional. I mean, It could decrease the production infrastructure usage for development proposes. Does it make sense?

javanlacerda
javanlacerda approved these changes Feb 23, 2024
View reviewed changes
Copy link
Contributor

@javanlacerda javanlacerda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

@jku
Copy link
Member Author

jku commented Feb 23, 2024 •
edited
Loading

I am just wondering if we could have staging as default and production as optional. I mean, It could decrease the production infrastructure usage for development proposes. Does it make sense?

I think once we're sure this works the default should be to test everything (and not testing with production seems like a decision that could bite a client developer in the behind). Whether there is a use case to ever test only staging I don't know...

woodruffw
woodruffw approved these changes Feb 23, 2024
View reviewed changes
Copy link
Member

@woodruffw woodruffw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jku, LGTM!

@woodruffw
Copy link
Member

Whether there is a use case to ever test only staging I don't know...

I personally can't think of one 🙂 -- IMO we can save ourselves a state here and make staging tests "additive" rather than dealing with different combinations.

@woodruffw woodruffw added the component:tests Unit and integration tests label Feb 23, 2024
@woodruffw woodruffw merged commit 36c89ee into sigstore:main Feb 23, 2024
3 checks passed
@bdehamer bdehamer mentioned this pull request Mar 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@javanlacerda javanlacerda javanlacerda approved these changes

@woodruffw woodruffw woodruffw approved these changes

Assignees
No one assigned
Labels
component:cicd CI/CD component:tests Unit and integration tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Test against staging too
3 participants
@jku @javanlacerda @woodruffw