Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow expensive tests to be disabled #574

Open
niklasdewally opened this issue Jan 8, 2025 · 1 comment
Open

Allow expensive tests to be disabled #574

niklasdewally opened this issue Jan 8, 2025 · 1 comment
Labels
kind::testing Testing and Correctness

Comments

@niklasdewally
Copy link
Collaborator

niklasdewally commented Jan 8, 2025

Summary

Allow the selection of which tests to run depending on how expensive (time consuming they are). This should take the form of a flag to the tester. For example, in Conjure, --limit_time=5 runs only those tests that have an expected run time of <=5 seconds.

CI should be modified to run both a quick version of the tests, and all the tests.

Background

In #572, I proposed using Rust's builtin #[ignore] attribute 1 to disable by default a test case that took too long to run.

However, we want a more fine grained approach to select which tests to run than whether a test is ignored or not.

Oz suggested the following:

I like the idea of having infrastructure to select which test to run depending on how expensive the test is though. In Conjure I optionally add a rough expected time (in seconds) for each test case. Then running --limit_time=5 runs only those tests that have an expected run time of <=5 seconds. Default value is 0.

Could be better than the coarse grained quick/all. Quick can then be an alias for <=1 or whatever and all can be actually all.

@niklasdewally
Copy link
Collaborator Author

CC: @EEJDempster @Soph1514 @ozgurakgun

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind::testing Testing and Correctness
Projects
None yet
Development

No branches or pull requests

1 participant