Improve runtime of tests #1854
Merged
Improve runtime of tests #1854
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tbro
force-pushed
the
tb/next-test-or-not
branch
from
86739ba to
ed19b80
Compare
this way we can compare
Get more information on how much time we spend on each task to better plan CI runtime optimizations.
This reverts commit 6c09212.
Only run that workflow on push to main (instead of PR)
This will be replaced by `nextest.yml` in before merge.
tbro
force-pushed
the
tb/next-test-or-not
branch
from
ed19b80 to
c1cd5f5
Compare
tbro
requested review from
nomaxg,
philippecamacho,
ImJeremyHe,
sveitser,
jbearer and
imabdulbasit
as code owners
sveitser
reviewed
Aug 14, 2024
We think this will not be required on PR since it is a new workflow.
sveitser
reviewed
Aug 14, 2024
| run: | | ||
| export RUSTFLAGS="$RUSTFLAGS --cfg hotshot_example" | ||
| export PATH="$PWD/target/release:$PATH" | ||
| cargo nextest run --locked --release --workspace --all-features --retries 2 --verbose -E '!test(slow_)' |
There was a problem hiding this comment.
I think it would be nice to add just recipies to mirror the new test commands locally since it's also nice to run the tests quickly locally. And sometimes we want to re-run the same tests as in a CI job locally.
sveitser
approved these changes
Aug 14, 2024
sveitser
reviewed
Aug 14, 2024
| @@ -0,0 +1,64 @@ | |||
| name: SlowEst | |||
There was a problem hiding this comment.
Suggested change
| name: SlowEst | |
| name: Slow Tests |
sveitser
reviewed
Aug 14, 2024
| RUST_LOG: info,libp2p=off,node=error | ||
|
|
||
| jobs: | ||
| slowest: |
There was a problem hiding this comment.
Or is "slowest" chosen on purpose?
There was a problem hiding this comment.
Yes. I was it was a play on nextest plus these are in fact the slowest tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1848
This PR improves CI runtime by working around slow tests. The last commit on main, tests took 36min. Tests are currently completing in this PR in 12m.
--retries(retries and marks tests that succeed upon retry as FLAKYslowdev-node tests to separate workflow, only run that workflow on push to 'main'Note that slow tests would now only be run on main, but the only tests currently considerd slow are
dev-nodetests. If this is merged more or less as is, I will create an issue to look into why these tests are slow and explore possible solutions.