Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve runtime of tests #1854

Merged
merged 32 commits into from
Aug 14, 2024
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
58ec3cd
split build and test
tbro Aug 13, 2024
8e223df
add nextest
tbro Aug 13, 2024
a31ccef
add nextest
tbro Aug 13, 2024
45c3d79
try again
tbro Aug 13, 2024
4190d16
apparently nextest doesn't not have `--locked` param
tbro Aug 13, 2024
25de9ea
fix command
tbro Aug 13, 2024
b3e14f0
fix
tbro Aug 13, 2024
4d3e749
Add nextest to flake.nix
sveitser Aug 13, 2024
97f477e
Separate building of tests from running of tests
sveitser Aug 13, 2024
8ef47fa
CI: install nextest before using it
sveitser Aug 13, 2024
5abed7f
prefix `slow_` tests
tbro Aug 13, 2024
4fb32e2
remove just
tbro Aug 13, 2024
76117bb
Separate slow tests
tbro Aug 13, 2024
8e91d53
remove async flags
tbro Aug 13, 2024
d40f2fa
format
tbro Aug 13, 2024
c225e4d
put back the flags
tbro Aug 13, 2024
10005c6
more time for the test
tbro Aug 13, 2024
7622602
reduce slow timeout
tbro Aug 13, 2024
9d394c3
add slow_ to test_restart
tbro Aug 13, 2024
f1796ef
remove --test-threads option
tbro Aug 14, 2024
aed01e9
add retry
tbro Aug 14, 2024
41278cf
reduce timeout
tbro Aug 14, 2024
b672cfc
Revert "add slow_ to test_restart"
tbro Aug 14, 2024
284d97f
move slow tests to own workflow
tbro Aug 14, 2024
c8f18a4
remove `slow_` from test_wait_for_finalized_block
tbro Aug 14, 2024
c093376
remove `test.yml`
tbro Aug 14, 2024
0874381
change names from `NexTest` to `Test`
tbro Aug 14, 2024
c1cd5f5
rename nextest.yml -> test.yml
tbro Aug 14, 2024
39d2e88
add pull_request
tbro Aug 14, 2024
5a609df
do not cancel job on `main`
tbro Aug 14, 2024
a7d14fd
add just commands
tbro Aug 14, 2024
5494139
typo
tbro Aug 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/slowtest.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: SlowEst
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: SlowEst
name: Slow Tests


on:
push:
branches:
- main
- release-*
tags:
# YYYYMMDD
- "20[0-9][0-9][0-1][0-9][0-3][0-9]*"
schedule:
- cron: "0 0 * * 1"
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
sveitser marked this conversation as resolved.
Show resolved Hide resolved

env:
RUSTFLAGS: '--cfg async_executor_impl="async-std" --cfg async_channel_impl="async-std"'
RUST_LOG: info,libp2p=off,node=error

jobs:
slowest:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or is "slowest" chosen on purpose?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I was it was a play on nextest plus these are in fact the slowest tests.

runs-on: ubuntu-latest
steps:
- name: Fix submodule permissions check
run: |
git config --global --add safe.directory '*'

- name: Install Foundry
uses: foundry-rs/foundry-toolchain@v1
with:
version: nightly

- uses: taiki-e/install-action@nextest

- name: Checkout Repository
uses: actions/checkout@v4
with:
submodules: recursive

- name: Enable Rust Caching
uses: Swatinem/rust-cache@v2

- name: Build
# Build test binary with `testing` feature, which requires `hotshot_example` config
run: |
export RUSTFLAGS="$RUSTFLAGS --cfg hotshot_example"
export PATH="$PWD/target/release:$PATH"
cargo build --locked --bin diff-test --release
cargo nextest run --locked --release --workspace --all-features --no-run
timeout-minutes: 90

- name: SlowEst
env:
CARGO_TERM_COLOR: always
# Build test binary with `testing` feature, which requires `hotshot_example` config
run: |
export RUSTFLAGS="$RUSTFLAGS --cfg hotshot_example"
export PATH="$PWD/target/release:$PATH"
cargo nextest run --locked --release --workspace --all-features --verbose -E 'test(slow_)'
timeout-minutes: 25
20 changes: 13 additions & 7 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,7 @@ jobs:
with:
version: nightly

- name: Install just command runner
run: |
sudo snap install --edge --classic just
just --version
- uses: taiki-e/install-action@nextest

- name: Checkout Repository
uses: actions/checkout@v4
Expand All @@ -47,12 +44,21 @@ jobs:
- name: Enable Rust Caching
uses: Swatinem/rust-cache@v2

- name: Test
- name: Build
# Build test binary with `testing` feature, which requires `hotshot_example` config
run: |
export RUSTFLAGS="$RUSTFLAGS --cfg hotshot_example"
export PATH="$PWD/target/release:$PATH"
cargo build --locked --bin diff-test --release
cargo test --locked --release --workspace --all-features --no-run
cargo test --locked --release --workspace --all-features --verbose -- --test-threads 1 --nocapture
cargo nextest run --locked --release --workspace --all-features --no-run
timeout-minutes: 90

- name: Test
env:
CARGO_TERM_COLOR: always
# Build test binary with `testing` feature, which requires `hotshot_example` config
run: |
export RUSTFLAGS="$RUSTFLAGS --cfg hotshot_example"
export PATH="$PWD/target/release:$PATH"
cargo nextest run --locked --release --workspace --all-features --retries 2 --verbose -E '!test(slow_)'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to add just recipies to mirror the new test commands locally since it's also nice to run the tests quickly locally. And sometimes we want to re-run the same tests as in a CI job locally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

timeout-minutes: 5
1 change: 1 addition & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@
cargo-audit
cargo-edit
cargo-sort
cargo-nextest
typos
just
fenix.packages.${system}.rust-analyzer
Expand Down
4 changes: 2 additions & 2 deletions sequencer/src/bin/espresso-dev-node.rs
Original file line number Diff line number Diff line change
Expand Up @@ -541,7 +541,7 @@ mod tests {
// - APIs update
// - Types (like `Header`) update
#[async_std::test]
async fn dev_node_test() {
async fn slow_dev_node_test() {
setup_test();

let builder_port = pick_unused_port().unwrap();
Expand Down Expand Up @@ -849,7 +849,7 @@ mod tests {
}

#[async_std::test]
async fn dev_node_multiple_lc_providers_test() {
async fn slow_dev_node_multiple_lc_providers_test() {
setup_test();

let builder_port = pick_unused_port().unwrap();
Expand Down
Loading