Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Significant performance differences since benchmarks were last posted #51

Open
ItsDoot opened this issue Dec 15, 2023 · 1 comment
Open

Comments

@ItsDoot
Copy link

ItsDoot commented Dec 15, 2023

Decided to compare the benchmark results from c757a17 (when benchmark results were last posted) until 2869f9d (latest commit at time of writing).

I noticed some significant performance differences, and am curious why that could be:

     Running benches/all.rs (target/release/deps/all-164e82f8f06f0a40)
Gnuplot not found, using plotters backend
All<> - 1 match         time:   [86.178 µs 86.584 µs 87.044 µs]
                        change: [+63.099% +63.800% +64.533%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high severe

10000
All<> - 2 matches       time:   [115.23 µs 115.41 µs 115.63 µs]
                        change: [+33.806% +34.788% +35.940%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

20000
All<> - 1-2 matches     time:   [99.420 µs 99.742 µs 100.10 µs]
                        change: [+41.814% +43.307% +44.795%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

15000
     Running benches/concrete.rs (target/release/deps/concrete-fdfa9e097d2b391d)
Gnuplot not found, using plotters backend
concrete - 1 match      time:   [10.453 µs 10.460 µs 10.468 µs]
                        change: [+30.343% +31.453% +32.302%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

10000
concrete - 2 matches    time:   [10.536 µs 10.571 µs 10.609 µs]
                        change: [+21.170% +22.323% +23.419%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  5 (5.00%) high mild
  6 (6.00%) high severe

10000
     Running benches/fragmented.rs (target/release/deps/fragmented-599b12242e573cfa)
Gnuplot not found, using plotters backend
concrete - fragmented   time:   [1.0395 µs 1.0597 µs 1.0862 µs]
                        change: [-3.1553% -1.5826% -0.1409%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

4497169560
One<> - fragmented      time:   [3.1337 ns 3.1424 ns 3.1598 ns]
                        change: [-40.444% -40.207% -39.959%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe

0
All<> - fragmented      time:   [5.5079 µs 5.5820 µs 5.6784 µs]
                        change: [-0.0460% +0.7549% +1.6465%] (p = 0.08 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high severe

2014870000
     Running benches/one.rs (target/release/deps/one-6682cb865896f1c3)
Gnuplot not found, using plotters backend
One<>                   time:   [27.594 µs 27.610 µs 27.630 µs]
                        change: [+0.1968% +1.0798% +2.2258%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

10000
One<> - filtering       time:   [14.427 µs 14.456 µs 14.496 µs]
                        change: [-0.8238% -0.6224% -0.3342%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

5000
@ItsDoot
Copy link
Author

ItsDoot commented Dec 15, 2023

Here's 3 more runs, on Windows 11 this time (Linux Mint prior): https://gist.github.com/ItsDoot/571107bd776bdc4a4178c520d77e612a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant