Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unreliable test: zipCode performance tests #6117

Closed
justinmk3 opened this issue Nov 28, 2024 · 2 comments · Fixed by #6120
Closed

unreliable test: zipCode performance tests #6117

justinmk3 opened this issue Nov 28, 2024 · 2 comments · Fixed by #6120
Labels
tests-ci-cd

Comments

@justinmk3
Copy link
Contributor

Test Details

  • Name of test: zipCode performance tests
  • OS of failure: Linux
  • VSCode version of failure (minimum/insider/stable): Integration Tests - insiders
  • Link to failing run: Integration Tests - insiders on 5766e4b

Log of Test Failure

  1) zipCode
       performance tests
         zipCode performance tests
           "after all" hook for "zipCode - test run 10":

      AssertionError [ERR_ASSERTION]: Expected total user CPU usage for zipCode to be less than 200. Actual user CPU usage was 277.1372267951132
      + expected - actual

      -false
      +true
@Hweinstock Hweinstock mentioned this issue Nov 29, 2024
Merged
@justinmk3
Copy link
Contributor Author 

  1) zipCode
       performance tests
         zipCode performance tests
           "after all" hook for "zipCode - test run 10":

      AssertionError [ERR_ASSERTION]: Expected total user CPU usage for zipCode to be less than 350. Actual user CPU usage was 388.65274407871664
      + expected - actual

f004173

@justinmk3 justinmk3 reopened this Jan 22, 2025
This was referenced Jan 29, 2025
Closed
Merged
Hweinstock added a commit that referenced this issue Jan 31, 2025
@Hweinstock
test(flaky): skip all performance tests. (#6459)
82cae41 
## Problem
#6117
#6457

These issues keep popping up, regardless of the the thresholds. The
performance tests are causing more problems then they are solving. Real
performance regressions would be captured in telemetry.

## Solution
skip all performance tests, revisit once there is less flakiness. 

---

- Treat all work as PUBLIC. Private `feature/x` branches will not be
squash-merged at release time.
- Your code changes must meet the guidelines in
[CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines).
- License: I confirm that my contribution is made under the terms of the
Apache 2.0 license.
@Hweinstock
Copy link
Contributor

Skipped in #6459

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests-ci-cd
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test(flaky): unreliable test: zipCode performance tests Hweinstock/aws-toolkit-vscode
2 participants
@Hweinstock @justinmk3