Test phrase passed even though Test cases failed #2745

andychoww · 2022-05-10T09:37:28Z

Before opening, please confirm:

I have searched for duplicate or closed issues and discussions.
I have read the guide for submitting bug reports.
I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

Not applicable

Amplify Categories

hosting

Environment information

I am using Jest as the Test runner

Describe the bug

The Jest test cases fail on Test Phrase in Amplify pipeline, but it still continue to proceed instead of stopping.

Expected behavior

The Test Phrase should be failed and stop proceeding to Deploy Phrase

Reproduction steps

commit the changes in branch and push to remote
The pipelines in Amplify runs
The Jest test cases fail in Test Phrase, but the pipeline still continues
The application deploys successfully

Code Snippet

// Put your code below this line.

Jest.config.ts

export default {
  preset: 'ts-jest',
  coverageProvider: 'v8',
  globals: {
    'ts-jest': {
      isolatedModules: true,
    },
  },
  testEnvironment: 'node',
  transform: {
    'node_modules/variables/.+\\.(j|t)sx?$': 'ts-jest',
  },
  transformIgnorePatterns: ['node_modules/(?!variables/.*)'],
};

amplify.yml

version: 2
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
    test:
      commands:
        - 'npm test'
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'

Log output

// Put your logs below this line
2022-05-10T09:14:52.261Z [WARNING]: FAIL src/utils/formatters/formatters.test.ts
2022-05-10T09:14:52.286Z [WARNING]: ● formatters test › decimalPlacesRegex test › should return true - when the decimal of the string equals to decimalPlaces
                                    expect(received).toEqual(expected) // deep equality
                                    Expected: false
                                    Received: true
                                    28 |     it('should return true - when the decimal of the string equals to decimalPlaces', () => {
                                    29 |       const result = decimalPlacesRegex(DEFAULT_AMOUNT, DEFAULT_DECIMAL);
                                    > 30 |       expect(result).toEqual(false);
                                    |                      ^
                                    31 |     });
                                    32 |     it('should return false - when the decimal of the string is different from decimalPlaces', () => {
                                    33 |       const result = decimalPlacesRegex(DEFAULT_AMOUNT, 2);
                                    at Object.<anonymous> (src/utils/formatters/formatters.test.ts:30:22)
2022-05-10T09:14:52.301Z [WARNING]: Test Suites: 1 failed, 1 passed, 2 total
                                    Tests:       1 failed, 16 passed, 17 total
                                    Snapshots:   0 total
                                    Time:        1.309 s
                                    Ran all test suites.
2022-05-10T09:14:52.376Z [WARNING]: npm
2022-05-10T09:14:52.377Z [WARNING]: ERR! Test failed.  See above for more details.
2022-05-10T09:14:52.380Z [INFO]: # Completed phase: test
2022-05-10T09:14:52.380Z [INFO]: ## Tests completed successfully
2022-05-10T09:14:52.382Z [INFO]: # Starting test artifact upload process...
2022-05-10T09:14:59.382Z [INFO]: # Uploading build artifact 'testConfig.json'...
2022-05-10T09:14:59.950Z [INFO]: # Build artifact is: 6MB
2022-05-10T09:14:59.951Z [INFO]: # Uploading build artifact 'testArtifacts.zip'...
2022-05-10T09:15:00.172Z [INFO]: # Test artifact upload completed
2022-05-10T09:15:00.175Z [INFO]: No custom headers found.
2022-05-10T09:15:00.194Z [INFO]: # Starting build artifact upload process...
2022-05-10T09:15:00.667Z [INFO]: # Build artifact is: 4MB
2022-05-10T09:15:00.668Z [INFO]: # Uploading build artifact '__artifacts.zip'...
2022-05-10T09:15:00.686Z [INFO]: # Build artifact is: 4MB
2022-05-10T09:15:00.686Z [INFO]: # Uploading build artifact '__artifactsHash.zip'...
2022-05-10T09:15:00.849Z [INFO]: # Build artifact upload completed
2022-05-10T09:15:00.849Z [INFO]: # Starting environment caching...
2022-05-10T09:15:00.849Z [INFO]: # Uploading environment cache artifact...
2022-05-10T09:15:00.928Z [INFO]: # Environment caching completed
Terminating logging...

The text was updated successfully, but these errors were encountered:

chrisbonifacio · 2022-05-10T19:26:59Z

Hi @andychoww 👋 thanks for raising this issue. This seems to be related to Amplify Hosting's CI/CD pipeline. I will transfer this over to the hosting team's repo for better support and feedback.

ghost · 2022-05-10T22:37:31Z

Hi @andychoww 👋🏽 can you try implementing the workaround suggested here please: #2409 (comment)

andychoww · 2022-05-11T01:38:56Z

Hi @hloriana tried but the test phrase still passes

version: 4
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
    test:
      commands:
        - 'npm test'
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'

andychoww · 2022-05-11T01:58:20Z

I even tried below but still not working

version: 5
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
    test:
      commands:
        - CI=true npm test
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

ghost · 2022-05-11T22:43:33Z

@andychoww please add - npx pm2 start npm -- start under preTest like in this example for running Cypress tests:

test:
  phases:
    preTest:
      commands:
        - npm ci
        - npm install wait-on
        - npm install pm2
        - npm install [email protected] mochawesome mochawesome-merge mochawesome-report-generator
        - npx pm2 start npm -- start
        - 'npx wait-on --timeout 60 http://localhost:3000'
    test:
      commands:
        - 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
        - npx pm2 kill
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'

andychoww · 2022-05-16T02:43:56Z

@hloriana I have added that in preTest but still not working

version: 6
frontend:
  phases:
    preBuild:
      commands:
        - npm install
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm install
        - npm install pm2
        - npx pm2 start npm -- start
    test:
      commands:
        - npm test
    postTest:
      commands:
        - npx pm2 kill
  artifacts:
    baseDirectory: ./
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

ghost · 2022-05-18T19:21:30Z

Thank you for testing that out @andychoww. We are tracking this as a bug and will update you once it's been fixed. In the mean time, we are still investigating a possible workaround. Thank you for your patience!

aleksey-moravio · 2022-09-19T18:31:44Z

Solution from @hloriana is about Cypress test, but @andychoww report about Jest tests.
That's why all previous conversation is useless.

ghost · 2022-11-01T20:09:57Z

Amplify Hosting only supports testing with Cypress at this time (see here). Please follow the feature request to support testing with the Jest framework on issue #464.

github-actions · 2022-11-01T20:10:25Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

github-actions · 2022-11-01T21:02:18Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

chrisbonifacio transferred this issue from aws-amplify/amplify-js May 10, 2022

chrisbonifacio added pending-prioritization Added to the backlog. Pending prioritization. e2e-cypress-tests Cypress/E2E test step issue deploy transferred This issue was transferred from another Amplify project labels May 10, 2022

ghost self-assigned this May 10, 2022

ghost added the UX label May 11, 2022

ghost closed this as completed Nov 1, 2022

github-actions bot added the archived This issue has been locked. label Nov 1, 2022

github-actions bot locked and limited conversation to collaborators Nov 1, 2022

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test phrase passed even though Test cases failed #2745

Test phrase passed even though Test cases failed #2745

andychoww commented May 10, 2022 •

edited

Loading

chrisbonifacio commented May 10, 2022

ghost commented May 10, 2022

andychoww commented May 11, 2022 •

edited

Loading

andychoww commented May 11, 2022

ghost commented May 11, 2022

andychoww commented May 16, 2022 •

edited

Loading

ghost commented May 18, 2022

aleksey-moravio commented Sep 19, 2022

ghost commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

Test phrase passed even though Test cases failed #2745

Test phrase passed even though Test cases failed #2745

Comments

andychoww commented May 10, 2022 • edited Loading

Before opening, please confirm:

JavaScript Framework

Amplify APIs

Amplify Categories

Environment information

Describe the bug

Expected behavior

Reproduction steps

Code Snippet

Log output

chrisbonifacio commented May 10, 2022

ghost commented May 10, 2022

andychoww commented May 11, 2022 • edited Loading

andychoww commented May 11, 2022

ghost commented May 11, 2022

andychoww commented May 16, 2022 • edited Loading

ghost commented May 18, 2022

aleksey-moravio commented Sep 19, 2022

ghost commented Nov 1, 2022

github-actions bot commented Nov 1, 2022

⚠️COMMENT VISIBILITY WARNING⚠️

github-actions bot commented Nov 1, 2022

andychoww commented May 10, 2022 •

edited

Loading

andychoww commented May 11, 2022 •

edited

Loading

andychoww commented May 16, 2022 •

edited

Loading