QE-15097 report after hook errors as passed (#439)

Issue: After hook errors are reported as 'Errored' even when the test passes. This is causing an issue and blocking the release/pr when basic smoke tests or sanity tests are marked as 'Errored' because of after hook errors. Solution: Mark the test as 'Passed' even when the after scenario hook fails. For before scenario hook failures the tests status will be 'Errored' Jira: [QE-15097](https://dominodatalab.atlassian.net/browse/QE-15097) [QE-15097]: https://dominodatalab.atlassian.net/browse/QE-15097?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
dominodatalab · Mar 1, 2024 · 3800256 · 3800256
1 parent b2e95b5
commit 3800256
Show file tree

Hide file tree

Showing 6 changed files with 19 additions and 20 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,9 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project closely adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.189.0
+- Change - after-hook failures will report as 'passed' instead of 'failed/errored' to testrail and html reports
+
 ## 0.188.0
 - Fix - how element size is gotten in fuzzy find
 

diff --git a/features/cli/report_basics.feature b/features/cli/report_basics.feature
@@ -87,7 +87,7 @@ Feature: Report basics
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
      Then I should see a table that matches the following:
        | Started at | Feature                    | Total | Passed | Failed | Skipped | Errored | Status | Duration |
-       | .*         | Feature with mixed results | 6     | 2      | 3      | 0       | 1       | failed | .*       |
+       | .*         | Feature with mixed results | 6     | 3      | 3      | 0       | 0       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
       | Offset | Scenario                            | Steps | Status  | Duration |
@@ -96,7 +96,7 @@ Feature: Report basics
       | .*     | Scenario that fails                 | 2     | failed  | .*       |
       | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
       | .*     | Scenario that passes                | 1     | passed  | .*       |
-      | .*     | Scenario with after-hook error      | 2     | errored | .*       |
+      | .*     | Scenario with after-hook error      | 2     | passed  | .*       |
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
      When I click the button "Index"
@@ -112,7 +112,7 @@ Feature: Report basics
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
      Then I should see a table that matches the following:
        | Started at | Feature                    | Total | Passed | Failed | Skipped | Errored | Status | Duration |
-       | .*         | Feature with mixed results | 7     | 2      | 3      | 1       | 1       | failed | .*       |
+       | .*         | Feature with mixed results | 7     | 3      | 3      | 1       | 0       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
       | Offset | Scenario                            | Steps | Status  | Duration |
@@ -122,7 +122,7 @@ Feature: Report basics
       | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
       | .*     | Scenario that is skipped            | 1     | skipped | .*       |
       | .*     | Scenario that passes                | 1     | passed  | .*       |
-      | .*     | Scenario with after-hook error      | 2     | errored | .*       |
+      | .*     | Scenario with after-hook error      | 2     | passed  | .*       |
 
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
@@ -224,7 +224,7 @@ Feature: Report basics
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
      Then I should see a table that matches the following:
       | Started at | Feature                    | Total | Passed | Failed | Skipped | Errored | Status | Duration |
-      | .*         | Feature with mixed results | 6     | 2      | 3      | 0       | 1       | failed | .*       |
+      | .*         | Feature with mixed results | 6     | 3      | 3      | 0       | 0       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
       | Offset | Scenario                            | Steps | Status  | Duration |
@@ -233,7 +233,7 @@ Feature: Report basics
       | .*     | Scenario that fails                 | 2     | failed  | .*       |
       | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
       | .*     | Scenario that passes                | 1     | passed  | .*       |
-      | .*     | Scenario with after-hook error      | 2     | errored | .*       |
+      | .*     | Scenario with after-hook error      | 2     | passed  | .*       |
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
      When I click the button "Index"

diff --git a/features/cli/run_outputs.feature b/features/cli/run_outputs.feature
@@ -144,7 +144,7 @@ Feature: Run outputs
      Then I should see a file at "{CUCU_RESULTS_DIR}/validate_junit_xml_results/Feature with mixed results.xml"
       And I should see the file at "{CUCU_RESULTS_DIR}/validate_junit_xml_results/Feature with mixed results.xml" matches the following:
       """
-      <testsuite name="Feature with mixed results" tests="7" errors="1" failures="3" skipped="1" timestamp=".*" tags="mixed">
+      <testsuite name="Feature with mixed results" tests="7" errors="0" failures="3" skipped="1" timestamp=".*" tags="mixed">
        <testcase classname="Feature with mixed results" name="Scenario that passes" status="passed" time=".*">
        </testcase>
        <testcase classname="Feature with mixed results" name="Scenario that fails" status="failed" time=".*">
@@ -157,7 +157,7 @@ Feature: Run outputs
         [\s\S]*
         </failure>
        </testcase>
-       <testcase classname="Feature with mixed results" name="Scenario with after-hook error" status="errored" time=".*">
+       <testcase classname="Feature with mixed results" name="Scenario with after-hook error" status="passed" time=".*">
        </testcase>
        <testcase classname="Feature with mixed results" name="Scenario that also passes" status="passed" time=".*">
        </testcase>

diff --git a/features/cli/run_with_hooks.feature b/features/cli/run_with_hooks.feature
@@ -147,7 +147,7 @@ Feature: Run with hooks
         Scenario: Hello world scenario
           Given I echo "Hello World"
       """
-     When I run the command "cucu run {CUCU_RESULTS_DIR}/failing_custom_hooks/echo.feature --results {CUCU_RESULTS_DIR}/failing_custom_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/failing_custom_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "1"
+     When I run the command "cucu run {CUCU_RESULTS_DIR}/failing_custom_hooks/echo.feature --results {CUCU_RESULTS_DIR}/failing_custom_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/failing_custom_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "0"
      Then I should see "{STDOUT}" contains the following
       """
       HOOK-ERROR in after_scenario_fail: RuntimeError: boom
@@ -160,7 +160,7 @@ Feature: Run with hooks
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
       And I click the link "Feature that fails due to after scenario hook"
       And I click the link "Hello world scenario"
-     Then I should see the text "HOOK-ERROR in after_scenario_fail: RuntimeError: boom"
+     Then I should see the text "Hello World"
 
   Scenario: User gets expected output when running a scenario with failing before hooks
     Given I create a file at "{CUCU_RESULTS_DIR}/failing_before_hooks/environment.py" with the following:
@@ -223,7 +223,7 @@ Feature: Run with hooks
         Scenario: Hello world scenario
           Given I echo "Hello World"
       """
-     When I run the command "cucu run {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks/echo.feature --results {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "1"
+     When I run the command "cucu run {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks/echo.feature --results {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/mixed_results_fail_pass_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "0"
      Then I should see "{STDOUT}" contains the following
       """
       HOOK-ERROR in after_scenario_fail: RuntimeError: boom
@@ -240,7 +240,7 @@ Feature: Run with hooks
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
       And I click the link "Feature that has failing and passing after scenario hooks"
       And I click the link "Hello world scenario"
-     Then I should see the text "HOOK-ERROR in after_scenario_fail: RuntimeError: boom"
+     Then I should see the text "Hello World"
 
   Scenario: User gets expected output when running a scenario with multiple after hooks passing and failing in order
     Given I create a file at "{CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks/environment.py" with the following:
@@ -270,7 +270,7 @@ Feature: Run with hooks
         Scenario: Hello world scenario
           Given I echo "Hello World"
       """
-     When I run the command "cucu run {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks/echo.feature --results {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "1"
+     When I run the command "cucu run {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks/echo.feature --results {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks_results/ -l debug --no-color-output --generate-report --report {CUCU_RESULTS_DIR}/mixed_results_pass_fail_hooks_report/" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "0"
      Then I should see "{STDOUT}" contains the following
       """
       HOOK-ERROR in after_scenario_fail: RuntimeError: boom
@@ -287,4 +287,4 @@ Feature: Run with hooks
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
       And I click the link "Feature that has failing and passing after scenario hooks"
       And I click the link "Hello world scenario"
-     Then I should see the text "HOOK-ERROR in after_scenario_fail: RuntimeError: boom"
+     Then I should see the text "Hello World"
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "cucu"
-version = "0.188.0"
+version = "0.189.0"
 license = "MIT"
 description = "Easy BDD web testing"
 authors = ["Domino Data Lab <[email protected]>"]

diff --git a/src/cucu/environment.py b/src/cucu/environment.py
@@ -124,15 +124,11 @@ def before_scenario(ctx, scenario):
 
 
 def run_after_scenario_hook(ctx, scenario, hook):
-    scenario_status = scenario.status.name
     try:
         hook(ctx)
         logger.debug(f"HOOK {hook.__name__}: passed ✅")
     except Exception as e:
-        # scenario 'failed' status takes precedence over 'errored' status
-        if scenario_status != "failed":
-            scenario.hook_failed = True
-        # attach error_message to scenario
+        # For any after scenario hooks,'hook_failed' status will be 'False' but will attach the error message to scenario.
         error_message = (
             f"HOOK-ERROR in {hook.__name__}: {e.__class__.__name__}: {e}\n"
         )