Integration of basic functional test suite developed with FTR into the Create Environment workflow #2219

gurevichdmitry · 2024-05-20T14:37:16Z

Summary of your changes

This PR adds the ability to run test suites written in the Kibana repository using the Functional Test Runner (FTR). The integration is achieved through the Create Environment workflow. When the UI sanity check option is selected, the Create Environment workflow will execute UI sanity checks from the Kibana repository. The main branch of Kibana will be used for checkout.

Running Kibana tests using FTR was implemented as a separate GitHub action kibana-ftr, which is reused in the Create Environment workflow. This action checks out the specified Kibana branch without history, then installs necessary tools and performs a bootstrap. After preparation, FTR is executed with the specified configuration.

Screenshot/Data

This PR introduces two input parameters for the Create Environment workflow: run-ui-sanity-tests flag, which allows you to select whether to run UI sanity tests, and ref_commit_sha, which allows you to choose the appropriate Kibana branch, tag, or commit SHA.

New Step in the workflow

Report Example

Related Issues

Checklist

I have added tests that prove my fix is effective or that my feature works
I have added the necessary README/documentation (if appropriate)

Introducing a new rule?

Generate rule metadata using this script
Add relevant unit tests
Generate relevant rule templates using this script, and open a PR in elastic/packages/cloud_security_posture

mergify · 2024-05-20T14:37:58Z

This pull request does not have a backport label. Could you fix it @gurevichdmitry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

github-actions · 2024-05-20T14:57:11Z

📊 Allure Report - 💚 No failures were reported.

Result	Count
🟥 Failed	0
🟩 Passed	359
⬜ Skipped	33

gurevichdmitry · 2024-05-22T11:33:14Z

.github/workflows/test-environment.yml

+      run-ui-sanity-tests:
+          description: "Run UI sanity tests after provision"
+          default: false
+          type: boolean


I created a new input parameter to provide flexibility in selecting which tests to execute. This can also be combined with a checkbox to run both integration sanity checks and UI sanity checks.

gurevichdmitry · 2024-05-23T07:32:01Z

.github/workflows/test-environment.yml

+          es_version: ${{ env.STACK_VERSION }}
+          ref_commit_sha: ${{ inputs.ref_commit_sha }}


Version control for Kibana UI tests has become relevant because the UI controls implementation can differ between versions. Currently, I assume that the user executing the workflow will select the correct Kibana version and branch, for example, 8.15.0-SNAPSHOT and main, or in the future 8.15.0 and v8.15.0 in Kibana.
I am considering whether we should implement logic to prevent execution if versions do not match correctly. In the current create environment workflow, we allow installing different versions, so we need to decide if we should add additional logic to enforce version consistency, or if we will simply allow the UI tests to fail if an incorrect branch/tag is provided.

oren-zohar · 2024-05-23T14:48:03Z

dev-docs/Cloud-Env-Testing.md

@@ -45,6 +45,10 @@ Follow these steps to run the workflow:

    - **`run-sanity-tests`** (**optional**): Set to `true` to run sanity tests after the environment is set up. Default: `false`

+    - **`run-ui-sanity-tests`** (**optional**): Set to `true` to run Kibana UI sanity tests after the environment is set up. Default: `false`


Maybe link the tests docs here (.github/actions/kibana-ftr/README.md)

Linked to the kibana-ftr action documentation

oren-zohar · 2024-05-23T14:50:27Z

dev-docs/Cloud-Env-Testing.md

@@ -45,6 +45,10 @@ Follow these steps to run the workflow:

    - **`run-sanity-tests`** (**optional**): Set to `true` to run sanity tests after the environment is set up. Default: `false`

+    - **`run-ui-sanity-tests`** (**optional**): Set to `true` to run Kibana UI sanity tests after the environment is set up. Default: `false`
+
+    - **`ref_commit_sha`** (**optional**): Specifies the Kibana branch, tag, or commit SHA to check out for the UI sanity tests, which will be executed after the environment is set up. This should correspond to the version of the `elk-stack-version` provisioned by this workflow. Note that for the current version in development, use Kibana's `main` branch. Default: `main`


field name is too generic IMO, consider making it more clear that it's Kibana's commit, e.g. kbn_ref_commit_sha, also since it's not referring only the commit sha, but also the branch name, maybe more generic on that side, so kibana_ref

changed to kibana_ref

oren-zohar · 2024-05-23T14:52:42Z

.github/workflows/test-environment.yml

-          export CSPM_PUBLIC_IP=$(terraform output -raw ec2_cspm_public_ip)
-          echo "::add-mask::$CSPM_PUBLIC_IP"
-          echo "CSPM_PUBLIC_IP=$CSPM_PUBLIC_IP" >> $GITHUB_ENV
+        run: ../../.ci/scripts/set_environment_output.sh


Is this a generic script that will be used in other workflows? If not, I would use a less generic name

renamed to set_cloud_env_params

oren-zohar · 2024-05-23T14:53:44Z

dev-docs/Cloud-Env-Testing.md

@@ -45,6 +45,10 @@ Follow these steps to run the workflow:

    - **`run-sanity-tests`** (**optional**): Set to `true` to run sanity tests after the environment is set up. Default: `false`

+    - **`run-ui-sanity-tests`** (**optional**): Set to `true` to run Kibana UI sanity tests after the environment is set up. Default: `false`
+
+    - **`ref_commit_sha`** (**optional**): Specifies the Kibana branch, tag, or commit SHA to check out for the UI sanity tests, which will be executed after the environment is set up. This should correspond to the version of the `elk-stack-version` provisioned by this workflow. Note that for the current version in development, use Kibana's `main` branch. Default: `main`


Can you add example supported values for each option? Just to make sure the format is clear for users

added examples for branch, tag and commit SHA.

oren-zohar · 2024-05-23T14:55:52Z

.github/actions/kibana-ftr/README.md

+          ref_commit_sha: ${{ github.event.inputs.ref_commit_sha || 'main' }}
+          test_kibana_url: ${{ secrets.TEST_KIBANA_URL }} # required
+          test_es_url: ${{ secrets.TEST_ES_URL }} # required
+          es_version: 8.15.0-SNAPSHOT # required


why is this hard-coded? Or is it only for the example?

es_version in the README is just an example.

oren-zohar · 2024-05-23T14:56:56Z

.github/actions/kibana-ftr/README.md

+          test_kibana_url: ${{ secrets.TEST_KIBANA_URL }} # required
+          test_es_url: ${{ secrets.TEST_ES_URL }} # required
+          es_version: 8.15.0-SNAPSHOT # required
+          test_browser_headless: '1'


why are we using '1' as a boolean?

This originates from the FTR runner, where setting the environment variable to '1' activates headless mode. However, as per @orouz's suggestion, I'll eliminate this input variable. Since all tests are executed in headless mode on the CI, no further adjustments are necessary.

.github/actions/kibana-ftr/action.yml

gurevichdmitry · 2024-05-27T06:07:51Z

This is the latest run after addressing review comments.

gurevichdmitry added 2 commits May 20, 2024 17:27

add kibana ftr action

2d2808d

Merge branch 'main' into dg-ftr-integration

e5ec296

mergify bot assigned gurevichdmitry May 20, 2024

mergify bot added the backport-skip label May 20, 2024

gurevichdmitry added 11 commits May 21, 2024 15:21

ftr action integration into create env workflow

9e3e5a3

Merge branch 'main' into dg-ftr-integration

372ae5f

update workflow

cfc5b15

update script path

8774745

Merge branch 'main' into dg-ftr-integration

55eee0c

update script and workflow

f3e36c3

add input param for ref_commit_sha

92f4c73

add input param UI sanity tests

60730b9

change params order

14bd60a

update input param description

9030537

update workflow documentation

616efd4

gurevichdmitry commented May 22, 2024

View reviewed changes

gurevichdmitry added 2 commits May 22, 2024 15:01

update picture

5b4cf6e

Merge branch 'main' into dg-ftr-integration

8912f68

gurevichdmitry marked this pull request as ready for review May 22, 2024 12:03

gurevichdmitry requested a review from a team as a code owner May 22, 2024 12:03

gurevichdmitry commented May 23, 2024

View reviewed changes

oren-zohar requested changes May 23, 2024

View reviewed changes

orouz reviewed May 26, 2024

View reviewed changes

gurevichdmitry added 2 commits May 26, 2024 14:27

Merge branch 'main' into dg-ftr-integration

96034e1

fix review comments

7b915ed

gurevichdmitry requested review from oren-zohar and orouz May 26, 2024 14:42

oren-zohar approved these changes May 27, 2024

View reviewed changes

gurevichdmitry merged commit 443cbdb into main May 28, 2024
25 checks passed

gurevichdmitry deleted the dg-ftr-integration branch May 28, 2024 06:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration of basic functional test suite developed with FTR into the Create Environment workflow #2219

Integration of basic functional test suite developed with FTR into the Create Environment workflow #2219

gurevichdmitry commented May 20, 2024 •

edited by oren-zohar

Loading

mergify bot commented May 20, 2024

github-actions bot commented May 20, 2024 •

edited

Loading

gurevichdmitry May 22, 2024

gurevichdmitry May 23, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

oren-zohar May 23, 2024

gurevichdmitry May 26, 2024

gurevichdmitry commented May 27, 2024

		es_version: ${{ env.STACK_VERSION }}
		ref_commit_sha: ${{ inputs.ref_commit_sha }}

		@@ -45,6 +45,10 @@ Follow these steps to run the workflow:

		- `run-sanity-tests` (optional): Set to `true` to run sanity tests after the environment is set up. Default: `false`

		- `run-ui-sanity-tests` (optional): Set to `true` to run Kibana UI sanity tests after the environment is set up. Default: `false`

Integration of basic functional test suite developed with FTR into the Create Environment workflow #2219

Integration of basic functional test suite developed with FTR into the Create Environment workflow #2219

Conversation

gurevichdmitry commented May 20, 2024 • edited by oren-zohar Loading

Summary of your changes

Screenshot/Data

Related Issues

Checklist

Introducing a new rule?

mergify bot commented May 20, 2024

github-actions bot commented May 20, 2024 • edited Loading

📊 Allure Report - 💚 No failures were reported.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurevichdmitry commented May 27, 2024

gurevichdmitry commented May 20, 2024 •

edited by oren-zohar

Loading

github-actions bot commented May 20, 2024 •

edited

Loading