diff --git a/CNF_TESTSUITE_YML_USAGE.md b/CNF_TESTSUITE_YML_USAGE.md index c96165c18..e6a5f2459 100644 --- a/CNF_TESTSUITE_YML_USAGE.md +++ b/CNF_TESTSUITE_YML_USAGE.md @@ -4,7 +4,7 @@ The cnf-testsuite.yml is used by `cnf_setup` in order to install the CNF to be tested onto an existing K8s cluster. -The information in the cnf-testsuite.yml is also used for additional configuration of some tests e.g. `allowlist_helm_chart_container_names` is used for exculding containers from the [privileged](https://github.com/cncf/cnf-testsuite/blob/main/src/tasks/workload/security.cr#L196) container test. +The information in the cnf-testsuite.yml is also used for additional configuration of some tests e.g. `allowlist_helm_chart_container_names` is used for exculding containers from the [privileged](https://github.com/cnti-testcatalog/testsuite/blob/main/src/tasks/workload/security.cr#L196) container test. ### Table of Contents @@ -16,7 +16,7 @@ The information in the cnf-testsuite.yml is also used for additional configurati ### Overview of all cnf-testsuite.yml -The following is a basic working example cnf-testsuite.yml file that can be found in the cnf-testsuite respository: [cnf-testsuite.example.yml](https://github.com/cncf/cnf-testsuite/blob/CNF_TESTSUITE_YML%231357/example-cnfs/coredns/cnf-testsuite.yml) +The following is a basic working example cnf-testsuite.yml file that can be found in the cnf-testsuite respository: [cnf-testsuite.example.yml](https://github.com/cnti-testcatalog/testsuite/blob/CNF_TESTSUITE_YML%231357/example-cnfs/coredns/cnf-testsuite.yml) ```yaml= --- @@ -94,7 +94,7 @@ This is the path to the helm chart directory (relative to the location of the cn An example of a helm directory can be found [here](https://github.com/helm/charts/tree/master/stable/coredns). -The PATH is also relative to the location of the cnf-testsuite.yml. So if the cnf-testsuite.yml is in the directory `sample-cnfs/sample_coredns/cnf-testsuite.yml` and helm_directory is set to `helm_directory: chart` the test suite would expect to find the chart under [`sample-cnfs/sample_coredns/chart`](https://github.com/cncf/cnf-testsuite/tree/main/sample-cnfs/sample_coredns/chart) +The PATH is also relative to the location of the cnf-testsuite.yml. So if the cnf-testsuite.yml is in the directory `sample-cnfs/sample_coredns/cnf-testsuite.yml` and helm_directory is set to `helm_directory: chart` the test suite would expect to find the chart under [`sample-cnfs/sample_coredns/chart`](https://github.com/cnti-testcatalog/testsuite/tree/main/sample-cnfs/sample_coredns/chart) Example Setting: @@ -117,9 +117,9 @@ Example Setting: This is the path to a directory of manifest files for installing the cnf (relative to the location of the cnf-testsuite.yml). This, or [helm_chart](#helm_chart), or [helm_directory](#helm_directory) must be set, but only one **(mutually exclusive)**. This argument is used by cnf_setup in order to deploy the CNF being tested onto an existing K8s cluster. -An example of a manifest directory can be found [here](https://github.com/cncf/cnf-testsuite/tree/main/sample-cnfs/sample_nonroot/manifests). +An example of a manifest directory can be found [here](https://github.com/cnti-testcatalog/testsuite/tree/main/sample-cnfs/sample_nonroot/manifests). -The PATH is also relative to the location of the cnf-testsuite.yml. So if the cnf-testsuite.yml is in the directory `sample-cnfs/sample_nonroot/cnf-testsuite.yml` and manifest_directory is set to `manifest_directory: manifests` the test suite would expect to find the manifest files under [`sample-cnfs/sample_nonroot/manifests`](https://github.com/cncf/cnf-testsuite/tree/main/sample-cnfs/sample_nonroot/manifests) +The PATH is also relative to the location of the cnf-testsuite.yml. So if the cnf-testsuite.yml is in the directory `sample-cnfs/sample_nonroot/cnf-testsuite.yml` and manifest_directory is set to `manifest_directory: manifests` the test suite would expect to find the manifest files under [`sample-cnfs/sample_nonroot/manifests`](https://github.com/cnti-testcatalog/testsuite/tree/main/sample-cnfs/sample_nonroot/manifests) Example Setting: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 09a1e0d66..3ccb61e52 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -27,10 +27,10 @@ Issues are used as the primary method for tracking items in the CNF Test Suite i ### Issue Templates **1. New Features:** -To request an enhancement, please create a new issue using the [**Feature Request**](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=enhancement&template=feature-request.md&title=%5BFeature%5D) Template +To request an enhancement, please create a new issue using the [**Feature Request**](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=enhancement&template=feature-request.md&title=%5BFeature%5D) Template **2. Report Bugs:** -To report a bug, please create a new issue using the [**Bug Report**](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=bug&template=bug-report.md&title=%5BBUG%5D) Template. Check out [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html.). +To report a bug, please create a new issue using the [**Bug Report**](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=bug&template=bug-report.md&title=%5BBUG%5D) Template. Check out [How to Report Bugs Effectively](https://www.chiark.greenend.org.uk/~sgtatham/bugs.html.). NOTE: To help with debugging, you can enable higher logging level output via the command line or env var @@ -55,11 +55,11 @@ Check [usage documentation](USAGE.md) for more info about invoking commands and **3. New CNF Test Suite Tests:** -- To request a new workload test, please create a new issue using the [**New Workload Test**](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=workload&template=new-workload-test.md&title=%5BWorkload%5D) Template -- To request a new platform test, please create a new issue using the [**New Platform Test**](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=platform&template=new-platform-test.md&title=%5BPlatform%5D) Template +- To request a new workload test, please create a new issue using the [**New Workload Test**](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=workload&template=new-workload-test.md&title=%5BWorkload%5D) Template +- To request a new platform test, please create a new issue using the [**New Platform Test**](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=platform&template=new-platform-test.md&title=%5BPlatform%5D) Template **4. New CNF Example:** -To suggest a new CNF, please create a GitHub issue using the [New Example CNF template](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=example+CNF&template=new-example-cnf.md&title=%5BCNF%5D). +To suggest a new CNF, please create a GitHub issue using the [New Example CNF template](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=example+CNF&template=new-example-cnf.md&title=%5BCNF%5D). To install the CNF Test Suite and run a CNF, follow instructions at: @@ -75,7 +75,7 @@ The test framework and tests (using upstream tools) are written in the human rea Outline of what a contributor's workflow looks like: -1. Fork it (https://github.com/cncf/cnf-testsuite/fork) +1. Fork it (https://github.com/cnti-testcatalog/testsuite/fork) 1. Create a branch from where you want to base your work (usually main). Example `git checkout -b my-new-feature)` 1. Read the [INSTALL.md](install for build and test instructions) 1. Make your changes and arrange them in readable commits. @@ -109,7 +109,7 @@ For general advice on how to submit a pull request, please see [Creating a pull `git clone git@github.com:/cnf-testsuite.git` `cd cnf-testsuite` 3. Add the original cnf-testsuite repo. - `git remote add cncf git@github.com:cncf/cnf-testsuite.git` + `git remote add cncf git@github.com:cnti-testcatalog/testsuite.git` 4. Checkout the pull request's branch. `git checkout ` 5. Push the branch to the original cnf-testsuite repo. diff --git a/CONTRIBUTION_LADDER.md b/CONTRIBUTION_LADDER.md index be3214351..0b3989e49 100644 --- a/CONTRIBUTION_LADDER.md +++ b/CONTRIBUTION_LADDER.md @@ -25,7 +25,7 @@ How you can learn more? * Read the [install guide](INSTALL.md), [usage guide](USAGE.md), [FAQ](FAQ.md), and other documentation * Request a live demonstration and introduction of the test suite ([book a time on Calendly](https://calendly.com/cnftestsuite)) -* Come chat with us in [Slack](https://cloud-native.slack.com/archives/C014TNCEX8R) or start a [discussion](https://github.com/cncf/cnf-testsuite/discussions) +* Come chat with us in [Slack](https://cloud-native.slack.com/archives/C014TNCEX8R) or start a [discussion](https://github.com/cnti-testcataglog/testsuite/discussions) ## Community Participant @@ -40,7 +40,7 @@ How you can be more involved and participate in the community: * Helping other users * Commenting on issues * Trying out new releases -* Attending [community meetings](https://github.com/cncf/cnf-testsuite#communication-and-community-meetings) and events +* Attending [community meetings](https://github.com/cnti-testcatalog/testsuite#communication-and-community-meetings) and events ## Contributor diff --git a/EXAMPLE-CNFS.md b/EXAMPLE-CNFS.md index fc49abb82..703d6b58e 100644 --- a/EXAMPLE-CNFS.md +++ b/EXAMPLE-CNFS.md @@ -13,7 +13,7 @@ This is a preliminary list of CNF samples for each layer in the [OSI model](http ## [Layer 7 - Application](https://en.wikipedia.org/wiki/Application_layer) -- [CoreDNS Sample CNF](https://github.com/cncf/cnf-testsuite/tree/main/sample-cnfs/sample-coredns-cnf) +- [CoreDNS Sample CNF](https://github.com/cnti-testcatalog/testsuite/tree/main/sample-cnfs/sample-coredns-cnf) - [NFF Go Deep Packet Inspection example](https://github.com/intel-go/nff-go/tree/master/examples/dpi) example ## [Layer 6 - Presentation](https://en.wikipedia.org/wiki/Presentation_layer) diff --git a/FAQ.md b/FAQ.md index 5e9986b0a..7df187315 100644 --- a/FAQ.md +++ b/FAQ.md @@ -37,7 +37,7 @@
If I found a bug or I think it's a bug, how do I report it?

-- If you would like to report a bug, please create a [new issue](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=bug&template=bug-report.md&title=%5BBUG%5D) (using the **Bug Report** Template). +- If you would like to report a bug, please create a [new issue](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=bug&template=bug-report.md&title=%5BBUG%5D) (using the **Bug Report** Template).

@@ -45,7 +45,7 @@
How do I request a new feature?

-- If you would like to request an enhancement, please create a [new issue](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=enhancement&template=feature-request.md&title=%5BFeature%5D) (using the **Feature Request** Template). +- If you would like to request an enhancement, please create a [new issue](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=enhancement&template=feature-request.md&title=%5BFeature%5D) (using the **Feature Request** Template).

@@ -53,7 +53,7 @@
Can I request a new workload or platform test for the CNF Test Suite?

-- Yes. If you would like to request a new workload test, please create a [new issue](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=workload&template=new-workload-test.md&title=%5BWorkload%5D) (using the **New Workload Test** Template) or create a [new issue](https://github.com/cncf/cnf-testsuite/issues/new?assignees=&labels=platform&template=new-platform-test.md&title=%5BPlatform%5D) (using the **New Platform Test** Template). +- Yes. If you would like to request a new workload test, please create a [new issue](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=workload&template=new-workload-test.md&title=%5BWorkload%5D) (using the **New Workload Test** Template) or create a [new issue](https://github.com/cnti-testcatalog/testsuite/issues/new?assignees=&labels=platform&template=new-platform-test.md&title=%5BPlatform%5D) (using the **New Platform Test** Template).

@@ -79,7 +79,7 @@
How are points assigned for tests?

-- Points are different for each test and workload but in general terms, pass defaults to 5 and fail is a -1. See [points.yml](https://github.com/cncf/cnf-testsuite/blob/main/embedded_files/points.yml) for more details on the different points for default scoring. +- Points are different for each test and workload but in general terms, pass defaults to 5 and fail is a -1. See [points.yml](https://github.com/cnti-testcatalog/testsuite/blob/main/embedded_files/points.yml) for more details on the different points for default scoring.

diff --git a/INSTALL.md b/INSTALL.md index 9ad9a3c38..62f2072be 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -76,7 +76,7 @@ cd tools/ && git clone https://github.com/crosscloudci/k8s-infra.git We support the following methods of installing the cnf-testsuite: - [Curl installation](#Curl-Binary-Installation) (via latest binary release) -- [Latest Binary](https://github.com/cncf/cnf-testsuite/releases/latest) (manual download) +- [Latest Binary](https://github.com/cnti-testcatalog/testsuite/releases/latest) (manual download) - From [**Source**](#Source-Install) on github. - [Air Gapped](#Air-Gapped) @@ -88,7 +88,7 @@ There are two methods to install via curl, we prefer the first method (the other - This first command using curl will download, install, and export the path automatically (recommended method): ``` -source <(curl -s https://raw.githubusercontent.com/cncf/cnf-testsuite/main/curl_install.sh) +source <(curl -s https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/curl_install.sh) ```
Click here for the alternate curl and manual install method @@ -96,12 +96,12 @@ source <(curl -s https://raw.githubusercontent.com/cncf/cnf-testsuite/main/curl_ - The other curl method to download and install requires you to export the PATH to the location of the executable: ``` -curl -s https://raw.githubusercontent.com/cncf/cnf-testsuite/main/curl_install.sh | bash +curl -s https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/curl_install.sh | bash ``` - The Latest Binary (or you can select a previous release if desired) can be pulled down with wget, curl or you're own preferred method. Once downloaded you'll need to make the binary executable and manually add to your path: ``` -wget https://github.com/cncf/cnf-testsuite/releases/download/latest/latest.tar.gz +wget https://github.com/cnti-testcatalog/testsuite/releases/download/latest/latest.tar.gz tar xzf latest.tar.gz cd cnf-testsuite chmod +x cnf-testsuite @@ -120,7 +120,7 @@ This is a brief summary for source installations and [does have requirements](#R Follow these steps to checkout the source from github and compile a cnf-testsuite binary: ``` -git clone https://github.com/cncf/cnf-testsuite.git +git clone https://github.com/cnti-testcatalog/testsuite.git cd cnf-testsuite/ shards install crystal build src/cnf-testsuite.cr @@ -209,14 +209,14 @@ Now cnf-testsuite is setup, we're ready to configure it to point at a CNF to tes - If you want to use an example CNF, you can download our CoreDNS example CNF by doing the following: ``` -wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cncf/cnf-testsuite/main/example-cnfs/coredns/cnf-testsuite.yml +wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/example-cnfs/coredns/cnf-testsuite.yml ``` - The wget gets a working config file, now tell cnf-testsuite to use it by doing the following: ``` cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml ``` -- There are other examples in the [examples-cnfs](https://github.com/cncf/cnf-testsuite/tree/master/example-cnfs) directory that can be used for testing as well. +- There are other examples in the [examples-cnfs](https://github.com/cnti-testcatalog/testsuite/tree/master/example-cnfs) directory that can be used for testing as well. #### Bring Your Own CNF diff --git a/KIND-INSTALL.md b/KIND-INSTALL.md index a39c36450..227012dd7 100644 --- a/KIND-INSTALL.md +++ b/KIND-INSTALL.md @@ -56,4 +56,4 @@ for node in $(kind get nodes); do done ``` -Now you can use CNFs that require Multus and CNIs, e.g. [examples/ip-forwarder](https://github.com/cncf/cnf-testsuite/tree/main/example-cnfs/ip-forwarder) +Now you can use CNFs that require Multus and CNIs, e.g. [examples/ip-forwarder](https://github.com/cnti-testcatalog/testsuite/tree/main/example-cnfs/ip-forwarder) diff --git a/README.md b/README.md index 738dd33b2..50e01b439 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ | Main | | ------------------------------------------------------------------------------------------------------------------------------------------- | -| [![Build Status](https://github.com/cncf/cnf-testsuite/workflows/Crystal%20Specs/badge.svg)](https://github.com/cncf/cnf-testsuite/actions) | +| [![Build Status](https://github.com/cnti-testcatalog/gtestsuite/workflows/Crystal%20Specs/badge.svg)](https://github.com/cnti-testcatalog/testsuite/actions) | The CNTI Test Catalog is a tool that validates telco application's adherence to [cloud native principles](https://networking.cloud-native-principles.org/) and best practices. @@ -16,9 +16,9 @@ To get the CNTI Test Catalog up and running, see the [Installation Guide](INSTAL Prereqs: kubernetes cluster, wget, curl, helm 3.1.1 or greater on your system already. -1. Install the latest test suite binary: `source <(curl -s https://raw.githubusercontent.com/cncf/cnf-testsuite/main/curl_install.sh)` +1. Install the latest test suite binary: `source <(curl -s https://raw.githubusercontent.com/cnti-testcatalog/gtestsuite/main/curl_install.sh)` 2. Run `setup` to prepare the cnf-testsuite: `cnf-testsuite setup` -3. Pull down an example CNF configuration to try: `curl -o cnf-testsuite.yml https://raw.githubusercontent.com/cncf/cnf-testsuite/main/example-cnfs/coredns/cnf-testsuite.yml` +3. Pull down an example CNF configuration to try: `curl -o cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/gtestsuite/main/example-cnfs/coredns/cnf-testsuite.yml` 4. Initialize the test suite for using the CNF: `cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml` 5. Run all of application/workload tests: `cnf-testsuite workload` @@ -45,7 +45,7 @@ See the [Test Categories Documentation](TEST-CATEGORIES.md) for a complete overv Welcome! We gladly accept contributions on new tests, example CNFs, updates to documentation, enhancements, bug reports, and more. - [Contributing guide](CONTRIBUTING.md) -- [Good first issues](https://github.com/cncf/cnf-testsuite/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) +- [Good first issues](https://github.com/cnti-testcatalog/testsuite/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) ## Communication and community meetings @@ -66,11 +66,11 @@ Welcome! We gladly accept contributions on new tests, example CNFs, updates to d **CNF Test Suite Demo** - [Recording](https://drive.google.com/file/d/1SBHE5Dqx6Sa-m83WODbCEbbdiB2_l_U2/view?usp=sharing) -- [Slides](https://github.com/cncf/cnf-testsuite/files/6857515/SHARED-COMMON.CNF.Test.Suite.Demo.and.CNF.initiatives.overview.2021-06-29.pdf) (PDF) +- [Slides](https://github.com/cnti-testcatalog/testsuite/files/6857515/SHARED-COMMON.CNF.Test.Suite.Demo.and.CNF.initiatives.overview.2021-06-29.pdf) (PDF) **Crystal in the Cloud: A cloud native journey at Crystal 1.0 Conference** - [Recording](https://youtu.be/n8g60VglyUw) -- [Slides](https://github.com/cncf/cnf-testsuite/files/6785788/Crystal.1.0.Crystal.in.the.Cloud_.CNF.Test.Suite.pdf) (PDF) +- [Slides](https://github.com/cnti-testcatalog/testsuite/files/6785788/Crystal.1.0.Crystal.in.the.Cloud_.CNF.Test.Suite.pdf) (PDF) ## Implementation overview @@ -79,7 +79,7 @@ The CNF Test Suite leverages upstream tools such as [OPA Gatekeeper](https://git The test framework and tests (using the upstream tools) are written in the human-readable, compiled language, [Crystal](https://crystal-lang.org/). Common capabilities like dependencies between tests and categories are supported. -Setup of vanilla upstream K8s on [Equinix Metal](https://metal.equinix.com/) is done with the [CNF Testbed](https://github.com/cncf/cnf-testbed/) platform tool chain, which includes [k8s-infra](https://github.com/crosscloudci/k8s-infra), [Kubespray](https://kubespray.io/). To add support for other providers, please submit a [Pull Request](https://github.com/cncf/cnf-testbed/pulls) to the [CNF Testbed](https://github.com/cncf/cnf-testbed/) repo. +Setup of vanilla upstream K8s on [Equinix Metal](https://metal.equinix.com/) is done with the [CNF Testbed](https://github.com/cnti-testcatalog/gtestbed/) platform tool chain, which includes [k8s-infra](https://github.com/crosscloudci/k8s-infra), [Kubespray](https://kubespray.io/). To add support for other providers, please submit a [Pull Request](https://github.com/cncf/cnf-testbed/pulls) to the [CNF Testbed](https://github.com/cncf/cnf-testbed/) repo. ## Code of Conduct diff --git a/RELEASE.md b/RELEASE.md index 7fb8a6ff0..feac61bfa 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -10,7 +10,7 @@ git push --tags ``` - Wait for github actions to complete the build -- Go to https://github.com/cncf/cnf-testsuite/releases +- Go to https://github.com/cnti-testcatalog/testsuite/releases - Locate the draft release for the build - Modify the release notes to reflect the contents for the release - Mark the release as non-draft @@ -66,7 +66,7 @@ _Note: this covers both workload (ie. application) and platform tests_ **[Manually create builds]** based on [SOURCE_INSTALL.md#optional-build-binary](SOURCE_INSTALL.md#installation) and [Minimal instructions to run the tests from source (as of 2020-06-23)](https://hackmd.io/hcHoJEKaRWuyf_fZ7ITxLw) -- Download source: `git clone https://github.com/cncf/cnf-testsuite.git` +- Download source: `git clone https://github.com/cnti-testcatalog/testsuite.git` - `cd cnf-testsuite` - Install dependencies: `shards install` - Create a static binary: `crystal build src/cnf-testsuite.cr --release --static --link-flags "-lxml2 -llzma"` diff --git a/ROADMAP.md b/ROADMAP.md index b1e08ac01..a3ac628fe 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -5,21 +5,21 @@ This document defines a high level roadmap for the CNF Test Suite. The following is a selection of some of the major features the CNF Test Suite team plans to explore. This roadmap will continue to be updated as priorities evolve. -To get a more complete overview of planned features and current work see the [project board](https://github.com/cncf/cnf-testsuite/projects/1), [issue tracker](https://github.com/cncf/cnf-testsuite/issues) and [milestones](https://github.com/cncf/cnf-testsuite/milestones) in GitHub. +To get a more complete overview of planned features and current work see the [project board](https://github.com/cnti-testcatalog/testsuite/projects/1), [issue tracker](https://github.com/cnti-testcatalog/testsuite/issues) and [milestones](https://github.com/cnti-testcatalog/testsuite/milestones) in GitHub. ### Create tests - Build tests for Kubernetes best practices that address issues voiced by the end users, including: - On-boarding (day 1) items - CNF WG best practices -- Build [resilience tests](https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md#resilience-tests) using [LitmusChaos](https://litmuschaos.io/) experiments -- Create [observability tests](https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md#observability-tests) to check for cloud native monitoring -- Create [state tests](https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md#state-tests) to check cloud native data handling -- Create [security tests](https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md#security-tests) +- Build [resilience tests](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md#resilience-tests) using [LitmusChaos](https://litmuschaos.io/) experiments +- Create [observability tests](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md#observability-tests) to check for cloud native monitoring +- Create [state tests](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md#state-tests) to check cloud native data handling +- Create [security tests](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md#security-tests) ### Enhance the functionality of the test suite framework -- Add support for [air gapped](https://github.com/cncf/cnf-testsuite/labels/air-gapped) environments +- Add support for [air gapped](https://github.com/cnti-testcatalog/testsuite/labels/air-gapped) environments - Add best practice suggestions to test results - Split libraries out into different repositories under a single organization @@ -28,7 +28,7 @@ To get a more complete overview of planned features and current work see the [pr - Document a Governance structure for maintainers - Document a Contributor Ladder for maintainer levels - Publish a "Getting Started with the CNF Test Suite" blog post -- Create "Office Hours" and [good first issues](https://github.com/cncf/cnf-testsuite/labels/good%20first%20issue) to help beginners +- Create "Office Hours" and [good first issues](https://github.com/cnti-testcatalog/testsuite/labels/good%20first%20issue) to help beginners - Engage CNCF-hosted projects and propose test ideas using their software - Enlist help from other communities - Crystal Lang @@ -40,6 +40,6 @@ To get a more complete overview of planned features and current work see the [pr - Create ADOPTERS.md file to list users of the CNF Test Suite - Set up [Calendly](https://calendly.com/cnftestsuite) to schedule presentations - Offer presentations, demonstrations and assistance on the CNF Test Suite to Service Providers and CNF Developers - - To request a presentation, please open an [issue](https://github.com/cncf/cnf-testsuite/issues/new) or schedule via [Calendly](https://calendly.com/cnftestsuite) + - To request a presentation, please open an [issue](https://github.com/cnti-testcatalog/testsuite/issues/new) or schedule via [Calendly](https://calendly.com/cnftestsuite) - Promote test suite with new groups and communities - Present at Network of the Future (NoF) seminar diff --git a/SOURCE_INSTALL.md b/SOURCE_INSTALL.md index 62b73e10b..df7f8573f 100644 --- a/SOURCE_INSTALL.md +++ b/SOURCE_INSTALL.md @@ -86,7 +86,7 @@ _NOTE: Currently Mac OS X users will need to ensure helm 3.1.1 or greater is ins - cnf-testsuite needs helm-3.1.1 or greater but is optional as the prerequisite checks will install if not found. You can install helm by checking their [installation methods](https://helm.sh/docs/helm/helm_install/). - Checkout the source code with git: ``` - git clone git@github.com:cncf/cnf-testsuite.git + git clone git@github.com:cnti-testcatalog/testsuite.git ``` - Change directory into the source: ``` @@ -173,7 +173,7 @@ To use CoreDNS as an example CNF. Download the testsuite configuration to test C - Make sure you are in your cnf-testsuite/ source repo checkout directory and do the following: ``` - curl -o cnf-testsuite.yml https://raw.githubusercontent.com/cncf/cnf-testsuite/main/example-cnfs/coredns/cnf-testsuite.yml + curl -o cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/example-cnfs/coredns/cnf-testsuite.yml ``` - Prepare the test suite to use the CNF by running: ``` @@ -218,7 +218,7 @@ The following would run only the platform tests: You can also run via `crystal` by replacing the `./cnf-testsuite` with `crystal spec src/cnf-testsuite.cr` and then the argument. -#### More Example Usage (also see the [complete usage documentation](https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md)) +#### More Example Usage (also see the [complete usage documentation](https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md)) ``` # These assume you've already run the cnf_setup pointing at a cnf-testsuite.yml config above. You can always specify your config at the end of each command as well, eg: @@ -276,7 +276,7 @@ protoc -I. --crystal_out . openmetrics_data_model.proto ### Ready to Bring Your Own CNF? -You can check out our [CNF_TESTSUITE_YML_USAGE.md](https://github.com/cncf/cnf-testsuite/blob/main/CNF_TESTSUITE_YML_USAGE.md) document on what is required to bring or use your own CNF. +You can check out our [CNF_TESTSUITE_YML_USAGE.md](https://github.com/cnti-testcatalog/testsuite/blob/main/CNF_TESTSUITE_YML_USAGE.md) document on what is required to bring or use your own CNF. - Follow the [INSTALL](INSTALL.md) or [SOURCE-INSTALL](SOURCE-INSTALL.md) to build the binary. -- Now head over to [CNF_TESTSUITE_YML_USAGE.md](https://github.com/cncf/cnf-testsuite/blob/main/CNF_TESTSUITE_YML_USAGE.md) for more detailed steps. +- Now head over to [CNF_TESTSUITE_YML_USAGE.md](https://github.com/cnti-testcatalog/testsuite/blob/main/CNF_TESTSUITE_YML_USAGE.md) for more detailed steps. diff --git a/USAGE.md b/USAGE.md index 89cf9e216..419330b0e 100644 --- a/USAGE.md +++ b/USAGE.md @@ -163,17 +163,17 @@ crystal bin/ameba.cr ./cnf-testsuite compatibility ``` -## [Increase decrease capacity:](https://github.com/cncf/cnf-testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#increase-decrease-capacity) +## [Increase decrease capacity:](https://github.com/cnti-testcatalog/testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#increase-decrease-capacity) ##### To run both increase and decrease tests, you can use the alias command that calls them both: ``` ./cnf-testsuite increase_decrease_capacity ``` -### [Increase capacity](https://github.com/cncf/cnf-testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#increase-capacity) +### [Increase capacity](https://github.com/cnti-testcatalog/testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#increase-capacity) ##### Or, they can be called individually using the following commands: ``` ./cnf-testsuite increase_capacity ``` -### [Decrease capacity](https://github.com/cncf/cnf-testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#decrease-capacity) +### [Decrease capacity](https://github.com/cnti-testcatalog/testsuite/blob/refactor_usage_doc%231371/docs/LIST_OF_TESTS.md#decrease-capacity) ``` ./cnf-testsuite decrease_capacity diff --git a/docs/LIST_OF_TESTS.md b/docs/LIST_OF_TESTS.md index 6ed4b25c9..3a87e2796 100644 --- a/docs/LIST_OF_TESTS.md +++ b/docs/LIST_OF_TESTS.md @@ -17,7 +17,7 @@ List of Workload Tests -## [Increase decrease capacity:](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L168) +## [Increase decrease capacity:](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L168) The increase and decrease capacity tests: HPA (horizonal pod autoscale) will autoscale replicas to accommodate when there is an increase of CPU, memory or other configured metrics to prevent disruption by allowing more requests by balancing out the utilisation across all of the pods. @@ -46,13 +46,13 @@ You can read more about horizonal pod autoscaling to create replicas [here](http -### [Increase capacity](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L184) +### [Increase capacity](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L184) - Expectation: The number of replicas for a Pod increases **What's tested:** The pod is increased and replicated to 3 for the CNF image or release being tested. -### [Decrease capacity](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L213) +### [Decrease capacity](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L213) - Expectation: The number of replicas for a Pod decreases **What's tested:** After `increase_capacity` increases the replicas to 3, it decreases back to 1. @@ -64,7 +64,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-test-the-increasing-and-decreasing-of-capacity-increase_decrease_capacity) -## [Helm chart published](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L406) +## [Helm chart published](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L406) - Expectation: The Helm chart is published in a Helm Repsitory. **What's tested:** Checks if the helm chart is found in a remote repository when running [`helm search`](https://helm.sh/docs/helm/helm_search_repo/). @@ -74,7 +74,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-helm-chart-is-published-helm_chart_published) -## [Helm chart valid](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L449) +## [Helm chart valid](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L449) - Expectation: No syntax or validation problems are found in the chart. **What's tested:** Checks the syntax & validity of the chart using [`helm lint`](https://helm.sh/docs/helm/helm_lint/) @@ -94,7 +94,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-helm-deploys-helm_deploy) -## [Rollback:](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L87) +## [Rollback:](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L87) - Expectation: The CNF Software version can be successfully incremented, then rolled back. **What's tested:** Checks if the Pod can be upgraded to a new software version, then restored back to the orginal software version by using the [Kubectl Set Image](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-image-em-) & [Kubectl Rollout Undo](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#rollout) commands. @@ -104,7 +104,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-a-cnf-version-can-be-rolled-back-rollback) -### [Rolling update](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) +### [Rolling update](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) - Expectation: The CNF Software version can be successfully incremented. **What's tested:** Checks if the Pod can be upgraded to a new software version by using the [Kubectl Set Image](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-image-em-) @@ -112,7 +112,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-the-cnf-can-perform-a-rolling-update-rolling_update) -### [Rolling version change](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) +### [Rolling version change](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) - Expectation: The CNF Software version is successfully rolled back to its original version. **What's tested:** Checks if the Pod can be rolled back to the original software version by using the [Kubectl Set Image](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-image-em-) to perform a rollback. @@ -120,7 +120,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-a-cnf-version-can-be-downgraded-through-a-rolling_version_change-rolling_version_change) -### [Rolling downgrade](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) +### [Rolling downgrade](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L8) - Expectation: The CNF Software version is successfully downgraded to a software version older than the orginal installation version. **What's tested:** Checks if the Pod can be rolled back older software version(Older than the original software version) by using the [Kubectl Set Image](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-image-em-) to perform a downgrade. @@ -130,7 +130,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http -## [CNI compatible](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L588) +## [CNI compatible](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/compatibility.cr#L588) - Expectation: CNF should be compatible with multiple and different CNIs **What's tested:** This installs temporary kind clusters and will test the CNF against both Calico and Cilium CNIs. @@ -141,7 +141,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http -## [Kubernetes Alpha APIs - Proof of Concept](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L499) +## [Kubernetes Alpha APIs - Proof of Concept](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L499) - Expectation: CNF should not use Kubernetes alpha APIs **What's tested:** This checks if a CNF uses alpha or unstable versions of Kubernetes APIs @@ -153,7 +153,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http # Microservice Category -## [Reasonable image size](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L200) +## [Reasonable image size](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L200) - Expectation: CNF image size is under 5 gigs **What's tested:** Checks the size of the image used. @@ -163,7 +163,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-has-a-reasonable-image-size-reasonable_image_size) -## [Reasonable startup time](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L109) +## [Reasonable startup time](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L109) - Expectation: CNF starts up under one minute **What's tested:** Checks how long the it takes for the CNF to pass a Readiness Probe and reach a ready/running state. @@ -173,7 +173,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-have-a-reasonable-startup-time-reasonable_startup_time) -## [Single process type in one container](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L300) +## [Single process type in one container](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L300) - Expectation: CNF container has one process type **What's tested:** This verifies that there is only one process type within one container. This does not count against child processes. Example would be nginx or httpd could have a parent process and then 10 child processes but if both nginx and httpd were running, this test would fail. @@ -183,7 +183,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-has-multiple-process-types-within-one-container-single_process_type) -## [Service discovery](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L405) +## [Service discovery](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L405) - Expectation: CNFs accessible to other applications should be exposed via a Service. **What's tested:** This tests and checks if the containers within a CNF have services exposed via a Kubernetes Service resource. Application access for microservices within a cluster should be exposed via a Service. Read more about K8s Service [here](https://kubernetes.io/docs/concepts/services-networking/service/). @@ -193,7 +193,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-exposes-any-of-its-containers-as-a-service-service_discovery-service_discovery) -## [Shared database](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L26) +## [Shared database](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L26) - Expectation: Multiple microservices should not share the same database. **What's tested:** This tests if multiple CNFs are using the same database. @@ -202,7 +202,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-uses-a-shared-database-shared_database) -## [Specialized Init Systems](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L462) +## [Specialized Init Systems](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/microservice.cr#L462) - Expectation: Container images should use specialized init systems for containers. **What's tested:** This tests if containers in pods have dumb-init, tini or s6-overlay as init processes. @@ -211,7 +211,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http # State Category -## [Node drain](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L209) +## [Node drain](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L209) - Expectation: All workload resources are successfully rescheduled onto other available node(s). **What's tested:** A node is drained and workload resources rescheduled to another node, passing with a liveness and readiness check. This will skip when the cluster only has a single node. @@ -221,7 +221,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-node-drain-occurs-node_drain) -## [Volume hostpath not found](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L419) +## [Volume hostpath not found](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L419) - Expectation: Volume host path configurations should not be used. **What's tested:** This tests if volume host paths are configured and used by the CNF. @@ -231,7 +231,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-the-cnf-uses-a-volume-host-path-volume_hostpath_not_found) -## [No local volume configuration](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L457) +## [No local volume configuration](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L457) - Expectation: Local storage should not be used or configured. **What's tested:** This tests if local volumes are being used for the CNF. @@ -241,7 +241,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-the-cnf-uses-local-storage-no_local_volume_configuration) -## [Elastic volumes](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L321) +## [Elastic volumes](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L321) - Expectation: Elastic persistent volumes should be configured for statefulness. **What's tested:** This checks for elastic persistent volumes in use by the CNF. @@ -251,7 +251,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-the-cnf-uses-elastic-volumes-elastic_volumes) -## [Database persistence](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L358) +## [Database persistence](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/state.cr#L358) - Expectation: Elastic volumes and or statefulsets should be used for databases to maintain a minimum resilience level in K8s clusters. **What's tested:** This checks if elastic volumes and stateful sets are used for MySQL databases. If no MySQL database is found, the test is skipped. @@ -263,7 +263,7 @@ You can read more about horizonal pod autoscaling to create replicas [here](http # Reliability, Resilience and Availability Category -## [CNF under network latency](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L231) +## [CNF under network latency](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L231) - Expectation: The CNF should continue to function when network latency occurs **What's tested:** [This experiment](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-network-latency/) causes network degradation without the pod being marked unhealthy/unworthy of traffic by kube-proxy (unless you have a liveness probe of sorts that measures latency and restarts/crashes the container). The idea of this experiment is to simulate issues within your pod network OR microservice communication across services in different availability zones/regions etc. @@ -276,7 +276,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack -## [CNF with host disk fill](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L390) +## [CNF with host disk fill](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L390) - Expectation: The CNF should continue to function when disk fill occurs and pods should not be evicted to another node. **What's tested:** [This experiment](https://litmuschaos.github.io/litmus/experiments/categories/pods/disk-fill/) stresses the disk with continuous and heavy IO to cause degradation in the shared disk. This experiment also reduces the amount of scratch space available on a node which can lead to a lack of space for newer containers to get scheduled. This can cause (Kubernetes gives up by applying an "eviction" taint like "disk-pressure") a wholesale movement of all pods to other nodes. @@ -286,7 +286,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-disk-fill-occurs-disk_fill) -## [Pod delete](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L441) +## [Pod delete](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L441) - Expectation: The CNF should continue to function when pod delete occurs **What's tested:** [This experiment](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-delete/) helps to simulate such a scenario with forced/graceful pod failure on specific or random replicas of an application resource and checks the deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application. @@ -296,7 +296,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-pod-delete-occurs-pod_delete) -## [Memory hog](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L495) +## [Memory hog](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L495) - Expectation: The CNF should continue to function when pod memory hog occurs **What's tested:** The [pod-memory hog](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-memory-hog/) experiment launches a stress process within the target container - which can cause either the primary process in the container to be resource constrained in cases where the limits are enforced OR eat up available system memory on the node in cases where the limits are not specified. @@ -307,7 +307,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack -## [IO Stress](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L549) +## [IO Stress](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L549) - Expectation: The CNF should continue to function when pod io stress occurs **What's tested:** The [pod-io stress](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-io-stress/) experiment the disk with continuous and heavy IO to cause degradation in reads/writes by other microservices that use this shared disk. @@ -318,7 +318,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack -## [Network corruption](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L284) +## [Network corruption](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L284) - Expectation: The CNF should be resilient to a lossy/flaky network and should continue to provide some level of availability. **What's tested:** The [pod-network corruption](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-network-corruption/) experiment injects packet corruption on the CNF by starting a traffic control (tc) process with netem rules to add egress packet corruption. @@ -328,7 +328,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-corruption-occurs-pod_network_corruption) -## [Network duplication](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L337) +## [Network duplication](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L337) - Expectation: The CNF should continue to function and be resilient to a duplicate network. **What's tested:** The [pod-network duplication](https://litmuschaos.github.io/litmus/experiments/categories/pods/pod-network-duplication/) experiment injects network duplication into the CNF by starting a traffic control (tc) process with netem rules to add egress delays. @@ -337,7 +337,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-pod-network-duplication-occurs-pod_network_duplication) -## [Pod DNS errors](https://github.com/cncf/cnf-testsuite/blob/v0.26.0/src/tasks/workload/reliability.cr#L604) +## [Pod DNS errors](https://github.com/cnti-testcatalog/testsuite/blob/v0.26.0/src/tasks/workload/reliability.cr#L604) - :heavy_check_mark: Added to CNF Test Suite in release v0.26.0 - Expectation: That the CNF dosen't crash is resilient to DNS resolution failures. @@ -348,7 +348,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#test-if-the-cnf-crashes-when-dns-errors-occur-pod_dns_errors) -## [Helm chart liveness entry](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L15) +## [Helm chart liveness entry](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L15) - Expectation: The Helm chart should have a liveness probe configured. **What's tested:** This test scans all of the CNFs workload resources and check if a Liveness Probe has been configuered for each container. @@ -358,7 +358,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-there-is-a-liveness-entry-in-the-helm-chart-liveness) -## [Helm chart readiness entry](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L45) +## [Helm chart readiness entry](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/reliability.cr#L45) - Expectation: The Helm chart should have a readiness probe configured. **What's tested:** This test scans all of the CNFs workload resources and check if a Readiness Probe has been configuered for each container. @@ -370,7 +370,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack # Observability and Diagnostic Category -## [Use stdout/stderr for logs](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L13) +## [Use stdout/stderr for logs](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L13) - Expectation: Resource output logs should be sent to STDOUT/STDERR **What's tested:** This checks and verifies that STDOUT/STDERR logging is configured for the CNF. @@ -379,7 +379,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-logs-are-being-sent-to-stdoutstderr-standard-out-standard-error-instead-of-a-log-file-log_output) -## [Prometheus installed](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L42) +## [Prometheus installed](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L42) - Expectation: The CNF is configured and sending metrics to a Prometheus server. **What's tested:** Tests for the presence of [Prometheus](https://prometheus.io/) and if the CNF configured to sent metrics to the prometheus server. @@ -389,7 +389,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-prometheus-is-installed-and-configured-for-the-cnf-prometheus_traffic) -## [Routed logs](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L170) +## [Routed logs](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L170) - Expectation: Fluentd or FluentBit is installed and capturing logs for the CNF. **What's tested:** Checks for presence of a Unified Logging Layer and if the CNFs logs are being captured by the Unified Logging Layer. fluentd and fluentbit are currently supported. @@ -399,7 +399,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-logs-and-data-are-being-routed-through-a-unified-logging-layer-routed_logs) -## [OpenMetrics compatible](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L146) +## [OpenMetrics compatible](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L146) - Expectation: CNF should emit OpenMetrics compatible traffic. **What's tested:** Checks if the CNFs metrics are [OpenMetrics](https://openmetrics.io/) compliant. @@ -408,7 +408,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-openmetrics-is-being-used-and-or-compatible-open_metrics) -## [Jaeger tracing](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L203) +## [Jaeger tracing](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/observability.cr#L203) - Expectation: The CNF is sending traces to Jaeger. **What's tested:** Checks if Jaeger is installed and the CNF is configured to send traces to the Jaeger Server. @@ -420,7 +420,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack # Security Category -## [Container socket mounts](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L51) +## [Container socket mounts](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L51) - :heavy_check_mark: Added to CNF Test Suite in release v0.27.0 - Expectation: Container runtime sockets should not be mounted as volumes @@ -431,7 +431,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-cnf-performs-a-cri-socket-mount-container_sock_mounts) -## [Privileged Containers](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L420) +## [Privileged Containers](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L420) - Expectation: Containers should not run in privileged mode **What's tested:** Checks if any containers are running in privileged mode (using [Kubescape](https://hub.armo.cloud/docs/c-0057)) @@ -441,7 +441,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-there-are-any-privileged-containers-privileged_containers) -## [External IPs](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L31) +## [External IPs](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L31) - :heavy_check_mark: Added to CNF Test Suite in release v0.27.0 - Expectation: A CNF should not run services with external IPs @@ -452,7 +452,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-external-ips-are-used-for-services-external_ips) -## [Selinux Options](https://github.com/cncf/cnf-testsuite/blob/v0.30.1/src/tasks/workload/security.cr#L91) +## [Selinux Options](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.1/src/tasks/workload/security.cr#L91) - Expectation: A CNF should not have any 'seLinuxOptions' configured that allow privilege escalation. **What's tested:** Checks if the CNF has escalatory seLinuxOptions configured. @@ -462,7 +462,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-selinux-has-been-configured-properly-selinux_options) -## [Sysctls](https://github.com/cncf/cnf-testsuite/blob/v0.30.1/src/tasks/workload/security.cr#L39) +## [Sysctls](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.1/src/tasks/workload/security.cr#L39) - Expectation: The CNF should only have "safe" sysctls mechanisms configured, that are isolated from other Pods. **What's tested:** Checks the CNF for usage of non-namespaced sysctls mechanisms that can affect the entire host. @@ -472,7 +472,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-any-pods-in-the-cnf-use-sysctls-with-restricted-values-sysctls) -## [Privilege escalation](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L156) +## [Privilege escalation](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L156) - Expectation: Containers should not allow [privilege escalation](https://bit.ly/C0016_privilege_escalation) **What's tested:** Check that the allowPrivilegeEscalation field in the securityContext of each container is set to false. @@ -482,7 +482,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-any-containers-allow-for-privilege-escalation-privilege_escalation) -## [Symlink file system](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L175) +## [Symlink file system](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L175) - Expectation: No vulnerable K8s version being used in conjunction with the [subPath](https://bit.ly/C0058_symlink_filesystem) feature. **What's tested:** This test checks for vulnerable K8s versions and the actual usage of the subPath feature for all Pods in the CNF. @@ -492,7 +492,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-an-attacker-can-use-a-symlink-for-arbitrary-host-file-system-access-cve-2021-25741-symlink_file_system) -## [Application credentials](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L194) +## [Application credentials](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L194) - Exepectation: Application credentials should not be found in the CNFs configuration files **What's tested:** Checks the CNF for sensitive information in environment variables, by using list of known sensitive key names. Also checks for configmaps with sensitive information. @@ -502,7 +502,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-there-are-applications-credentials-in-configuration-files-application_credentials) -## [Host network](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L213) +## [Host network](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L213) - Expectation: The CNF should not have access to the host systems network. **What's tested:** Checks if there is a [host network](https://bit.ly/C0041_hostNetwork) attached to any of the Pods in the CNF. @@ -512,7 +512,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-there-is-a-host-network-attached-to-a-pod-host_network) -## [Service account mapping](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L232) +## [Service account mapping](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L232) - Expectation: The [automatic mapping](https://bit.ly/C0034_service_account_mapping) of service account tokens should be disabled. **What's tested:** Check if the CNF is using service accounts that are automatically mapped. @@ -522,7 +522,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-there-is-automatic-mapping-of-service-accounts-service_account_mapping) -## [Ingress and Egress blocked](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L335) +## [Ingress and Egress blocked](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L335) - Expectation: [Ingress and Egress traffic should be blocked on Pods](https://bit.ly/3bhT10s). **What's tested:** Checks each Pod in the CNF for a defined ingress and egress policy. @@ -532,7 +532,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-there-is-an-ingress-and-egress-policy-defined-ingress_egress_blocked) -## [Insecure capabilities](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L272) +## [Insecure capabilities](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L272) - Expectation: Containers should not have insecure capabilities enabled. **What's tested:** Checks the CNF for any usage of insecure capabilities using the following [deny list](https://man7.org/linux/man-pages/man7/capabilities.7.html) @@ -542,7 +542,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-for-insecure-capabilities-insecure_capabilities) -## [Non-root containers](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L377) +## [Non-root containers](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L377) - Expectation: Containers should run with non-root user and allowPrivilegeEscalation should be set to false. **What's tested:** Checks if the CNF has runAsUser and runAsGroup set to a user id greater than 999. Also checks that the allowPrivilegeEscalation field is set to false for the CNF. Read more at [ARMO-C0013](https://bit.ly/2Zzlts3) @@ -552,7 +552,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-containers-are-running-with-non-root-user-with-non-root-membership-non_root_containers) -## [Host PID/IPC privileges](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L356) +## [Host PID/IPC privileges](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L356) - Expectation: Containers should not have hostPID and hostIPC privileges **What's tested:** Checks if containers are running with hostPID or hostIPC privileges. Read more at [ARMO-C0038](https://bit.ly/3nGvpIQ) @@ -562,7 +562,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-containers-are-running-with-hostpid-or-hostipc-privileges-host_pid_ipc_privileges) -## [Linux hardening](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L251) +## [Linux hardening](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L251) - Expectation: Security services are being used to harden application. **What's tested:** Check if there are AppArmor, Seccomp, SELinux or Capabilities defined in the securityContext of the CNF's containers and pods. Read more at [ARMO-C0055](https://bit.ly/2ZKOjpJ) @@ -573,7 +573,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack -## [Resource policies](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L314) +## [Resource policies](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L314) - Expectation: Containers should have resource limits defined **What's tested:** Check if there is a ‘limits’ field defined for the CNF. Check for each limitrange/resourcequota if there is a max/hard field defined, respectively. Read more at [ARMO-C0009](https://bit.ly/3Ezxkps). @@ -583,7 +583,7 @@ The applications may stall or get corrupted while they wait endlessly for a pack [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-containers-have-resource-limits-defined-resource_policies) -## [Immutable File Systems](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L441) +## [Immutable File Systems](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L441) - Expectation: Containers should use an immutable file system when possible. **What's tested:** @@ -594,7 +594,7 @@ Checks whether the readOnlyRootFilesystem field in the SecurityContext is set to [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-containers-have-immutable-file-systems-immutable_file_systems) -## [HostPath Mounts](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L462) +## [HostPath Mounts](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/security.cr#L462) - Expectation: Containers should not have hostPath mounts **What's tested:** Checks the CNF's POD spec for any hostPath volumes, if found it checks the volume for the field mount.readOnly == false (or if it doesn’t exist). @@ -607,7 +607,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) # Configuration Category -## [Default namespaces](https://github.com/cncf/cnf-testsuite/blob/v0.30.0/src/tasks/workload/configuration.cr#L56) +## [Default namespaces](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.0/src/tasks/workload/configuration.cr#L56) - Expectation: Resources should not be deployed in the default namespace. **What's tested:** Checks if any of the CNF's resources are deployed in the default namespace. @@ -617,7 +617,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-a-cnf-is-using-the-default-namespace-default_namespace) -## [Latest tag](https://github.com/cncf/cnf-testsuite/blob/v0.30.0/src/tasks/workload/configuration.cr#L79) +## [Latest tag](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.0/src/tasks/workload/configuration.cr#L79) - Expectation: The CNF should use an immutable tag that maps to a symantic version of the application. @@ -628,7 +628,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-mutable-tags-being-used-for-image-versioningusing-kyverno-latest_tag-latest_tag) -## [Require labels](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L18) +## [Require labels](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L18) - :heavy_check_mark: Added to CNF Test Suite in release v0.27.0 - Expectation: Checks if pods are using the 'app.kubernetes.io/name' label @@ -639,7 +639,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-the-recommended-labels-are-being-used-to-describe-resources-required_labels) -## [Versioned tag](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L80) +## [Versioned tag](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L80) - Expectation: The CNF should use an immutable tag that maps to a symantic version of the application. **What's tested:** Checks if the CNF is using a 'latest' tag instead of a semantic version using OPA Gatekeeper. @@ -649,7 +649,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-there-are-versioned-tags-on-all-images-using-opa-gatekeeper-versioned_tag) -## [nodePort not used](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L131) +## [nodePort not used](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L131) - Expectation: The nodePort configuration field is not found in any of the CNF's services. **What's tested:** Checks the CNF for any associated K8s Services that configured to expose the CNF by using a nodePort. @@ -659,7 +659,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-there-are-node-ports-used-in-the-service-configuration-nodeport_not_used) -## [hostPort not used](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L166) +## [hostPort not used](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L166) - Expectation: The hostPort configuration field is not found in any of the defined containers. **What's tested:** Checks the CNF's workload resources for any containers using the hostPort configuration field to expose the application. @@ -669,7 +669,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-there-are-host-ports-used-in-the-service-configuration-hostport_not_used) -## [Hardcoded IP addresses in K8s runtime configuration](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L213) +## [Hardcoded IP addresses in K8s runtime configuration](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L213) - Expectation: That no hardcoded IP addresses or subnet masks are found in the Kubernetes workload resources for the CNF. **What's tested:** The hardcoded ip address test will scan all of the CNF's workload resources and check for any static, hardcoded ip addresses being used in the configuration. @@ -679,7 +679,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-test-if-there-are-any-non-declarative-hardcoded-ip-addresses-or-subnet-masks-in-the-k8s-runtime-configuration-hardcoded_ip_addresses_in_k8s_runtime_configuration) -## [Secrets used](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L257) +## [Secrets used](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L257) - Expectation: The CNF is using K8s secrets for the management of sensitive data. **What's tested:** The secrets used test will scan all the Kubernetes workload resources to see if K8s secrets are being used. @@ -689,7 +689,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-a-cnf-uses-k8s-secrets-secrets_used) -## [Immutable configmap](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L362) +## [Immutable configmap](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/workload/configuration.cr#L362) - Expectation: Immutable configmaps are being used for non-mutable data. **What's tested:** The immutable configmap test will scan the CNF's workload resources and see if immutable configmaps are being used. @@ -701,7 +701,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) # 5g Category -## [smf_upf_core_validator](https://github.com/cncf/cnf-testsuite/blob/v0.30.0/src/tasks/workload/5g_validator.cr#L9) +## [smf_upf_core_validator](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.0/src/tasks/workload/5g_validator.cr#L9) - Expectation: 5g core should continue to function during various CNF tests. **What's tested:** Checks the pfcp heartbeat between the smf and upf to make sure it remains close to baseline. @@ -710,7 +710,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) [**Rationale & Reasoning**](../RATIONALE.md#to-validate-a-5g-core) -## [suci_enabled](https://github.com/cncf/cnf-testsuite/blob/v0.30.0/src/tasks/workload/5g_validator.cr#L20) +## [suci_enabled](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.0/src/tasks/workload/5g_validator.cr#L20) - Expectation: 5g core should use suci concealment. **What's tested:** Checks to see if the 5g core supports suci concealment. @@ -722,7 +722,7 @@ Read more at [ARMO-C0045](https://bit.ly/3EvltIL) # Ran Category -## [oran_e2_connection](https://github.com/cncf/cnf-testsuite/blob/v0.30.0/src/tasks/workload/ran.cr#L10) +## [oran_e2_connection](https://github.com/cnti-testcatalog/testsuite/blob/v0.30.0/src/tasks/workload/ran.cr#L10) - Expectation: An ORAN RIC should use an e2 connection. **What's tested:** Checks if a RIC uses a oran compatible e2 connection. @@ -738,7 +738,7 @@ List of Platform Tests --- -## [K8s Conformance](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/platform.cr#L21) +## [K8s Conformance](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/platform.cr#L21) - Expectation: The K8s cluster passes the K8s conformance tests **What's tested:** Check if your platform passes the K8s conformance test. See https://github.com/cncf/k8s-conformance for details on what is tested. @@ -748,7 +748,7 @@ List of Platform Tests [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-a-cnf-version-uses-immutable-configmaps-immutable_configmap) -## [ClusterAPI enabled](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/platform.cr#L88) +## [ClusterAPI enabled](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/platform.cr#L88) - Expectation: The cluster has Cluster API enabled which manages at least one Node. **What's tested:** Checks the platforms Kubernetes Nodes to see if they were instansiated by ClusterAPI. @@ -758,7 +758,7 @@ List of Platform Tests [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-plateform-is-being-managed-by-clusterapi-clusterapi-enabled) -## [OCI Compliant](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/hardware_and_scheduling.cr#L15) +## [OCI Compliant](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/hardware_and_scheduling.cr#L15) - Expectation: All worker nodes are using an OCI compliant run-time. **What's tested:** Inspects all worker nodes and checks if the run-time being used for scheduling is OCI compliant. @@ -768,7 +768,7 @@ List of Platform Tests [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-plateform-is-using-an-oci-compliant-runtime-oci-compliant) -## (PoC) [Worker reboot recovery](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/resilience.cr#L15) +## (PoC) [Worker reboot recovery](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/resilience.cr#L15) - Expectation: Pods should reschedule after a node failure. - **WARNING**: this is a destructive test and will reboot your _host_ node! Do not run this unless you have completely separate cluster, e.g. development or test cluster. @@ -780,7 +780,7 @@ List of Platform Tests -## [Cluster admin](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L33) +## [Cluster admin](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L33) - Expectation: The [cluster admin role should not be bound to a Pod](https://bit.ly/C0035_cluster_admin) **What's tested:** Check which subjects have cluster-admin RBAC permissions – either by being bound to the cluster-admin clusterrole, or by having equivalent high privileges. @@ -790,7 +790,7 @@ List of Platform Tests [**Rationale & Reasoning**](../RATIONALE.md#to-check-if-the-plateform-has-a-default-cluster-admin-role-cluster-admin) -## [Control plane hardening](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L13) +## [Control plane hardening](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L13) - Expectation: That the the k8s control plane is secure and not hosted on an [insecure port](https://bit.ly/C0005_Control_Plane) **What's tested:** Checks if the insecure-port flag is set for the K8s API Server. @@ -800,7 +800,7 @@ List of Platform Tests [**Rationale & Reasoning**](../RATIONALE.md#check-if-the-plateform-is-using-insecure-ports-for-the-api-server-control_plane_hardening) -## [Dashboard exposed](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L54) +## [Dashboard exposed](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L54) - Expectation: The K8s dashboard should not exposed to the public internet when the software version is older than v2.0.1 **What's tested:** Checks if Kubernetes dashboard exists and exposed externally as a service (nodeport/loadbalancer) and if the software version of the container image is older than v2.0.1. @@ -811,7 +811,7 @@ List of Platform Tests -## [Tiller images](https://github.com/cncf/cnf-testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L75) +## [Tiller images](https://github.com/cnti-testcatalog/testsuite/blob/v0.27.0/src/tasks/platform/security.cr#L75) - Added in release v0.27.0 - Expectation: The platform should be using Helm v3+ without Tiller. diff --git a/src/cnf-testsuite.cr b/src/cnf-testsuite.cr index b3d410710..e963b35a1 100644 --- a/src/cnf-testsuite.cr +++ b/src/cnf-testsuite.cr @@ -110,7 +110,7 @@ end # https://www.thegeekstuff.com/2013/12/bash-completion-complete/ # https://kubernetes.io/docs/tasks/tools/install-kubectl/#enable-kubectl-autocompletion # https://stackoverflow.com/questions/43794270/disable-or-unset-specific-bash-completion -desc "Install Shell Completion: check https://github.com/cncf/cnf-testsuite/blob/main/USAGE.md for usage" +desc "Install Shell Completion: check https://github.com/cnti-testcatalog/testsuite/blob/main/USAGE.md for usage" task "completion" do |_| # assumes bash completion feel free to make a pr for zsh and check an arg for it diff --git a/src/tasks/constants.cr b/src/tasks/constants.cr index a6e6b487e..82c6b6295 100644 --- a/src/tasks/constants.cr +++ b/src/tasks/constants.cr @@ -32,6 +32,7 @@ EmbeddedFileManager.chaos_network_loss EmbeddedFileManager.chaos_cpu_hog EmbeddedFileManager.chaos_container_kill EmbeddedFileManager.points_yml +EmbeddedFileManager.points_yml_write_file EmbeddedFileManager.enforce_image_tag EmbeddedFileManager.constraint_template EmbeddedFileManager.disable_cni diff --git a/src/tasks/workload/configuration.cr b/src/tasks/workload/configuration.cr index b76df3b88..eb24b6836 100644 --- a/src/tasks/workload/configuration.cr +++ b/src/tasks/workload/configuration.cr @@ -287,7 +287,7 @@ task "hostport_not_used" do |_, args| k8s_resource = KubectlClient::Get.resource(resource[:kind], resource[:name], resource[:namespace]) Log.for(testsuite_task).debug { "resource: #{k8s_resource}" } - # per examaple https://github.com/cncf/cnf-testsuite/issues/164#issuecomment-904890977 + # per examaple https://github.com/cnti-testcatalog/testsuite/issues/164#issuecomment-904890977 containers = k8s_resource.dig?("spec", "template", "spec", "containers") Log.for(testsuite_task).debug { "containers: #{containers}" } @@ -585,7 +585,7 @@ task "immutable_configmap" do |_, args| # https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/ # feature test to see if immutable_configmaps are enabled - # https://github.com/cncf/cnf-testsuite/issues/508#issuecomment-758438413 + # https://github.com/cnti-testcatalog/testsuite/issues/508#issuecomment-758438413 test_config_map_filename = "#{destination_cnf_dir}/config_maps/test_config_map.yml"; diff --git a/tools/curl_install_tester_docker_setup/Dockerfile b/tools/curl_install_tester_docker_setup/Dockerfile index a6574f3b8..21f0c33e3 100644 --- a/tools/curl_install_tester_docker_setup/Dockerfile +++ b/tools/curl_install_tester_docker_setup/Dockerfile @@ -20,5 +20,5 @@ from base as ci_test # add binary AND ONLY the binary in this container so we can fail if binary is broken without the rest of the source COPY cnf-testsuite . RUN ./cnf-testsuite setup verbose -RUN wget https://raw.githubusercontent.com/cncf/cnf-testsuite/main/example-cnfs/coredns/cnf-testsuite.yml +RUN wget https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/example-cnfs/coredns/cnf-testsuite.yml RUN ./cnf-testsuite ip_addresses cnf-config=./cnf-testsuite.yml verbose diff --git a/tools/curl_install_tester_docker_setup/README.md b/tools/curl_install_tester_docker_setup/README.md index bd7272110..77abb4bde 100644 --- a/tools/curl_install_tester_docker_setup/README.md +++ b/tools/curl_install_tester_docker_setup/README.md @@ -17,15 +17,15 @@ docker build -t curl_install_tester_docker_setup --target base . docker run --rm -it curl_install_tester_docker_setup -source <(curl https://raw.githubusercontent.com/cncf/cnf-testsuite/main/curl_install.sh) +source <(curl https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/curl_install.sh) cd # to make sure you are in home dir ./cnf-testsuite setup -wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cncf/cnf-testsuite/main/example-cnfs/coredns/cnf-testsuite.yml +wget -O cnf-testsuite.yml https://raw.githubusercontent.com/cnti-testcatalog/testsuite/main/example-cnfs/coredns/cnf-testsuite.yml -./cnf-testsutie cnf_setup cnf-config=./cnf-testsuite.yml +./cnf-testsuite cnf_setup cnf-config=./cnf-testsuite.yml ``` diff --git a/tools/ephemeral_env/README.md b/tools/ephemeral_env/README.md index dba5cbc6b..35d8d8047 100644 --- a/tools/ephemeral_env/README.md +++ b/tools/ephemeral_env/README.md @@ -10,7 +10,7 @@ Prereqs: [Kind Install](../../KIND-INSTALL.md) 1. Obtain the ephemeral binary -- Download the binary from https://github.com/cncf/cnf-testsuite/releases +- Download the binary from https://github.com/cnti-testcatalog/testsuite/releases - OPTIONAL: build the binary yourself using crystal ``` diff --git a/tools/github-runner/cleanup_runners.sh b/tools/github-runner/cleanup_runners.sh index 38c7bb2e9..f45e4489f 100755 --- a/tools/github-runner/cleanup_runners.sh +++ b/tools/github-runner/cleanup_runners.sh @@ -1,11 +1,18 @@ #!/bin/bash +TOKEN=$1 + RUNNERS=( 136.144.55.87 136.144.55.243 139.178.69.151) +if [ -z "$1" ]; then + echo "USAGE: ./cleanup_runners.sh " + exit 1 +fi + #TOKEN=$(curl -XPOST -H "authorization: token ${GITHUB_TOKEN}" https://api.github.com/repos/cncf/cnf-testsuite/actions/runners/registration-token | jq -r .token) for node in "${RUNNERS[@]}"; do diff --git a/tools/github-runner/create_runners.sh b/tools/github-runner/create_runners.sh index 9bd565153..4645adb27 100755 --- a/tools/github-runner/create_runners.sh +++ b/tools/github-runner/create_runners.sh @@ -10,6 +10,13 @@ VIPS=( 147.75.108.64/28 147.75.202.0/28) +TOKEN=$1 + +if [ -z "$1" ]; then + echo "USAGE: sh ./create_runners.sh " + exit 1 +fi + RUNNER_COUNT=0 for node in "${!RUNNERS[@]}"; do diff --git a/tools/github-runner/delete-runners-on-github.sh b/tools/github-runner/delete-runners-on-github.sh index f80a5582f..1e864bf09 100644 --- a/tools/github-runner/delete-runners-on-github.sh +++ b/tools/github-runner/delete-runners-on-github.sh @@ -10,9 +10,9 @@ # curl -u $GH_USER:$GH_TOKEN -H "Accept: application/vnd.github.v3+json" \ - https://api.github.com/repos/cncf/cnf-testsuite/actions/runners?per_page=100 | \ + https://api.github.com/repos/cnti-testcatalog/testsuite/actions/runners?per_page=100 | \ jq -r '.runners[].id' | \ while IFS='' read -r line; do - echo "Calling url https://api.github.com/repos/cncf/cnf-testsuite/actions/runners/$line" - curl -X DELETE -u $GH_USER:$GH_TOKEN -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/cncf/cnf-testsuite/actions/runners/$line" + echo "Calling url https://api.github.com/repos/cnti-testcatalog/testsuite/actions/runners/$line" + curl -X DELETE -u $GH_USER:$GH_TOKEN -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/cnti-testcatalog/testsuite/actions/runners/$line" done