PR tests reporting

[szczecha]

Scope of the change

I want to have test reports available in PRs and measure flaky tests
This PR adds a new PW reporter https://github.com/ctrf-io/playwright-ctrf-json-reporter
and publish test summary https://github.com/ctrf-io/github-test-reporter to PRs

Next, we will collect data form PRs and enable option for historical data:
fail-rate-report
flaky-rate-report
previous-results-report
and add reporting for other workflows

[changeset-bot]

🦋 Changeset detected

Latest commit: f51a4ea

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
saleor-dashboard	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Differences Found

⚠️ 1 packages or licenses were added.

Expand

License	Package
MIT	playwright-ctrf-json-reporter

Summary

Expand

License Name	Package Count	Packages
0BSD	1	Packages tslib
BlueOak-1.0.0	1	Packages path-scurry
CC0-1.0	1	Packages type-fest
MIT/X11	1	Packages nub
MPL-1.1	1	Packages harmony-reflect
MPL-2.0	1	Packages dompurify
Public Domain	1	Packages jsonify
Python-2.0	1	Packages argparse
WTFPL	1	Packages opener
CC-BY-4.0	2	Packages @saleor/macaw-ui caniuse-lite
<<missing>>	3	Packages busboy saleor-dashboard streamsearch
BSD-2-Clause	24	Packages @typescript-eslint/parser @typescript-eslint/typescript-estree browser-process-hrtime css-select css-what domelementtype domhandler domutils dotenv dotenv-expand entities escodegen eslint-scope espree esprima esrecurse estraverse esutils nth-check stringify-object And 4 more...
BSD-3-Clause	45	Packages @humanwhocodes/object-schema @sentry/cli @sentry/cli-darwin @sentry/cli-linux-arm @sentry/cli-linux-arm64 @sentry/cli-linux-i686 @sentry/cli-linux-x64 @sentry/cli-win32-i686 @sentry/cli-win32-x64 @sinonjs/commons @sinonjs/fake-timers abab asn1js babel-plugin-istanbul charenc chroma-js crypt diff esquery exenv And 25 more...
Apache-2.0	46	Packages @ampproject/remapping @editorjs/editorjs @humanwhocodes/config-array @humanwhocodes/module-importer @opentelemetry/api @opentelemetry/semantic-conventions @playwright/test @swc/core @swc/core-darwin-arm64 @swc/core-darwin-x64 @swc/core-linux-arm-gnueabihf @swc/core-linux-arm64-gnu @swc/core-linux-arm64-musl @swc/core-linux-x64-gnu @swc/core-linux-x64-musl @swc/core-win32-arm64-msvc @swc/core-win32-ia32-msvc @swc/core-win32-x64-msvc @swc/counter aria-query And 26 more...
ISC	50	Packages @esbuild-plugins/node-globals-polyfill @istanbuljs/load-nyc-config @saleor/app-sdk anymatch boolbase browser-stdout cli-width cliui electron-to-chromium fastq flatted fs.realpath get-caller-file get-own-enumerable-property-symbols glob glob-parent graceful-fs inflight inherits ini And 30 more...
MIT	1124	Packages @adobe/css-tools @apollo/client @ardatan/relay-compiler @ardatan/sync-fetch @babel/code-frame @babel/compat-data @babel/core @babel/generator @babel/helper-annotate-as-pure @babel/helper-compilation-targets @babel/helper-create-class-features-plugin @babel/helper-member-expression-to-functions @babel/helper-module-imports @babel/helper-module-transforms @babel/helper-optimise-call-expression @babel/helper-plugin-utils @babel/helper-replace-supers @babel/helper-simple-access @babel/helper-skip-transparent-expression-wrappers @babel/helper-string-parser And 1104 more...

[karola312]

What if test if flaky? Now we have info only about test case status in merge reports, before I saw a "flaky" field

[szczecha]

What if test if flaky? Now we have info only about test case status in merge reports, before I saw a "flaky" field

@karola312 I disabled it for now because when we post failed/flaky or all results we have tags in the test name with @ so each time we notify someone for our PRs which isn't a great vibe. We need to change that and then we can enable more options
Here is an example test report where we called additional people who has GitHub nick same as our test tags:
https://github.com/saleor/saleor-dashboard/actions/runs/12868849980