Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add all conformance runner flags to the documentation #285

Merged
merged 2 commits into from
Dec 19, 2024
Merged

Add all conformance runner flags to the documentation #285

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
159b252
Add all conformance runner flags to the documentation
timostamm Dec 19, 2024
3efb8b6
Merge remote-tracking branch 'origin/main' into tstamm/Add-all-confor…
timostamm Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 14 additions & 12 deletions docs/conformance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,19 @@ make conformance

You can customize the behavior of the tests using the following flags:

| Flag | Description | Default Value |
|------------------------|------------------------------------------------------|---------------|
| `--suite <regex>` | Filter suites using the provided regular expression. | None |
| `--case <regex>` | Filter cases using the provided regular expression. | None |
| `--timeout <duration>` | Set a per-suite timeout. | 5 seconds |
| `--verbose`, `-v` | Enable verbose output. | `false` |
| `--strict` | Enable strict mode. | `false` |
| `--json` | Return results as JSON to stdout. | `false` |
| `--proto` | Return results as binary serialized proto to stdout. | `false` |
| `--dump` | Output the expected results, without a command. | `false` |
| `--expected_failures` | `.yaml` file containing list of expected failures | None |
| Flag | Description | Default Value |
|------------------------|-----------------------------------------------------------------------------------------------------|---------------|
| `--suite <regex>` | Filter suites using the provided regular expression. | None |
| `--case <regex>` | Filter cases using the provided regular expression. | None |
| `--timeout <duration>` | Set a per-suite timeout. | 5 seconds |
| `--verbose`, `-v` | Enable verbose output. | `false` |
| `--strict` | Enable strict mode. Require that the violation type is an exact match. | `false` |
| `--strict_message` | Require that violation messages match the expected message exactly. | `false` |
| `--strict_error` | Require that compile-time errors are distinguished from runtime errors. | `false` |
| `--json` | Return results as JSON serialized message `buf.validate.conformance.harness.ResultSet` to stdout. | `false` |
| `--proto` | Return results as binary serialized message `buf.validate.conformance.harness.ResultSet` to stdout. | `false` |
| `--dump` | Output the expected results, without a command. | `false` |
| `--expected_failures` | `.yaml` file containing list of expected failures | None |

## Components

Expand Down Expand Up @@ -59,7 +61,7 @@ For a reference implementation, see [Go's conformance test executor][pv-go].
The test messages are defined in [`.proto` files][cases-proto]. These are
utilized to make concrete message _test cases_ which are organized into groups
known as [_test suites_][suites]. Which tests are passed to the executor can be
controlled through the `-suite` and `-case` flags mentioned above.
controlled through the `--suite` and `--case` flags mentioned above.

### Expected Failures

Expand Down
2 changes: 1 addition & 1 deletion tools/protovalidate-conformance/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ func parseFlags() (*config, error) {
flag.StringVar(&cfg.caseFilterPattern, "case", cfg.caseFilterPattern, "regex to filter cases")
flag.DurationVar(&cfg.suiteTimeout, "timeout", cfg.suiteTimeout, "per-suite timeout")
flag.BoolVarP(&cfg.verbose, "verbose", "v", cfg.verbose, "verbose output")
flag.BoolVar(&cfg.strict, "strict", cfg.strict, "strict mode")
flag.BoolVar(&cfg.strict, "strict", cfg.strict, "require that the violation type is an exact match")
flag.BoolVar(&cfg.strictMessage, "strict_message", cfg.strictMessage, "require that violation messages match the expected message exactly")
flag.BoolVar(&cfg.strictError, "strict_error", cfg.strictError, "require that compile-time errors are distinguished from runtime errors")
flag.BoolVar(&cfg.json, "json", cfg.json, "return results as JSON to stdout")
Expand Down
Loading