Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REG-1872] add validation rules + update validations in general #79

Merged
merged 11 commits into from
Aug 12, 2024
Merged
Binary file added docs/validations/img/building-a-scenario.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/validations/img/create-validation-rule.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/validations/img/run-details.png
Binary file not shown.
Binary file added docs/validations/img/suite-run-details.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 21 additions & 9 deletions docs/validations/validations-getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,37 +6,49 @@ sidebar_label: 'Getting Started with Validations'

![Screenshot of a scenario](./img/scenario-example.png)

**Validations** are used to assert states within you game. You can create the scenarios you want to verify using our Scenario Builder,
and can run collections of scenarios in Unity.
**Validations** are used to assert states within you game. You can create the Scenarios you want to verify using our Scenario Builder,
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved
and can run collections of Scenarios in Unity.
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved

## Validation Suites

**Validation Suites** are a collection of scenarios that you want to run together. You can create a Suite starting with a name and description. At the core of a Suite is its Base Gameplay Recording. This recording is a gameplay session recording used to create the scenarios that will be run, and the inputs used during the recording will be used when running the scenarios.
**Validation Suites** are a collection of Scenarios and Rules that you want to run together. You can create a Suite starting with a name and description. At the core of a Suite is its Base Gameplay Recording. This recording is a gameplay session recording used to create the Scenarios and Rules that will be run, and the inputs used during the recording will be used when running the scenarios.

### Sharing Suites

Validation Suites can be shared with other Regression Games users, in a read-only mode. If a Suite is shared with you, you can view the Suite and its scenarios, but you cannot edit the Suite or its scenarios.
Validation Suites can be shared with other Regression Games users, in a read-only mode. If a Suite is shared with you, you can view the Suite and its Scenarios, but you cannot edit the Suite, its Scenarios, or its Rules.

![Screenshot of a scenario](./img/share.png)

## Scenario Builder

**Scenarios** are composed of steps that can be asserted throughout a gameplay recording. You can wait for specific mouse or keyboard events, check for specific game objects, or even check for specific values in the game state. The available options when creating Scenarios are drawn from the Base Gameplay Recording associated with a Suite.

Along with Scenario steps, you can also view the state, and view a screenshot, at each tick during the recording. This can help you understand the state of the game at each point in time, and can also help create the scenarios you want to run.
Along with Scenario steps, you can also view the state, and view a screenshot, at each tick during the recording. This can help you understand the state of the game at each point in time, and can also help create the Scenarios you want to run.

![Building a Scenario](./img/building-a-scenario.png)

Check out the [full Scenario Builder reference](./validations-scenario-builder-reference) for more details

## Rules

**Rules** are used to assert that the gameplay recording meets specific criteria, such as the average frames per second during the recording. When viewing a specific Validation Suite, you can add Rules to the Suite that will be asserted each time the Suite is run.

Rules can be added to a Suite by clicking the "Create Rule" button on the Suite's page. You can then select the Rule type you want to add, and configure the Rule's parameters. After adding a Rule to a Suite, you can always edit it later.

![Creating a Validation Rule](./img/create-validation-rule.png)

Check out the [full Rules reference](./validations-rules-reference) for more details

## Running Validations

You can either run **all** of the Scenarios within a Suite, or an individual Scenario. To run a Suite, click the "Run Suite" button, and select a Gameplay Session to test against. When the Suite is finished running, you can view the results of each Scenario within the Suite on the Test Runs page.
You can either run **all** of the Scenarios within a Suite, or test run an individual Scenario. To run a Suite, click the "Run Suite" button, and select a Gameplay Session to test against. When the Suite is finished running, you can view the results of each Scenario and Rule within the Suite on the Test Runs page.
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved

To run an individual Scenario, click the "Run Scenario" button within the Scenario's menu. The Scenario's results will be shown in a dialog.
To test run an individual Scenario, click the "Run Scenario" button within the Scenario's menu. The Scenario's results will be shown in a dialog, and will not be persisted.

![Screenshot of a scenario](./img/run-scenario.png)

## Viewing Results

To access the results of your Suite or Scenario, navigate to the Test Runs page. Here you can see the results of each Scenario within a Suite. You can also view the results of a Scenario's steps, and the state of the game at each tick during the Scenario.
To access the results of your Suite, navigate to the Test Runs page. Here you can see the results of each Scenario and Rule within a Suite. You can also view the results of a Scenario's steps, and the state of the game at each tick during the Scenario.
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved

![Screenshot of a scenario](./img/run-details.png)
![Screenshot of a scenario](./img/suite-run-details.png)
24 changes: 24 additions & 0 deletions docs/validations/validations-rules-reference.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
sidebar_label: 'Rules Reference'
---

# Rules Reference

Rules can be asserted against a gameplay recording to ensure that the recording meets specific criteria. We provide a set of rule types that can be used to construct assertions based on various performance metrics.
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved

Rules belong to a Validation Suite, and are asserted each time the Validation Suite is run.

## Rule Types

### Frames Per Second (with percentile)

Assert that the frames per second of the gameplay recording are above a threshold, within a specified percentile.
addisonbgross marked this conversation as resolved.
Show resolved Hide resolved

### Average Frames Per Second

Assert that the average frames per second of the gameplay recording is above a threshold.
svolokh marked this conversation as resolved.
Show resolved Hide resolved

### Maximum Frame Time

Assert that the maximum frame time of the gameplay recording is below a threshold.
svolokh marked this conversation as resolved.
Show resolved Hide resolved

13 changes: 6 additions & 7 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -50,24 +50,23 @@ const sidebars = {
items: [
"gameplay-sessions/gameplay-sessions-getting-started",
"gameplay-sessions/gameplay-sessions-usage",
"gameplay-sessions/gameplay-sessions-reference"
]
"gameplay-sessions/gameplay-sessions-reference",
],
},
{
type: "category",
label: "Validations",
items: [
"validations/validations-getting-started",
"validations/validations-scenario-builder-reference",
"validations/validations-rules-reference",
],
},
"authenticating-with-api-keys",
{
type: "category",
label: "Generic Bots",
items: [
"generic-bots/monkey-bot"
]
items: ["generic-bots/monkey-bot"],
},
{
type: "category",
Expand All @@ -76,8 +75,8 @@ const sidebars = {
// DISABLED For MVP Launch.
// "creating-bots/csharp/agent-builder",
"creating-bots/csharp/adaptive-bots",
]
}
],
},
],
automatedTesting: [
"automated-testing/overview",
Expand Down
Loading