Skip to content

Latest commit

 

History

History
121 lines (88 loc) · 9.32 KB

TEST_DEV_GUIDE.md

File metadata and controls

121 lines (88 loc) · 9.32 KB

Test Development Guidelines

Contents

Writing Tests in Gherkin

The Aries Agent Test Harness utilizes a behavioural driven approach to testing. The Python toolset Behave is used to actualize this approach. [Gherkin] is the language syntax used to define test preconditions and context, actions and events, and expected results and outcomes.

The first step in developing a suite of tests for an Aries RFC is to write plain english Gherkin definitions, before any code is written. The only input to the test cases should be the RFC. The test cases should not be driven by agent or agent framework implementations.

The priority is to do "happy path" type tests first, leaving the exception & negative testing until there are multiple suites across protocols of happy path acceptance tests. Write one main scenario then get peers and others familiar with the RFC to review the test. This is important because the structure and language of this initial test may guide the rest of the tests in the suite.

Initial writing of the Gherkin tests themselves are done in a .feature file or in a GitHub issue detailing the test development work to be accomplished. If no GitHub issue exists for the test development work, create one.

To keep test definitions immune to code changes or nomenclatures in code, it is best to express the RFC in high level terms from the user level based on predefined persona, currently Acme, Bob and Mallory, that can be interpreted at the business level without revealing implementation details. For example, When Acme requests a connection with Bob instead of When Acme sends a connection-request to Bob. Sometimes this may be cumbersome, so just make it as high level as makes sense. A full example from the connection protocol might look something like this;

Scenario Outline: establish a connection between two agents
Given we have "2" agents
| name | role |
| Acme | inviter |
| Bob | invitee |
When "Acme" generates a connection invitation
And "Bob" receives the connection invitation
And "Bob" sends a connection request to "Acme"
And "Acme" receives the connection request
And "Acme" sends a connection response to "Bob"
And "Bob" receives the connection response
And "Bob" sends <message> to "Acme"
Then "Acme" and "Bob" have a connection

Examples:
| message |
| trustping |
| ack |

Utilize data tables and examples in the Gherkin definition where possible.

The test cases should use the test persona as follows:

  • Acme: an enterprise agent with issuer and verifier capabilities
  • Bob: a holder/prover person
  • Mallory: a malicious holder/prover person

As necessary, other persona will be added. We expect adding Carol (another holder/prover person), Faber (another enterprise issuer and verifier) and perhaps Thing (an IOT thing, likely an issuer and verifier). Note that each additional persona requires updates to the running of tests (via the ./manage script) and introduce operational overhead, so thought should be given before introducing new characters into the test suite.

Using Tags

The test harness run script supports the use of tags in the feature files to be able to narrow down a test set to be executed. The general tags currently utilized are as follows:

  • @AcceptanceTest - Tests based on requirements specifically stated in the RFC. These are tests that will run to verify confirmance of the implemented protocol in the Agents.
  • @DerivedFunctionalTest - Tests derived on requirements but not specifically stated in the RFC.
  • @P1 - Test Priority
  • @P2 - Test Priority
  • @P3 - Test Priority
  • @P4 - Test Priority
  • @NegativeTest - Test that attempts to break the software. ie. change workflow order, use invalid data, etc.
  • @ExceptionTest - Tests that are based on requirements that suggest exception cases.
  • @NegativeTest - Tests specifically designed to try and break the software
  • @SmokeTest - Tests that can be used as a builds smoke or sanity tests.
  • @NeedsReview - Tests that have not been reviewed or approved.
  • @ReviewedApproved - obvious
  • @wip - Tests that are a work in progress and incomplete
  • @Done - Finished tests that are expected to Pass if executed against an Agent.
  • @AIP10 - Aries Interop Profile version the tests are written for
  • @T01-API10-RFC0160 - Test Unique Identifier - Please use T for the test cases number, the API version number the test is written for, and the RFC number for the Protocol under test.
  • @WillFail - Tests completed but because of an outstanding bug, they are expected to fail. The test harness looks for this to report on the next tag
  • @OutstandingBug..###..url - When using @WillFail, this tag much also be used. It is comprised of 3 components, OutstandingBug, the bug number, and the bug url. These must be separated by double periods.

Defining Protocol Suite Specific Tags

There will be cases where there will be a need for Protocol specific tags. This will usually reveal itself when there are optional implementations or where implementations can diverge into 2 or more options. Tests will need to be tagged with the least common option, where no tag means the other option. For example in the connection protocol there are specific tests that exercise the behaviour of the protocol using a Multi Use Invite and a Single Use Invite. The tag @MultiUseInvite is used to differentiate the two, and by default it is expected that MultiUseInvite is the least common option.

Currently Existing Connection Protocol Tags

  • @MultiUseInvite - Test utilizes a multi-use invite. Not using this tag and the test expects the invite to be single use.
  • @SingleTryOnException
  • @RetryableOnException

Defining specific tags should be discussed with the Aries Protocol test community.

Defining Backchannel Operations

Defining test steps require using and extending the commands and operations to be implemented by the backchannels. The commands and operations are documented in a Google Sheet, located here. As test developers add new steps to test cases, document the new operations oon which they depend in the spreadsheet. The data from the Google Sheet is available to the backchannels and are used by some to implement the backchannel capabilities. Making the data available is currently a manual process that is done using the following process:

  • prepare a branch in your fork of the repo that will be a PR
  • export the Google Sheet, saving it into this file in the repo: steps/Mapping Aries Protocols for Testing - Aries Agent Test Scripts.csv
  • run the command in the aries-test-harness/steps folder: ./genBCdata Mapping\ Aries\ Protocols\ for\ Testing\ -\ Aries\ Agent\ Test\ Scripts.csv
    • this will replace the current data file in the aries-backchannels/data folder
  • submit a PR to update the file

To Do: This should be changed to a github action that is run on each commit to the repo

Implementing Test Steps

Follow standard best practices for implementing test steps in Behave, writing the test steps as if the feature is fully supported and then adding code at all levels to supprt the new test. The process is something like this:

  • Study the RFC and define the tests needed to cover at least the happy path requirements of the RFC
    • Raise as issues in the Aries RFCs repo any deficiencies in the RFC
  • Define Gherkin tests using the existing persona
    • Add new persona only if really, really needed
  • Execute the new tests, grabbing the skeleton code for the new, unimplemented steps from the behave output, and adding them to the steps Python code
  • Implement the steps code, defining as needed commands and operations to be added to the backchannel interface
  • Update the Google Sheet to add the new commands and operations and regenerate the backchannel operations data file (see subsection above)
  • If you are also responsible for implementing one or more backchannels, extend the backchannel(s) to support the new commands and operations
  • Notify the community of backchannel maintainers of the new tests, commands and operations

Existing backchannels will throw a "NotImplementedException" for any steps that are not implemented in the backchannels, and should include information from the above-mentioned data file.

Implementing the Backchannel

See the README in the aries-backchannels folder for details on writing backchannels.