Skip to content

Latest commit

 

History

History
78 lines (49 loc) · 4.52 KB

results-logs.md

File metadata and controls

78 lines (49 loc) · 4.52 KB

Understanding Results and Logs

This section describes how to view and interpret IDT result reports and logs.

Viewing Results

While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it writes a test run summary to the console and generates two test reports. These reports can be found in <devicetester-extract-location>/results/<execution-id>/. Both reports capture the results from the qualification test suite execution.

The awsiotdevicetester_report.xml is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements:

  • The IDT for Amazon FreeRTOS version.
  • The Amazon FreeRTOS version that was tested.
  • The features of Amazon FreeRTOS that are supported by the device based on the tests passed.
  • The SKU and the device name specified in the device.json file.
  • The features of the device specified in the device.json file.
  • The aggregate summary of test case results.
  • A breakdown of test case results by libraries that were tested based on the device features (for example, FullWiFi, FullMQTT, and so on).

The AFQ_Report.xml is a report in standard JUnit XML format. You can integrate it into CI/CD platforms likeJenkins, Bamboo, and so on. The report contains the following elements:

  • An aggregate summary of test case results.
  • A breakdown of test case results by libraries that were tested based on the device features.

Interpreting IDT for Amazon FreeRTOS Results

The report section in awsiotdevicetester_report.xml or AFQ_Report.xml lists the tests that were run and the results of the tests.

The first XML tag <testsuites> contains the overall summary of the test execution. For example:

<testsuites name="AFQ results" time="5633" tests="184" failures="0" errors="0" disabled="0">Attributes used in the <testsuites> tag

name
The name of the test suite.

time
The time, in seconds, it took to run the qualification suite.

tests
The number of test cases executed.

failures
The number of test cases that were run, but did not pass.

errors
The number of test cases that IDT for Amazon FreeRTOS couldn't execute.

disabled
This attribute is not used and can be ignored.

If there are no test case failures or errors, your device meets the technical requirements to run Amazon FreeRTOS and can interoperate with AWS IoT services. If you choose to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence.

In the event of test case failures or errors, you can identify the test case that failed by reviewing the <testsuites> XML tags. The <testsuite> XML tags inside the <testsuites> tag shows the test case result summary for a test group.

<testsuite name="FullMQTT" package="" tests="16" failures="0" time="76" disabled="0" errors="0" skipped="0">

The format is similar to the <testsuites> tag, but with an attribute called skipped that is not used and can be ignored. Inside each <testsuite> XML tag, there are <testcase> tags for each of the test cases that were executed for a test group. For example:

<testcase classname="mcu.Full_MQTT" name="AFQP_MQTT_Connect_HappyCase" attempts="1"></testcase>Attributes used in the <testcase> tag

name
The name of the test case.

attempts
The number of times IDT for Amazon FreeRTOS executed the test case.

When a test fails or an error occurs, <failure> or <error> tags are added to the <testcase> tag with information for troubleshooting. For example:

<testcase classname="mcu.Full_MQTT" name="AFQP_MQTT_Connect_HappyCase"> <failure type="Failure">Reason for the test case failure</failure> <error>Reason for the test case execution error</error> </testcase>

For more information, see Troubleshooting.

Viewing Logs

You can find logs that IDT for Amazon FreeRTOS generates from test execution in <devicetester-extract-location>/results/<execution-id>/logs. Two sets of logs are generated:

test_manager.log
Contains logs generated from IDT for Amazon FreeRTOS (for example, logs related configuration and report generation).

<test_group_name>.log (for example, Full_MQTT.log)
The logs of the test group, including logs from the device under test.