[DRUNX] Distributed Runtime and bUild for NuttX Test Environment

[cederom]

• In order to maintain high quality of the code releases we need not only Build but also Runtime Automated Testing.
• GitHub CI provides automated Pull Request verification (@lupyuen manages).
• We need to test built code on a physical devices in order to validate drivers etc.
• Solution should be distributed, so anyone can test what they have at hand/home/work/desk.
• It's called Distributed Test Environment because it enables independent decentralized cross-check examination.
• Distributed design provides availability redundancy.
• Some sort of central Dashboard is required for build/runtime logs storage, analysis, reporting (@lupyuen created working prototype).
• It will objectively indicate which change broke (or is going to break) what and where, so project can react accordingly.
• Use case: Some old PC as local CI machine, attach available board(s), build and runtime test master, report problems to Dashboard.
• List of external and internal tests needs to be created for all boards, and boards/configuration specific tests. This can be implemented as selftest board configuration. Every board should at least provide mandatory base selftest, and optional selftest-extended, selftest-specific, selftest-custom, etc (see below).
• We need a building blocks for base / extended / specific / custom test scenarios. Base tests will be mandatory and cover all boards (i.e. nsh + help + ostest). Extended may include benchmarks, stress tests, will be optional (may not be possible on small platforms) but may provide additional results like performance improvement or degradation. Specific tests will be optional too and would cover arch / board specific tests. There must be a way to implement Custom scenarios for closed testing of custom hardware etc.
• All tests must return PASS, FAIL, TIMEOUT, UNAVAILABLE results for every single test case plus detailed logs.
• At first nuttx and nuttx-apps master will be tested. Check if PR can be also intercepted and verified that way in order to provide runtime feedback from a physical boards.

[raiden00pl]

@cederom hi, do we need this long label DRUNX Distrubutes.... ? Can we use here some standardized way of using labels? We have already Area: Qualification Tests, so maybe this will be enough? We don't want to have a mess in the labels, otherwise they don't fulfill their purpose :)

[cederom]

@raiden00pl: @cederom hi, do we need this long label DRUNX Distrubutes.... ? Can we use here some standardized way of using labels? We have already Area: Qualification Tests, so maybe this will be enough? We don't want to have a mess in the labels, otherwise they don't fulfill their purpose :)

Allright, sorry, juest testing the projects :-) It displays well in other places. What do you propose? drunx: core, drunx: dashboard? Testing? Please update as you please you will choose the best one :-)

[fdcavalcanti]

@cederom there are plenty of QEMU tests on Espressif side, all pytest based.
If there is computing power available for running more QEMU tests, we could think of a way to integrate those tests to Nuttx.
Had a brief talk about this with @lupyuen, so I'm tagging him here.

[cederom]

@fdcavalcanti: @cederom there are plenty of QEMU tests on Espressif side, all pytest based. If there is computing power available for running more QEMU tests, we could think of a way to integrate those tests to Nuttx. Had a brief talk about this with @lupyuen, so I'm tagging him here.

Thank you @fdcavalcanti that would be great!!

Ultimately DRUNX will work on home computers of interested users, @lupyuen already has working prototype that simply run the CI scripts that we run on GitHub. There is also Dashboard that sums up the builds from various places. Ultimately there will be as many computing powers as tehere are users testing :-)

PyTest is a well standardized choice :-)

Are those QEMU images available to fetch from some public repo or even build by hand?

What platforms are supported?

Do you have some sort of bootstrap for them?

ESP QEMU tests would perfectly fit the build and runtime in a situation where no hardware is available or necessary.. maybe even GH CI if not too expensive to run :-)

How much place would it take to place them in nuttx-apps/testing?

I have created dedicated issue here: #15733 :-)

Thank you!! :-)

[fdcavalcanti]

We use QEMU available for ESP32, ESP32S3 and ESP32C3. You can find them here.

All tests are Pytest based, but heavily use the pytest-embeded plugin, more specifically the pytest-embedded-nuttx plugin (supports Nuttshell parsing, flashing, etc).

Simple tests like building, booting and testing if nsh> is responsive exist. Which is enough to know if the build is at least working to this extent. This allows testing most defconfigs that do not require external hardware, WiFi and BLE.
We also test bootloader combinations: reuse all defconfigs and apply MCUBoot instead of simple boot.

However there are more complex tests that use some customized application similar to nuttx-apps. Most would be Espressif related since they target our boards and don't follow some rules (like directly calling GPIO instead of using gpio.h when testing motor driver, capture driver, etc). That's something to be analyzed internally first.

I'm afraid the current NuttX CI image is not compatible. We use a Docker image that has the build system and all plugins required for the tests.

[cederom]

Thank you @fdcavalcanti :-) Lets move ESP QEMU discussion to dedicated thread #15733 :-)

[acassis]

@cederom hi, do we need this long label DRUNX Distrubutes.... ? Can we use here some standardized way of using labels? We have already Area: Qualification Tests, so maybe this will be enough? We don't want to have a mess in the labels, otherwise they don't fulfill their purpose :)

@cederom also although you seem to stick to this "DRUNX" name, it seems already a name of other project: https://github.com/alxolr/drunx

I know you want to use a funny name that sounds like "drunk", but it is better to use unique name and more "serious" name. Think about the message we are sending for a company that want to use NuttX!

[cederom]

@acassis: @cederom also although you seem to stick to this "DRUNX" name, it seems already a name of other project: https://github.com/alxolr/drunx

I know you want to use a funny name that sounds like "drunk", but it is better to use unique name and more "serious" name. Think about the message we are sending for a company that want to use NuttX!

Thanks Alan :-) Just a working slur until we figure out better name, ideas welcome!! :-)

[lupyuen]

Now testing our new PR Test Bot, that will Build a Test a PR on Real Hardware :-) (Oz64 SG2000 RISC-V SBC)

• [Feedback] PR Test Bot #15779

[cederom]

@lupyuen Now testing our new PR Test Bot:, that will Build a Test a PR on Real Hardware :-) (Oz64 SG2000 RISC-V SBC)

I saw it already commenting on PRs wow AMAZING WORK @lupyuen as always!! Big Thank You!! :-)