Skip to content

Latest commit

 

History

History
423 lines (312 loc) · 18 KB

Contributing.asciidoc

File metadata and controls

423 lines (312 loc) · 18 KB

openQA developer guide

Table of Contents

Introduction

openQA is an automated test tool that makes it possible to test the whole installation process of an operating system. It’s free software released under the GPLv2 license. The source code and documentation are hosted in the os-autoinst organization on GitHub.

This document provides the information needed to start contributing to the openQA development improving the tool, fixing bugs and implementing new features. For information about writing or improving openQA tests, refer to the Tests Developer Guide. In both documents it’s assumed that the reader is already familiar with openQA and has already read the Starter Guide. All those documents are available at the official repository.

Development guidelines

As mentioned, the central point of development is the os-autoinst organization on GitHub where several repositories can be found:

As in most projects hosted on GitHub, pull request are always welcome and are the right way to contribute improvements and fixes.

Rules for commits

  • Every commit is checked by Travis CI as soon as you create a pull request but you should run the tidy script locally, i.e. before every commit call:

./script/tidy

to ensure your Perl code changes are consistent with the style rules.

  • You may also run local tests on your machine or in your own development environment to verify everything works as expected. Call:

make test

for unit and integration tests.

To execute a single test, one can use prove. You must set TEST_PG so the database can be found. If you set a custom base directory, be sure to unset it when running tests. Example:

TEST_PG='DBI:Pg:dbname=openqa_test;host=/dev/shm/tpg' OPENQA_BASEDIR= prove -v t/14-grutasks.t

To speed up the test initialization, start PostgreSQL using t/test_postgresql instead of using the system service. Eg.

t/test_postgresql /dev/shm/tpg

  • For git commit messages use the rules stated on How to Write a Git Commit Message as a reference

  • Every pull request is reviewed in a peer review to give feedback on possible implications and how we can help each other to improve

If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal.

Getting involved into development

But developers willing to get really involved into the development of openQA or people interested in following the always-changing roadmap should take a look at the openQAv3 project in openSUSE’s project management tool. This Redmine instance is used to coordinate the main development effort organizing the existing issues (bugs and desired features) into 'target versions'.

Currently developers meet in IRC channel #opensuse-factory and in a daily jangouts call of the core developer team.

In addition to the ones representing development sprints, two other versions are always open. Easy hacks lists issues that are not specially urgent and that are considered to be easy to implement by newcomers. Developers looking for a place to start contributing are encouraged to simply go to that list and assign any open issue to themselves. Future improvements groups features that are in the developers' and users' wish list but that have little chances to be addressed in the short term, either because the return of investment is not worth it or because they are out of the current scope of the development.

openQA and os-autoinst repositories also include test suites aimed at preventing bugs and regressions in the software. codecov is configured in the repositories to encourage contributors to raise the tests coverage with every commit and pull request. New features and bug fixes are expected to be backed with the corresponding tests.

Technologies

Everything in openQA, from os-autoinst to the web frontend and from the tests to the support scripts is written in Perl. So having some basic knowledge about that language is really desirable in order to understand and develop openQA. Of course, in addition to bare Perl, several libraries and additional tools are required. The easiest way to install all needed dependencies is using the available os-autoinst and openQA packages, as described in the Installation Guide.

In the case of os-autoinst, only a few CPAN modules are required. Basically Carp::Always, Data::Dump. JSON and YAML. On the other hand, several external tools are needed including QEMU, Tesseract and OptiPNG. Last but not least, the OpenCV library is the core of the openQA image matching mechanism, so it must be available on the system.

The openQA package is built on top of Mojolicious, an excellent Perl framework for web development that will be extremely familiar to developers coming from other modern web frameworks like Sinatra and that have nice and comprehensive documentation available at its home page.

In addition to Mojolicious and its dependencies, several other CPAN modules are required by the openQA package. For a full list of hard dependencies, see the file cpanfile at the root of the openQA repository.

openQA relies on PostgreSQL to store the information. It used to support SQLite, but that is no longer possible.

As stated in the previous section, every feature implemented in both packages should be backed by proper tests. Test::More is used to implement those tests. As usual, tests are located under the /t/ directory. In the openQA package, one of the tests consists of a call to Perltidy to ensure that the contributed code follows the most common Perl style conventions.

Starting the webserver from local Git checkout

  • To start the webserver for development, use the scripts/openqa daemon.

  • openQA will pull the required asssets on the first run.

  • openQA uses SASS, so Ruby development files are required. Under openSUSE, installing the packages devel_C_C++ and ruby-devel should be sufficient. openQA will install the required files automatically under .gem. Add .gem/ruby/2.4.0/bin to the PATH variable to let it find the sass/scss binaries. I also had to create symlinks of those binaries without .ruby2.4 suffix so openQA could find them.

  • It is also useful to start openQA with morbo which allows applying changes without restarting the server: morbo -m development -w assets -w lib -w templates -l http://localhost:9526 script/openqa daemon

Managing the database

During the development process there are cases in which the database schema needs to be changed. there are some steps that have to be followed so that new database instances and upgrades include those changes.

When is it required to update the database schema?

After modifying files in lib/OpenQA/Schema/Result. However, not all changes require to update the schema. Adding just another method or altering/adding functions like has_many doesn’t require an update. However, adding new columns, modifying or removing existing ones requires to follow the steps mentioned above.

How to update the database schema

  1. First, you need to increase the database version number in the $VERSION variable in the lib/OpenQA/Schema.pm file. Note that it’s recommended to notify the other developers before doing so, to synchronize in case there are more developers wanting to increase the version number at the same time.

  2. Then you need to generate the deployment files for new installations, this is done by running ./script/initdb --prepare_init.

  3. Afterwards you need to generate the deployment files for existing installations, this is done by running ./script/upgradedb --prepare_upgrade. After doing so, the directories dbicdh/$ENGINE/deploy/<new version> and dbicdh/$ENGINE/upgrade/<prev version>-<new version> for PosgreSQL should have been created with some SQL files inside containing the statements to initialize the schema and to upgrade from one version to the next in the corresponding database engine.

  4. Migration scripts to upgrade from previous versions can be added under dbicdh/_common/upgrade. Create a <prev_version>-<new_version> directory and put some files there with DBIx commands for the migration. For examples just have a look at the migrations which are already there.

The above steps are only for preparing the required SQL statements, but do not actually alter the database. Before doing so, it is recommended to backup your database to be able to downgrade again if something goes wrong or you just need to continue working on another branch. To do so, the following command can be used to create a copy:

createdb -O ownername -T originaldb newdb

To actually create or update the database (after creating a backup as described), you should run either ./script/initdb --init_database or ./script/upgradedb --upgrade_database. This is also required when the changes are installed in a production server.

How to add fixtures to the database

Note: This section is not about the fixtures for the testsuite. Those are located under t/fixtures.

Note: This section might not be relevant anymore. At least there are currently none of the mentioned directories with files containing SQL statements present.

Fixtures (initial data stored in tables at installation time) are stored in files into the dbicdh/_common/deploy/_any/<version> and dbicdh/_common/upgrade/<prev_version>-<next_version> directories.

You can create as many files as you want in each directory. These files contain SQL statements that will be executed when initializing or upgrading a database. Note that those files (and directories) have to be created manually.

Executed SQL statements can be traced by setting the DBIC_TRACE environment variable.

export DBIC_TRACE=1

How to overwrite config files

It can be necessary during development to change the config files in etc/. For example you have to edit etc/openqa/database.ini to use another database. Or to increase the log level it’s useful to set the loglevel to debug in etc/openqa/openqa.ini.

To avoid these changes getting in your git workflow, copy them to a new directory and set OPENQA_CONFIG in your shell setup files.

cp -ar etc/openqa etc/mine
export OPENQA_CONFIG=$PWD/etc/mine

Note that OPENQA_CONFIG points to the directory containing openqa.ini, database.ini, client.conf and workers.ini.

How to setup PostgreSQL to test locally with production data

  1. Install PosgreSQL - under openSUSE the following package are required: postgresql-server postgresql-init

  2. Start the server: systemctl start postgresql

  3. The following steps need to be done by the user postgres: su - postgres

  4. Create user: createuser your_username where your_username must be the same as the UNIX user you start your local openQA instance with.

  5. Create database: createdb -O your_username openqa

  6. The next steps must be done by the user you start your local openQA instance with.

  7. Import dump: pg_restore -c -d openqa path/to/dump

  8. Configure openQA to use PostgreSQL as described in the section Database of the installation guide. User name and password are not required.

Adding new authentication module

OpenQA comes with three authentication modules providing authentication methods: OpenID, iChain and Fake (see User authentication).

All authentication modules reside in lib/OpenQA/Auth directory. During OpenQA start, [auth]/method section of /etc/openqa/openqa.ini is read and according to its value (or default OpenID) OpenQA tries to require OpenQA::WebAPI::Auth::$method. If successful, module for given method is imported or the OpenQA ends with error.

Each authentication module is expected to export auth_login and auth_logout functions. In case of request-response mechanism (as in OpenID), auth_response is imported on demand.

Currently there is no login page because all implemented methods use either 3rd party page or none.

Authentication module is expected to return HASH:

%res = (
    # error = 1 signals auth error
    error => 0|1
    # where to redirect the user
    redirect => ''
);

Authentication module is expected to create or update user entry in OpenQA database after user validation. See included modules for inspiration.

Customize base directory

It is possible to customize the openQA base directory by setting the environment variable OPENQA_BASEDIR. The default value is /var/lib.

Running tests of openQA itself

There’s two ways of executing the testsuite locally:

  1. with docker
    The goal of running the tests with docker is to have consistent tests results (as sometimes the tests have different outcomes because missing packages or different package versions amongst other reasons). This is the preferred way if the user wants to run a full test battery or if it needs to setup a test database

  2. without docker

How to run tests with docker

To run them in docker please be sure that docker is installed and the docker daemon is running. To launch the test suite first it’s required to pull the docker image:

docker pull registry.opensuse.org/devel/openqa/containers/openqa_dev:latest

Build the image using Makefile target:

make docker-test-build

Launch the tests using Makefile target:

make docker-test-run

Run tests by invoking Docker manually:

docker run -v OPENQA_LOCAL_CODE:/opt/openqa -v /var/run/dbus:/var/run/dbus -e VAR1=1 -e VAR2=1 openqa:latest make docker-tests

Replace OPENQA_LOCAL_CODE to the location where you have the openqa code.

Replace VAR1 and VAR2 in -e switch to match a test battery of the test matrix:

FULLSTACK=0

UITESTS=0

FULLSTACK=0

UITESTS=1

GH_PUBLISH=true

FULLSTACK=1

SCHEDULER_FULLSTACK=1

DEVELOPER_FULLSTACK=1

tips

Running commands will be executed after the initialization script (database creation and so on..). So if there’s the need to run an interactive session after it just do:

docker run -it -v OPENQA_LOCAL_CODE:/opt/openqa -v /var/run/dbus:/var/run/dbus registry.opensuse.org/devel/openqa/containers/openqa_dev bash

Of course you can also use make docker-tests \; bash to run the tests first and then open a shell for further investigation.

There’s also the possibility to change the initialization scripts with the --entrypoint switch. This allows us to go into an interactive session without any initialization script run:

docker run -it --entrypoint /bin/bash -v OPENQA_LOCAL_CODE:/opt/openqa -v /var/run/dbus:/var/run/dbus registry.opensuse.org/devel/openqa/containers/openqa_dev

In case there’s the need to follow what’s happening in the current running docker (the execution will terminate the session):

docker exec -ti $(docker ps | awk '!/CONTAINER/{print $1}') /bin/bash

Running UI tests in non-headless mode is also possible, eg.:

xhost +local:root
docker run --rm -ti --name openqa-testsuite -v /tmp/.X11-unix:/tmp/.X11-unix:rw -e DISPLAY="$DISPLAY" -e NOT_HEADLESS=1 prove -v t/ui/14-dashboard.t
xhost -local:root

It is also possible to use a custom os-autoinst checkout using the following arguments:

docker run ... -e CUSTOM_OS_AUTOINST=1 -v /path/to/your/os-autoinst:/opt/os-autoinst make docker-tests

By default, configure and make are still executed (so a clean checkout is expected). If your checkout is already prepared to use, set CUSTOM_OS_AUTOINST_SKIP_BUILD to prevent this. Be aware that the build produced outside of the container might not work inside the container if both environments provide different, incompatible library versions (eg. OpenCV).

It is also important to mention that your local repositories will be copied into the container. This can take very long if those are big, eg. when the openQA repo contains a lot of profiling data because you enabled Mojolicious::Plugin::NYTProf.

In general, if starting the tests via Docker seems to hang, it is a good idea to inspect the process tree to see which command is currently executed.

How to run tests without docker

To execute the testsuite locally without docker, use make test. It is also possible to run a particular test for example prove t/api/01-workers.t.

To run UI tests the package perl-Selenium-Remote-Driver is required. Note that the version provided by Leap 42.2 is too old. The version from the repository devel-languages-perl can be used instead.

You need to install chromedriver and either chrome or chromium for the ui tests.