I've been working a lot with openQA lately, openSUSE's great operating system testing tool, and I think some of what I've been getting up is kind of interesting.
This is likely to be the first of a series of blog posts about my crazy adventures with this awesome tool.
What is openQA?
openQA is a fully featured operating system testing tool.
It takes an operating system ISO or Disk Image, and using its Domain Specific Language (DSL) conducts a series of tests consisting of keyboard and mouse inputs while analysing the screen output for specific matches (called 'needles') to ensure the operating system is behaving as intended.
A typical, simple example of an openQA test for looks like this
#Header, doesn't normally need to be changed
use base "basetest";
use strict;
use testapi;
#run() subroutine, where the test code goes
sub run() {
#test functions
assert_screen "inst-bootmenu", 6;
send_key 'ret';
}
1;
This test is used to check a boot loader. The 'assert_screen' function in this case is looking for a matching 'needle' (screenshot) which has a tag named 'inst-bootmenu'. If it doesn't find a matching needle within the defined timeout (6 seconds) the test fails. If it finds a matching needle within 6 seconds, the test continues, and presses the return key with the 'send_key' function.
The results of these tests are provided in the form of logs, screenshots, and video recordings delivered over the openQA web interface. You can have a look at openSUSE's production instance at http://openqa.opensuse.org and an example of a single set of test results HERE.
Screenshots? Isn't it a pain in the arse maintaining the Needles?
Not at all. The openQA webUI contains a needle editor (example HERE) which lets testers see both the reference screenshots and the accompanying JSON file, and edit both with a WYSIWYG editor.
Select the areas of the UI you're interested in, click save, done. It takes seconds to update your needles to accept intended changes to a UI.
As needles are only looking for specific UI elements on the screen, well crafted needles can often allow regular UI changes to pass unmolested. If you're not interested in confirming the version number drawn on the screen, don't include the version number in your needles, and you'll have a version neutral needle - like in the example HERE.
The openQA web interface can automatically commit and (if you want) push your needles to a git repository (openSUSE's are HERE) making it really easy for testers/test developers to work together on keeping your needle collection up to date.
Is openQA just for openSUSE?
NO! While it's mainly used and contributed to by the openSUSE Project for testing Tumbleweed and the openSUSE Distribution, it's always been designed to be not only distribution-neutral (ie. it can test any Linux Distribution) but technically operating system-neutral - Linux, BSD, Windows, openQA doesn't care about what it's testing.
Yeah, right...prove it.
I haven't been working on openQA for a very long time. For a few weeks I've been working on new openQA tests for openSUSE and SUSE Linux Enterprise, but as we already have a huge collection of tests in our public git repository, a good number of my new tests are borrowing code and ideas from our existing test library.
So I got wondering, how hard is it to test a totally differing operating system, with a totally different installation workflow?
So, after reading the recent Register review of Fedora 21 that had a subtitle mentioning 'install woes', I decided to pick on our friends at the Fedora Project and downloaded a copy of Fedora 21 Workstation.
'Designing' tests for openQA is easy. I fired up the ISO in a VM and kept notes on which screens would make good needles and which keys to press. Each of the screens became an 'assert_screen' call in my tests, and each keypress became a 'send_key'.
NOTE: I could have used openQA's 'assert_and_click' function to check the screen for a needled area and click in the middle of it - it's a great for pressing buttons, but I decided to take a very keyboard centric approach.
In a few hours I had thrown together a basic set of tests, which you can get on GitHub
These tests successfully complete a full installation of Fedora 21 Workstation, as you can see from the openQA-produced Video I uploaded to YouTube
It wasn't perfect though, no QA tool is any good if it doesn't find bugs - luckily even these really simple tests were able to reliably reproduce a known issue in anaconda which apparently had been causing problems since October and hasn't yet been fixed.
Wanting to prove the tests and needles were generic enough that they could be reused for future testing, I've also run these against a recent Rawhide build.
Everything looks really promising, but a bug in Rawhide with Keyboard Shortcuts needs to be fixed before my tests can complete a full run.
Not bad for a few hours work if I say so myself, and I'd really like it if other people want to pick up where I've left off and extend these tests to provide more comprehensive coverage.
I'm interested, where can I find out more?
The main openQA website on GitHub is at http://os-autoinst.github.io/openQA/ and contains all the Documentation and Downloads you need to get started.
It takes about 1 hour to setup openQA on a basic openSUSE installation, and it doesn't have any crazy hardware requirements. If you can run KVM virtualisation, you can run openQA. All these Fedora tests have been run on a local instance of openQA on my X220 Laptop running openSUSE Tumbleweed.
I'd especially recommend the Getting Started Guide which explains a lot more about the fundamentals of openQA
For more help, the best place is the #opensuse-factory IRC channel on irc.freenode.org (where I can be found as ilmehtar and/or sysrich) or the opensuse-factory@opensuse.org mailinglist
Have a lot of fun!