While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

• The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
• There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
• Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
• Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
• A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !