Arrfab's Blog Linux tips and tricks …


Ansible as an alternative to puppet/chef/cfengine and others …

I already know that i'll be criticized for this post, but i don't care :-) . Strangely my last blog post (which is *very* old ...) was about a puppet dashboard, so why speaking about another tool ? Well, first i got a new job and some prerequisites have changed. I still like puppet (and I'd even want to be able to use puppet but that's another story ...) but I was faced to some constraints when being in front of a new project. For that specific project,  I had to configure a bunch of new Virtual Machines (RHEL6) coming as OVF files. Problem number one was that I can't alter or modify the base image so i can't push packages (from the distro or third-party repositories). Second issue is that I can't install nor have a daemon/agent running on those machines. I had a look at the different config tools available but they all require either a daemon to be started, or at least having extra packages to be installed on each managed node. (so not possible to have puppetd nor puppetrun or invoke puppet directly through ssh , as puppet can't even be installed, same for saltstack). That's why i decided to give Ansible a try. It was already on my "TO-test" list for a long time but it seems it was really fitting the bill for that specific project and constraints : using the 'already-in-place' ssh authorization, no packages to be installed on the managed nodes, and last-but-no-least, a learning curve that is really thin (compared to puppet and others, but that's my personal opinion/experience).

The other good thing with Ansible is that you can start very easily and then slowly add 'complexity' to your playbooks/tasks. I'm still using for example a flat inventory file, but already organized to reflect what we can do in the future (hostnames included in groups, themselves included in parents groups - aka nested groups). Same for the variables inheritance : at the group level and down to the host level, host variables overwriting those defined at the group level , etc ...)

The Yaml syntax is really easy to understand so you can have quickly your first playbook being played on a bunch of machines simultaneously (thanks to paramiko/parallel ssh). The number of modules is less than the puppet resources, but is quickly growing. I also just tested to tie the execution of ansible playbook with Jenkins so that people not having access to the ansible inventory/playbooks/tasks (stored in a vcs, subversion in my case) can use it from a gui.. More to come on Ansible in the future


Puppet, Foreman and selinux on CentOS

We implemented Puppet as a configuration management system at $work ,  and Puppet is a great tool. Then I heard about some dashboards that could be used on top of it. I've heard about different dashboards ($management_people *like* dashboards) like Puppet-dashboard and Foreman.

I was advised by several people to give Foreman a try and it's really simple to install. Their wiki covers basic installation and there is even a yum repo that can be used (Epel has to be enabled too). As i have a small network to manage, I decided to setup Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is easy and missing parts can be configured just by looking at the Foreman website wiki/FAQ. But troubles came when I enabled reports : puppetmasterd config was changed to include :

reports = store, foreman

and the foreman.rb script (copied and modified from /usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb) integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports dir. (Note : don't forget to update $foreman_url).

But no reports were coming in Foreman. hmmm .... error message was :

Report foreman failed: Could not send report to Foreman at Permission denied - connect(2)

That was not an iptables issue, but selinux one :

type=AVC msg=audit(1329830711.788:28372): avc:  denied  { name_connect } for  pid=13144 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket

Here is my locally generated selinux for Foreman :

module foreman 1.0;

require {
type puppetmaster_t;
type http_port_t;
type ntop_port_t;
class tcp_socket name_connect;

#============= puppetmaster_t ==============
allow puppetmaster_t http_port_t:tcp_socket name_connect;
allow puppetmaster_t ntop_port_t:tcp_socket name_connect;

Things work really better after I added my foreman.pp selinux module on that host. If you don't know how to compile selinux custom policies, please read the nice Selinux page on the CentOS wiki, and especially the "Manually customizing selinux policies" section. Tools like sealert (from setroubleshoot-server package) and audit2allow are really helpful when there is no pre-defined selinux boolean that can be used.

Hope this helps .. and now going back enjoying reports, including error reports by mail (nice feature)


CentOS Automated QA explained …

While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

  • The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
  • There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
  • Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
  • Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
  • A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !

Filed under: CentOS, Fun, Linux Comments Off

Monitoring DRBD resources with Zabbix on CentOS

We use DRBD at work on several CentOS 5.x nodes to replicate data between our two computer rooms (in different buildings but linked with Gigabit fiber). It's true that you can know if something wrong happens at the DRBD level if you have configured the correct 'handlers' and the appropriate notifications scripts (Have a look for example at the Split Brain notification script). Those scripts are 'cool' but what if you could 'plumb' the DRBD status in your actual monitoring solution ? We use Zabbix at $work and I was asked to centralize events from differents sources and Zabbix doesn't support directly monitoring DRBD devices. But one of the cool thing with Zabbix is that it's like a Lego system : you can extend what it does if you know what to query and how to do it. If you want to monitor DRBD devices, the best that Zabbix can do (on the agent side, when using the zabbix agent running as a simple zabbix user with /sbin/nologin as shell) is to query and parse /proc/drbd . So here we go : we need to modify the Zabbix agent to use Flexible User Parameters, like this (in /etc/zabbix/zabbix_agentd.conf) :

UserParameter=drbd.cstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep cs|cut -f 2 -d ':'|grep Connected |wc -l
UserParameter=drbd.dstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep ds|cut -f 2 -d ':'|cut -f 1 -d '/'|grep UpToDate|wc -l

We just need to inform the Zabbix server of the actual Connection State (cs) and Disk State (ds) . For that we just need to create Application/Items and Triggers .. but what if we could just create a Zabbix Template so that we can just link that template to a DRBD host ? I attach to this post the DRBD Zabbix template (xml file that you can import in your zabbix setup) and you can just link it to your drbd hosts. Here is the link . That XML file contains both two Items (cstate and dstate) and the associated triggers. Of course you can extend it, especially if you use multiple resources , drbd disks. Because we used the Flexible parameters, you can for example in the Zabbix item, create a new one (based on the template) and monitor the /dev/drbd1 device just by using the drbd.dstate[1] key in that zabbix item.

Happy Monitoring and DRBD'ing ...


CentOS 6 LiveCD and LiveDVD tools

The number of questions I received from different people regarding the LiveCD/LiveDVD tools and the kickstart files used to produce the ISO images was quite "high". People looking at the normal place will be disappointed because we haven't used the original livecd subversion repo to produce the actual Live medias.  So in the meantime, if people want to use the livecd-creator tool, they can fetch the SRPM here : . I've just copied also the two kickstart files used for both LiveCD and LiveDVD here :

Hope that people will be satisfied .. faster to push those files there than to change the whole 'used behind the scene' infra

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 ISO spins

As you've probably seen if you're subscribed to the CentOS announce list (or if you just rsync/mirror the whole CentOS tree) , the CentOS 6.0 LiveCD was released last monday. This is the first of our CentOS custom spins ! While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way to the external mirrors too and will normally be announced shortly (when enough mirrors will have it) ! It will be the second CentOS respin and we have more in the pipe for you ! As Karanbir announced it in the 6.0 release mail , we planned also to provide two other spins : the minimal one and the lws one. Good news is that the minimal one is almost finished and being intensively tested. If things don't change (or bugs appear during QA), the iso image will be only ~250Mb for the i386 arch and ~300Mb for the x86_64 one. It's meant to be used as a real basic CentOS system (even less packages that the @core group on a normal install if used with the proper kickstart invocation !) : 186 packages only on your disk. You'll have a very basic CentOS system with only openssh-server and yum. We are even testing the luks/lvm/md devices combination to be sure to meet your needs.

The next custom respin (LWS code name - for LightWeigth Server edition) will still be a CD iso image (but pushed to the limit) that will include basic server packages, more or less in the idea of the ServerCD that existed during the CentOS 4.x days ... That one still needs to be finished while work has already being done.

Stay tuned for more informations when it will be pushed to mirrors and announced .. all that at the same time as 6.1 and 5.7 (in parallel) builds ..Interesting times ! :-)

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 on the iMac

I decided to put CentOS 6 on my iMac. It was running in dual-boot mode with OSX and CentOS 5. Installing through the network (from a NFS share) was really easy and no bug encountered but at the end of the install, when it asked me to reboot, nothing : after having selected the Linux partition in the rEfit boot manager screen, nothing. hmm ....

I restarted the install process to see if at least anaconda tried to install grub on the first sector of the /boot partition and not in the MBR but that was correctly seen and chosen by anaconda . So the issue was somewhere else. I had a /boot ext3 partition (on /dev/sda3) while /dev/sda4 is the VolumeGroup in which I had defined my Logical Volumes. There was a big rewrite in Anaconda for the storage part and el6/CentOS 6 suffers from one bug found on the upstream bugzilla when having to deal with Apple computers *and* using rEfit at the same time :

Long story short : to have CentOS 6 running on your iMac (if using refit as the EFI boot manager) :

  • install CentOS 6 as usual (check that grub will be installed on the first sector of /boot and not in the MBR , normally correctly seen/proposed by Anaconda)
  • on the first reboot, enter the rEFIt shell and launch 'gptsync' (it will say that it has to 'sync' the gpt, accept the sync)
  • select now the Linux partition : it will fail with a black screen
  • power down the iMac and start it up : select Linux in the refit boot manager and enjoy your CentOS 6 installation on the iMac
Filed under: CentOS, Fun Comments Off

Modifying Anaconda behaviour without rebuilding the whole install media

One thing that I had to have a look at (during CentOS 6 QA), is the way anaconda (the Red Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to those kind of install know what I'm talking about : the "Mininal", "Desktop", "Basic Server" and other choices you have during setup. From that first selection, you can decide (or not) to customize the software selection which then leads you to a screen containing categories / groups / packages defined in the comps.xml file present under /repodata on the tree/install media.

If you don't 'see' which screen i'm talking about, a small screenshot of the upcoming CentOS 6 will explain better than words :

Those pre-defined tasks aren't defined in the comps.xml file but rather at build time within anaconda. Fine but how can you 'modify' anaconda behaviour and test it without having to patch anaconda SRPM, rebuild it and launch a new build to generate the tree and install medias ? Easy , thanks to a simple file on the tree !

People wanting to modify anaconda behaviour at install time without having to regenerate the whole tree can just create a small file (updates.img) , put it in the /images directory in the tree. Anaconda (when installing over the network, http/ftp/nfs) always try to see if an updates.img file exists, and if so, use it. Fine, so I could easily try to "patch" it without having to modify the whole tree.

Creating that updates.img (it's just a ext2 filesystem on top) is really easy :

dd if=/dev/zero of=/tmp/updates.img bs=1k count=1440
losetup `losetup -f` /tmp/updates.img

losetup -a|grep updates.img
mkfs.ext2 /dev/loop3           # was loop3 in my case
mkdir /mnt/loop ; mount -o loop /tmp/updates.img /mnt/loop/ ; ll /mnt/loop
drwx------. 2 root root 12288 Jun 11 15:43 lost+found

From now, it's just a matter of putting the new files that you want to test and that will "overwrite" at run-time the defaults anaconda ones.

(in our current example, it was the installclasses/ that needed to be modified, so I just had to create a installclasses dir and drop my version of in there on the loop device)

When you're done, umount the updates.img, copy it to /path/to/your/install/tree/images , restart a http install (verify that permissions and selinux contexts are of course correct !) and enjoy !

Easier and faster. Thanks to the Anaconda team which decided to permit modifying the anaconda behaviour at run-time with a simple file :-)

Filed under: CentOS, Linux Comments Off

IPV6 world day !

It seems quite a lot of people blogged about IPV6 day . It's true that it's always a good idea to speak about IPV6. I'm using IPV6 natively on my server hosted at Hetzner (they offer a /64 IPV6 subnet, which is more than enough for a CentOS server hosting several xen domU Virtual Machines). At home, that's another story. I use a free tunnel to be able to reach ipv6 hosts. Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ? that's indeed a good question. Even if my CentOS ipv6 tunnel end-point/router/radvd at home is working correctly, I decided to ask my belgian provider if they had plans on implementing native IPV6. Well, not for my home connection, as I already know that Belgacom (the biggest provider in belgium) doesn't support IPV6 on their BBOX2 modems that they give to customers when ordering a DSL connection at home (while i'm talking about Belgacom, please stop sending me direct advertisement to my mailbox - the real one and not the electronic one - with your invoices about a service - VDSL2/BelgacomTV - that you *can't* offer to all your customers ... thanks) . So I decided to ask their 'professional services' because we have two 'professional and business' lines that we used at $work. Long story short (to avoid explaining how much emails/cases I had to send/open to have an answer) : "no, even on the business lines we can't support IPV6 and we have no plans (*sic*, I hope that guy was just kidding or probably doesn't know the real answer ..) nor dates about future implementation of the IPV6 services/connectivity " ..

Nice .. now /me goes back to CentOS QA mode ...

Filed under: CentOS, Fun, Linux Comments Off

What do you want to see ? CentOS 5.6 or CentOS 6.0 ?

As you probably know (if you are interested in the Enterprise Linux market), Red Hat released earlier today 5.6 . So automatically some CentOS QA team members started to discuss about that in the appropriate IRC channel. As CentOS 6.0 isn't (yet) released nor ready, the discussion was about putting 5.6 build & release as priority number one or not. Karanbir on his side asked on Twitter about thoughts on the matter, and a discussion was started too on the centos-devel list about that topic. My personal opinion (and shared by some people too) seems to give 5.6 the priority for quite some reasons :

  • The centos 5.x install base is there while there is (obviously) no centos 6 install base.
  • So those people having machines in production, faced to the net (, etc, etc, ...)  would prefer having their machines patched and up2date (security first !)
  • People running CentOS 5.x on servers and willing to install php53 packages, now officially included
  • On the build side, the el5 build process is clearly identified and known since 2007 : packages with branding issues are already identified and patches/artwork is already there, meaning that it will be probably (no, surely !) faster to have 5.6 out of the door than 6
  • Same rule for the QA process : people from the QA team can "blindly" focus on their previous tests, and just have a look eventually at some newer packages (a few, like php53 but not that much in comparison with el6)

Please notice that it's still my personal opinion on that question and isn't the (to be defined) official CentOS position.

Filed under: CentOS, Linux Comments Off