Arrfab's Blog Linux tips and tricks …


Automatic laptop backup with NetworkManager (and correct selinux policies …)

Those days, almost everyone uses a laptop as his primary (work)station : I don't remember when I was using something else than a laptop for both work and home usage. I admit that I'm using what I'll describe in the following sentences for quite some time, but it seems some people I spoke to don't know what can be done around NetworkManager, and because I encountered a (small) issue with that process (because of updated selinux policies), I thought it would be a good time to speak about it.

Let me first discuss a (little) bit about NetworkManager : almost everyone (using CentOS/Fedora or other distributions) knows what it's all about : helping you to quickly switch from one network to another, that network being a wired one, a Wifi hotpot, or even a 3G connection through your 3G usb modem or your smartphone being used as a modem, etc, etc .... That's the "visible" part of NetworkManager.  While some people don't seem to like it, I admit myself that I really appreciate it and I use it on a daily basis for $work and $home usage (switching from wired to wireless, and so on). A quick read in the NetworkManager man page shows that you can "script" events based on the actual status of your network interface : basically all executables scripts found by NetworkManager under /etc/NetworkManager/dispatcher.d/ will be executed on network change. When I discovered that (was quite some time ago now ...), I decided that it would be good to launch backup script for my laptop, depending on the network my laptop is connected, and using different profiles. For example, (the "head" of ) a simple script can look like :


if [[ "$IF" = "eth0" && "$STATUS" = "up" ]] ; then

NET=$(/sbin/ip -4 route show dev eth0|awk '{print $1}'|grep -v default)
if [ "$NET" = "" ] ; then # and now the rest up to you ....

You've got the idea, so it's now just a matter of writing the whole script. One thing that I like when writing some small scripts is the fact that I can be notified on my laptop when something happens (or doesn't, because of errors). I use also quite often notify-send for that, but because all scripts under dispatcher.d are executed under root, I prefer from there "jumping" to my user account with a "su - $my_user_name -c $".

Of course, my script needs several things to "interact" with my desktop session : the DISPLAY to use and also the dbus-session I currently use (because I also have to use gvfs-mount to automatically mount in my gnome session some remote folders, like , (yeah, don't shoot me for that, not my idea) CIFS shares for $work).

So that backup script needs some variables like this :

export DISPLAY=":0"
export DBUS_SESSION_BUS_ADDRESS=$(cat /proc/$(pidof nautilus)/environ|tr '\0' '\n'|grep DBUS_SESSION_BUS|cut -f2- -d '=')

If I started that blog post, it's not to speak about NetworkManager at first (well, I still thought that some people would benefit of those unknown/unused dispatcher.d scripts ....) but because I encountered an issue with the recent updates to CentOS 6.4 (and to be precise, newer selinux-policy-3.7.19-195.el6_4.3.noarch package). So it was time to dive into that issue , and *yes*, i run selinux everywhere, including on my laptop ...

Long story short : because I use rsync for my backup scripts (why having to reinvent the wheel ? ), I had to enable two selinux booleans :

setsebool -P rsync_client 1
setsebool -P rsync_export_all_ro 1

But that was still not enough. sealert/audit.log/audit2allow to the rescue (read the Selinux page on the CentOS wiki) and finally I created a custom policy that suits my needs. Here it is :

module rsync-client.pol 1.0;

require {
type initrc_tmp_t;

type user_home_t;
type rsync_t;
class sock_file getattr;
class file write;

#============= rsync_t ==============
allow rsync_t initrc_tmp_t:file write;

allow rsync_t user_home_t:sock_file getattr;

Now, everytime I connect my laptop to a (recognized) network, my laptop auto-backups itself :

Filed under: CentOS, Fun Comments Off

Using Openssh as transport for Ansible instead of default paramiko

You've probably read that Ansible uses by default paramiko for the SSH connections to the host(s) you want to manage. But since 0.5 (quite some ago now ...) Ansible can use plain openssh binary as a transport. Why ? simple reasons : you sometimes have complex scenario and you can for example declare a ProxyCommand in your ~/.ssh/config if you need to use a JumpHost to reach the real host you want to connect to. That's fine and I was using that for some of the hosts i have to managed (specifying -c ssh when calling ansible, but having switched to a bash alias containing that string and also -i /path/to/my/inventory for those hosts).

It's great but it can lead to strange results if you don't have a full look at what's happening in the background. Here is the situation I just had yesterday : one of the remote hosts is reachable, but not a standard port (aka tcp/22) so an entry in my ~/.ssh/config was containing both HostName (for the known FQDN of the host I had to point to, not the host i wanted to reach) and Port.

Host myremotehost
Port 2222

With such entry, I was able to just "ssh user@myremotehost" and was directly on the remote box. "ansible -c ssh  -m ping myremotehost" was happy, but in fact was not reaching the host I was thinking : running "ansible -c ssh -m setup myremotehost -vvv" showed me that ansible_fqdn (one of the ansible facts) wasn't the correct one but instead the host in front of that machine (the one declared with HostName in ~/.ssh/config). The verbose mode showed me that even if you specify the Port in your ~/.ssh/config, ansible will *always* use port 22 :

<myremotehost> EXEC ['ssh', '-tt', '-q', '-o', 'AddressFamily=inet', '-o', 'ControlMaster=auto', '-o', 'ControlPath=/tmp/ansible-ssh-%h-%p-%r', '-o', 'StrictHostKeyChecking=no', '-o', 'Port=22', '-o', 'User=root', 'myremotehost', 'mkdir -p /var/tmp/ansible-1351603527.81-16435744643257 && echo /var/tmp/ansible-1351603527.81-16435744643257']

Hmm, quickly resolved : a quick discussion with people hanging in the #ansible IRC channel (on explained the issue to me : Port is *never* being looked at in your ~/.ssh/config, even when using -c ssh. Solution is to specify the port in your inventory file, as a variable for that host :

myremotehost ansible_ssh_port=9999

In the same vein, you can also use ansible_ssh_host , this one corresponding to the HostName of your  ~/.ssh/config.

Hope that it can save you time, if you encounter the same "issue" one day ...


Ansible as an alternative to puppet/chef/cfengine and others …

I already know that i'll be criticized for this post, but i don't care :-) . Strangely my last blog post (which is *very* old ...) was about a puppet dashboard, so why speaking about another tool ? Well, first i got a new job and some prerequisites have changed. I still like puppet (and I'd even want to be able to use puppet but that's another story ...) but I was faced to some constraints when being in front of a new project. For that specific project,  I had to configure a bunch of new Virtual Machines (RHEL6) coming as OVF files. Problem number one was that I can't alter or modify the base image so i can't push packages (from the distro or third-party repositories). Second issue is that I can't install nor have a daemon/agent running on those machines. I had a look at the different config tools available but they all require either a daemon to be started, or at least having extra packages to be installed on each managed node. (so not possible to have puppetd nor puppetrun or invoke puppet directly through ssh , as puppet can't even be installed, same for saltstack). That's why i decided to give Ansible a try. It was already on my "TO-test" list for a long time but it seems it was really fitting the bill for that specific project and constraints : using the 'already-in-place' ssh authorization, no packages to be installed on the managed nodes, and last-but-no-least, a learning curve that is really thin (compared to puppet and others, but that's my personal opinion/experience).

The other good thing with Ansible is that you can start very easily and then slowly add 'complexity' to your playbooks/tasks. I'm still using for example a flat inventory file, but already organized to reflect what we can do in the future (hostnames included in groups, themselves included in parents groups - aka nested groups). Same for the variables inheritance : at the group level and down to the host level, host variables overwriting those defined at the group level , etc ...)

The Yaml syntax is really easy to understand so you can have quickly your first playbook being played on a bunch of machines simultaneously (thanks to paramiko/parallel ssh). The number of modules is less than the puppet resources, but is quickly growing. I also just tested to tie the execution of ansible playbook with Jenkins so that people not having access to the ansible inventory/playbooks/tasks (stored in a vcs, subversion in my case) can use it from a gui.. More to come on Ansible in the future


Puppet, Foreman and selinux on CentOS

We implemented Puppet as a configuration management system at $work ,  and Puppet is a great tool. Then I heard about some dashboards that could be used on top of it. I've heard about different dashboards ($management_people *like* dashboards) like Puppet-dashboard and Foreman.

I was advised by several people to give Foreman a try and it's really simple to install. Their wiki covers basic installation and there is even a yum repo that can be used (Epel has to be enabled too). As i have a small network to manage, I decided to setup Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is easy and missing parts can be configured just by looking at the Foreman website wiki/FAQ. But troubles came when I enabled reports : puppetmasterd config was changed to include :

reports = store, foreman

and the foreman.rb script (copied and modified from /usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb) integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports dir. (Note : don't forget to update $foreman_url).

But no reports were coming in Foreman. hmmm .... error message was :

Report foreman failed: Could not send report to Foreman at Permission denied - connect(2)

That was not an iptables issue, but selinux one :

type=AVC msg=audit(1329830711.788:28372): avc:  denied  { name_connect } for  pid=13144 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket

Here is my locally generated selinux for Foreman :

module foreman 1.0;

require {
type puppetmaster_t;
type http_port_t;
type ntop_port_t;
class tcp_socket name_connect;

#============= puppetmaster_t ==============
allow puppetmaster_t http_port_t:tcp_socket name_connect;
allow puppetmaster_t ntop_port_t:tcp_socket name_connect;

Things work really better after I added my foreman.pp selinux module on that host. If you don't know how to compile selinux custom policies, please read the nice Selinux page on the CentOS wiki, and especially the "Manually customizing selinux policies" section. Tools like sealert (from setroubleshoot-server package) and audit2allow are really helpful when there is no pre-defined selinux boolean that can be used.

Hope this helps .. and now going back enjoying reports, including error reports by mail (nice feature)


CentOS Automated QA explained …

While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

  • The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
  • There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
  • Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
  • Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
  • A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !

Filed under: CentOS, Fun, Linux Comments Off

Monitoring DRBD resources with Zabbix on CentOS

We use DRBD at work on several CentOS 5.x nodes to replicate data between our two computer rooms (in different buildings but linked with Gigabit fiber). It's true that you can know if something wrong happens at the DRBD level if you have configured the correct 'handlers' and the appropriate notifications scripts (Have a look for example at the Split Brain notification script). Those scripts are 'cool' but what if you could 'plumb' the DRBD status in your actual monitoring solution ? We use Zabbix at $work and I was asked to centralize events from differents sources and Zabbix doesn't support directly monitoring DRBD devices. But one of the cool thing with Zabbix is that it's like a Lego system : you can extend what it does if you know what to query and how to do it. If you want to monitor DRBD devices, the best that Zabbix can do (on the agent side, when using the zabbix agent running as a simple zabbix user with /sbin/nologin as shell) is to query and parse /proc/drbd . So here we go : we need to modify the Zabbix agent to use Flexible User Parameters, like this (in /etc/zabbix/zabbix_agentd.conf) :

UserParameter=drbd.cstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep cs|cut -f 2 -d ':'|grep Connected |wc -l
UserParameter=drbd.dstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep ds|cut -f 2 -d ':'|cut -f 1 -d '/'|grep UpToDate|wc -l

We just need to inform the Zabbix server of the actual Connection State (cs) and Disk State (ds) . For that we just need to create Application/Items and Triggers .. but what if we could just create a Zabbix Template so that we can just link that template to a DRBD host ? I attach to this post the DRBD Zabbix template (xml file that you can import in your zabbix setup) and you can just link it to your drbd hosts. Here is the link . That XML file contains both two Items (cstate and dstate) and the associated triggers. Of course you can extend it, especially if you use multiple resources , drbd disks. Because we used the Flexible parameters, you can for example in the Zabbix item, create a new one (based on the template) and monitor the /dev/drbd1 device just by using the drbd.dstate[1] key in that zabbix item.

Happy Monitoring and DRBD'ing ...


CentOS 6 LiveCD and LiveDVD tools

The number of questions I received from different people regarding the LiveCD/LiveDVD tools and the kickstart files used to produce the ISO images was quite "high". People looking at the normal place will be disappointed because we haven't used the original livecd subversion repo to produce the actual Live medias.  So in the meantime, if people want to use the livecd-creator tool, they can fetch the SRPM here : . I've just copied also the two kickstart files used for both LiveCD and LiveDVD here :

Hope that people will be satisfied .. faster to push those files there than to change the whole 'used behind the scene' infra

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 ISO spins

As you've probably seen if you're subscribed to the CentOS announce list (or if you just rsync/mirror the whole CentOS tree) , the CentOS 6.0 LiveCD was released last monday. This is the first of our CentOS custom spins ! While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way to the external mirrors too and will normally be announced shortly (when enough mirrors will have it) ! It will be the second CentOS respin and we have more in the pipe for you ! As Karanbir announced it in the 6.0 release mail , we planned also to provide two other spins : the minimal one and the lws one. Good news is that the minimal one is almost finished and being intensively tested. If things don't change (or bugs appear during QA), the iso image will be only ~250Mb for the i386 arch and ~300Mb for the x86_64 one. It's meant to be used as a real basic CentOS system (even less packages that the @core group on a normal install if used with the proper kickstart invocation !) : 186 packages only on your disk. You'll have a very basic CentOS system with only openssh-server and yum. We are even testing the luks/lvm/md devices combination to be sure to meet your needs.

The next custom respin (LWS code name - for LightWeigth Server edition) will still be a CD iso image (but pushed to the limit) that will include basic server packages, more or less in the idea of the ServerCD that existed during the CentOS 4.x days ... That one still needs to be finished while work has already being done.

Stay tuned for more informations when it will be pushed to mirrors and announced .. all that at the same time as 6.1 and 5.7 (in parallel) builds ..Interesting times ! :-)

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 on the iMac

I decided to put CentOS 6 on my iMac. It was running in dual-boot mode with OSX and CentOS 5. Installing through the network (from a NFS share) was really easy and no bug encountered but at the end of the install, when it asked me to reboot, nothing : after having selected the Linux partition in the rEfit boot manager screen, nothing. hmm ....

I restarted the install process to see if at least anaconda tried to install grub on the first sector of the /boot partition and not in the MBR but that was correctly seen and chosen by anaconda . So the issue was somewhere else. I had a /boot ext3 partition (on /dev/sda3) while /dev/sda4 is the VolumeGroup in which I had defined my Logical Volumes. There was a big rewrite in Anaconda for the storage part and el6/CentOS 6 suffers from one bug found on the upstream bugzilla when having to deal with Apple computers *and* using rEfit at the same time :

Long story short : to have CentOS 6 running on your iMac (if using refit as the EFI boot manager) :

  • install CentOS 6 as usual (check that grub will be installed on the first sector of /boot and not in the MBR , normally correctly seen/proposed by Anaconda)
  • on the first reboot, enter the rEFIt shell and launch 'gptsync' (it will say that it has to 'sync' the gpt, accept the sync)
  • select now the Linux partition : it will fail with a black screen
  • power down the iMac and start it up : select Linux in the refit boot manager and enjoy your CentOS 6 installation on the iMac
Filed under: CentOS, Fun Comments Off

Modifying Anaconda behaviour without rebuilding the whole install media

One thing that I had to have a look at (during CentOS 6 QA), is the way anaconda (the Red Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to those kind of install know what I'm talking about : the "Mininal", "Desktop", "Basic Server" and other choices you have during setup. From that first selection, you can decide (or not) to customize the software selection which then leads you to a screen containing categories / groups / packages defined in the comps.xml file present under /repodata on the tree/install media.

If you don't 'see' which screen i'm talking about, a small screenshot of the upcoming CentOS 6 will explain better than words :

Those pre-defined tasks aren't defined in the comps.xml file but rather at build time within anaconda. Fine but how can you 'modify' anaconda behaviour and test it without having to patch anaconda SRPM, rebuild it and launch a new build to generate the tree and install medias ? Easy , thanks to a simple file on the tree !

People wanting to modify anaconda behaviour at install time without having to regenerate the whole tree can just create a small file (updates.img) , put it in the /images directory in the tree. Anaconda (when installing over the network, http/ftp/nfs) always try to see if an updates.img file exists, and if so, use it. Fine, so I could easily try to "patch" it without having to modify the whole tree.

Creating that updates.img (it's just a ext2 filesystem on top) is really easy :

dd if=/dev/zero of=/tmp/updates.img bs=1k count=1440
losetup `losetup -f` /tmp/updates.img

losetup -a|grep updates.img
mkfs.ext2 /dev/loop3           # was loop3 in my case
mkdir /mnt/loop ; mount -o loop /tmp/updates.img /mnt/loop/ ; ll /mnt/loop
drwx------. 2 root root 12288 Jun 11 15:43 lost+found

From now, it's just a matter of putting the new files that you want to test and that will "overwrite" at run-time the defaults anaconda ones.

(in our current example, it was the installclasses/ that needed to be modified, so I just had to create a installclasses dir and drop my version of in there on the loop device)

When you're done, umount the updates.img, copy it to /path/to/your/install/tree/images , restart a http install (verify that permissions and selinux contexts are of course correct !) and enjoy !

Easier and faster. Thanks to the Anaconda team which decided to permit modifying the anaconda behaviour at run-time with a simple file :-)

Filed under: CentOS, Linux Comments Off