Arrfab's Blog Linux tips and tricks …


Monitoring DRBD resources with Zabbix on CentOS

We use DRBD at work on several CentOS 5.x nodes to replicate data between our two computer rooms (in different buildings but linked with Gigabit fiber). It's true that you can know if something wrong happens at the DRBD level if you have configured the correct 'handlers' and the appropriate notifications scripts (Have a look for example at the Split Brain notification script). Those scripts are 'cool' but what if you could 'plumb' the DRBD status in your actual monitoring solution ? We use Zabbix at $work and I was asked to centralize events from differents sources and Zabbix doesn't support directly monitoring DRBD devices. But one of the cool thing with Zabbix is that it's like a Lego system : you can extend what it does if you know what to query and how to do it. If you want to monitor DRBD devices, the best that Zabbix can do (on the agent side, when using the zabbix agent running as a simple zabbix user with /sbin/nologin as shell) is to query and parse /proc/drbd . So here we go : we need to modify the Zabbix agent to use Flexible User Parameters, like this (in /etc/zabbix/zabbix_agentd.conf) :

UserParameter=drbd.cstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep cs|cut -f 2 -d ':'|grep Connected |wc -l
UserParameter=drbd.dstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep ds|cut -f 2 -d ':'|cut -f 1 -d '/'|grep UpToDate|wc -l

We just need to inform the Zabbix server of the actual Connection State (cs) and Disk State (ds) . For that we just need to create Application/Items and Triggers .. but what if we could just create a Zabbix Template so that we can just link that template to a DRBD host ? I attach to this post the DRBD Zabbix template (xml file that you can import in your zabbix setup) and you can just link it to your drbd hosts. Here is the link . That XML file contains both two Items (cstate and dstate) and the associated triggers. Of course you can extend it, especially if you use multiple resources , drbd disks. Because we used the Flexible parameters, you can for example in the Zabbix item, create a new one (based on the template) and monitor the /dev/drbd1 device just by using the drbd.dstate[1] key in that zabbix item.

Happy Monitoring and DRBD'ing ...


CentOS 6 LiveCD and LiveDVD tools

The number of questions I received from different people regarding the LiveCD/LiveDVD tools and the kickstart files used to produce the ISO images was quite "high". People looking at the normal place will be disappointed because we haven't used the original livecd subversion repo to produce the actual Live medias.  So in the meantime, if people want to use the livecd-creator tool, they can fetch the SRPM here : . I've just copied also the two kickstart files used for both LiveCD and LiveDVD here :

Hope that people will be satisfied .. faster to push those files there than to change the whole 'used behind the scene' infra

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 ISO spins

As you've probably seen if you're subscribed to the CentOS announce list (or if you just rsync/mirror the whole CentOS tree) , the CentOS 6.0 LiveCD was released last monday. This is the first of our CentOS custom spins ! While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way to the external mirrors too and will normally be announced shortly (when enough mirrors will have it) ! It will be the second CentOS respin and we have more in the pipe for you ! As Karanbir announced it in the 6.0 release mail , we planned also to provide two other spins : the minimal one and the lws one. Good news is that the minimal one is almost finished and being intensively tested. If things don't change (or bugs appear during QA), the iso image will be only ~250Mb for the i386 arch and ~300Mb for the x86_64 one. It's meant to be used as a real basic CentOS system (even less packages that the @core group on a normal install if used with the proper kickstart invocation !) : 186 packages only on your disk. You'll have a very basic CentOS system with only openssh-server and yum. We are even testing the luks/lvm/md devices combination to be sure to meet your needs.

The next custom respin (LWS code name - for LightWeigth Server edition) will still be a CD iso image (but pushed to the limit) that will include basic server packages, more or less in the idea of the ServerCD that existed during the CentOS 4.x days ... That one still needs to be finished while work has already being done.

Stay tuned for more informations when it will be pushed to mirrors and announced .. all that at the same time as 6.1 and 5.7 (in parallel) builds ..Interesting times ! :-)

Filed under: CentOS, Fun, Linux Comments Off

CentOS 6 on the iMac

I decided to put CentOS 6 on my iMac. It was running in dual-boot mode with OSX and CentOS 5. Installing through the network (from a NFS share) was really easy and no bug encountered but at the end of the install, when it asked me to reboot, nothing : after having selected the Linux partition in the rEfit boot manager screen, nothing. hmm ....

I restarted the install process to see if at least anaconda tried to install grub on the first sector of the /boot partition and not in the MBR but that was correctly seen and chosen by anaconda . So the issue was somewhere else. I had a /boot ext3 partition (on /dev/sda3) while /dev/sda4 is the VolumeGroup in which I had defined my Logical Volumes. There was a big rewrite in Anaconda for the storage part and el6/CentOS 6 suffers from one bug found on the upstream bugzilla when having to deal with Apple computers *and* using rEfit at the same time :

Long story short : to have CentOS 6 running on your iMac (if using refit as the EFI boot manager) :

  • install CentOS 6 as usual (check that grub will be installed on the first sector of /boot and not in the MBR , normally correctly seen/proposed by Anaconda)
  • on the first reboot, enter the rEFIt shell and launch 'gptsync' (it will say that it has to 'sync' the gpt, accept the sync)
  • select now the Linux partition : it will fail with a black screen
  • power down the iMac and start it up : select Linux in the refit boot manager and enjoy your CentOS 6 installation on the iMac
Filed under: CentOS, Fun Comments Off

Modifying Anaconda behaviour without rebuilding the whole install media

One thing that I had to have a look at (during CentOS 6 QA), is the way anaconda (the Red Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to those kind of install know what I'm talking about : the "Mininal", "Desktop", "Basic Server" and other choices you have during setup. From that first selection, you can decide (or not) to customize the software selection which then leads you to a screen containing categories / groups / packages defined in the comps.xml file present under /repodata on the tree/install media.

If you don't 'see' which screen i'm talking about, a small screenshot of the upcoming CentOS 6 will explain better than words :

Those pre-defined tasks aren't defined in the comps.xml file but rather at build time within anaconda. Fine but how can you 'modify' anaconda behaviour and test it without having to patch anaconda SRPM, rebuild it and launch a new build to generate the tree and install medias ? Easy , thanks to a simple file on the tree !

People wanting to modify anaconda behaviour at install time without having to regenerate the whole tree can just create a small file (updates.img) , put it in the /images directory in the tree. Anaconda (when installing over the network, http/ftp/nfs) always try to see if an updates.img file exists, and if so, use it. Fine, so I could easily try to "patch" it without having to modify the whole tree.

Creating that updates.img (it's just a ext2 filesystem on top) is really easy :

dd if=/dev/zero of=/tmp/updates.img bs=1k count=1440
losetup `losetup -f` /tmp/updates.img

losetup -a|grep updates.img
mkfs.ext2 /dev/loop3           # was loop3 in my case
mkdir /mnt/loop ; mount -o loop /tmp/updates.img /mnt/loop/ ; ll /mnt/loop
drwx------. 2 root root 12288 Jun 11 15:43 lost+found

From now, it's just a matter of putting the new files that you want to test and that will "overwrite" at run-time the defaults anaconda ones.

(in our current example, it was the installclasses/ that needed to be modified, so I just had to create a installclasses dir and drop my version of in there on the loop device)

When you're done, umount the updates.img, copy it to /path/to/your/install/tree/images , restart a http install (verify that permissions and selinux contexts are of course correct !) and enjoy !

Easier and faster. Thanks to the Anaconda team which decided to permit modifying the anaconda behaviour at run-time with a simple file :-)

Filed under: CentOS, Linux Comments Off

IPV6 world day !

It seems quite a lot of people blogged about IPV6 day . It's true that it's always a good idea to speak about IPV6. I'm using IPV6 natively on my server hosted at Hetzner (they offer a /64 IPV6 subnet, which is more than enough for a CentOS server hosting several xen domU Virtual Machines). At home, that's another story. I use a free tunnel to be able to reach ipv6 hosts. Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ? that's indeed a good question. Even if my CentOS ipv6 tunnel end-point/router/radvd at home is working correctly, I decided to ask my belgian provider if they had plans on implementing native IPV6. Well, not for my home connection, as I already know that Belgacom (the biggest provider in belgium) doesn't support IPV6 on their BBOX2 modems that they give to customers when ordering a DSL connection at home (while i'm talking about Belgacom, please stop sending me direct advertisement to my mailbox - the real one and not the electronic one - with your invoices about a service - VDSL2/BelgacomTV - that you *can't* offer to all your customers ... thanks) . So I decided to ask their 'professional services' because we have two 'professional and business' lines that we used at $work. Long story short (to avoid explaining how much emails/cases I had to send/open to have an answer) : "no, even on the business lines we can't support IPV6 and we have no plans (*sic*, I hope that guy was just kidding or probably doesn't know the real answer ..) nor dates about future implementation of the IPV6 services/connectivity " ..

Nice .. now /me goes back to CentOS QA mode ...

Filed under: CentOS, Fun, Linux Comments Off

What do you want to see ? CentOS 5.6 or CentOS 6.0 ?

As you probably know (if you are interested in the Enterprise Linux market), Red Hat released earlier today 5.6 . So automatically some CentOS QA team members started to discuss about that in the appropriate IRC channel. As CentOS 6.0 isn't (yet) released nor ready, the discussion was about putting 5.6 build & release as priority number one or not. Karanbir on his side asked on Twitter about thoughts on the matter, and a discussion was started too on the centos-devel list about that topic. My personal opinion (and shared by some people too) seems to give 5.6 the priority for quite some reasons :

  • The centos 5.x install base is there while there is (obviously) no centos 6 install base.
  • So those people having machines in production, faced to the net (, etc, etc, ...)  would prefer having their machines patched and up2date (security first !)
  • People running CentOS 5.x on servers and willing to install php53 packages, now officially included
  • On the build side, the el5 build process is clearly identified and known since 2007 : packages with branding issues are already identified and patches/artwork is already there, meaning that it will be probably (no, surely !) faster to have 5.6 out of the door than 6
  • Same rule for the QA process : people from the QA team can "blindly" focus on their previous tests, and just have a look eventually at some newer packages (a few, like php53 but not that much in comparison with el6)

Please notice that it's still my personal opinion on that question and isn't the (to be defined) official CentOS position.

Filed under: CentOS, Linux Comments Off

CentOS team @ Fosdem 2011

Some members of the CentOS team will be present at the Fosdem . Feel free to come at our booth just to discuss ...

More informations on our wiki and on the Fosdem website

Filed under: CentOS, Fun, Linux Comments Off

Enabling IPv6 for guests on an Hetzner CentOS 5.5 xen dom0

I was playing with IPv6 in the last days (started to use a tunnel from as my current ISP doesn't support  native IPv6 and doesn't plan to support it in a short time) and wanted to add IPv6 to some of my CentOS Xen domU's running on a Hetzner box. This part was a little bit more difficult than for a standard network. Due to their internal network design, Hetzner only allow 'routed' xen networks and not standard 'bridged' ones. What I used for IPv4 was just binding the public IPs on the dom0 and configured all my iptables rules there to forward/SNAT/DNAT to the appropriate domU. But you know that NAT is gone with IPv6 so normally it's supposed to be easier, right ? Well, yes and no, depending on your network layout. Even after  having enabled ipv6 forwarding (net.ipv6.conf.all.forwarding=1 ), I was just able to ping the dom0 but not the guests behind. Hmm, that reminds me the proxy ARP that was used for IPv4 but not existing anymore for IPv6 (gone too ...) . ARP was (more or less, not technically correct but read the RFCs if you enough time) replaced by NDP but I don't see such option for IPv6. Well, a kernel feature called proxy_ndp (net.ipv6.conf.all.proxy_ndp=1) exists on newer kernels (like for example the 2.6.32.x that is used on RHEL6 , and so in CentOS 6) but not on CentOS 5.5 (using a 2.6.18.x) kernel .. Hmmm ...

On the other side, I was searching for a 'workaround' probably given by libvirt, but the version included in RHEL5/CentOS5 doesn't know what to do with IPv6. Okay so let's have a look at the Xen and kernel side at the same time. If the proxy_ndp kernel feature is not present on my CentOS 5.5 dom0, I can still 'advertise' my neighbors with the ip command : yes, it supports it : " ip -6 neighbor add proxy your:ipv6:long:address::1 dev eth0"

So we just need to create a modified vif-route script (in fact I decided to call it vif-route6) that will be used for ipv6 guests :

# /etc/xen/scripts/vif-route6
# Script for configuring a vif in routed mode for IPv6 only
# Based on existing vif-route script in /etc/xen/scripts and adapted for ipv6

dir=$(dirname "$0")
. "$dir/"

main_ip6=$(ip -6 addr show eth0|grep 'scope global'|sort|head -n 1|awk '{print $2}'|cut -f 1 -d '/')

case "$command" in
ifconfig ${vif} ${main_ip} netmask up
ip -6 addr add ${main_ip6} dev ${vif}
do_without_error ifdown ${vif}

if [ "${ip}" ] ; then
# If we've been given a list of IP addresses, then add routes from dom0 to
# the guest using those addresses.
for addr in ${ip} ; do
${cmdprefix} ip -6 neighbor ${ipcmd} proxy ${addr} dev ${netdev:-eth0} 2>&1
result=`${cmdprefix} ip -6 route ${ipcmd} ${addr} dev ${vif} src ${main_ip6} 2>&1`


log debug "Successful vif-route $command for $vif."
if [ "$command" = "online" ]

Ok, so we have just now to modify our xen domU's config to add a vif that will use that specific script and give it the IPv6 address that we'll assign to that domU (from /etc/xen/your-domU-name):

vif = [ <snip of the first vif> , "mac=00:16:36:38:31:b8,vifname=test.ipv6,script=vif-route6,ip=2a01:4f8:100:4363::dead" ]

You can now start your domU and configure it normally for IPv6 (using obviously that 2a01:4f8:100:4363::dead IPv6 address and choosing the dom0 main IPv6 address as gateway ...

Hope it will help some people in the same situation (using a routed and not a bridged network layout for xen)


Zabbix crashes when using IPMI checks

Working for an IBM Business Partner for quite some years, I was used to deploy and configure (and even teach for IBM) IBM Director as a monitoring solution (for both hardware/operating systems/snmp devices/etc/etc ...). Now that I work as a sysadmin, I have to maintain one IBM director 5.20.3 setup I had myself installed and configured quite some time ago (as a consultant then). But I didn't want to update to 6.2 because it simply kills the machine on which it runs .. needs too much processor, too much memory .. and just to give you an idea : it's a Websphere/java thing that you have to install now ... I wanted to go the opensource way instead, but with something that can still monitor Linux/Windows/snmp devices and IPMI devices (we have quite some IBM servers and/or BladeCenter).

I tested Zabbix and directly felt in love with it : the agent memory footprint is really small (in comparison with that java-based agent on the Director side) and the way to build Items and Triggers is really great. I deployed it in our environment but focused first on the OS/services side (as the 'other' monitoring solution was still there for the hardware layer monitoring). I wanted then to use the integrated IPMI features of Zabbix and started to poll data from our IBM servers ... until .. crash !

From the zabbix_server.log :

2774:20101217:100001.893 IPMI Host []: first network error, wait for 15 seconds
2774:20101217:100002.894 Got signal [signal:11(SIGSEGV),reason:2,refaddr:0x34a3f52a38]. Crashing ...

Hmm, not good when the monitoring application crashes itself. I disabled all my IPMI checks and then the server was back without any issue. I repeated the above steps vice and versa to proove that it was really IPMI related and it's the case. Browsing the Zabbix support website returned me quite some interesting answers, including that one (ZBX-2898) and surely that one (ZBX-633) . Ok so that confirms that IPMI checks have to be disabled now and let's wait for Zabbix 1.8.4 to appear .. In the meantime I'll write some scripts (type External Check) to return values in Zabbix that can be used to create Triggers ... that's also one of the advantages in Zabbix : you can still write many plugins/scripts to do the same things :-)