IPv6 connectivity status within the infra

Recently, some people started to ask proper IPv6/AAAA record for some of our public mirror infrastructure, like, and also

Reason is that a lot of people are now using IPv6 wherever possible and from a CentOS point of view, we should ensure that everybody can have content over (legacy) ipv4 and ipv6. Funny that I call ipv4 "legacy" as we still have to admit that it's still the default everywhere, even in 2016 with the available pools now exhausted.

While we had already some AAAA records for some of our public nodes (like as an example), I started to "chase" after proper and native ipv6 connectivity for our nodes. That's where I had to take contact with all our valuable sponsors. First thing to say is that we'd like to thank them all for their support for the CentOS Project over the years : it wouldn't have been possible to deliver multiple terrabytes of data per month without their sponsorship !

WRT ipv6 connectivity that's where the results of my quest where really different : while some DCs support ipv6 natively, and even answer you in 5 minutes ...

Kernel 3.10.0-327 issue on AMD Neo processor

As CentOS 7 (1511) was released, I thought it would be a good idea to update several of my home machines (including kids' workstations) with that version, and also newer kernel. Usually that's just a smooth operation, but sometimes some backported features/new features, especially in the kernel, can lead to some strange issues. That's what happened for my older Thinkpad Edge : That's a cheap/small thinkpad that Lenovo did several years ago ( circa 2011 ), and that I used a lot just when travelling, as it only has a AMD Athlon(tm) II Neo K345 Dual-Core Processor. So basically not a lot of horse power, but still something convenient just to read your mails, remotely connect through ssh, or browse the web. When rebooting on the newer kernel, it panics directly.

Two bug reports are open for this, one on the CentOS Bug tracker, linked also to the upstream one. Current status is that there is no kernel update that will fix this, but there is a easy to implement workaround :

  • boot with the initcall_blacklist=clocksource_done_booting kernel parameter added (or reboot on previous kernel)
  • once booted, add the same parameter at the end of the GRUB_CMDLINE_LINUX=" .." line ...

Kernel IO wait and megaraid controller

Last friday, while working on something else (working on "CentOS 7 userland" release for Armv7hl boards), I got notifications coming from our Zabbix monitoring instance complaining about web scenarios failing (errors due to time outs) , and also then also about "Disk I/O is overloaded" triggers (checking the cpu iowait time). Usually you'd verify what happens in the Virtual Machine itself, but even connecting to the VM was difficult and slow. But once connected, nothing strange, and no real activity , not even on the disk (Plenty of tools for this, but iotop is helpful to see which process is reading/writing to the disk in that case), but iowait was almost at 100%).

As said, it was happening suddenly for all Virtual Machines on the same hypervisor (CentOS 6 x86_64 KVM host), and even the hypervisor was suddenly complaining (but less in comparison with the VMs) about iowait too. So obviously, it wasn't really something not being optimized at the hypervisor/VMS, but something else. That rang a bell, as if you have a raid controller, and that battery for example is to be replaced, the controller can decide to stop all read/write cache, so slowing down ...

CentOS AltArch SIG status

Recently I had (from an Infra side) to start deploying KVM guests for the ppc64 and ppc64le arches, so that AltArch SIGs contributors could start bootstrapping CentOS 7 rebuild for those arches. I'll probably write a tech review about Power8 and the fact you can just use libvirt/virt-install to quickly provision new VMs on PowerKVM , but I'll do that in a separate post.

Parallel to ppc64/ppc64le, armv7hl interested some Community members, and the discussion/activity about that arch is discussed on the dedicated mailing list. It's slowly coming and some users already reported having used that on some boards (but still unsigned and no updates packages -yet- )

Last (but not least) in this AltArch list is i686 : Johnny built all packages and are already publicly available on , each time in parallel to the x86_64 version. It seems that respinning the ISO for that arch and last tests would be the only things to do.

If you're interested in participating in AltArch (and have special interesting a specific arch/platform), feel free to discuss that on the centos-devel list !

CentOS Dojo in Barcelona

So, thanks to the folks from Opennebula, we'll have another CentOS Dojo in Barcelona on Tuesday 20th October 2015. That even will be colocated with the Opennebulaconf happening the days after that Dojo. If you're attending the OpennebulaConf, or if you're just in the area and would like to attend the CentOS Dojo, feel free to register

Regarding the Dojo content, I'll be myself giving a presentation about Selinux : covering a little bit of intro (still needed for some folks afraid of using it , don't know why but we'll change that ...) about selinux itself, how to run it on bare-metal, virtual machines and there will be some slides for the mandatory container hype thing. But we'll also cover managing selinux booleans/contexts, etc through your config management solution. (We'll cover puppet and ansible as those are the two I'm using on a daily basis) and also how to build and deploy custom selinux policies with your config management solution.

On the other hand, if you're a CentOS user and would like yourself to give a talk during that Dojo, feel free to submit a talk ! More informations about the Dojo ...

Ext4 limitation with GDT blocks number

In the last days, I encountered a strange issue^Wlimitation with Ext4 that I wouldn't have thought of. I've used ext2/ext3/ext4 for quite some time and so I've been used to resize the filesystem "online" (while "mounted"). In the past you had to use ext2online for that, then it was integrated into resize2fs itself.

The logic is simple and always the same : extend your underlaying block device (or add another one), then modify the LVM Volume Group (if needed), then the Logical Volume and finally the resize2fs operation, so something like

lvextend -L +${added_size}G /dev/mapper/${name_of_your_logical_volume} 
resize2fs /dev/mapper/${name_of_your_logical_volume}

I don't know how much times I've used that, but this time resize2fs wasn't happy :

resize2fs: Operation not permitted While trying to add group #16384

I remember having had in the past an issue because of the journal size not being big enough. But this wasn't the case here.

FWIW, you can always verify your journal size with dumpe2fs /dev/mapper/${name_of_your_logical_volume} |grep "Journal Size"

Small note : if you need to increase the journal size, you have to do it "offline" as you have to remove the journal and ...

Implementing TLS for postfix

As some initiatives (like Let's Encrypt as one example) try to force TLS usage everywhere. We thought about doing the same for the infra. Obviously we already had some x509 certificates, but not for every httpd server that was serving content for CentOS users. So we decided to enforce TLS usage on those servers. But TLS can be used obviously on other things than a web server.

That's why we considered implementing something for our Postfix nodes. The interesting part is that it's really easy (depending of course at the security level one may want to reach/use). There are two parts in the postfix that can be configured :

  • outgoing mails (aka your server sends mail to other SMTPD servers)
  • incoming mails (aka remote clients/servers send mail to your postfix/smtpd server)

Let's start with the client/outgoing part : just adding those lines in your will automatically configure it to use TLS when possible, but otherwise fall back on clear if remote server doesn't support TLS :

# TLS - client part
smtp_tls_security_level = may
smtp_tls_loglevel = 1
smtp_tls_session_cache_database = btree:/var/lib/postfix/smtp_scache 

The interesting ...

CentOS 7 armv7hl build in progress

As more and more people were showing interest in CentOS on the ARM platform, we thought that it would be a good idea to start trying building CentOS 7 for that platform. Jim started with arm64/aarch64 and got an alpha build ready and installable.

On my end, I configured some armv7hl nodes, "donated" to the project by Scaleway. The first goal was to init some Plague builders to distribute the jobs on those nodes, which is now done. Then working on a "self-contained" buildroot , so that all other packages can be rebuilt only against that buildroot. So building first gcc from CentOS 7 (latest release, better arm support), then glibc, etc, etc ... That buildroot is now done and is available here.

Now the fun started (meaning that 4 armv7hl nodes are currently (re)building a bunch of SRPMS) and you can follow the status on the Arm-dev List if you're interested, or even better, if you're willing to join the party and have a look at the build logs for packages that failed to rebuild. The first target would be to have a "minimal" install working, so basically having sshd/yum working. Then try other things like ...

Hacking initrd.img for fun and profit

During my presentation at Loadays 2015 , I was mentioning some tips and tricks around Anaconda and kickstart, and so how to deploy CentOS , fully automated. I asked the audience about where to store the kickstart, that would be used then by anaconda to install CentOS (same works for RHEL/Fedora), and I got several answers, like "on the http server", or "on the ftp server", which is where most people will put their kickstart files. Some would generate those files files "dynamically" (through $cfgmgmt - I use Ansible with Jinja2 template for this - ) as a bonus point.

But it's not mandatory to host your kickstart file on a publicly available http/ftp/nfs server, and surely not when having to reinstall nodes not in the same DC. Within the infra, I sometimes have to reinstall remote nodes ("donated" to the Project) that are running CentOS 5 or 6 to 7. That's how injecting your ks file directly into the initrd.img really helps. (yes, so network server needed). Just as an intro, here is how you can remotely trigger a CentOS install, without any medium/iso/pxe environment : basically you just need to download the pxeboot images ...

More builders available for Koji/CBS

As you probably know, the CentOS Project now hosts the CBS effort, (aka Community Build System), that is used to build all packages for the CentOSSIGs.

There was already one physical node dedicated to Koji Web and Koji Hub, and another node dedicated to the build threads (koji-builder). As we have now more people building packages, we thought it was time to add more builders to the mix, and here we go: lists now two added machines that are dedicated to Koji/CBS.

Those added nodes have 2 * Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz with 8cores/sockets (+ Hyperthreading activated)  , and 32Gb of RAM. Let's see how the SIGs members will keep those builders busy and throwing a bunch of interesting packages for the CentOS Community :-) . Have a nice week-end