Arrfab's blog - Linuxhttps://arrfab.net/2017-05-16T00:00:00+02:00Some tips and tricks, mostly around CentOSLinking Foreman with Zabbix through MQTT2017-05-16T00:00:00+02:002017-05-16T00:00:00+02:00Fabian Arrotintag:arrfab.net,2017-05-16:/posts/2017/May/16/linking-foreman-with-zabbix-through-mqtt/<p>It's been a while since I thought about this design, but I finally had time to implement it the proper way, and "just in time" as I needed recently to migrate our <a href="https://www.theforeman.org">Foreman</a> instance to another host (from CentOS 6 to CentOS 7)</p>
<p>Within the CentOS Infra, we use Foreman as an <a href="https://docs.puppet.com/puppet/4.10/nodes_external.html#what-is-an-enc">ENC</a> for our Puppet environments (multiple ones). For full automation between configuration management and monitoring, you need some "glue". The idea is that whatever you describe at the configuration management level should be authoritative and so automatically configuring the monitoring solution you have in place in your Infra.</p>
<p>In our case, that means that we have Foreman/puppet on one side, and <a href="http://www.zabbix.com">Zabbix</a> on the other side. Let's see how we can "link" the two sides.</p>
<p>What I've seen so far is that you use <a href="https://docs.puppet.com/puppet/4.10/lang_exported.html">exported resources</a> on each node, store that in another <a href="https://docs.puppet.com/puppetdb/4.4/install_via_module.html">PuppetDB</a>, and then on the monitoring node, reapply all those resources. Problem with such solution is that it's "expensive" and when one thinks about it, a little bit strange to export the "knowledge" from Foreman back into another DB, and then let puppet compiles a <em>huge</em> catalog at the monitoring side, even if nothing …</p><p>It's been a while since I thought about this design, but I finally had time to implement it the proper way, and "just in time" as I needed recently to migrate our <a href="https://www.theforeman.org">Foreman</a> instance to another host (from CentOS 6 to CentOS 7)</p>
<p>Within the CentOS Infra, we use Foreman as an <a href="https://docs.puppet.com/puppet/4.10/nodes_external.html#what-is-an-enc">ENC</a> for our Puppet environments (multiple ones). For full automation between configuration management and monitoring, you need some "glue". The idea is that whatever you describe at the configuration management level should be authoritative and so automatically configuring the monitoring solution you have in place in your Infra.</p>
<p>In our case, that means that we have Foreman/puppet on one side, and <a href="http://www.zabbix.com">Zabbix</a> on the other side. Let's see how we can "link" the two sides.</p>
<p>What I've seen so far is that you use <a href="https://docs.puppet.com/puppet/4.10/lang_exported.html">exported resources</a> on each node, store that in another <a href="https://docs.puppet.com/puppetdb/4.4/install_via_module.html">PuppetDB</a>, and then on the monitoring node, reapply all those resources. Problem with such solution is that it's "expensive" and when one thinks about it, a little bit strange to export the "knowledge" from Foreman back into another DB, and then let puppet compiles a <em>huge</em> catalog at the monitoring side, even if nothing was changed.</p>
<p>One issue is also that in our Zabbix setup, we also have some nodes that aren't really managed by Foreman/puppet (but other automation around <a href="http://www.ansible.com">Ansible</a>, so I had to use an intermediate step that other tools can also use/abuse for the same reason.</p>
<p>The other reason also is that I admit that I'm a fan of "event driven" configuration change, so my idea was :</p>
<ul>
<li>update a host in Foreman (or groups of hosts, etc)</li>
<li>publish that change on a secure network through a message queue (so asynchronous so that it doesn't slow down the foreman update operation itself)</li>
<li>let Zabbix server know that change and apply it (like linking a template to a host)</li>
</ul>
<p>So the good news is that it can be done really easily with several components :</p>
<ul>
<li><a href="https://github.com/theforeman/foreman_hooks">foreman hooks</a></li>
<li><a href="https://mosquitto.org/">Mosquitto</a>, a very lightweight MQTT broker/pub/sub client</li>
<li><a href="https://github.com/usit-gd/zabbix-cli">zabbix-cli</a> , to let us talk to the Zabbix API</li>
</ul>
<p>Here is a small overview of the process :</p>
<p><img alt="Foreman MQTT Zabbix" src="/images/mqtt-foreman-zabbix.png" title="Foreman MQTT Zabbix"></p>
<h2>Foreman hooks</h2>
<p>Setting up foreman hooks is really easy: just install the pkg itself (tfm-rubygem-foreman_hooks.noarch), read the <a href="https://github.com/theforeman/foreman_hooks">Documentation</a>, and then create your scripts. There are some examples for Bash and python in the <a href="https://github.com/theforeman/foreman_hooks/tree/master/examples">examples</a> directory, but basically you just need to place some scripts at specific place[s].
In my case I wanted to "trigger" an event in the case of a node update (like adding a puppet class, or variable/paramater change) so I just had to place it under /usr/share/foreman/config/hooks/host/managed/update/.</p>
<p>One little remark though : if you put a new file, don't forget to restart foreman itself, so that it picks that hooks file, otherwise it would still be ignored and so not ran.</p>
<h2>Mosquitto</h2>
<p>Mosquitto itself is available in your favorite rpm repo, so installing it is a breeze. Reason why I selected mosquitto is that it's very lightweight (package size is under 200Kb), it supports TLS and ACL out-of-the box.</p>
<p>For an introduction to MQTT/Mosquitto, I'd suggest you to read <a href="https://twitter.com/jpmens">Jan-Piet Mens</a> dedicated <a href="http://jpmens.net/2013/02/25/lots-of-messages-mqtt-pub-sub-and-the-mosquitto-broker/">blog post</a> around it
I even admit that I discovered it by attending one of his talks on the topic, back in the <a href="http://loadays.org/">Loadays.org</a> days :-)</p>
<h2>Zabbix-cli</h2>
<p>While one can always discuss <a href="https://www.zabbix.com/documentation/3.0/manual/api">"Raw API"</a> with Zabbix, I found it useful to use a tool I was already using for various tasks around Zabbix : <a href="https://github.com/usit-gd/zabbix-cli">zabbix-cli</a>
For people interested in using it on CentOS 6 or 7, I built the packages and they are on <a href="https://cbs.centos.org/koji/packageinfo?packageID=4477">CBS</a></p>
<p>So I plumbed it in a systemd unit file that subscribe to specific MQTT topic, parse the needed informations (like hostname and zabbix templates to link, unlink, etc) and then it updates that in Zabbix itself (from the log output):</p>
<div class="highlight"><pre><span></span><span class="o">[</span><span class="n">+</span><span class="o">]</span><span class="w"> </span><span class="mi">20170516</span><span class="o">-</span><span class="mi">11</span><span class="err">:</span><span class="mi">43</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">Adding</span><span class="w"> </span><span class="n">zabbix</span><span class="w"> </span><span class="n">template</span><span class="w"> </span><span class="ss">"Template CentOS - https SSL Cert Check External"</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">host</span><span class="w"> </span><span class="ss">"dev-registry.lon1.centos.org"</span><span class="w"> </span>
<span class="o">[</span><span class="n">Done</span><span class="o">]</span><span class="err">:</span><span class="w"> </span><span class="n">Templates</span><span class="w"> </span><span class="n">Template</span><span class="w"> </span><span class="n">CentOS</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">https</span><span class="w"> </span><span class="n">SSL</span><span class="w"> </span><span class="n">Cert</span><span class="w"> </span><span class="k">Check</span><span class="w"> </span><span class="k">External</span><span class="w"> </span><span class="p">(</span><span class="err">{</span><span class="ss">"templateid"</span><span class="err">:</span><span class="ss">"10105"</span><span class="err">}</span><span class="p">)</span><span class="w"> </span><span class="n">linked</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">these</span><span class="w"> </span><span class="nl">hosts</span><span class="p">:</span><span class="w"> </span><span class="n">dev</span><span class="o">-</span><span class="n">registry</span><span class="p">.</span><span class="n">lon1</span><span class="p">.</span><span class="n">centos</span><span class="p">.</span><span class="n">org</span><span class="w"> </span><span class="p">(</span><span class="err">{</span><span class="ss">"hostid"</span><span class="err">:</span><span class="ss">"10174"</span><span class="err">}</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>Cool, so now I don't have to worry about forgetting to tie a zabbix template to a host , as it's now done automatically. No need to say that the deployment of those tools was of course automated and coming from Puppet/foreman :-)</p>Zabbix, selinux and CentOS 7.3.16112016-11-25T00:00:00+01:002016-11-25T00:00:00+01:00Fabian Arrotintag:arrfab.net,2016-11-25:/posts/2016/Nov/25/zabbix-selinux-and-centos-731611/<p>If you're using CentOS, you probably noticed that we have a <a href="https://wiki.centos.org/AdditionalResources/Repositories/CR">CR repository</a> containing all the built packages for the next minor release, so that people can "opt-in" and already use those packages, before they are released with the full installable tree and iso images.</p>
<p>Using those packages on a subset of your nodes can be interesting, as it permits you to catch some errors/issues/conflicts before the official release (and so symlink on mirrors being changed to that new major.minor version)</p>
<p>For example, I tested myself some roles and found an issue with zabbix-agent refusing to start on a node fully updated/rebooted with CR pkgs (so what will become 7.3.1611 release). The issue was due to selinux denying something (that was allowed in previous policy)</p>
<p>Here is what selinux had to say about it : </p>
<div class="highlight"><pre><span></span><span class="nv">type</span><span class="o">=</span><span class="nv">AVC</span> <span class="nv">msg</span><span class="o">=</span><span class="nv">audit</span><span class="ss">(</span><span class="mi">1480001303</span>.<span class="mi">440</span>:<span class="mi">2626</span><span class="ss">)</span>: <span class="nv">avc</span>: <span class="nv">denied</span> { <span class="nv">setrlimit</span> } <span class="k">for</span> <span class="nv">pid</span><span class="o">=</span><span class="mi">22682</span> <span class="nv">comm</span><span class="o">=</span><span class="s2">"</span><span class="s">zabbix_agentd</span><span class="s2">"</span> <span class="nv">scontext</span><span class="o">=</span><span class="nv">system_u</span>:<span class="nv">system_r</span>:<span class="nv">zabbix_agent_t</span>:<span class="nv">s0</span> <span class="nv">tcontext</span><span class="o">=</span><span class="nv">system_u</span>:<span class="nv">system_r</span>:<span class="nv">zabbix_agent_t</span>:<span class="nv">s0</span> <span class="nv">tclass</span><span class="o">=</span><span class="nv">process</span>
</pre></div>
<p>It's true that there was an update for selinux policy : from selinux-policy-3.13.1-60.el7_2.9.noarch to selinux-policy-3.13.1-102.el7.noarch.</p>
<p>What's interesting is that I found the reported issue at …</p><p>If you're using CentOS, you probably noticed that we have a <a href="https://wiki.centos.org/AdditionalResources/Repositories/CR">CR repository</a> containing all the built packages for the next minor release, so that people can "opt-in" and already use those packages, before they are released with the full installable tree and iso images.</p>
<p>Using those packages on a subset of your nodes can be interesting, as it permits you to catch some errors/issues/conflicts before the official release (and so symlink on mirrors being changed to that new major.minor version)</p>
<p>For example, I tested myself some roles and found an issue with zabbix-agent refusing to start on a node fully updated/rebooted with CR pkgs (so what will become 7.3.1611 release). The issue was due to selinux denying something (that was allowed in previous policy)</p>
<p>Here is what selinux had to say about it : </p>
<div class="highlight"><pre><span></span><span class="nv">type</span><span class="o">=</span><span class="nv">AVC</span> <span class="nv">msg</span><span class="o">=</span><span class="nv">audit</span><span class="ss">(</span><span class="mi">1480001303</span>.<span class="mi">440</span>:<span class="mi">2626</span><span class="ss">)</span>: <span class="nv">avc</span>: <span class="nv">denied</span> { <span class="nv">setrlimit</span> } <span class="k">for</span> <span class="nv">pid</span><span class="o">=</span><span class="mi">22682</span> <span class="nv">comm</span><span class="o">=</span><span class="s2">"</span><span class="s">zabbix_agentd</span><span class="s2">"</span> <span class="nv">scontext</span><span class="o">=</span><span class="nv">system_u</span>:<span class="nv">system_r</span>:<span class="nv">zabbix_agent_t</span>:<span class="nv">s0</span> <span class="nv">tcontext</span><span class="o">=</span><span class="nv">system_u</span>:<span class="nv">system_r</span>:<span class="nv">zabbix_agent_t</span>:<span class="nv">s0</span> <span class="nv">tclass</span><span class="o">=</span><span class="nv">process</span>
</pre></div>
<p>It's true that there was an update for selinux policy : from selinux-policy-3.13.1-60.el7_2.9.noarch to selinux-policy-3.13.1-102.el7.noarch.</p>
<p>What's interesting is that I found the reported issue at Zabbix side, but for zabbix-server (here it's the agent, server is running fine) : <a href="https://support.zabbix.com/browse/ZBX-10542">ZBX-10542</a></p>
<p>Clearly something that was working before and now denied, so I created a <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1398721">bug report</a> and hopefully one fix will come in an updated selinux-policy package. But I doubt that it will be available soon.</p>
<p>So in the mean time, what you have to do is :</p>
<ul>
<li>either put zabbix_agent_t into permissive mode with <code>semanage permissive -a zabbix_agent_t</code></li>
<li>either build and distribute a custom selinux policy in your infra (preferred method for me)</li>
</ul>
<p>For those interested, the following .te (type enforcement) will allow you to build a custom .pp selinux policy file (that you can load with semodule) : </p>
<div class="highlight"><pre><span></span><span class="nt">module</span> <span class="nt">local-zabbix</span> <span class="nt">1</span><span class="p">.</span><span class="nc">0</span><span class="o">;</span>
<span class="nt">require</span> <span class="p">{</span>
<span class="err">type</span> <span class="err">zabbix_agent_t</span><span class="p">;</span>
<span class="err">class</span> <span class="err">process</span> <span class="err">setrlimit</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">#</span><span class="o">=============</span> <span class="nt">zabbix_agent_t</span> <span class="o">==============</span>
<span class="nt">allow</span> <span class="nt">zabbix_agent_t</span> <span class="nt">self</span><span class="p">:</span><span class="nd">process</span> <span class="nt">setrlimit</span><span class="o">;</span>
</pre></div>
<p>You can now use your configuration management platform to distribute that built .pp policy (you don't need to build it on every node). I'll not dive into details, but I wrote <a href="https://people.centos.org/arrfab/Events/Loadays-2014/managing%20selinux%20with%20your%20cfgmgmt%20solution.pdf">some slides</a> around this (for Ansible and Puppet) for a talk I gave some time ago, so feel free to read those, especially the last slides (with examples)</p>(ab)using Alias for Zabbix2016-10-21T00:00:00+02:002016-10-21T00:00:00+02:00Fabian Arrotintag:arrfab.net,2016-10-21:/posts/2016/Oct/21/abusing-alias-for-zabbix/<p>It's not a secret that we use Zabbix to monitor the CentOS.org infra. That's even a reason why we (re)build it for some other architectures, including aarch64,ppc64,ppc64le on <a href="https://cbs.centos.org/koji/packageinfo?packageID=15">CBS</a> and also <a href="http://armv7.dev.centos.org/repodir/c7-extras-1/zabbix/">armhfp</a></p>
<p>There are really cool things in Zabbix, including <a href="https://www.zabbix.com/documentation/3.0/manual/discovery/low_level_discovery">Low-Level Discovery</a>. With such discovery, you can create items/prototypes/triggers that will be applied "automagically" for each discovered network interface, or mounted filesystem. For example, the default template (if you still use it) has such item prototypes and also graph for each discovered network interface and show you the bandwidth usage on those network interfaces.</p>
<p>But what happens if you suddenly want to for example to create some <a href="https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/calculated">calculated item</a> on top of those ? Well, the issue is that from one node to the other, interface name can be eth0, or sometimes eth1, and with CentOS 7 things started to also move to the new naming scheme, so you can have something like enp4s0f0. I wanted to create a template that would fit-them-all, so I had a look at calculated item and thought "well, easy : let's have that calculated item use a user macro that would define the name of the interface we really want …</p><p>It's not a secret that we use Zabbix to monitor the CentOS.org infra. That's even a reason why we (re)build it for some other architectures, including aarch64,ppc64,ppc64le on <a href="https://cbs.centos.org/koji/packageinfo?packageID=15">CBS</a> and also <a href="http://armv7.dev.centos.org/repodir/c7-extras-1/zabbix/">armhfp</a></p>
<p>There are really cool things in Zabbix, including <a href="https://www.zabbix.com/documentation/3.0/manual/discovery/low_level_discovery">Low-Level Discovery</a>. With such discovery, you can create items/prototypes/triggers that will be applied "automagically" for each discovered network interface, or mounted filesystem. For example, the default template (if you still use it) has such item prototypes and also graph for each discovered network interface and show you the bandwidth usage on those network interfaces.</p>
<p>But what happens if you suddenly want to for example to create some <a href="https://www.zabbix.com/documentation/3.0/manual/config/items/itemtypes/calculated">calculated item</a> on top of those ? Well, the issue is that from one node to the other, interface name can be eth0, or sometimes eth1, and with CentOS 7 things started to also move to the new naming scheme, so you can have something like enp4s0f0. I wanted to create a template that would fit-them-all, so I had a look at calculated item and thought "well, easy : let's have that calculated item use a user macro that would define the name of the interface we really want to gather stats from ..." .. but it seems I was wrong. Zabbix <a href="https://www.zabbix.com/documentation/3.0/manual/config/macros/usermacros">user macros</a> can be used in multiple places, but not <a href="https://support.zabbix.com/browse/ZBX-11373">everywhere</a>. (It seems that I wasn't the only one not understanding the doc coverage for this, but at least that bug report will have an effect on the doc to clarify this)</p>
<p>That's when I discussed this in #zabbix (on irc.freenode.net) that <a href="https://twitter.com/real_richlv">RichLV</a> pointed me to something that could be interesting for my case : <a href="https://www.zabbix.com/documentation/3.0/manual/appendix/config/zabbix_agentd">Alias</a>. I must admit that it's the first time I was hearing about it, and I don't even know when it landed in Zabbix (or if I just overlooked it at first sight).</p>
<p>So cool, now I can just have our config mgmt pushing for example a /etc/zabbix/zabbix_agentd.d/interface-alias.conf file that looks like this and reload zabbix-agent : </p>
<div class="highlight"><pre><span></span><span class="nv">Alias</span><span class="o">=</span><span class="nv">net</span>.<span class="k">if</span>.<span class="nv">default</span>.<span class="nv">out</span>:<span class="nv">net</span>.<span class="k">if</span>.<span class="nv">out</span>[<span class="nv">enp4s0f0</span>]
<span class="nv">Alias</span><span class="o">=</span><span class="nv">net</span>.<span class="k">if</span>.<span class="nv">default</span>.<span class="nv">in</span>:<span class="nv">net</span>.<span class="k">if</span>.<span class="nv">in</span>[<span class="nv">enp4s0f0</span>]
</pre></div>
<p>That means that now, whatever the interface name will be (as puppet in our case will create that file for us) , we'll be able to get values from net.if.default.out and net.if.default.in keys, automatically. Cool</p>
<p>That also means that if you want to aggregate all this into a single key for a group of nodes (and so graph that too), you can do something always referencing those new keys (example for the total outgoing bandwidth for a group of hosts) :</p>
<div class="highlight"><pre><span></span><span class="nv">grpsum</span>[<span class="s2">"</span><span class="s">Your group name</span><span class="s2">"</span>,<span class="s2">"</span><span class="s">net.if.default.out</span><span class="s2">"</span>,<span class="nv">last</span>,<span class="mi">0</span>]
</pre></div>
<p>And from that point, you can easily also configure triggers, and graphs too.
Now going back to work on some other calculated items for total bandwith usage for a period of time and triggers based on some max_bw_usage user macro.</p>CentOS Infra public service dashboard2016-09-22T00:00:00+02:002016-09-22T00:00:00+02:00Fabian Arrotintag:arrfab.net,2016-09-22:/posts/2016/Sep/22/centos-infra-public-service-dashboard/<p>As soon as you're running some IT services, there is one thing that you already know : you'll have <a href="https://en.wikipedia.org/wiki/Downtime">downtimes</a>, despite all your efforts to avoid those...</p>
<p>As the old joke says : <code>"What's up ?" asked the Boss. "Hopefully everything !" answered the SysAdmin guy ....</code></p>
<p>You probably know that the CentOS infra is itself widespread, and subject to quick move too. Recently we had to <a href="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html">announce</a> an important DC relocation that impacts some of our crucial and publicly facing services. That one falls in the "scheduled and known outages" category, and can be prepared. For such "downtime" we always announced that through several mediums, like sending a mail to the centos-announce, centos-devel (and in this case , also to the ci-users) <a href="https://lists.centos.org">mailing lists</a>. But even when we announce that in advance, some people forget about it, or people using (sometimes "indirectly") the concerned service are surprized and then ask about it (usually in #centos or #centos-devel on irc.freenode.net).</p>
<p>In parallel to those "scheduled outages", we have also the worst ones : the unscheduled ones. For those ones, depending on the impact/criticity of the impacted service, and also the estimated <a href="https://en.wikipedia.org/wiki/Recovery_time_objective">RTO</a>, we also send a mail to the concerned mailing lists (or not …</p><p>As soon as you're running some IT services, there is one thing that you already know : you'll have <a href="https://en.wikipedia.org/wiki/Downtime">downtimes</a>, despite all your efforts to avoid those...</p>
<p>As the old joke says : <code>"What's up ?" asked the Boss. "Hopefully everything !" answered the SysAdmin guy ....</code></p>
<p>You probably know that the CentOS infra is itself widespread, and subject to quick move too. Recently we had to <a href="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html">announce</a> an important DC relocation that impacts some of our crucial and publicly facing services. That one falls in the "scheduled and known outages" category, and can be prepared. For such "downtime" we always announced that through several mediums, like sending a mail to the centos-announce, centos-devel (and in this case , also to the ci-users) <a href="https://lists.centos.org">mailing lists</a>. But even when we announce that in advance, some people forget about it, or people using (sometimes "indirectly") the concerned service are surprized and then ask about it (usually in #centos or #centos-devel on irc.freenode.net).</p>
<p>In parallel to those "scheduled outages", we have also the worst ones : the unscheduled ones. For those ones, depending on the impact/criticity of the impacted service, and also the estimated <a href="https://en.wikipedia.org/wiki/Recovery_time_objective">RTO</a>, we also send a mail to the concerned mailing lists (or not).</p>
<p>So we just decided to show a very simple and public dashboard for the CentOS Infra, but only covering the publicly facing services, to have a quick overview of that part of the Infra. It's now live and hosted on <a href="https://status.centos.org">https://status.centos.org</a>.</p>
<p>We use <a href="http://www.zabbix.com">Zabbix</a> to monitor our Infra (so we build it for multiple arches, like x86_64,i386,ppc64,ppc64le,aarch64 and also armhfp) , including through remote zabbix <a href="https://www.zabbix.com/documentation/3.0/manual/concepts/proxy">proxies</a> (because of our "distributed" network setup right now, with machines all around the world).
For some of those services listed on status.centos.org, we can "manually" announce a downtime/maintenance period, but Zabbix also updates on its own that dashboard.
The simple way to link those together was to use zabbix <a href="https://www.zabbix.com/documentation/3.0/manual/config/notifications/media/script">custom alertscripts</a> and you can even customize those to send specific <a href="https://www.zabbix.com/documentation/3.0/manual/appendix/macros/supported_by_location">macros</a> and have that alertscript just parsing and then updating the dashboard.</p>
<p>We hope to enhance that dashboard in the future, but it's a good start, and I have to thank again <a href="https://twitter.com/puiterwijkFP">Patrick Uiterwijk</a> who wrote that <a href="https://git.fedorahosted.org/git/fedora-status">tool</a> for Fedora initially (and that we adapted to our needs).</p>Generating multiple certificates with Letsencrypt from a single instance2016-05-03T00:00:00+02:002016-05-03T00:00:00+02:00Fabian Arrotintag:arrfab.net,2016-05-03:/posts/2016/May/03/generating-multiple-certificates-with-letsencrypt-from-a-single-instance/<p>Recently I was discussing with some people about TLS everywhere, and we then started to discuss about the <a href="https://letsencrypt.org/">Letsencrypt</a> initiative.
I had to admit that I just tested it some time ago (just for "fun") but I suddenly looked at it from a different angle : while the most used case is when you install/run the letsencrypt client on your node to directly configure it, I have to admit that it's something I didn't want to have to deal with. I still think that proper web server configuration has to happen through cfgmgmt, and not through another process. (and same for the key/cert distribution, something for a different blog post maybe).</p>
<p>If so you're (pushing|pulling) automatically your web servers configuration from $cfgmgmt, but that you want to use/deploy TLS certificates signed by letsencrypt, what can you do ? Well, the good news is that you don't have to be forced to let the letsencrypt client touch your configuration at all : you can use the "certonly" option to just generate the private key locally, send the <a href="https://en.wikipedia.org/wiki/Certificate_signing_request">csr</a> and get the signed cert back (and the whole chain too)
One thing to know about letsencrypt is that the validation/verification …</p><p>Recently I was discussing with some people about TLS everywhere, and we then started to discuss about the <a href="https://letsencrypt.org/">Letsencrypt</a> initiative.
I had to admit that I just tested it some time ago (just for "fun") but I suddenly looked at it from a different angle : while the most used case is when you install/run the letsencrypt client on your node to directly configure it, I have to admit that it's something I didn't want to have to deal with. I still think that proper web server configuration has to happen through cfgmgmt, and not through another process. (and same for the key/cert distribution, something for a different blog post maybe).</p>
<p>If so you're (pushing|pulling) automatically your web servers configuration from $cfgmgmt, but that you want to use/deploy TLS certificates signed by letsencrypt, what can you do ? Well, the good news is that you don't have to be forced to let the letsencrypt client touch your configuration at all : you can use the "certonly" option to just generate the private key locally, send the <a href="https://en.wikipedia.org/wiki/Certificate_signing_request">csr</a> and get the signed cert back (and the whole chain too)
One thing to know about letsencrypt is that the validation/verification process isn't the one that you can see in most of the companies providing CA/signing capabilities : as there is no ID/Paper verification (or something else) , the only validation for the domain/sub-domain that you want to generate a certificate for happens over http request (basically creating a file with a challenge , process a request from their "ACME" server[s] to retrieve that file back, and validate content)</p>
<p>So what are our options then ? The letsencrypt documentation mentions several <a href="http://letsencrypt.readthedocs.io/en/latest/using.html#plugins">plugins</a> like manual (involves you to then create the file with the challenge answer to the webserver, then launching the validation process) , or standalone (doesn't work if you already have a httpd/nginx process as there will be a port conflict) , or even webroot (working fine as it will then just write the file itself under /.well-kwown/ under the DocumentRoot)</p>
<p>The webroot seems easy, but as said, we don't want to even install letsencrypt on the web server[s]. Even worse, suppose (and that's the case I had in mind) that you have multiple web nodes configured in a kind of <a href="https://en.wikipedia.org/wiki/Content_delivery_network">CDN</a> way : you don't want to distribute that file on all the nodes for validation/verification (when using the "manual" plugin) and you'd have to do it on <em>all</em> the nodes (as you don't know in advance which one will be verified by the ACME server)</p>
<p>So what about something centralized (where you'd run the letsencrypt client locally) for all your certs (including some with <a href="https://en.wikipedia.org/wiki/SubjectAltName">SANs</a> ) in a transpartent way ? I so thought about something like this :</p>
<p><img alt="Single Letsencrypt node" src="/images/central-le-process.png" title="central letsencrypt node"></p>
<p>The idea would be to :</p>
<ul>
<li>use a central node : let's call it central.domain.com (vm, docker container, make-your-choice-here) to launch the letsencrypt client</li>
<li>have the ACME server hitting transparently one of the web servers without any changed/uploaded file</li>
<li>the server getting the GET request for that file using the letsencrypt central node as a backend node</li>
<li>ACME server being happy and so signed certificates being available automatically on the centralize letsencrypt node.</li>
</ul>
<p>The good news is that it's possible and even really easy to implement, through <a href="https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass">ProxyPass</a> (for httpd/Apache web server) or <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass">proxy_pass</a> (for nginx based setup)</p>
<p>For example, for the httpd vhost config for sub1.domain.com (three nodes in our example) we can just add this in the .conf file :</p>
<div class="highlight"><pre><span></span><span class="nt"><Location</span> <span class="err">"/.well-known/"</span><span class="nt">></span>
ProxyPass "http://central.domain.com/.well-known/"
<span class="nt"></Location></span>
</pre></div>
<p>So now, once in place everywhere, you can generate the cert for that domain on the central letsencrypt node (assuming that httpd is running on that node, and reachable from the "frontend" nodes, and that /var/www/html is indeed the DocumentRoot (default) for httpd on that node): </p>
<div class="highlight"><pre><span></span><span class="n">letsencrypt</span><span class="w"> </span><span class="n">certonly</span><span class="w"> </span><span class="c1">--webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email you@domain.com -d sub1.domain.com</span>
</pre></div>
<p>Same if you run nginx instead (let's assume this for sub2.domain.com and sub3.domain.com) , you just have to add a snippet in your vhost .conf file (and before the / definition too): </p>
<div class="highlight"><pre><span></span><span class="nt">location</span> <span class="o">/</span><span class="p">.</span><span class="nc">well-known</span><span class="o">/</span> <span class="p">{</span>
<span class="err">proxy_pass</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">central</span><span class="o">.</span><span class="n">domain</span><span class="o">.</span><span class="n">com</span><span class="o">/.</span><span class="n">well-known</span><span class="o">/</span> <span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>And then on the central node, do the same thing, but you can add multiple -d for multiple SubjectAltName in the same cert :</p>
<div class="highlight"><pre><span></span><span class="n">letsencrypt</span><span class="w"> </span><span class="n">certonly</span><span class="w"> </span><span class="c1">--webroot --webroot-path /var/www/html --manual-public-ip-logging-ok --agree-tos --email you@domain.com -d sub2.domain.com -d sub3.domain.com</span>
</pre></div>
<p>Transparent, smart, easy to do and even something you can deploy when you need to renew, and then remove to be back with initial config files too (if you don't want to have those ProxyPass directives active all the time)</p>
<p>The only thing you have also to know is that once you have proper TLS in place, it's usually better to redirect transpartently all requests to your http server to the https version. Most of the people will do that (next example for httpd/apache) like this : </p>
<div class="highlight"><pre><span></span> <span class="n">RewriteEngine</span> <span class="k">On</span>
<span class="n">RewriteCond</span> <span class="o">%</span><span class="err">{</span><span class="n">HTTPS</span><span class="err">}</span> <span class="o">!=</span><span class="k">on</span>
<span class="n">RewriteRule</span> <span class="o">^/?</span><span class="p">(.</span><span class="o">*</span><span class="p">)</span> <span class="n">https</span><span class="p">:</span><span class="o">//%</span><span class="err">{</span><span class="k">SERVER_NAME</span><span class="err">}</span><span class="o">/</span><span class="err">$</span><span class="mi">1</span> <span class="p">[</span><span class="n">R</span><span class="p">,</span><span class="n">L</span><span class="p">]</span>
</pre></div>
<p>It's good, but when you'll renew the certificate, you'll probably just want to be sure that the GET request for /.well-known/* will continue to work over http (from the ACME server) so we can tune a little bit those rules (<a href="https://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewritecond">RewriteCond</a> are cumulatives so it will not be redirect if url starts with .well-known: </p>
<div class="highlight"><pre><span></span> <span class="n">RewriteEngine</span> <span class="k">On</span>
<span class="n">RewriteCond</span> <span class="err">$</span><span class="mi">1</span> <span class="o">!^</span><span class="p">.</span><span class="n">well</span><span class="o">-</span><span class="n">known</span>
<span class="n">RewriteCond</span> <span class="o">%</span><span class="err">{</span><span class="n">HTTPS</span><span class="err">}</span> <span class="o">!=</span><span class="k">on</span>
<span class="n">RewriteRule</span> <span class="o">^/?</span><span class="p">(.</span><span class="o">*</span><span class="p">)</span> <span class="n">https</span><span class="p">:</span><span class="o">//%</span><span class="err">{</span><span class="k">SERVER_NAME</span><span class="err">}</span><span class="o">/</span><span class="err">$</span><span class="mi">1</span> <span class="p">[</span><span class="n">R</span><span class="p">,</span><span class="n">L</span><span class="p">]</span>
</pre></div>
<p>Different syntax, but same principle for nginx : (also snippet, not full configuration file for that server/vhost):</p>
<div class="highlight"><pre><span></span><span class="nt">location</span> <span class="o">/</span><span class="p">.</span><span class="nc">well-known</span><span class="o">/</span> <span class="p">{</span>
<span class="err">proxy_pass</span> <span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">central</span><span class="o">.</span><span class="n">domain</span><span class="o">.</span><span class="n">com</span><span class="o">/.</span><span class="n">well-known</span><span class="o">/</span> <span class="p">;</span>
<span class="p">}</span>
<span class="nt">location</span> <span class="o">/</span> <span class="p">{</span>
<span class="err">rewrite</span> <span class="err">^</span> <span class="n">https</span><span class="p">:</span><span class="o">//</span><span class="err">$</span><span class="n">server_name</span><span class="err">$</span><span class="n">request_uri</span><span class="o">?</span> <span class="n">permanent</span><span class="p">;</span>
<span class="p">}</span>
</pre></div>
<p>Hope that you'll have found that useful, especially if you don't want to deploy letsencrypt everywhere but still use it to generate locally your keys/certs. Once done, you can then distribute/push/pull (depending on your cfgmgmt) those files and don't forget to also implement proper monitoring for cert validity and automation around that too (consider that your homework)</p>IPv6 connectivity status within the CentOS.org infra2016-04-29T00:00:00+02:002016-04-29T00:00:00+02:00Fabian Arrotintag:arrfab.net,2016-04-29:/posts/2016/Apr/29/ipv6-connectivity-status-within-the-centosorg-infra/<p>Recently, some people started to ask proper IPv6/AAAA record for some of our public mirror infrastructure, like mirror.centos.org, and also <a href="https://wiki.centos.org/HowTos/CreatePublicMirrors">msync.centos.org</a> </p>
<p>Reason is that a lot of people are now using IPv6 wherever possible and from a CentOS point of view, we should ensure that everybody can have content over (legacy) ipv4 and ipv6. Funny that I call ipv4 "legacy" as we still have to admit that it's still the default everywhere, even in 2016 with the available pools now exhausted.</p>
<p>While we had already some AAAA records for some of our public nodes (like <a href="https://www.centos.org">www.centos.org</a> as an example), I started to "chase" after proper and native ipv6 connectivity for our nodes.
That's where I had to take contact with all our valuable <a href="https://www.centos.org/sponsors">sponsors</a>. First thing to say is that we'd like to thank them all for their support for the CentOS Project over the years : it wouldn't have been possible to deliver multiple terrabytes of data per month without their sponsorship !</p>
<p>WRT ipv6 connectivity that's where the results of my quest where really different : while some DCs support ipv6 natively, and even answer you in 5 minutes when asking for a /64 …</p><p>Recently, some people started to ask proper IPv6/AAAA record for some of our public mirror infrastructure, like mirror.centos.org, and also <a href="https://wiki.centos.org/HowTos/CreatePublicMirrors">msync.centos.org</a> </p>
<p>Reason is that a lot of people are now using IPv6 wherever possible and from a CentOS point of view, we should ensure that everybody can have content over (legacy) ipv4 and ipv6. Funny that I call ipv4 "legacy" as we still have to admit that it's still the default everywhere, even in 2016 with the available pools now exhausted.</p>
<p>While we had already some AAAA records for some of our public nodes (like <a href="https://www.centos.org">www.centos.org</a> as an example), I started to "chase" after proper and native ipv6 connectivity for our nodes.
That's where I had to take contact with all our valuable <a href="https://www.centos.org/sponsors">sponsors</a>. First thing to say is that we'd like to thank them all for their support for the CentOS Project over the years : it wouldn't have been possible to deliver multiple terrabytes of data per month without their sponsorship !</p>
<p>WRT ipv6 connectivity that's where the results of my quest where really different : while some DCs support ipv6 natively, and even answer you in 5 minutes when asking for a /64 subnet to be allocated , some other aren't still ipv6 ready : For the worst case the answer was "nothing ready and no plan for that" or for sometimes the received answer was something like "it's on the roadmap for 2018/2019").</p>
<p>The good news is that ~30% of our nodes behind msync.centos.org have now ipv6 connectivity, so the next step is now to test our various configurations (distributed by puppet) and then also our GeoIP redirection (done at the <a href="http://www.powerdns.com">PowerDNS</a> level for such records, for which we'll also then add proper AAAA record)</p>
<p>Hopefully we'll have that tested and then announced soon, and also for other public services that we're providing to you.</p>
<p>Stay tuned for more info about ipv6 deployment within centos.org !</p>Kernel 3.10.0-327 issue on AMD Neo processor2015-12-15T00:00:00+01:002015-12-15T00:00:00+01:00Fabian Arrotintag:arrfab.net,2015-12-15:/posts/2015/Dec/15/kernel-3100-327-issue-on-amd-neo-processor/<p>As CentOS 7 (1511) was released, I thought it would be a good idea to update several of my home machines (including kids' workstations) with that version, and also newer kernel.
Usually that's just a smooth operation, but sometimes some backported features/new features, especially in the kernel, can lead to some strange issues.
That's what happened for my older Thinkpad Edge : That's a cheap/small thinkpad that Lenovo did several years ago ( circa 2011 ), and that I used a lot just when travelling, as it only has a <a href="http://www.notebookcheck.net/AMD-Athlon-II-Neo-K325-Notebook-Processor.33886.0.html">AMD Athlon(tm) II Neo K345 Dual-Core Processor</a>.
So basically not a lot of horse power, but still something convenient just to read your mails, remotely connect through ssh, or browse the web.
When rebooting on the newer kernel, it panics directly.</p>
<p>Two bug reports are open for this, one on the <a href="https://bugs.centos.org/view.php?id=9860">CentOS Bug tracker</a>, linked also to the <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1285235">upstream one</a>. Current status is that there is no kernel update that will fix this, but there is a easy to implement workaround : </p>
<ul>
<li>boot with the initcall_blacklist=clocksource_done_booting kernel parameter added (or reboot on previous kernel)</li>
<li>once booted, add the same parameter at the end of the GRUB_CMDLINE_LINUX=" .." line , in the file …</li></ul><p>As CentOS 7 (1511) was released, I thought it would be a good idea to update several of my home machines (including kids' workstations) with that version, and also newer kernel.
Usually that's just a smooth operation, but sometimes some backported features/new features, especially in the kernel, can lead to some strange issues.
That's what happened for my older Thinkpad Edge : That's a cheap/small thinkpad that Lenovo did several years ago ( circa 2011 ), and that I used a lot just when travelling, as it only has a <a href="http://www.notebookcheck.net/AMD-Athlon-II-Neo-K325-Notebook-Processor.33886.0.html">AMD Athlon(tm) II Neo K345 Dual-Core Processor</a>.
So basically not a lot of horse power, but still something convenient just to read your mails, remotely connect through ssh, or browse the web.
When rebooting on the newer kernel, it panics directly.</p>
<p>Two bug reports are open for this, one on the <a href="https://bugs.centos.org/view.php?id=9860">CentOS Bug tracker</a>, linked also to the <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1285235">upstream one</a>. Current status is that there is no kernel update that will fix this, but there is a easy to implement workaround : </p>
<ul>
<li>boot with the initcall_blacklist=clocksource_done_booting kernel parameter added (or reboot on previous kernel)</li>
<li>once booted, add the same parameter at the end of the GRUB_CMDLINE_LINUX=" .." line , in the file /etc/default/grub</li>
<li>as root, run <code>grub2-mkconfig -o /etc/grub2.conf</code></li>
</ul>
<p>Hope it can help others too</p>Kernel IO wait and megaraid controller2015-12-01T00:00:00+01:002015-12-01T00:00:00+01:00Fabian Arrotintag:arrfab.net,2015-12-01:/posts/2015/Dec/01/kernel-io-wait-and-megaraid-controller/<p>Last friday, while working on something else (working on "CentOS 7 userland" release for Armv7hl boards), I got notifications coming from our <a href="http://www.zabbix.org">Zabbix</a> monitoring instance complaining about <a href="https://www.zabbix.com/documentation/2.4/manual/web_monitoring">web scenarios</a> failing (errors due to time outs) , and also then also about "Disk I/O is overloaded" triggers (checking the cpu iowait time). Usually you'd verify what happens in the Virtual Machine itself, but even connecting to the VM was difficult and slow. But once connected, nothing strange, and no real activity , not even on the disk (Plenty of tools for this, but <a href="http://guichaz.free.fr/iotop/">iotop</a> is helpful to see which process is reading/writing to the disk in that case), but iowait was almost at 100%).</p>
<p>As said, it was happening suddenly for all Virtual Machines on the same hypervisor (CentOS 6 x86_64 KVM host), and even the hypervisor was suddenly complaining (but less in comparison with the VMs) about iowait too. So obviously, it wasn't really something not being optimized at the hypervisor/VMS, but something else. That rang a bell, as if you have a raid controller, and that battery for example is to be replaced, the controller can decide to stop all read/write cache, so slowing down all IOs …</p><p>Last friday, while working on something else (working on "CentOS 7 userland" release for Armv7hl boards), I got notifications coming from our <a href="http://www.zabbix.org">Zabbix</a> monitoring instance complaining about <a href="https://www.zabbix.com/documentation/2.4/manual/web_monitoring">web scenarios</a> failing (errors due to time outs) , and also then also about "Disk I/O is overloaded" triggers (checking the cpu iowait time). Usually you'd verify what happens in the Virtual Machine itself, but even connecting to the VM was difficult and slow. But once connected, nothing strange, and no real activity , not even on the disk (Plenty of tools for this, but <a href="http://guichaz.free.fr/iotop/">iotop</a> is helpful to see which process is reading/writing to the disk in that case), but iowait was almost at 100%).</p>
<p>As said, it was happening suddenly for all Virtual Machines on the same hypervisor (CentOS 6 x86_64 KVM host), and even the hypervisor was suddenly complaining (but less in comparison with the VMs) about iowait too. So obviously, it wasn't really something not being optimized at the hypervisor/VMS, but something else. That rang a bell, as if you have a raid controller, and that battery for example is to be replaced, the controller can decide to stop all read/write cache, so slowing down all IOs going to the disk. </p>
<p>At first sight, there was no HDD issue, and array/logical volume was working fine (no failed HDD in that RAID10 volume), so it was time to dive deeper into analysis.</p>
<p>That server has the following raid adapter : </p>
<div class="highlight"><pre><span></span><span class="mi">03</span><span class="err">:</span><span class="mf">00.0</span><span class="w"> </span><span class="n">RAID</span><span class="w"> </span><span class="n">bus</span><span class="w"> </span><span class="nl">controller</span><span class="p">:</span><span class="w"> </span><span class="n">LSI</span><span class="w"> </span><span class="n">Logic</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="n">Symbios</span><span class="w"> </span><span class="n">Logic</span><span class="w"> </span><span class="n">MegaRAID</span><span class="w"> </span><span class="n">SAS</span><span class="w"> </span><span class="mi">2108</span><span class="w"> </span><span class="o">[</span><span class="n">Liberator</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="n">rev</span><span class="w"> </span><span class="mi">03</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>That means that you need to use the <a href="http://www.avagotech.com/cs/Satellite?pagename=AVG2/searchLayout&SearchKeyWord=megacli&searchType=type-AVG_Document_C~Downloads&locale=avg_en&srchradio=null">MegaCLI</a> tool for that.</p>
<p>A quick <strong><code>MegaCli64 -ShowSummary -a0</code></strong> showed me that indeed the underlying disk were active but I got my attention caught by the fact that there was a "Patrol Read" operation in progress on a disk. I then discovered a useful (bookmarked, as it's a gold mine) <a href="http://fibrevillage.com/storage/175-lsi-megaraid-patrol-read-and-consistency-check">page</a> explaining the issue with default settings and the "Patrol Read" operation.
While it seems a good idea to scan the disks in the background to discover disk error in advance (PFA), the default setting is really not optimized : (from that website) : "will take up to 30% of IO resources"</p>
<p>I decided to stop the currently running patrol read process with <strong><code>MegaCli64 -AdpPR -Stop -aALL</code></strong> and I directly saw Virtual Machines (and hypervisor) iowait going back to normal mode.
Here is the Zabbix graph for one of the impacted VM, and it's easy to guess when I stopped the underlying "Patrol read" process : </p>
<p><img alt="VM iowait" src="/images/iowait.png" title="VM iowait"></p>
<p>That "patrol read" operation is scheduled to run by default once a week (168h) so your real option is to either disable it completely (through <strong><code>MegaCli64 -AdpPR -Dsbl -aALL</code></strong>) or at least (adviced) change the IO impact (for example 5% : <strong><code>MegaCli64 -AdpSetProp PatrolReadRate 5 -aALL</code></strong>)</p>
<p><strong>Never</strong> understimate the power of Hardware settings (in the BIOS or in that case raid hardware controller). </p>
<p>Hope it can help others too</p>CentOS AltArch SIG status2015-09-24T00:00:00+02:002015-09-24T00:00:00+02:00Fabian Arrotintag:arrfab.net,2015-09-24:/posts/2015/Sep/24/centos-altarch-sig-status/<p>Recently I had (from an Infra side) to start deploying KVM guests for the <a href="https://en.wikipedia.org/wiki/Ppc64">ppc64</a> and <a href="https://en.wikipedia.org/wiki/Ppc64">ppc64le</a> arches, so that <a href="https://wiki.centos.org/SpecialInterestGroup/AltArch">AltArch</a> SIGs contributors could start bootstrapping CentOS 7 rebuild for those arches. I'll probably write a tech review about <a href="https://en.wikipedia.org/wiki/POWER8">Power8</a> and the fact you can just use libvirt/virt-install to quickly provision new VMs on <a href="http://www-03.ibm.com/systems/power/software/linux/powerkvm/">PowerKVM</a> , but I'll do that in a separate post.</p>
<p>Parallel to ppc64/ppc64le, <a href="https://en.wikipedia.org/wiki/ARM_architecture#32-bit_architecture">armv7hl</a> interested some Community members, and the discussion/activity about that arch is discussed on the <a href="https://lists.centos.org/mailman/listinfo/arm-dev">dedicated mailing list</a>. It's slowly coming and some users already reported having used that on some boards (but still unsigned and no updates packages -yet- )</p>
<p>Last (but not least) in this AltArch list is i686 : <a href="https://wiki.centos.org/JohnnyHughes">Johnny</a> built all packages and are already publicly available on <a href="http://buildlogs.centos.org/">buildlogs.centos.org</a> , each time in parallel to the x86_64 version. It seems that respinning the ISO for that arch and last tests would be the only things to do.</p>
<p>If you're interested in participating in AltArch (and have special interesting a specific arch/platform), feel free to discuss that on the <a href="https://lists.centos.org/mailman/listinfo/centos-devel">centos-devel</a> list !</p>CentOS Dojo in Barcelona2015-09-17T00:00:00+02:002015-09-17T00:00:00+02:00Fabian Arrotintag:arrfab.net,2015-09-17:/posts/2015/Sep/17/centos-dojo-in-barcelona/<p>So, thanks to the folks from Opennebula, we'll have another CentOS Dojo in Barcelona on Tuesday 20th October 2015. That even will be colocated with the <a href="http://2015.opennebulaconf.com/">Opennebulaconf</a> happening the days after that Dojo. If you're attending the OpennebulaConf, or if you're just in the area and would like to attend the CentOS Dojo, feel free to <a href="http://www.eventbrite.com/e/centos-dojo-barcelona-2015-tickets-18514955731">register</a></p>
<p>Regarding the Dojo content, I'll be myself giving a presentation about Selinux : covering a little bit of intro (still needed for some folks afraid of using it , don't know why but we'll change that ...) about selinux itself, how to run it on bare-metal, virtual machines <em>and</em> there will be some slides for the mandatory container hype thing.
But we'll also cover managing selinux booleans/contexts, etc through your config management solution. (We'll cover <a href="https://puppetlabs.com/">puppet</a> and <a href="http://www.ansible.com/">ansible</a> as those are the two I'm using on a daily basis) and also how to build and deploy custom selinux policies with your config management solution.</p>
<p>On the other hand, if you're a CentOS user and would like yourself to give a talk during that Dojo, feel free to submit a talk ! More informations about the Dojo on the <a href="https://wiki.centos.org/Events/Dojo/Barcelona2015">dedicated wiki page</a></p>
<p>See you there !</p>Ext4 limitation with GDT blocks number2015-09-10T00:00:00+02:002015-09-10T00:00:00+02:00Fabian Arrotintag:arrfab.net,2015-09-10:/posts/2015/Sep/10/ext4-limitation-with-gdt-blocks-number/<p>In the last days, I encountered a strange issue^Wlimitation with <a href="https://en.wikipedia.org/wiki/Ext4">Ext4</a> that I wouldn't have thought of. I've used ext2/ext3/ext4 for quite some time and so I've been used to resize the filesystem "online" (while "mounted"). In the past you had to use <a href="http://linux.die.net/man/8/ext2online">ext2online</a> for that, then it was integrated into <a href="http://linux.die.net/man/8/resize2fs">resize2fs</a> itself.</p>
<p>The logic is simple and always the same : extend your underlaying block device (or add another one), then modify the LVM Volume Group (if needed), then the Logical Volume and finally the resize2fs operation, so something like </p>
<div class="highlight"><pre><span></span>lvextend -L +<span class="cp">${</span><span class="n">added_size</span><span class="cp">}</span>G /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
resize2fs /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
</pre></div>
<p>I don't know how much times I've used that, but this time resize2fs wasn't happy :</p>
<div class="highlight"><pre><span></span><span class="nv">resize2fs</span>: <span class="nv">Operation</span> <span class="nv">not</span> <span class="nv">permitted</span> <span class="k">While</span> <span class="nv">trying</span> <span class="nv">to</span> <span class="nv">add</span> <span class="nv">group</span> <span class="sc">#16384</span>
</pre></div>
<p>I remember having had in the past an issue because of the journal size <a href="https://bugzilla.redhat.com/show_bug.cgi?id=160612#c27">not being big enough</a>. <code>But this wasn't the case here.</code></p>
<p>FWIW, you can always verify your journal size with <code>dumpe2fs /dev/mapper/${name_of_your_logical_volume} |grep "Journal Size"</code></p>
<p>Small note : if you need to increase the journal size, you have to do it "offline" as you have to remove the journal and then add it back with a bigger …</p><p>In the last days, I encountered a strange issue^Wlimitation with <a href="https://en.wikipedia.org/wiki/Ext4">Ext4</a> that I wouldn't have thought of. I've used ext2/ext3/ext4 for quite some time and so I've been used to resize the filesystem "online" (while "mounted"). In the past you had to use <a href="http://linux.die.net/man/8/ext2online">ext2online</a> for that, then it was integrated into <a href="http://linux.die.net/man/8/resize2fs">resize2fs</a> itself.</p>
<p>The logic is simple and always the same : extend your underlaying block device (or add another one), then modify the LVM Volume Group (if needed), then the Logical Volume and finally the resize2fs operation, so something like </p>
<div class="highlight"><pre><span></span>lvextend -L +<span class="cp">${</span><span class="n">added_size</span><span class="cp">}</span>G /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
resize2fs /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
</pre></div>
<p>I don't know how much times I've used that, but this time resize2fs wasn't happy :</p>
<div class="highlight"><pre><span></span><span class="nv">resize2fs</span>: <span class="nv">Operation</span> <span class="nv">not</span> <span class="nv">permitted</span> <span class="k">While</span> <span class="nv">trying</span> <span class="nv">to</span> <span class="nv">add</span> <span class="nv">group</span> <span class="sc">#16384</span>
</pre></div>
<p>I remember having had in the past an issue because of the journal size <a href="https://bugzilla.redhat.com/show_bug.cgi?id=160612#c27">not being big enough</a>. <code>But this wasn't the case here.</code></p>
<p>FWIW, you can always verify your journal size with <code>dumpe2fs /dev/mapper/${name_of_your_logical_volume} |grep "Journal Size"</code></p>
<p>Small note : if you need to increase the journal size, you have to do it "offline" as you have to remove the journal and then add it back with a bigger size (and that also takes time) :</p>
<div class="highlight"><pre><span></span>umount /<span class="nv">$path_where_that_fs_is_mounted</span>
tune2fs -O ^has_journal /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
# Assuming we want to increase to 128Mb
tune2fs -j -J size=128 /dev/mapper/<span class="cp">${</span><span class="n">name_of_your_logical_volume</span><span class="cp">}</span>
</pre></div>
<p>But in that case, as said, it wasn't really the root cause : while the <code>resize2fs: Operation not permitted</code> doesn't give much informations, <code>dmesg</code> was more explicit : </p>
<div class="highlight"><pre><span></span><span class="n">EXT4</span><span class="o">-</span><span class="n">fs</span> <span class="n">warning</span> <span class="p">(</span><span class="n">device</span> <span class="n">dm</span><span class="o">-</span><span class="mi">2</span><span class="p">):</span> <span class="n">ext4_group_add</span><span class="p">:</span> <span class="k">No</span> <span class="n">reserved</span> <span class="n">GDT</span> <span class="n">blocks</span><span class="p">,</span> <span class="n">can</span><span class="err">'</span><span class="n">t</span> <span class="n">resize</span>
</pre></div>
<p>The limitation is that when the initial Ext4 filesystem is created, the number of reserved/calculated GDT blocks for that filesystem will allow to grow it by a <a href="http://www.spinics.net/lists/linux-ext4/msg35015.html">factor of 1000</a>.</p>
<p>Ouch, that system (CentOS 6.7) I was working on had been provisioned in the past for a certain role, and that particular fs/mount point was set to 2G (installed like this through the <a href="https://en.wikipedia.org/wiki/Kickstart_(Linux)">Kickstart setup</a> ). But finally role changed and so the filesystem has been extended/resized some times, until I tried to extend it to more than 2TiB, which then caused resize2fs to complain ...</p>
<p>So two choices :</p>
<ul>
<li>you do it "offline" through <code>umount, e2fsck, resize2fs, e2fsck, mount</code> (but time consumming)</li>
<li>you still have plenty of space in the VG, and you just want to create another volume with correct size, format it, rsync content, umount old one and mount the new one.</li>
</ul>
<p>That means that I learned something new (one learns something new every day !), and also the fact that you then need to take that limitation in mind when using a kickstart (that doesn't include the <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-kickstart2-options.html">--grow</a> option, but a fixed size for the filesystem).</p>
<p>Hope that it can help</p>Implementing TLS for postfix2015-09-03T00:00:00+02:002015-09-03T00:00:00+02:00Fabian Arrotintag:arrfab.net,2015-09-03:/posts/2015/Sep/03/implementing-tls-for-postfix/<p>As some initiatives (like <a href="https://letsencrypt.org">Let's Encrypt</a> as one example) try to force <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> usage everywhere. We thought about doing the same for the CentOS.org infra. Obviously we already had some <a href="https://en.wikipedia.org/wiki/X.509">x509</a> certificates, but not for every httpd server that was serving content for CentOS users. So we decided to <a href="https://lists.centos.org/pipermail/centos-announce/2015-August/021341.html">enforce</a> TLS usage on those servers. But TLS can be used obviously on other things than a web server.</p>
<p>That's why we considered implementing something for our <a href="http://www.postfix.org">Postfix</a> nodes. The interesting part is that it's really easy (depending of course at the security level one may want to reach/use). There are two parts in the postfix main.cf that can be configured :</p>
<ul>
<li>outgoing mails (aka your server sends mail to other SMTPD servers)</li>
<li>incoming mails (aka remote clients/servers send mail to your postfix/smtpd server)</li>
</ul>
<p>Let's start with the client/outgoing part : just adding those lines in your main.cf will automatically configure it to use TLS when possible, but otherwise fall back on clear if remote server doesn't support TLS :</p>
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="n">TLS</span> <span class="o">-</span> <span class="n">client</span> <span class="n">part</span>
<span class="n">smtp_tls_CAfile</span><span class="o">=/</span><span class="n">etc</span><span class="o">/</span><span class="n">pki</span><span class="o">/</span><span class="n">tls</span><span class="o">/</span><span class="n">certs</span><span class="o">/</span><span class="n">ca</span><span class="o">-</span><span class="n">bundle</span><span class="p">.</span><span class="n">crt</span>
<span class="n">smtp_tls_security_level</span> <span class="o">=</span> <span class="n">may</span>
<span class="n">smtp_tls_loglevel</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">smtp_tls_session_cache_database</span> <span class="o">=</span> <span class="n">btree</span><span class="p">:</span><span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="k">postfix</span><span class="o">/</span><span class="n">smtp_scache</span>
</pre></div>
<p>The interesting part is the <code>smtp_tls_security_level …</code></p><p>As some initiatives (like <a href="https://letsencrypt.org">Let's Encrypt</a> as one example) try to force <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security">TLS</a> usage everywhere. We thought about doing the same for the CentOS.org infra. Obviously we already had some <a href="https://en.wikipedia.org/wiki/X.509">x509</a> certificates, but not for every httpd server that was serving content for CentOS users. So we decided to <a href="https://lists.centos.org/pipermail/centos-announce/2015-August/021341.html">enforce</a> TLS usage on those servers. But TLS can be used obviously on other things than a web server.</p>
<p>That's why we considered implementing something for our <a href="http://www.postfix.org">Postfix</a> nodes. The interesting part is that it's really easy (depending of course at the security level one may want to reach/use). There are two parts in the postfix main.cf that can be configured :</p>
<ul>
<li>outgoing mails (aka your server sends mail to other SMTPD servers)</li>
<li>incoming mails (aka remote clients/servers send mail to your postfix/smtpd server)</li>
</ul>
<p>Let's start with the client/outgoing part : just adding those lines in your main.cf will automatically configure it to use TLS when possible, but otherwise fall back on clear if remote server doesn't support TLS :</p>
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="n">TLS</span> <span class="o">-</span> <span class="n">client</span> <span class="n">part</span>
<span class="n">smtp_tls_CAfile</span><span class="o">=/</span><span class="n">etc</span><span class="o">/</span><span class="n">pki</span><span class="o">/</span><span class="n">tls</span><span class="o">/</span><span class="n">certs</span><span class="o">/</span><span class="n">ca</span><span class="o">-</span><span class="n">bundle</span><span class="p">.</span><span class="n">crt</span>
<span class="n">smtp_tls_security_level</span> <span class="o">=</span> <span class="n">may</span>
<span class="n">smtp_tls_loglevel</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">smtp_tls_session_cache_database</span> <span class="o">=</span> <span class="n">btree</span><span class="p">:</span><span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="k">postfix</span><span class="o">/</span><span class="n">smtp_scache</span>
</pre></div>
<p>The interesting part is the <code>smtp_tls_security_level</code> option : as you see, we decided to force it to <code>may</code> . That's what Postfix <a href="http://www.postfix.org/TLS_README.html#client_tls_may">official TLS documentation</a> calls "Opportunistic TLS" : in some words it will try TLS (even with untrusted remote certs !) and will only default to clear if no remote TLS support is available. That's the option we decided to use as it doesn't break anything, and even if the remote server has a self-signed cert, it's still better to use TLS with self-signed than clear text, right ?</p>
<p>Once you have reloaded your postfix configuration, you'll directly see in your maillog that it will start trying TLS and deliver mails to servers configured for it : </p>
<div class="highlight"><pre><span></span><span class="n">Sep</span> <span class="mi">3</span> <span class="mi">07</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">37</span> <span class="n">mailsrv</span> <span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="p">[</span><span class="mi">1936</span><span class="p">]:</span> <span class="n">setting</span> <span class="n">up</span> <span class="n">TLS</span> <span class="k">connection</span> <span class="k">to</span> <span class="n">ASPMX</span><span class="p">.</span><span class="n">L</span><span class="p">.</span><span class="n">GOOGLE</span><span class="p">.</span><span class="n">com</span><span class="p">[</span><span class="mi">173</span><span class="p">.</span><span class="mi">194</span><span class="p">.</span><span class="mi">207</span><span class="p">.</span><span class="mi">27</span><span class="p">]:</span><span class="mi">25</span>
<span class="n">Sep</span> <span class="mi">3</span> <span class="mi">07</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">37</span> <span class="n">mailsrv</span> <span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="p">[</span><span class="mi">1936</span><span class="p">]:</span> <span class="k">Trusted</span> <span class="n">TLS</span> <span class="k">connection</span> <span class="n">established</span> <span class="k">to</span> <span class="n">ASPMX</span><span class="p">.</span><span class="n">L</span><span class="p">.</span><span class="n">GOOGLE</span><span class="p">.</span><span class="n">com</span><span class="p">[</span><span class="mi">173</span><span class="p">.</span><span class="mi">194</span><span class="p">.</span><span class="mi">207</span><span class="p">.</span><span class="mi">27</span><span class="p">]:</span><span class="mi">25</span><span class="p">:</span> <span class="n">TLSv1</span><span class="p">.</span><span class="mi">2</span> <span class="k">with</span> <span class="n">cipher</span> <span class="n">ECDHE</span><span class="o">-</span><span class="n">RSA</span><span class="o">-</span><span class="n">AES128</span><span class="o">-</span><span class="n">GCM</span><span class="o">-</span><span class="n">SHA256</span> <span class="p">(</span><span class="mi">128</span><span class="o">/</span><span class="mi">128</span> <span class="n">bits</span><span class="p">)</span>
<span class="n">Sep</span> <span class="mi">3</span> <span class="mi">07</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mi">37</span> <span class="n">mailsrv</span> <span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="p">[</span><span class="mi">1936</span><span class="p">]:</span> <span class="n">DF584A00774</span><span class="p">:</span> <span class="k">to</span><span class="o">=<></span><span class="p">,</span> <span class="n">orig_to</span><span class="o">=<></span><span class="p">,</span> <span class="n">relay</span><span class="o">=</span><span class="n">ASPMX</span><span class="p">.</span><span class="n">L</span><span class="p">.</span><span class="n">GOOGLE</span><span class="p">.</span><span class="n">com</span><span class="p">[</span><span class="mi">173</span><span class="p">.</span><span class="mi">194</span><span class="p">.</span><span class="mi">207</span><span class="p">.</span><span class="mi">27</span><span class="p">]:</span><span class="mi">25</span><span class="p">,</span> <span class="n">delay</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">delays</span><span class="o">=</span><span class="mi">0</span><span class="o">/</span><span class="mi">0</span><span class="p">.</span><span class="mi">12</span><span class="o">/</span><span class="mi">0</span><span class="p">.</span><span class="mi">22</span><span class="o">/</span><span class="mi">0</span><span class="p">.</span><span class="mi">71</span><span class="p">,</span> <span class="n">dsn</span><span class="o">=</span><span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="n">sent</span> <span class="p">(</span><span class="mi">250</span> <span class="mi">2</span><span class="p">.</span><span class="mi">0</span><span class="p">.</span><span class="mi">0</span> <span class="n">OK</span> <span class="mi">1441266639</span> <span class="mi">79</span><span class="n">si29025652qku</span><span class="p">.</span><span class="mi">67</span> <span class="o">-</span> <span class="n">gsmtp</span><span class="p">)</span>
</pre></div>
<p>Now let's have a look at the other part : when you want your server to present the STARTTLS feature when remote servers/clients try to send you mails (still in postfix main.cf) :</p>
<div class="highlight"><pre><span></span><span class="x"># TLS - server part</span>
<span class="x">smtpd_tls_CAfile=/etc/pki/tls/certs/ca-bundle.crt</span>
<span class="x">smtpd_tls_cert_file = /etc/pki/tls/certs/</span><span class="cp"><%=</span> <span class="n">postfix_myhostname</span> <span class="cp">%></span><span class="x">-postfix.crt </span>
<span class="x">smtpd_tls_key_file = /etc/pki/tls/private/</span><span class="cp"><%=</span> <span class="n">postfix_myhostname</span> <span class="cp">%></span><span class="x">.key</span>
<span class="x">smtpd_tls_security_level = may</span>
<span class="x">smtpd_tls_loglevel = 1</span>
<span class="x">smtpd_tls_session_cache_database = btree:/var/lib/postfix/smtpd_scache</span>
</pre></div>
<p>Still easy, but here we also add our key/cert to the config but if you decide to use a signed by a trusted CA cert (like we do for centos.org infra), be sure that the cert is the concatenated/bundled version of both your cert and the CAChain cert. That's also documented in the <a href="http://www.postfix.org/TLS_README.html#server_cert_key">Postfix TLS guide</a>, and if you're already using <a href="http://www.nginx.org">Nginx</a>, you already know what I'm talking about as you <a href="http://nginx.org/en/docs/http/configuring_https_servers.html#chains">already have to do it</a> too.</p>
<p>If you've correctly configured your cert/keys and reloaded your postfix config, now remote SMTPD servers will also (if configured to do so) deliver mails to your server through TLS. Bonus point if you're using a cert signed by a trusted CA, as from a client side you'll see this : </p>
<div class="highlight"><pre><span></span><span class="n">Sep</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">16</span><span class="err">:</span><span class="mi">17</span><span class="err">:</span><span class="mi">22</span><span class="w"> </span><span class="n">hoth</span><span class="w"> </span><span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="o">[</span><span class="n">15329</span><span class="o">]</span><span class="err">:</span><span class="w"> </span><span class="n">setting</span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="n">TLS</span><span class="w"> </span><span class="k">connection</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mail</span><span class="p">.</span><span class="n">centos</span><span class="p">.</span><span class="n">org</span><span class="o">[</span><span class="n">72.26.200.203</span><span class="o">]</span><span class="err">:</span><span class="mi">25</span><span class="w"></span>
<span class="n">Sep</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">16</span><span class="err">:</span><span class="mi">17</span><span class="err">:</span><span class="mi">22</span><span class="w"> </span><span class="n">hoth</span><span class="w"> </span><span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="o">[</span><span class="n">15329</span><span class="o">]</span><span class="err">:</span><span class="w"> </span><span class="n">Trusted</span><span class="w"> </span><span class="n">TLS</span><span class="w"> </span><span class="k">connection</span><span class="w"> </span><span class="n">established</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">mail</span><span class="p">.</span><span class="n">centos</span><span class="p">.</span><span class="n">org</span><span class="o">[</span><span class="n">72.26.200.203</span><span class="o">]</span><span class="err">:</span><span class="mi">25</span><span class="err">:</span><span class="w"> </span><span class="n">TLSv1</span><span class="mf">.2</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">cipher</span><span class="w"> </span><span class="n">DHE</span><span class="o">-</span><span class="n">RSA</span><span class="o">-</span><span class="n">AES256</span><span class="o">-</span><span class="n">GCM</span><span class="o">-</span><span class="n">SHA384</span><span class="w"> </span><span class="p">(</span><span class="mi">256</span><span class="o">/</span><span class="mi">256</span><span class="w"> </span><span class="n">bits</span><span class="p">)</span><span class="w"></span>
<span class="n">Sep</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">16</span><span class="err">:</span><span class="mi">17</span><span class="err">:</span><span class="mi">23</span><span class="w"> </span><span class="n">hoth</span><span class="w"> </span><span class="k">postfix</span><span class="o">/</span><span class="n">smtp</span><span class="o">[</span><span class="n">15329</span><span class="o">]</span><span class="err">:</span><span class="w"> </span><span class="nl">CC8351C00C9</span><span class="p">:</span><span class="w"> </span><span class="k">to</span><span class="o">=<</span><span class="n">fake_one_for_blog_post</span><span class="nv">@centos</span><span class="p">.</span><span class="n">org</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">relay</span><span class="o">=</span><span class="n">mail</span><span class="p">.</span><span class="n">centos</span><span class="p">.</span><span class="n">org</span><span class="o">[</span><span class="n">72.26.200.203</span><span class="o">]</span><span class="err">:</span><span class="mi">25</span><span class="p">,</span><span class="w"> </span><span class="n">delay</span><span class="o">=</span><span class="mf">1.6</span><span class="p">,</span><span class="w"> </span><span class="n">delays</span><span class="o">=</span><span class="mf">0.19</span><span class="o">/</span><span class="mf">0.03</span><span class="o">/</span><span class="mf">1.1</span><span class="o">/</span><span class="mf">0.31</span><span class="p">,</span><span class="w"> </span><span class="n">dsn</span><span class="o">=</span><span class="mf">2.0.0</span><span class="p">,</span><span class="w"> </span><span class="n">status</span><span class="o">=</span><span class="n">sent</span><span class="w"> </span><span class="p">(</span><span class="mi">250</span><span class="w"> </span><span class="mf">2.0.0</span><span class="w"> </span><span class="nl">Ok</span><span class="p">:</span><span class="w"> </span><span class="n">queued</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">A7299A006E2</span><span class="p">)</span><span class="w"></span>
</pre></div>
<p>The <code>Trusted TLS connection established</code> part shows that your smtpd server presents a correct cert (bundle) and that the remote server sending you mails trusts the CA used to sign that cert.</p>
<p>There are a lot of TLS options that you can also add for tuning/security reasons, and all can be seen through <code>postconf |grep tls</code>, but also on the Postfix <a href="http://www.postfix.org/postconf.5.html#smtp_tls_ciphers">postconf doc</a></p>Updating to Gluster 3.6 packages on CentOS 62014-11-21T16:08:00+01:002014-11-21T16:08:00+01:00Fabian Arrotintag:arrfab.net,2014-11-21:/posts/2014/Nov/21/updating-to-gluster-3-6-packages-on-centos-6/<p>I had to do yesterday some maintenance yesterday on our
<a href="http://www.gluster.org">Gluster</a> nodes used within CentOS.org infra.
Basically I had to reconfigure some gluster volumes to use Infiniband
instead of Ethernet. (I'll write a dedicated blog post about that
migration later).</p>
<p>While a lot of people directly consume packages from Gluster.org (for
example
http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/CentOS/epel-6/x86_64/),
you'll be able (soon) to also install directly those packages on CentOS,
through packages built by the <a href="https://wiki.centos.org/SpecialInterestGroup/Storage/Proposal">Storage
SIG</a>. At
the moment I'm writing this blog post, gluster 3.6.1 packages are built
and available on our <a href="http://cbs.centos.org/koji/">Community Build Server Koji
setup</a> , but still in testing (and
unsigned).</p>
<p>"But wait, there are already glusterfs packages tagged 3.6 in CentOS
6.6, right ? " will you say. Well, yes, but not the full stack. What you
see in the [base] (or [updates]) repository are the client packages, as
for example a base CentOS 6.x can be a gluster client (through fuse, or
libgfapi - really interesting to speed up qemu-kvm instead of using the
default fuse mount point ..) , but the -server package isn't there. So
the reason why you can either use the …</p><p>I had to do yesterday some maintenance yesterday on our
<a href="http://www.gluster.org">Gluster</a> nodes used within CentOS.org infra.
Basically I had to reconfigure some gluster volumes to use Infiniband
instead of Ethernet. (I'll write a dedicated blog post about that
migration later).</p>
<p>While a lot of people directly consume packages from Gluster.org (for
example
http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/CentOS/epel-6/x86_64/),
you'll be able (soon) to also install directly those packages on CentOS,
through packages built by the <a href="https://wiki.centos.org/SpecialInterestGroup/Storage/Proposal">Storage
SIG</a>. At
the moment I'm writing this blog post, gluster 3.6.1 packages are built
and available on our <a href="http://cbs.centos.org/koji/">Community Build Server Koji
setup</a> , but still in testing (and
unsigned).</p>
<p>"But wait, there are already glusterfs packages tagged 3.6 in CentOS
6.6, right ? " will you say. Well, yes, but not the full stack. What you
see in the [base] (or [updates]) repository are the client packages, as
for example a base CentOS 6.x can be a gluster client (through fuse, or
libgfapi - really interesting to speed up qemu-kvm instead of using the
default fuse mount point ..) , but the -server package isn't there. So
the reason why you can either use the upstream gluster.org yum
repositories or the Storage SIG one to have access to the full stack,
and so run glusterd on CentOS.</p>
<p>Interested in testing those packages ? Wanting to test the update before
those packages will be released by the Storage SIG ? here we go :
<a href="http://cbs.centos.org/repos/storage6-testing/x86_64/os/Packages/">http://cbs.centos.org/repos/storage6-testing/x86_64/os/Packages/</a>
(packages available for <a href="http://cbs.centos.org/repos/storage7-testing/x86_64/os/Packages/">CentOS
7</a>
too)</p>
<p>By the way, if you never tested Gluster, it's really easy to setup and
play with, even within Virtual Machines. Interesting reading : (quick
start) :
<a href="http://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart">http://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart</a></p>CentOS Dojo Lyon (France)2014-03-15T15:00:00+01:002014-03-15T15:00:00+01:00Fabian Arrotintag:arrfab.net,2014-03-15:/posts/2014/Mar/15/centos-dojo-lyon-france/<p>Comme vous le savez peut-être (ou pas !), nous tiendrons un Dojo CentOS
à Lyon le vendredi 11 avril. Si donc vous avez envie de partager votre
expérience autour de CentOS, en donnant une présentation par exemple, ou
bien si vous désirez seulement venir passer un bon moment avec nous en
écoutant les présentations prévues (appel - subliminal - aux candidats
volontaires !), sentez-vous libre de vous inscrire.<br>
L'inscription est gratuite ! Plus d'informations sur la page Wiki :
<a href="http://wiki.centos.org/Events/Dojo/Lyon2014">http://wiki.centos.org/Events/Dojo/Lyon2014</a> .</p>
<dl>
<dt>Hi people, are you in the Lyon (France) area around April 11th ? Willing</dt>
<dt>to come to a CentOS Dojo ? (either to attend it or even better, present</dt>
<dt>something around CentOS ?) . Feel free to register for this free event !</dt>
<dd><a href="http://wiki.centos.org/Events/Dojo/Lyon2014">http://wiki.centos.org/Events/Dojo/Lyon2014</a></dd>
</dl>Rolling updates with Ansible and Apache reverse proxies2013-05-23T17:36:00+02:002013-05-23T17:36:00+02:00Fabian Arrotintag:arrfab.net,2013-05-23:/posts/2013/May/23/rolling-updates-with-ansible-and-apache-reverse-proxies/<p>It's not a secret anymore that I use <a href="http://ansible.cc/">Ansible</a> to do
a lot of things. That goes from simple "one shot" actions with ansible
on multiple nodes to "configuration management and deployment tasks"
with ansible-playbook. One of the thing I also really like with Ansible
is the fact that it's also a great
<a href="http://en.wikipedia.org/wiki/Orchestration_%28computing%29">orchestration</a>
tool.</p>
<p>For example, in some WSOA flows you can have a bunch of servers behind
load balancer nodes. When you want to put a backend node/web server node
in maintenance mode (to change configuration/update package/update
app/whatever), you just "remove" that node from the production flow, do
what you need to do, verify it's up again and put that node back in
production. The principle of "rolling updates" is then interesting as
you still have 24/7 flows in production.</p>
<p>But what if you're not in charge of the whole infrastructure ? AKA for
example you're in charge of some servers, but not the load balancers in
front of your infrastructure. Let's consider the following situation,
and how we'll use ansible to still disable/enable a backend server
behind Apache reverse proxies.</p>
<p><img alt="Apache LB" src="/images/Apache-LB1.png" title="Apache-LB"></p>
<p>So here is the (simplified) situation : two Apache reverse proxies
(using the …</p><p>It's not a secret anymore that I use <a href="http://ansible.cc/">Ansible</a> to do
a lot of things. That goes from simple "one shot" actions with ansible
on multiple nodes to "configuration management and deployment tasks"
with ansible-playbook. One of the thing I also really like with Ansible
is the fact that it's also a great
<a href="http://en.wikipedia.org/wiki/Orchestration_%28computing%29">orchestration</a>
tool.</p>
<p>For example, in some WSOA flows you can have a bunch of servers behind
load balancer nodes. When you want to put a backend node/web server node
in maintenance mode (to change configuration/update package/update
app/whatever), you just "remove" that node from the production flow, do
what you need to do, verify it's up again and put that node back in
production. The principle of "rolling updates" is then interesting as
you still have 24/7 flows in production.</p>
<p>But what if you're not in charge of the whole infrastructure ? AKA for
example you're in charge of some servers, but not the load balancers in
front of your infrastructure. Let's consider the following situation,
and how we'll use ansible to still disable/enable a backend server
behind Apache reverse proxies.</p>
<p><img alt="Apache LB" src="/images/Apache-LB1.png" title="Apache-LB"></p>
<p>So here is the (simplified) situation : two Apache reverse proxies
(using the
<a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">mod_proxy_balancer</a>
module) are used to load balance traffic to four backend nodes (Jboss in
our simplified case). We can't directly touch those upstream Apache
nodes, but we can still interact on them , thanks to the fact that
"<a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html#balancer_manager">balancer manager
suppor</a>t"
is active (and protected !)</p>
<p>Let's have a look at a (simplified) ansible inventory file :</p>
<div class="highlight"><pre><span></span><span class="k">[jboss-cluster]</span>
<span class="na">jboss-1</span>
<span class="na">jboss-2</span>
<span class="na">jboss-3</span>
<span class="na">jboss-4</span>
<span class="k">[apache-group-1]</span>
<span class="na">apache-node-1</span>
<span class="na">apache-node-2</span>
</pre></div>
<p>Let's now create a generic (write once/use it many) task to disable a
backend node from apache ! :</p>
<div class="highlight"><pre><span></span><span class="o">---</span>
##############################################################################
#
# <span class="nv">This</span> <span class="nv">task</span> <span class="nv">can</span> <span class="nv">be</span> <span class="nv">included</span> <span class="nv">in</span> <span class="nv">a</span> <span class="nv">playbook</span> <span class="nv">to</span> <span class="k">pause</span> <span class="nv">a</span> <span class="nv">backend</span> <span class="nv">node</span>
# <span class="nv">being</span> <span class="nv">load</span> <span class="nv">balanced</span> <span class="nv">by</span> <span class="nv">Apache</span> <span class="nv">Reverse</span> <span class="nv">Proxies</span>
# <span class="nv">Two</span> <span class="nv">variables</span> <span class="nv">need</span> <span class="nv">to</span> <span class="nv">be</span> <span class="nv">defined</span> :
# <span class="o">-</span> ${<span class="nv">apache_rp_backend_url</span>} : <span class="nv">the</span> <span class="nv">URL</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">backend</span> <span class="nv">server</span>, <span class="nv">as</span> <span class="nv">known</span> <span class="nv">by</span> <span class="nv">Apache</span> <span class="nv">server</span>
# <span class="o">-</span> ${<span class="nv">apache_rp_backend_cluster</span>} : <span class="nv">the</span> <span class="nv">name</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">cluster</span> <span class="nv">as</span> <span class="nv">defined</span> <span class="nv">on</span> <span class="nv">the</span> <span class="nv">Apache</span> <span class="nv">RP</span> <span class="ss">(</span><span class="nv">the</span> <span class="nv">group</span> <span class="nv">the</span> <span class="nv">node</span> <span class="nv">is</span> <span class="nv">member</span> <span class="nv">of</span><span class="ss">)</span> <span class="ss">(</span><span class="nv">internalasync</span><span class="ss">)</span>
# <span class="o">-</span> ${<span class="nv">apache_rp_group</span>} : <span class="nv">the</span> <span class="nv">name</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">group</span> <span class="nv">declared</span> <span class="nv">in</span> <span class="nv">hosts</span>.<span class="nv">cfg</span> <span class="nv">containing</span> <span class="nv">Apache</span> <span class="nv">Reverse</span> <span class="nv">Proxies</span>
# <span class="o">-</span> ${<span class="nv">apache_rp_user</span>}: <span class="nv">the</span> <span class="nv">username</span> <span class="nv">used</span> <span class="nv">to</span> <span class="nv">authenticate</span> <span class="nv">against</span> <span class="nv">the</span> <span class="nv">Apache</span> <span class="nv">balancer</span><span class="o">-</span><span class="nv">manager</span> <span class="ss">(</span><span class="nv">clusteradmin</span><span class="ss">)</span>
# <span class="o">-</span> ${<span class="nv">apache_rp_password</span>}: <span class="nv">the</span> <span class="nv">password</span> <span class="nv">used</span> <span class="nv">to</span> <span class="nv">authenticate</span> <span class="nv">against</span> <span class="nv">the</span> <span class="nv">Apache</span> <span class="nv">balancer</span><span class="o">-</span><span class="nv">manager</span> <span class="ss">(</span><span class="mi">5</span><span class="nv">added592b</span><span class="ss">)</span>
# <span class="o">-</span> ${<span class="nv">apache_rp_balancer_manager_uri</span>}: <span class="nv">the</span> <span class="nv">URI</span> <span class="nv">where</span> <span class="nv">to</span> <span class="nv">find</span> <span class="nv">the</span> <span class="nv">balancer</span><span class="o">-</span><span class="nv">manager</span> <span class="nv">Apache</span> <span class="nv">mod</span>
#
##############################################################################
<span class="o">-</span> <span class="nv">name</span>: <span class="nv">Disabling</span> <span class="nv">the</span> <span class="nv">worker</span> <span class="nv">in</span> <span class="nv">Apache</span> <span class="nv">Reverse</span> <span class="nv">Proxies</span>
<span class="nv">local_action</span>: <span class="nv">shell</span> <span class="o">/</span><span class="nv">usr</span><span class="o">/</span><span class="nv">bin</span><span class="o">/</span><span class="nv">curl</span> <span class="o">-</span><span class="nv">k</span> <span class="o">--</span><span class="nv">user</span> ${<span class="nv">apache_rp_user</span>}:${<span class="nv">apache_rp_password</span>} <span class="s2">"</span><span class="s">https://${item}/${apache_rp_balancer_manager_uri}?b=${apache_rp_backend_cluster}&w=${apache_rp_backend_url}&nonce=$(curl -k --user ${apache_rp_user}:${apache_rp_password} https://${item}/${apache_rp_balancer_manager_uri} |grep nonce|tail -n 1|cut -f 3 -d '&'|cut -f 2 -d '='|cut -f 1 -d '</span><span class="s2">"</span><span class="s1">'</span><span class="s">)&dw=Disable"</span>
<span class="nv">with_items</span>: ${<span class="nv">groups</span>.${<span class="nv">apache_rp_group</span>}}
<span class="o">-</span> <span class="nv">name</span>: <span class="nv">Waiting</span> <span class="mi">20</span> <span class="nv">seconds</span> <span class="nv">to</span> <span class="nv">be</span> <span class="nv">sure</span> <span class="nv">no</span> <span class="nv">traffic</span> <span class="nv">is</span> <span class="nv">being</span> <span class="nv">sent</span> <span class="nv">anymore</span> <span class="nv">to</span> <span class="nv">that</span> <span class="nv">worker</span> <span class="nv">backend</span> <span class="nv">node</span>
<span class="k">pause</span>: <span class="nv">seconds</span><span class="o">=</span><span class="mi">20</span>
</pre></div>
<p>The interesting bit is the with_items one : it will use the
apache_rp_group variable to know which apache servers are used
upstream (assuming you can have multiple nodes/clusters) and will play
that command for every host in the list obtained from the inventory !</p>
<p>We can now, in the "rolling-updates" playbook, just call the previous
tasks (assuming we saved it as ../tasks/apache-disable-worker.yml) :</p>
<div class="highlight"><pre><span></span><span class="o">---</span>
<span class="o">-</span> <span class="nv">hosts</span>: <span class="nv">jboss</span><span class="o">-</span><span class="nv">cluster</span>
<span class="nv">serial</span>: <span class="mi">1</span>
<span class="nv">user</span>: <span class="nv">root</span>
<span class="nv">tasks</span>:
<span class="o">-</span> <span class="k">include</span>: ..<span class="o">/</span><span class="nv">tasks</span><span class="o">/</span><span class="nv">apache</span><span class="o">-</span><span class="nv">disable</span><span class="o">-</span><span class="nv">worker</span>.<span class="nv">yml</span>
<span class="o">-</span> <span class="nv">etc</span><span class="o">/</span><span class="nv">etc</span> ...
<span class="o">-</span> <span class="nv">wait_for</span>: <span class="nv">port</span><span class="o">=</span><span class="mi">8443</span> <span class="nv">state</span><span class="o">=</span><span class="nv">started</span>
<span class="o">-</span> <span class="k">include</span>: ..<span class="o">/</span><span class="nv">tasks</span><span class="o">/</span><span class="nv">apache</span><span class="o">-</span><span class="nv">enable</span><span class="o">-</span><span class="nv">worker</span>.<span class="nv">yml</span>
</pre></div>
<p>But Wait ! As you've seen, we still need to declare some variables :
let's do that in the inventory, under group_vars and host_vars !</p>
<p>group_vars/jboss-cluster :</p>
<div class="highlight"><pre><span></span><span class="o">#</span> <span class="n">Apache</span> <span class="n">reverse</span> <span class="n">proxies</span> <span class="n">settins</span>
<span class="n">apache_rp_group</span><span class="p">:</span> <span class="n">apache</span><span class="o">-</span><span class="k">group</span><span class="o">-</span><span class="mi">1</span>
<span class="n">apache_rp_user</span><span class="p">:</span> <span class="n">my</span><span class="o">-</span><span class="k">admin</span><span class="o">-</span><span class="n">account</span>
<span class="n">apache_rp_password</span><span class="p">:</span> <span class="n">my</span><span class="o">-</span><span class="n">beautiful</span><span class="o">-</span><span class="n">pass</span>
<span class="n">apache_rp_balancer_manager_uri</span><span class="p">:</span>
<span class="n">balancer</span><span class="o">-</span><span class="n">manager</span><span class="o">-</span><span class="n">hidden</span><span class="o">-</span><span class="k">and</span><span class="o">-</span><span class="n">redirected</span>
</pre></div>
<p>host_vars/jboss-1 :</p>
<div class="highlight"><pre><span></span><span class="n">apache_rp_backend_url</span> <span class="p">:</span> <span class="s1">'https://jboss1.myinternal.domain.org:8443'</span>
<span class="n">apache_rp_backend_cluster</span> <span class="p">:</span> <span class="n">nameofmyclusterdefinedinapache</span>
</pre></div>
<p>Now when we'll use that playbook, we'll have a local action that will
interact with the balancer manager to disable that backend node while we
do maintainance.</p>
<p>I let you imagine (and create) a ../tasks/apache-enable-worker.yml file
to enable it (which you'll call at the end of your playbook).</p>Ansible as an alternative to puppet/chef/cfengine and others ...2012-10-26T15:02:00+02:002012-10-26T15:02:00+02:00Fabian Arrotintag:arrfab.net,2012-10-26:/posts/2012/Oct/26/ansible-as-an-alternative-to-puppetchefcfengine-and-others/<p>I already know that i'll be criticized for this post, but i don't care
:-) . Strangely my last blog post (which is *very* old ...) was about
a puppet dashboard, so why speaking about another tool ? Well, first i
got a new job and some prerequisites have changed. I still like puppet
(and I'd even want to be able to use puppet but that's another story
...) but I was faced to some constraints when being in front of a new
project. For that specific project, I had to configure a bunch of new
Virtual Machines (RHEL6) coming as OVF files. Problem number one was
that I can't alter or modify the base image so i can't push packages
(from the distro or third-party repositories). Second issue is that I
can't install nor have a daemon/agent running on those machines. I had a
look at the different config tools available but they all require either
a daemon to be started, or at least having extra packages to be
installed on each managed node. (so not possible to have puppetd nor
puppetrun or invoke puppet directly through ssh , as
<a href="http://www.puppetlabs.com/">puppet</a> can't even be installed, same for
<a href="http://saltstack.org/">saltstack</a>). That's why i decided to give …</p><p>I already know that i'll be criticized for this post, but i don't care
:-) . Strangely my last blog post (which is *very* old ...) was about
a puppet dashboard, so why speaking about another tool ? Well, first i
got a new job and some prerequisites have changed. I still like puppet
(and I'd even want to be able to use puppet but that's another story
...) but I was faced to some constraints when being in front of a new
project. For that specific project, I had to configure a bunch of new
Virtual Machines (RHEL6) coming as OVF files. Problem number one was
that I can't alter or modify the base image so i can't push packages
(from the distro or third-party repositories). Second issue is that I
can't install nor have a daemon/agent running on those machines. I had a
look at the different config tools available but they all require either
a daemon to be started, or at least having extra packages to be
installed on each managed node. (so not possible to have puppetd nor
puppetrun or invoke puppet directly through ssh , as
<a href="http://www.puppetlabs.com/">puppet</a> can't even be installed, same for
<a href="http://saltstack.org/">saltstack</a>). That's why i decided to give
<a href="http://ansible.cc">Ansible</a> a try. It was already on my "TO-test" list
for a long time but it seems it was really fitting the bill for that
specific project and constraints : using the 'already-in-place' ssh
authorization, no packages to be installed on the managed nodes, and
last-but-no-least, a learning curve that is really thin (compared to
puppet and others, but that's my personal opinion/experience).</p>
<p>The other good thing with Ansible is that you can start very easily and
then slowly add 'complexity' to your playbooks/tasks. I'm still using
for example a flat inventory file, but already organized to reflect what
we can do in the future (hostnames included in groups, themselves
included in parents groups - aka nested groups). Same for the variables
inheritance : at the group level and down to the host level, host
variables overwriting those defined at the group level , etc ...)</p>
<p>The Yaml syntax is really easy to understand so you can have quickly
your first playbook being played on a bunch of machines simultaneously
(thanks to paramiko/parallel ssh). The number of modules is less than
the puppet resources, but is quickly growing. I also just tested to tie
the execution of ansible playbook with<a href="http://jenkins-ci.org/">Jenkins</a>
so that people not having access to the ansible
inventory/playbooks/tasks (stored in a vcs, subversion in my case) can
use it from a gui.. More to come on Ansible in the future</p>Puppet, Foreman and selinux on CentOS2012-02-21T15:23:00+01:002012-02-21T15:23:00+01:00Fabian Arrotintag:arrfab.net,2012-02-21:/posts/2012/Feb/21/puppet-foreman-and-selinux-on-centos/<p>We implemented <a href="http://puppetlabs.com">Puppet</a> as a configuration
management system at \$work , and Puppet is a great tool. Then I heard
about some dashboards that could be used on top of it. I've heard about
different dashboards (\$management_people *like* dashboards) like
<a href="http://puppetlabs.com/puppet/related-projects/dashboard/">Puppet-dashboard</a>
and <a href="http://theforeman.org/">Foreman</a>.</p>
<p>I was advised by several people to give Foreman a try and it's really
simple to install. Their
<a href="http://theforeman.org/projects/foreman/wiki/Installation_instructions">wiki</a>
covers basic installation and there is even a<a href="http://yum.theforeman.org/">yum
repo</a> that can be used (Epel has to be
enabled too). As i have a small network to manage, I decided to setup
Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is
easy and missing parts can be configured just by looking at the Foreman
website wiki/FAQ. But troubles came when I enabled reports :
puppetmasterd config was changed to include :</p>
<blockquote>
<p>[master]<br>
reports = store, foreman</p>
</blockquote>
<p>and the foreman.rb script (copied and modified from
/usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb)
integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports
dir. (Note : don't forget to update \$foreman_url).</p>
<p>But no reports were coming in Foreman. hmmm .... error message was :</p>
<blockquote>
<p>Report foreman failed: Could not send report to Foreman at
http://puppetmaster.mybeautifuldomain.com …</p></blockquote><p>We implemented <a href="http://puppetlabs.com">Puppet</a> as a configuration
management system at \$work , and Puppet is a great tool. Then I heard
about some dashboards that could be used on top of it. I've heard about
different dashboards (\$management_people *like* dashboards) like
<a href="http://puppetlabs.com/puppet/related-projects/dashboard/">Puppet-dashboard</a>
and <a href="http://theforeman.org/">Foreman</a>.</p>
<p>I was advised by several people to give Foreman a try and it's really
simple to install. Their
<a href="http://theforeman.org/projects/foreman/wiki/Installation_instructions">wiki</a>
covers basic installation and there is even a<a href="http://yum.theforeman.org/">yum
repo</a> that can be used (Epel has to be
enabled too). As i have a small network to manage, I decided to setup
Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is
easy and missing parts can be configured just by looking at the Foreman
website wiki/FAQ. But troubles came when I enabled reports :
puppetmasterd config was changed to include :</p>
<blockquote>
<p>[master]<br>
reports = store, foreman</p>
</blockquote>
<p>and the foreman.rb script (copied and modified from
/usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb)
integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports
dir. (Note : don't forget to update \$foreman_url).</p>
<p>But no reports were coming in Foreman. hmmm .... error message was :</p>
<blockquote>
<p>Report foreman failed: Could not send report to Foreman at
http://puppetmaster.mybeautifuldomain.com:3000/reports/create?format=yml:
Permission denied - connect(2)</p>
</blockquote>
<p>That was not an iptables issue, but selinux one :</p>
<blockquote>
<p>type=AVC msg=audit(1329830711.788:28372): avc: denied {
name_connect } for pid=13144 comm="puppetmasterd" dest=3000
scontext=unconfined_u:system_r:puppetmaster_t:s0
tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket</p>
</blockquote>
<p>Here is my locally generated selinux for Foreman :</p>
<blockquote>
<p>module foreman 1.0;</p>
<p>require {<br>
type puppetmaster_t;<br>
type http_port_t;<br>
type ntop_port_t;<br>
class tcp_socket name_connect;<br>
}</p>
<p>#============= puppetmaster_t ==============<br>
allow puppetmaster_t http_port_t:tcp_socket name_connect;<br>
allow puppetmaster_t ntop_port_t:tcp_socket name_connect;</p>
</blockquote>
<p>Things work really better after I added my foreman.pp selinux module on
that host. If you don't know how to compile selinux custom policies,
please read the nice <a href="http://wiki.centos.org/HowTos/SELinux">Selinux page on the CentOS
wiki</a>, and especially the
<a href="http://wiki.centos.org/HowTos/SELinux#head-aa437f65e1c7873cddbafd9e9a73bbf9d102c072">"Manually customizing selinux
policies"</a>
section. Tools like sealert (from setroubleshoot-server package) and
audit2allow are really helpful when there is no pre-defined selinux
boolean that can be used.</p>
<p>Hope this helps .. and now going back enjoying reports, including error
reports by mail (nice feature)</p>CentOS Automated QA explained ...2012-01-09T15:41:00+01:002012-01-09T15:41:00+01:00Fabian Arrotintag:arrfab.net,2012-01-09:/posts/2012/Jan/09/centos-automated-qa-explained/<p>While <a href="http://centosnow.blogspot.com/2012/01/centos-in-2012.html">Johnny was
explaining</a>
to the rest of the world how CentOS 6.1 and 6.2 were released, I
received quite some questions about the QA tests and how they were
performed. Well, let me explain in some words how it's now organized.
Previously, there was only a Tests Matrix that was shared between the QA
team members : each member of that group had access to the QA bits,
could download/rsync the complete tree (with ISO images too) and do his
tests, and then reported the results in one way or the other (irc,
mailing-list). Of course it didn't scale out very well. Too much manual
intervention, and when someone was busy with personal (or work related)
issues, no feedback was coming back to the CentOS devteam.</p>
<p>So during <a href="http://archive.fosdem.org/2011/">Fosdem 2011</a>, I had a
meeting with <a href="http://www.karan.org/blog/index.php">Karanbir</a> to see how
we could solve that issue and put automation in the QA loop. We
dedicated some (old) machines to be used only for QA, and in a separate
VLAN. Basically, here are the steps from the built bits to the QA
reports.</p>
<ul>
<li>The CentOS buildfarm (using the newly build system called 'reimzul'
and using <a href="http://kr.github.com/beanstalkd/">beanstalkd</a> as a
queuing system …</li></ul><p>While <a href="http://centosnow.blogspot.com/2012/01/centos-in-2012.html">Johnny was
explaining</a>
to the rest of the world how CentOS 6.1 and 6.2 were released, I
received quite some questions about the QA tests and how they were
performed. Well, let me explain in some words how it's now organized.
Previously, there was only a Tests Matrix that was shared between the QA
team members : each member of that group had access to the QA bits,
could download/rsync the complete tree (with ISO images too) and do his
tests, and then reported the results in one way or the other (irc,
mailing-list). Of course it didn't scale out very well. Too much manual
intervention, and when someone was busy with personal (or work related)
issues, no feedback was coming back to the CentOS devteam.</p>
<p>So during <a href="http://archive.fosdem.org/2011/">Fosdem 2011</a>, I had a
meeting with <a href="http://www.karan.org/blog/index.php">Karanbir</a> to see how
we could solve that issue and put automation in the QA loop. We
dedicated some (old) machines to be used only for QA, and in a separate
VLAN. Basically, here are the steps from the built bits to the QA
reports.</p>
<ul>
<li>The CentOS buildfarm (using the newly build system called 'reimzul'
and using <a href="http://kr.github.com/beanstalkd/">beanstalkd</a> as a
queuing system) pushes automatically each new tree to the dedicated
QA hardware</li>
<li>There is a rsync post-xfer script that is launched from there that
also uses beanstalkd and some workers (so we can scale out easily if
we add machines)</li>
<li>Each built and pushed tree/ISOs set has its own BuildTag (that is
used to identify what was tested and when)</li>
<li>Some tools (hosted in an internal Git repository) are then used to
deploy some Virtual Machines (actually a mix of BareMetal and VMs :
blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step"
failed (VMs are installed through ISO/pxe boot/virt-install through
http/ftp/nfs methods)</li>
<li>A test suite (that we call the t_functional stack) is then copied
from the local git repo to those newly deployed machines and each
test is then ran. From that point a report is then automatically
sent to the QA mailing-list so that people can see the results,
while the full log is available on QA head node.</li>
</ul>
<p>The fact that we use two separate git repositories (one for the
deploy/provisioniong functions and another one for the tests themselves)
was really a good thing, as it permitted some people to include their
tests in the t_functional stack. For example ,
<a href="http://athmane.wordpress.com/">Athmane</a> did a great job writing/fixing
some tests used for 6.1 and 6.2.</p>
<p>More informations to come later about how you (yes, *you*) can
participate and contribute such CentOS QA auto-tests !</p>CentOS 6 LiveCD and LiveDVD tools2011-07-28T14:29:00+02:002011-07-28T14:29:00+02:00Fabian Arrotintag:arrfab.net,2011-07-28:/posts/2011/Jul/28/centos-6-livecd-and-livedvd-tools/<p>The number of questions I received from different people regarding the
LiveCD/LiveDVD tools and the kickstart files used to produce the ISO
images was quite "high". People looking at the normal place will be
disappointed because we haven't used the original <a href="https://projects.centos.org/svn/livecd/">livecd subversion
repo</a> to produce the actual
Live medias. So in the meantime, if people want to use the
livecd-creator tool, they can fetch the SRPM here :
<a href="http://people.centos.org/arrfab/CentOS6/SRPMS/livecd-tools-0.3.6-1.el6.src.rpm">http://people.centos.org/arrfab/CentOS6/SRPMS/livecd-tools-0.3.6-1.el6.src.rpm</a>
. I've just copied also the two kickstart files used for both LiveCD and
LiveDVD here :<a href="http://people.centos.org/arrfab/CentOS6/LiveCD-DVD/">http://people.centos.org/arrfab/CentOS6/LiveCD-DVD/</a></p>
<p>Hope that people will be satisfied .. faster to push those files there
than to change the whole 'used behind the scene' infra</p>CentOS 6 ISO spins2011-07-26T19:39:00+02:002011-07-26T19:39:00+02:00Fabian Arrotintag:arrfab.net,2011-07-26:/posts/2011/Jul/26/centos-6-iso-spins/<p>As you've probably seen if you're subscribed to the <a href="http://lists.centos.org/pipermail/centos-announce/2011-July/017658.html">CentOS announce
list</a>
(or if you just rsync/mirror the <a href="http://mirror.centos.org/centos/">whole CentOS
tree</a>) , the CentOS 6.0 LiveCD was
released last monday. This is the first of our CentOS custom spins !
While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way
to the external mirrors too and will normally be announced shortly (when
enough mirrors will have it) ! It will be the second CentOS respin and
we have more in the pipe for you ! As Karanbir announced it in the <a href="http://lists.centos.org/pipermail/centos-announce/2011-July/017645.html">6.0
release
mail</a>
, we planned also to provide two other spins : the minimal one and the
lws one. Good news is that the minimal one is almost finished and being
intensively tested. If things don't change (or bugs appear during QA),
the iso image will be only \~250Mb for the i386 arch and \~300Mb for the
x86_64 one. It's meant to be used as a real basic CentOS system (even
less packages that the @core group on a normal install if used with the
proper kickstart invocation !) : 186 packages only on your disk. You'll
have a very basic CentOS system with only openssh-server and yum …</p><p>As you've probably seen if you're subscribed to the <a href="http://lists.centos.org/pipermail/centos-announce/2011-July/017658.html">CentOS announce
list</a>
(or if you just rsync/mirror the <a href="http://mirror.centos.org/centos/">whole CentOS
tree</a>) , the CentOS 6.0 LiveCD was
released last monday. This is the first of our CentOS custom spins !
While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way
to the external mirrors too and will normally be announced shortly (when
enough mirrors will have it) ! It will be the second CentOS respin and
we have more in the pipe for you ! As Karanbir announced it in the <a href="http://lists.centos.org/pipermail/centos-announce/2011-July/017645.html">6.0
release
mail</a>
, we planned also to provide two other spins : the minimal one and the
lws one. Good news is that the minimal one is almost finished and being
intensively tested. If things don't change (or bugs appear during QA),
the iso image will be only \~250Mb for the i386 arch and \~300Mb for the
x86_64 one. It's meant to be used as a real basic CentOS system (even
less packages that the @core group on a normal install if used with the
proper kickstart invocation !) : 186 packages only on your disk. You'll
have a very basic CentOS system with only openssh-server and yum. We are
even testing the luks/lvm/md devices combination to be sure to meet your
needs.</p>
<p>The next custom respin (LWS code name - for LightWeigth Server edition)
will still be a CD iso image (but pushed to the limit) that will include
basic server packages, more or less in the idea of the ServerCD that
existed during the CentOS 4.x days ... That one still needs to be
finished while work has already being done.</p>
<p>Stay tuned for more informations when it will be pushed to mirrors and
announced .. all that at the same time as 6.1 and 5.7 (in parallel)
builds ..Interesting times ! :-)</p>Modifying Anaconda behaviour without rebuilding the whole install media2011-06-11T14:54:00+02:002011-06-11T14:54:00+02:00Fabian Arrotintag:arrfab.net,2011-06-11:/posts/2011/Jun/11/modifying-anaconda-behaviour-without-rebuilding-the-whole-install-media/<p>One thing that I had to have a look at (during CentOS 6 QA), is the way
<a href="http://fedoraproject.org/wiki/Anaconda">anaconda</a> (the Red
Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to
those kind of install know what I'm talking about : the "Mininal",
"Desktop", "Basic Server" and other choices you have during setup. From
that first selection, you can decide (or not) to customize the software
selection which then leads you to a screen containing categories /
groups / packages defined in the comps.xml file present under /repodata
on the tree/install media.</p>
<p>If you don't 'see' which screen i'm talking about, a small screenshot of
the upcoming CentOS 6 will explain better than words :</p>
<p><img alt="Anaconda in CentOS" src="/images/anaconda-centos.png" title="anaconda-centos"></p>
<p>Those pre-defined tasks aren't defined in the comps.xml file but rather
at build time within anaconda. Fine but how can you 'modify' anaconda
behaviour and test it without having to patch anaconda SRPM, rebuild it
and launch a new build to generate the tree and install medias ? Easy ,
thanks to a simple file on the tree !</p>
<p>People wanting to modify anaconda behaviour at install time without
having to regenerate the whole tree can just create a small file
(updates.img) , put it in the /images directory in …</p><p>One thing that I had to have a look at (during CentOS 6 QA), is the way
<a href="http://fedoraproject.org/wiki/Anaconda">anaconda</a> (the Red
Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to
those kind of install know what I'm talking about : the "Mininal",
"Desktop", "Basic Server" and other choices you have during setup. From
that first selection, you can decide (or not) to customize the software
selection which then leads you to a screen containing categories /
groups / packages defined in the comps.xml file present under /repodata
on the tree/install media.</p>
<p>If you don't 'see' which screen i'm talking about, a small screenshot of
the upcoming CentOS 6 will explain better than words :</p>
<p><img alt="Anaconda in CentOS" src="/images/anaconda-centos.png" title="anaconda-centos"></p>
<p>Those pre-defined tasks aren't defined in the comps.xml file but rather
at build time within anaconda. Fine but how can you 'modify' anaconda
behaviour and test it without having to patch anaconda SRPM, rebuild it
and launch a new build to generate the tree and install medias ? Easy ,
thanks to a simple file on the tree !</p>
<p>People wanting to modify anaconda behaviour at install time without
having to regenerate the whole tree can just create a small file
(updates.img) , put it in the /images directory in the tree. Anaconda
(when installing over the network, http/ftp/nfs) always try to see if an
updates.img file exists, and if so, use it. Fine, so I could easily try
to "patch" it without having to modify the whole tree.</p>
<p>Creating that updates.img (it's just a ext2 filesystem on top) is really
easy :</p>
<div class="highlight"><pre><span></span><span class="nv">dd</span> <span class="k">if</span><span class="o">=/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">zero</span> <span class="nv">of</span><span class="o">=/</span><span class="nv">tmp</span><span class="o">/</span><span class="nv">updates</span>.<span class="nv">img</span> <span class="nv">bs</span><span class="o">=</span><span class="mi">1</span><span class="nv">k</span> <span class="nv">count</span><span class="o">=</span><span class="mi">1440</span>
<span class="nv">losetup</span> \`<span class="nv">losetup</span> <span class="o">-</span><span class="nv">f</span>\` <span class="o">/</span><span class="nv">tmp</span><span class="o">/</span><span class="nv">updates</span>.<span class="nv">img</span>
<span class="nv">losetup</span> <span class="o">-</span><span class="nv">a</span><span class="o">|</span><span class="nv">grep</span> <span class="nv">updates</span>.<span class="nv">img</span>
<span class="nv">mkfs</span>.<span class="nv">ext2</span> <span class="o">/</span><span class="nv">dev</span><span class="o">/</span><span class="nv">loop3</span> \# <span class="nv">was</span> <span class="nv">loop3</span> <span class="nv">in</span> <span class="nv">my</span> <span class="nv">case</span>
<span class="nv">mkdir</span> <span class="o">/</span><span class="nv">mnt</span><span class="o">/</span><span class="k">loop</span> <span class="c1">; mount -o loop /tmp/updates.img /mnt/loop/ ; ll</span>
<span class="o">/</span><span class="nv">mnt</span><span class="o">/</span><span class="k">loop</span>
<span class="nv">drwx</span><span class="o">------</span>. <span class="mi">2</span> <span class="nv">root</span> <span class="nv">root</span> <span class="mi">12288</span> <span class="nv">Jun</span> <span class="mi">11</span> <span class="mi">15</span>:<span class="mi">43</span> <span class="nv">lost</span><span class="o">+</span><span class="nv">found</span>
</pre></div>
<p>From now, it's just a matter of putting the new files that you want to
test and that will "overwrite" at run-time the defaults anaconda ones.</p>
<p>(in our current example, it was the installclasses/rhel.py that needed
to be modified, so I just had to create a installclasses dir and drop my
version of rhel.py in there on the loop device)</p>
<p>When you're done, umount the updates.img, copy it to
/path/to/your/install/tree/images , restart a http install (verify that
permissions and selinux contexts are of course correct !) and enjoy !</p>
<p>Easier and faster. Thanks to the Anaconda team which decided to permit
modifying the anaconda behaviour at run-time with a simple file :-)</p>IPV6 world day !2011-06-09T21:24:00+02:002011-06-09T21:24:00+02:00Fabian Arrotintag:arrfab.net,2011-06-09:/posts/2011/Jun/09/ipv6-world-day/<p>It seems quite a lot of people blogged about<a href="http://www.worldipv6day.org/">IPV6
day</a> . It's true that it's always a good
idea to speak about IPV6. I'm using IPV6 natively on my server hosted
at<a href="http://www.hetzner.de/en/">Hetzner</a> (they offer a /64 IPV6 subnet,
which is more than enough for a <a href="http://www.arrfab.net/blog/?p=271">CentOS server hosting several xen domU
Virtual Machines</a>). At home, that's
another story. I use a<a href="http://ipv6.he.net/">HE.net</a> <a href="http://www.tunnelbroker.net/">free
tunnel</a> to be able to reach ipv6 hosts.
Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ?
that's indeed a good question. Even if my CentOS ipv6 tunnel
end-point/router/radvd at home is working correctly, I decided to ask my
belgian provider if they had plans on implementing native IPV6. Well,
not for my home connection, as I already know that
<a href="http://www.belgacom.be/privathttp://www.belgacom.be/private/hbsres/jsp/dynamic/homepage.jsp">Belgacom</a>
(the biggest provider in belgium) doesn't support IPV6 on their BBOX2
modems that they give to customers when ordering a DSL connection at
home (<em>while i'm talking about Belgacom, please stop sending me direct
advertisement to my mailbox - the real one and not the electronic one -
with your invoices about a service - VDSL2/BelgacomTV - that you
*can't* offer to all your customers ... thanks</em>) . So I decided to …</p><p>It seems quite a lot of people blogged about<a href="http://www.worldipv6day.org/">IPV6
day</a> . It's true that it's always a good
idea to speak about IPV6. I'm using IPV6 natively on my server hosted
at<a href="http://www.hetzner.de/en/">Hetzner</a> (they offer a /64 IPV6 subnet,
which is more than enough for a <a href="http://www.arrfab.net/blog/?p=271">CentOS server hosting several xen domU
Virtual Machines</a>). At home, that's
another story. I use a<a href="http://ipv6.he.net/">HE.net</a> <a href="http://www.tunnelbroker.net/">free
tunnel</a> to be able to reach ipv6 hosts.
Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ?
that's indeed a good question. Even if my CentOS ipv6 tunnel
end-point/router/radvd at home is working correctly, I decided to ask my
belgian provider if they had plans on implementing native IPV6. Well,
not for my home connection, as I already know that
<a href="http://www.belgacom.be/privathttp://www.belgacom.be/private/hbsres/jsp/dynamic/homepage.jsp">Belgacom</a>
(the biggest provider in belgium) doesn't support IPV6 on their BBOX2
modems that they give to customers when ordering a DSL connection at
home (<em>while i'm talking about Belgacom, please stop sending me direct
advertisement to my mailbox - the real one and not the electronic one -
with your invoices about a service - VDSL2/BelgacomTV - that you
*can't* offer to all your customers ... thanks</em>) . So I decided to ask
their 'professional services' because we have two 'professional and
business' lines that we used at \$work. Long story short (to avoid
explaining how much emails/cases I had to send/open to have an answer) :
"no, even on the business lines we can't support IPV6 and we have no
plans (*sic*, I hope that guy was just kidding or probably doesn't
know the real answer ..) nor dates about future implementation of the
IPV6 services/connectivity " ..</p>
<p>Nice .. now /me goes back to CentOS QA mode ...</p>Bye-Bye Nokia .. we loved you, until now2011-02-12T10:25:00+01:002011-02-12T10:25:00+01:00Fabian Arrotintag:arrfab.net,2011-02-12:/posts/2011/Feb/12/bye-bye-nokia-we-loved-you-until-now/<p>I couldn't believe what I read yesterday .. but yes, Nokia has decided
to sign a special partnership with Microsoft to load WM7 on their
mobiles phones in the future .. The end of Symbian and Meego/Maemo ..
sad, sad .. time for me to think about my next mobile phone. Official
Nokia announce here
:<a href="http://conversations.nokia.com/nokia-strategy-2011/">http://conversations.nokia.com/nokia-strategy-2011/</a></p>What do you want to see ? CentOS 5.6 or CentOS 6.0 ?2011-01-13T22:17:00+01:002011-01-13T22:17:00+01:00Fabian Arrotintag:arrfab.net,2011-01-13:/posts/2011/Jan/13/what-do-you-want-to-see-centos-5-6-or-centos-6-0/<p>As you probably know (if you are interested in the Enterprise Linux
market), <a href="https://www.redhat.com/archives/rhelv5-announce/2011-January/msg00000.html">Red Hat released earlier today
5.6</a>
. So automatically some CentOS QA team members started to discuss about
that in the appropriate IRC channel. As CentOS 6.0 isn't (yet) released
nor ready, the discussion was about putting 5.6 build & release as
priority number one or not. Karanbir on his side <a href="http://twitter.com/CentOS/status/25505187548368897">asked on
Twitter</a> about
thoughts on the matter, and a discussion was started too on the
<a href="http://lists.centos.org/pipermail/centos-devel/2011-January/006511.html">centos-devel</a>
list about that topic. My personal opinion (and shared by some people
too) seems to give 5.6 the priority for quite some reasons :</p>
<ul>
<li>The centos 5.x install base is there while there is (obviously) no
centos 6 install base.</li>
<li>So those people having machines in production, faced to the net (,
etc, etc, ...) would prefer having their machines patched and
up2date (security first !)</li>
<li>People running CentOS 5.x on servers and willing to install php53
packages, now officially included</li>
<li>On the build side, the el5 build process is clearly identified and
known since 2007 : packages with branding issues are already
identified and patches/artwork is already there, meaning that it
will be <span style="text-decoration: line-through;">probably</span>
(no, surely !) faster to …</li></ul><p>As you probably know (if you are interested in the Enterprise Linux
market), <a href="https://www.redhat.com/archives/rhelv5-announce/2011-January/msg00000.html">Red Hat released earlier today
5.6</a>
. So automatically some CentOS QA team members started to discuss about
that in the appropriate IRC channel. As CentOS 6.0 isn't (yet) released
nor ready, the discussion was about putting 5.6 build & release as
priority number one or not. Karanbir on his side <a href="http://twitter.com/CentOS/status/25505187548368897">asked on
Twitter</a> about
thoughts on the matter, and a discussion was started too on the
<a href="http://lists.centos.org/pipermail/centos-devel/2011-January/006511.html">centos-devel</a>
list about that topic. My personal opinion (and shared by some people
too) seems to give 5.6 the priority for quite some reasons :</p>
<ul>
<li>The centos 5.x install base is there while there is (obviously) no
centos 6 install base.</li>
<li>So those people having machines in production, faced to the net (,
etc, etc, ...) would prefer having their machines patched and
up2date (security first !)</li>
<li>People running CentOS 5.x on servers and willing to install php53
packages, now officially included</li>
<li>On the build side, the el5 build process is clearly identified and
known since 2007 : packages with branding issues are already
identified and patches/artwork is already there, meaning that it
will be <span style="text-decoration: line-through;">probably</span>
(no, surely !) faster to have 5.6 out of the door than 6</li>
<li>Same rule for the QA process : people from the QA team can "blindly"
focus on their previous tests, and just have a look eventually at
some newer packages (a few, like php53 but not that much in
comparison with el6)</li>
</ul>
<p>Please notice that it's still my <em>personal opinion</em> on that question and
isn't the (to be defined) official CentOS position.</p>CentOS team @ Fosdem 20112011-01-10T15:49:00+01:002011-01-10T15:49:00+01:00Fabian Arrotintag:arrfab.net,2011-01-10:/posts/2011/Jan/10/centos-team-fosdem-2011/<p>Some members of the CentOS team will be present at the Fosdem . Feel
free to come at our booth just to discuss ...</p>
<p>More informations on <a href="http://wiki.centos.org/Events/Fosdem2011">our
wiki</a> and on the
<a href="http://www.fosdem.org/">Fosdem</a> website</p>Enabling IPv6 for guests on an Hetzner CentOS 5.5 xen dom02010-12-31T11:28:00+01:002010-12-31T11:28:00+01:00Fabian Arrotintag:arrfab.net,2010-12-31:/posts/2010/Dec/31/enabling-ipv6-for-guests-on-an-hetzner-centos-5-5-xen-dom0/<p>I was playing with IPv6 in the last days (started to use a tunnel from
<a href="http://www.tunnelbroker.net/">he.net</a> as my current ISP doesn't
support native IPv6 and doesn't plan to support it in a short time) and
wanted to add IPv6 to some of my CentOS Xen domU's running on a
<a href="http://www.hetzner.de">Hetzner</a> box. This part was a little bit more
difficult than for a standard network. Due to their internal network
design, Hetzner <a href="http://translate.google.be/translate?u=http%3A%2F%2Fwiki.hetzner.de%2Findex.php%2FZusaetzliche_IP-Adressen&sl=de&tl=en&hl=&ie=UTF-8">only
allow</a>
'routed' xen networks and not standard 'bridged' ones. What I used for
IPv4 was just binding the public IPs on the dom0 and configured all my
iptables rules there to forward/SNAT/DNAT to the appropriate domU. But
you know that NAT is gone with IPv6 so normally it's supposed to be
easier, right ? Well, yes and no, depending on your network layout. Even
after having enabled ipv6 forwarding (net.ipv6.conf.all.forwarding=1 ),
I was just able to ping the dom0 but not the guests behind. Hmm, that
reminds me the <a href="http://en.wikipedia.org/wiki/Proxy_arp">proxy ARP</a> that
was used for IPv4 but not existing anymore for IPv6 (gone too ...) . ARP
was (more or less, not technically correct but read the RFCs if you
enough time) replaced by …</p><p>I was playing with IPv6 in the last days (started to use a tunnel from
<a href="http://www.tunnelbroker.net/">he.net</a> as my current ISP doesn't
support native IPv6 and doesn't plan to support it in a short time) and
wanted to add IPv6 to some of my CentOS Xen domU's running on a
<a href="http://www.hetzner.de">Hetzner</a> box. This part was a little bit more
difficult than for a standard network. Due to their internal network
design, Hetzner <a href="http://translate.google.be/translate?u=http%3A%2F%2Fwiki.hetzner.de%2Findex.php%2FZusaetzliche_IP-Adressen&sl=de&tl=en&hl=&ie=UTF-8">only
allow</a>
'routed' xen networks and not standard 'bridged' ones. What I used for
IPv4 was just binding the public IPs on the dom0 and configured all my
iptables rules there to forward/SNAT/DNAT to the appropriate domU. But
you know that NAT is gone with IPv6 so normally it's supposed to be
easier, right ? Well, yes and no, depending on your network layout. Even
after having enabled ipv6 forwarding (net.ipv6.conf.all.forwarding=1 ),
I was just able to ping the dom0 but not the guests behind. Hmm, that
reminds me the <a href="http://en.wikipedia.org/wiki/Proxy_arp">proxy ARP</a> that
was used for IPv4 but not existing anymore for IPv6 (gone too ...) . ARP
was (more or less, not technically correct but read the RFCs if you
enough time) replaced by
<a href="http://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol">NDP</a> but I
don't see such option for IPv6. Well, a kernel feature called proxy_ndp
(net.ipv6.conf.all.proxy_ndp=1) exists on newer kernels (like for
example the 2.6.32.x that is used on RHEL6 , and so in CentOS 6) but not
on CentOS 5.5 (using a 2.6.18.x) kernel .. Hmmm ...</p>
<p>On the other side, I was searching for a 'workaround' probably given by
libvirt, but the version included in RHEL5/CentOS5 doesn't know what to
do with IPv6. Okay so let's have a look at the Xen and kernel side at
the same time. If the proxy_ndp kernel feature is not present on my
CentOS 5.5 dom0, I can still 'advertise' my neighbors with the ip
command : yes, it supports it : " ip -6 neighbor add proxy
your:ipv6:long:address::1 dev eth0"</p>
<p>So we just need to create a modified vif-route script (in fact I decided
to call it vif-route6) that will be used for ipv6 guests :</p>
<blockquote>
<p>#!/bin/bash </p>
<p>#============================================================================<br>
# /etc/xen/scripts/vif-route6<br>
# Script for configuring a vif in routed mode for IPv6 only<br>
# Based on existing vif-route script in /etc/xen/scripts and adapted
for ipv6 </p>
<p>#============================================================================</p>
<p>dir=\$(dirname "\$0")<br>
. "\$dir/vif-common.sh"</p>
<p>main_ip=\$(dom0_ip)<br>
main_ip6=\$(ip -6 addr show eth0|grep 'scope global'|sort|head -n
1|awk '{print \$2}'|cut -f 1 -d '/')</p>
<p>case "\$command" in<br>
online)<br>
ifconfig \${vif} \${main_ip} netmask 255.255.255.255 up<br>
ip -6 addr add \${main_ip6} dev \${vif}<br>
ipcmd='add'<br>
cmdprefix=''<br>
;;<br>
offline)<br>
do_without_error ifdown \${vif}<br>
ipcmd='del'<br>
cmdprefix='do_without_error'<br>
;;<br>
esac</p>
<p>if [ "\${ip}" ] ; then<br>
# If we've been given a list of IP addresses, then add routes from
dom0 to<br>
# the guest using those addresses.<br>
for addr in \${ip} ; do<br>
\${cmdprefix} ip -6 neighbor \${ipcmd} proxy \${addr} dev
\${netdev:-eth0} 2>&1<br>
result=`\${cmdprefix} ip -6 route \${ipcmd} \${addr} dev \${vif} src
\${main_ip6} 2>&1`<br>
done<br>
fi</p>
<p>handle_iptable</p>
<p>log debug "Successful vif-route \$command for \$vif."<br>
if [ "\$command" = "online" ]<br>
then<br>
success<br>
fi</p>
</blockquote>
<p>Ok, so we have just now to modify our xen domU's config to add a vif
that will use that specific script and give it the IPv6 address that
we'll assign to that domU (from /etc/xen/your-domU-name):</p>
<blockquote>
<p>vif = [ \<snip of the first vif> ,
"mac=00:16:36:38:31:b8,vifname=test.ipv6,script=vif-route6,ip=2a01:4f8:100:4363::dead"
]</p>
</blockquote>
<p>You can now start your domU and configure it normally for IPv6 (using
obviously that 2a01:4f8:100:4363::dead IPv6 address and choosing the
dom0 main IPv6 address as gateway ...</p>
<p>Hope it will help some people in the same situation (using a routed and
not a bridged network layout for xen)</p>