OpenStack with Open vSwitch and Quantum (Folsom) on Scientific Linux

We decided to run OpenStack on Scientific Linux 6, a Red Hat derivative. Although this was not a reference platform (this was before Red Hat announced it would support OpenStack), we decided not to go with Ubuntu due to various factors. Knowledge and availability of an in-house infrastructure tailored to Red Hat based machines were the most important deciding factors.

With the availability of OpenStack packages through the EPEL repository it made running OpenStack on Scienfic Linux a manageable endeavour. Installing was basically just following the install guide.

Quantum networking and Open vSwitch

At this point it gets interesting. We probably spent more time on setting up Quantum than all other components combined. We decided on running with Quantum because it adds a lot of flexibility to the networking setup and a lot of interesting developments are coming out of the Quantum project (e.g. loadbalancer functionality is added in Grizzly). We are using the ability to set up a private network for a project setup with GRE tunnels. This makes it possible to create tenant networks “on the fly” without the need of setting up VLANs for network segmentation. Segmentation and connectivity between hosts is handled by the Open vSwitch. GRE is only supported with the Open vSwitch plugin, so we also had to get familiar with Open vSwitch at the same time as figuring out the Quantum parts

Installing and an RPM shopping list

Installing configuring in itself is not that hard, so start by following the manual. In our specific setup we noticed a few things that are not immediately or obviously addressed in the manual so we describe those here.

Up to Scientific Linux 6.3 there was no support for Open vSwitch in the kernel.
We built our own package from source with dkms support to keep it working with kernel upgrades. From Scientific Linux 6.4 and upwards Open vSwitch is part of the kernel, however it’s currently missing patch-port / GRE tunnel functionality. This meant we decided to keep using the module from upstream.

Small tip: The nice way to override loading your own module instead of the one shipped by Scientific Linux is by creating a depmod rule in /etc/depmod.d/  (e.g. override openvswitch * weak-updates/openvswitch)

The libvirt version which comes with Scientific Linux 6.3 (at our time of installation) does not come with Open vSwitch support. Make sure you use libvirt >= 0.9.11.

If you run into issues with a gateway ip not being set on the DHCP clients, make sure you run a recent version of dnsmasq. We are using version 2.63.

Troubleshooting Open vSwitch

Open vSwitch was new technology for us. Since there are so many possibilities with this software switch it takes some time to get acquainted and to find ways to get the most useful information from it. I found these two commands usually get you the information you want:

ovs-vsctl show

This prints a brief overview of the Open vSwitch database. With this command you can see which switches and ports are configured.

ovs-ofctl dump-flows <switchname>

Prints the (open)flow entries. This is where a lot of the magic happens, e.g. here is defined what traffic is routed where and what is allowed. Another thing the flows can do is changing properties of the traffic, e.g. changing VLAN ids which is used for bridge mapped networks.

Other things we ran into

There are a few other things we noticed during installation. They are documented but you might overlook them, so here is a list of the most important things to remember or to check if things do not work as expected.

Namespaces

Namespaces are not supported due to lack of kernel support on Scientific Linux 6. This is why overlapping IP segments are not allowed. You can find documentation here

MTU issues

There is an MTU issue with bridge networking. This is why we create an ifcfg mtu configuration for the patch interface. You can test whether you are effected by this issue by doing a ping with a larger packet size. Small packets will work, larger packets (>1500) will not.

https://bugs.launchpad.net/quantum/+bug/1075336

Broken networking after reboot

There is a known issue with broken networking after a reboot: Open vSwitch interfaces are not properly brought up, which results in a non-functional quantum-dhcp-agent and quantum-l3-agent.

Make sure you run the quantum-ovs-cleanup utility on reboot prior to the quantum-dhcp and quantum-l3 agent startup.

TCP packets not arriving

We had a hard nut to crack with the machine we installed the quantum-l3-router on. Pings were working fine, but any regular traffic just did not work. After some time we noticed that UDP traffic worked but TCP traffic would only go one way. TCP packets sent to the machine never arrived at the interface. The issues we had seem to be related to TCP offloading, so we tried disabling TCP offloading wherever possible, but unfortunately we did not get it to work.
Eventually we switched to different hardware, which solved the problem.
The ethernet controller we had this issue with was a Emulex Corporation OneConnect 10Gb NIC (be3) (rev 01) in a HP ProLiant BL460c G7.

Init scripts

The init scripts from EPEL have a hard-coded log entry: It includes a –logdir which will make the application always log to file. When syslog is configured it will still log to the local filesystem. There are two bugreports for this:
https://bugzilla.redhat.com/show_bug.cgi?id=892626
https://bugzilla.redhat.com/show_bug.cgi?id=892656

Bridge-mapped networks with IP ranges

Creating bridge mapped networks with a limited ip range did not work through the quantum commandline tool for us.
We used the Python script below to set this up.