Openstack installation gotchas - kennethjiang/Wolke GitHub Wiki

In some instance you might want to use a VM as router, e.g., when you use VM as a OpenVPN server, in either routed or bridged mode. However this won't work with default Openstack setting. This is because Openstack (thru nwfilter in libvirt) sets up ebtables rules that check Mac address and IP src address in all packets from a VM against its record to see if they match, and drop any of those that don't.

When a VM is set up as a router, it will send/receive packets with IP src addresses that don't belong to it. All these packets are therefore blocked by ebtables.

One of the workarounds is, using OpenVPN as example, to use SNAT for routing purpose. SNAT has limitations compared with IP routing, but it satisfied Openstack's rules of matching Mac/IP src address.

Another issue related with ebtables was DHCP packets initiated from a Windows VMs. Occasionally for some reason Windows VMs would send a couple of DHCP renew request to broadcast MAC address. Some ebtables rules established by one of these configuration files would block this type of packets:

/etc/libvirt/nwfilter/allow-dhcp.xml
/etc/libvirt/nwfilter/allow-dhcp-server.xml
/etc/libvirt/nwfilter/nova-allow-dhcp-server.xml (programmatically generated by /usr/lib/python2.7/dist-packages/nova/virt/libvirt/firewall.py)

Workaround:

virsh nwfilter-edit allow-dhcp-server

remove {srcipaddr='0.0.0.0' dstipaddr='255.255.255.255'} from

<rule action='accept' direction='out' priority='100'>
    <ip srcipaddr='0.0.0.0' dstipaddr='255.255.255.255' protocol='udp' srcportstart='68' dstportstart='67'/>
</rule> 

Then run:

service libvirt-bin restart

Occasionally libvirtd stop responding for some reasons yet to be found out. This will block nova-compute process (nova-compute will also hangs). The symptom of this problem is that VMs requested will stay in "Build" status indefinitely. Check nova-scheduler.log and there will be this error:

(nova.rpc): TRACE:     raise driver.NoValidHost(_("Scheduler was unable to locate a host"
(nova.rpc): TRACE: NoValidHost: Scheduler was unable to locate a host for this request. Is the appropriate service running?

The ultimate solution is to find out why libvirtd hangs. Current work-around is to run this cron job:

0,10,20,30,40,50 * * * * if ! timeout 5s /usr/bin/virsh list >>/var/log/monitor-libvirt.log 2>&1; then PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin service libvirt-bin stop >>/var/log/monitor-libvirt.log 2>&1; PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin service libvirt-bin start >>/var/log/monitor-libvirt.log 2>&1; fi