Deploying OpenStack on AWS

A Cloud on a Cloud!

Yes! You read it correctly. This post will guide you to install an OpenStack on top of AWS EC2.

Installing OpenStack on nested hypervisor environment is not a big deal when we use the QEMU emulator for launching virtual machines inside a virtual machine. However, unlike a usual nested hypervisor setup, Installing OpenStack on AWS EC2 instances has a few restrictions on the networking part for the OpenStack setup to work properly. This blog outlines the limitations and their solutions to run OpenStack on top of an AWS EC2 VM.

Limitations

The AWS environment will allow packets to flow in their network only when the MAC address is known/registered in the AWS network environment.

Also, the MAC address and the IP address are tightly mapped, so the AWS environment will not allow packet flow if the MAC address registered for the given IP address is different.

You may wonder whether the above restrictions will impact the OpenStack setup on AWS EC2.

Yes! Yes, it will!

While configuring the Neutron networking, we would be creating a virtual bridge (say, br-ex ) for the provider network where all the VM's traffic will reach the Internet via the external bridge, followed by the actual physical NIC (say, eth1).

In that case, we usually configure the external interface (NIC) with a special type of configuration, as follows.

The provider interface uses a special configuration without an IP address assigned to it. Configure the second interface as the provider interface. Replace INTERFACE_NAME with the actual interface name. For example, eth1 or ens224. Edit the /etc/network/interfaces file to contain the following:

# The provider network interface
auto INTERFACE_NAME
iface INTERFACE_NAME inet manual
up ip link set dev $IFACE up
down ip link set dev $IFACE down

Ref: http://docs.openstack.org/mitaka/install-guide-ubuntu/environment-networking-controller.html

Due to this special type interface configuration, the restriction in AWS will hit OpenStack's networking. In a mainstream OpenStack setup, the above-mentioned provider interface would be configured with a special NIC configuration that would have no IP for that interface and would allow all packets via that specially configured NIC.

Moreover, the VM packets reaching Internet via this specially configured NIC would have the IP of OpenStack tenant router’s gateway IP address as the source IP address in each packet.

Like I mentioned earlier in the limitations above, AWS will only allow the packet flow when the MAC address is known/registered in their environment. Also, the IP address must match the MAC address.

In our case, the packet from the above-mentioned OpenStack tenant router will have the IP address of router’s Gateway in every single packet, and the packet source MAC address will be the MAC address of router’s interface.

Note: You could see these details using “ip netns show” followed by the “ip netns exec qr- ifconfig” command in the OpenStack controller’s terminal.

Since the MAC address in unknown/not registered in the AWS environment, the packets will be dropped when it reaches the AWS switch. To allow the VM packets to reach the Internet via AWS switch, we need to do some tricks/hacks in our OpenStack setup.

Make Use of What We Have

The possible ways are:

• Register the router’s MAC address and its IP address with AWS environment. However, this is not feasible. AWS currently does not have the features available to register any random MAC address and IP address inside the VPC. Moreover, allowing this type of functionality would be a severe security threat to the environment.

• The other method is to make use of what we have. Since we have used a special type of interface configuration for the provider NIC, you could note that the IP address assigned to the provider NIC (say, eth1) is left unused. We could use this available/unused IP address for the OpenStack router’s gateway.

The below command will do that trick,

neutron router-gateway-set router provider --fixed-ip ip_address=<Registered_IP_address*>

IP Address and MAC Address Mismatch

After configuring router gateway with the AWS-registered IP address, each packet from the router’s gateway will have the AWS-registered IP address as the source IP address — but with the OVS-generated, unregistered MAC address.

Like I mentioned in the AWS restriction session above, the IP address must match the MAC address registered else all the packets with mismatched mac and IP address will be dropped by the AWS switch.

To make the registered MAC address match the IP address, we need to change the MAC address of router’s interface. The below-mentioned steps will do that magic.

1. Install macchanger.

2. Note down the actual/original MAC address of the provider NIC (eth1).

3. Change the MAC address of the provider NIC (eth1).

4. Change the MAC address of the router’s gateway interface to the original MAC address of eth1.

5. Now, try to ping 8.8.8.8 from the router namespace.

If you get a successful ping response, then we are done with Cloud on Cloud setup.

Key Points to Remember

1. Change the MAC address: In my case, I had worked in a Ubuntu 14.04 LTS server, which has no issue in changing the MAC address using macchanger tool. However, when I tried in Ubuntu 16.04 LTS, I got an error saying, "No permission to modify the MAC address." I suspect the error was due to the cloud-init tool not allowing the MAC address configuration. So, before installing OpenStack setup, try changing the MAC address of the NIC.

2. Floating IP disabled: Associating a floating IP to any OpenStack VM will send the packet via the router’s gateway with the source IP address as a floating IP address. This will make the packets to hit the AWS switch with non-registered IP and MAC addresses, which results in dropping the packets. So, we could not use the floating IP functionality in this setup. However, still, we could access the VM publicly using the below mentioned NAT process.

3. NAT to access OpenStack VM: Like I mentioned above, we could access the OpenStack VM publicly using the registered IP address that we have assigned for our router’s gateway. Use the following NAT command to access the OpenStack VM using the AWS EC2 instance’s elastic IP:

   $ ip netns exec qrouter-f85bxxxx-61b2-xxxx-xxxx-xxxxba0xxxx iptables -t nat -A PREROUTING -p tcp -d 172.16.20.101 --dport 522 -j DNAT --to-destination 192.168.20.5:22

   
   

   Note: In the above command, I had a NAT for forwarding all packets for 172.16.20.101 with port 522. Using the above NAT command, all the packets reaching 172.16.20.101 with port number 522 will be forwarded to 192.168.20.5:22. Here, 172.16.20.101 is the registered IP address of the AWS EC2 instance, which was assigned for the router’s gateway, and 192.168.20.5 is the local IP of the OpenStack VM. Notably, 172.16.20.101 already has a NAT with the AWS Elastic IP, which means all the traffic that comes to the elastic IP (public IP) will be forwarded to this VPC local IP (172.16.20.101).
   
   In short: [Elastic IP]:522 > 172.16.20.101:522 > 192.168.20.5:22. This means you could SSH the OpenStack VM globally by using the elastic IP address and the respective port number.

4. Elastic IP address: For this type of customized OpenStack installation, we required at least two NICs for the AWS EC2 instance. You need one for accessing the VM terminal for the installation and for accessing the dashboard. In short, it acts as a management network/VM tunnel network/API network.
   
   The second one is for an external network with a unique type of interface configuration and mapped with the provider network bridge, say br-ex with eth1. AWS will not allow any packets to travel out of the VPC unless the elastic IP is attached with that IP address. To overcome this problem, we must attach the elastic IP for this NIC. The packets of OpenStack’s VM will reach the OpenStack router’s gateway and, from the gateway, the packets get embedded with the registered MAC address. Hence, the matching IP address will reach the AWS switch (VPC environment) via br-ex and eth1 (our special interface configuration) and then hits the AWS VPC gateway. From there, the packets will reach the Internet.

Other Cloud Platforms

In my analysis, I could see most cloud providers like Dreamhost or Auro-cloud having the same limitations for OpenStack networking. So we could use the tricks/hacks mentioned above in any of those cloud providers to run an OpenStack cloud on top of it.

Note: Since we are using a QEMU emulator without KVM for the nested hypervisor environment, the VM performance will be slow.