How To: Install Openstack Icehouse on CentOS 6.5 Using Packstack

openstack-software-diagram

A brief introduction of Openstack + My thoughts

Openstack is open-source software for building clouds. It was created in 2010 by people from Rackspace & NASA, but is currently managed by the non-profit Openstack Foundation, which includes members from the who’s who of the technology sector that have joined forces to continue to invest & develop Openstack (written in Python) for fun and profit( well not so much ). It is definitely fun though :-). In fact I have been having nothing, but a blast since the moment I meant Openstack in 2012. However, admittedly the first time I worked on Openstack it was on the Essex release and I felt it had a ways to go before it was ready for prime time. At that time it was not only hard to install, but most of the people around it were hard core Pythonista’s (Python developers), that were rapidly trying to a mature Openstack & it’s ecosystem that just wasn’t production ready yet. And while the latest release of Openstack ‘Icehouse‘ (which I am covering in this blog) is leaps and bounds better, it can still be quite a PITA(Pain In The Ass) for first timers. In fact I still struggle with the idea of running Openstack in production because it requires an incredible amount of resources, engineering skill & organizational persistence to do so. Companies willing to embark on this journey thus far have included Mercado Libre, Dell, HP, Redhat, Canonical (the Ubunut guys), eBay, PayPal and Symantec among many others. These companies were required to beef up their engineering staff & allocate large amounts of resources just to take on this challenge. In addition their commitments were probably tested over and over again and their leadership had to respond with tremendous faith when timelines began to slip. However, organizations that complete this journey recognize that the return at the end of the tunnel is a compounding return. For starters you aren’t locked into to a vendor like VmWare or AWS (Amazon Web Services) who are both incredibly expensive, but the compounding return comes from skills it will build internally and the culture that will be a bi-product.

Coming here to Tuxlabs means you have personally accepted this mission to learn Openstack and you are in need of some guidance don’t worry you are in good hands. I will show you the light, which can be hard to see through the darkness if you try to take on the documentation all by yourself : http://docs.openstack.org/

So sit back, relax, read, and type exactly as I do. In the end you will have a perfectly functioning Openstack Icehouse cloud and if you decide to bring this into your organization remember with great power comes great responsibility…

You will change everyone in your organization from a POSA (Plain Old Sys Admin’s) to Cloud Engineers and Architects jumping into the Openstack Python code base at the sniff of an issue.

Devstack

Devstack is a shell script used to quickly and easily deploy an all-in-one install of Openstack on any machine or VM for the purposes of trying, testing, and developing on Openstack. It is extremely easy to install, use and get going, if you have not used Devstack already I recommend trying it first because it is a quick and easy way to get a learning win + it will help you decide what you want to learn more about and whether or not Openstack is what you are looking for. However, you do not want to run Devstack in production for starters because production deployments should be multi-node setup’s not all Cloud services deployed on one machine like Devstack does.

The Book

My favorite Openstack book to date is by far is the Openstack Cloud Computing Cookbook written by Kevin Jackson. This book is so great because gets straight to business giving you the commands & understanding needing to get Openstack up and running quickly unlike most other book that bore you with unnecessary details. The first time I installed Openstack Essex I used this book and it worked like a charm, however, that was on Ubuntu, which this book was written for. If you want to install Openstack on Redhat you can do so from scratch or you can use RDO.

RDO

RDO technically doesn’t stand for anything ! http://openstack.redhat.com/Frequently_Asked_Questions#What_does_RDO_stand_for.3F, but I don’t really buy that and have found some people expanding the acronym to the Redhat Distribution of Openstack which I think sounds incredibly fitting. RDO has a website dedicated to a community of people running Openstack on Redhat, CentOS and Fedora. Because Redhat is still the leader in production Linux deployments in the enterprises of the world, I chose to use it in my own lab environment & for the purposes of this tutorial. However, because I cannot afford the license cost of Redhat Enterprise Linux I am using the free community release of RHEL known as CentOS or the Community Enterprise Operating System. For the purposes of this tutorial we will be installing CentOS 6.5 with a minimal install on a bare metal system and then installing Packstack according to the instructions on the RDO website. Then we will go a step further.

Getting Started

A production Openstack deployment has a minimum of 3 nodes one for the Controller, Network, and Compute nodes and that would not cover a Highly Available deployment where you would need the capability to failover functions to standby nodes when there was an outage or maintenance needed. However, for the purposes of this article I will be showing you what is called an all-in-one install where the Controller, Network, and Compute functions all live on the same node and in future articles we will expand on this knowledge to build a production capable multi-node deployment of Openstack.

Before We Begin

You will need…

  1. A system that you have root access to, to install Openstack on. I am using an SGI Rackable with 16GB of RAM and 4.5TB of usable disk ( in a hardware RAID 10). Openstack does not require a lot of resources, but if you’re going to spawn a lot VM’s you need a lot of Cores (vCPU’s), Memory and Storage. See this link for more info:  http://docs.openstack.org/grizzly/openstack-compute/install/yum/content/compute-system-requirements.html
  2. That system should have dual NIC’s, although I think it is possible to use virtual interfaces for a lab environment if you have to.
  3. To configure the hostname of your controller in DNS mine is diamond.tuxlabs.com, but most people go with controller.yourdomain.com (if you don’t have or use DNS, just make sure you configure your hosts file /etc/hosts with the information)
  4. The system you use should have CentOS 6.5 installed with a Minimal Install using the defaults no extra’s.
  5. Your brain, keyboard, fingers and a fresh beer and possibly an ice chest with more beer depending on how far the fridge is.

Assumptions

  • Our home network uses 192.168.1.0/24 and has access to the internet.
  • It has DHCP enabled, but only for 192.168.1.150-199.
  • We are not using 10.0.0.0/24 for anything so we can use it for our private network in Openstack.

If these are true we should be able to follow my examples exactly, but if you want to change your networks you will have to substitute them as needed.

Setting Up Our Initial System

First become root or login as root, then…configure the following configuration files to match.

Resolve.conf

This assumes our gateway device @ 192.168.1.1 runs DNS (like the Linksys’s do). If 192.168.1.1 doesn’t run DNS do not add it to resolve.conf.

Sudoers

Make sure wheel is uncommented.

Setup Your Primary Network Interface

 Restart Networking

Next Add A Local User (Cause logging in as root is lame)

Set your password to something good, then add the user to the wheel group.

Installing Dependencies

Next we need to install some basic dependencies so things like scp, yum, wget and nslookup work.

Turn Off SELinux

Yes security is important, but unfortunately to install Openstack without a headache you need to turn off SELinux.

Don’t forget to reboot or the setting will not take effect.

Configuring Networking

There are many ways to configure Openstack networking and it can be quite complicated, so we are going to use what I consider the simplest method using Openvswitch.

Before we begin, make sure you can ping yahoo.com before we go mucking with the configs. We are going to change how your primary network interface is configured and then configure two additional interfaces. Do your best to match these configurations exact, making a mistake here might cause Openvswitch to barf on itself and that would not be fun to troubleshoot.

 What Is This Doing ?

The above configuration(s) is setting eth0 as an OVSPort that is bridged to the interface br-ex. eth1 is configured normally for our private 10 network, and br-ex is configured as a OVSBridge with our actual IP information from / for eth0.

After Restarting Networking ifconfig Output Should Look Like This

For more information on Openstack networking here are some references that I used.

  • https://www.youtube.com/watch?v=afImoFeuDnY
  • http://openstack.redhat.com/Neutron_with_existing_external_network
  • http://openstack.redhat.com/forum/discussion/780/confused-with-neutron-and-external-network-access/p1
  • http://openstack.redhat.com/Networking_in_too_much_detail
  • http://docs.openstack.org/grizzly/openstack-network/admin/content/

Update Packages

Finally since we are running a minimum install, let’s update all of our packages before continuing.

 Installing Openstack

Finally we are ready to install Openstack Using Packstack.

The first command adds the required RDO repo to yum install packstack. The second downloads the openstack-packstack package. And the third and final command is magic. This command installs packstack. We are telling it to install all openstack components to one machine the –provision-all-in-one-ovs-bridge=n tells packstack we are going to be using a single node (although I am still not entirely sure this flag is absolutely necessary) and the final flag tells packstack not to deploy the demo project, because if you do you have to end up deleting it before you can delete the network information and re-create it correctly.

Packstack will take about 10 minutes to run. It uses puppet to deploy Openstack and it’s required configurations. In my experience it works pretty well, however if you do need to re-install packstack there isn’t an automated uninstall script included. So someone created one and here it is in case you need it.

Don’t forget to chmod +x to that bad boy to make it executable so you can run it, when you are ready to uninstall.

Now then, getting back to our install. If Packstack was successful you should see something like this.

Check Out Our Keystonerc_admin File

Just take note of what environment variables it sets. To login to our GUI we need these credentials and to use any command line functionality we have to source this file in our shell.

 Configuring Openstack Networking

Because we did not deploy the demo project, the default network configuration in Openstack should not exist. If it does we can login to our GUI @ http://diamond.tuxlabs.com/dashboard by sourcing the credentials above in our keystonerc_admin and delete the network configuration under

  • admin –> routers
  • admin –> networks

Then you are ready to re-create your network. You could use the GUI to do this, which is pretty straight forward…but since GUI’s are dirty we are going to use the command line.

In order to use any openstack command line utilities you must first source the keystonerc_admin file so the required environment variables are set in your shell. Like so…

 Seeing (keystone_admin) in your prompt means you are ready to run commands and here you go

Copy & pasting this will spit out a lot of messages showing the result set of each command.

The important thing to realize is that your public network has to be configured with an  external router.  This effectively NAT’s the 10 & 192 networks to make internet access available to your VM’s. When this is configured correctly your Network Topology diagram under the dirty GUI should resemble this.

2014-08-04 12.03.09 am

The router_gateway always shows DOWN. No idea why, but someone else said it’s a bug 😉

Next under your project (the admin project) you have to configure your Security Groups under Compute—>Access & Security—>Security Groups. Once there click Manage Rules for the default security group. Delete what’s there. Add Ingress/Egress for ALL ICMP, ALL TCP, and ALL UDP accepting all other defaults on the form. This will open up your firewall completely.

Time To Restart Openstack

Finally we need to restart Openstack, validating that openvswitch-agent starts.

Note: You can use openstack-service –full-restart to restart all openstack services.

If openvswitch-agent does not start run back through the above steps and / or consult the references or email me tuxninja [at] tuxlabs.com. This can be a real pain to figure out, but I finally have it down pat.

If openvswitch-agent is active we are good to go. Next we have to create an SSH Key.

Creating Our Cloud’s SSH Key

Our Openstack Cloud has no authentication system for our virtual machines by default. Eventually you could configure an LDAP server, and configure your images or configure in Puppet the required pam configurations to use LDAP for authentication by your VM’s, but for now Openstack allows for a post configuration step after building VM’s where it will add your SSH key. It does this by using the metadata service which cloud images will look for, if they can reach the metadata service @ http://169.254.169.254 then they can copy down the public key and install it on the VM for you allowing you to login to that VM using the VM’s operating system default account (i.e. ubuntu, fedora). Now before I continue there are alternatives such as using guestfish to modify an image’s configuration or using a post install cloud-init configuration to specific a password to an account, but the cleanest and simplest way is to use the metadata service after generating & importing the public key into openstack and assigning it to our VM guest.

To create the key…

I don’t enter a password to use password-less SSH, cause again this is a lab environment and typing passwords sucks.

Launching A VM

Next you want to launch a VM that we can use to import our SSH key into. Login to the dashboard @ http://diamond.tuxlabs.com/dashboard again using the keystonerc_admin credentials provided. Once inside navigate to Project—>Compute—>Instances and click the Launch Instance button in the upper right corner. You will be presented with a form. Here are the screenshots to guide you through.

Fill out the details tab…

2014-08-04 12.19.19 am

Click the Access & Security Tab

2014-08-04 12.19.44 am

Then Click The +

Now in order to fill this out we have to copy and paste the contents (bold part) of the public key we created.

2014-08-04 12.20.10 am

Then Click Networking and add the private network…

2014-08-04 12.20.43 am

Then click launch….and you should see something like this…

2014-08-04 12.21.22 am

See the status spawning ? When the virtual machine is done being built you will see status changed to ‘Running’, you can watch the machine boot, by clicking the name of the instance and then going to the Log or Console tab.

2014-08-04 12.28.05 am

This is great our VM is running and it has an IP address configured ! But that IP address is only used for Openstack communication, so we still need to associate a floating IP to this system so we can SSH to it. Click on the More dropdown and select Associate Floating IP.

2014-08-04 12.29.52 am

Then click the + …

2014-08-04 12.30.08 am

Click Allocate IP…

2014-08-04 12.30.25 am

And then click associate…

2014-08-04 12.30.40 am

Now you should see your VM has an internal IP (a 10 dot address) and an External IP on the 192.168.1.0/24 network. Now we can SSH to our VM.

We didn’t mention it earlier, but Openstack comes with only one Linux image by default, it’s called CirrOS and it’s just a tiny-minimal cloud image for testing. The login to this operating system is cirros / cubswin:) … which is visible from the console log of the machine once fully booted. Now you could SSH into the machine on 192.168.1.54 using that login and password like so…

 But that is lame ! We imported our SSH key remember ? So how do we use that ? Like this..

 Wow, that is cool.

Now what ? Let’s add some more images.

Installing More Operating System Images

Ok, Openstack is installed, Networking is working, Metadata service and our keys are working, we are happy campers, but to make this Cloud useful we are going to need some real Linux images. There is two ways to install images using glance (the openstack image service).

In the first method we download the image using wget & then we run the proper glance command. Here is this approach in action.

Don’t forget to source the rc file if you have not already

 Alternatively, we could do the same thing in one command !

Remember whenever you run openstack commands you are going the status or result set back from the command.

Once you have installed the Fedora and Ubuntu images you should have a real cloud on your hands. Now you can do things like expand your setup for multi-node or get LBaaS working for load balancing requests to your web servers for example. Or if you want to be amazeballs you could install things like Puppet inside of Docker, or Cloud Foundry and build your PaaS (Platform As A Service)!

Final Comments

Openstack can be tricky even for an experienced Sys Admin. While learning Openstack I found it difficult to find tutorials on exactly the setup I was looking for mainly in terms of how the network was being configured, and this often made me second guess myself when I would run into an issue as I learned Openstack. I wanted to write this article to give back and help to educate my brothers and students of life long learning as you embark on your Openstack adventure, Godspeed. Here are some commands of note to help you along your way.

How to launch a VM from command line

 How to run command on a network namespace (VM Instance)

How to restart only neutron services

Or a simpler method would be…

How to add an account / change a password using cloud-init

Stick this under Post Creation Customized Script to

How to edit the password file of an image

How to see the status of openstack services

How to check your log files for more clues

you can do this for every openstack service, but below I am just showing the two most popular to troubleshoot for getting your cloud up and going, nova and neutron aka controller, compute and networking logs.

Want to change which partition Openstack uses on local disk (ephemeral storage) to deploy VM’s ?

If the majority of your ephemeral (local) disk is under a different partition other than /var… For example, mine was under /home, then you need to change your state path and restart openstack services. Don’t forget to copy any existing files in /var/lib/nova/ to the new location.

Checking your openvswitch configuration

The Final Gem

The Most Popular Issue I ran Into When Installing Openstack

Looks like this in the console..

novalidhost

This is the generic error for I didn’t have resources to create the VM, which can happen for a multitude of reasons…Like for example, the Openvswitch-agent not running.

But why is Openvswitch Agent Failing ? 

Additionally, I have seen different errors presented referencing vif.py 

The fix for this could be multiple different things. What you should do first is verify your network interfaces are configured 100% correctly. Compare your ovs-vsctl output to mine above… Look different ? Shorter, stuff is missing ? Ok this error usually means Openstack is having difficulty inserting port’s into Openvswitch for whatever reason. What I found is the most common error is that you did not restart networking, before restarting openstack networking after making a configuration change to the interfaces. So to resolve this try…

Now when we run openstack-status the openvswitch-agent should be running !

 And Healthy ! Awesome. Last, but not least you can find all my blessed configs for this article as well as a script I wrote to snapshot those configs on my github. Thanks for learning with me.

 

 

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.