Tuxninja aka Jason Riedel has worked as a Systems & Network Administrator and Code Hacker since 1999. Since 2005 he has worked for PayPal with a focus on Operations Architecture. He is also CEO of Tuxlabs LLC where he dedicates his time to the experimentation and close study of new technologies and programming languages.

How To: Add A Compute Node To Openstack Icehouse Using Packstack



This article is a continuation on the previous article I wrote on how to do a single node all-in-one (AIO) Openstack Icehouse install using Redhat’s packstack. A working Openstack AIO installation using packstack is required for this article. If you do not already have a functioning AIO install of Openstack please refer to the previous article before continuing on to this articles steps.

Preparing Our Compute Node

Much like in our previous article we first need to go through and setup our system and network properly to work with Openstack. I started with a minimal CentOS 6.5 install, and then configured the following

  1. resolv.conf
  2. sudoers
  3. my network interfaces eth0(192) and eth1 (10)
    1. Hostname: ruby.tuxlabs.com ( I also setup DNS for this )
    2. EXT IP:
    3. INT IP:
  4. A local user + added him to wheel for sudo
  5. I installed these handy dependencies
    1. yum install y opensshclients
    2. yum install y yumutils
    3. yum install y wget
    4. yum install y bindutils
  6. And I disabled SELinux
    1. Don’t forget to reboot after

To see how I setup the above pre-requisites see the “Setting Up Our Initial System” section on the previous controller install here : http://tuxlabs.com/?p=82

Adding Our Compute Node Using PackStack

For starters we need to follow the steps in this link  https://openstack.redhat.com/Adding_a_compute_node

I am including the link for reference, but you don’t have to click it as I will be listing the steps below.

On your controller node ( diamond.tuxlabs.com )

First, locate your answers file from your previous packstack all-in-one install.

[root@diamond tuxninja]# ls *answers*
[root@diamond tuxninja]#

 Edit the answers file

Change lo to eth1 (assuming that is your private 10. interface) for both CONFIG_NOVA_COMPUTE_PRIVIF & CONFIG_NOVA_NETWORK_PRIVIF

[root@diamond tuxninja]# egrep 'CONFIG_NOVA_COMPUTE_PRIVIF|CONFIG_NOVA_NETWORK_PRIVIF' packstack-answers-20140802-125113.txt
[root@diamond tuxninja]#

Change CONFIG_COMPUTE_HOSTS to the ip address of the compute node you want to add. In our case ‘’. Additionally, validate the ip address for CONFIG_NETWORK_HOSTS is your controller’s ip since you do not run a separate network node.

[root@diamond tuxninja]# egrep 'CONFIG_COMPUTE_HOSTS|CONFIG_NETWORK_HOSTS' packstack-answers-20140802-125113.txt
[root@diamond tuxninja]#

That’s it. Now run packstack again on the controller

[tuxninja@diamond yum.repos.d]$ sudo packstack --answer-file=packstack-answers-20140802-125113.txt

When that completes, ssh into or switch terminals over to your compute node you just added.

On the compute node ( ruby.tuxlabs.com )

Validate that the relevant openstack compute services are running

[root@ruby ~]# openstack-status
== Nova services ==
openstack-nova-api:                     dead      (disabled on boot)
openstack-nova-compute:                 active
openstack-nova-network:                 dead      (disabled on boot)
openstack-nova-scheduler:               dead      (disabled on boot)
== neutron services ==
neutron-server:                         inactive  (disabled on boot)
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              active
== Ceilometer services ==
openstack-ceilometer-api:               dead      (disabled on boot)
openstack-ceilometer-central:           dead      (disabled on boot)
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         dead      (disabled on boot)
== Support services ==
libvirtd:                               active
openvswitch:                            active
messagebus:                             active
Warning novarc not sourced
[root@ruby ~]#

 Back on the controller ( diamond.tuxlabs.com )

We should now be able to validate that ruby.tuxlabs.com has been added as a compute node hypervisor.

[tuxninja@diamond ~]$ sudo -s
[root@diamond tuxninja]# source keystonerc_admin
[root@diamond tuxninja(keystone_admin)]# nova hypervisor-list
| ID | Hypervisor hostname |
| 1  | diamond.tuxlabs.com |
| 2  | ruby.tuxlabs.com    |
[root@diamond tuxninja(keystone_admin)]# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:34
nova-conductor   diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:35
nova-scheduler   diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:27
nova-compute     diamond.tuxlabs.com                  nova             enabled    :-)   2014-10-12 20:48:32
nova-cert        diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:31
nova-compute     ruby.tuxlabs.com                     nova             enabled    :-)   2014-10-12 20:48:35
[root@diamond tuxninja(keystone_admin)]#

Additionally, you can verify it in the Openstack Dashboard


Next we are going to try to boot an instance using the new ruby.tuxlabs.com hypervisor. To do this we will need a few pieces of information. First let’s get our OS images list.

[root@diamond tuxninja(keystone_admin)]# glance image-list
| ID                                   | Name                | Disk Format | Container Format | Size      | Status |
| 0b3f2474-73cc-4df2-ad0e-fdb7a7f7c8a1 | cirros              | qcow2       | bare             | 13147648  | active |
| 737a0060-6e80-415c-b66b-a20893d9888b | Fedora 6.4          | qcow2       | bare             | 210829312 | active |
| 952ac512-19da-47a7-81a4-cfede18c7f45 | ubuntu-server-12.04 | qcow2       | bare             | 260964864 | active |
[root@diamond tuxninja(keystone_admin)]#

Great, now we need the ID of our private network

[root@diamond tuxninja(keystone_admin)]# neutron net-show private
| Field                     | Value                                |
| admin_state_up            | True                                 |
| id                        | d1a89c10-0ae2-43f0-8cf2-f02c20e19618 |
| name                      | private                              |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 10                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | b8760f9b-3c0a-47c7-a5af-9cb533242f5b |
| tenant_id                 | 7bdf35c08112447b8d2d78cdbbbcfa09     |
[root@diamond tuxninja(keystone_admin)]#

Ok now we are ready to proceed with the nova boot command.

[root@diamond tuxninja(keystone_admin)]#  nova boot --flavor m1.small --image 'ubuntu-server-12.04' --key-name cloud --nic net-id=d1a89c10-0ae2-43f0-8cf2-f02c20e19618 --hint force_hosts=ruby.tuxlabs.com test
| Property                             | Value                                                      |
| OS-DCF:diskConfig                    | MANUAL                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                       |
| OS-EXT-SRV-ATTR:host                 | -                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000019                                          |
| OS-EXT-STS:power_state               | 0                                                          |
| OS-EXT-STS:task_state                | scheduling                                                 |
| OS-EXT-STS:vm_state                  | building                                                   |
| OS-SRV-USG:launched_at               | -                                                          |
| OS-SRV-USG:terminated_at             | -                                                          |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| adminPass                            | XHUumC5YbE3J                                               |
| config_drive                         |                                                            |
| created                              | 2014-10-12T20:59:47Z                                       |
| flavor                               | m1.small (2)                                               |
| hostId                               |                                                            |
| id                                   | f7b9e8bb-df45-4b94-a896-5600f47c269b                       |
| image                                | ubuntu-server-12.04 (952ac512-19da-47a7-81a4-cfede18c7f45) |
| key_name                             | cloud                                                      |
| metadata                             | {}                                                         |
| name                                 | test                                                       |
| os-extended-volumes:volumes_attached | []                                                         |
| progress                             | 0                                                          |
| security_groups                      | default                                                    |
| status                               | BUILD                                                      |
| tenant_id                            | 7bdf35c08112447b8d2d78cdbbbcfa09                           |
| updated                              | 2014-10-12T20:59:47Z                                       |
| user_id                              | 6bb8fcf3ce9446838e50a6b98fbb5afe                           |
[root@diamond tuxninja(keystone_admin)]#

Fantastic. That command should look familiar from our previous tutorial it is the standard command for launching new VM instances using the command line, with one exception ‘–hint force_hosts=ruby.tuxlabs.com’ this part of the command line forces the scheduler to use ruby.tuxlabs.com as it’s hypervisor.

Once the VM is building we can validate that it is on the right hypervisor like so.

[root@diamond tuxninja(keystone_admin)]# nova hypervisor-servers ruby.tuxlabs.com
| ID                                   | Name              | Hypervisor ID | Hypervisor Hostname |
| f7b9e8bb-df45-4b94-a896-5600f47c269b | instance-00000019 | 2             | ruby.tuxlabs.com    |
[root@diamond tuxninja(keystone_admin)]# nova hypervisor-servers diamond.tuxlabs.com
| ID                                   | Name              | Hypervisor ID | Hypervisor Hostname |
| a4c67465-d7ef-42b6-9c2a-439f3b13e841 | instance-00000017 | 1             | diamond.tuxlabs.com |
| 0c34028d-dfb6-4fdf-b9f7-daade66f2107 | instance-00000018 | 1             | diamond.tuxlabs.com |
[root@diamond tuxninja(keystone_admin)]#

You can see from the output above I have 2 VM’s on my existing controller ‘diamond.tuxlabs.com’ and the newly created instance is on ‘ruby.tuxlabs.com’ as instructed, awesome.

Now that you are sure you setup your compute node correctly, and can boot a VM on a specific hypervisor via command line, you might be wondering how this works using the GUI. The answer is a little differently 🙂

The Openstack Nova Scheduler

The Nova Scheduler in Openstack is responsible for determining, which compute node a VM should be created on. If you are familiar with VMware this is like DRS, except it only happens on initial creation, there is no rebalancing that happens as resources are consumed overtime. Using the Openstack Dashboard GUI I am unable to tell nova to boot off a specific hypervisor, to do that I have to use the command line above (if someone knows of a way to do this using the GUI let me know, I have a feeling if it is not added already, they will add the ability to send a hint to nova from the GUI in a later version). In theory you can trust the nova-scheduler service to automatically balance the usage of compute resources (CPU, Memory, Disk etc) based on it’s default configuration. However, if you want to ensure that certain VM’s live on certain hypervisors you will want to use the command line above. For more information on how the scheduler works see : http://cloudarchitectmusings.com/2013/06/26/openstack-for-vmware-admins-nova-compute-with-vsphere-part-2/

The End

That is all for now, hopefully this tutorial was helpful and accurately assisted you in expanding your Openstack compute resources & knowledge of Openstack. Until next time !


How To: Add A Compute Node To Openstack Icehouse Using Packstack Read More »

How To: Install The Foreman on CentOS 6.5

2014-08-16 02.03.35 pm


In this tutorial I will be demonstrating how to install The Foreman on the Linux CentOS 6.5 operating system. If you are not familiar with CentOS it is the Community version of RedHat Linux Enterprise, hence the name C(ommunity)ENT(erprise) O(perating)S(ystem). For more information on CentOS see : https://www.centos.org/. I chose CentOS over Ubuntu or other Linux flavors for this set of tutorials because of it’s ubiquity in enterprise environments and because of RedHat’s leadership in the Openstack ecosystem. We will discuss Openstack (www.openstack.org) in later tutorials, for now The Foreman represents a great first step to setting up a world class Cloud environment.

Why The Foreman

“Foreman is an open source project that helps system administrators manage servers throughout their lifecycle, from provisioning and configuration to orchestration and monitoring. Using Puppet or Chef and Foreman’s smart proxy architecture, you can easily automate repetitive tasks, quickly deploy applications, and proactively manage change, both on-premise with VMs and bare-metal or in the cloud.” From their website (www.theforeman.org).

In other words it builds on well known configuration management systems (puppet in our case) and fully automates ‘things’ to provision and manage bare metal (physical) systems and Cloud Virtual Machines. This is really helpful when you have an Openstack Cloud and need to add compute nodes quickly.

Starting The Install

My 3 machines in my lab, diamond, ruby and emerald are all running CentOS 6.5. To make sure I cover all the dependencies needed I did a minimal install on these machines. So you will see me install some convenience dependencies as well, which I will point out. The following steps are based loosely on the http://theforeman.org/manuals/1.5/quickstart_guide.html

You will want to install The Foreman on your ‘master’ or machine you intend to the be the ‘controller’ of your Cloud. In my case, diamond.tuxlabs.com

Installing Basic System Dependencies

yum install -y openssh-clients
yum install -y yum-utils
yum install scl-utils scl-utils-build
yum install -y wget
yum install -y bind-utils

scl-utils is used for managing the rails console, the rest should be pretty self explanatory.

Installing Foreman Dependencies

wget https://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
rpm -ivh epel-release-6-8.noarch.rpm

Add Yum Repos

yum-config-manager --enable rhel-6-server-optional-rpms rhel-server-rhscl-6-rpms

Check to make sure the Repo was added using

yum repolist

Installing The Foreman

yum -y install http://yum.theforeman.org/releases/1.5/el6/x86_64/foreman-release.rpm
yum -y install foreman-installer

 After grabbing the packages…finally…run the installer


This part is pretty cool because foreman’s automated installer seems to work quite well, it takes a few minutes, but when completed you should see something like this.

Preparing installation Done                                             
  * Foreman is running at https://diamond.tuxlabs.com
      Default credentials are 'admin:changeme'
  * Foreman Proxy is running at https://diamond.tuxlabs.com:8443
  * Puppetmaster is running at port 8140
  The full log is at /var/log/foreman-installer/foreman-installer.log
[root@diamond tuxninja]#


If you have iptables running, you will have to flush the rules or open the need ports.

iptables --flush


At this point Puppet should be installed on diamond (and your machine). You should test it by running the agents twice, yes twice 🙂 The first time clears some warnings.

puppet agent --test
puppet agent --test

Follow the instructions for installing the NTP module in Puppet


I could have re-wrote this or cropped it, but it was written very simply. Definitely do this though so you get a feel for how to add puppet modules and manage them in The Foreman.

The Clients

Now before we get into The Foreman Dashboard, let’s setup our clients so we have more interesting stuff to look at then this one node. ssh to each of your machines (ruby,emerald in my case), and sudo or become root and run…

rpm  -ivh http://yum.puppetlabs.com/el/6.4/products/x86_64/puppetlabs-release-6-7.noarch.rpm
yum install -y puppet

 Then start puppet

/etc/init.d/puppet start

 Sign The Certs

Puppet uses SSL certificates to authenticate clients that it manages. By default you must permit each client by manually signing the certificate before the client is authenticated with the puppet master. Back on the master server (diamond) run the following modified for the correct FQDN’s (full hostnames).

puppet cert --sign ruby.tuxlabs.com
puppet cert --sign emerald.tuxlabs.com

Auto Starting

Make sure Puppet starts on boot, do this on all machines

chkconfig puppet on

The Dashboard

Finally :-), If you have set everything up correctly you should be able to reach your console @ https://yourserversip.com or if you have DNS at the https://yourserversname.com (aka the FQDN Fully Qualified Domain Name).  Note the console uses SSL so you want to use HTTPS (on port 443 by default) not HTTP (which is port 80 by default), so just remember the HTTPS part and you should see a login screen, like this.


The first time Login is : admin and the Password is : changeme … make sure you do like the password says ! Also feel free to change you display name in your profile.

The End

Now that you have The Foreman installed. The next logical step is to install Openstack. I will cover that next time folks, I hope this helps out !

How To: Install The Foreman on CentOS 6.5 Read More »

Python & Fun with Checkio.org

My Brief Coding History

13 years ago, I fell in love with Perl. I liked Perl a lot better than my previous flings with HTML, Shell (Bash), PHP, ASP, Javascript, Sed, Awk and a few others, mainly because Perl was always willing to do what I wanted, when I wanted, making my life as a Sys Admin a lot easier and never asked much in return just to be loved. I loved Perl for years and years and in a way I will never stop loving Perl, after all Perl was my first love. But too often our first loves die hard no matter how hard we try to will their existence forward…

“I never thought I could love a language more than Perl, until I met Python”  (TuxNinja, 2012)

Enter Python

Old loves die hard.. I haven’t used Perl for about ~18 months, whatever the exact amount of time is I am not sure, but it matches the exact time at which I started learning Python.  I decided to start learning Python because it’s popularity seemed to be growing exponentially (and still does), once I learned basic syntax by reading and watching a couple of videos, I was up and running and it wasn’t long before I was solving real business critical problems with it.

[stextbox id=”grey” caption=”Best Way To Learn A New Language”]I have learned a lot of languages. By far the best way to learn a new language is to solve a real problem (preferably at your job) with it. The reason being is that inevitably when you solve the problem you will need to maintain the code (i.e. add features and fix bugs). This ‘recall’ you experience every time you re-visit and re-factor your code will burn into your brain valuable learning’s better than any amount of writing and/or reading alone. Suffices to say that troubleshooting is the best way to learn.[/stextbox]

Two Reasons

There were two major reasons once I started learning Python I decided it was my new #1.

  1. Python is the most readable language I have ever seen. It reads like pseudo code or plain english. If that isn’t important to you, go try to read someones competition winning obfuscated code ./facepalm. For the 13 years I coded Perl every time I took over a Perl script from someone else I re-wrote it. That is because Perl allows for coders to follow many different styles such as a C or Shell style of coding. It’s flexibility is it’s greatest enemy in my opinion, there are 1000 ways to do something in Perl. In Python there are 10’s of ways, but there is usually only 1 pythonic way. I prefer a language that has a clear best way to do something and that encourages readability. Of course there are still some assholes out there who prefer lambda() and map() over list comprehension and to them I say… read a book (err website)
  2. Community support. I forever loved Perl. Two of the reasons was the incredible support I found on www.perlmonks.com and because of CPAN and the plethra of modules that existed. Well Python wins again. It’s seems most former monks now spend their time correcting peoples Python on StackOverflow.com and PyPi > CPAN, sorry but it’s true !

All that being said, I encourage you to learn Python, Ruby, and Node.js… these 3 seem to be the front runners of that the next generation is learning and therefore will continue to be abound now and in the future. But since this is a Python article and the language I know best, learn Python first !


Finally, the purpose of this article! About 3 months into learning Python a good friend of mine who is a programmer for a living told me about CheckIO.org, which is a website, that provides a fun & free way to learn Python in the form of a game that reminds me of Wii’s Super Mario. Now when I initially signed up ~15 months ago I never actually ended up playing the game. Until a couple of days ago when I got an email from them asking how I liked CheckIO. Well that was all the reminder I needed, looking to continue in life long learning and specifically my Python knowledge I started playing the game. It’s been 48 hours and I am pretty hooked ! The benefits to learning using CheckIO are unmatched by any other learning mechanisms I have used to date here’s why.

How It Works

CheckIO starts you at your ‘home’ and presents you with challenges that it gives you points and badges for completing. Once you complete enough you move on to the next challenge and eventually unlock other adventures and your next set of challenges. Yea, yea ok that’s cool so what…well when you solve your challenge, you have the option of publishing your code and reviewing / voting on the best solution of all time for that problem. This is an amazing thing. It combines not only exercising to solve a problem yourself, but ultimately showing you the most clean, efficient and pythonic way (caveat: not always but mostly the social voting keeps this true). This is an incredible about of knowledge in one place, and the challenges you are solving are universally applicable to some of the most common problems with data structures, sorting and counting that you will run into. Admittedly, I am a level 4 currently and only about 60% of the way through and have already unlocked other adventures, but I was having so much fun and learning so much that I decided I’m going to complete 100% of an adventure before moving on.

CheckIO Teaches You To Be An Efficient Programmer

Feel free to read that again. Yes it does. My whole programming life I have been focused on function…function, function, function. Making my code functional was always #1 because after all you are solving a problem and making your code functional has to be the right first step anyway, you can’t work on making an application efficient if you haven’t figured out a way to solve the problem programmaticly. So the only time I went back and re-wrote an program to make it more efficient was when it was heavily used so incredibly inefficient that it demanded to be FIXED!

Let There Be Light

What CheckIO does is encourages you to compete with other programmers solving the same challenges in the game format, keeping it fun. So finally 6 challenges in, I started to get it… instead of just solving the challenge, why not try to solve the challenge in the most efficient & cleanest way possible and then publish my code for voting with the community. I did just that and the difference in my code was a difference of 47 lines.  The example I am showing below is a basic one, but the story is what’s crucial. If you solve the CheckIO problems and challenge yourself to write the best possible solution each time and then compare against your peers you will ultimately be a far better programmer then most people. Not to mention you will have a nice repository for re-usable code for common problems you will encounter as a programmer.

Ok enough…it’s show and tell time.

The Problem

Full Problem Description: http://www.checkio.org/mission/roman-numerals/
Short Version: For this task, you should return a roman numeral using the specified integer value ranging from 1 to 3999.

Input: An integer ranging from 1 to 3999.
Output: A string in the form of a Roman numeral.

checkio(6) == 'VI'
checkio(76) == 'LXXVI'

Functional Solution

import random

numerals = { 1 : 'I', 4 : 'IV', 5 : 'V', 9 : 'IX', 10 : 'X', 40 : 'XL', 50 : 'L', 90 : 'XC', 100 : 'C', 400 : 'CD', 500 : 'D', 900 : 'CM', 1000 : 'M' }
number = random.randrange(4000)
string = ''

if (number / 1000):
    thousands = number / 1000 # thousands
    string += numerals[1000]*thousands
    number = number - (thousands*1000)
if (number / 900):
    ninehundreds = number / 900 # 900's
    string += numerals[900]*ninehundreds
    number = number - (ninehundreds*900)
if (number / 500):
    fivehundreds = number / 500 # 500's
    string += numerals[500]*fivehundreds
    number = number - (fivehundreds*500)
if (number / 400):
    fourhundreds = number / 400 # 400's
    string += numerals[400]*fourhundreds
    number = number - (fourhundreds*400)
if (number / 100):
    hundreds = number / 100 # 100's
    string += numerals[100]*hundreds
    number = number - (hundreds*100)
if (number / 90):
    nineties = number / 90 # 90's
    string += numerals[90]*nineties
    number = number - (nineties*90)
if (number / 50):
    fifties = number / 50 # 50's
    string += numerals[50]*fifties
    number = number - (fifties*50)
if (number / 40):
    forties = number / 40 # 40's
    string += numerals[40]*forties
    number = number - (forties*40)
if (number / 10):
    tens =  number / 10 # 10's
    string += numerals[10]*tens
    number = number - (tens*10)
if (number / 9):
    nines =  number / 9 # 9's
    string += numerals[9]*nines
    number = number - (nines*9)
if (number / 5):
    fives = number / 5 # 5's
    string += numerals[5]*fives
    number = number - (fives*5)
if (number / 4):
    fours = number / 4 # 4's
    string += numerals[4]*fours
    number = number - (fours*4)
if (number / 1):
    ones = number / 1 # 1's
    string += numerals[1]*ones
    number = number - (ones*1)

print string

My Think Twice Solution

It’s amazing how quickly you can improve code once you understand how it needs to function. This improvement took me a total of 2 minutes to finish.

import random

numerals = { 1 : 'I', 4 : 'IV', 5 : 'V', 9 : 'IX', 10 : 'X', 40 : 'XL', 50 : 'L', 90 : 'XC', 100 : 'C', 400 : 'CD', 500 : 'D', 900 : 'CM', 1000 : 'M' }
number = random.randrange(4000)
string = ''

for roman in sorted(numerals, reverse=True):
    if (number / roman):
        multiplier = number / roman
        string += numerals[roman]*multiplier
        number = number - (multiplier*roman)

print string

Published Solution

The above solutions were what I used to test and run before submitting, when you actually submit it has to be as a CheckIO function. The real code submitted can be found below here.




Python & Fun with Checkio.org Read More »