Python

Upgrading to Python 2.7 on CentOS 6.5

1 Comment / How To's, Programming, Python /

Hey Folks,

The systems running Tuxlabs are currently running CentOS 6.5 to emulate a production RHEL like setup for an Openstack Cloud. Running an operating system this old has it’s drawbacks such as dependencies. I was recently installing a well know Python framework and ran into compatibility issues. The framework required Python 2.7 and CentOS 6.5 comes with 2.6. The below is a step by step procedure for how to upgrade to Python 2.7 on CentOS 6.5 if  you ever should need it. However, as a reminder run a newer OS when possible and for god sakes if you don’t need Redhat support, run Ubuntu.

Step one, we verify we are indeed running Python 2.6

[tuxninja@diamond ~]$ python --version
Python 2.6.6
[tuxninja@diamond ~]$

Ok then, let’s upgrade Python to 2.7. First let’s update all of our system applications, just in case for version dependencies and it’s good for security etc.

[tuxninja@diamond ~]$ yum -y update

Next, we have to install Develop Tools, it is a required dependency to install Python.

[tuxninja@diamond ~]$ yum groupinstall "Development tools"

Additionally, we will need these…

[tuxninja@diamond ~]$ sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

Now, let’s install Python 2.7

[tuxninja@diamond ~]$ cd /opt
[tuxninja@diamond opt]$ sudo wget --no-check-certificate https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
[tuxninja@diamond opt]$ sudo tar xf Python-2.7.6.tar.xz 
[tuxninja@diamond opt]$ cd Python-2.7.6
[tuxninja@diamond Python-2.7.6]$ sudo ./configure --prefix=/usr/local
[tuxninja@diamond Python-2.7.6]$ sudo make && sudo make altinstall

It is important to use ‘altinstall’ otherwise you will end up with two different versions of Python on your filesystem, both named ‘python’.

You can verify the install like so

[tuxninja@diamond Python-2.7.6]$ ls -la /usr/local/bin/python2.7*
-rwxr-xr-x 1 root root 6214493 Apr 30 15:14 /usr/local/bin/python2.7
-rwxr-xr-x 1 root root    1674 Apr 30 15:14 /usr/local/bin/python2.7-config
[tuxninja@diamond Python-2.7.6]$ /usr/local/bin/python2.7 --version
Python 2.7.6
[tuxninja@diamond Python-2.7.6]$

That’s it ! Enjoy.

Upgrading to Python 2.7 on CentOS 6.5 Read More »

Runner: Multi-threaded SSH with Sudo support using Python & Paramiko

Leave a Comment / Programming, Python, Systems Administration /

Example of Runner

$ runner -r web1 -c "whoami" -s
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'web1'
RUNNER: 1 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - logs/runner.log.2015-01-17.03:10:00
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 5 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INFO]: SUDO IS ON
RUNNER [INPUT]: Please Enter Site Pass: 
web1.tuxlabs.com: 
web1.tuxlabs.com: root
web1.tuxlabs.com: [tuxninja@web1 ~]$ 

RUNNER [RESULT]: Successfully logged into 1/1 hosts and ran your commands in 0:00:08 second(s)
RUNNER [RESULT]: There were 0 login failures.

Why Runner ?

I have been working as a Systems & Network Administrator since 1999. In that time I have repeatedly had the need for rapidly executing commands across thousands of servers. There are many applications out there that solve this problem in various ways…to name a few…pdsh, Ansible, Salt, Chef, Puppet (mcollective),  even Cfengine and more. Some require agents running on the machines, some use SSH, but require keys…or learning curves. Alternatively, you can write your own code to solve this problem, which is what I did mostly for fun. I don’t recommend re-inventing the wheel if you need this for your job, just use what is already out there, or download runner and hack it to your hearts content for your purposes.

Fabric vs. Paramiko

Because I use Python for most of my work code these days, I decided to write my multi-threaded SSH command runner in Python this way I can use Runner for parallel SSH transport & easily bolt on my other Python scripts for additional functionality. Python has fantastic support for SSH via two libraries Fabric & Paramiko. Fabric is built on top of Paramiko. Fabric provides a simpler interface than Paramiko does for doing just about anything you can think of. Create a fabfile run it, and wolla instant results from commands ran via SSH. Fabric is really great for running & re-running a set of commands to automate an install or reporting for example. All that being said I still chose to use Paramiko over Fabric for three reasons.

  1. I don’t like abstraction. Fabric hides the ugly-ness of Paramiko, which I prefer to understand better.
  2. Writing this using Paramiko lent itself better to a command line utility used for adhoc commands than Fabric did.
  3. I wasn’t sure if Fabric’s abstraction would limit me later based on needing custom functionality. So for Runner I chose Paramiko, but to be clear, 9 times out of 10 I think I would choose Fabric.

Bastions

A bastion or jump box is a machine that is used as the gatekeeper of access to the rest of the machines in your network. In secure environments where your Corp network is separate from your Production network, you will have to SSH into a bastion, which usually has some form of 2-factor authentication (at least it should !) and then from there you may SSH into other hosts. A bastion can throw a real wrench in trying to manage thousands of machines in seconds, because you would have to authenticate to the bastion 1000 times ! The way around this, is by setting up your SSH config to proxy commands.

ProxyCommand & Sconnect

Sconnect (or connect.c) is a binary that is most commonly used as the proxy command for SSH. You can download / read more about sconnect here : https://bitbucket.org/gotoh/connect/wiki/Home and it will also tell you how to setup your SSH config. Using a ProxyCommand with Runner is required, you can however use any ProxyCommand you would like. Really quickly here is what you basically need to do.

  1. Download / Compile connect.c
  2. Copy it to /usr/local/bin/sconnect and set executable permissions
  3. In your SSH Config (.ssh/config) add…
    1. Host <ssh-config-profile-name>
      User tuxninja
      ForwardAgent yes
      HostName <bastion_name>
      DynamicForward 8081 (any uncommon port is fine)
    2. Host *.tuxlabs.com
      User tuxninja
      ProxyCommand /usr/local/bin/sconnect -4 -w 4 -S localhost:8081 %h.tuxlabs.com %p

That is basically it. Then you should start a screen session so you can background the SSH session, since you will leave this open for other SSH sessions to proxy through so you don’t have to go through 2-factor authentication more than once. So something like…

screen -S sshsession
ssh <ssh-config-profile-name>

After you authenticate, detach yourself from the screen using CTRL A then D. Now you can ssh to anything @ domain name in my case tuxlab.com and it will forward through the bastion. At this point you still have to authenticate using a username / password, which is fine. Runner deals with this.

Hosts

Runner requires a hosts file to run. By default it is configured to look in hosts/hosts-all for a list of all hosts. I use a script called ‘update-runner-hosts.pl’, which is included in my github to gather hosts from a URL and update the required hosts file. Once you have populated hosts/hosts-all with the FQDN for your hosts, you are ready to use Runner.

Note: You can use ‘-f’ to provide a custom location for your hosts file.

Great Flags / Features

So some of the really great features of Runner are threading (-t), sudo (-s), list only mode (-l) and the regular expression (-r). -r is for pattern matching your hosts lists, which is incredibly handy and absolutely required in an environment with hundreds to thousands of hosts and you only want to select hosts with -r ‘web’ in them.

(-1) one host per pool mode is a great feature, however it is dependent on understanding your environments hostname pattern so you will have to modify the regular expression in the code to make sure it works for you. It is currently setup to identify hostnames in pools when the naming convention is something like apache1234.tuxlabs.com.

Ok I could go on and on about runner, but it’s better to just share the code at this point and let you go! Note the statically defined proxy_command in the code, you may need to change this if you didn’t use sconnect or the same port.

Note: by default runner uses the user you are logged in as to SSH, you can prompt input for a different user with ‘-u’.

All code and accessories are available for download on github : https://github.com/jasonriedel/tuxlabs/tree/master/runner

Email tuxninja@tuxlabs.com with any question ! Happy SSH’ing admins!

Note: In various versions of this code I had a ‘-h’ allowing you to pass a CSV list of hosts, somehow I let that drop out of this version, sorry ! Feel free to re-add it !

The Runner Code

#!/usr/bin/env python
#Author: Jason Riedel

import paramiko
import getpass
import Queue
import threading
import argparse
import os.path
import time
import logging
import re
import datetime

## SETUP AVAILABLE ARGUMENTS ##
parser = argparse.ArgumentParser()
parser.add_argument('-f', action="store", dest="file_path", required=False, help="Specify your own path to a hosts file")
parser.add_argument('-l', action="store_true", dest="list_only", required=False, help="List all known hosts")
parser.add_argument('-q', action="store_true", dest="quiet_mode", required=False, help="Quiet mode: turns off RUNNER INFO messages.")
parser.add_argument('-qq', action="store_true", dest="super_quiet_mode", required=False, help="Super Quiet mode: turns off ALL RUNNER messages except [INPUT].")
parser.add_argument('-r', action="store", dest="host_match", required=False, help="Select Hosts matching supplied pattern")
parser.add_argument('-c', action="store", dest="command_string", required=False, help="Command to run")
parser.add_argument('-s', action="store_true", dest="sudo", required=False, help="Run command inside root shell using sudo") 
parser.add_argument('-t', action="store", dest="connect_timeout", required=False, help="ssh timeout to hosts in seconds")
parser.add_argument('-T', action="store", dest="threads", required=False, help="# of threads to run (don't get crazy)")
parser.add_argument('-u', action="store", dest="site_user", required=False, help="Specify a username (by default I use who you are logged in as)")
parser.add_argument('-1', action="store_true", dest="host_per_pool", required=False, help="One host per pool")
args = parser.parse_args()

##GLOBAL##
logging.getLogger('paramiko.transport').addHandler(logging.NullHandler())

stime = time.time()

## SET TIMEOUT ##
connect_timeout = 5
if args.connect_timeout:
    connect_timeout = args.connect_timeout

## SET THREADS / WORKERS ##
workers = 20
if args.threads:
    workers = int(args.threads)

## SET USER / PASS ##
site_user = getpass.getuser()
site_passwd = ''
if args.site_user:
    site_user = args.site_user

failed_logins = []
successful_logins = []

tstamp = datetime.datetime.now().strftime("%Y-%m-%d.%H:%M:%S")
logfile_dir = 'logs'
if not os.path.exists(logfile_dir):
    os.makedirs(logfile_dir)
logfile_path = '%s/runner.log.%s' % (logfile_dir, tstamp)
logfile = open(logfile_path, 'w')

## END GLOBAL ##

def ssh_to_host(hosts, site_passwd):
    for i in range(workers):
        t = threading.Thread(target=worker, args=(site_user, site_passwd))
        t.daemon = True
        t.start()

    for hostname in hosts:
        hostname = hostname.rstrip()
        q.put(hostname)

    q.join()

def worker(site_user, site_passwd):
    while True:
        hostname = q.get()
        node_shell(hostname, site_user, site_passwd)
        q.task_done()


def node_shell(hostname, site_user, site_passwd):
    ssh = paramiko.SSHClient()
    proxy_command = "sconnect -4 -w 4 -S localhost:8081 %s %s" % (hostname,'22')
    proxy_sock = paramiko.ProxyCommand(proxy_command)
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    try:
        ssh.connect(hostname, username=site_user, password=site_passwd, timeout=connect_timeout, sock=proxy_sock)
        transport = ssh.get_transport()
        transport.set_keepalive(1)

        cmd = args.command_string
	if args.sudo: 
		try: 
			## have to use invoke shell for sudo due to ssh config on machines requirng a TTY
			channel = ssh.invoke_shell() 
			sudocmd = 'sudo ' + cmd

			channel.send(sudocmd + '\n') 

			buff = ''
			while not '[sudo] password' in buff: 
				resp = channel.recv(9999)
				buff += resp

			channel.send(site_passwd + '\n') 

			buff = ''
			while not buff.endswith('$ '):
				resp = channel.recv(9999)
				buff += resp

			for line in buff.split('\n'):
				log_and_print("%s: %s" % (hostname, line))

		except Exception as e:
			log_and_print("ERROR: Sudo failed: %s" % (e))  
  
	else: 
        	(stdin, stdout, stderr) = ssh.exec_command(cmd)

		## stdout 
        	for line in stdout.readlines():
            		line = line.rstrip()
            		log_and_print("%s: %s" % (hostname, line))
		## stderr
        	for line in stderr.readlines():
            		line = line.rstrip()
            		log_and_print("%s: %s" % (hostname, line))

        successful_logins.append(hostname)
        ssh.close()

    except Exception as e:
        log_and_print("%s: failed to login : %s" % (hostname, e))
        failed_logins.append(hostname)
        ssh.close()

def log_and_print(message):
    if args.super_quiet_mode or args.list_only:
        if "RUNNER [INPUT]" in message or "RUNNER [ERROR]" in message or "RUNNER" not in message:
            print message
            logfile.write(message + '\n')
    elif args.quiet_mode or args.list_only:
        if "RUNNER [INFO]" not in message:
            print message
            logfile.write(message + '\n')
    else:
        print message
        if not args.list_only:
            logfile.write(message + '\n')

def get_hosts(file_path):
    if os.path.exists(file_path):
        hosts = open(file_path)
        selected_hosts = []
        if not args.host_match:
            selected_hosts = list(hosts)
            log_and_print("RUNNER [INFO]: SELECTING ALL HOSTS")
        else:
            host_match = args.host_match
            for host in hosts:
                if re.search(host_match, host):
                    selected_hosts.append(host)
            log_and_print("RUNNER [INFO]: MATCHING HOSTNAMES WITH '%s'" % (host_match))
    else:
        log_and_print("RUNNER [ERROR]: %s does not exist ! Try running ./update-runner-hosts" % (file_path))
        exit()

    ## Select one host per pool
    if args.host_per_pool:
        seen = {}
        host_per_pool = []
        for host in selected_hosts:
	    # Here strip values that make hostnames unique like #'s
	    # That way the dict matches after 1 host per pool has been seen 
            nhost = re.sub("\d+?\.", ".", host) #Removing #'s in a hostname like host1234.tuxlabs.com
            if not nhost in seen:
                seen[nhost] = 1
                host_per_pool.append(host)
        selected_hosts = host_per_pool

    log_and_print("RUNNER: %s HOSTS HAVE BEEN SELECTED" % (len(selected_hosts)))
    return selected_hosts

if __name__ == "__main__":
    file_path = 'hosts/hosts-all' ## update-hosts-all creates the DIR 

    if args.file_path:
        file_path = args.file_path
        if '~' in file_path:
            print "RUNNER [ERROR]: -f does not support '~'"
            exit()

    if args.list_only or args.command_string:
        selected_hosts = get_hosts(file_path)
        if args.list_only:
            for host in selected_hosts:
                host = host.rstrip()
                log_and_print(host)
            log_and_print("\nThere were %s hosts listed." % (len(selected_hosts)))
            exit()

        else:
            log_and_print("RUNNER [INFO]: LOGFILE SET - %s" % (logfile_path))
            log_and_print("RUNNER [INFO]: USER SET - %s" % (site_user))
            log_and_print("RUNNER [INFO]: SSH CONNECT TIMEOUT is: %s seconds" % (connect_timeout))
            log_and_print("RUNNER [INFO]: THREADS SET - %s" % (workers))
	    if args.sudo:
		log_and_print("RUNNER [INFO]: SUDO IS ON") 

            site_passwd = getpass.getpass("RUNNER [INPUT]: Please Enter Site Pass: ")

            q = Queue.Queue()

            ssh_to_host(selected_hosts,site_passwd)

            etime=time.time()
            run_time = int(etime-stime)

            timestamp = str(datetime.timedelta(seconds=run_time))
            log_and_print("\nRUNNER [RESULT]: Successfully logged into %s/%s hosts and ran your commands in %s second(s)" % (len(successful_logins), len(selected_hosts), timestamp))
            log_and_print("RUNNER [RESULT]: There were %s login failures.\n" % (len(failed_logins)))
            if len(failed_logins) > 0:
                for failed_host in failed_logins:
                    log_and_print("RUNNER [RESULT]: Failed to login to: %s" % (failed_host))
    else:
        parser.print_help()
        output = "\nRUNNER [INFO]: Either -l (list hosts only) or -s (Run cmd string) is required.\n"
        log_and_print(output)

Runner: Multi-threaded SSH with Sudo support using Python & Paramiko Read More »

How To: Install Openstack Icehouse on CentOS 6.5 Using Packstack

Leave a Comment / Cloud, How To's, Openstack, Systems Administration /

openstack-software-diagram

A brief introduction of Openstack + My thoughts

Openstack is open-source software for building clouds. It was created in 2010 by people from Rackspace & NASA, but is currently managed by the non-profit Openstack Foundation, which includes members from the who’s who of the technology sector that have joined forces to continue to invest & develop Openstack (written in Python) for fun and profit( well not so much ). It is definitely fun though :-). In fact I have been having nothing, but a blast since the moment I meant Openstack in 2012. However, admittedly the first time I worked on Openstack it was on the Essex release and I felt it had a ways to go before it was ready for prime time. At that time it was not only hard to install, but most of the people around it were hard core Pythonista’s (Python developers), that were rapidly trying to a mature Openstack & it’s ecosystem that just wasn’t production ready yet. And while the latest release of Openstack ‘Icehouse‘ (which I am covering in this blog) is leaps and bounds better, it can still be quite a PITA(Pain In The Ass) for first timers. In fact I still struggle with the idea of running Openstack in production because it requires an incredible amount of resources, engineering skill & organizational persistence to do so. Companies willing to embark on this journey thus far have included Mercado Libre, Dell, HP, Redhat, Canonical (the Ubunut guys), eBay, PayPal and Symantec among many others. These companies were required to beef up their engineering staff & allocate large amounts of resources just to take on this challenge. In addition their commitments were probably tested over and over again and their leadership had to respond with tremendous faith when timelines began to slip. However, organizations that complete this journey recognize that the return at the end of the tunnel is a compounding return. For starters you aren’t locked into to a vendor like VmWare or AWS (Amazon Web Services) who are both incredibly expensive, but the compounding return comes from skills it will build internally and the culture that will be a bi-product.

Coming here to Tuxlabs means you have personally accepted this mission to learn Openstack and you are in need of some guidance don’t worry you are in good hands. I will show you the light, which can be hard to see through the darkness if you try to take on the documentation all by yourself : http://docs.openstack.org/

So sit back, relax, read, and type exactly as I do. In the end you will have a perfectly functioning Openstack Icehouse cloud and if you decide to bring this into your organization remember with great power comes great responsibility…

You will change everyone in your organization from a POSA (Plain Old Sys Admin’s) to Cloud Engineers and Architects jumping into the Openstack Python code base at the sniff of an issue.

Devstack

Devstack is a shell script used to quickly and easily deploy an all-in-one install of Openstack on any machine or VM for the purposes of trying, testing, and developing on Openstack. It is extremely easy to install, use and get going, if you have not used Devstack already I recommend trying it first because it is a quick and easy way to get a learning win + it will help you decide what you want to learn more about and whether or not Openstack is what you are looking for. However, you do not want to run Devstack in production for starters because production deployments should be multi-node setup’s not all Cloud services deployed on one machine like Devstack does.

The Book

My favorite Openstack book to date is by far is the Openstack Cloud Computing Cookbook written by Kevin Jackson. This book is so great because gets straight to business giving you the commands & understanding needing to get Openstack up and running quickly unlike most other book that bore you with unnecessary details. The first time I installed Openstack Essex I used this book and it worked like a charm, however, that was on Ubuntu, which this book was written for. If you want to install Openstack on Redhat you can do so from scratch or you can use RDO.

RDO

RDO technically doesn’t stand for anything ! http://openstack.redhat.com/Frequently_Asked_Questions#What_does_RDO_stand_for.3F, but I don’t really buy that and have found some people expanding the acronym to the Redhat Distribution of Openstack which I think sounds incredibly fitting. RDO has a website dedicated to a community of people running Openstack on Redhat, CentOS and Fedora. Because Redhat is still the leader in production Linux deployments in the enterprises of the world, I chose to use it in my own lab environment & for the purposes of this tutorial. However, because I cannot afford the license cost of Redhat Enterprise Linux I am using the free community release of RHEL known as CentOS or the Community Enterprise Operating System. For the purposes of this tutorial we will be installing CentOS 6.5 with a minimal install on a bare metal system and then installing Packstack according to the instructions on the RDO website. Then we will go a step further.

Getting Started

A production Openstack deployment has a minimum of 3 nodes one for the Controller, Network, and Compute nodes and that would not cover a Highly Available deployment where you would need the capability to failover functions to standby nodes when there was an outage or maintenance needed. However, for the purposes of this article I will be showing you what is called an all-in-one install where the Controller, Network, and Compute functions all live on the same node and in future articles we will expand on this knowledge to build a production capable multi-node deployment of Openstack.

Before We Begin

You will need…

  1. A system that you have root access to, to install Openstack on. I am using an SGI Rackable with 16GB of RAM and 4.5TB of usable disk ( in a hardware RAID 10). Openstack does not require a lot of resources, but if you’re going to spawn a lot VM’s you need a lot of Cores (vCPU’s), Memory and Storage. See this link for more info:  http://docs.openstack.org/grizzly/openstack-compute/install/yum/content/compute-system-requirements.html
  2. That system should have dual NIC’s, although I think it is possible to use virtual interfaces for a lab environment if you have to.
  3. To configure the hostname of your controller in DNS mine is diamond.tuxlabs.com, but most people go with controller.yourdomain.com (if you don’t have or use DNS, just make sure you configure your hosts file /etc/hosts with the information)
  4. The system you use should have CentOS 6.5 installed with a Minimal Install using the defaults no extra’s.
  5. Your brain, keyboard, fingers and a fresh beer and possibly an ice chest with more beer depending on how far the fridge is.

Assumptions

  • Our home network uses 192.168.1.0/24 and has access to the internet.
  • It has DHCP enabled, but only for 192.168.1.150-199.
  • We are not using 10.0.0.0/24 for anything so we can use it for our private network in Openstack.

If these are true we should be able to follow my examples exactly, but if you want to change your networks you will have to substitute them as needed.

Setting Up Our Initial System

First become root or login as root, then…configure the following configuration files to match.

Resolve.conf

This assumes our gateway device @ 192.168.1.1 runs DNS (like the Linksys’s do). If 192.168.1.1 doesn’t run DNS do not add it to resolve.conf.

[root@diamond tuxninja]# cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 192.168.1.1
[root@diamond tuxninja]#

Sudoers

Make sure wheel is uncommented.

[root@diamond tuxninja]# cat /etc/sudoers | grep '%wheel' | grep -v '#'
%wheel    ALL=(ALL)    NOPASSWD: ALL
[root@diamond tuxninja]#

Setup Your Primary Network Interface

[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:15:17:65:F9:98
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
NAMESERVER=192.168.1.1
[root@diamond tuxninja]#

 Restart Networking

/etc/init.d/network restart

Next Add A Local User (Cause logging in as root is lame)

adduser tuxninja
passwd tuxninja

Set your password to something good, then add the user to the wheel group.

usermod -G tuxninja,wheel tuxninja

Installing Dependencies

Next we need to install some basic dependencies so things like scp, yum, wget and nslookup work.

yum install -y openssh-clients
yum install -y yum-utils
yum install -y wget
yum install -y bind-utils

Turn Off SELinux

Yes security is important, but unfortunately to install Openstack without a headache you need to turn off SELinux.

[root@diamond tuxninja]# cat /etc/sysconfig/selinux  | grep SELINU | grep -v '#'
SELINUX=disabled
SELINUXTYPE=targeted
[root@diamond tuxninja]# reboot

Don’t forget to reboot or the setting will not take effect.

Configuring Networking

There are many ways to configure Openstack networking and it can be quite complicated, so we are going to use what I consider the simplest method using Openvswitch.

Before we begin, make sure you can ping yahoo.com before we go mucking with the configs. We are going to change how your primary network interface is configured and then configure two additional interfaces. Do your best to match these configurations exact, making a mistake here might cause Openvswitch to barf on itself and that would not be fun to troubleshoot.

[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:15:17:65:F9:98
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-ex
ONBOOT=yes
[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:15:17:65:F9:99
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
NETWORK=10.0.0.0
IPADDR=10.0.0.1
NETMASK=255.255.255.0
[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
ONBOOT=yes
[root@diamond tuxninja]# /etc/init.d/network restart

 What Is This Doing ?

The above configuration(s) is setting eth0 as an OVSPort that is bridged to the interface br-ex. eth1 is configured normally for our private 10 network, and br-ex is configured as a OVSBridge with our actual IP information from / for eth0.

After Restarting Networking ifconfig Output Should Look Like This

[root@diamond tuxninja]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:15:17:65:F9:98 
          inet6 addr: fe80::215:17ff:fe65:f998/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:322092 errors:0 dropped:0 overruns:0 frame:0
          TX packets:187420 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:436592198 (416.3 MiB)  TX bytes:15605772 (14.8 MiB)
          Interrupt:18 Memory:b8820000-b8840000

[root@diamond tuxninja]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:15:17:65:F9:99 
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe65:f999/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:552 (552.0 b)
          Interrupt:19 Memory:b8800000-b8820000

[root@diamond tuxninja]# ifconfig br-ex
br-ex     Link encap:Ethernet  HWaddr 00:15:17:65:F9:98 
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a8db:16ff:fed6:f4c4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:293 errors:0 dropped:0 overruns:0 frame:0
          TX packets:151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43799 (42.7 KiB)  TX bytes:15786 (15.4 KiB)

[root@diamond tuxninja]#

For more information on Openstack networking here are some references that I used.

  • https://www.youtube.com/watch?v=afImoFeuDnY
  • http://openstack.redhat.com/Neutron_with_existing_external_network
  • http://openstack.redhat.com/forum/discussion/780/confused-with-neutron-and-external-network-access/p1
  • http://openstack.redhat.com/Networking_in_too_much_detail
  • http://docs.openstack.org/grizzly/openstack-network/admin/content/

Update Packages

Finally since we are running a minimum install, let’s update all of our packages before continuing.

[root@diamond tuxninja]# sudo yum update -y

 Installing Openstack

Finally we are ready to install Openstack Using Packstack.

[root@diamond tuxninja]# sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm
[root@diamond tuxninja]# sudo yum install -y openstack-packstack
[root@diamond tuxninja]# packstack --allinone --provision-all-in-one-ovs-bridge=n --provision-demo=n

The first command adds the required RDO repo to yum install packstack. The second downloads the openstack-packstack package. And the third and final command is magic. This command installs packstack. We are telling it to install all openstack components to one machine the –provision-all-in-one-ovs-bridge=n tells packstack we are going to be using a single node (although I am still not entirely sure this flag is absolutely necessary) and the final flag tells packstack not to deploy the demo project, because if you do you have to end up deleting it before you can delete the network information and re-create it correctly.

Packstack will take about 10 minutes to run. It uses puppet to deploy Openstack and it’s required configurations. In my experience it works pretty well, however if you do need to re-install packstack there isn’t an automated uninstall script included. So someone created one and here it is in case you need it.

[root@diamond tuxninja]# cat packstack_uninstall.sh
#!/bin/bash

# Warning! Dangerous step! Destroys VMs
for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do
    virsh destroy $x ;
    virsh undefine $x ;
done ;

# Warning! Dangerous step! Removes lots of packages
yum remove -y nrpe "*nagios*" puppet "*ntp*" "*openstack*" \
"*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" \
mysql mysql-server httpd "*memcache*" scsi-target-utils \
iscsi-initiator-utils perl-DBI perl-DBD-MySQL ;

# Warning! Dangerous step! Deletes local application data
rm -rf /etc/nagios /etc/yum.repos.d/packstack_* /root/.my.cnf \
/var/lib/mysql/ /var/lib/glance /var/lib/nova /etc/nova /etc/swift \
/srv/node/device*/* /var/lib/cinder/ /etc/rsync.d/frag* \
/var/cache/swift /var/log/keystone /var/log/cinder/ /var/log/nova/ \
/var/log/httpd /var/log/glance/ /var/log/nagios/ /var/log/quantum/ ;

umount /srv/node/device* ;
killall -9 dnsmasq tgtd httpd ;

vgremove -f cinder-volumes ;
losetup -a | sed -e 's/:.*//g' | xargs losetup -d ;
find /etc/pki/tls -name "ssl_ps*" | xargs rm -rf ;
for x in $(df | grep "/lib/" | sed -e 's/.* //g') ; do
    umount $x ;
done

Don’t forget to chmod +x to that bad boy to make it executable so you can run it, when you are ready to uninstall.

Now then, getting back to our install. If Packstack was successful you should see something like this.

 **** Installation completed successfully ******


Additional information:
 * A new answerfile was created in: /root/packstack-answers-20140802-125113.txt
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.1.10. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://192.168.1.10/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * To use Nagios, browse to http://192.168.1.10/nagios username: nagiosadmin, password: 4bf73f1116544975
 * The installation log file is available at: /var/tmp/packstack/20140802-125113-RzCDrE/openstack-setup.log
 * The generated manifests are available at: /var/tmp/packstack/20140802-125113-RzCDrE/manifests
[root@diamond tuxninja]#

Check Out Our Keystonerc_admin File

Just take note of what environment variables it sets. To login to our GUI we need these credentials and to use any command line functionality we have to source this file in our shell.

[root@diamond tuxninja]# cat ~/keystonerc_admin
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PASSWORD=a57c83f56ccc41f5
export OS_AUTH_URL=http://192.168.1.10:5000/v2.0/
export PS1='[\u@\h \W(keystone_admin)]\$ '
[root@diamond tuxninja]#

 Configuring Openstack Networking

Because we did not deploy the demo project, the default network configuration in Openstack should not exist. If it does we can login to our GUI @ http://diamond.tuxlabs.com/dashboard by sourcing the credentials above in our keystonerc_admin and delete the network configuration under

  • admin –> routers
  • admin –> networks

Then you are ready to re-create your network. You could use the GUI to do this, which is pretty straight forward…but since GUI’s are dirty we are going to use the command line.

In order to use any openstack command line utilities you must first source the keystonerc_admin file so the required environment variables are set in your shell. Like so…

[root@diamond ~]# source keystonerc_admin 
[root@diamond ~(keystone_admin)]#

 Seeing (keystone_admin) in your prompt means you are ready to run commands and here you go

neutron router-create router1
neutron net-create private
neutron subnet-create private 10.0.0.0/24 --name private_subnet
neutron router-interface-add router1 private_subnet
neutron net-create public --router:external=True
neutron subnet-create public 192.168.1.0/24 --name public_subnet --enable_dhcp=False --allocation-pool start=192.168.1.51,end=192.168.1.99 --gateway=192.168.1.1
neutron router-gateway-set router1 public

Copy & pasting this will spit out a lot of messages showing the result set of each command.

The important thing to realize is that your public network has to be configured with an  external router.  This effectively NAT’s the 10 & 192 networks to make internet access available to your VM’s. When this is configured correctly your Network Topology diagram under the dirty GUI should resemble this.

2014-08-04 12.03.09 am

The router_gateway always shows DOWN. No idea why, but someone else said it’s a bug 😉

Next under your project (the admin project) you have to configure your Security Groups under Compute—>Access & Security—>Security Groups. Once there click Manage Rules for the default security group. Delete what’s there. Add Ingress/Egress for ALL ICMP, ALL TCP, and ALL UDP accepting all other defaults on the form. This will open up your firewall completely.

Time To Restart Openstack

Finally we need to restart Openstack, validating that openvswitch-agent starts.

[root@diamond tuxninja(keystone_admin)]# /etc/init.d/network restart;openstack-service --full-restart;openstack-status | grep 'openvswitch-agent' 
neutron-openvswitch-agent:              active

Note: You can use openstack-service –full-restart to restart all openstack services.

If openvswitch-agent does not start run back through the above steps and / or consult the references or email me tuxninja [at] tuxlabs.com. This can be a real pain to figure out, but I finally have it down pat.

If openvswitch-agent is active we are good to go. Next we have to create an SSH Key.

Creating Our Cloud’s SSH Key

Our Openstack Cloud has no authentication system for our virtual machines by default. Eventually you could configure an LDAP server, and configure your images or configure in Puppet the required pam configurations to use LDAP for authentication by your VM’s, but for now Openstack allows for a post configuration step after building VM’s where it will add your SSH key. It does this by using the metadata service which cloud images will look for, if they can reach the metadata service @ http://169.254.169.254 then they can copy down the public key and install it on the VM for you allowing you to login to that VM using the VM’s operating system default account (i.e. ubuntu, fedora). Now before I continue there are alternatives such as using guestfish to modify an image’s configuration or using a post install cloud-init configuration to specific a password to an account, but the cleanest and simplest way is to use the metadata service after generating & importing the public key into openstack and assigning it to our VM guest.

To create the key…

[root@diamond tuxninja]# ssh-keygen -t rsa -f cloud.key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in cloud.key.
Your public key has been saved in cloud.key.pub.
The key fingerprint is:
59:e0:b3:0f:c9:d0:8a:67:ba:86:39:0e:fe:b2:93:ce root@diamond.tuxlabs.com
The key's randomart image is:
+--[ RSA 2048]----+
|        .        |
|       o .       |
|      . + .      |
|     . + *       |
|    . + S        |
|     +   o       |
|. .o.     .      |
|o=+ ..           |
|.E*+.            |
+-----------------+
[root@diamond tuxninja]#

I don’t enter a password to use password-less SSH, cause again this is a lab environment and typing passwords sucks.

Launching A VM

Next you want to launch a VM that we can use to import our SSH key into. Login to the dashboard @ http://diamond.tuxlabs.com/dashboard again using the keystonerc_admin credentials provided. Once inside navigate to Project—>Compute—>Instances and click the Launch Instance button in the upper right corner. You will be presented with a form. Here are the screenshots to guide you through.

Fill out the details tab…

2014-08-04 12.19.19 am

Click the Access & Security Tab…

2014-08-04 12.19.44 am

Then Click The + …

Now in order to fill this out we have to copy and paste the contents (bold part) of the public key we created.

[root@diamond tuxninja]# cat cloud.key.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzs46kFxia6q9gfwmmszFHGFSk5UAQFjLhEOE08I/2F7+Qwh0AFBuO99c/OCITr2HvVfDc7DF5JlRQGohTA9aowtINWyEUvEKSPG8gEtL95GZ5EZH/3pqWFaklIfoP3fbgKPrpPSGarYQY5UH+qm419x+A3tAkmumdd3BtQHLRF1qq9Ui69IQ383MctuOu+sF9Ws6+NT1+MbND++Qp2VNli/+cpZTnv4g9i5C2mXbUQQJghEoUvIXiBh+r27kVOlJ6L3TeT8WIwD1N916ZRgHrfbAEQ+108Zdyc5/TXYRbbWs4WO9PVcTp+6zcLOXyds0sFMyu53rWOS22iEFL3XGRQ== root@diamond.tuxlabs.com
[root@diamond tuxninja]#

2014-08-04 12.20.10 am

Then Click Networking and add the private network…

2014-08-04 12.20.43 am

Then click launch….and you should see something like this…

2014-08-04 12.21.22 am

See the status spawning ? When the virtual machine is done being built you will see status changed to ‘Running’, you can watch the machine boot, by clicking the name of the instance and then going to the Log or Console tab.

2014-08-04 12.28.05 am

This is great our VM is running and it has an IP address configured ! But that IP address is only used for Openstack communication, so we still need to associate a floating IP to this system so we can SSH to it. Click on the More dropdown and select Associate Floating IP.

2014-08-04 12.29.52 am

Then click the + …

2014-08-04 12.30.08 am

Click Allocate IP…

2014-08-04 12.30.25 am

And then click associate…

2014-08-04 12.30.40 am

Now you should see your VM has an internal IP (a 10 dot address) and an External IP on the 192.168.1.0/24 network. Now we can SSH to our VM.

We didn’t mention it earlier, but Openstack comes with only one Linux image by default, it’s called CirrOS and it’s just a tiny-minimal cloud image for testing. The login to this operating system is cirros / cubswin:) … which is visible from the console log of the machine once fully booted. Now you could SSH into the machine on 192.168.1.54 using that login and password like so…

[root@diamond tuxninja]# ssh cirros@192.168.1.54
The authenticity of host '192.168.1.54 (192.168.1.54)' can't be established.
RSA key fingerprint is d1:19:89:1d:f0:4b:20:27:a3:7c:99:0d:75:fe:d0:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.54' (RSA) to the list of known hosts.
cirros@192.168.1.54's password: 
$

 But that is lame ! We imported our SSH key remember ? So how do we use that ? Like this..

[root@diamond tuxninja]# ssh -i cloud.key cirros@192.168.1.54
$

 Wow, that is cool.

Now what ? Let’s add some more images.

Installing More Operating System Images

Ok, Openstack is installed, Networking is working, Metadata service and our keys are working, we are happy campers, but to make this Cloud useful we are going to need some real Linux images. There is two ways to install images using glance (the openstack image service).

In the first method we download the image using wget & then we run the proper glance command. Here is this approach in action.

Don’t forget to source the rc file if you have not already

[root@diamond ~]# source keystonerc_admin
[root@diamond ~(keystone_admin)]# wget http://download.fedoraproject.org/pub/fedora/linux/updates/20/Images/x86_64/Fedora-x86_64-20-20140407-sda.qcow2
[root@diamond ~(keystone_admin)]# glance image-create --name='Fedora 6.4' --is-public=true \ --container-format=bare --disk-format=qcow2 < Fedora-x86_64-20-20140407-sda.qcow2

 Alternatively, we could do the same thing in one command !

[root@diamond ~(keystone_admin)]# glance image-create --name ubuntu-server-12.04 --is-public true --container-format bare --disk-format qcow2 --copy-from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

Remember whenever you run openstack commands you are going the status or result set back from the command.

Once you have installed the Fedora and Ubuntu images you should have a real cloud on your hands. Now you can do things like expand your setup for multi-node or get LBaaS working for load balancing requests to your web servers for example. Or if you want to be amazeballs you could install things like Puppet inside of Docker, or Cloud Foundry and build your PaaS (Platform As A Service)!

Final Comments

Openstack can be tricky even for an experienced Sys Admin. While learning Openstack I found it difficult to find tutorials on exactly the setup I was looking for mainly in terms of how the network was being configured, and this often made me second guess myself when I would run into an issue as I learned Openstack. I wanted to write this article to give back and help to educate my brothers and students of life long learning as you embark on your Openstack adventure, Godspeed. Here are some commands of note to help you along your way.

How to launch a VM from command line

[root@diamond ~(keystone_admin)]# nova boot --flavor m1.small --image 'Fedora 6.4' --security-groups default --key-name openstack --nic net-id=975a58eb-d3cb-4c5b-956b-ed1bdb0820d2 fed1

 How to run command on a network namespace (VM Instance)

[root@diamond tuxninja]# ip netns exec qrouter-71064321-eb87-4d28-85ee-05f6ddc75968 iptables -L -t nat | grep 169
REDIRECT tcp -- anywhere= 169.254.169.254 tcp dpt:http redir ports 9697
[root@diamond tuxninja]# ip netns exec `ip netns list | grep qrouter` netstat -anpt

How to restart only neutron services

[root@diamond tuxninja]# cd /etc/init.d/; for i in $(ls -1 neutron-* | cut -d \. -f 1); do sudo service $i restart; done

Or a simpler method would be… 

[root@diamond tuxninja]# openstack-service restart neutron
Stopping neutron-dhcp-agent: [FAILED]
Starting neutron-dhcp-agent: [  OK  ]
Stopping neutron-l3-agent: [  OK  ]
Starting neutron-l3-agent: [  OK  ]
Stopping neutron-metadata-agent: [  OK  ]
Starting neutron-metadata-agent: [  OK  ]
Stopping neutron-openvswitch-agent: [  OK  ]
Starting neutron-openvswitch-agent: [  OK  ]
Stopping neutron: [  OK  ]
Starting neutron: [  OK  ]
[root@diamond tuxninja]#

How to add an account / change a password using cloud-init

Stick this under Post Creation Customized Script to

#cloud-config
password: mysecret
chpasswd: { expire: False }
ssh_pwauth: True

How to edit the password file of an image

[root@diamond tuxninja]# guestfish -a disk.img -i vi /etc/passwd

How to see the status of openstack services

[root@diamond tuxninja]# openstack-status

How to check your log files for more clues

you can do this for every openstack service, but below I am just showing the two most popular to troubleshoot for getting your cloud up and going, nova and neutron aka controller, compute and networking logs.

[root@diamond tuxninja]# tail /var/log/nova/*.log
[root@diamond tuxninja]# tail /var/log/neutron/*.log

Want to change which partition Openstack uses on local disk (ephemeral storage) to deploy VM’s ?

If the majority of your ephemeral (local) disk is under a different partition other than /var… For example, mine was under /home, then you need to change your state path and restart openstack services. Don’t forget to copy any existing files in /var/lib/nova/ to the new location.

[root@diamond ~(keystone_admin)]# grep state_path /etc/nova/nova.conf | grep -v '#' | grep -v ^$ 
state_path=/home/openstack/var/lib/nova
[root@diamond ~(keystone_admin)]#

Checking your openvswitch configuration

[root@diamond ~(keystone_admin)]# ovs-vsctl show
ee001583-903b-4397-8805-40c18b528d2c
    Bridge br-int
        Port "qvo0cadc326-6b"
            tag: 3
            Interface "qvo0cadc326-6b"
        Port "tap86f8ac73-41"
            tag: 3
            Interface "tap86f8ac73-41"
                type: internal
        Port "qr-11baa878-fb"
            tag: 3
            Interface "qr-11baa878-fb"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qvo6673d677-ed"
            tag: 3
            Interface "qvo6673d677-ed"
        Port "qvoc8a2b994-a1"
            tag: 3
            Interface "qvoc8a2b994-a1"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth0"
            Interface "eth0"
        Port "qg-82e238cc-3d"
            Interface "qg-82e238cc-3d"
                type: internal
    ovs_version: "1.11.0"
[root@diamond ~(keystone_admin)]#

The Final Gem

The Most Popular Issue I ran Into When Installing Openstack

Looks like this in the console..

novalidhost

This is the generic error for I didn’t have resources to create the VM, which can happen for a multitude of reasons…Like for example, the Openvswitch-agent not running.

[root@diamond metadata]# openstack-status
...
neutron-openvswitch-agent:              dead

But why is Openvswitch Agent Failing ? 

[root@diamond metadata]# tail /var/log/neutron/openvswitch-agent.log
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2014-07-27 12:47:31.072 17288 TRACE neutron     sys.exit(main())
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1394, in main
2014-07-27 12:47:31.072 17288 TRACE neutron     agent = OVSNeutronAgent(**agent_config)
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 216, in __init__
2014-07-27 12:47:31.072 17288 TRACE neutron     self.setup_tunnel_br(tun_br)
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 707, in setup_tunnel_br
2014-07-27 12:47:31.072 17288 TRACE neutron     if int(self.patch_tun_ofport) < 0 or int(self.patch_int_ofport) < 0:
2014-07-27 12:47:31.072 17288 TRACE neutron TypeError: int() argument must be a string or a number, not 'NoneType'
2014-07-27 12:47:31.072 17288 TRACE neutron

Additionally, I have seen different errors presented referencing vif.py 

The fix for this could be multiple different things. What you should do first is verify your network interfaces are configured 100% correctly. Compare your ovs-vsctl output to mine above… Look different ? Shorter, stuff is missing ? Ok this error usually means Openstack is having difficulty inserting port’s into Openvswitch for whatever reason. What I found is the most common error is that you did not restart networking, before restarting openstack networking after making a configuration change to the interfaces. So to resolve this try…

[root@diamond metadata]# /etc/init.d/network restart
Shutting down interface br-ex:                             [  OK  ]
Shutting down interface br-int:                            [  OK  ]
Shutting down interface br-tun:                            [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface br-ex:  Determining if ip address 192.168.1.10 is already in use for device br-ex...
                                                           [  OK  ]
Bringing up interface br-int:                              [  OK  ]
Bringing up interface br-tun:                              [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]
Bringing up interface eth1:  Determining if ip address 10.0.0.1 is already in use for device eth1...
                                                           [  OK  ]
[root@diamond metadata]# openstack-service --full-restart

Stopping openstack-ceilometer-alarm-evaluator:             [  OK  ]
Starting openstack-ceilometer-alarm-evaluator:             [  OK  ]
Stopping openstack-ceilometer-alarm-notifier:              [  OK  ]
Starting openstack-ceilometer-alarm-notifier:              [  OK  ]
Stopping openstack-ceilometer-api:                         [  OK  ]
Starting openstack-ceilometer-api:                         [  OK  ]
Stopping openstack-ceilometer-central:                     [  OK  ]
Starting openstack-ceilometer-central:                     [  OK  ]
Stopping openstack-ceilometer-collector:                   [  OK  ]
Starting openstack-ceilometer-collector:                   [  OK  ]
Stopping openstack-ceilometer-compute:                     [  OK  ]
Starting openstack-ceilometer-compute:                     [  OK  ]
Stopping openstack-ceilometer-agent-notification:          [  OK  ]
Starting openstack-ceilometer-agent-notification:          [  OK  ]
Stopping openstack-cinder-api:                             [  OK  ]
Starting openstack-cinder-api:                             [  OK  ]
Stopping openstack-cinder-backup:                          [  OK  ]
Starting openstack-cinder-backup:                          [  OK  ]
Stopping openstack-cinder-scheduler:                       [  OK  ]
Starting openstack-cinder-scheduler:                       [  OK  ]
Stopping openstack-cinder-volume:                          [  OK  ]
Starting openstack-cinder-volume:                          [  OK  ]
Stopping openstack-glance-api:                             [  OK  ]
Starting openstack-glance-api:                             [  OK  ]
Stopping openstack-glance-registry:                        [  OK  ]
Starting openstack-glance-registry:                        [  OK  ]
Stopping openstack-glance-scrubber:                        [  OK  ]
Starting openstack-glance-scrubber:                        [  OK  ]
Stopping keystone:                                         [  OK  ]
Starting keystone:                                         [  OK  ]
Stopping openstack-nova-api:                               [  OK  ]
Starting openstack-nova-api:                               [  OK  ]
Stopping openstack-nova-cert:                              [  OK  ]
Starting openstack-nova-cert:                              [  OK  ]
Stopping openstack-nova-compute:                           [  OK  ]
Starting openstack-nova-compute:                           [  OK  ]
Stopping openstack-nova-conductor:                         [  OK  ]
Starting openstack-nova-conductor:                         [  OK  ]
Stopping openstack-nova-console:                           [  OK  ]
Starting openstack-nova-console:                           [  OK  ]
Stopping openstack-nova-consoleauth:                       [  OK  ]
Starting openstack-nova-consoleauth:                       [  OK  ]
Stopping openstack-nova-metadata-api:                      [FAILED]
Starting openstack-nova-metadata-api:                      [  OK  ]
Stopping openstack-nova-novncproxy:                        [  OK  ]
Starting openstack-nova-novncproxy:                        [  OK  ]
Stopping openstack-nova-scheduler:                         [  OK  ]
Starting openstack-nova-scheduler:                         [  OK  ]
Stopping openstack-nova-spicehtml5proxy:                   [FAILED]
Starting openstack-nova-spicehtml5proxy:                   [  OK  ]
Stopping openstack-nova-xvpvncproxy:                       [  OK  ]
Starting openstack-nova-xvpvncproxy:                       [  OK  ]
Stopping openstack-swift-account:                          [  OK  ]
Starting openstack-swift-account:                          [  OK  ]
Stopping openstack-swift-account-auditor:                  [  OK  ]
Starting openstack-swift-account-auditor:                  [  OK  ]
Stopping openstack-swift-account-reaper:                   [  OK  ]
Starting openstack-swift-account-reaper:                   [  OK  ]
Stopping openstack-swift-account-replicator:               [  OK  ]
Starting openstack-swift-account-replicator:               [  OK  ]
Stopping openstack-swift-container:                        [  OK  ]
Starting openstack-swift-container:                        [  OK  ]
Stopping openstack-swift-container-auditor:                [  OK  ]
Starting openstack-swift-container-auditor:                [  OK  ]
Stopping openstack-swift-container-replicator:             [  OK  ]
Starting openstack-swift-container-replicator:             [  OK  ]
Stopping openstack-swift-container-updater:                [  OK  ]
Starting openstack-swift-container-updater:                [  OK  ]
Stopping openstack-swift-object:                           [  OK  ]
Starting openstack-swift-object:                           [  OK  ]
Stopping openstack-swift-object-auditor:                   [  OK  ]
Starting openstack-swift-object-auditor:                   [  OK  ]
Stopping openstack-swift-object-expirer:                   [  OK  ]
Starting openstack-swift-object-expirer:                   [  OK  ]
Stopping openstack-swift-object-replicator:                [  OK  ]
Starting openstack-swift-object-replicator:                [  OK  ]
Stopping openstack-swift-object-updater:                   [  OK  ]
Starting openstack-swift-object-updater:                   [  OK  ]
Stopping openstack-swift-proxy:                            [  OK  ]
Starting openstack-swift-proxy:                            [  OK  ]
Stopping neutron-dhcp-agent:                               [  OK  ]
Starting neutron-dhcp-agent:                               [  OK  ]
Stopping neutron-l3-agent:                                 [  OK  ]
Starting neutron-l3-agent:                                 [  OK  ]
Stopping neutron-lbaas-agent:                              [FAILED]
Starting neutron-lbaas-agent:                              [  OK  ]
Stopping neutron-metadata-agent:                           [  OK  ]
Starting neutron-metadata-agent:                           [  OK  ]
Stopping neutron-openvswitch-agent:                        [FAILED]
Starting neutron-openvswitch-agent:                        [  OK  ]
Stopping neutron-openvswitch-agent:                        [  OK  ]
Starting neutron-openvswitch-agent:                        [  OK  ]
Stopping neutron:                                          [  OK  ]
Starting neutron:                                          [  OK  ]
[root@diamond metadata]#

Now when we run openstack-status the openvswitch-agent should be running !

[root@diamond metadata]# openstack-status
...
neutron-openvswitch-agent:              active


[root@diamond metadata]# tail /var/log/neutron/openvswitch-agent.log
2014-07-27 12:50:13.793 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent initialized successfully, now running...
2014-07-27 12:50:13.800 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent out of sync with plugin!
2014-07-27 12:50:13.801 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent tunnel out of sync with plugin!
2014-07-27 12:50:13.851 24985 INFO neutron.openstack.common.rpc.common [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Connected to AMQP server on 192.168.1.10:5672
2014-07-27 12:50:14.329 24985 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'e44457a8-c88a-4a3a-818a-c96c75e16979'])
2014-07-27 12:50:15.023 24985 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Device e44457a8-c88a-4a3a-818a-c96c75e16979 not defined on plugin
2014-07-27 12:50:18.119 24985 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'9cdf8669-9e09-48bb-873e-84cfad27993f'])
2014-07-27 12:50:18.812 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 9cdf8669-9e09-48bb-873e-84cfad27993f updated. Details: {u'admin_state_up': True, u'network_id': u'975a58eb-d3cb-4c5b-956b-ed1bdb0820d2', u'segmentation_id': 10, u'physical_network': None, u'device': u'9cdf8669-9e09-48bb-873e-84cfad27993f', u'port_id': u'9cdf8669-9e09-48bb-873e-84cfad27993f', u'network_type': u'vxlan'}
2014-07-27 12:50:18.812 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Assigning 1 as local vlan for net-id=975a58eb-d3cb-4c5b-956b-ed1bdb0820d2
2014-07-27 12:50:19.205 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Configuration for device 9cdf8669-9e09-48bb-873e-84cfad27993f completed.
[root@diamond metadata]#

 And Healthy ! Awesome. Last, but not least you can find all my blessed configs for this article as well as a script I wrote to snapshot those configs on my github. Thanks for learning with me.

 

 

How To: Install Openstack Icehouse on CentOS 6.5 Using Packstack Read More »

Python & Fun with Checkio.org

Leave a Comment / Programming, Python /

My Brief Coding History

13 years ago, I fell in love with Perl. I liked Perl a lot better than my previous flings with HTML, Shell (Bash), PHP, ASP, Javascript, Sed, Awk and a few others, mainly because Perl was always willing to do what I wanted, when I wanted, making my life as a Sys Admin a lot easier and never asked much in return just to be loved. I loved Perl for years and years and in a way I will never stop loving Perl, after all Perl was my first love. But too often our first loves die hard no matter how hard we try to will their existence forward…

“I never thought I could love a language more than Perl, until I met Python”  (TuxNinja, 2012)

Enter Python

Old loves die hard.. I haven’t used Perl for about ~18 months, whatever the exact amount of time is I am not sure, but it matches the exact time at which I started learning Python.  I decided to start learning Python because it’s popularity seemed to be growing exponentially (and still does), once I learned basic syntax by reading and watching a couple of videos, I was up and running and it wasn’t long before I was solving real business critical problems with it.

[stextbox id=”grey” caption=”Best Way To Learn A New Language”]I have learned a lot of languages. By far the best way to learn a new language is to solve a real problem (preferably at your job) with it. The reason being is that inevitably when you solve the problem you will need to maintain the code (i.e. add features and fix bugs). This ‘recall’ you experience every time you re-visit and re-factor your code will burn into your brain valuable learning’s better than any amount of writing and/or reading alone. Suffices to say that troubleshooting is the best way to learn.[/stextbox]

Two Reasons

There were two major reasons once I started learning Python I decided it was my new #1.

  1. Python is the most readable language I have ever seen. It reads like pseudo code or plain english. If that isn’t important to you, go try to read someones competition winning obfuscated code ./facepalm. For the 13 years I coded Perl every time I took over a Perl script from someone else I re-wrote it. That is because Perl allows for coders to follow many different styles such as a C or Shell style of coding. It’s flexibility is it’s greatest enemy in my opinion, there are 1000 ways to do something in Perl. In Python there are 10’s of ways, but there is usually only 1 pythonic way. I prefer a language that has a clear best way to do something and that encourages readability. Of course there are still some assholes out there who prefer lambda() and map() over list comprehension and to them I say… read a book (err website)
  2. Community support. I forever loved Perl. Two of the reasons was the incredible support I found on www.perlmonks.com and because of CPAN and the plethra of modules that existed. Well Python wins again. It’s seems most former monks now spend their time correcting peoples Python on StackOverflow.com and PyPi > CPAN, sorry but it’s true !

All that being said, I encourage you to learn Python, Ruby, and Node.js… these 3 seem to be the front runners of that the next generation is learning and therefore will continue to be abound now and in the future. But since this is a Python article and the language I know best, learn Python first !

CheckIO

Finally, the purpose of this article! About 3 months into learning Python a good friend of mine who is a programmer for a living told me about CheckIO.org, which is a website, that provides a fun & free way to learn Python in the form of a game that reminds me of Wii’s Super Mario. Now when I initially signed up ~15 months ago I never actually ended up playing the game. Until a couple of days ago when I got an email from them asking how I liked CheckIO. Well that was all the reminder I needed, looking to continue in life long learning and specifically my Python knowledge I started playing the game. It’s been 48 hours and I am pretty hooked ! The benefits to learning using CheckIO are unmatched by any other learning mechanisms I have used to date here’s why.

How It Works

CheckIO starts you at your ‘home’ and presents you with challenges that it gives you points and badges for completing. Once you complete enough you move on to the next challenge and eventually unlock other adventures and your next set of challenges. Yea, yea ok that’s cool so what…well when you solve your challenge, you have the option of publishing your code and reviewing / voting on the best solution of all time for that problem. This is an amazing thing. It combines not only exercising to solve a problem yourself, but ultimately showing you the most clean, efficient and pythonic way (caveat: not always but mostly the social voting keeps this true). This is an incredible about of knowledge in one place, and the challenges you are solving are universally applicable to some of the most common problems with data structures, sorting and counting that you will run into. Admittedly, I am a level 4 currently and only about 60% of the way through and have already unlocked other adventures, but I was having so much fun and learning so much that I decided I’m going to complete 100% of an adventure before moving on.

CheckIO Teaches You To Be An Efficient Programmer

Feel free to read that again. Yes it does. My whole programming life I have been focused on function…function, function, function. Making my code functional was always #1 because after all you are solving a problem and making your code functional has to be the right first step anyway, you can’t work on making an application efficient if you haven’t figured out a way to solve the problem programmaticly. So the only time I went back and re-wrote an program to make it more efficient was when it was heavily used so incredibly inefficient that it demanded to be FIXED!

Let There Be Light

What CheckIO does is encourages you to compete with other programmers solving the same challenges in the game format, keeping it fun. So finally 6 challenges in, I started to get it… instead of just solving the challenge, why not try to solve the challenge in the most efficient & cleanest way possible and then publish my code for voting with the community. I did just that and the difference in my code was a difference of 47 lines.  The example I am showing below is a basic one, but the story is what’s crucial. If you solve the CheckIO problems and challenge yourself to write the best possible solution each time and then compare against your peers you will ultimately be a far better programmer then most people. Not to mention you will have a nice repository for re-usable code for common problems you will encounter as a programmer.

Ok enough…it’s show and tell time.

The Problem

Full Problem Description: http://www.checkio.org/mission/roman-numerals/
Short Version: For this task, you should return a roman numeral using the specified integer value ranging from 1 to 3999.

Input: An integer ranging from 1 to 3999.
Output: A string in the form of a Roman numeral.
Example:

checkio(6) == 'VI'
checkio(76) == 'LXXVI'

Functional Solution

import random

numerals = { 1 : 'I', 4 : 'IV', 5 : 'V', 9 : 'IX', 10 : 'X', 40 : 'XL', 50 : 'L', 90 : 'XC', 100 : 'C', 400 : 'CD', 500 : 'D', 900 : 'CM', 1000 : 'M' }
number = random.randrange(4000)
string = ''

if (number / 1000):
    thousands = number / 1000 # thousands
    string += numerals[1000]*thousands
    number = number - (thousands*1000)
if (number / 900):
    ninehundreds = number / 900 # 900's
    string += numerals[900]*ninehundreds
    number = number - (ninehundreds*900)
if (number / 500):
    fivehundreds = number / 500 # 500's
    string += numerals[500]*fivehundreds
    number = number - (fivehundreds*500)
if (number / 400):
    fourhundreds = number / 400 # 400's
    string += numerals[400]*fourhundreds
    number = number - (fourhundreds*400)
if (number / 100):
    hundreds = number / 100 # 100's
    string += numerals[100]*hundreds
    number = number - (hundreds*100)
if (number / 90):
    nineties = number / 90 # 90's
    string += numerals[90]*nineties
    number = number - (nineties*90)
if (number / 50):
    fifties = number / 50 # 50's
    string += numerals[50]*fifties
    number = number - (fifties*50)
if (number / 40):
    forties = number / 40 # 40's
    string += numerals[40]*forties
    number = number - (forties*40)
if (number / 10):
    tens =  number / 10 # 10's
    string += numerals[10]*tens
    number = number - (tens*10)
if (number / 9):
    nines =  number / 9 # 9's
    string += numerals[9]*nines
    number = number - (nines*9)
if (number / 5):
    fives = number / 5 # 5's
    string += numerals[5]*fives
    number = number - (fives*5)
if (number / 4):
    fours = number / 4 # 4's
    string += numerals[4]*fours
    number = number - (fours*4)
if (number / 1):
    ones = number / 1 # 1's
    string += numerals[1]*ones
    number = number - (ones*1)

print string

My Think Twice Solution

It’s amazing how quickly you can improve code once you understand how it needs to function. This improvement took me a total of 2 minutes to finish.

import random

numerals = { 1 : 'I', 4 : 'IV', 5 : 'V', 9 : 'IX', 10 : 'X', 40 : 'XL', 50 : 'L', 90 : 'XC', 100 : 'C', 400 : 'CD', 500 : 'D', 900 : 'CM', 1000 : 'M' }
number = random.randrange(4000)
string = ''

for roman in sorted(numerals, reverse=True):
    if (number / roman):
        multiplier = number / roman
        string += numerals[roman]*multiplier
        number = number - (multiplier*roman)

print string

Published Solution

The above solutions were what I used to test and run before submitting, when you actually submit it has to be as a CheckIO function. The real code submitted can be found below here.

 

 

 

Python & Fun with Checkio.org Read More »