Consul for Service Discovery

Why Service Discovery ?

Service Discovery effectively replaces the process of having to manually assign or automate your own DNS entries for nodes on your network. Service Discovery aims to move even further away from treating VM’s like pets to cattle, by getting rid of the age old practice of Hostname & FQDN having contextual value. Instead when using services discovery nodes are automatically registered by an agent and automatically are configured in DNS for both nodes and services running on the machine.

Consul

Consul by Hashicorp is becoming the de-facto standard for Service Discovery. Consul’s full features & simplistic deployment model make it an optimal choice for organizations looking to quickly deploy Service Discovery capabilities in their environment.

Components of Consul

The Consul Agent
An optional JSON config file for each service located under /etc/consul.d/<service>.json
1. If you do not specific a JSON file, consul can still start and will provide discovery for the nodes (they will have DNS as well)

A Quick Example of Consul

How easy is it to deploy console ?

Download / Decompress and install the Consul agent – https://www.consul.io/downloads.html
Define services in a JSON file (if you want) – https://www.consul.io/intro/getting-started/services.html
Start the agent on the nodes – https://www.consul.io/intro/getting-started/join.html
Make 1 node join 1 other node (does not matter which node) to join the cluster, which gets you access to all cluster metadata

Steps 1 and 2 Above

After downloading the Consul binary to each machine and decompressing it, copy it to /usr/local/bin/ so it’s in your path.
Create the directory
sudo mkdir /etc/consul.d
Optionally, run the following to create a JSON file defining a fake service running

echo '{"service": {"name": "web", "tags": ["rails"], "port": 80}}' \
    >/etc/consul.d/web.json

Step 3 Above

Run the agent on each node, changing IP accordingly.

tuxninja@consul-d415:~$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=agent-one -bind=10.73.172.110 -config-dir /etc/consul.d

Step 4 Above

tuxninja@consul-d415:~$ consul join 10.73.172.108
Successfully joined cluster by contacting 1 nodes.

Wow, simple…ok now for the examples….

Show cluster members

tuxninja@consul-dcb3:~$ consul join 10.73.172.110
Successfully joined cluster by contacting 1 nodes.

Look up DNS for a node

tuxninja@consul-dcb3:~$ dig @127.0.0.1 -p 8600 agent-one.node.consul
; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 agent-one.node.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2450
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;agent-one.node.consul.		IN	A
;; ANSWER SECTION:
agent-one.node.consul.	0	IN	A	10.73.172.110
;; Query time: 1 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Tue May 03 21:43:47 UTC 2016
;; MSG SIZE  rcvd: 76
tuxninja@consul-dcb3:~$

Lookup DNS for a service

tuxninja@consul-dcb3:~$  dig @127.0.0.1 -p 8600 web.service.consul
; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> @127.0.0.1 -p 8600 web.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55798
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;web.service.consul.		IN	A
;; ANSWER SECTION:
web.service.consul.	0	IN	A	10.73.172.110
;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Tue May 03 21:46:54 UTC 2016
;; MSG SIZE  rcvd: 70
tuxninja@consul-dcb3:~$

Query the REST API for Nodes

tuxninja@consul-dcb3:~$ curl localhost:8500/v1/catalog/nodes
[{"Node":"agent-one","Address":"10.73.172.110","TaggedAddresses":{"wan":"10.73.172.110"},"CreateIndex":3,"ModifyIndex":1311},{"Node":"agent-two","Address":"10.73.172.108","TaggedAddresses":{"wan":"10.73.172.108"},"CreateIndex":1338,"ModifyIndex":1339}

Query the REST API for Services

tuxninja@consul-dcb3:~$ curl http://localhost:8500/v1/catalog/service/web
[{"Node":"agent-one","Address":"10.73.172.110","ServiceID":"web","ServiceName":"web","ServiceTags":["rails"],"ServiceAddress":"","ServicePort":80,"ServiceEnableTagOverride":false,"CreateIndex":5,"ModifyIndex":772}

How To: curl the Openstack API’s (v3 Keystone Auth)

While Openstack provides a python client(s) for interactions….

[root@diamond ~]# source keystonerc_tuxninja
[root@diamond ~(keystone_tuxninja)]# openstack server list
+--------------------------------------+-------+--------+----------------------------------------+
| ID                                   | Name  | Status | Networks                               |
+--------------------------------------+-------+--------+----------------------------------------+
| e5b35d6a-a9ba-4714-a9e1-6361706bd047 | spin1 | ACTIVE | private_tuxlabs=10.0.0.8, 192.168.1.52 |
+--------------------------------------+-------+--------+----------------------------------------+
[root@diamond ~(keystone_tuxninja)]#

I frequently, finding myself needing to get data out of it without the pain of awk/sed’ing out the ASCII art.

Thus to quickly access the raw data, we can directly query the API’s using curl & parsing JSON instead, which is much better 🙂

Authentication

Before we can interact with the other Openstack API’s we need to authenticate to Keystone openstack’s identity service. After authenticating we receive a token to use with our subequent API requests. So step 1 we are going to create a JSON object with the required authentication details.

Create a file called ‘token-request.json’ with an object that looks like this.

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "id": "default"
                    },
                    "name": "tuxninja",
                    "password": "put_your_openstack_pass"
                }
            }
        }
    }
}

Btw, if you followed my tutorial on how to install Openstack Kilo, your authentication details for ‘admin’ is in your keystonerc_admin file.

Now we can use this file to authenticate like so:

export TOKEN=`curl -si -d @token-request.json -H "Content-type: application/json" http://localhost:35357/v3/auth/tokens | awk '/X-Subject-Token/ {print $2}'`

The token is actually returned in the header of the HTTP response, so this is why we need ‘-i’ when curling. Notice we are parsing out the token and returning the value to an environment variable $TOKEN.

Now we can include this $TOKEN and run whatever API commands we want (assuming admin privileges for the tenant/project)

Curl Commands (Numerous Examples!)

# list domains
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/domains

# create a domain
curl  -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" -d '{"domain": {"description": "--optional--", "enabled": true, "name": "dom1"}}'  http://localhost:35357/v3/domains


# list users
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users

# To create a users, create file named create_user.json file like this:

{
    "user": {
           "default_project_id": "18ed894bb8b84a5b9144c129fc754722",
            "description": "Description",
            "domain_id": "default",
            "email": "tuxninja@tuxlabs.com",
            "enabled": true,
            "name": "tuxninja",
            "password": "changeme" }
}

# then run
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users -d @create_user.json


# list images in nova
                                                                                             <tenant_id>
curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/images | python -m json.tool

# list servers (vms)

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool

# neutron networks

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

# neutron subnets

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

I sometimes pipe the output to python -m json.tool, which provides formatting for JSON. Lets take a closer look at an example.

Listing servers (vm’s)

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool
{
    "servers": [
        {
            "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047",
            "links": [
                {
                    "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "bookmark"
                }
            ],
            "name": "spin1"
        }
    ]
}
[root@diamond ~(keystone_tuxninja)]#

I only have 1 VM currently called spin1, but for the tutorials sake, if I had ten’s or hundred’s of VM’s and all I cared about was the VM name or ID, I would still need to parse this JSON object to avoid getting all this other meta-data.

My favorite command line way to do that without going full Python is using the handy JQ tool.

Here is how to use it !

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .
{
  "servers": [
    {
      "name": "spin1",
      "links": [
        {
          "rel": "self",
          "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        },
        {
          "rel": "bookmark",
          "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        }
      ],
      "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047"
    }
  ]
}
[root@diamond ~(keystone_tuxninja)]#
[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .servers[0].name -r
spin1
[root@diamond ~(keystone_tuxninja)]#

The first command just takes whatever the STDOUT from curl is and indent’s and color’s the JSON making it pretty (colors gives it +1 vs. python -m json.tool).

The second example we actually parse what were after. As you can see it is pretty simple, but jq’s query language may not be 100% intuitive at first, but I promise it is pretty easy to understand if you have ever parsed JSON before. Read up more on JQ @ https://stedolan.github.io/jq/ & check out the Openstack docs for more API commands http://developer.openstack.org/api-ref.html

Hope you enjoyed this post ! Until next time.

Installing Openstack Kilo on Centos 7

In a previous article I wrote about how to install Openstack Icehouse on CentOS 6.5 in great detail. In this article, I am going to keep verbosity to a minimum and just give you the commands ! I am hoping this will be refreshing for my audience. If you are curious however, about the what, when and why please read my previous article.

Pre-requisites

You need a machine with x86_64 architecture with at least 4 GB of memory & 2 NIC’s.
On this machine you need to install CentOS 7 as a minimal install
You should create a user with admin privileges (i.e. wheel, in my case ‘tuxninja’ was created)
Disable SELinux
1. vi /etc/sysconfig/selinux
2. SELINUX=disabled
3. save changes

Jumping Right In

Here are the commands you need to run.

sudo yum update -y
sudo yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-kilo/rdo-release-kilo-1.noarch.rpm
sudo yum install epel-release
sudo yum install -y openstack-packstack

Now at this point if you ran ‘packstack’ you would run into a bug with this message

ERROR : Error appeared during Puppet run: 192.168.1.10_prescript.pp
Error: Could not find data item CONFIG_USE_SUBNETS in any Hiera data file and no default supplied at /var/tmp/packstack/053c9a3614de4404b906141268c08f0a/manifests/192.168.1.10_prescript.pp:2 on node diamond.tuxlabs.com

The workaround for this bug is as follows

sudo rpm -e puppet
sudo rpm rpm -e hiera
curl -O https://yum.puppetlabs.com/el/7/products/x86_64/hiera-1.3.4-1.el7.noarch.rpm
sudo rpm -ivh hiera-1.3.4-1.el7.noarch.rpm
vi /etc/yum.repos.d/epel.repo
1. At the bottom of the [epel] section, after the gpgkey add a newline with: exclude=hiera*
2. Save the file
sud0 yum install -y puppet-3.6.2-3.el7.noarch
reboot
sudo rm /etc/puppet/hiera.yaml
sudo packstack –allinone

This should successfully install. Godspeed.

Networking

Now that Openstack is setup, we still have to setup our network with private & public routed networks, so we can turn this into a real multi-node setup and ssh to our hosts and let them reach the internet etc. To do this, much like my previous post you need to modify your /etc/sysconfig/network-scripts/ files to reflect this.

[tuxninja@diamond network-scripts]$ cat ifcfg-enp4s0f0
NAME="enp4s0f0"
UUID="e0c3929c-1f9b-44d1-9c59-6c8872f603bd"
DEVICE="enp4s0f0"
TYPE="OVSPort"
NM_CONTROLLED="no"
DEVICETYPE="ovs"
OVS_BRIDGE="br-ex"
BOOTPROTO="none"
ONBOOT="yes"
[tuxninja@diamond network-scripts]$ cat ifcfg-enp4s0f1
NAME=enp4s0f1
UUID=ed50b4b6-2c29-4307-bbb0-f3c923f6552a
DEVICE=enp4s0f1
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
NETWORK=10.0.0.0
IPADDR=10.0.0.1
NETMASK=255.255.255.0
[tuxninja@diamond network-scripts]$ cat ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
DNS3=192.168.1.1
ONBOOT=yes
[tuxninja@diamond network-scripts]$

Note: I deleted all the IPV6 crap, I think it messes some stuff up. When your done making the changes with your favorite editor, restart networking : sudo /etc/init.d/network restart or sudo systemctl restart network

Next go into in the Horizon Dashboard GUI and delete the demo project. See my previous article for details on how.

Back On the All-In-One Node Console

[root@diamond ~]# source keystonerc_admin 
[root@diamond ~(keystone_admin)]# neutron router-create router1
[root@diamond ~(keystone_admin)]# neutron net-create private
[root@diamond ~(keystone_admin)]# neutron subnet-create private 10.0.0.0/24 --name private_subnet
[root@diamond ~(keystone_admin)]# neutron router-interface-add router1 private_subnet
[root@diamond ~(keystone_admin)]# neutron net-create public --router:external
[root@diamond ~(keystone_admin)]# neutron subnet-create public 192.168.1.0/24 --name public_subnet --enable_dhcp=False --allocation-pool start=192.168.1.51,end=192.168.1.99 --gateway=192.168.1.1
[root@diamond ~(keystone_admin)]# neutron router-gateway-set router1 public

Next ‘reboot’ or restart all openstack services :

for service in `openstack-service list`; do openstack-service restart $service; done

Note: it appears the –full-restart flag is gone, used to work !

When logging into your dashboard located at http://192.168.1.10/dashboard at some point you might hit a bug that prevent you from logging into the Horizon dashboard see : https://bugzilla.redhat.com/show_bug.cgi?id=1218894 … the work-around for this is to clear your browser cookies.

You’re Done

That’s it. Next steps would be to create a project & new admin user, re-create the required network mappings in openstack using the above commands (modify the names to make them unique) and create your ssh key, import it, download some images, import them using glance, and create some VM’s. Also I like to delete the demo project (you can also prevent this from being created with a flag on the packstack command). Make sure you delete all default security rules and add back ICMP, TCP, and UDP allow ingress / egress rules for 0.0.0.0 aka any/any, again you can see my article on CentOS 6.5 with more specifics on how to do this. Additionally, I have an article on how to add additional compute nodes as well.

As always I can be reached for assistance @ tuxninja [at] tuxlabs.com

Happying Stacking !

How To: Add A Compute Node To Openstack Icehouse Using Packstack

Pre-requisites

This article is a continuation on the previous article I wrote on how to do a single node all-in-one (AIO) Openstack Icehouse install using Redhat’s packstack. A working Openstack AIO installation using packstack is required for this article. If you do not already have a functioning AIO install of Openstack please refer to the previous article before continuing on to this articles steps.

Preparing Our Compute Node

Much like in our previous article we first need to go through and setup our system and network properly to work with Openstack. I started with a minimal CentOS 6.5 install, and then configured the following

resolv.conf
sudoers
my network interfaces eth0(192) and eth1 (10)
1. Hostname: ruby.tuxlabs.com ( I also setup DNS for this )
2. EXT IP: 192.168.1.11
3. INT IP: 10.0.0.2
A local user + added him to wheel for sudo
I installed these handy dependencies
1. yum install –y openssh–clients
2. yum install –y yum–utils
3. yum install –y wget
4. yum install –y bind–utils
And I disabled SELinux
1. Don’t forget to reboot after

To see how I setup the above pre-requisites see the “Setting Up Our Initial System” section on the previous controller install here : http://tuxlabs.com/?p=82

Adding Our Compute Node Using PackStack

For starters we need to follow the steps in this link https://openstack.redhat.com/Adding_a_compute_node

I am including the link for reference, but you don’t have to click it as I will be listing the steps below.

On your controller node ( diamond.tuxlabs.com )

First, locate your answers file from your previous packstack all-in-one install.

[root@diamond tuxninja]# ls *answers*
packstack-answers-20140802-125113.txt
[root@diamond tuxninja]#

Edit the answers file

Change lo to eth1 (assuming that is your private 10. interface) for both CONFIG_NOVA_COMPUTE_PRIVIF & CONFIG_NOVA_NETWORK_PRIVIF

[root@diamond tuxninja]# egrep 'CONFIG_NOVA_COMPUTE_PRIVIF|CONFIG_NOVA_NETWORK_PRIVIF' packstack-answers-20140802-125113.txt
CONFIG_NOVA_COMPUTE_PRIVIF=eth1
CONFIG_NOVA_NETWORK_PRIVIF=eth1
[root@diamond tuxninja]#

Change CONFIG_COMPUTE_HOSTS to the ip address of the compute node you want to add. In our case ‘192.168.1.11’. Additionally, validate the ip address for CONFIG_NETWORK_HOSTS is your controller’s ip since you do not run a separate network node.

[root@diamond tuxninja]# egrep 'CONFIG_COMPUTE_HOSTS|CONFIG_NETWORK_HOSTS' packstack-answers-20140802-125113.txt
CONFIG_COMPUTE_HOSTS=192.168.1.11
CONFIG_NETWORK_HOSTS=192.168.1.10
[root@diamond tuxninja]#

That’s it. Now run packstack again on the controller

[tuxninja@diamond yum.repos.d]$ sudo packstack --answer-file=packstack-answers-20140802-125113.txt

When that completes, ssh into or switch terminals over to your compute node you just added.

On the compute node ( ruby.tuxlabs.com )

Validate that the relevant openstack compute services are running

[root@ruby ~]# openstack-status
== Nova services ==
openstack-nova-api:                     dead      (disabled on boot)
openstack-nova-compute:                 active
openstack-nova-network:                 dead      (disabled on boot)
openstack-nova-scheduler:               dead      (disabled on boot)
== neutron services ==
neutron-server:                         inactive  (disabled on boot)
neutron-dhcp-agent:                     inactive  (disabled on boot)
neutron-l3-agent:                       inactive  (disabled on boot)
neutron-metadata-agent:                 inactive  (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              active
== Ceilometer services ==
openstack-ceilometer-api:               dead      (disabled on boot)
openstack-ceilometer-central:           dead      (disabled on boot)
openstack-ceilometer-compute:           active
openstack-ceilometer-collector:         dead      (disabled on boot)
== Support services ==
libvirtd:                               active
openvswitch:                            active
messagebus:                             active
Warning novarc not sourced
[root@ruby ~]#

Back on the controller ( diamond.tuxlabs.com )

We should now be able to validate that ruby.tuxlabs.com has been added as a compute node hypervisor.

[tuxninja@diamond ~]$ sudo -s
[root@diamond tuxninja]# source keystonerc_admin
[root@diamond tuxninja(keystone_admin)]# nova hypervisor-list
+----+---------------------+
| ID | Hypervisor hostname |
+----+---------------------+
| 1  | diamond.tuxlabs.com |
| 2  | ruby.tuxlabs.com    |
+----+---------------------+
[root@diamond tuxninja(keystone_admin)]# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:34
nova-conductor   diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:35
nova-scheduler   diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:27
nova-compute     diamond.tuxlabs.com                  nova             enabled    :-)   2014-10-12 20:48:32
nova-cert        diamond.tuxlabs.com                  internal         enabled    :-)   2014-10-12 20:48:31
nova-compute     ruby.tuxlabs.com                     nova             enabled    :-)   2014-10-12 20:48:35
[root@diamond tuxninja(keystone_admin)]#

Additionally, you can verify it in the Openstack Dashboard

Next we are going to try to boot an instance using the new ruby.tuxlabs.com hypervisor. To do this we will need a few pieces of information. First let’s get our OS images list.

[root@diamond tuxninja(keystone_admin)]# glance image-list
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+
| 0b3f2474-73cc-4df2-ad0e-fdb7a7f7c8a1 | cirros              | qcow2       | bare             | 13147648  | active |
| 737a0060-6e80-415c-b66b-a20893d9888b | Fedora 6.4          | qcow2       | bare             | 210829312 | active |
| 952ac512-19da-47a7-81a4-cfede18c7f45 | ubuntu-server-12.04 | qcow2       | bare             | 260964864 | active |
+--------------------------------------+---------------------+-------------+------------------+-----------+--------+
[root@diamond tuxninja(keystone_admin)]#

Great, now we need the ID of our private network

[root@diamond tuxninja(keystone_admin)]# neutron net-show private
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | d1a89c10-0ae2-43f0-8cf2-f02c20e19618 |
| name                      | private                              |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 10                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | b8760f9b-3c0a-47c7-a5af-9cb533242f5b |
| tenant_id                 | 7bdf35c08112447b8d2d78cdbbbcfa09     |
+---------------------------+--------------------------------------+
[root@diamond tuxninja(keystone_admin)]#

Ok now we are ready to proceed with the nova boot command.

[root@diamond tuxninja(keystone_admin)]#  nova boot --flavor m1.small --image 'ubuntu-server-12.04' --key-name cloud --nic net-id=d1a89c10-0ae2-43f0-8cf2-f02c20e19618 --hint force_hosts=ruby.tuxlabs.com test
+--------------------------------------+------------------------------------------------------------+
| Property                             | Value                                                      |
+--------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                       |
| OS-EXT-SRV-ATTR:host                 | -                                                          |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000019                                          |
| OS-EXT-STS:power_state               | 0                                                          |
| OS-EXT-STS:task_state                | scheduling                                                 |
| OS-EXT-STS:vm_state                  | building                                                   |
| OS-SRV-USG:launched_at               | -                                                          |
| OS-SRV-USG:terminated_at             | -                                                          |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| adminPass                            | XHUumC5YbE3J                                               |
| config_drive                         |                                                            |
| created                              | 2014-10-12T20:59:47Z                                       |
| flavor                               | m1.small (2)                                               |
| hostId                               |                                                            |
| id                                   | f7b9e8bb-df45-4b94-a896-5600f47c269b                       |
| image                                | ubuntu-server-12.04 (952ac512-19da-47a7-81a4-cfede18c7f45) |
| key_name                             | cloud                                                      |
| metadata                             | {}                                                         |
| name                                 | test                                                       |
| os-extended-volumes:volumes_attached | []                                                         |
| progress                             | 0                                                          |
| security_groups                      | default                                                    |
| status                               | BUILD                                                      |
| tenant_id                            | 7bdf35c08112447b8d2d78cdbbbcfa09                           |
| updated                              | 2014-10-12T20:59:47Z                                       |
| user_id                              | 6bb8fcf3ce9446838e50a6b98fbb5afe                           |
+--------------------------------------+------------------------------------------------------------+
[root@diamond tuxninja(keystone_admin)]#

Fantastic. That command should look familiar from our previous tutorial it is the standard command for launching new VM instances using the command line, with one exception ‘–hint force_hosts=ruby.tuxlabs.com’ this part of the command line forces the scheduler to use ruby.tuxlabs.com as it’s hypervisor.

Once the VM is building we can validate that it is on the right hypervisor like so.

[root@diamond tuxninja(keystone_admin)]# nova hypervisor-servers ruby.tuxlabs.com
+--------------------------------------+-------------------+---------------+---------------------+
| ID                                   | Name              | Hypervisor ID | Hypervisor Hostname |
+--------------------------------------+-------------------+---------------+---------------------+
| f7b9e8bb-df45-4b94-a896-5600f47c269b | instance-00000019 | 2             | ruby.tuxlabs.com    |
+--------------------------------------+-------------------+---------------+---------------------+
[root@diamond tuxninja(keystone_admin)]# nova hypervisor-servers diamond.tuxlabs.com
+--------------------------------------+-------------------+---------------+---------------------+
| ID                                   | Name              | Hypervisor ID | Hypervisor Hostname |
+--------------------------------------+-------------------+---------------+---------------------+
| a4c67465-d7ef-42b6-9c2a-439f3b13e841 | instance-00000017 | 1             | diamond.tuxlabs.com |
| 0c34028d-dfb6-4fdf-b9f7-daade66f2107 | instance-00000018 | 1             | diamond.tuxlabs.com |
+--------------------------------------+-------------------+---------------+---------------------+
[root@diamond tuxninja(keystone_admin)]#

You can see from the output above I have 2 VM’s on my existing controller ‘diamond.tuxlabs.com’ and the newly created instance is on ‘ruby.tuxlabs.com’ as instructed, awesome.

Now that you are sure you setup your compute node correctly, and can boot a VM on a specific hypervisor via command line, you might be wondering how this works using the GUI. The answer is a little differently 🙂

The Openstack Nova Scheduler

The Nova Scheduler in Openstack is responsible for determining, which compute node a VM should be created on. If you are familiar with VMware this is like DRS, except it only happens on initial creation, there is no rebalancing that happens as resources are consumed overtime. Using the Openstack Dashboard GUI I am unable to tell nova to boot off a specific hypervisor, to do that I have to use the command line above (if someone knows of a way to do this using the GUI let me know, I have a feeling if it is not added already, they will add the ability to send a hint to nova from the GUI in a later version). In theory you can trust the nova-scheduler service to automatically balance the usage of compute resources (CPU, Memory, Disk etc) based on it’s default configuration. However, if you want to ensure that certain VM’s live on certain hypervisors you will want to use the command line above. For more information on how the scheduler works see : http://cloudarchitectmusings.com/2013/06/26/openstack-for-vmware-admins-nova-compute-with-vsphere-part-2/

The End

That is all for now, hopefully this tutorial was helpful and accurately assisted you in expanding your Openstack compute resources & knowledge of Openstack. Until next time !

How To: Install Openstack Icehouse on CentOS 6.5 Using Packstack

A brief introduction of Openstack + My thoughts

Openstack is open-source software for building clouds. It was created in 2010 by people from Rackspace & NASA, but is currently managed by the non-profit Openstack Foundation, which includes members from the who’s who of the technology sector that have joined forces to continue to invest & develop Openstack (written in Python) for fun and profit( well not so much ). It is definitely fun though :-). In fact I have been having nothing, but a blast since the moment I meant Openstack in 2012. However, admittedly the first time I worked on Openstack it was on the Essex release and I felt it had a ways to go before it was ready for prime time. At that time it was not only hard to install, but most of the people around it were hard core Pythonista’s (Python developers), that were rapidly trying to a mature Openstack & it’s ecosystem that just wasn’t production ready yet. And while the latest release of Openstack ‘Icehouse‘ (which I am covering in this blog) is leaps and bounds better, it can still be quite a PITA(Pain In The Ass) for first timers. In fact I still struggle with the idea of running Openstack in production because it requires an incredible amount of resources, engineering skill & organizational persistence to do so. Companies willing to embark on this journey thus far have included Mercado Libre, Dell, HP, Redhat, Canonical (the Ubunut guys), eBay, PayPal and Symantec among many others. These companies were required to beef up their engineering staff & allocate large amounts of resources just to take on this challenge. In addition their commitments were probably tested over and over again and their leadership had to respond with tremendous faith when timelines began to slip. However, organizations that complete this journey recognize that the return at the end of the tunnel is a compounding return. For starters you aren’t locked into to a vendor like VmWare or AWS (Amazon Web Services) who are both incredibly expensive, but the compounding return comes from skills it will build internally and the culture that will be a bi-product.

Coming here to Tuxlabs means you have personally accepted this mission to learn Openstack and you are in need of some guidance don’t worry you are in good hands. I will show you the light, which can be hard to see through the darkness if you try to take on the documentation all by yourself : http://docs.openstack.org/

So sit back, relax, read, and type exactly as I do. In the end you will have a perfectly functioning Openstack Icehouse cloud and if you decide to bring this into your organization remember with great power comes great responsibility…

You will change everyone in your organization from a POSA (Plain Old Sys Admin’s) to Cloud Engineers and Architects jumping into the Openstack Python code base at the sniff of an issue.

Devstack

Devstack is a shell script used to quickly and easily deploy an all-in-one install of Openstack on any machine or VM for the purposes of trying, testing, and developing on Openstack. It is extremely easy to install, use and get going, if you have not used Devstack already I recommend trying it first because it is a quick and easy way to get a learning win + it will help you decide what you want to learn more about and whether or not Openstack is what you are looking for. However, you do not want to run Devstack in production for starters because production deployments should be multi-node setup’s not all Cloud services deployed on one machine like Devstack does.

The Book

My favorite Openstack book to date is by far is the Openstack Cloud Computing Cookbook written by Kevin Jackson. This book is so great because gets straight to business giving you the commands & understanding needing to get Openstack up and running quickly unlike most other book that bore you with unnecessary details. The first time I installed Openstack Essex I used this book and it worked like a charm, however, that was on Ubuntu, which this book was written for. If you want to install Openstack on Redhat you can do so from scratch or you can use RDO.

RDO

RDO technically doesn’t stand for anything ! http://openstack.redhat.com/Frequently_Asked_Questions#What_does_RDO_stand_for.3F, but I don’t really buy that and have found some people expanding the acronym to the Redhat Distribution of Openstack which I think sounds incredibly fitting. RDO has a website dedicated to a community of people running Openstack on Redhat, CentOS and Fedora. Because Redhat is still the leader in production Linux deployments in the enterprises of the world, I chose to use it in my own lab environment & for the purposes of this tutorial. However, because I cannot afford the license cost of Redhat Enterprise Linux I am using the free community release of RHEL known as CentOS or the Community Enterprise Operating System. For the purposes of this tutorial we will be installing CentOS 6.5 with a minimal install on a bare metal system and then installing Packstack according to the instructions on the RDO website. Then we will go a step further.

Getting Started

A production Openstack deployment has a minimum of 3 nodes one for the Controller, Network, and Compute nodes and that would not cover a Highly Available deployment where you would need the capability to failover functions to standby nodes when there was an outage or maintenance needed. However, for the purposes of this article I will be showing you what is called an all-in-one install where the Controller, Network, and Compute functions all live on the same node and in future articles we will expand on this knowledge to build a production capable multi-node deployment of Openstack.

Before We Begin

You will need…

A system that you have root access to, to install Openstack on. I am using an SGI Rackable with 16GB of RAM and 4.5TB of usable disk ( in a hardware RAID 10). Openstack does not require a lot of resources, but if you’re going to spawn a lot VM’s you need a lot of Cores (vCPU’s), Memory and Storage. See this link for more info: http://docs.openstack.org/grizzly/openstack-compute/install/yum/content/compute-system-requirements.html
That system should have dual NIC’s, although I think it is possible to use virtual interfaces for a lab environment if you have to.
To configure the hostname of your controller in DNS mine is diamond.tuxlabs.com, but most people go with controller.yourdomain.com (if you don’t have or use DNS, just make sure you configure your hosts file /etc/hosts with the information)
The system you use should have CentOS 6.5 installed with a Minimal Install using the defaults no extra’s.
Your brain, keyboard, fingers and a fresh beer and possibly an ice chest with more beer depending on how far the fridge is.

Assumptions

Our home network uses 192.168.1.0/24 and has access to the internet.
It has DHCP enabled, but only for 192.168.1.150-199.
We are not using 10.0.0.0/24 for anything so we can use it for our private network in Openstack.

If these are true we should be able to follow my examples exactly, but if you want to change your networks you will have to substitute them as needed.

Setting Up Our Initial System

First become root or login as root, then…configure the following configuration files to match.

Resolve.conf

This assumes our gateway device @ 192.168.1.1 runs DNS (like the Linksys’s do). If 192.168.1.1 doesn’t run DNS do not add it to resolve.conf.

[root@diamond tuxninja]# cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 192.168.1.1
[root@diamond tuxninja]#

Sudoers

Make sure wheel is uncommented.

[root@diamond tuxninja]# cat /etc/sudoers | grep '%wheel' | grep -v '#'
%wheel    ALL=(ALL)    NOPASSWD: ALL
[root@diamond tuxninja]#

Setup Your Primary Network Interface

[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:15:17:65:F9:98
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
NAMESERVER=192.168.1.1
[root@diamond tuxninja]#

Restart Networking

/etc/init.d/network restart

Next Add A Local User (Cause logging in as root is lame)

adduser tuxninja
passwd tuxninja

Set your password to something good, then add the user to the wheel group.

usermod -G tuxninja,wheel tuxninja

Installing Dependencies

Next we need to install some basic dependencies so things like scp, yum, wget and nslookup work.

yum install -y openssh-clients
yum install -y yum-utils
yum install -y wget
yum install -y bind-utils

Turn Off SELinux

Yes security is important, but unfortunately to install Openstack without a headache you need to turn off SELinux.

[root@diamond tuxninja]# cat /etc/sysconfig/selinux  | grep SELINU | grep -v '#'
SELINUX=disabled
SELINUXTYPE=targeted
[root@diamond tuxninja]# reboot

Don’t forget to reboot or the setting will not take effect.

Configuring Networking

There are many ways to configure Openstack networking and it can be quite complicated, so we are going to use what I consider the simplest method using Openvswitch.

Before we begin, make sure you can ping yahoo.com before we go mucking with the configs. We are going to change how your primary network interface is configured and then configure two additional interfaces. Do your best to match these configurations exact, making a mistake here might cause Openvswitch to barf on itself and that would not be fun to troubleshoot.

[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:15:17:65:F9:98
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-ex
ONBOOT=yes
[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
HWADDR=00:15:17:65:F9:99
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
NETWORK=10.0.0.0
IPADDR=10.0.0.1
NETMASK=255.255.255.0
[root@diamond tuxninja]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
ONBOOT=yes
[root@diamond tuxninja]# /etc/init.d/network restart

What Is This Doing ?

The above configuration(s) is setting eth0 as an OVSPort that is bridged to the interface br-ex. eth1 is configured normally for our private 10 network, and br-ex is configured as a OVSBridge with our actual IP information from / for eth0.

After Restarting Networking ifconfig Output Should Look Like This

[root@diamond tuxninja]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:15:17:65:F9:98 
          inet6 addr: fe80::215:17ff:fe65:f998/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:322092 errors:0 dropped:0 overruns:0 frame:0
          TX packets:187420 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:436592198 (416.3 MiB)  TX bytes:15605772 (14.8 MiB)
          Interrupt:18 Memory:b8820000-b8840000

[root@diamond tuxninja]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:15:17:65:F9:99 
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::215:17ff:fe65:f999/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:552 (552.0 b)
          Interrupt:19 Memory:b8800000-b8820000

[root@diamond tuxninja]# ifconfig br-ex
br-ex     Link encap:Ethernet  HWaddr 00:15:17:65:F9:98 
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a8db:16ff:fed6:f4c4/64 Scope:Link
          UP BROADCAST RUNNING  MTU:1500  Metric:1
          RX packets:293 errors:0 dropped:0 overruns:0 frame:0
          TX packets:151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:43799 (42.7 KiB)  TX bytes:15786 (15.4 KiB)

[root@diamond tuxninja]#

For more information on Openstack networking here are some references that I used.

https://www.youtube.com/watch?v=afImoFeuDnY
http://openstack.redhat.com/Neutron_with_existing_external_network
http://openstack.redhat.com/forum/discussion/780/confused-with-neutron-and-external-network-access/p1
http://openstack.redhat.com/Networking_in_too_much_detail
http://docs.openstack.org/grizzly/openstack-network/admin/content/

Update Packages

Finally since we are running a minimum install, let’s update all of our packages before continuing.

[root@diamond tuxninja]# sudo yum update -y

Installing Openstack

Finally we are ready to install Openstack Using Packstack.

[root@diamond tuxninja]# sudo yum install -y http://rdo.fedorapeople.org/rdo-release.rpm
[root@diamond tuxninja]# sudo yum install -y openstack-packstack
[root@diamond tuxninja]# packstack --allinone --provision-all-in-one-ovs-bridge=n --provision-demo=n

The first command adds the required RDO repo to yum install packstack. The second downloads the openstack-packstack package. And the third and final command is magic. This command installs packstack. We are telling it to install all openstack components to one machine the –provision-all-in-one-ovs-bridge=n tells packstack we are going to be using a single node (although I am still not entirely sure this flag is absolutely necessary) and the final flag tells packstack not to deploy the demo project, because if you do you have to end up deleting it before you can delete the network information and re-create it correctly.

Packstack will take about 10 minutes to run. It uses puppet to deploy Openstack and it’s required configurations. In my experience it works pretty well, however if you do need to re-install packstack there isn’t an automated uninstall script included. So someone created one and here it is in case you need it.

[root@diamond tuxninja]# cat packstack_uninstall.sh
#!/bin/bash

# Warning! Dangerous step! Destroys VMs
for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do
    virsh destroy $x ;
    virsh undefine $x ;
done ;

# Warning! Dangerous step! Removes lots of packages
yum remove -y nrpe "*nagios*" puppet "*ntp*" "*openstack*" \
"*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" \
mysql mysql-server httpd "*memcache*" scsi-target-utils \
iscsi-initiator-utils perl-DBI perl-DBD-MySQL ;

# Warning! Dangerous step! Deletes local application data
rm -rf /etc/nagios /etc/yum.repos.d/packstack_* /root/.my.cnf \
/var/lib/mysql/ /var/lib/glance /var/lib/nova /etc/nova /etc/swift \
/srv/node/device*/* /var/lib/cinder/ /etc/rsync.d/frag* \
/var/cache/swift /var/log/keystone /var/log/cinder/ /var/log/nova/ \
/var/log/httpd /var/log/glance/ /var/log/nagios/ /var/log/quantum/ ;

umount /srv/node/device* ;
killall -9 dnsmasq tgtd httpd ;

vgremove -f cinder-volumes ;
losetup -a | sed -e 's/:.*//g' | xargs losetup -d ;
find /etc/pki/tls -name "ssl_ps*" | xargs rm -rf ;
for x in $(df | grep "/lib/" | sed -e 's/.* //g') ; do
    umount $x ;
done

Don’t forget to chmod +x to that bad boy to make it executable so you can run it, when you are ready to uninstall.

Now then, getting back to our install. If Packstack was successful you should see something like this.

 **** Installation completed successfully ******


Additional information:
 * A new answerfile was created in: /root/packstack-answers-20140802-125113.txt
 * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components.
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.1.10. To use the command line tools you need to source the file.
 * To access the OpenStack Dashboard browse to http://192.168.1.10/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * To use Nagios, browse to http://192.168.1.10/nagios username: nagiosadmin, password: 4bf73f1116544975
 * The installation log file is available at: /var/tmp/packstack/20140802-125113-RzCDrE/openstack-setup.log
 * The generated manifests are available at: /var/tmp/packstack/20140802-125113-RzCDrE/manifests
[root@diamond tuxninja]#

Check Out Our Keystonerc_admin File

Just take note of what environment variables it sets. To login to our GUI we need these credentials and to use any command line functionality we have to source this file in our shell.

[root@diamond tuxninja]# cat ~/keystonerc_admin
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PASSWORD=a57c83f56ccc41f5
export OS_AUTH_URL=http://192.168.1.10:5000/v2.0/
export PS1='[\u@\h \W(keystone_admin)]\$ '
[root@diamond tuxninja]#

Configuring Openstack Networking

Because we did not deploy the demo project, the default network configuration in Openstack should not exist. If it does we can login to our GUI @ http://diamond.tuxlabs.com/dashboard by sourcing the credentials above in our keystonerc_admin and delete the network configuration under

admin –> routers
admin –> networks

Then you are ready to re-create your network. You could use the GUI to do this, which is pretty straight forward…but since GUI’s are dirty we are going to use the command line.

In order to use any openstack command line utilities you must first source the keystonerc_admin file so the required environment variables are set in your shell. Like so…

[root@diamond ~]# source keystonerc_admin 
[root@diamond ~(keystone_admin)]#

Seeing (keystone_admin) in your prompt means you are ready to run commands and here you go

neutron router-create router1
neutron net-create private
neutron subnet-create private 10.0.0.0/24 --name private_subnet
neutron router-interface-add router1 private_subnet
neutron net-create public --router:external=True
neutron subnet-create public 192.168.1.0/24 --name public_subnet --enable_dhcp=False --allocation-pool start=192.168.1.51,end=192.168.1.99 --gateway=192.168.1.1
neutron router-gateway-set router1 public

Copy & pasting this will spit out a lot of messages showing the result set of each command.

The important thing to realize is that your public network has to be configured with an external router. This effectively NAT’s the 10 & 192 networks to make internet access available to your VM’s. When this is configured correctly your Network Topology diagram under the dirty GUI should resemble this.

The router_gateway always shows DOWN. No idea why, but someone else said it’s a bug 😉

Next under your project (the admin project) you have to configure your Security Groups under Compute—>Access & Security—>Security Groups. Once there click Manage Rules for the default security group. Delete what’s there. Add Ingress/Egress for ALL ICMP, ALL TCP, and ALL UDP accepting all other defaults on the form. This will open up your firewall completely.

Time To Restart Openstack

Finally we need to restart Openstack, validating that openvswitch-agent starts.

[root@diamond tuxninja(keystone_admin)]# /etc/init.d/network restart;openstack-service --full-restart;openstack-status | grep 'openvswitch-agent' 
neutron-openvswitch-agent:              active

Note: You can use openstack-service –full-restart to restart all openstack services.

If openvswitch-agent does not start run back through the above steps and / or consult the references or email me tuxninja [at] tuxlabs.com. This can be a real pain to figure out, but I finally have it down pat.

If openvswitch-agent is active we are good to go. Next we have to create an SSH Key.

Creating Our Cloud’s SSH Key

Our Openstack Cloud has no authentication system for our virtual machines by default. Eventually you could configure an LDAP server, and configure your images or configure in Puppet the required pam configurations to use LDAP for authentication by your VM’s, but for now Openstack allows for a post configuration step after building VM’s where it will add your SSH key. It does this by using the metadata service which cloud images will look for, if they can reach the metadata service @ http://169.254.169.254 then they can copy down the public key and install it on the VM for you allowing you to login to that VM using the VM’s operating system default account (i.e. ubuntu, fedora). Now before I continue there are alternatives such as using guestfish to modify an image’s configuration or using a post install cloud-init configuration to specific a password to an account, but the cleanest and simplest way is to use the metadata service after generating & importing the public key into openstack and assigning it to our VM guest.

To create the key…

[root@diamond tuxninja]# ssh-keygen -t rsa -f cloud.key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in cloud.key.
Your public key has been saved in cloud.key.pub.
The key fingerprint is:
59:e0:b3:0f:c9:d0:8a:67:ba:86:39:0e:fe:b2:93:ce root@diamond.tuxlabs.com
The key's randomart image is:
+--[ RSA 2048]----+
|        .        |
|       o .       |
|      . + .      |
|     . + *       |
|    . + S        |
|     +   o       |
|. .o.     .      |
|o=+ ..           |
|.E*+.            |
+-----------------+
[root@diamond tuxninja]#

I don’t enter a password to use password-less SSH, cause again this is a lab environment and typing passwords sucks.

Launching A VM

Next you want to launch a VM that we can use to import our SSH key into. Login to the dashboard @ http://diamond.tuxlabs.com/dashboard again using the keystonerc_admin credentials provided. Once inside navigate to Project—>Compute—>Instances and click the Launch Instance button in the upper right corner. You will be presented with a form. Here are the screenshots to guide you through.

Fill out the details tab…

Click the Access & Security Tab…

Then Click The + …

Now in order to fill this out we have to copy and paste the contents (bold part) of the public key we created.

[root@diamond tuxninja]# cat cloud.key.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzs46kFxia6q9gfwmmszFHGFSk5UAQFjLhEOE08I/2F7+Qwh0AFBuO99c/OCITr2HvVfDc7DF5JlRQGohTA9aowtINWyEUvEKSPG8gEtL95GZ5EZH/3pqWFaklIfoP3fbgKPrpPSGarYQY5UH+qm419x+A3tAkmumdd3BtQHLRF1qq9Ui69IQ383MctuOu+sF9Ws6+NT1+MbND++Qp2VNli/+cpZTnv4g9i5C2mXbUQQJghEoUvIXiBh+r27kVOlJ6L3TeT8WIwD1N916ZRgHrfbAEQ+108Zdyc5/TXYRbbWs4WO9PVcTp+6zcLOXyds0sFMyu53rWOS22iEFL3XGRQ== root@diamond.tuxlabs.com
[root@diamond tuxninja]#

Then Click Networking and add the private network…

Then click launch….and you should see something like this…

See the status spawning ? When the virtual machine is done being built you will see status changed to ‘Running’, you can watch the machine boot, by clicking the name of the instance and then going to the Log or Console tab.

This is great our VM is running and it has an IP address configured ! But that IP address is only used for Openstack communication, so we still need to associate a floating IP to this system so we can SSH to it. Click on the More dropdown and select Associate Floating IP.

Then click the + …

Click Allocate IP…

And then click associate…

Now you should see your VM has an internal IP (a 10 dot address) and an External IP on the 192.168.1.0/24 network. Now we can SSH to our VM.

We didn’t mention it earlier, but Openstack comes with only one Linux image by default, it’s called CirrOS and it’s just a tiny-minimal cloud image for testing. The login to this operating system is cirros / cubswin:) … which is visible from the console log of the machine once fully booted. Now you could SSH into the machine on 192.168.1.54 using that login and password like so…

[root@diamond tuxninja]# ssh cirros@192.168.1.54
The authenticity of host '192.168.1.54 (192.168.1.54)' can't be established.
RSA key fingerprint is d1:19:89:1d:f0:4b:20:27:a3:7c:99:0d:75:fe:d0:35.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.54' (RSA) to the list of known hosts.
cirros@192.168.1.54's password: 
$

But that is lame ! We imported our SSH key remember ? So how do we use that ? Like this..

[root@diamond tuxninja]# ssh -i cloud.key cirros@192.168.1.54
$

Wow, that is cool.

Now what ? Let’s add some more images.

Installing More Operating System Images

Ok, Openstack is installed, Networking is working, Metadata service and our keys are working, we are happy campers, but to make this Cloud useful we are going to need some real Linux images. There is two ways to install images using glance (the openstack image service).

In the first method we download the image using wget & then we run the proper glance command. Here is this approach in action.

Don’t forget to source the rc file if you have not already

[root@diamond ~]# source keystonerc_admin
[root@diamond ~(keystone_admin)]# wget http://download.fedoraproject.org/pub/fedora/linux/updates/20/Images/x86_64/Fedora-x86_64-20-20140407-sda.qcow2
[root@diamond ~(keystone_admin)]# glance image-create --name='Fedora 6.4' --is-public=true \ --container-format=bare --disk-format=qcow2 < Fedora-x86_64-20-20140407-sda.qcow2

Alternatively, we could do the same thing in one command !

[root@diamond ~(keystone_admin)]# glance image-create --name ubuntu-server-12.04 --is-public true --container-format bare --disk-format qcow2 --copy-from http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

Remember whenever you run openstack commands you are going the status or result set back from the command.

Once you have installed the Fedora and Ubuntu images you should have a real cloud on your hands. Now you can do things like expand your setup for multi-node or get LBaaS working for load balancing requests to your web servers for example. Or if you want to be amazeballs you could install things like Puppet inside of Docker, or Cloud Foundry and build your PaaS (Platform As A Service)!

Final Comments

Openstack can be tricky even for an experienced Sys Admin. While learning Openstack I found it difficult to find tutorials on exactly the setup I was looking for mainly in terms of how the network was being configured, and this often made me second guess myself when I would run into an issue as I learned Openstack. I wanted to write this article to give back and help to educate my brothers and students of life long learning as you embark on your Openstack adventure, Godspeed. Here are some commands of note to help you along your way.

How to launch a VM from command line

[root@diamond ~(keystone_admin)]# nova boot --flavor m1.small --image 'Fedora 6.4' --security-groups default --key-name openstack --nic net-id=975a58eb-d3cb-4c5b-956b-ed1bdb0820d2 fed1

How to run command on a network namespace (VM Instance)

[root@diamond tuxninja]# ip netns exec qrouter-71064321-eb87-4d28-85ee-05f6ddc75968 iptables -L -t nat | grep 169
REDIRECT tcp -- anywhere= 169.254.169.254 tcp dpt:http redir ports 9697
[root@diamond tuxninja]# ip netns exec `ip netns list | grep qrouter` netstat -anpt

How to restart only neutron services

[root@diamond tuxninja]# cd /etc/init.d/; for i in $(ls -1 neutron-* | cut -d \. -f 1); do sudo service $i restart; done

Or a simpler method would be…

[root@diamond tuxninja]# openstack-service restart neutron
Stopping neutron-dhcp-agent: [FAILED]
Starting neutron-dhcp-agent: [  OK  ]
Stopping neutron-l3-agent: [  OK  ]
Starting neutron-l3-agent: [  OK  ]
Stopping neutron-metadata-agent: [  OK  ]
Starting neutron-metadata-agent: [  OK  ]
Stopping neutron-openvswitch-agent: [  OK  ]
Starting neutron-openvswitch-agent: [  OK  ]
Stopping neutron: [  OK  ]
Starting neutron: [  OK  ]
[root@diamond tuxninja]#

How to add an account / change a password using cloud-init

Stick this under Post Creation Customized Script to

#cloud-config
password: mysecret
chpasswd: { expire: False }
ssh_pwauth: True

How to edit the password file of an image

[root@diamond tuxninja]# guestfish -a disk.img -i vi /etc/passwd

How to see the status of openstack services

[root@diamond tuxninja]# openstack-status

How to check your log files for more clues

you can do this for every openstack service, but below I am just showing the two most popular to troubleshoot for getting your cloud up and going, nova and neutron aka controller, compute and networking logs.

[root@diamond tuxninja]# tail /var/log/nova/*.log
[root@diamond tuxninja]# tail /var/log/neutron/*.log

Want to change which partition Openstack uses on local disk (ephemeral storage) to deploy VM’s ?

If the majority of your ephemeral (local) disk is under a different partition other than /var… For example, mine was under /home, then you need to change your state path and restart openstack services. Don’t forget to copy any existing files in /var/lib/nova/ to the new location.

[root@diamond ~(keystone_admin)]# grep state_path /etc/nova/nova.conf | grep -v '#' | grep -v ^$ 
state_path=/home/openstack/var/lib/nova
[root@diamond ~(keystone_admin)]#

Checking your openvswitch configuration

[root@diamond ~(keystone_admin)]# ovs-vsctl show
ee001583-903b-4397-8805-40c18b528d2c
    Bridge br-int
        Port "qvo0cadc326-6b"
            tag: 3
            Interface "qvo0cadc326-6b"
        Port "tap86f8ac73-41"
            tag: 3
            Interface "tap86f8ac73-41"
                type: internal
        Port "qr-11baa878-fb"
            tag: 3
            Interface "qr-11baa878-fb"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
        Port "qvo6673d677-ed"
            tag: 3
            Interface "qvo6673d677-ed"
        Port "qvoc8a2b994-a1"
            tag: 3
            Interface "qvoc8a2b994-a1"
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "eth0"
            Interface "eth0"
        Port "qg-82e238cc-3d"
            Interface "qg-82e238cc-3d"
                type: internal
    ovs_version: "1.11.0"
[root@diamond ~(keystone_admin)]#

The Final Gem

The Most Popular Issue I ran Into When Installing Openstack

Looks like this in the console..

This is the generic error for I didn’t have resources to create the VM, which can happen for a multitude of reasons…Like for example, the Openvswitch-agent not running.

[root@diamond metadata]# openstack-status
...
neutron-openvswitch-agent:              dead

But why is Openvswitch Agent Failing ?

[root@diamond metadata]# tail /var/log/neutron/openvswitch-agent.log
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2014-07-27 12:47:31.072 17288 TRACE neutron     sys.exit(main())
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1394, in main
2014-07-27 12:47:31.072 17288 TRACE neutron     agent = OVSNeutronAgent(**agent_config)
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 216, in __init__
2014-07-27 12:47:31.072 17288 TRACE neutron     self.setup_tunnel_br(tun_br)
2014-07-27 12:47:31.072 17288 TRACE neutron   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 707, in setup_tunnel_br
2014-07-27 12:47:31.072 17288 TRACE neutron     if int(self.patch_tun_ofport) < 0 or int(self.patch_int_ofport) < 0:
2014-07-27 12:47:31.072 17288 TRACE neutron TypeError: int() argument must be a string or a number, not 'NoneType'
2014-07-27 12:47:31.072 17288 TRACE neutron

Additionally, I have seen different errors presented referencing vif.py

The fix for this could be multiple different things. What you should do first is verify your network interfaces are configured 100% correctly. Compare your ovs-vsctl output to mine above… Look different ? Shorter, stuff is missing ? Ok this error usually means Openstack is having difficulty inserting port’s into Openvswitch for whatever reason. What I found is the most common error is that you did not restart networking, before restarting openstack networking after making a configuration change to the interfaces. So to resolve this try…

[root@diamond metadata]# /etc/init.d/network restart
Shutting down interface br-ex:                             [  OK  ]
Shutting down interface br-int:                            [  OK  ]
Shutting down interface br-tun:                            [  OK  ]
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface br-ex:  Determining if ip address 192.168.1.10 is already in use for device br-ex...
                                                           [  OK  ]
Bringing up interface br-int:                              [  OK  ]
Bringing up interface br-tun:                              [  OK  ]
Bringing up interface eth0:  RTNETLINK answers: File exists
                                                           [  OK  ]
Bringing up interface eth1:  Determining if ip address 10.0.0.1 is already in use for device eth1...
                                                           [  OK  ]
[root@diamond metadata]# openstack-service --full-restart

Stopping openstack-ceilometer-alarm-evaluator:             [  OK  ]
Starting openstack-ceilometer-alarm-evaluator:             [  OK  ]
Stopping openstack-ceilometer-alarm-notifier:              [  OK  ]
Starting openstack-ceilometer-alarm-notifier:              [  OK  ]
Stopping openstack-ceilometer-api:                         [  OK  ]
Starting openstack-ceilometer-api:                         [  OK  ]
Stopping openstack-ceilometer-central:                     [  OK  ]
Starting openstack-ceilometer-central:                     [  OK  ]
Stopping openstack-ceilometer-collector:                   [  OK  ]
Starting openstack-ceilometer-collector:                   [  OK  ]
Stopping openstack-ceilometer-compute:                     [  OK  ]
Starting openstack-ceilometer-compute:                     [  OK  ]
Stopping openstack-ceilometer-agent-notification:          [  OK  ]
Starting openstack-ceilometer-agent-notification:          [  OK  ]
Stopping openstack-cinder-api:                             [  OK  ]
Starting openstack-cinder-api:                             [  OK  ]
Stopping openstack-cinder-backup:                          [  OK  ]
Starting openstack-cinder-backup:                          [  OK  ]
Stopping openstack-cinder-scheduler:                       [  OK  ]
Starting openstack-cinder-scheduler:                       [  OK  ]
Stopping openstack-cinder-volume:                          [  OK  ]
Starting openstack-cinder-volume:                          [  OK  ]
Stopping openstack-glance-api:                             [  OK  ]
Starting openstack-glance-api:                             [  OK  ]
Stopping openstack-glance-registry:                        [  OK  ]
Starting openstack-glance-registry:                        [  OK  ]
Stopping openstack-glance-scrubber:                        [  OK  ]
Starting openstack-glance-scrubber:                        [  OK  ]
Stopping keystone:                                         [  OK  ]
Starting keystone:                                         [  OK  ]
Stopping openstack-nova-api:                               [  OK  ]
Starting openstack-nova-api:                               [  OK  ]
Stopping openstack-nova-cert:                              [  OK  ]
Starting openstack-nova-cert:                              [  OK  ]
Stopping openstack-nova-compute:                           [  OK  ]
Starting openstack-nova-compute:                           [  OK  ]
Stopping openstack-nova-conductor:                         [  OK  ]
Starting openstack-nova-conductor:                         [  OK  ]
Stopping openstack-nova-console:                           [  OK  ]
Starting openstack-nova-console:                           [  OK  ]
Stopping openstack-nova-consoleauth:                       [  OK  ]
Starting openstack-nova-consoleauth:                       [  OK  ]
Stopping openstack-nova-metadata-api:                      [FAILED]
Starting openstack-nova-metadata-api:                      [  OK  ]
Stopping openstack-nova-novncproxy:                        [  OK  ]
Starting openstack-nova-novncproxy:                        [  OK  ]
Stopping openstack-nova-scheduler:                         [  OK  ]
Starting openstack-nova-scheduler:                         [  OK  ]
Stopping openstack-nova-spicehtml5proxy:                   [FAILED]
Starting openstack-nova-spicehtml5proxy:                   [  OK  ]
Stopping openstack-nova-xvpvncproxy:                       [  OK  ]
Starting openstack-nova-xvpvncproxy:                       [  OK  ]
Stopping openstack-swift-account:                          [  OK  ]
Starting openstack-swift-account:                          [  OK  ]
Stopping openstack-swift-account-auditor:                  [  OK  ]
Starting openstack-swift-account-auditor:                  [  OK  ]
Stopping openstack-swift-account-reaper:                   [  OK  ]
Starting openstack-swift-account-reaper:                   [  OK  ]
Stopping openstack-swift-account-replicator:               [  OK  ]
Starting openstack-swift-account-replicator:               [  OK  ]
Stopping openstack-swift-container:                        [  OK  ]
Starting openstack-swift-container:                        [  OK  ]
Stopping openstack-swift-container-auditor:                [  OK  ]
Starting openstack-swift-container-auditor:                [  OK  ]
Stopping openstack-swift-container-replicator:             [  OK  ]
Starting openstack-swift-container-replicator:             [  OK  ]
Stopping openstack-swift-container-updater:                [  OK  ]
Starting openstack-swift-container-updater:                [  OK  ]
Stopping openstack-swift-object:                           [  OK  ]
Starting openstack-swift-object:                           [  OK  ]
Stopping openstack-swift-object-auditor:                   [  OK  ]
Starting openstack-swift-object-auditor:                   [  OK  ]
Stopping openstack-swift-object-expirer:                   [  OK  ]
Starting openstack-swift-object-expirer:                   [  OK  ]
Stopping openstack-swift-object-replicator:                [  OK  ]
Starting openstack-swift-object-replicator:                [  OK  ]
Stopping openstack-swift-object-updater:                   [  OK  ]
Starting openstack-swift-object-updater:                   [  OK  ]
Stopping openstack-swift-proxy:                            [  OK  ]
Starting openstack-swift-proxy:                            [  OK  ]
Stopping neutron-dhcp-agent:                               [  OK  ]
Starting neutron-dhcp-agent:                               [  OK  ]
Stopping neutron-l3-agent:                                 [  OK  ]
Starting neutron-l3-agent:                                 [  OK  ]
Stopping neutron-lbaas-agent:                              [FAILED]
Starting neutron-lbaas-agent:                              [  OK  ]
Stopping neutron-metadata-agent:                           [  OK  ]
Starting neutron-metadata-agent:                           [  OK  ]
Stopping neutron-openvswitch-agent:                        [FAILED]
Starting neutron-openvswitch-agent:                        [  OK  ]
Stopping neutron-openvswitch-agent:                        [  OK  ]
Starting neutron-openvswitch-agent:                        [  OK  ]
Stopping neutron:                                          [  OK  ]
Starting neutron:                                          [  OK  ]
[root@diamond metadata]#

Now when we run openstack-status the openvswitch-agent should be running !

[root@diamond metadata]# openstack-status
...
neutron-openvswitch-agent:              active


[root@diamond metadata]# tail /var/log/neutron/openvswitch-agent.log
2014-07-27 12:50:13.793 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent initialized successfully, now running...
2014-07-27 12:50:13.800 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent out of sync with plugin!
2014-07-27 12:50:13.801 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Agent tunnel out of sync with plugin!
2014-07-27 12:50:13.851 24985 INFO neutron.openstack.common.rpc.common [req-f9d72adc-3043-4c3e-a24c-fc32bfa1d835 None] Connected to AMQP server on 192.168.1.10:5672
2014-07-27 12:50:14.329 24985 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'e44457a8-c88a-4a3a-818a-c96c75e16979'])
2014-07-27 12:50:15.023 24985 WARNING neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Device e44457a8-c88a-4a3a-818a-c96c75e16979 not defined on plugin
2014-07-27 12:50:18.119 24985 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'9cdf8669-9e09-48bb-873e-84cfad27993f'])
2014-07-27 12:50:18.812 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port 9cdf8669-9e09-48bb-873e-84cfad27993f updated. Details: {u'admin_state_up': True, u'network_id': u'975a58eb-d3cb-4c5b-956b-ed1bdb0820d2', u'segmentation_id': 10, u'physical_network': None, u'device': u'9cdf8669-9e09-48bb-873e-84cfad27993f', u'port_id': u'9cdf8669-9e09-48bb-873e-84cfad27993f', u'network_type': u'vxlan'}
2014-07-27 12:50:18.812 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Assigning 1 as local vlan for net-id=975a58eb-d3cb-4c5b-956b-ed1bdb0820d2
2014-07-27 12:50:19.205 24985 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Configuration for device 9cdf8669-9e09-48bb-873e-84cfad27993f completed.
[root@diamond metadata]#

And Healthy ! Awesome. Last, but not least you can find all my blessed configs for this article as well as a script I wrote to snapshot those configs on my github. Thanks for learning with me.