TuxLabs LLC

All things DevOps

Category: AWS

How to use Boto to Audit your AWS EC2 instance security groups

Published / by tuxninja / Leave a Comment

Boto is a Software Development Kit for accessing the AWS API’s using Python.

https://github.com/boto/boto3

Recently, I needed to determine how many of my EC2 instances were spawned in a public subnet, that also had security groups with wide open access on any port via any protocol to the instances. Because I have an IGW (Internet Gateway) in my VPC’s and properly setup routing tables, instances launched in the public subnets with wide open security groups (allowing ingress traffic from any source) is a really bad thing 🙂

Here is the code I wrote to identify these naughty instances. It will require slight modifications in your own environment, to match your public subnet IP Ranges, EC2 Tags, and Account names.

To run this you also need to setup your .aws/config and .aws/credentials file.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Email me tuxninja [at] tuxlabs.com if you have any issues.
Boto is awesome 🙂 Note so is the AWS CLI, but requires some shell scripting as opposed to Python to do cool stuff.

The github for this code here https://github.com/jasonriedel/AWS/blob/master/sg-audit.py

Enjoy !

Consul for Service Discovery

Published / by tuxninja / Leave a Comment

Why Service Discovery ?

Service Discovery effectively replacing having to manually assign or automate your own DNS entries for nodes on your network. Service Discovery aims to move even further away from treating VM’s like pets to cattle, by getting rid of the age old practice Hostname & FQDN having contextual value. Instead when using services discovery nodes are automatically registered by an agent and automatically are configured in DNS for both nodes and services running on the machine.

Consul

Consul by Hashicorp is becoming the de-facto standard for Service Discovery. Consul’s full features & simplistic deployment model make it an optimal choice for organizations looking to quickly deploy Service Discovery capabilities in their environment.

Components of Consul

  1. The Consul Agent
  2. An optional JSON config file for each service located under /etc/consul.d/<service>.json
    1. If you do not specific a JSON file, consul can still start and will provide discovery for the nodes (they will have DNS as well)

A Quick Example of Consul

How easy is it to deploy console ?

  1. Download / Decompress and install the Consul agent – https://www.consul.io/downloads.html
  2. Define services in a JSON file (if you want) – https://www.consul.io/intro/getting-started/services.html
  3. Start the agent on the nodes – https://www.consul.io/intro/getting-started/join.html
  4.  Make 1 node join 1 other node (does not matter which node) to join the cluster, which gets you access to all cluster metadata

Steps 1 and 2 Above

  1. After downloading the Consul binary to each machine and decompressing it, copy it to /usr/local/bin/ so it’s in your path.
  2. Create the directory
  3. Optionally, run the following to create a JSON file defining a fake service running

Step 3 Above

Run the agent on each node, changing IP accordingly.

Step 4 Above

Wow, simple…ok now for the examples….

Show cluster members

Look up DNS for a node

Lookup DNS for a service

Query the REST API for Nodes

Query the REST API for Services

Fun with Python, Tabular & AWS IP ranges

Published / by tuxninja / Leave a Comment

I have been spending a lot of time designing a Hybrid Cloud that consists of Openstack and public cloud platforms. In particular I have been spending a lot of time designing the AWS portion of the Hybrid Cloud Platform. Today I found myself continually needing to look up AWS public address space and then parsing out regions & services. Then I remembered something a mentor of mine told me…

if you are going to do something more than once, there is probably value in automating it.

I love writing command line tools and thus a short Python script was born. Since I rarely share Python code, even though I didn’t spend a lot time on this, and I certainly didn’t optimize it for DRY etc. I am sharing it anyway for others to use, enjoy and hack on,

but mainly to learn, which is…The entire purpose of the Tuxlabs site

I should mention I have strong views about when to use Python vs. Go a language I find myself writing in more and more and this tool falls under my rules for things that I should write in Go. So later as a follow up I will likely re-code this in Go and post the code for review & learning. For now here’s the Python code, enjoy !

Listing all IP Ranges

Filtering

The code