Programming

How to use Boto to Audit your AWS EC2 instance security groups

Boto is a Software Development Kit for accessing the AWS API’s using Python.

https://github.com/boto/boto3

Recently, I needed to determine how many of my EC2 instances were spawned in a public subnet, that also had security groups with wide open access on any port via any protocol to the instances. Because I have an IGW (Internet Gateway) in my VPC’s and properly setup routing tables, instances launched in the public subnets with wide open security groups (allowing ingress traffic from any source) is a really bad thing 🙂

Here is the code I wrote to identify these naughty instances. It will require slight modifications in your own environment, to match your public subnet IP Ranges, EC2 Tags, and Account names.

#!/usr/bin/env python
__author__ = 'Jason Riedel'
__description__ = 'Find EC2 instances provisioned in a public subnet that have security groups allowing ingress traffic from any source.'
__date__ = 'June 5th 2016'
__version__ = '1.0'

import boto3

def find_public_addresses(ec2):
    public_instances = {}
    instance_public_ips = {}
    instance_private_ips = {}
    instance_ident = {}
    instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running'] }])

    # Ranges that you define as public subnets in AWS go here.
    public_subnet_ranges = ['10.128.0', '192.168.0', '172.16.0']

    for instance in instances:
        # I only care if the private address falls into a public subnet range
        # because if it doesnt Internet ingress cant reach it directly anyway even with a public IP
        if any(cidr in instance.private_ip_address for cidr in public_subnet_ranges):
            owner_tag = "None"
            instance_name = "None"
            if instance.tags:
                for i in range(len(instance.tags)):
                    #comment OwnerEmail tag out if you do not tag your instances with it.
                    if instance.tags[i]['Key'] == "OwnerEmail":
                        owner_tag = instance.tags[i]['Value']
                    if instance.tags[i]['Key'] == "Name":
                        instance_name = instance.tags[i]['Value']
            instance_ident[instance.id] = "Name: %s\n\tKeypair: %s\n\tOwner: %s" % (instance_name, instance.key_name, owner_tag)
            if instance.public_ip_address is not None:
                values=[]
                for i in range(len(instance.security_groups)):
                    values.append(instance.security_groups[i]['GroupId'])
                public_instances[instance.id] = values
                instance_public_ips[instance.id] = instance.public_ip_address
                instance_private_ips[instance.id] = instance.private_ip_address

    return (public_instances, instance_public_ips,instance_private_ips, instance_ident)

def inspect_security_group(ec2, sg_id):
    sg = ec2.SecurityGroup(sg_id)

    open_cidrs = []
    for i in range(len(sg.ip_permissions)):
        to_port = ''
        ip_proto = ''
        if 'ToPort' in sg.ip_permissions[i]:
            to_port = sg.ip_permissions[i]['ToPort']
        if 'IpProtocol' in sg.ip_permissions[i]:
            ip_proto = sg.ip_permissions[i]['IpProtocol']
            if '-1' in ip_proto:
                ip_proto = 'All'
        for j in range(len(sg.ip_permissions[i]['IpRanges'])):
            cidr_string = "%s %s %s" % (sg.ip_permissions[i]['IpRanges'][j]['CidrIp'], ip_proto, to_port)

            if sg.ip_permissions[i]['IpRanges'][j]['CidrIp'] == '0.0.0.0/0':
                #preventing an instance being flagged for only ICMP being open
                if ip_proto != 'icmp':
                    open_cidrs.append(cidr_string)

    return open_cidrs


if __name__ == "__main__":
    #loading profiles from ~/.aws/config & credentials
    profile_names = ['de', 'pe', 'pde', 'ppe']
    #Translates profile name to a more friendly name i.e. Account Name
    translator = {'de': 'Platform Dev', 'pe': 'Platform Prod', 'pde': 'Products Dev', 'ppe': 'Products Prod'}
    for profile_name in profile_names:
        session = boto3.Session(profile_name=profile_name)
        ec2 = session.resource('ec2')

        (public_instances, instance_public_ips, instance_private_ips, instance_ident) = find_public_addresses(ec2)

        for instance in public_instances:
            for sg_id in public_instances[instance]:
                open_cidrs = inspect_security_group(ec2, sg_id)
                if open_cidrs: #only print if there are open cidrs
                    print "=================================="
                    print " %s, %s" % (instance, translator[profile_name])
                    print "=================================="
                    print "\tprivate ip: %s\n\tpublic ip: %s\n\t%s" % (instance_private_ips[instance], instance_public_ips[instance], instance_ident[instance])
                    print "\t=========================="
                    print "\t open ingress rules"
                    print "\t=========================="
                    for cidr in open_cidrs:
                        print "\t\t" + cidr

To run this you also need to setup your .aws/config and .aws/credentials file.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Email me tuxninja [at] tuxlabs.com if you have any issues.
Boto is awesome 🙂 Note so is the AWS CLI, but requires some shell scripting as opposed to Python to do cool stuff.

The github for this code here https://github.com/jasonriedel/AWS/blob/master/sg-audit.py

Enjoy !

How to use Boto to Audit your AWS EC2 instance security groups Read More »

Fun with Python, Tabular & AWS IP ranges

I have been spending a lot of time designing a Hybrid Cloud that consists of Openstack and public cloud platforms. In particular I have been spending a lot of time designing the AWS portion of the Hybrid Cloud Platform. Today I found myself continually needing to look up AWS public address space and then parsing out regions & services. Then I remembered something a mentor of mine told me…

if you are going to do something more than once, there is probably value in automating it.

I love writing command line tools and thus a short Python script was born. Since I rarely share Python code, even though I didn’t spend a lot time on this, and I certainly didn’t optimize it for DRY etc. I am sharing it anyway for others to use, enjoy and hack on,

but mainly to learn, which is…The entire purpose of the Tuxlabs site

I should mention I have strong views about when to use Python vs. Go a language I find myself writing in more and more and this tool falls under my rules for things that I should write in Go. So later as a follow up I will likely re-code this in Go and post the code for review & learning. For now here’s the Python code, enjoy !

Listing all IP Ranges

(env) ➜  aws python aws-ranges.py
region          ip_prefix          service
--------------  -----------------  --------------------
us-east-1       23.20.0.0/14       AMAZON
ap-northeast-1  27.0.0.0/22        AMAZON
ap-southeast-1  43.250.192.0/24    AMAZON
ap-southeast-1  43.250.193.0/24    AMAZON
eu-west-1       46.51.128.0/18     AMAZON
eu-west-1       46.51.192.0/20     AMAZON
ap-southeast-1  46.51.216.0/21     AMAZON
ap-northeast-1  46.51.224.0/19     AMAZON
eu-west-1       46.137.0.0/17      AMAZON
eu-west-1       46.137.128.0/18    AMAZON
ap-southeast-1  46.137.192.0/19    AMAZON
ap-southeast-1  46.137.224.0/19    AMAZON
us-east-1       50.16.0.0/15       AMAZON
us-west-1       50.18.0.0/16       AMAZON
us-east-1       50.19.0.0/16       AMAZON
us-west-2       50.112.0.0/16      AMAZON
us-east-1       52.0.0.0/15        AMAZON
us-east-1       52.2.0.0/15        AMAZON
us-east-1       52.4.0.0/14        AMAZON
us-west-1       52.8.0.0/16        AMAZON
us-west-1       52.9.0.0/16        AMAZON
us-west-2       52.10.0.0/15       AMAZON
us-west-2       52.12.0.0/15       AMAZON
eu-west-1       52.16.0.0/15       AMAZON
eu-west-1       52.18.0.0/15       AMAZON
us-east-1       52.20.0.0/14       AMAZON
us-west-2       52.24.0.0/14       AMAZON
eu-central-1    52.28.0.0/16       AMAZON
eu-central-1    52.29.0.0/16       AMAZON
eu-west-1       52.30.0.0/15       AMAZON
us-west-2       52.32.0.0/14       AMAZON
us-west-2       52.36.0.0/14       AMAZON
us-west-2       52.40.0.0/14       AMAZON
eu-west-1       52.48.0.0/14       AMAZON
us-west-1       52.52.0.0/15       AMAZON
eu-central-1    52.58.0.0/15       AMAZON
ap-southeast-2  52.62.0.0/15       AMAZON
ap-southeast-2  52.64.0.0/17       AMAZON
ap-southeast-2  52.64.128.0/17     AMAZON
ap-southeast-2  52.65.0.0/16       AMAZON
sa-east-1       52.67.0.0/16       AMAZON
ap-northeast-1  52.68.0.0/15       AMAZON
us-east-1       52.70.0.0/15       AMAZON
us-east-1       52.72.0.0/15       AMAZON
ap-southeast-1  52.74.0.0/16       AMAZON
ap-southeast-1  52.76.0.0/17       AMAZON
ap-southeast-1  52.76.128.0/17     AMAZON
ap-southeast-1  52.77.0.0/16       AMAZON
ap-northeast-2  52.79.0.0/16       AMAZON
GLOBAL          52.84.0.0/15       AMAZON
us-east-1       52.86.0.0/15       AMAZON
us-west-2       52.88.0.0/15       AMAZON
us-east-1       52.90.0.0/15       AMAZON
ap-northeast-2  52.92.0.0/20       AMAZON
us-east-1       52.92.16.0/20      AMAZON
us-west-2       52.92.32.0/22      AMAZON
sa-east-1       52.92.39.0/24      AMAZON
eu-west-1       52.92.40.0/21      AMAZON
us-west-1       52.92.48.0/22      AMAZON
ap-southeast-2  52.92.52.0/22      AMAZON
ap-southeast-1  52.92.56.0/22      AMAZON
ap-northeast-1  52.92.60.0/22      AMAZON
sa-east-1       52.92.64.0/22      AMAZON
eu-central-1    52.92.68.0/22      AMAZON
sa-east-1       52.92.72.0/22      AMAZON
us-gov-west-1   52.92.252.0/22     AMAZON
eu-west-1       52.93.0.0/24       AMAZON
us-east-1       52.93.1.0/24       AMAZON
eu-west-1       52.93.2.0/24       AMAZON
us-east-1       52.93.3.0/24       AMAZON
ap-southeast-1  52.93.8.0/22       AMAZON
us-east-1       52.94.0.0/22       AMAZON
eu-west-1       52.94.5.0/24       AMAZON
ap-northeast-2  52.94.6.0/24       AMAZON
sa-east-1       52.94.7.0/24       AMAZON
ap-northeast-1  52.94.8.0/24       AMAZON
us-gov-west-1   52.94.9.0/24       AMAZON
us-west-2       52.94.10.0/24      AMAZON
ap-southeast-1  52.94.11.0/24      AMAZON
us-west-1       52.94.12.0/24      AMAZON
ap-southeast-2  52.94.13.0/24      AMAZON
us-west-1       52.94.14.0/24      AMAZON
us-east-1       52.94.254.0/23     AMAZON
ap-northeast-1  52.95.30.0/23      AMAZON
ap-northeast-1  52.95.34.0/24      AMAZON
ap-southeast-1  52.95.35.0/24      AMAZON
ap-southeast-2  52.95.36.0/22      AMAZON
us-east-1       52.95.48.0/22      AMAZON
us-east-1       52.95.52.0/22      AMAZON
ap-northeast-1  52.95.56.0/22      AMAZON
eu-west-1       52.95.60.0/24      AMAZON
eu-west-1       52.95.61.0/24      AMAZON
us-east-1       52.95.62.0/24      AMAZON
us-east-1       52.95.63.0/24      AMAZON
ap-northeast-2  52.95.192.0/20     AMAZON
ap-southeast-1  52.95.212.0/22     AMAZON
sa-east-1       52.95.240.0/24     AMAZON
ap-southeast-2  52.95.241.0/24     AMAZON
ap-southeast-1  52.95.242.0/24     AMAZON
ap-northeast-1  52.95.243.0/24     AMAZON
eu-west-1       52.95.244.0/24     AMAZON
us-east-1       52.95.245.0/24     AMAZON
us-west-1       52.95.246.0/24     AMAZON
us-west-2       52.95.247.0/24     AMAZON
eu-central-1    52.95.248.0/24     AMAZON
cn-north-1      52.95.249.0/24     AMAZON
ap-northeast-2  52.95.252.0/24     AMAZON
sa-east-1       52.95.255.0/28     AMAZON
ap-southeast-2  52.95.255.16/28    AMAZON
ap-southeast-1  52.95.255.32/28    AMAZON
ap-northeast-1  52.95.255.48/28    AMAZON
eu-west-1       52.95.255.64/28    AMAZON
us-east-1       52.95.255.80/28    AMAZON
us-west-1       52.95.255.96/28    AMAZON
us-west-2       52.95.255.112/28   AMAZON
eu-central-1    52.95.255.128/28   AMAZON
cn-north-1      52.95.255.144/28   AMAZON
ap-northeast-1  52.192.0.0/15      AMAZON
ap-northeast-1  52.196.0.0/14      AMAZON
us-east-1       52.200.0.0/13      AMAZON
eu-west-1       52.208.0.0/13      AMAZON
ap-southeast-1  52.220.0.0/15      AMAZON
ap-northeast-1  54.64.0.0/15       AMAZON
ap-southeast-2  54.66.0.0/16       AMAZON
us-west-1       54.67.0.0/16       AMAZON
us-west-2       54.68.0.0/14       AMAZON
eu-west-1       54.72.0.0/15       AMAZON
eu-west-1       54.74.0.0/15       AMAZON
eu-west-1       54.76.0.0/15       AMAZON
eu-west-1       54.78.0.0/16       AMAZON
ap-southeast-2  54.79.0.0/16       AMAZON
us-east-1       54.80.0.0/13       AMAZON
us-east-1       54.88.0.0/14       AMAZON
ap-northeast-1  54.92.0.0/17       AMAZON
us-east-1       54.92.128.0/17     AMAZON
eu-central-1    54.93.0.0/16       AMAZON
sa-east-1       54.94.0.0/16       AMAZON
ap-northeast-1  54.95.0.0/16       AMAZON
us-east-1       54.144.0.0/14      AMAZON
us-west-2       54.148.0.0/15      AMAZON
ap-northeast-1  54.150.0.0/16      AMAZON
us-west-1       54.151.0.0/17      AMAZON
ap-southeast-1  54.151.128.0/17    AMAZON
us-east-1       54.152.0.0/16      AMAZON
us-west-1       54.153.0.0/17      AMAZON
ap-southeast-2  54.153.128.0/17    AMAZON
eu-west-1       54.154.0.0/16      AMAZON
eu-west-1       54.155.0.0/16      AMAZON
us-east-1       54.156.0.0/14      AMAZON
us-east-1       54.160.0.0/13      AMAZON
ap-northeast-1  54.168.0.0/16      AMAZON
ap-southeast-1  54.169.0.0/16      AMAZON
eu-west-1       54.170.0.0/15      AMAZON
us-east-1       54.172.0.0/15      AMAZON
us-east-1       54.174.0.0/15      AMAZON
us-west-1       54.176.0.0/15      AMAZON
ap-northeast-1  54.178.0.0/16      AMAZON
ap-southeast-1  54.179.0.0/16      AMAZON
GLOBAL          54.182.0.0/16      AMAZON
us-west-1       54.183.0.0/16      AMAZON
us-west-2       54.184.0.0/13      AMAZON
GLOBAL          54.192.0.0/16      AMAZON
us-west-1       54.193.0.0/16      AMAZON
eu-west-1       54.194.0.0/15      AMAZON
us-east-1       54.196.0.0/15      AMAZON
us-east-1       54.198.0.0/16      AMAZON
ap-northeast-1  54.199.0.0/16      AMAZON
us-west-2       54.200.0.0/15      AMAZON
us-west-2       54.202.0.0/15      AMAZON
us-east-1       54.204.0.0/15      AMAZON
ap-southeast-2  54.206.0.0/16      AMAZON
sa-east-1       54.207.0.0/16      AMAZON
us-east-1       54.208.0.0/15      AMAZON
us-east-1       54.210.0.0/15      AMAZON
us-west-2       54.212.0.0/15      AMAZON
us-west-2       54.214.0.0/16      AMAZON
us-west-1       54.215.0.0/16      AMAZON
eu-west-1       54.216.0.0/15      AMAZON
us-west-2       54.218.0.0/16      AMAZON
us-west-1       54.219.0.0/16      AMAZON
eu-west-1       54.220.0.0/16      AMAZON
us-east-1       54.221.0.0/16      AMAZON
cn-north-1      54.222.0.0/19      AMAZON
cn-north-1      54.222.128.0/17    AMAZON
cn-north-1      54.223.0.0/16      AMAZON
us-east-1       54.224.0.0/15      AMAZON
us-east-1       54.226.0.0/15      AMAZON
eu-west-1       54.228.0.0/16      AMAZON
eu-west-1       54.229.0.0/16      AMAZON
GLOBAL          54.230.0.0/16      AMAZON
us-east-1       54.231.0.0/17      AMAZON
eu-west-1       54.231.128.0/19    AMAZON
us-west-2       54.231.160.0/19    AMAZON
eu-central-1    54.231.192.0/20    AMAZON
cn-north-1      54.231.208.0/20    AMAZON
ap-northeast-1  54.231.224.0/21    AMAZON
us-west-1       54.231.232.0/21    AMAZON
ap-southeast-1  54.231.240.0/22    AMAZON
us-east-1       54.231.244.0/22    AMAZON
ap-southeast-2  54.231.248.0/22    AMAZON
ap-southeast-2  54.231.252.0/24    AMAZON
sa-east-1       54.231.253.0/24    AMAZON
us-gov-west-1   54.231.254.0/24    AMAZON
sa-east-1       54.232.0.0/16      AMAZON
sa-east-1       54.233.0.0/18      AMAZON
sa-east-1       54.233.64.0/18     AMAZON
sa-east-1       54.233.128.0/17    AMAZON
us-east-1       54.234.0.0/15      AMAZON
us-east-1       54.236.0.0/15      AMAZON
ap-northeast-1  54.238.0.0/16      AMAZON
us-west-2       54.239.2.0/23      AMAZON
eu-central-1    54.239.4.0/22      AMAZON
us-east-1       54.239.8.0/21      AMAZON
us-east-1       54.239.16.0/20     AMAZON
eu-west-1       54.239.32.0/21     AMAZON
us-west-2       54.239.48.0/22     AMAZON
ap-northeast-1  54.239.52.0/23     AMAZON
eu-central-1    54.239.54.0/23     AMAZON
eu-central-1    54.239.56.0/21     AMAZON
ap-northeast-1  54.239.96.0/24     AMAZON
us-east-1       54.239.98.0/24     AMAZON
eu-west-1       54.239.99.0/24     AMAZON
eu-west-1       54.239.100.0/23    AMAZON
us-east-1       54.239.104.0/23    AMAZON
us-east-1       54.239.108.0/22    AMAZON
eu-west-1       54.239.114.0/24    AMAZON
ap-northeast-2  54.239.116.0/22    AMAZON
ap-northeast-2  54.239.120.0/21    AMAZON
GLOBAL          54.239.128.0/18    AMAZON
GLOBAL          54.239.192.0/19    AMAZON
GLOBAL          54.240.128.0/18    AMAZON
ap-southeast-2  54.240.192.0/22    AMAZON
us-east-1       54.240.196.0/24    AMAZON
eu-west-1       54.240.197.0/24    AMAZON
us-west-1       54.240.198.0/24    AMAZON
ap-southeast-1  54.240.199.0/24    AMAZON
ap-northeast-1  54.240.200.0/24    AMAZON
us-east-1       54.240.202.0/24    AMAZON
ap-southeast-2  54.240.203.0/24    AMAZON
ap-southeast-2  54.240.204.0/22    AMAZON
us-east-1       54.240.208.0/22    AMAZON
us-west-1       54.240.212.0/22    AMAZON
us-east-1       54.240.216.0/22    AMAZON
eu-west-1       54.240.220.0/22    AMAZON
ap-northeast-1  54.240.225.0/24    AMAZON
ap-southeast-1  54.240.226.0/24    AMAZON
ap-southeast-1  54.240.227.0/24    AMAZON
us-east-1       54.240.228.0/23    AMAZON
us-west-2       54.240.230.0/23    AMAZON
us-east-1       54.240.232.0/22    AMAZON
ap-northeast-2  54.240.236.0/22    AMAZON
eu-central-1    54.240.240.0/24    AMAZON
sa-east-1       54.240.244.0/22    AMAZON
us-west-2       54.240.248.0/21    AMAZON
us-west-1       54.241.0.0/16      AMAZON
us-east-1       54.242.0.0/15      AMAZON
us-west-2       54.244.0.0/16      AMAZON
us-west-2       54.245.0.0/16      AMAZON
eu-west-1       54.246.0.0/16      AMAZON
eu-west-1       54.247.0.0/16      AMAZON
ap-northeast-1  54.248.0.0/15      AMAZON
ap-northeast-1  54.250.0.0/16      AMAZON
ap-southeast-1  54.251.0.0/16      AMAZON
ap-southeast-2  54.252.0.0/16      AMAZON
ap-southeast-2  54.253.0.0/16      AMAZON
ap-southeast-1  54.254.0.0/16      AMAZON
ap-southeast-1  54.255.0.0/16      AMAZON
us-east-1       67.202.0.0/18      AMAZON
us-east-1       72.21.192.0/19     AMAZON
us-east-1       72.44.32.0/19      AMAZON
us-east-1       75.101.128.0/17    AMAZON
eu-west-1       79.125.0.0/17      AMAZON
eu-west-1       87.238.80.0/21     AMAZON
us-gov-west-1   96.127.0.0/17      AMAZON
ap-northeast-1  103.4.8.0/21       AMAZON
ap-southeast-1  103.246.148.0/23   AMAZON
ap-northeast-1  103.246.150.0/23   AMAZON
us-east-1       107.20.0.0/14      AMAZON
ap-southeast-1  122.248.192.0/18   AMAZON
us-east-1       172.96.97.0/24     AMAZON
us-east-1       174.129.0.0/16     AMAZON
ap-southeast-1  175.41.128.0/18    AMAZON
ap-northeast-1  175.41.192.0/18    AMAZON
ap-northeast-1  176.32.64.0/19     AMAZON
us-east-1       176.32.96.0/21     AMAZON
eu-west-1       176.32.104.0/21    AMAZON
us-west-1       176.32.112.0/21    AMAZON
us-east-1       176.32.120.0/22    AMAZON
us-west-2       176.32.125.0/25    AMAZON
ap-northeast-1  176.34.0.0/19      AMAZON
ap-northeast-1  176.34.32.0/19     AMAZON
eu-west-1       176.34.64.0/18     AMAZON
eu-west-1       176.34.128.0/17    AMAZON
sa-east-1       177.71.128.0/17    AMAZON
sa-east-1       177.72.240.0/21    AMAZON
eu-west-1       178.236.0.0/20     AMAZON
us-west-1       184.72.0.0/18      AMAZON
us-east-1       184.72.64.0/18     AMAZON
us-east-1       184.72.128.0/17    AMAZON
us-east-1       184.73.0.0/16      AMAZON
us-west-1       184.169.128.0/17   AMAZON
eu-west-1       185.48.120.0/22    AMAZON
ap-southeast-1  203.83.220.0/22    AMAZON
us-west-1       204.236.128.0/18   AMAZON
us-east-1       204.236.192.0/18   AMAZON
us-west-1       204.246.160.0/22   AMAZON
GLOBAL          204.246.164.0/22   AMAZON
GLOBAL          204.246.168.0/22   AMAZON
GLOBAL          204.246.174.0/23   AMAZON
GLOBAL          204.246.176.0/20   AMAZON
GLOBAL          205.251.192.0/19   AMAZON
us-east-1       205.251.224.0/22   AMAZON
us-west-1       205.251.228.0/22   AMAZON
us-west-2       205.251.232.0/22   AMAZON
us-gov-west-1   205.251.236.0/22   AMAZON
us-east-1       205.251.240.0/22   AMAZON
us-east-1       205.251.244.0/23   AMAZON
us-east-1       205.251.247.0/24   AMAZON
us-east-1       205.251.248.0/24   AMAZON
GLOBAL          205.251.249.0/24   AMAZON
GLOBAL          205.251.250.0/23   AMAZON
GLOBAL          205.251.252.0/23   AMAZON
GLOBAL          205.251.254.0/24   AMAZON
us-east-1       205.251.255.0/24   AMAZON
us-east-1       207.171.160.0/20   AMAZON
us-east-1       207.171.176.0/20   AMAZON
GLOBAL          216.137.32.0/19    AMAZON
us-east-1       216.182.224.0/20   AMAZON
us-east-1       23.20.0.0/14       EC2
eu-west-1       46.51.128.0/18     EC2
eu-west-1       46.51.192.0/20     EC2
ap-southeast-1  46.51.216.0/21     EC2
ap-northeast-1  46.51.224.0/19     EC2
eu-west-1       46.137.0.0/17      EC2
eu-west-1       46.137.128.0/18    EC2
ap-southeast-1  46.137.192.0/19    EC2
ap-southeast-1  46.137.224.0/19    EC2
us-east-1       50.16.0.0/15       EC2
us-west-1       50.18.0.0/16       EC2
us-east-1       50.19.0.0/16       EC2
us-west-2       50.112.0.0/16      EC2
us-east-1       52.0.0.0/15        EC2
us-east-1       52.2.0.0/15        EC2
us-east-1       52.4.0.0/14        EC2
us-west-1       52.8.0.0/16        EC2
us-west-1       52.9.0.0/16        EC2
us-west-2       52.10.0.0/15       EC2
us-west-2       52.12.0.0/15       EC2
eu-west-1       52.16.0.0/15       EC2
eu-west-1       52.18.0.0/15       EC2
us-east-1       52.20.0.0/14       EC2
us-west-2       52.24.0.0/14       EC2
eu-central-1    52.28.0.0/16       EC2
eu-central-1    52.29.0.0/16       EC2
eu-west-1       52.30.0.0/15       EC2
us-west-2       52.32.0.0/14       EC2
us-west-2       52.36.0.0/14       EC2
us-west-2       52.40.0.0/14       EC2
eu-west-1       52.48.0.0/14       EC2
us-west-1       52.52.0.0/15       EC2
eu-central-1    52.58.0.0/15       EC2
ap-southeast-2  52.62.0.0/15       EC2
ap-southeast-2  52.64.0.0/17       EC2
ap-southeast-2  52.64.128.0/17     EC2
ap-southeast-2  52.65.0.0/16       EC2
sa-east-1       52.67.0.0/16       EC2
ap-northeast-1  52.68.0.0/15       EC2
us-east-1       52.70.0.0/15       EC2
us-east-1       52.72.0.0/15       EC2
ap-southeast-1  52.74.0.0/16       EC2
ap-southeast-1  52.76.0.0/17       EC2
ap-southeast-1  52.76.128.0/17     EC2
ap-southeast-1  52.77.0.0/16       EC2
ap-northeast-2  52.79.0.0/16       EC2
us-east-1       52.86.0.0/15       EC2
us-west-2       52.88.0.0/15       EC2
us-east-1       52.90.0.0/15       EC2
sa-east-1       52.95.240.0/24     EC2
ap-southeast-2  52.95.241.0/24     EC2
ap-southeast-1  52.95.242.0/24     EC2
ap-northeast-1  52.95.243.0/24     EC2
eu-west-1       52.95.244.0/24     EC2
us-east-1       52.95.245.0/24     EC2
us-west-1       52.95.246.0/24     EC2
us-west-2       52.95.247.0/24     EC2
eu-central-1    52.95.248.0/24     EC2
cn-north-1      52.95.249.0/24     EC2
ap-northeast-2  52.95.252.0/24     EC2
sa-east-1       52.95.255.0/28     EC2
ap-southeast-2  52.95.255.16/28    EC2
ap-southeast-1  52.95.255.32/28    EC2
ap-northeast-1  52.95.255.48/28    EC2
eu-west-1       52.95.255.64/28    EC2
us-east-1       52.95.255.80/28    EC2
us-west-1       52.95.255.96/28    EC2
us-west-2       52.95.255.112/28   EC2
eu-central-1    52.95.255.128/28   EC2
cn-north-1      52.95.255.144/28   EC2
ap-northeast-1  52.192.0.0/15      EC2
ap-northeast-1  52.196.0.0/14      EC2
us-east-1       52.200.0.0/13      EC2
eu-west-1       52.208.0.0/13      EC2
ap-southeast-1  52.220.0.0/15      EC2
ap-northeast-1  54.64.0.0/15       EC2
ap-southeast-2  54.66.0.0/16       EC2
us-west-1       54.67.0.0/16       EC2
us-west-2       54.68.0.0/14       EC2
eu-west-1       54.72.0.0/15       EC2
eu-west-1       54.74.0.0/15       EC2
eu-west-1       54.76.0.0/15       EC2
eu-west-1       54.78.0.0/16       EC2
ap-southeast-2  54.79.0.0/16       EC2
us-east-1       54.80.0.0/13       EC2
us-east-1       54.88.0.0/14       EC2
ap-northeast-1  54.92.0.0/17       EC2
us-east-1       54.92.128.0/17     EC2
eu-central-1    54.93.0.0/16       EC2
sa-east-1       54.94.0.0/16       EC2
ap-northeast-1  54.95.0.0/16       EC2
us-east-1       54.144.0.0/14      EC2
us-west-2       54.148.0.0/15      EC2
ap-northeast-1  54.150.0.0/16      EC2
us-west-1       54.151.0.0/17      EC2
ap-southeast-1  54.151.128.0/17    EC2
us-east-1       54.152.0.0/16      EC2
us-west-1       54.153.0.0/17      EC2
ap-southeast-2  54.153.128.0/17    EC2
eu-west-1       54.154.0.0/16      EC2
eu-west-1       54.155.0.0/16      EC2
us-east-1       54.156.0.0/14      EC2
us-east-1       54.160.0.0/13      EC2
ap-northeast-1  54.168.0.0/16      EC2
ap-southeast-1  54.169.0.0/16      EC2
eu-west-1       54.170.0.0/15      EC2
us-east-1       54.172.0.0/15      EC2
us-east-1       54.174.0.0/15      EC2
us-west-1       54.176.0.0/15      EC2
ap-northeast-1  54.178.0.0/16      EC2
ap-southeast-1  54.179.0.0/16      EC2
us-west-1       54.183.0.0/16      EC2
us-west-2       54.184.0.0/13      EC2
us-west-1       54.193.0.0/16      EC2
eu-west-1       54.194.0.0/15      EC2
us-east-1       54.196.0.0/15      EC2
us-east-1       54.198.0.0/16      EC2
ap-northeast-1  54.199.0.0/16      EC2
us-west-2       54.200.0.0/15      EC2
us-west-2       54.202.0.0/15      EC2
us-east-1       54.204.0.0/15      EC2
ap-southeast-2  54.206.0.0/16      EC2
sa-east-1       54.207.0.0/16      EC2
us-east-1       54.208.0.0/15      EC2
us-east-1       54.210.0.0/15      EC2
us-west-2       54.212.0.0/15      EC2
us-west-2       54.214.0.0/16      EC2
us-west-1       54.215.0.0/16      EC2
eu-west-1       54.216.0.0/15      EC2
us-west-2       54.218.0.0/16      EC2
us-west-1       54.219.0.0/16      EC2
eu-west-1       54.220.0.0/16      EC2
us-east-1       54.221.0.0/16      EC2
cn-north-1      54.222.128.0/17    EC2
cn-north-1      54.223.0.0/16      EC2
us-east-1       54.224.0.0/15      EC2
us-east-1       54.226.0.0/15      EC2
eu-west-1       54.228.0.0/16      EC2
eu-west-1       54.229.0.0/16      EC2
sa-east-1       54.232.0.0/16      EC2
sa-east-1       54.233.0.0/18      EC2
sa-east-1       54.233.64.0/18     EC2
sa-east-1       54.233.128.0/17    EC2
us-east-1       54.234.0.0/15      EC2
us-east-1       54.236.0.0/15      EC2
ap-northeast-1  54.238.0.0/16      EC2
us-west-1       54.241.0.0/16      EC2
us-east-1       54.242.0.0/15      EC2
us-west-2       54.244.0.0/16      EC2
us-west-2       54.245.0.0/16      EC2
eu-west-1       54.246.0.0/16      EC2
eu-west-1       54.247.0.0/16      EC2
ap-northeast-1  54.248.0.0/15      EC2
ap-northeast-1  54.250.0.0/16      EC2
ap-southeast-1  54.251.0.0/16      EC2
ap-southeast-2  54.252.0.0/16      EC2
ap-southeast-2  54.253.0.0/16      EC2
ap-southeast-1  54.254.0.0/16      EC2
ap-southeast-1  54.255.0.0/16      EC2
us-east-1       67.202.0.0/18      EC2
us-east-1       72.44.32.0/19      EC2
us-east-1       75.101.128.0/17    EC2
eu-west-1       79.125.0.0/17      EC2
us-gov-west-1   96.127.0.0/17      EC2
ap-northeast-1  103.4.8.0/21       EC2
us-east-1       107.20.0.0/14      EC2
ap-southeast-1  122.248.192.0/18   EC2
us-east-1       174.129.0.0/16     EC2
ap-southeast-1  175.41.128.0/18    EC2
ap-northeast-1  175.41.192.0/18    EC2
ap-northeast-1  176.32.64.0/19     EC2
ap-northeast-1  176.34.0.0/19      EC2
ap-northeast-1  176.34.32.0/19     EC2
eu-west-1       176.34.64.0/18     EC2
eu-west-1       176.34.128.0/17    EC2
sa-east-1       177.71.128.0/17    EC2
us-west-1       184.72.0.0/18      EC2
us-east-1       184.72.64.0/18     EC2
us-east-1       184.72.128.0/17    EC2
us-east-1       184.73.0.0/16      EC2
us-west-1       184.169.128.0/17   EC2
eu-west-1       185.48.120.0/22    EC2
us-west-1       204.236.128.0/18   EC2
us-east-1       204.236.192.0/18   EC2
us-east-1       216.182.224.0/20   EC2
GLOBAL          205.251.192.0/21   ROUTE53
us-west-1       54.183.255.128/26  ROUTE53_HEALTHCHECKS
eu-west-1       54.228.16.0/26     ROUTE53_HEALTHCHECKS
sa-east-1       54.232.40.64/26    ROUTE53_HEALTHCHECKS
us-west-1       54.241.32.64/26    ROUTE53_HEALTHCHECKS
us-east-1       54.243.31.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.244.52.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.245.168.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.248.220.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.250.253.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.251.31.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.79.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.254.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.255.254.192/26  ROUTE53_HEALTHCHECKS
us-east-1       107.23.255.0/26    ROUTE53_HEALTHCHECKS
eu-west-1       176.34.159.192/26  ROUTE53_HEALTHCHECKS
sa-east-1       177.71.207.128/26  ROUTE53_HEALTHCHECKS
GLOBAL          52.84.0.0/15       CLOUDFRONT
GLOBAL          54.182.0.0/16      CLOUDFRONT
GLOBAL          54.192.0.0/16      CLOUDFRONT
GLOBAL          54.230.0.0/16      CLOUDFRONT
GLOBAL          54.239.128.0/18    CLOUDFRONT
GLOBAL          54.239.192.0/19    CLOUDFRONT
GLOBAL          54.240.128.0/18    CLOUDFRONT
GLOBAL          204.246.164.0/22   CLOUDFRONT
GLOBAL          204.246.168.0/22   CLOUDFRONT
GLOBAL          204.246.174.0/23   CLOUDFRONT
GLOBAL          204.246.176.0/20   CLOUDFRONT
GLOBAL          205.251.192.0/19   CLOUDFRONT
GLOBAL          205.251.249.0/24   CLOUDFRONT
GLOBAL          205.251.250.0/23   CLOUDFRONT
GLOBAL          205.251.252.0/23   CLOUDFRONT
GLOBAL          205.251.254.0/24   CLOUDFRONT
GLOBAL          216.137.32.0/19    CLOUDFRONT
(env) ➜  aws

Filtering

(env) ➜  aws python aws-ranges.py -r sa-east-1
region     ip_prefix          service
---------  -----------------  --------------------
sa-east-1  52.67.0.0/16       AMAZON
sa-east-1  52.92.39.0/24      AMAZON
sa-east-1  52.92.64.0/22      AMAZON
sa-east-1  52.92.72.0/22      AMAZON
sa-east-1  52.94.7.0/24       AMAZON
sa-east-1  52.95.240.0/24     AMAZON
sa-east-1  52.95.255.0/28     AMAZON
sa-east-1  54.94.0.0/16       AMAZON
sa-east-1  54.207.0.0/16      AMAZON
sa-east-1  54.231.253.0/24    AMAZON
sa-east-1  54.232.0.0/16      AMAZON
sa-east-1  54.233.0.0/18      AMAZON
sa-east-1  54.233.64.0/18     AMAZON
sa-east-1  54.233.128.0/17    AMAZON
sa-east-1  54.240.244.0/22    AMAZON
sa-east-1  177.71.128.0/17    AMAZON
sa-east-1  177.72.240.0/21    AMAZON
sa-east-1  52.67.0.0/16       EC2
sa-east-1  52.95.240.0/24     EC2
sa-east-1  52.95.255.0/28     EC2
sa-east-1  54.94.0.0/16       EC2
sa-east-1  54.207.0.0/16      EC2
sa-east-1  54.232.0.0/16      EC2
sa-east-1  54.233.0.0/18      EC2
sa-east-1  54.233.64.0/18     EC2
sa-east-1  54.233.128.0/17    EC2
sa-east-1  177.71.128.0/17    EC2
sa-east-1  54.232.40.64/26    ROUTE53_HEALTHCHECKS
sa-east-1  177.71.207.128/26  ROUTE53_HEALTHCHECKS
(env) ➜  aws python aws-ranges.py -r sa-east-1 -s ec2
region     ip_prefix        service
---------  ---------------  ---------
sa-east-1  52.67.0.0/16     EC2
sa-east-1  52.95.240.0/24   EC2
sa-east-1  52.95.255.0/28   EC2
sa-east-1  54.94.0.0/16     EC2
sa-east-1  54.207.0.0/16    EC2
sa-east-1  54.232.0.0/16    EC2
sa-east-1  54.233.0.0/18    EC2
sa-east-1  54.233.64.0/18   EC2
sa-east-1  54.233.128.0/17  EC2
sa-east-1  177.71.128.0/17  EC2
(env) ➜  aws

The code

#!/usr/bin/env python
#March 8th 2016

__author__ = 'Jason Riedel'
__description__ = 'Grabs IP ranges from amazon'
__version__ = '1.0'

url = 'https://ip-ranges.amazonaws.com/ip-ranges.json'

import requests
import json
import argparse
from tabulate import tabulate

parser = argparse.ArgumentParser()
parser.add_argument('-r', '--region', action="store", dest="regionFilter", required=False, help="Region to filter/return data for: us-west, ap-southeast-1, us-east-1")
parser.add_argument('-s', '--service', action="store", dest="serviceFilter", required=False, help="Service to filter/return data for: AMAZON, CLOUDFRONT, EC2")
parser.add_argument('-rl', action="store_true", dest="listRegions", required=False, help="List known regions")
parser.add_argument('-sl', action="store_true", dest="listService", required=False, help="List known services")
args = parser.parse_args()


def get_json(url):
    try:
        r = requests.get(url)
        jdata = r.json()

    except Exception, e:
        print "ERROR - %s failed:  %s" % (url, e)

    return jdata

def list_filter_json(jdata, filter_on):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        filtered_data.append(jdata['prefixes'][i][filter_on])
    return filtered_data

def filter_json(jdata, filters):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        if 'region' in filters and 'service' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower() and filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'region' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'service' in filters:
            if filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
    return filtered_data

def list_regions(jdata):
    regions = set()
    filtered_data = list_filter_json(jdata, 'region')
    for region in filtered_data:
        regions.add(region)
    for region in sorted(regions):
        print region

def list_services(jdata):
    services = set()
    filtered_data = list_filter_json(jdata, 'service')
    for service in filtered_data:
        services.add(service)
    for service in sorted(services):
        print service

def table_it(filtered_data):
    print tabulate(filtered_data, headers="keys")

if __name__ == "__main__":
    jdata = get_json(url)
    if args.listRegions or args.listService:
        if args.listRegions:
            list_regions(jdata)
        elif args.listService:
            list_services(jdata)
        else:
            parser.print_help()
    else:
        if args.regionFilter or args.serviceFilter:
            if args.regionFilter and args.serviceFilter:
                filters = {'region': args.regionFilter, 'service': args.serviceFilter}
            else:
                if args.regionFilter:
                    filters = {'region': args.regionFilter}
                if args.serviceFilter:
                    filters = {'service': args.serviceFilter}
            filtered_data = filter_json(jdata, filters)
            table_it(filtered_data)
        else:
            table_it(jdata['prefixes'])

Fun with Python, Tabular & AWS IP ranges Read More »

Object-Oriented Programming With Python : Inheritance (2/3)

Continuing on our Object-Oriented Programming with Python series, this is article 2 of 3 on the 2nd pillar, Inheritance. You can find the first article about Encapsulation in this series here.

Inheritance – The Second Pillar

Inheritance in OOP is the ability to have one class inherit the attributes of another class.

Inheritance is simple to implement, but arguably the most powerful pillar in OOP, providing the most value to a programmer. It is the pillar that allows for effective re-use of code organized in classes reducing the amount of lines a programmer has to write to do something useful in their program.

In case you don’t remember from the previous article in this series on encapsulation, a class is a blueprint or template that contains methods, objects and attributes. Once defined a class can be inherited by another class. At which point the child class (aka derived class or subclass) contains all the goodness that the parent class ( aka super class or base class) provides.

Here is an example of inheritance :

#!/usr/bin/env python

import requests

class Tuxlabs(object):

    def setter(self, website):
        if 'http' in website:
            self.website = website
        else:
            print 'ERROR: Unable to set website to %s.\nRequired format is: http://www.example.com' % (website)

    def getter(self):
        return self.website

class Crawl(Tuxlabs):

    def download(self, website):
        self.setter(website)
        crawlTarget = self.getter()
        r = requests.get(crawlTarget)
        print r.text

website = 'http://www.tuxlabs.com'
crawl = Crawl()
crawl.download(website)

The code above downloads the webpage ‘tuxlabs.com’ and outputs it’s contents to STDOUT using the ‘requests’ module.

<rant>If you are a Python developer who is not already familiar with the requests module you should be ashamed of yourself 🙂 It is the most fantastically simple implementation of Python HTTP client functions available.</end rant>

In this example we have defined two classes, one called ‘Tuxlabs’ and one called ‘Crawl’. Tuxlabs inherits a base class ‘object’ and the ‘Crawl’ class inherits ‘Tuxlabs’. If you’re wondering why Tuxlabs inherits from object see here.  Our ‘Tuxlabs’ class has two methods a setter() & a getter() you will remember from the previous article on encapsulation. Now let’s focus on the real point in this example, the ‘Crawl’ class.

The ‘Crawl’ class as previously mention inherits the Tuxlabs class when it is defined or implemented ‘class Crawl(Tuxlabs):’. This means when an instance of Crawl initialized it will inherit all the attributes from the Tuxlabs class in addition to it’s own methods defined in the ‘Crawl’ class. The ‘Crawl’ class has only one method defined ‘download’ that takes a single argument, website.

Inside the download method in the crawl class we see ‘self.setter(website)’, but this is interesting, because we are using ‘self’ to call ‘setter()’, but self is actually referring to the ‘Crawl’ class and the ‘Crawl’ class  only has download defined. So shouldn’t an error be thrown when we try to access ‘self.setter(website)’ inside the ‘Crawl’ class, when setter() lives in ‘Tuxlabs’ ?

No, and the answer is because self, which refers to our ‘Crawl’ class, which is actually inheriting from our ‘Tuxlabs’ class. So this actually works ! Now from within ‘Crawl’ we can access all methods and attributes in Tuxlabs or Crawl, nifty, handy and powerful are the words that come to mind. Before moving on to some even cooler features, let’s show the abbreviated output of this script (I can’t show the entire webpage output, yuck!).

➜  tuxlabs  python example.py 
...
</body>
</html>
➜  tuxlabs

Nothing magical there, but I did want to show you that this code works !

Now then, we should talk about __init__ & super().

__init__() if you don’t already know is the initial method called in a class. It is what other languages might refer to as a constructor. When you create an instance object of a class the __init__() method within that class is called and whatever code is in it is executed. Init can be good for setting up default/static values used within the scope of the class and other methods. A typical example of an init class might look like this.

class Foundation(object):
    def __init__(self, sqlCreds):
        self.sqlHost = sqlCreds['host']
        self.sqlUser = sqlCreds['user']
        self.sqlPass = sqlCreds['pass']
        self.sqlDb = sqlCreds['db']
        if 'port' in sqlCreds:
            self.sqlPort = int(sqlCreds['port'])
        else:
            self.sqlPort = 3306
        self.sqlQuery = ''

Here we are defining a class ‘Foundation’ and an init constructor that accepts a dictionary as an argument containing SQL Credentials. Then within the init constructor we define several attributes who’s values are populated from that dictionary. We also made ‘port’ an optional argument with a default port of ‘3306’ the MySQL default port. This is a very useful start to a class, and it is my actual code from a project. From here we would build on top of this class many other methods like SQL queries we frequently used and other data processing techniques we frequently need like parsing JSON or XML.

Now let’s assume for a moment, this is a larger scale project. One that requires many classes. Classes that are inheriting from each other. Because the ‘Foundation’ class above is so generic, it can be used over and over again in other classes. However, those other classes will have slightly different purposes. In the case of databases, perhaps each class represents a different database. This makes sense, but we would still want to make use of the lovely constructor code inside of ‘Foundation’, thanks to it’s well written generic-ness. Here is an example of how we would do that, using something called super().

class Foundation(object):
    def __init__(self, sqlCreds):
        self.sqlHost = sqlCreds['host']
        self.sqlUser = sqlCreds['user']
        self.sqlPass = sqlCreds['pass']
        self.sqlDb = sqlCreds['db']
        if 'port' in sqlCreds:
            self.sqlPort = int(sqlCreds['port'])
        else:
            self.sqlPort = 3306
        self.sqlQuery = ''
...

class Cloud(Foundation):
    def __init__(self, sqlCredsCloud):
        super(Cloud, self).__init__(sqlCredsCloud)

...

We have our original class ‘Foundation’, and a new child class ‘Cloud’ that inherits ‘Foundation’. In addition ‘Cloud’ is defining it’s own __init__ constructor and accepting a dictionary for SQL credentials ‘sqlCredsCloud’. Finally, we call super() and pass our ‘Cloud’ class to the inherited class ‘Foundation’  (in this case self) and pass our SQL Creds to the ‘Foundation’ class constructor.

Whew.. I think I did an ok job explaining that, but you may need to read it back a couple of times and re-visit after trying to use this concept. Let’s take a look at how we would use these classes.

import sys
sys.path.append('/Users/jriedel/PycharmProjects')
from mymodules import foundation

def setup_cloud_db(sqlCredsCloud):
    Cloud = foundation.Cloud(sqlCredsCloud)
    CloudCursor = Cloud.SqlConnect()

    return (CloudCursor, Cloud)

if __name__ == '__main__':
    sqlCredsCloud = {'host': 'localhost', 'user': 'root', 'pass': '', 'db': 'cloud'}
    (CloudCursor, Cloud) = setup_cloud_db(sqlCredsCloud)
    Cloud.TruncateTable('cloud_machine_counts')

First we modify our Python path to locate the foundation module and import it. Then we create a subroutine for setting up our connection to the database and creating our instance object of our Cloud class. Notice we are passing sqlCredsCloud to the Cloud class and remember internally this is passing sqlCreds to the Foundation class constructor.  Finally, in the main routine of the program, after we setup the credentials and after we setup our database connection, we call ‘TruncateTable’, which is a method within the Cloud class that I have not shown in our code above, but you get the idea. Well I think that is about all I can muster, you now know what I know about inheritance.

For more information on super() click here

See ya next time !
Jason Riedel

Object-Oriented Programming With Python : Inheritance (2/3) Read More »

Introducing Vault

Vault

Vault is a command line utility for encrypting & decrypting things. Those things are stored on disk in hidden files, meaning in *nix they simply have a ‘.’ in front and don’t show up unless you type ls -la 😉 But anyway, who cares if someone can locate the files ! They are AES encrypted !

I wrote Vault for a couple of reasons.

  1. To learn how to use encryption in Python
  2. To store passwords for stuff
  3. To turn Vault into methods in my Utilities class that I use for my daily programming activities. First write a command line utility, then write a class 🙂

Vault Usage

I literally wrote vault earlier today and even though it is super simple, I still find it cool & useful so I had to share. Here’s a demo on it’s simple usage !

Keys

AES / Vault supports 16, 24, or 32 byte encryption keys. A byte is one character. Your key should be something ambiguous that you can remember and you should not use the ‘0123…’ example below. What would really be paranoid is to encrypt your keys in a separate vault somewhere 🙂 Note, you will need to use the same key to encrypt as decrypt for a given stored piece of encrypted data.

Encrypting

➜  vault git:(master) python store.py --name tuxlabs -t 'Well I could store a password or b00bs, cause everyone likes b00bs even the newbs' --key 01234567890987654321abcd
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w== was stored as tuxlabs
➜  vault git:(master) ✗

Can You Read The File Contents ?

➜  vault git:(master) ✗ cat .tuxlabs 
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w==%                                                                                                                     ➜  vault git:(master) ✗

Decrypting

➜  vault git:(master) ✗ python retrieve.py --name tuxlabs --key 01234567890987654321abcd
Well I could store a password or b00bs, cause everyone likes b00bs even the newbs
➜  vault git:(master) ✗

Alternatively, you can hide the key from the command line, by not specifying it. 

➜  vault  python store.py --name JimmyJohns --text 'Really Fast'
Key:
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ= was stored as JimmyJohns
➜  vault  python retrieve.py --name JimmyJohns
Key:
Really Fast
➜  vault  cat .JimmyJohns 
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ=%                                                                                                                                                                                                                                 ➜  vault

Clone Vault on github

Enjoy !
Jason Riedel

Introducing Vault Read More »

Runner Features Have Been Updated !

Runner Reminder

Runner is a command line tool for running commands on thousands of devices that support SSH. I wrote Runner and use it every single day, because unlike Ansible, Runner truly has no dependencies on the client or server side other than SSH. I have used Runner to build entire datacenters, so it is proven and tested and has a lot of well thought out features, which brings me to todays post. Since I initially debuted Runner I have added a lot of features, but I had yet to check them into github, until now. Here is a run down of Runner’s features.

Features

  • Runner takes your login credentials & doesn’t require you to setup SSH keys on the client machines/devices.
  • Runner can be used through a bastion/jump host via an SSH tunnel (see prunner.py)
  • Runner reads it’s main host list from a file ~/.runner/hosts/hosts-all
  • Runner can accept custom hosts lists via -f
  • -e can be used to echo a command before it is run, this is useful for running commands on F5 load balancers for example, when no output is returned on success.
  • -T will allow you to tune the number of threads, but be careful you can easily exhaust your system or site resources (I.E. do NOT DOS your LDAP authentication servers by trying to do hundreds of threads across thousands of machines, unless you know they can handle it 😉 ).
  • -s is for sudo for those users who have permissions in the sudoers file.
  • -1 reduces any host list down to one host per pool. It uses a regex, which you will likely have to modify for your own host / device naming standard.
  • -r can be used to supply a regular expression for matching hosts. Remember sometimes you have to quote the regex and/or escape the shell when using certain characters.
  • -c will run a single command on many hosts, but -cf will run a series of commands listed in a file on any hosts specified. This is particularly useful for automations. For example, I used it to build out load balancer virtuals and pools on an F5.
  • -p enables you to break apart the number of hosts to run at a time using a percentage. This is a handy & more humanized way to ensure you do not kill your machine or the infrastructure you are managing when you crank threads through the roof 😉

Now that I have taken the time to explain some of those cool features, here’s an example of what it looks like in action.

Runner Demo

Host List

➜  ~  runner -l -r tuxlabs                               
tuxlabs.com
old.tuxlabs.com

There were 2 hosts listed.
➜

Basic Run Using Only -c, -u and defaults

Note: User defaults to the user you are logged in as if you don’t specify -u . Since I am logged in as ‘jriedel’ I have specified the user tuxninja instead.

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),27(sudo)
old.tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),114(sambashare),1001(admin)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
➜  ~

The Same Run Using Sudo 

Note: I just realized if you do not prompt for a password for sudo it will fail, I will have to fix that ! Whoops ! P.S. You should always prompt for a password when using sudo !

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja -s
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INFO]: SUDO IS ON
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)
old.tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
➜  ~

Runner with a command file in super quiet mode  !

➜  ~  cat lets-run-these 
uptime
who
date
uptime -s
➜  ~  

➜  ~  runner -r tuxlabs -cf lets-run-these -T 2 -p 50 -qq -u tuxninja
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com:  05:17:51 up 14 days,  4:45,  0 users,  load average: 0.00, 0.01, 0.05
old.tuxlabs.com:  03:17:52 up 80 days, 47 min,  1 user,  load average: 0.42, 0.69, 0.78
old.tuxlabs.com: root     pts/0        2015-08-25 02:23 (173.224.162.99)
tuxlabs.com: Tue Aug 25 05:17:52 EDT 2015
old.tuxlabs.com: Tue Aug 25 03:17:53 CST 2015
tuxlabs.com: 2015-08-11 00:32:18
old.tuxlabs.com: 2015-06-06 02:30:42
➜  ~

Example of a simple regex & a failure

➜  ~  runner -r old.* -l
zsh: no matches found: old.*
➜  ~  

➜  ~  runner -r 'old.*' -l
old.tuxlabs.com

There was 1 host listed.
➜  ~

I hope you enjoyed the overview and new features. You can clone Runner on github.

Enjoy,
Jason Riedel

Runner Features Have Been Updated ! Read More »