Python

How to setup Flask and Apache on an Ubuntu VM in DigitalOcean with a Custom Domain

Leave a Comment / Cloud, Digital Ocean, How To's, Python, Systems Administration /

In this video I show how setup Flask and Apache on an Ubuntu VM in Digital Ocean with a custom domain. This was made after someone in the comments on my other DigitalOcean video requested it. If there is something else anyone would like to see, please just let me know I am happy to provide these walk through’s.

Note: I hit a number of challenges with DNS in this one, I think it’s fun to watch me struggle. Enjoy!

How to setup Flask and Apache on an Ubuntu VM in DigitalOcean with a Custom Domain Read More »

How To: Create An AWS Lambda Function To Backup/Snapshot Your EBS Volumes

Leave a Comment / AWS, Cloud, How To's, Programming, Python, Systems Administration /

AWS Lambda functions are a great way to run some code on a trigger/schedule without needing a whole server dedicated to it. They can be cost effective, but be careful depending on how long they run, and the number of executions per hour, they can be quite costly as well.

For my use case, I wanted to create snapshot backups of EBS volumes for a Mongo Database every day. I originally implemented this using only CloudWatch, which is a monitoring service, but because it’s focused on scheduling, AWS also uses it for other things that require scheduling/cron like features. Unfortunately, the CloudWatch implementation of snapshot backups was very limited. I could not ‘tag’ the backups, which was certainly something I needed for easy finding and cleanups later (past a retention period).

Anyway, there were a couple pitfalls I ran into when creating this function.

Pitfalls

  1. Make sure you security group allows you to communicate to the Internet for any AWS API’s you need to talk to.
  2. Make sure your time-out is set to 1 minute or greater depending on your use case. The default is seconds, and is likely not high enough.
  3. “The Lambda function execution role must have permissions to create, describe and delete ENIs. AWS Lambda provides a permissions policy, AWSLambdaVPCAccessExecutionRole, with permissions for the necessary EC2 actions (ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces, and ec2:DeleteNetworkInterface) that you can use when creating a role”
    1. Personally, I did inline permissions and included the specific actions.
  4. Upload your zip file and make sure your handler section is configured with the exact file_name.method_in_your_code_for_the_handler
  5. Also this one is more of an FYI, Lambda Function have a maximum TTL of 5 minutes ( 300 seconds).

I think that was it, after that everything worked fine. To finish this short article off, screenshots and the code!

Screenshots

 

 

And finally the code…

Function Code

# Backup cis volumes

import boto3


def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    reg = 'us-east-1'

    # Connect to region
    ec2 = boto3.client('ec2', region_name=reg)

    response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']},
                                               {'Name': 'tag-key', 'Values': ['Name']},
                                               {'Name': 'tag-value', 'Values': ['cis-mongo*']},
                                               ])

    for r in response['Reservations']:
        for i in r['Instances']:
            for mapping in i['BlockDeviceMappings']:
                volId = mapping['Ebs']['VolumeId']

                # Create snapshot
                result = ec2.create_snapshot(VolumeId=volId,
                                             Description='Created by Lambda backup function ebs-snapshots')

                # Get snapshot resource
                ec2resource = boto3.resource('ec2', region_name=reg)
                snapshot = ec2resource.Snapshot(result['SnapshotId'])

                # Add volume name to snapshot for easier identification
                snapshot.create_tags(Tags=[{'Key': 'Name', 'Value': 'cis-mongo-snapshot-backup'}])

And here is an additional function to add for cleanup

import boto3
from datetime import timedelta, datetime


def lambda_handler(event, context):
    # if older than days delete
    days = 14

    filters = [{'Name': 'tag:Name', 'Values': ['cis-mongo-snapshot-backup']}]

    ec2 = boto3.setup_default_session(region_name='us-east-1')
    client = boto3.client('ec2')
    snapshots = client.describe_snapshots(Filters=filters)

    for snapshot in snapshots["Snapshots"]:
        start_time = snapshot["StartTime"]
        delete_time = datetime.now(start_time.tzinfo) - timedelta(days=days)

        if start_time < delete_time:
            print 'Deleting {id}'.format(id=snapshot["SnapshotId"])
            client.delete_snapshot(SnapshotId=snapshot["SnapshotId"], DryRun=False)

The end, happy server-lessing (ha !)

 

How To: Create An AWS Lambda Function To Backup/Snapshot Your EBS Volumes Read More »

How to use Boto to Audit your AWS EC2 instance security groups

Leave a Comment / AWS, Cloud, Programming, Python, Security, Systems Administration /

Boto is a Software Development Kit for accessing the AWS API’s using Python.

https://github.com/boto/boto3

Recently, I needed to determine how many of my EC2 instances were spawned in a public subnet, that also had security groups with wide open access on any port via any protocol to the instances. Because I have an IGW (Internet Gateway) in my VPC’s and properly setup routing tables, instances launched in the public subnets with wide open security groups (allowing ingress traffic from any source) is a really bad thing 🙂

Here is the code I wrote to identify these naughty instances. It will require slight modifications in your own environment, to match your public subnet IP Ranges, EC2 Tags, and Account names.

#!/usr/bin/env python
__author__ = 'Jason Riedel'
__description__ = 'Find EC2 instances provisioned in a public subnet that have security groups allowing ingress traffic from any source.'
__date__ = 'June 5th 2016'
__version__ = '1.0'

import boto3

def find_public_addresses(ec2):
    public_instances = {}
    instance_public_ips = {}
    instance_private_ips = {}
    instance_ident = {}
    instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running'] }])

    # Ranges that you define as public subnets in AWS go here.
    public_subnet_ranges = ['10.128.0', '192.168.0', '172.16.0']

    for instance in instances:
        # I only care if the private address falls into a public subnet range
        # because if it doesnt Internet ingress cant reach it directly anyway even with a public IP
        if any(cidr in instance.private_ip_address for cidr in public_subnet_ranges):
            owner_tag = "None"
            instance_name = "None"
            if instance.tags:
                for i in range(len(instance.tags)):
                    #comment OwnerEmail tag out if you do not tag your instances with it.
                    if instance.tags[i]['Key'] == "OwnerEmail":
                        owner_tag = instance.tags[i]['Value']
                    if instance.tags[i]['Key'] == "Name":
                        instance_name = instance.tags[i]['Value']
            instance_ident[instance.id] = "Name: %s\n\tKeypair: %s\n\tOwner: %s" % (instance_name, instance.key_name, owner_tag)
            if instance.public_ip_address is not None:
                values=[]
                for i in range(len(instance.security_groups)):
                    values.append(instance.security_groups[i]['GroupId'])
                public_instances[instance.id] = values
                instance_public_ips[instance.id] = instance.public_ip_address
                instance_private_ips[instance.id] = instance.private_ip_address

    return (public_instances, instance_public_ips,instance_private_ips, instance_ident)

def inspect_security_group(ec2, sg_id):
    sg = ec2.SecurityGroup(sg_id)

    open_cidrs = []
    for i in range(len(sg.ip_permissions)):
        to_port = ''
        ip_proto = ''
        if 'ToPort' in sg.ip_permissions[i]:
            to_port = sg.ip_permissions[i]['ToPort']
        if 'IpProtocol' in sg.ip_permissions[i]:
            ip_proto = sg.ip_permissions[i]['IpProtocol']
            if '-1' in ip_proto:
                ip_proto = 'All'
        for j in range(len(sg.ip_permissions[i]['IpRanges'])):
            cidr_string = "%s %s %s" % (sg.ip_permissions[i]['IpRanges'][j]['CidrIp'], ip_proto, to_port)

            if sg.ip_permissions[i]['IpRanges'][j]['CidrIp'] == '0.0.0.0/0':
                #preventing an instance being flagged for only ICMP being open
                if ip_proto != 'icmp':
                    open_cidrs.append(cidr_string)

    return open_cidrs


if __name__ == "__main__":
    #loading profiles from ~/.aws/config & credentials
    profile_names = ['de', 'pe', 'pde', 'ppe']
    #Translates profile name to a more friendly name i.e. Account Name
    translator = {'de': 'Platform Dev', 'pe': 'Platform Prod', 'pde': 'Products Dev', 'ppe': 'Products Prod'}
    for profile_name in profile_names:
        session = boto3.Session(profile_name=profile_name)
        ec2 = session.resource('ec2')

        (public_instances, instance_public_ips, instance_private_ips, instance_ident) = find_public_addresses(ec2)

        for instance in public_instances:
            for sg_id in public_instances[instance]:
                open_cidrs = inspect_security_group(ec2, sg_id)
                if open_cidrs: #only print if there are open cidrs
                    print "=================================="
                    print " %s, %s" % (instance, translator[profile_name])
                    print "=================================="
                    print "\tprivate ip: %s\n\tpublic ip: %s\n\t%s" % (instance_private_ips[instance], instance_public_ips[instance], instance_ident[instance])
                    print "\t=========================="
                    print "\t open ingress rules"
                    print "\t=========================="
                    for cidr in open_cidrs:
                        print "\t\t" + cidr

To run this you also need to setup your .aws/config and .aws/credentials file.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Email me tuxninja [at] tuxlabs.com if you have any issues.
Boto is awesome 🙂 Note so is the AWS CLI, but requires some shell scripting as opposed to Python to do cool stuff.

The github for this code here https://github.com/jasonriedel/AWS/blob/master/sg-audit.py

Enjoy !

How to use Boto to Audit your AWS EC2 instance security groups Read More »

Fun with Python, Tabular & AWS IP ranges

Leave a Comment / AWS, Cloud, Programming, Python /

I have been spending a lot of time designing a Hybrid Cloud that consists of Openstack and public cloud platforms. In particular I have been spending a lot of time designing the AWS portion of the Hybrid Cloud Platform. Today I found myself continually needing to look up AWS public address space and then parsing out regions & services. Then I remembered something a mentor of mine told me…

if you are going to do something more than once, there is probably value in automating it.

I love writing command line tools and thus a short Python script was born. Since I rarely share Python code, even though I didn’t spend a lot time on this, and I certainly didn’t optimize it for DRY etc. I am sharing it anyway for others to use, enjoy and hack on,

but mainly to learn, which is…The entire purpose of the Tuxlabs site

I should mention I have strong views about when to use Python vs. Go a language I find myself writing in more and more and this tool falls under my rules for things that I should write in Go. So later as a follow up I will likely re-code this in Go and post the code for review & learning. For now here’s the Python code, enjoy !

Listing all IP Ranges

(env) ➜  aws python aws-ranges.py
region          ip_prefix          service
--------------  -----------------  --------------------
us-east-1       23.20.0.0/14       AMAZON
ap-northeast-1  27.0.0.0/22        AMAZON
ap-southeast-1  43.250.192.0/24    AMAZON
ap-southeast-1  43.250.193.0/24    AMAZON
eu-west-1       46.51.128.0/18     AMAZON
eu-west-1       46.51.192.0/20     AMAZON
ap-southeast-1  46.51.216.0/21     AMAZON
ap-northeast-1  46.51.224.0/19     AMAZON
eu-west-1       46.137.0.0/17      AMAZON
eu-west-1       46.137.128.0/18    AMAZON
ap-southeast-1  46.137.192.0/19    AMAZON
ap-southeast-1  46.137.224.0/19    AMAZON
us-east-1       50.16.0.0/15       AMAZON
us-west-1       50.18.0.0/16       AMAZON
us-east-1       50.19.0.0/16       AMAZON
us-west-2       50.112.0.0/16      AMAZON
us-east-1       52.0.0.0/15        AMAZON
us-east-1       52.2.0.0/15        AMAZON
us-east-1       52.4.0.0/14        AMAZON
us-west-1       52.8.0.0/16        AMAZON
us-west-1       52.9.0.0/16        AMAZON
us-west-2       52.10.0.0/15       AMAZON
us-west-2       52.12.0.0/15       AMAZON
eu-west-1       52.16.0.0/15       AMAZON
eu-west-1       52.18.0.0/15       AMAZON
us-east-1       52.20.0.0/14       AMAZON
us-west-2       52.24.0.0/14       AMAZON
eu-central-1    52.28.0.0/16       AMAZON
eu-central-1    52.29.0.0/16       AMAZON
eu-west-1       52.30.0.0/15       AMAZON
us-west-2       52.32.0.0/14       AMAZON
us-west-2       52.36.0.0/14       AMAZON
us-west-2       52.40.0.0/14       AMAZON
eu-west-1       52.48.0.0/14       AMAZON
us-west-1       52.52.0.0/15       AMAZON
eu-central-1    52.58.0.0/15       AMAZON
ap-southeast-2  52.62.0.0/15       AMAZON
ap-southeast-2  52.64.0.0/17       AMAZON
ap-southeast-2  52.64.128.0/17     AMAZON
ap-southeast-2  52.65.0.0/16       AMAZON
sa-east-1       52.67.0.0/16       AMAZON
ap-northeast-1  52.68.0.0/15       AMAZON
us-east-1       52.70.0.0/15       AMAZON
us-east-1       52.72.0.0/15       AMAZON
ap-southeast-1  52.74.0.0/16       AMAZON
ap-southeast-1  52.76.0.0/17       AMAZON
ap-southeast-1  52.76.128.0/17     AMAZON
ap-southeast-1  52.77.0.0/16       AMAZON
ap-northeast-2  52.79.0.0/16       AMAZON
GLOBAL          52.84.0.0/15       AMAZON
us-east-1       52.86.0.0/15       AMAZON
us-west-2       52.88.0.0/15       AMAZON
us-east-1       52.90.0.0/15       AMAZON
ap-northeast-2  52.92.0.0/20       AMAZON
us-east-1       52.92.16.0/20      AMAZON
us-west-2       52.92.32.0/22      AMAZON
sa-east-1       52.92.39.0/24      AMAZON
eu-west-1       52.92.40.0/21      AMAZON
us-west-1       52.92.48.0/22      AMAZON
ap-southeast-2  52.92.52.0/22      AMAZON
ap-southeast-1  52.92.56.0/22      AMAZON
ap-northeast-1  52.92.60.0/22      AMAZON
sa-east-1       52.92.64.0/22      AMAZON
eu-central-1    52.92.68.0/22      AMAZON
sa-east-1       52.92.72.0/22      AMAZON
us-gov-west-1   52.92.252.0/22     AMAZON
eu-west-1       52.93.0.0/24       AMAZON
us-east-1       52.93.1.0/24       AMAZON
eu-west-1       52.93.2.0/24       AMAZON
us-east-1       52.93.3.0/24       AMAZON
ap-southeast-1  52.93.8.0/22       AMAZON
us-east-1       52.94.0.0/22       AMAZON
eu-west-1       52.94.5.0/24       AMAZON
ap-northeast-2  52.94.6.0/24       AMAZON
sa-east-1       52.94.7.0/24       AMAZON
ap-northeast-1  52.94.8.0/24       AMAZON
us-gov-west-1   52.94.9.0/24       AMAZON
us-west-2       52.94.10.0/24      AMAZON
ap-southeast-1  52.94.11.0/24      AMAZON
us-west-1       52.94.12.0/24      AMAZON
ap-southeast-2  52.94.13.0/24      AMAZON
us-west-1       52.94.14.0/24      AMAZON
us-east-1       52.94.254.0/23     AMAZON
ap-northeast-1  52.95.30.0/23      AMAZON
ap-northeast-1  52.95.34.0/24      AMAZON
ap-southeast-1  52.95.35.0/24      AMAZON
ap-southeast-2  52.95.36.0/22      AMAZON
us-east-1       52.95.48.0/22      AMAZON
us-east-1       52.95.52.0/22      AMAZON
ap-northeast-1  52.95.56.0/22      AMAZON
eu-west-1       52.95.60.0/24      AMAZON
eu-west-1       52.95.61.0/24      AMAZON
us-east-1       52.95.62.0/24      AMAZON
us-east-1       52.95.63.0/24      AMAZON
ap-northeast-2  52.95.192.0/20     AMAZON
ap-southeast-1  52.95.212.0/22     AMAZON
sa-east-1       52.95.240.0/24     AMAZON
ap-southeast-2  52.95.241.0/24     AMAZON
ap-southeast-1  52.95.242.0/24     AMAZON
ap-northeast-1  52.95.243.0/24     AMAZON
eu-west-1       52.95.244.0/24     AMAZON
us-east-1       52.95.245.0/24     AMAZON
us-west-1       52.95.246.0/24     AMAZON
us-west-2       52.95.247.0/24     AMAZON
eu-central-1    52.95.248.0/24     AMAZON
cn-north-1      52.95.249.0/24     AMAZON
ap-northeast-2  52.95.252.0/24     AMAZON
sa-east-1       52.95.255.0/28     AMAZON
ap-southeast-2  52.95.255.16/28    AMAZON
ap-southeast-1  52.95.255.32/28    AMAZON
ap-northeast-1  52.95.255.48/28    AMAZON
eu-west-1       52.95.255.64/28    AMAZON
us-east-1       52.95.255.80/28    AMAZON
us-west-1       52.95.255.96/28    AMAZON
us-west-2       52.95.255.112/28   AMAZON
eu-central-1    52.95.255.128/28   AMAZON
cn-north-1      52.95.255.144/28   AMAZON
ap-northeast-1  52.192.0.0/15      AMAZON
ap-northeast-1  52.196.0.0/14      AMAZON
us-east-1       52.200.0.0/13      AMAZON
eu-west-1       52.208.0.0/13      AMAZON
ap-southeast-1  52.220.0.0/15      AMAZON
ap-northeast-1  54.64.0.0/15       AMAZON
ap-southeast-2  54.66.0.0/16       AMAZON
us-west-1       54.67.0.0/16       AMAZON
us-west-2       54.68.0.0/14       AMAZON
eu-west-1       54.72.0.0/15       AMAZON
eu-west-1       54.74.0.0/15       AMAZON
eu-west-1       54.76.0.0/15       AMAZON
eu-west-1       54.78.0.0/16       AMAZON
ap-southeast-2  54.79.0.0/16       AMAZON
us-east-1       54.80.0.0/13       AMAZON
us-east-1       54.88.0.0/14       AMAZON
ap-northeast-1  54.92.0.0/17       AMAZON
us-east-1       54.92.128.0/17     AMAZON
eu-central-1    54.93.0.0/16       AMAZON
sa-east-1       54.94.0.0/16       AMAZON
ap-northeast-1  54.95.0.0/16       AMAZON
us-east-1       54.144.0.0/14      AMAZON
us-west-2       54.148.0.0/15      AMAZON
ap-northeast-1  54.150.0.0/16      AMAZON
us-west-1       54.151.0.0/17      AMAZON
ap-southeast-1  54.151.128.0/17    AMAZON
us-east-1       54.152.0.0/16      AMAZON
us-west-1       54.153.0.0/17      AMAZON
ap-southeast-2  54.153.128.0/17    AMAZON
eu-west-1       54.154.0.0/16      AMAZON
eu-west-1       54.155.0.0/16      AMAZON
us-east-1       54.156.0.0/14      AMAZON
us-east-1       54.160.0.0/13      AMAZON
ap-northeast-1  54.168.0.0/16      AMAZON
ap-southeast-1  54.169.0.0/16      AMAZON
eu-west-1       54.170.0.0/15      AMAZON
us-east-1       54.172.0.0/15      AMAZON
us-east-1       54.174.0.0/15      AMAZON
us-west-1       54.176.0.0/15      AMAZON
ap-northeast-1  54.178.0.0/16      AMAZON
ap-southeast-1  54.179.0.0/16      AMAZON
GLOBAL          54.182.0.0/16      AMAZON
us-west-1       54.183.0.0/16      AMAZON
us-west-2       54.184.0.0/13      AMAZON
GLOBAL          54.192.0.0/16      AMAZON
us-west-1       54.193.0.0/16      AMAZON
eu-west-1       54.194.0.0/15      AMAZON
us-east-1       54.196.0.0/15      AMAZON
us-east-1       54.198.0.0/16      AMAZON
ap-northeast-1  54.199.0.0/16      AMAZON
us-west-2       54.200.0.0/15      AMAZON
us-west-2       54.202.0.0/15      AMAZON
us-east-1       54.204.0.0/15      AMAZON
ap-southeast-2  54.206.0.0/16      AMAZON
sa-east-1       54.207.0.0/16      AMAZON
us-east-1       54.208.0.0/15      AMAZON
us-east-1       54.210.0.0/15      AMAZON
us-west-2       54.212.0.0/15      AMAZON
us-west-2       54.214.0.0/16      AMAZON
us-west-1       54.215.0.0/16      AMAZON
eu-west-1       54.216.0.0/15      AMAZON
us-west-2       54.218.0.0/16      AMAZON
us-west-1       54.219.0.0/16      AMAZON
eu-west-1       54.220.0.0/16      AMAZON
us-east-1       54.221.0.0/16      AMAZON
cn-north-1      54.222.0.0/19      AMAZON
cn-north-1      54.222.128.0/17    AMAZON
cn-north-1      54.223.0.0/16      AMAZON
us-east-1       54.224.0.0/15      AMAZON
us-east-1       54.226.0.0/15      AMAZON
eu-west-1       54.228.0.0/16      AMAZON
eu-west-1       54.229.0.0/16      AMAZON
GLOBAL          54.230.0.0/16      AMAZON
us-east-1       54.231.0.0/17      AMAZON
eu-west-1       54.231.128.0/19    AMAZON
us-west-2       54.231.160.0/19    AMAZON
eu-central-1    54.231.192.0/20    AMAZON
cn-north-1      54.231.208.0/20    AMAZON
ap-northeast-1  54.231.224.0/21    AMAZON
us-west-1       54.231.232.0/21    AMAZON
ap-southeast-1  54.231.240.0/22    AMAZON
us-east-1       54.231.244.0/22    AMAZON
ap-southeast-2  54.231.248.0/22    AMAZON
ap-southeast-2  54.231.252.0/24    AMAZON
sa-east-1       54.231.253.0/24    AMAZON
us-gov-west-1   54.231.254.0/24    AMAZON
sa-east-1       54.232.0.0/16      AMAZON
sa-east-1       54.233.0.0/18      AMAZON
sa-east-1       54.233.64.0/18     AMAZON
sa-east-1       54.233.128.0/17    AMAZON
us-east-1       54.234.0.0/15      AMAZON
us-east-1       54.236.0.0/15      AMAZON
ap-northeast-1  54.238.0.0/16      AMAZON
us-west-2       54.239.2.0/23      AMAZON
eu-central-1    54.239.4.0/22      AMAZON
us-east-1       54.239.8.0/21      AMAZON
us-east-1       54.239.16.0/20     AMAZON
eu-west-1       54.239.32.0/21     AMAZON
us-west-2       54.239.48.0/22     AMAZON
ap-northeast-1  54.239.52.0/23     AMAZON
eu-central-1    54.239.54.0/23     AMAZON
eu-central-1    54.239.56.0/21     AMAZON
ap-northeast-1  54.239.96.0/24     AMAZON
us-east-1       54.239.98.0/24     AMAZON
eu-west-1       54.239.99.0/24     AMAZON
eu-west-1       54.239.100.0/23    AMAZON
us-east-1       54.239.104.0/23    AMAZON
us-east-1       54.239.108.0/22    AMAZON
eu-west-1       54.239.114.0/24    AMAZON
ap-northeast-2  54.239.116.0/22    AMAZON
ap-northeast-2  54.239.120.0/21    AMAZON
GLOBAL          54.239.128.0/18    AMAZON
GLOBAL          54.239.192.0/19    AMAZON
GLOBAL          54.240.128.0/18    AMAZON
ap-southeast-2  54.240.192.0/22    AMAZON
us-east-1       54.240.196.0/24    AMAZON
eu-west-1       54.240.197.0/24    AMAZON
us-west-1       54.240.198.0/24    AMAZON
ap-southeast-1  54.240.199.0/24    AMAZON
ap-northeast-1  54.240.200.0/24    AMAZON
us-east-1       54.240.202.0/24    AMAZON
ap-southeast-2  54.240.203.0/24    AMAZON
ap-southeast-2  54.240.204.0/22    AMAZON
us-east-1       54.240.208.0/22    AMAZON
us-west-1       54.240.212.0/22    AMAZON
us-east-1       54.240.216.0/22    AMAZON
eu-west-1       54.240.220.0/22    AMAZON
ap-northeast-1  54.240.225.0/24    AMAZON
ap-southeast-1  54.240.226.0/24    AMAZON
ap-southeast-1  54.240.227.0/24    AMAZON
us-east-1       54.240.228.0/23    AMAZON
us-west-2       54.240.230.0/23    AMAZON
us-east-1       54.240.232.0/22    AMAZON
ap-northeast-2  54.240.236.0/22    AMAZON
eu-central-1    54.240.240.0/24    AMAZON
sa-east-1       54.240.244.0/22    AMAZON
us-west-2       54.240.248.0/21    AMAZON
us-west-1       54.241.0.0/16      AMAZON
us-east-1       54.242.0.0/15      AMAZON
us-west-2       54.244.0.0/16      AMAZON
us-west-2       54.245.0.0/16      AMAZON
eu-west-1       54.246.0.0/16      AMAZON
eu-west-1       54.247.0.0/16      AMAZON
ap-northeast-1  54.248.0.0/15      AMAZON
ap-northeast-1  54.250.0.0/16      AMAZON
ap-southeast-1  54.251.0.0/16      AMAZON
ap-southeast-2  54.252.0.0/16      AMAZON
ap-southeast-2  54.253.0.0/16      AMAZON
ap-southeast-1  54.254.0.0/16      AMAZON
ap-southeast-1  54.255.0.0/16      AMAZON
us-east-1       67.202.0.0/18      AMAZON
us-east-1       72.21.192.0/19     AMAZON
us-east-1       72.44.32.0/19      AMAZON
us-east-1       75.101.128.0/17    AMAZON
eu-west-1       79.125.0.0/17      AMAZON
eu-west-1       87.238.80.0/21     AMAZON
us-gov-west-1   96.127.0.0/17      AMAZON
ap-northeast-1  103.4.8.0/21       AMAZON
ap-southeast-1  103.246.148.0/23   AMAZON
ap-northeast-1  103.246.150.0/23   AMAZON
us-east-1       107.20.0.0/14      AMAZON
ap-southeast-1  122.248.192.0/18   AMAZON
us-east-1       172.96.97.0/24     AMAZON
us-east-1       174.129.0.0/16     AMAZON
ap-southeast-1  175.41.128.0/18    AMAZON
ap-northeast-1  175.41.192.0/18    AMAZON
ap-northeast-1  176.32.64.0/19     AMAZON
us-east-1       176.32.96.0/21     AMAZON
eu-west-1       176.32.104.0/21    AMAZON
us-west-1       176.32.112.0/21    AMAZON
us-east-1       176.32.120.0/22    AMAZON
us-west-2       176.32.125.0/25    AMAZON
ap-northeast-1  176.34.0.0/19      AMAZON
ap-northeast-1  176.34.32.0/19     AMAZON
eu-west-1       176.34.64.0/18     AMAZON
eu-west-1       176.34.128.0/17    AMAZON
sa-east-1       177.71.128.0/17    AMAZON
sa-east-1       177.72.240.0/21    AMAZON
eu-west-1       178.236.0.0/20     AMAZON
us-west-1       184.72.0.0/18      AMAZON
us-east-1       184.72.64.0/18     AMAZON
us-east-1       184.72.128.0/17    AMAZON
us-east-1       184.73.0.0/16      AMAZON
us-west-1       184.169.128.0/17   AMAZON
eu-west-1       185.48.120.0/22    AMAZON
ap-southeast-1  203.83.220.0/22    AMAZON
us-west-1       204.236.128.0/18   AMAZON
us-east-1       204.236.192.0/18   AMAZON
us-west-1       204.246.160.0/22   AMAZON
GLOBAL          204.246.164.0/22   AMAZON
GLOBAL          204.246.168.0/22   AMAZON
GLOBAL          204.246.174.0/23   AMAZON
GLOBAL          204.246.176.0/20   AMAZON
GLOBAL          205.251.192.0/19   AMAZON
us-east-1       205.251.224.0/22   AMAZON
us-west-1       205.251.228.0/22   AMAZON
us-west-2       205.251.232.0/22   AMAZON
us-gov-west-1   205.251.236.0/22   AMAZON
us-east-1       205.251.240.0/22   AMAZON
us-east-1       205.251.244.0/23   AMAZON
us-east-1       205.251.247.0/24   AMAZON
us-east-1       205.251.248.0/24   AMAZON
GLOBAL          205.251.249.0/24   AMAZON
GLOBAL          205.251.250.0/23   AMAZON
GLOBAL          205.251.252.0/23   AMAZON
GLOBAL          205.251.254.0/24   AMAZON
us-east-1       205.251.255.0/24   AMAZON
us-east-1       207.171.160.0/20   AMAZON
us-east-1       207.171.176.0/20   AMAZON
GLOBAL          216.137.32.0/19    AMAZON
us-east-1       216.182.224.0/20   AMAZON
us-east-1       23.20.0.0/14       EC2
eu-west-1       46.51.128.0/18     EC2
eu-west-1       46.51.192.0/20     EC2
ap-southeast-1  46.51.216.0/21     EC2
ap-northeast-1  46.51.224.0/19     EC2
eu-west-1       46.137.0.0/17      EC2
eu-west-1       46.137.128.0/18    EC2
ap-southeast-1  46.137.192.0/19    EC2
ap-southeast-1  46.137.224.0/19    EC2
us-east-1       50.16.0.0/15       EC2
us-west-1       50.18.0.0/16       EC2
us-east-1       50.19.0.0/16       EC2
us-west-2       50.112.0.0/16      EC2
us-east-1       52.0.0.0/15        EC2
us-east-1       52.2.0.0/15        EC2
us-east-1       52.4.0.0/14        EC2
us-west-1       52.8.0.0/16        EC2
us-west-1       52.9.0.0/16        EC2
us-west-2       52.10.0.0/15       EC2
us-west-2       52.12.0.0/15       EC2
eu-west-1       52.16.0.0/15       EC2
eu-west-1       52.18.0.0/15       EC2
us-east-1       52.20.0.0/14       EC2
us-west-2       52.24.0.0/14       EC2
eu-central-1    52.28.0.0/16       EC2
eu-central-1    52.29.0.0/16       EC2
eu-west-1       52.30.0.0/15       EC2
us-west-2       52.32.0.0/14       EC2
us-west-2       52.36.0.0/14       EC2
us-west-2       52.40.0.0/14       EC2
eu-west-1       52.48.0.0/14       EC2
us-west-1       52.52.0.0/15       EC2
eu-central-1    52.58.0.0/15       EC2
ap-southeast-2  52.62.0.0/15       EC2
ap-southeast-2  52.64.0.0/17       EC2
ap-southeast-2  52.64.128.0/17     EC2
ap-southeast-2  52.65.0.0/16       EC2
sa-east-1       52.67.0.0/16       EC2
ap-northeast-1  52.68.0.0/15       EC2
us-east-1       52.70.0.0/15       EC2
us-east-1       52.72.0.0/15       EC2
ap-southeast-1  52.74.0.0/16       EC2
ap-southeast-1  52.76.0.0/17       EC2
ap-southeast-1  52.76.128.0/17     EC2
ap-southeast-1  52.77.0.0/16       EC2
ap-northeast-2  52.79.0.0/16       EC2
us-east-1       52.86.0.0/15       EC2
us-west-2       52.88.0.0/15       EC2
us-east-1       52.90.0.0/15       EC2
sa-east-1       52.95.240.0/24     EC2
ap-southeast-2  52.95.241.0/24     EC2
ap-southeast-1  52.95.242.0/24     EC2
ap-northeast-1  52.95.243.0/24     EC2
eu-west-1       52.95.244.0/24     EC2
us-east-1       52.95.245.0/24     EC2
us-west-1       52.95.246.0/24     EC2
us-west-2       52.95.247.0/24     EC2
eu-central-1    52.95.248.0/24     EC2
cn-north-1      52.95.249.0/24     EC2
ap-northeast-2  52.95.252.0/24     EC2
sa-east-1       52.95.255.0/28     EC2
ap-southeast-2  52.95.255.16/28    EC2
ap-southeast-1  52.95.255.32/28    EC2
ap-northeast-1  52.95.255.48/28    EC2
eu-west-1       52.95.255.64/28    EC2
us-east-1       52.95.255.80/28    EC2
us-west-1       52.95.255.96/28    EC2
us-west-2       52.95.255.112/28   EC2
eu-central-1    52.95.255.128/28   EC2
cn-north-1      52.95.255.144/28   EC2
ap-northeast-1  52.192.0.0/15      EC2
ap-northeast-1  52.196.0.0/14      EC2
us-east-1       52.200.0.0/13      EC2
eu-west-1       52.208.0.0/13      EC2
ap-southeast-1  52.220.0.0/15      EC2
ap-northeast-1  54.64.0.0/15       EC2
ap-southeast-2  54.66.0.0/16       EC2
us-west-1       54.67.0.0/16       EC2
us-west-2       54.68.0.0/14       EC2
eu-west-1       54.72.0.0/15       EC2
eu-west-1       54.74.0.0/15       EC2
eu-west-1       54.76.0.0/15       EC2
eu-west-1       54.78.0.0/16       EC2
ap-southeast-2  54.79.0.0/16       EC2
us-east-1       54.80.0.0/13       EC2
us-east-1       54.88.0.0/14       EC2
ap-northeast-1  54.92.0.0/17       EC2
us-east-1       54.92.128.0/17     EC2
eu-central-1    54.93.0.0/16       EC2
sa-east-1       54.94.0.0/16       EC2
ap-northeast-1  54.95.0.0/16       EC2
us-east-1       54.144.0.0/14      EC2
us-west-2       54.148.0.0/15      EC2
ap-northeast-1  54.150.0.0/16      EC2
us-west-1       54.151.0.0/17      EC2
ap-southeast-1  54.151.128.0/17    EC2
us-east-1       54.152.0.0/16      EC2
us-west-1       54.153.0.0/17      EC2
ap-southeast-2  54.153.128.0/17    EC2
eu-west-1       54.154.0.0/16      EC2
eu-west-1       54.155.0.0/16      EC2
us-east-1       54.156.0.0/14      EC2
us-east-1       54.160.0.0/13      EC2
ap-northeast-1  54.168.0.0/16      EC2
ap-southeast-1  54.169.0.0/16      EC2
eu-west-1       54.170.0.0/15      EC2
us-east-1       54.172.0.0/15      EC2
us-east-1       54.174.0.0/15      EC2
us-west-1       54.176.0.0/15      EC2
ap-northeast-1  54.178.0.0/16      EC2
ap-southeast-1  54.179.0.0/16      EC2
us-west-1       54.183.0.0/16      EC2
us-west-2       54.184.0.0/13      EC2
us-west-1       54.193.0.0/16      EC2
eu-west-1       54.194.0.0/15      EC2
us-east-1       54.196.0.0/15      EC2
us-east-1       54.198.0.0/16      EC2
ap-northeast-1  54.199.0.0/16      EC2
us-west-2       54.200.0.0/15      EC2
us-west-2       54.202.0.0/15      EC2
us-east-1       54.204.0.0/15      EC2
ap-southeast-2  54.206.0.0/16      EC2
sa-east-1       54.207.0.0/16      EC2
us-east-1       54.208.0.0/15      EC2
us-east-1       54.210.0.0/15      EC2
us-west-2       54.212.0.0/15      EC2
us-west-2       54.214.0.0/16      EC2
us-west-1       54.215.0.0/16      EC2
eu-west-1       54.216.0.0/15      EC2
us-west-2       54.218.0.0/16      EC2
us-west-1       54.219.0.0/16      EC2
eu-west-1       54.220.0.0/16      EC2
us-east-1       54.221.0.0/16      EC2
cn-north-1      54.222.128.0/17    EC2
cn-north-1      54.223.0.0/16      EC2
us-east-1       54.224.0.0/15      EC2
us-east-1       54.226.0.0/15      EC2
eu-west-1       54.228.0.0/16      EC2
eu-west-1       54.229.0.0/16      EC2
sa-east-1       54.232.0.0/16      EC2
sa-east-1       54.233.0.0/18      EC2
sa-east-1       54.233.64.0/18     EC2
sa-east-1       54.233.128.0/17    EC2
us-east-1       54.234.0.0/15      EC2
us-east-1       54.236.0.0/15      EC2
ap-northeast-1  54.238.0.0/16      EC2
us-west-1       54.241.0.0/16      EC2
us-east-1       54.242.0.0/15      EC2
us-west-2       54.244.0.0/16      EC2
us-west-2       54.245.0.0/16      EC2
eu-west-1       54.246.0.0/16      EC2
eu-west-1       54.247.0.0/16      EC2
ap-northeast-1  54.248.0.0/15      EC2
ap-northeast-1  54.250.0.0/16      EC2
ap-southeast-1  54.251.0.0/16      EC2
ap-southeast-2  54.252.0.0/16      EC2
ap-southeast-2  54.253.0.0/16      EC2
ap-southeast-1  54.254.0.0/16      EC2
ap-southeast-1  54.255.0.0/16      EC2
us-east-1       67.202.0.0/18      EC2
us-east-1       72.44.32.0/19      EC2
us-east-1       75.101.128.0/17    EC2
eu-west-1       79.125.0.0/17      EC2
us-gov-west-1   96.127.0.0/17      EC2
ap-northeast-1  103.4.8.0/21       EC2
us-east-1       107.20.0.0/14      EC2
ap-southeast-1  122.248.192.0/18   EC2
us-east-1       174.129.0.0/16     EC2
ap-southeast-1  175.41.128.0/18    EC2
ap-northeast-1  175.41.192.0/18    EC2
ap-northeast-1  176.32.64.0/19     EC2
ap-northeast-1  176.34.0.0/19      EC2
ap-northeast-1  176.34.32.0/19     EC2
eu-west-1       176.34.64.0/18     EC2
eu-west-1       176.34.128.0/17    EC2
sa-east-1       177.71.128.0/17    EC2
us-west-1       184.72.0.0/18      EC2
us-east-1       184.72.64.0/18     EC2
us-east-1       184.72.128.0/17    EC2
us-east-1       184.73.0.0/16      EC2
us-west-1       184.169.128.0/17   EC2
eu-west-1       185.48.120.0/22    EC2
us-west-1       204.236.128.0/18   EC2
us-east-1       204.236.192.0/18   EC2
us-east-1       216.182.224.0/20   EC2
GLOBAL          205.251.192.0/21   ROUTE53
us-west-1       54.183.255.128/26  ROUTE53_HEALTHCHECKS
eu-west-1       54.228.16.0/26     ROUTE53_HEALTHCHECKS
sa-east-1       54.232.40.64/26    ROUTE53_HEALTHCHECKS
us-west-1       54.241.32.64/26    ROUTE53_HEALTHCHECKS
us-east-1       54.243.31.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.244.52.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.245.168.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.248.220.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.250.253.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.251.31.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.79.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.254.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.255.254.192/26  ROUTE53_HEALTHCHECKS
us-east-1       107.23.255.0/26    ROUTE53_HEALTHCHECKS
eu-west-1       176.34.159.192/26  ROUTE53_HEALTHCHECKS
sa-east-1       177.71.207.128/26  ROUTE53_HEALTHCHECKS
GLOBAL          52.84.0.0/15       CLOUDFRONT
GLOBAL          54.182.0.0/16      CLOUDFRONT
GLOBAL          54.192.0.0/16      CLOUDFRONT
GLOBAL          54.230.0.0/16      CLOUDFRONT
GLOBAL          54.239.128.0/18    CLOUDFRONT
GLOBAL          54.239.192.0/19    CLOUDFRONT
GLOBAL          54.240.128.0/18    CLOUDFRONT
GLOBAL          204.246.164.0/22   CLOUDFRONT
GLOBAL          204.246.168.0/22   CLOUDFRONT
GLOBAL          204.246.174.0/23   CLOUDFRONT
GLOBAL          204.246.176.0/20   CLOUDFRONT
GLOBAL          205.251.192.0/19   CLOUDFRONT
GLOBAL          205.251.249.0/24   CLOUDFRONT
GLOBAL          205.251.250.0/23   CLOUDFRONT
GLOBAL          205.251.252.0/23   CLOUDFRONT
GLOBAL          205.251.254.0/24   CLOUDFRONT
GLOBAL          216.137.32.0/19    CLOUDFRONT
(env) ➜  aws

Filtering

(env) ➜  aws python aws-ranges.py -r sa-east-1
region     ip_prefix          service
---------  -----------------  --------------------
sa-east-1  52.67.0.0/16       AMAZON
sa-east-1  52.92.39.0/24      AMAZON
sa-east-1  52.92.64.0/22      AMAZON
sa-east-1  52.92.72.0/22      AMAZON
sa-east-1  52.94.7.0/24       AMAZON
sa-east-1  52.95.240.0/24     AMAZON
sa-east-1  52.95.255.0/28     AMAZON
sa-east-1  54.94.0.0/16       AMAZON
sa-east-1  54.207.0.0/16      AMAZON
sa-east-1  54.231.253.0/24    AMAZON
sa-east-1  54.232.0.0/16      AMAZON
sa-east-1  54.233.0.0/18      AMAZON
sa-east-1  54.233.64.0/18     AMAZON
sa-east-1  54.233.128.0/17    AMAZON
sa-east-1  54.240.244.0/22    AMAZON
sa-east-1  177.71.128.0/17    AMAZON
sa-east-1  177.72.240.0/21    AMAZON
sa-east-1  52.67.0.0/16       EC2
sa-east-1  52.95.240.0/24     EC2
sa-east-1  52.95.255.0/28     EC2
sa-east-1  54.94.0.0/16       EC2
sa-east-1  54.207.0.0/16      EC2
sa-east-1  54.232.0.0/16      EC2
sa-east-1  54.233.0.0/18      EC2
sa-east-1  54.233.64.0/18     EC2
sa-east-1  54.233.128.0/17    EC2
sa-east-1  177.71.128.0/17    EC2
sa-east-1  54.232.40.64/26    ROUTE53_HEALTHCHECKS
sa-east-1  177.71.207.128/26  ROUTE53_HEALTHCHECKS
(env) ➜  aws python aws-ranges.py -r sa-east-1 -s ec2
region     ip_prefix        service
---------  ---------------  ---------
sa-east-1  52.67.0.0/16     EC2
sa-east-1  52.95.240.0/24   EC2
sa-east-1  52.95.255.0/28   EC2
sa-east-1  54.94.0.0/16     EC2
sa-east-1  54.207.0.0/16    EC2
sa-east-1  54.232.0.0/16    EC2
sa-east-1  54.233.0.0/18    EC2
sa-east-1  54.233.64.0/18   EC2
sa-east-1  54.233.128.0/17  EC2
sa-east-1  177.71.128.0/17  EC2
(env) ➜  aws

The code

#!/usr/bin/env python
#March 8th 2016

__author__ = 'Jason Riedel'
__description__ = 'Grabs IP ranges from amazon'
__version__ = '1.0'

url = 'https://ip-ranges.amazonaws.com/ip-ranges.json'

import requests
import json
import argparse
from tabulate import tabulate

parser = argparse.ArgumentParser()
parser.add_argument('-r', '--region', action="store", dest="regionFilter", required=False, help="Region to filter/return data for: us-west, ap-southeast-1, us-east-1")
parser.add_argument('-s', '--service', action="store", dest="serviceFilter", required=False, help="Service to filter/return data for: AMAZON, CLOUDFRONT, EC2")
parser.add_argument('-rl', action="store_true", dest="listRegions", required=False, help="List known regions")
parser.add_argument('-sl', action="store_true", dest="listService", required=False, help="List known services")
args = parser.parse_args()


def get_json(url):
    try:
        r = requests.get(url)
        jdata = r.json()

    except Exception, e:
        print "ERROR - %s failed:  %s" % (url, e)

    return jdata

def list_filter_json(jdata, filter_on):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        filtered_data.append(jdata['prefixes'][i][filter_on])
    return filtered_data

def filter_json(jdata, filters):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        if 'region' in filters and 'service' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower() and filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'region' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'service' in filters:
            if filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
    return filtered_data

def list_regions(jdata):
    regions = set()
    filtered_data = list_filter_json(jdata, 'region')
    for region in filtered_data:
        regions.add(region)
    for region in sorted(regions):
        print region

def list_services(jdata):
    services = set()
    filtered_data = list_filter_json(jdata, 'service')
    for service in filtered_data:
        services.add(service)
    for service in sorted(services):
        print service

def table_it(filtered_data):
    print tabulate(filtered_data, headers="keys")

if __name__ == "__main__":
    jdata = get_json(url)
    if args.listRegions or args.listService:
        if args.listRegions:
            list_regions(jdata)
        elif args.listService:
            list_services(jdata)
        else:
            parser.print_help()
    else:
        if args.regionFilter or args.serviceFilter:
            if args.regionFilter and args.serviceFilter:
                filters = {'region': args.regionFilter, 'service': args.serviceFilter}
            else:
                if args.regionFilter:
                    filters = {'region': args.regionFilter}
                if args.serviceFilter:
                    filters = {'service': args.serviceFilter}
            filtered_data = filter_json(jdata, filters)
            table_it(filtered_data)
        else:
            table_it(jdata['prefixes'])

Fun with Python, Tabular & AWS IP ranges Read More »