Fun with Python, Tabular & AWS IP ranges

I have been spending a lot of time designing a Hybrid Cloud that consists of Openstack and public cloud platforms. In particular I have been spending a lot of time designing the AWS portion of the Hybrid Cloud Platform. Today I found myself continually needing to look up AWS public address space and then parsing out regions & services. Then I remembered something a mentor of mine told me…

if you are going to do something more than once, there is probably value in automating it.

I love writing command line tools and thus a short Python script was born. Since I rarely share Python code, even though I didn’t spend a lot time on this, and I certainly didn’t optimize it for DRY etc. I am sharing it anyway for others to use, enjoy and hack on,

but mainly to learn, which is…The entire purpose of the Tuxlabs site

I should mention I have strong views about when to use Python vs. Go a language I find myself writing in more and more and this tool falls under my rules for things that I should write in Go. So later as a follow up I will likely re-code this in Go and post the code for review & learning. For now here’s the Python code, enjoy !

Listing all IP Ranges

(env) ➜  aws python aws-ranges.py
region          ip_prefix          service
--------------  -----------------  --------------------
us-east-1       23.20.0.0/14       AMAZON
ap-northeast-1  27.0.0.0/22        AMAZON
ap-southeast-1  43.250.192.0/24    AMAZON
ap-southeast-1  43.250.193.0/24    AMAZON
eu-west-1       46.51.128.0/18     AMAZON
eu-west-1       46.51.192.0/20     AMAZON
ap-southeast-1  46.51.216.0/21     AMAZON
ap-northeast-1  46.51.224.0/19     AMAZON
eu-west-1       46.137.0.0/17      AMAZON
eu-west-1       46.137.128.0/18    AMAZON
ap-southeast-1  46.137.192.0/19    AMAZON
ap-southeast-1  46.137.224.0/19    AMAZON
us-east-1       50.16.0.0/15       AMAZON
us-west-1       50.18.0.0/16       AMAZON
us-east-1       50.19.0.0/16       AMAZON
us-west-2       50.112.0.0/16      AMAZON
us-east-1       52.0.0.0/15        AMAZON
us-east-1       52.2.0.0/15        AMAZON
us-east-1       52.4.0.0/14        AMAZON
us-west-1       52.8.0.0/16        AMAZON
us-west-1       52.9.0.0/16        AMAZON
us-west-2       52.10.0.0/15       AMAZON
us-west-2       52.12.0.0/15       AMAZON
eu-west-1       52.16.0.0/15       AMAZON
eu-west-1       52.18.0.0/15       AMAZON
us-east-1       52.20.0.0/14       AMAZON
us-west-2       52.24.0.0/14       AMAZON
eu-central-1    52.28.0.0/16       AMAZON
eu-central-1    52.29.0.0/16       AMAZON
eu-west-1       52.30.0.0/15       AMAZON
us-west-2       52.32.0.0/14       AMAZON
us-west-2       52.36.0.0/14       AMAZON
us-west-2       52.40.0.0/14       AMAZON
eu-west-1       52.48.0.0/14       AMAZON
us-west-1       52.52.0.0/15       AMAZON
eu-central-1    52.58.0.0/15       AMAZON
ap-southeast-2  52.62.0.0/15       AMAZON
ap-southeast-2  52.64.0.0/17       AMAZON
ap-southeast-2  52.64.128.0/17     AMAZON
ap-southeast-2  52.65.0.0/16       AMAZON
sa-east-1       52.67.0.0/16       AMAZON
ap-northeast-1  52.68.0.0/15       AMAZON
us-east-1       52.70.0.0/15       AMAZON
us-east-1       52.72.0.0/15       AMAZON
ap-southeast-1  52.74.0.0/16       AMAZON
ap-southeast-1  52.76.0.0/17       AMAZON
ap-southeast-1  52.76.128.0/17     AMAZON
ap-southeast-1  52.77.0.0/16       AMAZON
ap-northeast-2  52.79.0.0/16       AMAZON
GLOBAL          52.84.0.0/15       AMAZON
us-east-1       52.86.0.0/15       AMAZON
us-west-2       52.88.0.0/15       AMAZON
us-east-1       52.90.0.0/15       AMAZON
ap-northeast-2  52.92.0.0/20       AMAZON
us-east-1       52.92.16.0/20      AMAZON
us-west-2       52.92.32.0/22      AMAZON
sa-east-1       52.92.39.0/24      AMAZON
eu-west-1       52.92.40.0/21      AMAZON
us-west-1       52.92.48.0/22      AMAZON
ap-southeast-2  52.92.52.0/22      AMAZON
ap-southeast-1  52.92.56.0/22      AMAZON
ap-northeast-1  52.92.60.0/22      AMAZON
sa-east-1       52.92.64.0/22      AMAZON
eu-central-1    52.92.68.0/22      AMAZON
sa-east-1       52.92.72.0/22      AMAZON
us-gov-west-1   52.92.252.0/22     AMAZON
eu-west-1       52.93.0.0/24       AMAZON
us-east-1       52.93.1.0/24       AMAZON
eu-west-1       52.93.2.0/24       AMAZON
us-east-1       52.93.3.0/24       AMAZON
ap-southeast-1  52.93.8.0/22       AMAZON
us-east-1       52.94.0.0/22       AMAZON
eu-west-1       52.94.5.0/24       AMAZON
ap-northeast-2  52.94.6.0/24       AMAZON
sa-east-1       52.94.7.0/24       AMAZON
ap-northeast-1  52.94.8.0/24       AMAZON
us-gov-west-1   52.94.9.0/24       AMAZON
us-west-2       52.94.10.0/24      AMAZON
ap-southeast-1  52.94.11.0/24      AMAZON
us-west-1       52.94.12.0/24      AMAZON
ap-southeast-2  52.94.13.0/24      AMAZON
us-west-1       52.94.14.0/24      AMAZON
us-east-1       52.94.254.0/23     AMAZON
ap-northeast-1  52.95.30.0/23      AMAZON
ap-northeast-1  52.95.34.0/24      AMAZON
ap-southeast-1  52.95.35.0/24      AMAZON
ap-southeast-2  52.95.36.0/22      AMAZON
us-east-1       52.95.48.0/22      AMAZON
us-east-1       52.95.52.0/22      AMAZON
ap-northeast-1  52.95.56.0/22      AMAZON
eu-west-1       52.95.60.0/24      AMAZON
eu-west-1       52.95.61.0/24      AMAZON
us-east-1       52.95.62.0/24      AMAZON
us-east-1       52.95.63.0/24      AMAZON
ap-northeast-2  52.95.192.0/20     AMAZON
ap-southeast-1  52.95.212.0/22     AMAZON
sa-east-1       52.95.240.0/24     AMAZON
ap-southeast-2  52.95.241.0/24     AMAZON
ap-southeast-1  52.95.242.0/24     AMAZON
ap-northeast-1  52.95.243.0/24     AMAZON
eu-west-1       52.95.244.0/24     AMAZON
us-east-1       52.95.245.0/24     AMAZON
us-west-1       52.95.246.0/24     AMAZON
us-west-2       52.95.247.0/24     AMAZON
eu-central-1    52.95.248.0/24     AMAZON
cn-north-1      52.95.249.0/24     AMAZON
ap-northeast-2  52.95.252.0/24     AMAZON
sa-east-1       52.95.255.0/28     AMAZON
ap-southeast-2  52.95.255.16/28    AMAZON
ap-southeast-1  52.95.255.32/28    AMAZON
ap-northeast-1  52.95.255.48/28    AMAZON
eu-west-1       52.95.255.64/28    AMAZON
us-east-1       52.95.255.80/28    AMAZON
us-west-1       52.95.255.96/28    AMAZON
us-west-2       52.95.255.112/28   AMAZON
eu-central-1    52.95.255.128/28   AMAZON
cn-north-1      52.95.255.144/28   AMAZON
ap-northeast-1  52.192.0.0/15      AMAZON
ap-northeast-1  52.196.0.0/14      AMAZON
us-east-1       52.200.0.0/13      AMAZON
eu-west-1       52.208.0.0/13      AMAZON
ap-southeast-1  52.220.0.0/15      AMAZON
ap-northeast-1  54.64.0.0/15       AMAZON
ap-southeast-2  54.66.0.0/16       AMAZON
us-west-1       54.67.0.0/16       AMAZON
us-west-2       54.68.0.0/14       AMAZON
eu-west-1       54.72.0.0/15       AMAZON
eu-west-1       54.74.0.0/15       AMAZON
eu-west-1       54.76.0.0/15       AMAZON
eu-west-1       54.78.0.0/16       AMAZON
ap-southeast-2  54.79.0.0/16       AMAZON
us-east-1       54.80.0.0/13       AMAZON
us-east-1       54.88.0.0/14       AMAZON
ap-northeast-1  54.92.0.0/17       AMAZON
us-east-1       54.92.128.0/17     AMAZON
eu-central-1    54.93.0.0/16       AMAZON
sa-east-1       54.94.0.0/16       AMAZON
ap-northeast-1  54.95.0.0/16       AMAZON
us-east-1       54.144.0.0/14      AMAZON
us-west-2       54.148.0.0/15      AMAZON
ap-northeast-1  54.150.0.0/16      AMAZON
us-west-1       54.151.0.0/17      AMAZON
ap-southeast-1  54.151.128.0/17    AMAZON
us-east-1       54.152.0.0/16      AMAZON
us-west-1       54.153.0.0/17      AMAZON
ap-southeast-2  54.153.128.0/17    AMAZON
eu-west-1       54.154.0.0/16      AMAZON
eu-west-1       54.155.0.0/16      AMAZON
us-east-1       54.156.0.0/14      AMAZON
us-east-1       54.160.0.0/13      AMAZON
ap-northeast-1  54.168.0.0/16      AMAZON
ap-southeast-1  54.169.0.0/16      AMAZON
eu-west-1       54.170.0.0/15      AMAZON
us-east-1       54.172.0.0/15      AMAZON
us-east-1       54.174.0.0/15      AMAZON
us-west-1       54.176.0.0/15      AMAZON
ap-northeast-1  54.178.0.0/16      AMAZON
ap-southeast-1  54.179.0.0/16      AMAZON
GLOBAL          54.182.0.0/16      AMAZON
us-west-1       54.183.0.0/16      AMAZON
us-west-2       54.184.0.0/13      AMAZON
GLOBAL          54.192.0.0/16      AMAZON
us-west-1       54.193.0.0/16      AMAZON
eu-west-1       54.194.0.0/15      AMAZON
us-east-1       54.196.0.0/15      AMAZON
us-east-1       54.198.0.0/16      AMAZON
ap-northeast-1  54.199.0.0/16      AMAZON
us-west-2       54.200.0.0/15      AMAZON
us-west-2       54.202.0.0/15      AMAZON
us-east-1       54.204.0.0/15      AMAZON
ap-southeast-2  54.206.0.0/16      AMAZON
sa-east-1       54.207.0.0/16      AMAZON
us-east-1       54.208.0.0/15      AMAZON
us-east-1       54.210.0.0/15      AMAZON
us-west-2       54.212.0.0/15      AMAZON
us-west-2       54.214.0.0/16      AMAZON
us-west-1       54.215.0.0/16      AMAZON
eu-west-1       54.216.0.0/15      AMAZON
us-west-2       54.218.0.0/16      AMAZON
us-west-1       54.219.0.0/16      AMAZON
eu-west-1       54.220.0.0/16      AMAZON
us-east-1       54.221.0.0/16      AMAZON
cn-north-1      54.222.0.0/19      AMAZON
cn-north-1      54.222.128.0/17    AMAZON
cn-north-1      54.223.0.0/16      AMAZON
us-east-1       54.224.0.0/15      AMAZON
us-east-1       54.226.0.0/15      AMAZON
eu-west-1       54.228.0.0/16      AMAZON
eu-west-1       54.229.0.0/16      AMAZON
GLOBAL          54.230.0.0/16      AMAZON
us-east-1       54.231.0.0/17      AMAZON
eu-west-1       54.231.128.0/19    AMAZON
us-west-2       54.231.160.0/19    AMAZON
eu-central-1    54.231.192.0/20    AMAZON
cn-north-1      54.231.208.0/20    AMAZON
ap-northeast-1  54.231.224.0/21    AMAZON
us-west-1       54.231.232.0/21    AMAZON
ap-southeast-1  54.231.240.0/22    AMAZON
us-east-1       54.231.244.0/22    AMAZON
ap-southeast-2  54.231.248.0/22    AMAZON
ap-southeast-2  54.231.252.0/24    AMAZON
sa-east-1       54.231.253.0/24    AMAZON
us-gov-west-1   54.231.254.0/24    AMAZON
sa-east-1       54.232.0.0/16      AMAZON
sa-east-1       54.233.0.0/18      AMAZON
sa-east-1       54.233.64.0/18     AMAZON
sa-east-1       54.233.128.0/17    AMAZON
us-east-1       54.234.0.0/15      AMAZON
us-east-1       54.236.0.0/15      AMAZON
ap-northeast-1  54.238.0.0/16      AMAZON
us-west-2       54.239.2.0/23      AMAZON
eu-central-1    54.239.4.0/22      AMAZON
us-east-1       54.239.8.0/21      AMAZON
us-east-1       54.239.16.0/20     AMAZON
eu-west-1       54.239.32.0/21     AMAZON
us-west-2       54.239.48.0/22     AMAZON
ap-northeast-1  54.239.52.0/23     AMAZON
eu-central-1    54.239.54.0/23     AMAZON
eu-central-1    54.239.56.0/21     AMAZON
ap-northeast-1  54.239.96.0/24     AMAZON
us-east-1       54.239.98.0/24     AMAZON
eu-west-1       54.239.99.0/24     AMAZON
eu-west-1       54.239.100.0/23    AMAZON
us-east-1       54.239.104.0/23    AMAZON
us-east-1       54.239.108.0/22    AMAZON
eu-west-1       54.239.114.0/24    AMAZON
ap-northeast-2  54.239.116.0/22    AMAZON
ap-northeast-2  54.239.120.0/21    AMAZON
GLOBAL          54.239.128.0/18    AMAZON
GLOBAL          54.239.192.0/19    AMAZON
GLOBAL          54.240.128.0/18    AMAZON
ap-southeast-2  54.240.192.0/22    AMAZON
us-east-1       54.240.196.0/24    AMAZON
eu-west-1       54.240.197.0/24    AMAZON
us-west-1       54.240.198.0/24    AMAZON
ap-southeast-1  54.240.199.0/24    AMAZON
ap-northeast-1  54.240.200.0/24    AMAZON
us-east-1       54.240.202.0/24    AMAZON
ap-southeast-2  54.240.203.0/24    AMAZON
ap-southeast-2  54.240.204.0/22    AMAZON
us-east-1       54.240.208.0/22    AMAZON
us-west-1       54.240.212.0/22    AMAZON
us-east-1       54.240.216.0/22    AMAZON
eu-west-1       54.240.220.0/22    AMAZON
ap-northeast-1  54.240.225.0/24    AMAZON
ap-southeast-1  54.240.226.0/24    AMAZON
ap-southeast-1  54.240.227.0/24    AMAZON
us-east-1       54.240.228.0/23    AMAZON
us-west-2       54.240.230.0/23    AMAZON
us-east-1       54.240.232.0/22    AMAZON
ap-northeast-2  54.240.236.0/22    AMAZON
eu-central-1    54.240.240.0/24    AMAZON
sa-east-1       54.240.244.0/22    AMAZON
us-west-2       54.240.248.0/21    AMAZON
us-west-1       54.241.0.0/16      AMAZON
us-east-1       54.242.0.0/15      AMAZON
us-west-2       54.244.0.0/16      AMAZON
us-west-2       54.245.0.0/16      AMAZON
eu-west-1       54.246.0.0/16      AMAZON
eu-west-1       54.247.0.0/16      AMAZON
ap-northeast-1  54.248.0.0/15      AMAZON
ap-northeast-1  54.250.0.0/16      AMAZON
ap-southeast-1  54.251.0.0/16      AMAZON
ap-southeast-2  54.252.0.0/16      AMAZON
ap-southeast-2  54.253.0.0/16      AMAZON
ap-southeast-1  54.254.0.0/16      AMAZON
ap-southeast-1  54.255.0.0/16      AMAZON
us-east-1       67.202.0.0/18      AMAZON
us-east-1       72.21.192.0/19     AMAZON
us-east-1       72.44.32.0/19      AMAZON
us-east-1       75.101.128.0/17    AMAZON
eu-west-1       79.125.0.0/17      AMAZON
eu-west-1       87.238.80.0/21     AMAZON
us-gov-west-1   96.127.0.0/17      AMAZON
ap-northeast-1  103.4.8.0/21       AMAZON
ap-southeast-1  103.246.148.0/23   AMAZON
ap-northeast-1  103.246.150.0/23   AMAZON
us-east-1       107.20.0.0/14      AMAZON
ap-southeast-1  122.248.192.0/18   AMAZON
us-east-1       172.96.97.0/24     AMAZON
us-east-1       174.129.0.0/16     AMAZON
ap-southeast-1  175.41.128.0/18    AMAZON
ap-northeast-1  175.41.192.0/18    AMAZON
ap-northeast-1  176.32.64.0/19     AMAZON
us-east-1       176.32.96.0/21     AMAZON
eu-west-1       176.32.104.0/21    AMAZON
us-west-1       176.32.112.0/21    AMAZON
us-east-1       176.32.120.0/22    AMAZON
us-west-2       176.32.125.0/25    AMAZON
ap-northeast-1  176.34.0.0/19      AMAZON
ap-northeast-1  176.34.32.0/19     AMAZON
eu-west-1       176.34.64.0/18     AMAZON
eu-west-1       176.34.128.0/17    AMAZON
sa-east-1       177.71.128.0/17    AMAZON
sa-east-1       177.72.240.0/21    AMAZON
eu-west-1       178.236.0.0/20     AMAZON
us-west-1       184.72.0.0/18      AMAZON
us-east-1       184.72.64.0/18     AMAZON
us-east-1       184.72.128.0/17    AMAZON
us-east-1       184.73.0.0/16      AMAZON
us-west-1       184.169.128.0/17   AMAZON
eu-west-1       185.48.120.0/22    AMAZON
ap-southeast-1  203.83.220.0/22    AMAZON
us-west-1       204.236.128.0/18   AMAZON
us-east-1       204.236.192.0/18   AMAZON
us-west-1       204.246.160.0/22   AMAZON
GLOBAL          204.246.164.0/22   AMAZON
GLOBAL          204.246.168.0/22   AMAZON
GLOBAL          204.246.174.0/23   AMAZON
GLOBAL          204.246.176.0/20   AMAZON
GLOBAL          205.251.192.0/19   AMAZON
us-east-1       205.251.224.0/22   AMAZON
us-west-1       205.251.228.0/22   AMAZON
us-west-2       205.251.232.0/22   AMAZON
us-gov-west-1   205.251.236.0/22   AMAZON
us-east-1       205.251.240.0/22   AMAZON
us-east-1       205.251.244.0/23   AMAZON
us-east-1       205.251.247.0/24   AMAZON
us-east-1       205.251.248.0/24   AMAZON
GLOBAL          205.251.249.0/24   AMAZON
GLOBAL          205.251.250.0/23   AMAZON
GLOBAL          205.251.252.0/23   AMAZON
GLOBAL          205.251.254.0/24   AMAZON
us-east-1       205.251.255.0/24   AMAZON
us-east-1       207.171.160.0/20   AMAZON
us-east-1       207.171.176.0/20   AMAZON
GLOBAL          216.137.32.0/19    AMAZON
us-east-1       216.182.224.0/20   AMAZON
us-east-1       23.20.0.0/14       EC2
eu-west-1       46.51.128.0/18     EC2
eu-west-1       46.51.192.0/20     EC2
ap-southeast-1  46.51.216.0/21     EC2
ap-northeast-1  46.51.224.0/19     EC2
eu-west-1       46.137.0.0/17      EC2
eu-west-1       46.137.128.0/18    EC2
ap-southeast-1  46.137.192.0/19    EC2
ap-southeast-1  46.137.224.0/19    EC2
us-east-1       50.16.0.0/15       EC2
us-west-1       50.18.0.0/16       EC2
us-east-1       50.19.0.0/16       EC2
us-west-2       50.112.0.0/16      EC2
us-east-1       52.0.0.0/15        EC2
us-east-1       52.2.0.0/15        EC2
us-east-1       52.4.0.0/14        EC2
us-west-1       52.8.0.0/16        EC2
us-west-1       52.9.0.0/16        EC2
us-west-2       52.10.0.0/15       EC2
us-west-2       52.12.0.0/15       EC2
eu-west-1       52.16.0.0/15       EC2
eu-west-1       52.18.0.0/15       EC2
us-east-1       52.20.0.0/14       EC2
us-west-2       52.24.0.0/14       EC2
eu-central-1    52.28.0.0/16       EC2
eu-central-1    52.29.0.0/16       EC2
eu-west-1       52.30.0.0/15       EC2
us-west-2       52.32.0.0/14       EC2
us-west-2       52.36.0.0/14       EC2
us-west-2       52.40.0.0/14       EC2
eu-west-1       52.48.0.0/14       EC2
us-west-1       52.52.0.0/15       EC2
eu-central-1    52.58.0.0/15       EC2
ap-southeast-2  52.62.0.0/15       EC2
ap-southeast-2  52.64.0.0/17       EC2
ap-southeast-2  52.64.128.0/17     EC2
ap-southeast-2  52.65.0.0/16       EC2
sa-east-1       52.67.0.0/16       EC2
ap-northeast-1  52.68.0.0/15       EC2
us-east-1       52.70.0.0/15       EC2
us-east-1       52.72.0.0/15       EC2
ap-southeast-1  52.74.0.0/16       EC2
ap-southeast-1  52.76.0.0/17       EC2
ap-southeast-1  52.76.128.0/17     EC2
ap-southeast-1  52.77.0.0/16       EC2
ap-northeast-2  52.79.0.0/16       EC2
us-east-1       52.86.0.0/15       EC2
us-west-2       52.88.0.0/15       EC2
us-east-1       52.90.0.0/15       EC2
sa-east-1       52.95.240.0/24     EC2
ap-southeast-2  52.95.241.0/24     EC2
ap-southeast-1  52.95.242.0/24     EC2
ap-northeast-1  52.95.243.0/24     EC2
eu-west-1       52.95.244.0/24     EC2
us-east-1       52.95.245.0/24     EC2
us-west-1       52.95.246.0/24     EC2
us-west-2       52.95.247.0/24     EC2
eu-central-1    52.95.248.0/24     EC2
cn-north-1      52.95.249.0/24     EC2
ap-northeast-2  52.95.252.0/24     EC2
sa-east-1       52.95.255.0/28     EC2
ap-southeast-2  52.95.255.16/28    EC2
ap-southeast-1  52.95.255.32/28    EC2
ap-northeast-1  52.95.255.48/28    EC2
eu-west-1       52.95.255.64/28    EC2
us-east-1       52.95.255.80/28    EC2
us-west-1       52.95.255.96/28    EC2
us-west-2       52.95.255.112/28   EC2
eu-central-1    52.95.255.128/28   EC2
cn-north-1      52.95.255.144/28   EC2
ap-northeast-1  52.192.0.0/15      EC2
ap-northeast-1  52.196.0.0/14      EC2
us-east-1       52.200.0.0/13      EC2
eu-west-1       52.208.0.0/13      EC2
ap-southeast-1  52.220.0.0/15      EC2
ap-northeast-1  54.64.0.0/15       EC2
ap-southeast-2  54.66.0.0/16       EC2
us-west-1       54.67.0.0/16       EC2
us-west-2       54.68.0.0/14       EC2
eu-west-1       54.72.0.0/15       EC2
eu-west-1       54.74.0.0/15       EC2
eu-west-1       54.76.0.0/15       EC2
eu-west-1       54.78.0.0/16       EC2
ap-southeast-2  54.79.0.0/16       EC2
us-east-1       54.80.0.0/13       EC2
us-east-1       54.88.0.0/14       EC2
ap-northeast-1  54.92.0.0/17       EC2
us-east-1       54.92.128.0/17     EC2
eu-central-1    54.93.0.0/16       EC2
sa-east-1       54.94.0.0/16       EC2
ap-northeast-1  54.95.0.0/16       EC2
us-east-1       54.144.0.0/14      EC2
us-west-2       54.148.0.0/15      EC2
ap-northeast-1  54.150.0.0/16      EC2
us-west-1       54.151.0.0/17      EC2
ap-southeast-1  54.151.128.0/17    EC2
us-east-1       54.152.0.0/16      EC2
us-west-1       54.153.0.0/17      EC2
ap-southeast-2  54.153.128.0/17    EC2
eu-west-1       54.154.0.0/16      EC2
eu-west-1       54.155.0.0/16      EC2
us-east-1       54.156.0.0/14      EC2
us-east-1       54.160.0.0/13      EC2
ap-northeast-1  54.168.0.0/16      EC2
ap-southeast-1  54.169.0.0/16      EC2
eu-west-1       54.170.0.0/15      EC2
us-east-1       54.172.0.0/15      EC2
us-east-1       54.174.0.0/15      EC2
us-west-1       54.176.0.0/15      EC2
ap-northeast-1  54.178.0.0/16      EC2
ap-southeast-1  54.179.0.0/16      EC2
us-west-1       54.183.0.0/16      EC2
us-west-2       54.184.0.0/13      EC2
us-west-1       54.193.0.0/16      EC2
eu-west-1       54.194.0.0/15      EC2
us-east-1       54.196.0.0/15      EC2
us-east-1       54.198.0.0/16      EC2
ap-northeast-1  54.199.0.0/16      EC2
us-west-2       54.200.0.0/15      EC2
us-west-2       54.202.0.0/15      EC2
us-east-1       54.204.0.0/15      EC2
ap-southeast-2  54.206.0.0/16      EC2
sa-east-1       54.207.0.0/16      EC2
us-east-1       54.208.0.0/15      EC2
us-east-1       54.210.0.0/15      EC2
us-west-2       54.212.0.0/15      EC2
us-west-2       54.214.0.0/16      EC2
us-west-1       54.215.0.0/16      EC2
eu-west-1       54.216.0.0/15      EC2
us-west-2       54.218.0.0/16      EC2
us-west-1       54.219.0.0/16      EC2
eu-west-1       54.220.0.0/16      EC2
us-east-1       54.221.0.0/16      EC2
cn-north-1      54.222.128.0/17    EC2
cn-north-1      54.223.0.0/16      EC2
us-east-1       54.224.0.0/15      EC2
us-east-1       54.226.0.0/15      EC2
eu-west-1       54.228.0.0/16      EC2
eu-west-1       54.229.0.0/16      EC2
sa-east-1       54.232.0.0/16      EC2
sa-east-1       54.233.0.0/18      EC2
sa-east-1       54.233.64.0/18     EC2
sa-east-1       54.233.128.0/17    EC2
us-east-1       54.234.0.0/15      EC2
us-east-1       54.236.0.0/15      EC2
ap-northeast-1  54.238.0.0/16      EC2
us-west-1       54.241.0.0/16      EC2
us-east-1       54.242.0.0/15      EC2
us-west-2       54.244.0.0/16      EC2
us-west-2       54.245.0.0/16      EC2
eu-west-1       54.246.0.0/16      EC2
eu-west-1       54.247.0.0/16      EC2
ap-northeast-1  54.248.0.0/15      EC2
ap-northeast-1  54.250.0.0/16      EC2
ap-southeast-1  54.251.0.0/16      EC2
ap-southeast-2  54.252.0.0/16      EC2
ap-southeast-2  54.253.0.0/16      EC2
ap-southeast-1  54.254.0.0/16      EC2
ap-southeast-1  54.255.0.0/16      EC2
us-east-1       67.202.0.0/18      EC2
us-east-1       72.44.32.0/19      EC2
us-east-1       75.101.128.0/17    EC2
eu-west-1       79.125.0.0/17      EC2
us-gov-west-1   96.127.0.0/17      EC2
ap-northeast-1  103.4.8.0/21       EC2
us-east-1       107.20.0.0/14      EC2
ap-southeast-1  122.248.192.0/18   EC2
us-east-1       174.129.0.0/16     EC2
ap-southeast-1  175.41.128.0/18    EC2
ap-northeast-1  175.41.192.0/18    EC2
ap-northeast-1  176.32.64.0/19     EC2
ap-northeast-1  176.34.0.0/19      EC2
ap-northeast-1  176.34.32.0/19     EC2
eu-west-1       176.34.64.0/18     EC2
eu-west-1       176.34.128.0/17    EC2
sa-east-1       177.71.128.0/17    EC2
us-west-1       184.72.0.0/18      EC2
us-east-1       184.72.64.0/18     EC2
us-east-1       184.72.128.0/17    EC2
us-east-1       184.73.0.0/16      EC2
us-west-1       184.169.128.0/17   EC2
eu-west-1       185.48.120.0/22    EC2
us-west-1       204.236.128.0/18   EC2
us-east-1       204.236.192.0/18   EC2
us-east-1       216.182.224.0/20   EC2
GLOBAL          205.251.192.0/21   ROUTE53
us-west-1       54.183.255.128/26  ROUTE53_HEALTHCHECKS
eu-west-1       54.228.16.0/26     ROUTE53_HEALTHCHECKS
sa-east-1       54.232.40.64/26    ROUTE53_HEALTHCHECKS
us-west-1       54.241.32.64/26    ROUTE53_HEALTHCHECKS
us-east-1       54.243.31.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.244.52.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.245.168.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.248.220.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.250.253.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.251.31.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.79.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.254.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.255.254.192/26  ROUTE53_HEALTHCHECKS
us-east-1       107.23.255.0/26    ROUTE53_HEALTHCHECKS
eu-west-1       176.34.159.192/26  ROUTE53_HEALTHCHECKS
sa-east-1       177.71.207.128/26  ROUTE53_HEALTHCHECKS
GLOBAL          52.84.0.0/15       CLOUDFRONT
GLOBAL          54.182.0.0/16      CLOUDFRONT
GLOBAL          54.192.0.0/16      CLOUDFRONT
GLOBAL          54.230.0.0/16      CLOUDFRONT
GLOBAL          54.239.128.0/18    CLOUDFRONT
GLOBAL          54.239.192.0/19    CLOUDFRONT
GLOBAL          54.240.128.0/18    CLOUDFRONT
GLOBAL          204.246.164.0/22   CLOUDFRONT
GLOBAL          204.246.168.0/22   CLOUDFRONT
GLOBAL          204.246.174.0/23   CLOUDFRONT
GLOBAL          204.246.176.0/20   CLOUDFRONT
GLOBAL          205.251.192.0/19   CLOUDFRONT
GLOBAL          205.251.249.0/24   CLOUDFRONT
GLOBAL          205.251.250.0/23   CLOUDFRONT
GLOBAL          205.251.252.0/23   CLOUDFRONT
GLOBAL          205.251.254.0/24   CLOUDFRONT
GLOBAL          216.137.32.0/19    CLOUDFRONT
(env) ➜  aws

Filtering

(env) ➜  aws python aws-ranges.py -r sa-east-1
region     ip_prefix          service
---------  -----------------  --------------------
sa-east-1  52.67.0.0/16       AMAZON
sa-east-1  52.92.39.0/24      AMAZON
sa-east-1  52.92.64.0/22      AMAZON
sa-east-1  52.92.72.0/22      AMAZON
sa-east-1  52.94.7.0/24       AMAZON
sa-east-1  52.95.240.0/24     AMAZON
sa-east-1  52.95.255.0/28     AMAZON
sa-east-1  54.94.0.0/16       AMAZON
sa-east-1  54.207.0.0/16      AMAZON
sa-east-1  54.231.253.0/24    AMAZON
sa-east-1  54.232.0.0/16      AMAZON
sa-east-1  54.233.0.0/18      AMAZON
sa-east-1  54.233.64.0/18     AMAZON
sa-east-1  54.233.128.0/17    AMAZON
sa-east-1  54.240.244.0/22    AMAZON
sa-east-1  177.71.128.0/17    AMAZON
sa-east-1  177.72.240.0/21    AMAZON
sa-east-1  52.67.0.0/16       EC2
sa-east-1  52.95.240.0/24     EC2
sa-east-1  52.95.255.0/28     EC2
sa-east-1  54.94.0.0/16       EC2
sa-east-1  54.207.0.0/16      EC2
sa-east-1  54.232.0.0/16      EC2
sa-east-1  54.233.0.0/18      EC2
sa-east-1  54.233.64.0/18     EC2
sa-east-1  54.233.128.0/17    EC2
sa-east-1  177.71.128.0/17    EC2
sa-east-1  54.232.40.64/26    ROUTE53_HEALTHCHECKS
sa-east-1  177.71.207.128/26  ROUTE53_HEALTHCHECKS
(env) ➜  aws python aws-ranges.py -r sa-east-1 -s ec2
region     ip_prefix        service
---------  ---------------  ---------
sa-east-1  52.67.0.0/16     EC2
sa-east-1  52.95.240.0/24   EC2
sa-east-1  52.95.255.0/28   EC2
sa-east-1  54.94.0.0/16     EC2
sa-east-1  54.207.0.0/16    EC2
sa-east-1  54.232.0.0/16    EC2
sa-east-1  54.233.0.0/18    EC2
sa-east-1  54.233.64.0/18   EC2
sa-east-1  54.233.128.0/17  EC2
sa-east-1  177.71.128.0/17  EC2
(env) ➜  aws

The code

#!/usr/bin/env python
#March 8th 2016

__author__ = 'Jason Riedel'
__description__ = 'Grabs IP ranges from amazon'
__version__ = '1.0'

url = 'https://ip-ranges.amazonaws.com/ip-ranges.json'

import requests
import json
import argparse
from tabulate import tabulate

parser = argparse.ArgumentParser()
parser.add_argument('-r', '--region', action="store", dest="regionFilter", required=False, help="Region to filter/return data for: us-west, ap-southeast-1, us-east-1")
parser.add_argument('-s', '--service', action="store", dest="serviceFilter", required=False, help="Service to filter/return data for: AMAZON, CLOUDFRONT, EC2")
parser.add_argument('-rl', action="store_true", dest="listRegions", required=False, help="List known regions")
parser.add_argument('-sl', action="store_true", dest="listService", required=False, help="List known services")
args = parser.parse_args()


def get_json(url):
    try:
        r = requests.get(url)
        jdata = r.json()

    except Exception, e:
        print "ERROR - %s failed:  %s" % (url, e)

    return jdata

def list_filter_json(jdata, filter_on):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        filtered_data.append(jdata['prefixes'][i][filter_on])
    return filtered_data

def filter_json(jdata, filters):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        if 'region' in filters and 'service' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower() and filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'region' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'service' in filters:
            if filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
    return filtered_data

def list_regions(jdata):
    regions = set()
    filtered_data = list_filter_json(jdata, 'region')
    for region in filtered_data:
        regions.add(region)
    for region in sorted(regions):
        print region

def list_services(jdata):
    services = set()
    filtered_data = list_filter_json(jdata, 'service')
    for service in filtered_data:
        services.add(service)
    for service in sorted(services):
        print service

def table_it(filtered_data):
    print tabulate(filtered_data, headers="keys")

if __name__ == "__main__":
    jdata = get_json(url)
    if args.listRegions or args.listService:
        if args.listRegions:
            list_regions(jdata)
        elif args.listService:
            list_services(jdata)
        else:
            parser.print_help()
    else:
        if args.regionFilter or args.serviceFilter:
            if args.regionFilter and args.serviceFilter:
                filters = {'region': args.regionFilter, 'service': args.serviceFilter}
            else:
                if args.regionFilter:
                    filters = {'region': args.regionFilter}
                if args.serviceFilter:
                    filters = {'service': args.serviceFilter}
            filtered_data = filter_json(jdata, filters)
            table_it(filtered_data)
        else:
            table_it(jdata['prefixes'])

How To: curl the Openstack API’s (v3 Keystone Auth)

While Openstack provides a python client(s) for interactions….

[root@diamond ~]# source keystonerc_tuxninja
[root@diamond ~(keystone_tuxninja)]# openstack server list
+--------------------------------------+-------+--------+----------------------------------------+
| ID                                   | Name  | Status | Networks                               |
+--------------------------------------+-------+--------+----------------------------------------+
| e5b35d6a-a9ba-4714-a9e1-6361706bd047 | spin1 | ACTIVE | private_tuxlabs=10.0.0.8, 192.168.1.52 |
+--------------------------------------+-------+--------+----------------------------------------+
[root@diamond ~(keystone_tuxninja)]#

I frequently, finding myself needing to get data out of it without the pain of awk/sed’ing out the ASCII art.

Thus to quickly access the raw data, we can directly query the API’s using curl & parsing JSON instead, which is much better 🙂

Authentication

Before we can interact with the other Openstack API’s we need to authenticate to Keystone openstack’s identity service. After authenticating we receive a token to use with our subequent API requests. So step 1 we are going to create a JSON object with the required authentication details.

Create a file called ‘token-request.json’ with an object that looks like this.

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "id": "default"
                    },
                    "name": "tuxninja",
                    "password": "put_your_openstack_pass"
                }
            }
        }
    }
}

Btw, if you followed my tutorial on how to install Openstack Kilo, your authentication details for ‘admin’ is in your keystonerc_admin file.

Now we can use this file to authenticate like so:

export TOKEN=`curl -si -d @token-request.json -H "Content-type: application/json" http://localhost:35357/v3/auth/tokens | awk '/X-Subject-Token/ {print $2}'`

The token is actually returned in the header of the HTTP response, so this is why we need ‘-i’ when curling. Notice we are parsing out the token and returning the value to an environment variable $TOKEN.

Now we can include this $TOKEN and run whatever API commands we want (assuming admin privileges for the tenant/project)

Curl Commands (Numerous Examples!)

# list domains
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/domains

# create a domain
curl  -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" -d '{"domain": {"description": "--optional--", "enabled": true, "name": "dom1"}}'  http://localhost:35357/v3/domains


# list users
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users

# To create a users, create file named create_user.json file like this:

{
    "user": {
           "default_project_id": "18ed894bb8b84a5b9144c129fc754722",
            "description": "Description",
            "domain_id": "default",
            "email": "tuxninja@tuxlabs.com",
            "enabled": true,
            "name": "tuxninja",
            "password": "changeme" }
}

# then run
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users -d @create_user.json


# list images in nova
                                                                                             <tenant_id>
curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/images | python -m json.tool

# list servers (vms)

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool

# neutron networks

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

# neutron subnets

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

I sometimes pipe the output to python -m json.tool, which provides formatting for JSON. Lets take a closer look at an example.

Listing servers (vm’s)

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool
{
    "servers": [
        {
            "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047",
            "links": [
                {
                    "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "bookmark"
                }
            ],
            "name": "spin1"
        }
    ]
}
[root@diamond ~(keystone_tuxninja)]#

I only have 1 VM currently called spin1, but for the tutorials sake, if I had ten’s or hundred’s of VM’s and all I cared about was the VM name or ID, I would still need to parse this JSON object to avoid getting all this other meta-data.

My favorite command line way to do that without going full Python is using the handy JQ tool.

Here is how to use it !

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .
{
  "servers": [
    {
      "name": "spin1",
      "links": [
        {
          "rel": "self",
          "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        },
        {
          "rel": "bookmark",
          "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        }
      ],
      "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047"
    }
  ]
}
[root@diamond ~(keystone_tuxninja)]#
[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .servers[0].name -r
spin1
[root@diamond ~(keystone_tuxninja)]#

The first command just takes whatever the STDOUT from curl is and indent’s and color’s the JSON making it pretty (colors gives it +1 vs. python -m json.tool).

The second example we actually parse what were after. As you can see it is pretty simple, but jq’s query language may not be 100% intuitive at first, but I promise it is pretty easy to understand if you have ever parsed JSON before. Read up more on JQ @ https://stedolan.github.io/jq/ & check out the Openstack docs for more API commands http://developer.openstack.org/api-ref.html

Hope you enjoyed this post ! Until next time.

Introducing Vault

Vault

Vault is a command line utility for encrypting & decrypting things. Those things are stored on disk in hidden files, meaning in *nix they simply have a ‘.’ in front and don’t show up unless you type ls -la 😉 But anyway, who cares if someone can locate the files ! They are AES encrypted !

I wrote Vault for a couple of reasons.

To learn how to use encryption in Python
To store passwords for stuff
To turn Vault into methods in my Utilities class that I use for my daily programming activities. First write a command line utility, then write a class 🙂

Vault Usage

I literally wrote vault earlier today and even though it is super simple, I still find it cool & useful so I had to share. Here’s a demo on it’s simple usage !

Keys

AES / Vault supports 16, 24, or 32 byte encryption keys. A byte is one character. Your key should be something ambiguous that you can remember and you should not use the ‘0123…’ example below. What would really be paranoid is to encrypt your keys in a separate vault somewhere 🙂 Note, you will need to use the same key to encrypt as decrypt for a given stored piece of encrypted data.

Encrypting

➜  vault git:(master) python store.py --name tuxlabs -t 'Well I could store a password or b00bs, cause everyone likes b00bs even the newbs' --key 01234567890987654321abcd
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w== was stored as tuxlabs
➜  vault git:(master) ✗

Can You Read The File Contents ?

➜  vault git:(master) ✗ cat .tuxlabs 
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w==%                                                                                                                     ➜  vault git:(master) ✗

Decrypting

➜  vault git:(master) ✗ python retrieve.py --name tuxlabs --key 01234567890987654321abcd
Well I could store a password or b00bs, cause everyone likes b00bs even the newbs
➜  vault git:(master) ✗

Alternatively, you can hide the key from the command line, by not specifying it.

➜  vault  python store.py --name JimmyJohns --text 'Really Fast'
Key:
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ= was stored as JimmyJohns
➜  vault  python retrieve.py --name JimmyJohns
Key:
Really Fast
➜  vault  cat .JimmyJohns 
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ=%                                                                                                                                                                                                                                 ➜  vault

Clone Vault on github

Enjoy !
Jason Riedel

Runner Features Have Been Updated !

Runner Reminder

Runner is a command line tool for running commands on thousands of devices that support SSH. I wrote Runner and use it every single day, because unlike Ansible, Runner truly has no dependencies on the client or server side other than SSH. I have used Runner to build entire datacenters, so it is proven and tested and has a lot of well thought out features, which brings me to todays post. Since I initially debuted Runner I have added a lot of features, but I had yet to check them into github, until now. Here is a run down of Runner’s features.

Features

Runner takes your login credentials & doesn’t require you to setup SSH keys on the client machines/devices.
Runner can be used through a bastion/jump host via an SSH tunnel (see prunner.py)
Runner reads it’s main host list from a file ~/.runner/hosts/hosts-all
Runner can accept custom hosts lists via -f
-e can be used to echo a command before it is run, this is useful for running commands on F5 load balancers for example, when no output is returned on success.
-T will allow you to tune the number of threads, but be careful you can easily exhaust your system or site resources (I.E. do NOT DOS your LDAP authentication servers by trying to do hundreds of threads across thousands of machines, unless you know they can handle it 😉 ).
-s is for sudo for those users who have permissions in the sudoers file.
-1 reduces any host list down to one host per pool. It uses a regex, which you will likely have to modify for your own host / device naming standard.
-r can be used to supply a regular expression for matching hosts. Remember sometimes you have to quote the regex and/or escape the shell when using certain characters.
-c will run a single command on many hosts, but -cf will run a series of commands listed in a file on any hosts specified. This is particularly useful for automations. For example, I used it to build out load balancer virtuals and pools on an F5.
-p enables you to break apart the number of hosts to run at a time using a percentage. This is a handy & more humanized way to ensure you do not kill your machine or the infrastructure you are managing when you crank threads through the roof 😉

Now that I have taken the time to explain some of those cool features, here’s an example of what it looks like in action.

Runner Demo

Host List

➜  ~  runner -l -r tuxlabs                               
tuxlabs.com
old.tuxlabs.com

There were 2 hosts listed.
➜

Basic Run Using Only -c, -u and defaults

Note: User defaults to the user you are logged in as if you don’t specify -u . Since I am logged in as ‘jriedel’ I have specified the user tuxninja instead.

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),27(sudo)
old.tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),114(sambashare),1001(admin)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
➜  ~

The Same Run Using Sudo

Note: I just realized if you do not prompt for a password for sudo it will fail, I will have to fix that ! Whoops ! P.S. You should always prompt for a password when using sudo !

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja -s
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INFO]: SUDO IS ON
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)
old.tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
➜  ~

Runner with a command file in super quiet mode !

➜  ~  cat lets-run-these 
uptime
who
date
uptime -s
➜  ~  

➜  ~  runner -r tuxlabs -cf lets-run-these -T 2 -p 50 -qq -u tuxninja
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com:  05:17:51 up 14 days,  4:45,  0 users,  load average: 0.00, 0.01, 0.05
old.tuxlabs.com:  03:17:52 up 80 days, 47 min,  1 user,  load average: 0.42, 0.69, 0.78
old.tuxlabs.com: root     pts/0        2015-08-25 02:23 (173.224.162.99)
tuxlabs.com: Tue Aug 25 05:17:52 EDT 2015
old.tuxlabs.com: Tue Aug 25 03:17:53 CST 2015
tuxlabs.com: 2015-08-11 00:32:18
old.tuxlabs.com: 2015-06-06 02:30:42
➜  ~

Example of a simple regex & a failure

➜  ~  runner -r old.* -l
zsh: no matches found: old.*
➜  ~  

➜  ~  runner -r 'old.*' -l
old.tuxlabs.com

There was 1 host listed.
➜  ~

I hope you enjoyed the overview and new features. You can clone Runner on github.

Enjoy,
Jason Riedel

Object-Oriented Programming With Python : Encapsulation (1/3)

Justification For Learning OOP

linux-python-logo1 Since 1995, I have written some form of code, but over the years my career has never taken shape with a primary focus on development. Instead in my various roles of Systems, Network, Application and Datacenter engineering & architecture, I have found excuses to let the inner code hacker in me, out. Allowing me to consistently pursue my passion for coding and automation within the scope of my daily duties. However, this approach has never allowed me to work on a development team and thus the necessity for using object oriented programming rarely has occurred.

Sure I’ve written classes and modules for re-use, but I have never had to place a tremendous amount of forethought into the design for flexibility or inflexibility of the interfaces those classes provide because I was the only one using them. This hasn’t change, but as of late, I find myself realizing my career may be ready for a change. It is likely the next step for me in my career is a full blown software development role. This is partly because of boredom and mastery in my existing field after decades of experience, but mostly because of my own career planning and the apparent industry trends.

Simply put more jobs are moving to software development, and Systems Engineering is on a steady decline due to the advent of Cloud computing, Containers and more. Due to the amazing products, platforms and tools that are now available less and less Systems Engineers are required and I would say Systems Engineers that don’t code are already obsolete, whether they realize it yet or not.

In todays world if you are or are going to be a developer, knowing how to write object oriented code is a requirement and more importantly thinking ahead about who and how your code will be used becomes paramount when designing effective software. This article is intended to help you get a better understanding of how to write object oriented programming using Python.

Procedural Programming vs. Object Oriented Programming

Procedural programming is a more antiquated paradigm for software design that can be described as:

Code that processes data
The data we process is stored in variables
Then we create functions to process the data in those variables

In the object oriented programming paradigm:

We organize data into Objects
Functions become Methods
And The design of these objects & methods is specified in a Class, which is essentially a blue print.

Object oriented programming is the primary software design paradigm in existence today because it provides programmers with a way to effectively organize and share code for re-use while to protecting the integrity of the existing code. To understand how this works we need to jump right into it with The Three Pillars of Object Oriented Programming.

The Three Pillars (Encapsulation, Inheritance, and Polymorphism)

Encapsulation

Encapsulation is about ensuring the safe storage of data as attributes in an instance.

Encapsulation tells us that :

Data should only be accessed through instance methods.
Data should always be correct based on the validation requirement set in the class methods.
And Data should be safe from changes by external processes.

Great so what does that mean 🙂 It means we should be using ‘setter’ and ‘getter’ methods to access object attribute values. Here’s an example of a class that uses a setter and getter method (note the name setter & getter is not actually required) to set and get the variable/attribute value.

#!/usr/bin/env python

class Tuxlabs(object):
    def setter(self, website):
        self.website = website

    def getter(self):
        return self.website


tuxlabs = Tuxlabs()

tuxlabs.setter('http://www.tuxlabs.com')
print (tuxlabs.getter())

tuxlabs.setter('http://www.google.com')
print (tuxlabs.getter())

The output of this code is:

➜  tuxlabs  python example.py 
http://www.tuxlabs.com
http://www.google.com
➜  tuxlabs

It’s important to recognize that encapsulation is not enforced by Python. So a programmer using a class is not required to access the data through the getter and setter methods. For example, in the class above someone could set website, without ever interacting with the defined setter and getter classes. This is called breaking encapsulation, and it’s bad form / practice for a programmer to this because the class author & maintainer can no longer validate the data that is being accessed and this can cause unforeseen problems with a program that is dependent on the class it is inheriting from. Here’s a modified example where we are breaking encapsulation.

#!/usr/bin/env python

class Tuxlabs(object):
    def setter(self, website):
        if 'http' in website:
            self.website = website
        else:
            print 'ERROR: Unable to set website to %s.\nRequired format is: http://www.example.com' % (website)

    def getter(self):
        output = 'Website is set to: ' + str(self.website) + '\n'
        return output

tuxlabs = Tuxlabs()

website = 'http://www.tuxlabs.com'
print "Attempting to set website to: \"%s\"" % (website)
tuxlabs.setter(website)
print (tuxlabs.getter())

website = 'http://www.google.com'
print "Attempting to set website to: \"%s\"" % (website)
tuxlabs.setter(website)
print (tuxlabs.getter())

website = 'I should not be accessing website in the class directly!, but since I am breaking encapsulation it does still work, it\'s just naughty!'
print "Attempting to set website to: \"%s\"" % (website)
tuxlabs.website = website
print tuxlabs.website ## Printing without the getter doh !

print ""
website = 'This is not a website, so an error is thrown!'
print "Attempting to set website to: \"%s\"" % (website)
tuxlabs.setter(website)

Here is the output of the above example:

➜ tuxlabs python example.py
Attempting to set website to: "http://www.tuxlabs.com"
Website is set to: http://www.tuxlabs.com

Attempting to set website to: "http://www.google.com"
Website is set to: http://www.google.com

Attempting to set website to: "I should not be accessing website in the class directly!, but since I am breaking encapsulation it does still work, it's just naughty!"
I should not be accessing website in the class directly!, but since I am breaking encapsulation it does still work, it's just naughty!

Attempting to set website to: "This is not a website, so an error is thrown!"
ERROR: Unable to set website to This is not a website, so an error is thrown!.
Required format is: http://www.example.com
➜ tuxlabs

A number of things have changed.

We added a variable website to make the example more clear and easy to follow.
We added validation to our setter() method, it now checks to make sure ‘http’ is contained in the value that we are setting for website.
If ‘http’ is not in website the setter() method will throw an error and will not set website!
We added some additional formatting to our getter() method so we can tell when it is being used verses when encapsulation is being broken. getter() now pre-pends any website passed with ‘Website is now set to:’

So let’s walk thru how these changes effected the program and what actually happened. First, we tried to set website to ‘http://www.tuxlabs.com’ and then we changed it to ‘http://www.google.com’. In both of these examples we followed the rules of encapsulation, setting and getting the attribute value through methods in the class. These examples are correct and set website to the value passed and outputed them with the correct formatting we implemented in getter().

Next, we tried setting website to a really long string, without using setter() ! So we did tuxlabs.website = website directly on the instance of our class ! We also circumvented using getter() and just printed in exactly the same, broken way print tuxlabs.website. Unfortunately, as previously mentioned nothing requires the programmer to use our setter and getter methods directly, and thus website is not being checked for containing ‘http’ here and so it can printed our very long string (“I should not be accessing website in the class directly!, but since I am breaking encapsulation it does still work, it\’s just naughty!”).

The last and final example, is an example of encapsulations value proposition. We try to set website to something that does not contain ‘http’ and therefore is not a website. An error is thrown and website is not set, which is exactly what we want.

Encapsulation is simple, easy to follow and powerful when your programming on or with a team of developers that are going to share code amongst each other. In these scenarios encapsulation is obviously a must promoting collaboration through protecting programmers from potentially mis-using each others code. It should be easy to see now that breaking encapsulation is bad and very poor practice because code that perfectly functions one day, could be broken the next without any changes from you, because you are violating the contract that encapsulation methods of the class provide.

This concludes our learning about encapsulation. Still to come, Inheritance, Polymorphism, and some real world examples!

Until next time…Keep learning,
Jason Riedel