tuxninja

Tuxninja aka Jason Riedel has worked as a Systems & Network Administrator and Code Hacker since 1999. Since 2005 he has worked for PayPal with a focus on Operations Architecture. He is also CEO of Tuxlabs LLC where he dedicates his time to the experimentation and close study of new technologies and programming languages.

Fun with Python, Tabular & AWS IP ranges

Leave a Comment / AWS, Cloud, Programming, Python / tuxninja

I have been spending a lot of time designing a Hybrid Cloud that consists of Openstack and public cloud platforms. In particular I have been spending a lot of time designing the AWS portion of the Hybrid Cloud Platform. Today I found myself continually needing to look up AWS public address space and then parsing out regions & services. Then I remembered something a mentor of mine told me…

if you are going to do something more than once, there is probably value in automating it.

I love writing command line tools and thus a short Python script was born. Since I rarely share Python code, even though I didn’t spend a lot time on this, and I certainly didn’t optimize it for DRY etc. I am sharing it anyway for others to use, enjoy and hack on,

but mainly to learn, which is…The entire purpose of the Tuxlabs site

I should mention I have strong views about when to use Python vs. Go a language I find myself writing in more and more and this tool falls under my rules for things that I should write in Go. So later as a follow up I will likely re-code this in Go and post the code for review & learning. For now here’s the Python code, enjoy !

Listing all IP Ranges

(env) ➜  aws python aws-ranges.py
region          ip_prefix          service
--------------  -----------------  --------------------
us-east-1       23.20.0.0/14       AMAZON
ap-northeast-1  27.0.0.0/22        AMAZON
ap-southeast-1  43.250.192.0/24    AMAZON
ap-southeast-1  43.250.193.0/24    AMAZON
eu-west-1       46.51.128.0/18     AMAZON
eu-west-1       46.51.192.0/20     AMAZON
ap-southeast-1  46.51.216.0/21     AMAZON
ap-northeast-1  46.51.224.0/19     AMAZON
eu-west-1       46.137.0.0/17      AMAZON
eu-west-1       46.137.128.0/18    AMAZON
ap-southeast-1  46.137.192.0/19    AMAZON
ap-southeast-1  46.137.224.0/19    AMAZON
us-east-1       50.16.0.0/15       AMAZON
us-west-1       50.18.0.0/16       AMAZON
us-east-1       50.19.0.0/16       AMAZON
us-west-2       50.112.0.0/16      AMAZON
us-east-1       52.0.0.0/15        AMAZON
us-east-1       52.2.0.0/15        AMAZON
us-east-1       52.4.0.0/14        AMAZON
us-west-1       52.8.0.0/16        AMAZON
us-west-1       52.9.0.0/16        AMAZON
us-west-2       52.10.0.0/15       AMAZON
us-west-2       52.12.0.0/15       AMAZON
eu-west-1       52.16.0.0/15       AMAZON
eu-west-1       52.18.0.0/15       AMAZON
us-east-1       52.20.0.0/14       AMAZON
us-west-2       52.24.0.0/14       AMAZON
eu-central-1    52.28.0.0/16       AMAZON
eu-central-1    52.29.0.0/16       AMAZON
eu-west-1       52.30.0.0/15       AMAZON
us-west-2       52.32.0.0/14       AMAZON
us-west-2       52.36.0.0/14       AMAZON
us-west-2       52.40.0.0/14       AMAZON
eu-west-1       52.48.0.0/14       AMAZON
us-west-1       52.52.0.0/15       AMAZON
eu-central-1    52.58.0.0/15       AMAZON
ap-southeast-2  52.62.0.0/15       AMAZON
ap-southeast-2  52.64.0.0/17       AMAZON
ap-southeast-2  52.64.128.0/17     AMAZON
ap-southeast-2  52.65.0.0/16       AMAZON
sa-east-1       52.67.0.0/16       AMAZON
ap-northeast-1  52.68.0.0/15       AMAZON
us-east-1       52.70.0.0/15       AMAZON
us-east-1       52.72.0.0/15       AMAZON
ap-southeast-1  52.74.0.0/16       AMAZON
ap-southeast-1  52.76.0.0/17       AMAZON
ap-southeast-1  52.76.128.0/17     AMAZON
ap-southeast-1  52.77.0.0/16       AMAZON
ap-northeast-2  52.79.0.0/16       AMAZON
GLOBAL          52.84.0.0/15       AMAZON
us-east-1       52.86.0.0/15       AMAZON
us-west-2       52.88.0.0/15       AMAZON
us-east-1       52.90.0.0/15       AMAZON
ap-northeast-2  52.92.0.0/20       AMAZON
us-east-1       52.92.16.0/20      AMAZON
us-west-2       52.92.32.0/22      AMAZON
sa-east-1       52.92.39.0/24      AMAZON
eu-west-1       52.92.40.0/21      AMAZON
us-west-1       52.92.48.0/22      AMAZON
ap-southeast-2  52.92.52.0/22      AMAZON
ap-southeast-1  52.92.56.0/22      AMAZON
ap-northeast-1  52.92.60.0/22      AMAZON
sa-east-1       52.92.64.0/22      AMAZON
eu-central-1    52.92.68.0/22      AMAZON
sa-east-1       52.92.72.0/22      AMAZON
us-gov-west-1   52.92.252.0/22     AMAZON
eu-west-1       52.93.0.0/24       AMAZON
us-east-1       52.93.1.0/24       AMAZON
eu-west-1       52.93.2.0/24       AMAZON
us-east-1       52.93.3.0/24       AMAZON
ap-southeast-1  52.93.8.0/22       AMAZON
us-east-1       52.94.0.0/22       AMAZON
eu-west-1       52.94.5.0/24       AMAZON
ap-northeast-2  52.94.6.0/24       AMAZON
sa-east-1       52.94.7.0/24       AMAZON
ap-northeast-1  52.94.8.0/24       AMAZON
us-gov-west-1   52.94.9.0/24       AMAZON
us-west-2       52.94.10.0/24      AMAZON
ap-southeast-1  52.94.11.0/24      AMAZON
us-west-1       52.94.12.0/24      AMAZON
ap-southeast-2  52.94.13.0/24      AMAZON
us-west-1       52.94.14.0/24      AMAZON
us-east-1       52.94.254.0/23     AMAZON
ap-northeast-1  52.95.30.0/23      AMAZON
ap-northeast-1  52.95.34.0/24      AMAZON
ap-southeast-1  52.95.35.0/24      AMAZON
ap-southeast-2  52.95.36.0/22      AMAZON
us-east-1       52.95.48.0/22      AMAZON
us-east-1       52.95.52.0/22      AMAZON
ap-northeast-1  52.95.56.0/22      AMAZON
eu-west-1       52.95.60.0/24      AMAZON
eu-west-1       52.95.61.0/24      AMAZON
us-east-1       52.95.62.0/24      AMAZON
us-east-1       52.95.63.0/24      AMAZON
ap-northeast-2  52.95.192.0/20     AMAZON
ap-southeast-1  52.95.212.0/22     AMAZON
sa-east-1       52.95.240.0/24     AMAZON
ap-southeast-2  52.95.241.0/24     AMAZON
ap-southeast-1  52.95.242.0/24     AMAZON
ap-northeast-1  52.95.243.0/24     AMAZON
eu-west-1       52.95.244.0/24     AMAZON
us-east-1       52.95.245.0/24     AMAZON
us-west-1       52.95.246.0/24     AMAZON
us-west-2       52.95.247.0/24     AMAZON
eu-central-1    52.95.248.0/24     AMAZON
cn-north-1      52.95.249.0/24     AMAZON
ap-northeast-2  52.95.252.0/24     AMAZON
sa-east-1       52.95.255.0/28     AMAZON
ap-southeast-2  52.95.255.16/28    AMAZON
ap-southeast-1  52.95.255.32/28    AMAZON
ap-northeast-1  52.95.255.48/28    AMAZON
eu-west-1       52.95.255.64/28    AMAZON
us-east-1       52.95.255.80/28    AMAZON
us-west-1       52.95.255.96/28    AMAZON
us-west-2       52.95.255.112/28   AMAZON
eu-central-1    52.95.255.128/28   AMAZON
cn-north-1      52.95.255.144/28   AMAZON
ap-northeast-1  52.192.0.0/15      AMAZON
ap-northeast-1  52.196.0.0/14      AMAZON
us-east-1       52.200.0.0/13      AMAZON
eu-west-1       52.208.0.0/13      AMAZON
ap-southeast-1  52.220.0.0/15      AMAZON
ap-northeast-1  54.64.0.0/15       AMAZON
ap-southeast-2  54.66.0.0/16       AMAZON
us-west-1       54.67.0.0/16       AMAZON
us-west-2       54.68.0.0/14       AMAZON
eu-west-1       54.72.0.0/15       AMAZON
eu-west-1       54.74.0.0/15       AMAZON
eu-west-1       54.76.0.0/15       AMAZON
eu-west-1       54.78.0.0/16       AMAZON
ap-southeast-2  54.79.0.0/16       AMAZON
us-east-1       54.80.0.0/13       AMAZON
us-east-1       54.88.0.0/14       AMAZON
ap-northeast-1  54.92.0.0/17       AMAZON
us-east-1       54.92.128.0/17     AMAZON
eu-central-1    54.93.0.0/16       AMAZON
sa-east-1       54.94.0.0/16       AMAZON
ap-northeast-1  54.95.0.0/16       AMAZON
us-east-1       54.144.0.0/14      AMAZON
us-west-2       54.148.0.0/15      AMAZON
ap-northeast-1  54.150.0.0/16      AMAZON
us-west-1       54.151.0.0/17      AMAZON
ap-southeast-1  54.151.128.0/17    AMAZON
us-east-1       54.152.0.0/16      AMAZON
us-west-1       54.153.0.0/17      AMAZON
ap-southeast-2  54.153.128.0/17    AMAZON
eu-west-1       54.154.0.0/16      AMAZON
eu-west-1       54.155.0.0/16      AMAZON
us-east-1       54.156.0.0/14      AMAZON
us-east-1       54.160.0.0/13      AMAZON
ap-northeast-1  54.168.0.0/16      AMAZON
ap-southeast-1  54.169.0.0/16      AMAZON
eu-west-1       54.170.0.0/15      AMAZON
us-east-1       54.172.0.0/15      AMAZON
us-east-1       54.174.0.0/15      AMAZON
us-west-1       54.176.0.0/15      AMAZON
ap-northeast-1  54.178.0.0/16      AMAZON
ap-southeast-1  54.179.0.0/16      AMAZON
GLOBAL          54.182.0.0/16      AMAZON
us-west-1       54.183.0.0/16      AMAZON
us-west-2       54.184.0.0/13      AMAZON
GLOBAL          54.192.0.0/16      AMAZON
us-west-1       54.193.0.0/16      AMAZON
eu-west-1       54.194.0.0/15      AMAZON
us-east-1       54.196.0.0/15      AMAZON
us-east-1       54.198.0.0/16      AMAZON
ap-northeast-1  54.199.0.0/16      AMAZON
us-west-2       54.200.0.0/15      AMAZON
us-west-2       54.202.0.0/15      AMAZON
us-east-1       54.204.0.0/15      AMAZON
ap-southeast-2  54.206.0.0/16      AMAZON
sa-east-1       54.207.0.0/16      AMAZON
us-east-1       54.208.0.0/15      AMAZON
us-east-1       54.210.0.0/15      AMAZON
us-west-2       54.212.0.0/15      AMAZON
us-west-2       54.214.0.0/16      AMAZON
us-west-1       54.215.0.0/16      AMAZON
eu-west-1       54.216.0.0/15      AMAZON
us-west-2       54.218.0.0/16      AMAZON
us-west-1       54.219.0.0/16      AMAZON
eu-west-1       54.220.0.0/16      AMAZON
us-east-1       54.221.0.0/16      AMAZON
cn-north-1      54.222.0.0/19      AMAZON
cn-north-1      54.222.128.0/17    AMAZON
cn-north-1      54.223.0.0/16      AMAZON
us-east-1       54.224.0.0/15      AMAZON
us-east-1       54.226.0.0/15      AMAZON
eu-west-1       54.228.0.0/16      AMAZON
eu-west-1       54.229.0.0/16      AMAZON
GLOBAL          54.230.0.0/16      AMAZON
us-east-1       54.231.0.0/17      AMAZON
eu-west-1       54.231.128.0/19    AMAZON
us-west-2       54.231.160.0/19    AMAZON
eu-central-1    54.231.192.0/20    AMAZON
cn-north-1      54.231.208.0/20    AMAZON
ap-northeast-1  54.231.224.0/21    AMAZON
us-west-1       54.231.232.0/21    AMAZON
ap-southeast-1  54.231.240.0/22    AMAZON
us-east-1       54.231.244.0/22    AMAZON
ap-southeast-2  54.231.248.0/22    AMAZON
ap-southeast-2  54.231.252.0/24    AMAZON
sa-east-1       54.231.253.0/24    AMAZON
us-gov-west-1   54.231.254.0/24    AMAZON
sa-east-1       54.232.0.0/16      AMAZON
sa-east-1       54.233.0.0/18      AMAZON
sa-east-1       54.233.64.0/18     AMAZON
sa-east-1       54.233.128.0/17    AMAZON
us-east-1       54.234.0.0/15      AMAZON
us-east-1       54.236.0.0/15      AMAZON
ap-northeast-1  54.238.0.0/16      AMAZON
us-west-2       54.239.2.0/23      AMAZON
eu-central-1    54.239.4.0/22      AMAZON
us-east-1       54.239.8.0/21      AMAZON
us-east-1       54.239.16.0/20     AMAZON
eu-west-1       54.239.32.0/21     AMAZON
us-west-2       54.239.48.0/22     AMAZON
ap-northeast-1  54.239.52.0/23     AMAZON
eu-central-1    54.239.54.0/23     AMAZON
eu-central-1    54.239.56.0/21     AMAZON
ap-northeast-1  54.239.96.0/24     AMAZON
us-east-1       54.239.98.0/24     AMAZON
eu-west-1       54.239.99.0/24     AMAZON
eu-west-1       54.239.100.0/23    AMAZON
us-east-1       54.239.104.0/23    AMAZON
us-east-1       54.239.108.0/22    AMAZON
eu-west-1       54.239.114.0/24    AMAZON
ap-northeast-2  54.239.116.0/22    AMAZON
ap-northeast-2  54.239.120.0/21    AMAZON
GLOBAL          54.239.128.0/18    AMAZON
GLOBAL          54.239.192.0/19    AMAZON
GLOBAL          54.240.128.0/18    AMAZON
ap-southeast-2  54.240.192.0/22    AMAZON
us-east-1       54.240.196.0/24    AMAZON
eu-west-1       54.240.197.0/24    AMAZON
us-west-1       54.240.198.0/24    AMAZON
ap-southeast-1  54.240.199.0/24    AMAZON
ap-northeast-1  54.240.200.0/24    AMAZON
us-east-1       54.240.202.0/24    AMAZON
ap-southeast-2  54.240.203.0/24    AMAZON
ap-southeast-2  54.240.204.0/22    AMAZON
us-east-1       54.240.208.0/22    AMAZON
us-west-1       54.240.212.0/22    AMAZON
us-east-1       54.240.216.0/22    AMAZON
eu-west-1       54.240.220.0/22    AMAZON
ap-northeast-1  54.240.225.0/24    AMAZON
ap-southeast-1  54.240.226.0/24    AMAZON
ap-southeast-1  54.240.227.0/24    AMAZON
us-east-1       54.240.228.0/23    AMAZON
us-west-2       54.240.230.0/23    AMAZON
us-east-1       54.240.232.0/22    AMAZON
ap-northeast-2  54.240.236.0/22    AMAZON
eu-central-1    54.240.240.0/24    AMAZON
sa-east-1       54.240.244.0/22    AMAZON
us-west-2       54.240.248.0/21    AMAZON
us-west-1       54.241.0.0/16      AMAZON
us-east-1       54.242.0.0/15      AMAZON
us-west-2       54.244.0.0/16      AMAZON
us-west-2       54.245.0.0/16      AMAZON
eu-west-1       54.246.0.0/16      AMAZON
eu-west-1       54.247.0.0/16      AMAZON
ap-northeast-1  54.248.0.0/15      AMAZON
ap-northeast-1  54.250.0.0/16      AMAZON
ap-southeast-1  54.251.0.0/16      AMAZON
ap-southeast-2  54.252.0.0/16      AMAZON
ap-southeast-2  54.253.0.0/16      AMAZON
ap-southeast-1  54.254.0.0/16      AMAZON
ap-southeast-1  54.255.0.0/16      AMAZON
us-east-1       67.202.0.0/18      AMAZON
us-east-1       72.21.192.0/19     AMAZON
us-east-1       72.44.32.0/19      AMAZON
us-east-1       75.101.128.0/17    AMAZON
eu-west-1       79.125.0.0/17      AMAZON
eu-west-1       87.238.80.0/21     AMAZON
us-gov-west-1   96.127.0.0/17      AMAZON
ap-northeast-1  103.4.8.0/21       AMAZON
ap-southeast-1  103.246.148.0/23   AMAZON
ap-northeast-1  103.246.150.0/23   AMAZON
us-east-1       107.20.0.0/14      AMAZON
ap-southeast-1  122.248.192.0/18   AMAZON
us-east-1       172.96.97.0/24     AMAZON
us-east-1       174.129.0.0/16     AMAZON
ap-southeast-1  175.41.128.0/18    AMAZON
ap-northeast-1  175.41.192.0/18    AMAZON
ap-northeast-1  176.32.64.0/19     AMAZON
us-east-1       176.32.96.0/21     AMAZON
eu-west-1       176.32.104.0/21    AMAZON
us-west-1       176.32.112.0/21    AMAZON
us-east-1       176.32.120.0/22    AMAZON
us-west-2       176.32.125.0/25    AMAZON
ap-northeast-1  176.34.0.0/19      AMAZON
ap-northeast-1  176.34.32.0/19     AMAZON
eu-west-1       176.34.64.0/18     AMAZON
eu-west-1       176.34.128.0/17    AMAZON
sa-east-1       177.71.128.0/17    AMAZON
sa-east-1       177.72.240.0/21    AMAZON
eu-west-1       178.236.0.0/20     AMAZON
us-west-1       184.72.0.0/18      AMAZON
us-east-1       184.72.64.0/18     AMAZON
us-east-1       184.72.128.0/17    AMAZON
us-east-1       184.73.0.0/16      AMAZON
us-west-1       184.169.128.0/17   AMAZON
eu-west-1       185.48.120.0/22    AMAZON
ap-southeast-1  203.83.220.0/22    AMAZON
us-west-1       204.236.128.0/18   AMAZON
us-east-1       204.236.192.0/18   AMAZON
us-west-1       204.246.160.0/22   AMAZON
GLOBAL          204.246.164.0/22   AMAZON
GLOBAL          204.246.168.0/22   AMAZON
GLOBAL          204.246.174.0/23   AMAZON
GLOBAL          204.246.176.0/20   AMAZON
GLOBAL          205.251.192.0/19   AMAZON
us-east-1       205.251.224.0/22   AMAZON
us-west-1       205.251.228.0/22   AMAZON
us-west-2       205.251.232.0/22   AMAZON
us-gov-west-1   205.251.236.0/22   AMAZON
us-east-1       205.251.240.0/22   AMAZON
us-east-1       205.251.244.0/23   AMAZON
us-east-1       205.251.247.0/24   AMAZON
us-east-1       205.251.248.0/24   AMAZON
GLOBAL          205.251.249.0/24   AMAZON
GLOBAL          205.251.250.0/23   AMAZON
GLOBAL          205.251.252.0/23   AMAZON
GLOBAL          205.251.254.0/24   AMAZON
us-east-1       205.251.255.0/24   AMAZON
us-east-1       207.171.160.0/20   AMAZON
us-east-1       207.171.176.0/20   AMAZON
GLOBAL          216.137.32.0/19    AMAZON
us-east-1       216.182.224.0/20   AMAZON
us-east-1       23.20.0.0/14       EC2
eu-west-1       46.51.128.0/18     EC2
eu-west-1       46.51.192.0/20     EC2
ap-southeast-1  46.51.216.0/21     EC2
ap-northeast-1  46.51.224.0/19     EC2
eu-west-1       46.137.0.0/17      EC2
eu-west-1       46.137.128.0/18    EC2
ap-southeast-1  46.137.192.0/19    EC2
ap-southeast-1  46.137.224.0/19    EC2
us-east-1       50.16.0.0/15       EC2
us-west-1       50.18.0.0/16       EC2
us-east-1       50.19.0.0/16       EC2
us-west-2       50.112.0.0/16      EC2
us-east-1       52.0.0.0/15        EC2
us-east-1       52.2.0.0/15        EC2
us-east-1       52.4.0.0/14        EC2
us-west-1       52.8.0.0/16        EC2
us-west-1       52.9.0.0/16        EC2
us-west-2       52.10.0.0/15       EC2
us-west-2       52.12.0.0/15       EC2
eu-west-1       52.16.0.0/15       EC2
eu-west-1       52.18.0.0/15       EC2
us-east-1       52.20.0.0/14       EC2
us-west-2       52.24.0.0/14       EC2
eu-central-1    52.28.0.0/16       EC2
eu-central-1    52.29.0.0/16       EC2
eu-west-1       52.30.0.0/15       EC2
us-west-2       52.32.0.0/14       EC2
us-west-2       52.36.0.0/14       EC2
us-west-2       52.40.0.0/14       EC2
eu-west-1       52.48.0.0/14       EC2
us-west-1       52.52.0.0/15       EC2
eu-central-1    52.58.0.0/15       EC2
ap-southeast-2  52.62.0.0/15       EC2
ap-southeast-2  52.64.0.0/17       EC2
ap-southeast-2  52.64.128.0/17     EC2
ap-southeast-2  52.65.0.0/16       EC2
sa-east-1       52.67.0.0/16       EC2
ap-northeast-1  52.68.0.0/15       EC2
us-east-1       52.70.0.0/15       EC2
us-east-1       52.72.0.0/15       EC2
ap-southeast-1  52.74.0.0/16       EC2
ap-southeast-1  52.76.0.0/17       EC2
ap-southeast-1  52.76.128.0/17     EC2
ap-southeast-1  52.77.0.0/16       EC2
ap-northeast-2  52.79.0.0/16       EC2
us-east-1       52.86.0.0/15       EC2
us-west-2       52.88.0.0/15       EC2
us-east-1       52.90.0.0/15       EC2
sa-east-1       52.95.240.0/24     EC2
ap-southeast-2  52.95.241.0/24     EC2
ap-southeast-1  52.95.242.0/24     EC2
ap-northeast-1  52.95.243.0/24     EC2
eu-west-1       52.95.244.0/24     EC2
us-east-1       52.95.245.0/24     EC2
us-west-1       52.95.246.0/24     EC2
us-west-2       52.95.247.0/24     EC2
eu-central-1    52.95.248.0/24     EC2
cn-north-1      52.95.249.0/24     EC2
ap-northeast-2  52.95.252.0/24     EC2
sa-east-1       52.95.255.0/28     EC2
ap-southeast-2  52.95.255.16/28    EC2
ap-southeast-1  52.95.255.32/28    EC2
ap-northeast-1  52.95.255.48/28    EC2
eu-west-1       52.95.255.64/28    EC2
us-east-1       52.95.255.80/28    EC2
us-west-1       52.95.255.96/28    EC2
us-west-2       52.95.255.112/28   EC2
eu-central-1    52.95.255.128/28   EC2
cn-north-1      52.95.255.144/28   EC2
ap-northeast-1  52.192.0.0/15      EC2
ap-northeast-1  52.196.0.0/14      EC2
us-east-1       52.200.0.0/13      EC2
eu-west-1       52.208.0.0/13      EC2
ap-southeast-1  52.220.0.0/15      EC2
ap-northeast-1  54.64.0.0/15       EC2
ap-southeast-2  54.66.0.0/16       EC2
us-west-1       54.67.0.0/16       EC2
us-west-2       54.68.0.0/14       EC2
eu-west-1       54.72.0.0/15       EC2
eu-west-1       54.74.0.0/15       EC2
eu-west-1       54.76.0.0/15       EC2
eu-west-1       54.78.0.0/16       EC2
ap-southeast-2  54.79.0.0/16       EC2
us-east-1       54.80.0.0/13       EC2
us-east-1       54.88.0.0/14       EC2
ap-northeast-1  54.92.0.0/17       EC2
us-east-1       54.92.128.0/17     EC2
eu-central-1    54.93.0.0/16       EC2
sa-east-1       54.94.0.0/16       EC2
ap-northeast-1  54.95.0.0/16       EC2
us-east-1       54.144.0.0/14      EC2
us-west-2       54.148.0.0/15      EC2
ap-northeast-1  54.150.0.0/16      EC2
us-west-1       54.151.0.0/17      EC2
ap-southeast-1  54.151.128.0/17    EC2
us-east-1       54.152.0.0/16      EC2
us-west-1       54.153.0.0/17      EC2
ap-southeast-2  54.153.128.0/17    EC2
eu-west-1       54.154.0.0/16      EC2
eu-west-1       54.155.0.0/16      EC2
us-east-1       54.156.0.0/14      EC2
us-east-1       54.160.0.0/13      EC2
ap-northeast-1  54.168.0.0/16      EC2
ap-southeast-1  54.169.0.0/16      EC2
eu-west-1       54.170.0.0/15      EC2
us-east-1       54.172.0.0/15      EC2
us-east-1       54.174.0.0/15      EC2
us-west-1       54.176.0.0/15      EC2
ap-northeast-1  54.178.0.0/16      EC2
ap-southeast-1  54.179.0.0/16      EC2
us-west-1       54.183.0.0/16      EC2
us-west-2       54.184.0.0/13      EC2
us-west-1       54.193.0.0/16      EC2
eu-west-1       54.194.0.0/15      EC2
us-east-1       54.196.0.0/15      EC2
us-east-1       54.198.0.0/16      EC2
ap-northeast-1  54.199.0.0/16      EC2
us-west-2       54.200.0.0/15      EC2
us-west-2       54.202.0.0/15      EC2
us-east-1       54.204.0.0/15      EC2
ap-southeast-2  54.206.0.0/16      EC2
sa-east-1       54.207.0.0/16      EC2
us-east-1       54.208.0.0/15      EC2
us-east-1       54.210.0.0/15      EC2
us-west-2       54.212.0.0/15      EC2
us-west-2       54.214.0.0/16      EC2
us-west-1       54.215.0.0/16      EC2
eu-west-1       54.216.0.0/15      EC2
us-west-2       54.218.0.0/16      EC2
us-west-1       54.219.0.0/16      EC2
eu-west-1       54.220.0.0/16      EC2
us-east-1       54.221.0.0/16      EC2
cn-north-1      54.222.128.0/17    EC2
cn-north-1      54.223.0.0/16      EC2
us-east-1       54.224.0.0/15      EC2
us-east-1       54.226.0.0/15      EC2
eu-west-1       54.228.0.0/16      EC2
eu-west-1       54.229.0.0/16      EC2
sa-east-1       54.232.0.0/16      EC2
sa-east-1       54.233.0.0/18      EC2
sa-east-1       54.233.64.0/18     EC2
sa-east-1       54.233.128.0/17    EC2
us-east-1       54.234.0.0/15      EC2
us-east-1       54.236.0.0/15      EC2
ap-northeast-1  54.238.0.0/16      EC2
us-west-1       54.241.0.0/16      EC2
us-east-1       54.242.0.0/15      EC2
us-west-2       54.244.0.0/16      EC2
us-west-2       54.245.0.0/16      EC2
eu-west-1       54.246.0.0/16      EC2
eu-west-1       54.247.0.0/16      EC2
ap-northeast-1  54.248.0.0/15      EC2
ap-northeast-1  54.250.0.0/16      EC2
ap-southeast-1  54.251.0.0/16      EC2
ap-southeast-2  54.252.0.0/16      EC2
ap-southeast-2  54.253.0.0/16      EC2
ap-southeast-1  54.254.0.0/16      EC2
ap-southeast-1  54.255.0.0/16      EC2
us-east-1       67.202.0.0/18      EC2
us-east-1       72.44.32.0/19      EC2
us-east-1       75.101.128.0/17    EC2
eu-west-1       79.125.0.0/17      EC2
us-gov-west-1   96.127.0.0/17      EC2
ap-northeast-1  103.4.8.0/21       EC2
us-east-1       107.20.0.0/14      EC2
ap-southeast-1  122.248.192.0/18   EC2
us-east-1       174.129.0.0/16     EC2
ap-southeast-1  175.41.128.0/18    EC2
ap-northeast-1  175.41.192.0/18    EC2
ap-northeast-1  176.32.64.0/19     EC2
ap-northeast-1  176.34.0.0/19      EC2
ap-northeast-1  176.34.32.0/19     EC2
eu-west-1       176.34.64.0/18     EC2
eu-west-1       176.34.128.0/17    EC2
sa-east-1       177.71.128.0/17    EC2
us-west-1       184.72.0.0/18      EC2
us-east-1       184.72.64.0/18     EC2
us-east-1       184.72.128.0/17    EC2
us-east-1       184.73.0.0/16      EC2
us-west-1       184.169.128.0/17   EC2
eu-west-1       185.48.120.0/22    EC2
us-west-1       204.236.128.0/18   EC2
us-east-1       204.236.192.0/18   EC2
us-east-1       216.182.224.0/20   EC2
GLOBAL          205.251.192.0/21   ROUTE53
us-west-1       54.183.255.128/26  ROUTE53_HEALTHCHECKS
eu-west-1       54.228.16.0/26     ROUTE53_HEALTHCHECKS
sa-east-1       54.232.40.64/26    ROUTE53_HEALTHCHECKS
us-west-1       54.241.32.64/26    ROUTE53_HEALTHCHECKS
us-east-1       54.243.31.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.244.52.192/26   ROUTE53_HEALTHCHECKS
us-west-2       54.245.168.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.248.220.0/26    ROUTE53_HEALTHCHECKS
ap-northeast-1  54.250.253.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.251.31.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.79.128/26   ROUTE53_HEALTHCHECKS
ap-southeast-2  54.252.254.192/26  ROUTE53_HEALTHCHECKS
ap-southeast-1  54.255.254.192/26  ROUTE53_HEALTHCHECKS
us-east-1       107.23.255.0/26    ROUTE53_HEALTHCHECKS
eu-west-1       176.34.159.192/26  ROUTE53_HEALTHCHECKS
sa-east-1       177.71.207.128/26  ROUTE53_HEALTHCHECKS
GLOBAL          52.84.0.0/15       CLOUDFRONT
GLOBAL          54.182.0.0/16      CLOUDFRONT
GLOBAL          54.192.0.0/16      CLOUDFRONT
GLOBAL          54.230.0.0/16      CLOUDFRONT
GLOBAL          54.239.128.0/18    CLOUDFRONT
GLOBAL          54.239.192.0/19    CLOUDFRONT
GLOBAL          54.240.128.0/18    CLOUDFRONT
GLOBAL          204.246.164.0/22   CLOUDFRONT
GLOBAL          204.246.168.0/22   CLOUDFRONT
GLOBAL          204.246.174.0/23   CLOUDFRONT
GLOBAL          204.246.176.0/20   CLOUDFRONT
GLOBAL          205.251.192.0/19   CLOUDFRONT
GLOBAL          205.251.249.0/24   CLOUDFRONT
GLOBAL          205.251.250.0/23   CLOUDFRONT
GLOBAL          205.251.252.0/23   CLOUDFRONT
GLOBAL          205.251.254.0/24   CLOUDFRONT
GLOBAL          216.137.32.0/19    CLOUDFRONT
(env) ➜  aws

Filtering

(env) ➜  aws python aws-ranges.py -r sa-east-1
region     ip_prefix          service
---------  -----------------  --------------------
sa-east-1  52.67.0.0/16       AMAZON
sa-east-1  52.92.39.0/24      AMAZON
sa-east-1  52.92.64.0/22      AMAZON
sa-east-1  52.92.72.0/22      AMAZON
sa-east-1  52.94.7.0/24       AMAZON
sa-east-1  52.95.240.0/24     AMAZON
sa-east-1  52.95.255.0/28     AMAZON
sa-east-1  54.94.0.0/16       AMAZON
sa-east-1  54.207.0.0/16      AMAZON
sa-east-1  54.231.253.0/24    AMAZON
sa-east-1  54.232.0.0/16      AMAZON
sa-east-1  54.233.0.0/18      AMAZON
sa-east-1  54.233.64.0/18     AMAZON
sa-east-1  54.233.128.0/17    AMAZON
sa-east-1  54.240.244.0/22    AMAZON
sa-east-1  177.71.128.0/17    AMAZON
sa-east-1  177.72.240.0/21    AMAZON
sa-east-1  52.67.0.0/16       EC2
sa-east-1  52.95.240.0/24     EC2
sa-east-1  52.95.255.0/28     EC2
sa-east-1  54.94.0.0/16       EC2
sa-east-1  54.207.0.0/16      EC2
sa-east-1  54.232.0.0/16      EC2
sa-east-1  54.233.0.0/18      EC2
sa-east-1  54.233.64.0/18     EC2
sa-east-1  54.233.128.0/17    EC2
sa-east-1  177.71.128.0/17    EC2
sa-east-1  54.232.40.64/26    ROUTE53_HEALTHCHECKS
sa-east-1  177.71.207.128/26  ROUTE53_HEALTHCHECKS
(env) ➜  aws python aws-ranges.py -r sa-east-1 -s ec2
region     ip_prefix        service
---------  ---------------  ---------
sa-east-1  52.67.0.0/16     EC2
sa-east-1  52.95.240.0/24   EC2
sa-east-1  52.95.255.0/28   EC2
sa-east-1  54.94.0.0/16     EC2
sa-east-1  54.207.0.0/16    EC2
sa-east-1  54.232.0.0/16    EC2
sa-east-1  54.233.0.0/18    EC2
sa-east-1  54.233.64.0/18   EC2
sa-east-1  54.233.128.0/17  EC2
sa-east-1  177.71.128.0/17  EC2
(env) ➜  aws

The code

#!/usr/bin/env python
#March 8th 2016

__author__ = 'Jason Riedel'
__description__ = 'Grabs IP ranges from amazon'
__version__ = '1.0'

url = 'https://ip-ranges.amazonaws.com/ip-ranges.json'

import requests
import json
import argparse
from tabulate import tabulate

parser = argparse.ArgumentParser()
parser.add_argument('-r', '--region', action="store", dest="regionFilter", required=False, help="Region to filter/return data for: us-west, ap-southeast-1, us-east-1")
parser.add_argument('-s', '--service', action="store", dest="serviceFilter", required=False, help="Service to filter/return data for: AMAZON, CLOUDFRONT, EC2")
parser.add_argument('-rl', action="store_true", dest="listRegions", required=False, help="List known regions")
parser.add_argument('-sl', action="store_true", dest="listService", required=False, help="List known services")
args = parser.parse_args()


def get_json(url):
    try:
        r = requests.get(url)
        jdata = r.json()

    except Exception, e:
        print "ERROR - %s failed:  %s" % (url, e)

    return jdata

def list_filter_json(jdata, filter_on):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        filtered_data.append(jdata['prefixes'][i][filter_on])
    return filtered_data

def filter_json(jdata, filters):
    filtered_data = []
    for i in range(len(jdata['prefixes'])):
        if 'region' in filters and 'service' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower() and filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'region' in filters:
            if filters['region'].lower() in jdata['prefixes'][i]['region'].lower():
                filtered_data.append(jdata['prefixes'][i])
        elif 'service' in filters:
            if filters['service'].lower() in jdata['prefixes'][i]['service'].lower():
                filtered_data.append(jdata['prefixes'][i])
    return filtered_data

def list_regions(jdata):
    regions = set()
    filtered_data = list_filter_json(jdata, 'region')
    for region in filtered_data:
        regions.add(region)
    for region in sorted(regions):
        print region

def list_services(jdata):
    services = set()
    filtered_data = list_filter_json(jdata, 'service')
    for service in filtered_data:
        services.add(service)
    for service in sorted(services):
        print service

def table_it(filtered_data):
    print tabulate(filtered_data, headers="keys")

if __name__ == "__main__":
    jdata = get_json(url)
    if args.listRegions or args.listService:
        if args.listRegions:
            list_regions(jdata)
        elif args.listService:
            list_services(jdata)
        else:
            parser.print_help()
    else:
        if args.regionFilter or args.serviceFilter:
            if args.regionFilter and args.serviceFilter:
                filters = {'region': args.regionFilter, 'service': args.serviceFilter}
            else:
                if args.regionFilter:
                    filters = {'region': args.regionFilter}
                if args.serviceFilter:
                    filters = {'service': args.serviceFilter}
            filtered_data = filter_json(jdata, filters)
            table_it(filtered_data)
        else:
            table_it(jdata['prefixes'])

Fun with Python, Tabular & AWS IP ranges Read More »

How To: curl the Openstack API’s (v3 Keystone Auth)

Leave a Comment / Cloud, How To's, Openstack, Systems Administration / tuxninja

While Openstack provides a python client(s) for interactions….

[root@diamond ~]# source keystonerc_tuxninja
[root@diamond ~(keystone_tuxninja)]# openstack server list
+--------------------------------------+-------+--------+----------------------------------------+
| ID                                   | Name  | Status | Networks                               |
+--------------------------------------+-------+--------+----------------------------------------+
| e5b35d6a-a9ba-4714-a9e1-6361706bd047 | spin1 | ACTIVE | private_tuxlabs=10.0.0.8, 192.168.1.52 |
+--------------------------------------+-------+--------+----------------------------------------+
[root@diamond ~(keystone_tuxninja)]#

I frequently, finding myself needing to get data out of it without the pain of awk/sed’ing out the ASCII art.

Thus to quickly access the raw data, we can directly query the API’s using curl & parsing JSON instead, which is much better 🙂

Authentication

Before we can interact with the other Openstack API’s we need to authenticate to Keystone openstack’s identity service. After authenticating we receive a token to use with our subequent API requests. So step 1 we are going to create a JSON object with the required authentication details.

Create a file called ‘token-request.json’ with an object that looks like this.

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "id": "default"
                    },
                    "name": "tuxninja",
                    "password": "put_your_openstack_pass"
                }
            }
        }
    }
}

Btw, if you followed my tutorial on how to install Openstack Kilo, your authentication details for ‘admin’ is in your keystonerc_admin file.

Now we can use this file to authenticate like so:

export TOKEN=`curl -si -d @token-request.json -H "Content-type: application/json" http://localhost:35357/v3/auth/tokens | awk '/X-Subject-Token/ {print $2}'`

The token is actually returned in the header of the HTTP response, so this is why we need ‘-i’ when curling. Notice we are parsing out the token and returning the value to an environment variable $TOKEN.

Now we can include this $TOKEN and run whatever API commands we want (assuming admin privileges for the tenant/project)

Curl Commands (Numerous Examples!)

# list domains
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/domains

# create a domain
curl  -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" -d '{"domain": {"description": "--optional--", "enabled": true, "name": "dom1"}}'  http://localhost:35357/v3/domains


# list users
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users

# To create a users, create file named create_user.json file like this:

{
    "user": {
           "default_project_id": "18ed894bb8b84a5b9144c129fc754722",
            "description": "Description",
            "domain_id": "default",
            "email": "tuxninja@tuxlabs.com",
            "enabled": true,
            "name": "tuxninja",
            "password": "changeme" }
}

# then run
curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:35357/v3/users -d @create_user.json


# list images in nova
                                                                                             <tenant_id>
curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/images | python -m json.tool

# list servers (vms)

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool

# neutron networks

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

# neutron subnets

curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:9696/v2.0/networks | python -m json.tool

I sometimes pipe the output to python -m json.tool, which provides formatting for JSON. Lets take a closer look at an example.

Listing servers (vm’s)

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | python -m json.tool
{
    "servers": [
        {
            "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047",
            "links": [
                {
                    "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047",
                    "rel": "bookmark"
                }
            ],
            "name": "spin1"
        }
    ]
}
[root@diamond ~(keystone_tuxninja)]#

I only have 1 VM currently called spin1, but for the tutorials sake, if I had ten’s or hundred’s of VM’s and all I cared about was the VM name or ID, I would still need to parse this JSON object to avoid getting all this other meta-data.

My favorite command line way to do that without going full Python is using the handy JQ tool.

Here is how to use it !

[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .
{
  "servers": [
    {
      "name": "spin1",
      "links": [
        {
          "rel": "self",
          "href": "http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        },
        {
          "rel": "bookmark",
          "href": "http://localhost:8774/18ed894bb8b84a5b9144c129fc754722/servers/e5b35d6a-a9ba-4714-a9e1-6361706bd047"
        }
      ],
      "id": "e5b35d6a-a9ba-4714-a9e1-6361706bd047"
    }
  ]
}
[root@diamond ~(keystone_tuxninja)]#
[root@diamond ~(keystone_tuxninja)]# curl -s -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://localhost:8774/v2/18ed894bb8b84a5b9144c129fc754722/servers | jq .servers[0].name -r
spin1
[root@diamond ~(keystone_tuxninja)]#

The first command just takes whatever the STDOUT from curl is and indent’s and color’s the JSON making it pretty (colors gives it +1 vs. python -m json.tool).

The second example we actually parse what were after. As you can see it is pretty simple, but jq’s query language may not be 100% intuitive at first, but I promise it is pretty easy to understand if you have ever parsed JSON before. Read up more on JQ @ https://stedolan.github.io/jq/ & check out the Openstack docs for more API commands http://developer.openstack.org/api-ref.html

Hope you enjoyed this post ! Until next time.

How To: curl the Openstack API’s (v3 Keystone Auth) Read More »

Installing Openstack Kilo on Centos 7

1 Comment / Centos, Cloud, How To's, Openstack, Systems Administration / tuxninja

In a previous article I wrote about how to install Openstack Icehouse on CentOS 6.5 in great detail. In this article, I am going to keep verbosity to a minimum and just give you the commands ! I am hoping this will be refreshing for my audience. If you are curious however, about the what, when and why please read my previous article.

Pre-requisites

You need a machine with x86_64 architecture with at least 4 GB of memory & 2 NIC’s.
On this machine you need to install CentOS 7 as a minimal install
You should create a user with admin privileges (i.e. wheel, in my case ‘tuxninja’ was created)
Disable SELinux
1. vi /etc/sysconfig/selinux
2. SELINUX=disabled
3. save changes

Jumping Right In

Here are the commands you need to run.

sudo yum update -y
sudo yum install -y https://repos.fedorapeople.org/repos/openstack/openstack-kilo/rdo-release-kilo-1.noarch.rpm
sudo yum install epel-release
sudo yum install -y openstack-packstack

Now at this point if you ran ‘packstack’ you would run into a bug with this message

ERROR : Error appeared during Puppet run: 192.168.1.10_prescript.pp
Error: Could not find data item CONFIG_USE_SUBNETS in any Hiera data file and no default supplied at /var/tmp/packstack/053c9a3614de4404b906141268c08f0a/manifests/192.168.1.10_prescript.pp:2 on node diamond.tuxlabs.com

The workaround for this bug is as follows

sudo rpm -e puppet
sudo rpm rpm -e hiera
curl -O https://yum.puppetlabs.com/el/7/products/x86_64/hiera-1.3.4-1.el7.noarch.rpm
sudo rpm -ivh hiera-1.3.4-1.el7.noarch.rpm
vi /etc/yum.repos.d/epel.repo
1. At the bottom of the [epel] section, after the gpgkey add a newline with: exclude=hiera*
2. Save the file
sud0 yum install -y puppet-3.6.2-3.el7.noarch
reboot
sudo rm /etc/puppet/hiera.yaml
sudo packstack –allinone

This should successfully install. Godspeed.

Networking

Now that Openstack is setup, we still have to setup our network with private & public routed networks, so we can turn this into a real multi-node setup and ssh to our hosts and let them reach the internet etc. To do this, much like my previous post you need to modify your /etc/sysconfig/network-scripts/ files to reflect this.

[tuxninja@diamond network-scripts]$ cat ifcfg-enp4s0f0
NAME="enp4s0f0"
UUID="e0c3929c-1f9b-44d1-9c59-6c8872f603bd"
DEVICE="enp4s0f0"
TYPE="OVSPort"
NM_CONTROLLED="no"
DEVICETYPE="ovs"
OVS_BRIDGE="br-ex"
BOOTPROTO="none"
ONBOOT="yes"
[tuxninja@diamond network-scripts]$ cat ifcfg-enp4s0f1
NAME=enp4s0f1
UUID=ed50b4b6-2c29-4307-bbb0-f3c923f6552a
DEVICE=enp4s0f1
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
NETWORK=10.0.0.0
IPADDR=10.0.0.1
NETMASK=255.255.255.0
[tuxninja@diamond network-scripts]$ cat ifcfg-br-ex
DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
DNS2=8.8.4.4
DNS3=192.168.1.1
ONBOOT=yes
[tuxninja@diamond network-scripts]$

Note: I deleted all the IPV6 crap, I think it messes some stuff up. When your done making the changes with your favorite editor, restart networking : sudo /etc/init.d/network restart or sudo systemctl restart network

Next go into in the Horizon Dashboard GUI and delete the demo project. See my previous article for details on how.

Back On the All-In-One Node Console

[root@diamond ~]# source keystonerc_admin 
[root@diamond ~(keystone_admin)]# neutron router-create router1
[root@diamond ~(keystone_admin)]# neutron net-create private
[root@diamond ~(keystone_admin)]# neutron subnet-create private 10.0.0.0/24 --name private_subnet
[root@diamond ~(keystone_admin)]# neutron router-interface-add router1 private_subnet
[root@diamond ~(keystone_admin)]# neutron net-create public --router:external
[root@diamond ~(keystone_admin)]# neutron subnet-create public 192.168.1.0/24 --name public_subnet --enable_dhcp=False --allocation-pool start=192.168.1.51,end=192.168.1.99 --gateway=192.168.1.1
[root@diamond ~(keystone_admin)]# neutron router-gateway-set router1 public

Next ‘reboot’ or restart all openstack services :

for service in `openstack-service list`; do openstack-service restart $service; done

Note: it appears the –full-restart flag is gone, used to work !

When logging into your dashboard located at http://192.168.1.10/dashboard at some point you might hit a bug that prevent you from logging into the Horizon dashboard see : https://bugzilla.redhat.com/show_bug.cgi?id=1218894 … the work-around for this is to clear your browser cookies.

You’re Done

That’s it. Next steps would be to create a project & new admin user, re-create the required network mappings in openstack using the above commands (modify the names to make them unique) and create your ssh key, import it, download some images, import them using glance, and create some VM’s. Also I like to delete the demo project (you can also prevent this from being created with a flag on the packstack command). Make sure you delete all default security rules and add back ICMP, TCP, and UDP allow ingress / egress rules for 0.0.0.0 aka any/any, again you can see my article on CentOS 6.5 with more specifics on how to do this. Additionally, I have an article on how to add additional compute nodes as well.

As always I can be reached for assistance @ tuxninja [at] tuxlabs.com

Happying Stacking !

Installing Openstack Kilo on Centos 7 Read More »

Creating a bootable USB for Centos on Mac OS X

Leave a Comment / Centos, How To's, Systems Administration / tuxninja

I’m a huge Ubuntu fan. However, most of my ‘day job’ work requires CentOS or RHEL, thus I commonly have to re-image my on premise Cloud with the latest and greatest CentOS. My servers are 3 Rackables by SGI, two with more CPU & Memory and one (the controller node) with tons of Disk (12x1TB RAID 10) and then for off-premise I use Digital Ocean who has a fantastic product. Most modern servers do not have a CDROM and neither do my on premise systems. Therefore, I need to place the CentOS image on a USB drive so I can re-image my lab. Here are the steps do that on Mac OS X.

List the current Disks & Partitions

➜  ~  diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI                         209.7 MB   disk0s1
   2:          Apple_CoreStorage                         499.4 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *499.1 GB   disk1
/dev/disk2
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *2.0 GB     disk2
   1:                  Apple_HFS Untitled                1.9 GB     disk2s1
➜  ~

My USB drive is the 2GB drive at the bottom, we need to unmount that

➜  ~  diskutil unmountDisk /dev/disk2
Unmount of all volumes on disk2 was successful
➜  ~

Next we copy the CentOS image onto the unmounted USB disk.

➜  ~  sudo dd if=/Users/jriedel/Downloads/CentOS-7-x86_64-Minimal-1503-01.iso of=/dev/disk2
Password:
1302528+0 records in
1302528+0 records out
666894336 bytes transferred in 620.521343 secs (1074732 bytes/sec)
➜  ~

When that’s gets done your Mac will pop up a window asking you to initialize the drive, ignore that. Remove it, and your ready to boot off this USB!

Creating a bootable USB for Centos on Mac OS X Read More »

Preventing (bind9) DNS Naughty-ness (named.conf & iptables/ufw) on Ubuntu

Leave a Comment / How To's, Security, Systems Administration / tuxninja

If you run a DNS server on the Internet with a default configuration many people/robots will take advantage of you. The same is true for Mail, but that is another article. Needless to say if you are running a service on the Internet, the naughty goblins will find you. To thwart these dirty criminals all that’s necessary is to configure your named.conf properly. However, since these robotos are being naughty there is a high degree of certainty they are infected endpoints, and as such I really don’t want them coming anywhere near me or my machines. After all for humanity sake we don’t want to be infected by the deadly plague ! This article is short and sweet, here is how to protect your DNS server & your server in one article using named.conf & ufw (iptables).

Named.conf.options

Now a days named.conf is really just a file that inherits 3 other files, named.conf.local, named.conf.options, and named.conf.default-zones. The one we are going to fix is named.conf.options. The configuration below should only be applied in a scenario where you want to run an authorative nameserver, and a caching name server, but the key is you only want to allow people to query the cache that ‘you know personally or are you’ vs. allowing the entire internet, because then bad things happen. If this is not the setup you are going for, don’t do this 🙂 But if it is follow along.

Add the following section with the proper IP’s to the top fo the file

acl "trusted" {
192.241.206.98;
localhost;
localnets;
};

Note you can also add a CIDR for a subnet like 192.168.0.0/16

After that’s done under the options {} section… make it look like this

        allow-query { any; };
        allow-recursion { trusted; };
        allow-query-cache { trusted; };
        allow-transfer { 202.157.182.142; };

Note, allow transfer is necessary if you have a secondary nameserver that needs to receive updates. Now restart bind9

tuxninja@tlprod1:/etc/bind$ sudo service bind9 restart

Ok now all querying including behavior from non-trusted people will not be allowed. If it is working check your /var/log/syslog and you will see some denies like this

Nov 11 16:00:31 tlprod1 named[952]: client 192.163.221.224#80 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied
Nov 11 16:00:31 tlprod1 named[952]: client 192.163.221.224#80 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied
Nov 11 16:00:31 tlprod1 named[952]: client 104.37.29.110#4761 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied

Now the above is from my actual log file. I was quite annoyed that clients are basically abusing the hell out of hehehey.ru… so I decided I don’t want to talk to those people at all. To those people I should be a blackhole. To do this I used UFW which is short for uncomplicated firewall, which essentially makes dealing with Iptables much much nicer. It’s only my 2nd time using UFW, but I’ve been using Iptables for well over a decade. Anyway, here is my simple setup with UFW that I came up with.

tuxninja@tlprod1:/etc/bind$ sudo ufw default deny incoming
Default incoming policy changed to 'deny'
(be sure to update your rules accordingly)

tuxninja@tlprod1:/etc/bind$ sudo ufw default allow outgoing
Default outgoing policy changed to 'allow'
(be sure to update your rules accordingly)

tuxninja@tlprod1:/etc/bind$ sudo ufw allow ssh
Rules updated
Rules updated (v6)

tuxninja@tlprod1:/etc/bind$ sudo ufw allow 80
Rules updated
Rules updated (v6)

So we are configuring the default policy to deny all incoming traffic, allow outgoing, and then allow SSH & Apache/Web traffic basically. Next I created a script called block.sh to add ufw deny rules for bad actors I parsed out of my log, here’s what block.sh looks like

# cat block.sh 
#!/bin/bash

while read line; do
	ufw deny from $line
done

Don’t forget to chmod +x your shell script. Then I did this… blocking all bad actors…

root@tlprod1:~# cat /var/log/syslog | grep hehehey.ru | grep -v repeated | awk -F ' ' '{print $7}' | cut -d '#' -f 1 | ./block.sh

Note, use sudo if you don’t run this as root. This will go through my log and find all these bad requests, and block the requestor. It’s quite aggresive, so be careful, make sure you thoroughly limit your parsing with grep to only block things you really don’t want talking to your server, because this blocks ALL traffic from this requestor to your service, not just DNS.

Once that is complete you need to finally permit good DNS requests by running

ufw allow 53

And then finally enable your firewall

ufw enable

If you are successful you should see entries in your log that look like this

Nov 11 15:10:35 tlprod1 kernel: [1652178.544292] [UFW BLOCK] IN=eth0 OUT= MAC=04:01:63:57:8a:01:3c:8a:b0:0d:3f:f0:08:00 SRC=65.60.18.103 DST=192.241.206.198 LEN=72 TOS=0x00 PREC=0x00 TTL=247 ID=31303 PROTO=UDP SPT=20225 DPT=53 LEN=52

You can also view all your firewall rules by running

sudo ufw status numbered

Happy Blocking !

Preventing (bind9) DNS Naughty-ness (named.conf & iptables/ufw) on Ubuntu Read More »