How To: Launch A Jump Host In AWS Using Terraform

I have been a Hashicorp fan boy for a couple of years now. I am impressed, and happy with pretty much everything they have done from Vagrant to Consul and more. In short they make the DevOps world a better place. That being said this article is about the aptly named Terraform product. Here is how Hashicorp describes Terraform in their own words…

“Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.”

Interestingly, enough it doesn’t point out, but in a way implies it by omitting anything about providers that Terraform is multi-cloud (or cloud agnostic). Terraform works with AWS, GCP, Azure and Openstack. In this article we will be covering how to use Terraform with AWS.

Step 1, download Terraform, I am not going to cover that part 😉
https://www.terraform.io/downloads.html

Step 2, Configuration…

Configuration

Hashicorp uses their own configuration language for Terraform, it is fully JSON compatible, which is nice.. The details are covered here https://github.com/hashicorp/hcl.

After downloading and installing Terraform, its time to start generating the configs.

AWS IAM Keys

AWS keys are required to do anything with Terraform. You can read about how to generate an access key / secret key for a user here : http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey

Terraform Configuration Files Overview

When you execute the terraform commands, you should be within a directory containing terraform configuration files. Those files ending in a ‘.tf’ extension will be loaded by Terraform in alphabetical order.

Before we jump into our configuration files for our Jump box, it may be helpful to review a quick primer on the syntax here https://www.terraform.io/docs/configuration/syntax.html

& some more advanced features such as lookup here https://www.terraform.io/docs/configuration/interpolation.html

Most users of Terraform choose to make their configs modular resulting in multiple .tf files with names like main.tf, variables.tf and data.tf… This is not required…you can choose to put everything in one big Terraform file, but I caution you modularity/compartmentalization is always a better approach than one monolithic file. Let’s take a look at our main.tf

Main.tf

Typically if you only see one terraform config file, it is called main.tf, most commonly there will be at least one other file called variables.tf used specifically for providing values for variables used in other TF files such as main.tf. Let’s take a look at our main.tf file section by section.

Provider

The provider keyword is used to identify the platform (cloud) you will be talking to, whether it is AWS, or another cloud. In our case it is AWS, and define three configuration items, an access key, secret key, and a region all of which we are inserting variables for, which will later be looked up / translated into real values in variables.tf

provider "aws" {
        access_key = "${var.aws_access_key_id}"
        secret_key = "${var.aws_secret_access_key}"
        region = "${var.aws_region}"
}

Resource aws_instance

This section defines a resource, which is an “aws_instance” that we are calling “jump_box”. We define all configuration requirements for that instance. Substituting variable names where necessary, and in some cases we just hard code the value. Notice we are attaching two security groups to the instance to allow for ICMP & SSH. We are also tagging our instance, which is critical an AWS environment so your administrators/teammates have some idea about the machine that is spun up and what it is used for.

resource "aws_instance" "jump_box" {
        ami = "${lookup(var.ami_base, "centos-7")}"
        instance_type = "t2.medium"
        key_name = "${var.key_name}"
        vpc_security_group_ids = [
                "${lookup(var.security-groups, "allow-icmp-from-home")}",
                "${lookup(var.security-groups, "allow-ssh-from-home")}"
        ]
        subnet_id = "${element(var.subnets_private, 0)}"
        root_block_device {
                volume_size = 8
                volume_type = "standard"
        }
        user_data = <<-EOF
                                #!/bin/bash
                                yum -y update
                                EOF
        tags = {
                Name = "${var.instance_name_prefix}-jump"
                ApplicationName = "jump-box"
                ApplicationRole = "ops"
                Cluster = "${var.tags["Cluster"]}"
                Environment = "${var.tags["Environment"]}"
                Project = "${var.tags["Project"]}"
                BusinessUnit = "${var.tags["BusinessUnit"]}"
                OwnerEmail = "${var.tags["OwnerEmail"]}"
                SupportEmail = "${var.tags["SupportEmail"]}"
        }
}

Btw, the this resource type is provided by an AWS module found here https://www.terraform.io/docs/providers/aws/r/instance.html

You have to download the module using terraform get (which can be done once you write some config files and type terraform get 🙂 ).

Also, note the usage of ‘user_data’ here to update the machines packages at boot time. This is an AWS feature that is exposed through the AWS module in terraform.

Resource aws_security_group

Next we define a new security group (vs attaching an existing one in the above section). We are creating this new security group for other VM’s in the environment to attach later, such that it can be used to allow SSH from the Jump host to the VM’s in the environment.

Also notice under cidr_blocks we define a single IP address a /32 of our jump host…but more important is to notice how we determine that jump hosts IP address. Using .private_ip to access the attribute of the “jump_box” aws_instance we are creating/just created in AWS. That is pretty cool.

resource "aws_security_group" "jump_box_sg" {
        name = "${var.instance_name_prefix}-allow-ssh-from-jumphost"
        description = "Allow SSH from the jump host"
        vpc_id = "${var.vpc_id}"

        ingress {
                from_port = 22
                to_port = 22
                protocol = "tcp"
                cidr_blocks = ["${aws_instance.jump_box.private_ip}/32"]
        }

        tags = "${var.tags_infra_default}"
}

Resource aws_route53_record

The last entry in our main.tf creates a DNS entry for our jump host in Route53. Again notice we are specifying a name of jump. has a prefix to an entry, but the remainder of the FQDN is figured out by the lookup command.  The lookup command is used to lookup values inside of a map. In this case the map is defined in our variables.tf that we will review next.

resource "aws_route53_record" "jump_box_dns" {
        zone_id = "${lookup(var.route53_zone, "id")}"
        type = "A"
        ttl = "300"
        name = "jump.${lookup(var.route53_zone, "name")}"
        records = ["${aws_instance.jump_box.private_ip}"]
}

Variables.tf

I will attempt to match the section structure I used above for main.tf when explaining the variables in variables.tf though it is not really in as clear of a layout using sections.

Provider variables

When terraform is run it compiles all .tf files, and replaces any key that equals a variable, with the value it finds listed in the variables.tf file (in our case) with the variable keyword as a prefix. Notice that the first two variables are empty, they have no value defined. Why ? Terraform supports taking input at runtime, by leaving these values blank, Terraform will prompt us for the values. Region is pretty straight forward, default is the value returned and description in this case is really an unused value except as a comment.

variable "aws_access_key_id" {}
variable "aws_secret_access_key" {}

variable "aws_region" {
        description = "AWS region to create resources in"
        default = "us-east-1"
}

I would like to demonstrate the behavior of Terraform as described above, when the variables are left empty

➜  jump terraform plan
var.aws_access_key_id
  Enter a value: aaaa

var.aws_secret_access_key
  Enter a value: bbbb

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

...

At this phase you would enter your AWS key info, and terraform would ‘plan’ out your deployment. Meaning it would run through your configs, and print to your screen it’s plan, but not actually change any state in AWS. That is the difference between Terraform plan & Terraform apply.

Resource aws_instance variables

Here we define the values of our AMI, SSH Key, Instance prefix for the name, Tags, security groups, and subnets. Again this should be pretty straight forward, no magic here, just the use of string variables & maps where necessary.

variable "ami_base" {
        description = "AWS AMIs for base images"
        default = {
                "centos-7" = "ami-2af1ca3d"
                "ubuntu-14.04" = "ami-d79487c0"
        }
}

variable "key_name" {
        default = "tuxninja-rsa-2048"
}

variable "instance_name_prefix" {
        default = "tuxlabs-"
}

variable "tags" {
        type = "map"
        default = {
                        ApplicationName = "Jump"
                        ApplicationRole = "jump box - bastion"
                        Cluster = "Jump"
                        Environment = "Dev"
                        Project = "Jump"
                        BusinessUnit = "TuxLabs"
                        OwnerEmail = "tuxninja@tuxlabs.com"
                        SupportEmail = "tuxninja@tuxlabs.com"
        }
}

variable "tags_infra_default" {
        type = "map"
        default = {
                        ApplicationName = "Jump"
                        ApplicationRole = "jump box - bastion"
                        Cluster = "Jump"
                        Environment = "DEV"
                        Project = "Jump"
                        BusinessUnit = "TuxLabs"
                        OwnerEmail = "tuxninja@tuxlabs.com"
                        SupportEmail = "tuxninja@tuxlabs.com"
        }
}

variable "security-groups" {
        description = "maintained security groups"
        default = {
                "allow-icmp-from-home" = "sg-a1b75ddc"
                "allow-ssh-from-home" = "sg-aab75dd7"
        }
}

variable "vpc_id" {
        description = "VPC us-east-1-vpc-tuxlabs-dev01"
        default = "vpc-c229daa5"
}

variable "subnets_private" {
        description = "Private subnets within us-east-1-vpc-tuxlabs-dev01 vpc"
        default = ["subnet-78dfb852", "subnet-a67322d0", "subnet-7aa1cd22", "subnet-75005c48"]
}

variable "subnets_public" {
        description = "Public subnets within us-east-1-vpc-tuxlabs-dev01 vpc"
        default = ["subnet-7bdfb851", "subnet-a57322d3", "subnet-47a1cd1f", "subnet-73005c4e"]
}

It’s important to note the variables above are also used in other sections as needed, such as the aws_security_group section in main.tf …

Resource aws_route53_record variables

Here we define the ID & Name that are used in the ‘lookup’ functionality from our main.tf Route53 section above.

variable "route53_zone" {
        description = "Route53 zone used for DNS records"
        default = {
                id = "Z1ME2RCUVBYEW2"
                name = "tuxlabs.com"
        }
}

It’s important to note Terraform or TF files do not care when or where things are loaded. All files are loaded and variables require no specific order consistent with any other part of the configuration. All that is required is that for each variable you try to insert a value for, it has a value listed via the variable keyword in a TF file somewhere.

Output.tf

Again, I want to remind folks you can put these terraform syntax in one file if you wanted to, but I choose to split things up for readability and simplicity. So we have an output.tf file specifically for the output command, there is only one command, which lists the results of our terraform configurations upon success.

output "jump-box-details" {
	value = "${aws_route53_record.jump_box_dns.fqdn} - ${aws_instance.jump_box.private_ip} - ${aws_instance.jump_box.id} - ${aws_instance.jump_box.availability_zone}"
}

Ok so let’s run this and see how it looks…First a reminder, to test your config you can run Terraform plan first..It will tell you the changes its going to make…example

➜  jump terraform plan
var.aws_access_key_id
  Enter a value: blahblah

var.aws_secret_access_key
  Enter a value: blahblahblahblah

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_instance.jump_box
    ami:                                       "ami-2af1ca3d"
    associate_public_ip_address:               "<computed>"
    availability_zone:                         "<computed>"
    ebs_block_device.#:                        "<computed>"
    ephemeral_block_device.#:                  "<computed>"
    instance_state:                            "<computed>"
    instance_type:                             "t2.medium"

...

Plan: 3 to add, 0 to change, 0 to destroy.

If everything looks good & is green, you are ready to apply.

aws_security_group.jump_box_sg: Creation complete
aws_route53_record.jump_box_dns: Still creating... (10s elapsed)
aws_route53_record.jump_box_dns: Still creating... (20s elapsed)
aws_route53_record.jump_box_dns: Still creating... (30s elapsed)
aws_route53_record.jump_box_dns: Still creating... (40s elapsed)
aws_route53_record.jump_box_dns: Still creating... (50s elapsed)
aws_route53_record.jump_box_dns: Creation complete

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

jump-box-details = jump.tuxlabs.com - 10.10.195.46 - i-037f5b15bce6cc16d - us-east-1b
➜  jump

Congratulations, you now have a jump box in AWS using Terraform. Remember to attach the required security group to each machine you want to grant access to, and start locking down your jump box / bastion and VM’s.

Outro

Remember if you take the above config and try to run it, swapping out only the variables it will error something about a required module. Downloading the required modules is as simple as typing ‘terraform get‘ , which I believe the error message even tells you 🙂

So again this was a brief intro to Terraform it does a lot & is extremely powerful. One of the thing I did when setting up a Mongo cluster using Terraform, was to take advantage of a map to change node count per region. So if you wanted to deploy a different number of instances in different regions, your config might look something like…

main.tf

  count = "${var.region_instance_count[var.region_name]}"

variables.tf

variable "region_instance_count" {
  type = "map"
  default = {
    us-east-1 = 2
    us-west-2 = 1
    eu-central-1 = 1
    eu-west-1 = 1
  }
}

It also supports split if you want to multi-value a string variable.

Another couple things before I forget, Terraform apply, doesn’t just set up new infrastructure, it also can be used to modify existing infrastructure, which is a very powerful feature. So if you have something deployed and want to make a change, terraform apply is your friend.

And finally, when you are done with the infrastructure you spun up or it’s time to bring her down… ‘terraform destroy’

➜  jump terraform destroy
Do you really want to destroy?
  Terraform will delete all your managed infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

var.aws_access_key_id
  Enter a value: blahblah

var.aws_secret_access_key
  Enter a value: blahblahblahblah

...

Destroy complete! Resources: 3 destroyed.
➜  jump

I hope this article helps.

Happy Terraforming 😉

 

Storing passwords securely using Pass (GPG)

Today we live in an endless sea of passwords, which are a very inefficient and ineffective means of securing our data & environments. Many companies are trying to solve this problem using a variety of techniques that all revolve around various forms of multi-factor authentication.

However, in the mean time were all screwed 😉

Just kidding. Quick PSA though, use two factor authentication at a minimum everywhere you can ESPECIALLY your email, since it’s used for password recovery on other sites. Ok then moving on…

There are many password managers like LastPass and 1Password, which do a fairly effective job at providing convenience and prevent you from scribbling down your passwords on paper (STOP IT !!!). However, I personally can’t get passed the whole ‘store all my passwords in one super secure vault on the Internet’ thing. To be fair some of these password managers can be downloaded on your machine and ran locally, but there are two other drawbacks to those I found.

  1. Some of them are not free and…
  2. Some of them have ugly and clunky UI’s

So what do I like/use then ? I use something called ‘pass’. Which is a command line utility that wraps GPG. The reason I use it is because…

  1. I love using command line utilities over GUI, I find it far more convenient and…
  2. I was going to write this exact utility (a GPG wrapper) until I found out someone else did and…
  3. Because I like GPG.

At most of the organizations I have worked at, password management was done poorly i.e. everyone used different approaches and there was no governance or oversight. I hope with this article to make folks aware of what I feel is a simple, effective method that every unix savvy administrator should use.

FYI Pass provides migration scripts from the most popular password manager tools on their website.

Introducing Pass

From the Pass site “Password management should be simple and follow Unix philosophy. With pass, each password lives inside of a gpg encrypted file whose filename is the title of the website or resource that requires the password. These encrypted files may be organized into meaningful folder hierarchies, copied from computer to computer, and, in general, manipulated using standard command line file management utilities.”

Where Can You Get or Learn More About Pass ? 

https://www.passwordstore.org/

Installing Pass

Depending on your operating system there are various ways to install

Ubuntu/Debian

sudo apt-get install pass

Fedora / RHEL

sudo yum install pass

Mac

brew install pass
echo "source /usr/local/etc/bash_completion.d/password-store" >> ~/.bashrc

Since I already installed pass on my Mac a while back I will be installing it on a Docker container with Ubuntu 16.04.

root@0b415380eb80:/# apt-get install -y pass

After pass successfully installs, try running it

root@0b415380eb80:/# pass
Error: password store is empty. Try "pass init".
root@0b415380eb80:/#

Well that is pretty straight forward, it appears we need to initiliaze the db.

root@0b415380eb80:/# pass init
Usage: pass init [--path=subfolder,-p subfolder] gpg-id...
root@0b415380eb80:/#

Looks like we need to provide ‘key’…can that be just anything?

root@0b415380eb80:/# pass init "tuxlabs Password Key"
mkdir: created directory '/root/.password-store/'
Password store initialized for tuxlabs Password Key
root@0b415380eb80:/# pass
Password Store
root@0b415380eb80:/#

Now our password store looks initialized ! Let’s try inserting a password into the DB !

root@0b415380eb80:/# pass insert Gmail/myemail
mkdir: created directory '/root/.password-store/Gmail'
Enter password for Gmail/myemail:
Retype password for Gmail/myemail:
gpg: tuxlabs Password Key: skipped: No public key
gpg: [stdin]: encryption failed: No public key
root@0b415380eb80:/#

Uh oh what happened ? Well remember I said it uses GPG, and we not only don’t have a gpg key setup in our Docker container, but we initialized our Pass DB without using a GPG Key (the whole point) !

root@0b415380eb80:/# gpg --list-keys
root@0b415380eb80:/#

To remedy this we need to create a GPG key

Creating your GPG Key

root@0b415380eb80:/# gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Tuxninja
Email address: tuxninja@tuxlabs.com
Comment: TuxLabs
You selected this USER-ID:
    "Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: gpg-agent is not available in this session
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
............+++++
...................+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
...+++++
+++++
gpg: key 5B2F89A5 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/5B2F89A5 2016-12-14
      Key fingerprint = 5FF6 1717 4415 03FF D455  7516 CF8E 1BDC 5B2F 89A5
uid                  Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>
sub   4096R/EF0F232F 2016-12-14

root@0b415380eb80:/#

To view your GPG key run

root@0b415380eb80:/# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   4096R/5B2F89A5 2016-12-14
uid                  Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>
sub   4096R/EF0F232F 2016-12-14

root@0b415380eb80:/#

Now we can see we have one GPG key, with the ID 5B2F89A5

Let’s try re-initializing Pass. 

root@0b415380eb80:/# pass init "5B2F89A5"
Password store initialized for 5B2F89A5
root@0b415380eb80:/#

But we have a problem, re-initializing Pass doesn’t get rid of our previous insert into the db. As you can see here our Pass DB is effectively corrupt.

root@0b415380eb80:~# pass
Password Store
`-- Gmail
root@0b415380eb80:~# pass rm Gmail
Are you sure you would like to delete Gmail? [y/N] y
rm: cannot remove '/root/.password-store/Gmail': Is a directory
root@0b415380eb80:~# pass rm Gmail/myemail
Error: Gmail/myemail is not in the password store.
root@0b415380eb80:~#

Hmmm, what’s a guy to do….

root@0b415380eb80:~# rm -rf .password-store/Gmail/
root@0b415380eb80:~# pass
Password Store
root@0b415380eb80:~#

Yes it really was that simple, and that is one more reason why I love pass.

You can also initialize your password store using git for version control, see the passwordstore.org website for more info !

Now let’s insert some good stuff.

Inserting A Password into Pass

root@0b415380eb80:~# pass insert Gmail/myemail
Enter password for Gmail/myemail:
Retype password for Gmail/myemail:
root@0b415380eb80:~# pass
Password Store
`-- Gmail
    `-- myemail
root@0b415380eb80:~#

That seems to have worked. Let’s try to retrieve the pass.

Retrieving A Password In Pass

root@0b415380eb80:~# pass Gmail/myemail
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/root/.gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
testpass
root@0b415380eb80:~# pass Gmail/myemail
testpass
root@0b415380eb80:~#

Note, I retrieve the password twice using my GPG Passsword (You will be prompted through a curses interface to enter your passphrase). Then I run it again, because of the initial GPG migration messages just to show how it would normally work after you’ve used GPG once with Pass.

Now let’s say someone is standing over your shoulder, you want to access your passsword, but you don’t want them to see it. You can get it straight to your clipboard by using -c.

Copying Passwords To Your Clipboard

pass -c Gmail/myemail
Copied Gmail/myemail to clipboard. Will clear in 45 seconds.

Docker Issue ?

Notice the prompt is not included in the above example ? That is cause it didn’t actually work. Apparently, it doesn’t work in Docker due to not having display dependencies installed/configured. So what I show above is the output from my mac…but my actual Docker related error was.

root@0b415380eb80:~# pass -c Gmail/myemail
Error: Can't open display: (null)
Error: Could not copy data to the clipboard
root@0b415380eb80:~#

There might be an easy way to fix this (like install X), but I don’t usually use Docker for storing my passwords I just happen to be using it for this tutorial, so moving on !

Folders

It’s also important to note that Pass supports folder structures, as shown in my example I am creating a ‘Gmail’ folder and placing a password file called ‘myemail’ with my password in it. In reality I recommend not naming the file after your account/email and using the multiline version to encrypt those details as well. That way you can just stick to the site name for the name of the encrypted file in whatever folder or in the top level of Pass.

Multiline Encrypted Files with Pass

A common use case with Pass is adding an entire encrypted file so you can store more than just a password…

root@0b415380eb80:~# pass insert -m tuxlabs/databases
mkdir: created directory '/root/.password-store/tuxlabs'
Enter contents of tuxlabs/databases and press Ctrl+D when finished:

this is an example of a multiline
encrypted file
this way you can store more than just a password you can store user/pass/url etc
root@0b415380eb80:~# pass
Password Store
|-- Gmail
|   `-- myemail
`-- tuxlabs
    `-- databases
root@0b415380eb80:~#

Again retrieving it is as easy as..

root@0b415380eb80:~# pass tuxlabs/databases
this is an example of a multiline
encrypted file
this way you can store more than just a password you can store user/pass/url etc
root@0b415380eb80:~#

Finally if you no longer want the info to be stored in Pass…

If you want to copy you password to the clipboard from a multiline file, you must store your password on the first line of the file !

Deleting An Entry In Pass

root@0b415380eb80:~# pass rm Gmail/myemail
Are you sure you would like to delete Gmail/myemail? [y/N] y
removed '/root/.password-store/Gmail/myemail.gpg'
root@0b415380eb80:~# pass rm tuxlabs/databases
Are you sure you would like to delete tuxlabs/databases? [y/N] y
removed '/root/.password-store/tuxlabs/databases.gpg'
root@0b415380eb80:~# pass
Password Store
root@0b415380eb80:~#

Another thing, the output on my mac is much prettier than this `– thing I am getting in the Ubuntu Docker container… Not sure if that’s an Ubuntu issue or Docker, but on the Mac the output is much prettier, which can be seen on the passwordstore.org home page.

So that’s it, Pass is pretty straight forward, easy to work with, depends on GPG security and that is why I like it.

Stay secure, until next time !

 

How to use Boto to Audit your AWS EC2 instance security groups

Boto is a Software Development Kit for accessing the AWS API’s using Python.

https://github.com/boto/boto3

Recently, I needed to determine how many of my EC2 instances were spawned in a public subnet, that also had security groups with wide open access on any port via any protocol to the instances. Because I have an IGW (Internet Gateway) in my VPC’s and properly setup routing tables, instances launched in the public subnets with wide open security groups (allowing ingress traffic from any source) is a really bad thing 🙂

Here is the code I wrote to identify these naughty instances. It will require slight modifications in your own environment, to match your public subnet IP Ranges, EC2 Tags, and Account names.

#!/usr/bin/env python
__author__ = 'Jason Riedel'
__description__ = 'Find EC2 instances provisioned in a public subnet that have security groups allowing ingress traffic from any source.'
__date__ = 'June 5th 2016'
__version__ = '1.0'

import boto3

def find_public_addresses(ec2):
    public_instances = {}
    instance_public_ips = {}
    instance_private_ips = {}
    instance_ident = {}
    instances = ec2.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running'] }])

    # Ranges that you define as public subnets in AWS go here.
    public_subnet_ranges = ['10.128.0', '192.168.0', '172.16.0']

    for instance in instances:
        # I only care if the private address falls into a public subnet range
        # because if it doesnt Internet ingress cant reach it directly anyway even with a public IP
        if any(cidr in instance.private_ip_address for cidr in public_subnet_ranges):
            owner_tag = "None"
            instance_name = "None"
            if instance.tags:
                for i in range(len(instance.tags)):
                    #comment OwnerEmail tag out if you do not tag your instances with it.
                    if instance.tags[i]['Key'] == "OwnerEmail":
                        owner_tag = instance.tags[i]['Value']
                    if instance.tags[i]['Key'] == "Name":
                        instance_name = instance.tags[i]['Value']
            instance_ident[instance.id] = "Name: %s\n\tKeypair: %s\n\tOwner: %s" % (instance_name, instance.key_name, owner_tag)
            if instance.public_ip_address is not None:
                values=[]
                for i in range(len(instance.security_groups)):
                    values.append(instance.security_groups[i]['GroupId'])
                public_instances[instance.id] = values
                instance_public_ips[instance.id] = instance.public_ip_address
                instance_private_ips[instance.id] = instance.private_ip_address

    return (public_instances, instance_public_ips,instance_private_ips, instance_ident)

def inspect_security_group(ec2, sg_id):
    sg = ec2.SecurityGroup(sg_id)

    open_cidrs = []
    for i in range(len(sg.ip_permissions)):
        to_port = ''
        ip_proto = ''
        if 'ToPort' in sg.ip_permissions[i]:
            to_port = sg.ip_permissions[i]['ToPort']
        if 'IpProtocol' in sg.ip_permissions[i]:
            ip_proto = sg.ip_permissions[i]['IpProtocol']
            if '-1' in ip_proto:
                ip_proto = 'All'
        for j in range(len(sg.ip_permissions[i]['IpRanges'])):
            cidr_string = "%s %s %s" % (sg.ip_permissions[i]['IpRanges'][j]['CidrIp'], ip_proto, to_port)

            if sg.ip_permissions[i]['IpRanges'][j]['CidrIp'] == '0.0.0.0/0':
                #preventing an instance being flagged for only ICMP being open
                if ip_proto != 'icmp':
                    open_cidrs.append(cidr_string)

    return open_cidrs


if __name__ == "__main__":
    #loading profiles from ~/.aws/config & credentials
    profile_names = ['de', 'pe', 'pde', 'ppe']
    #Translates profile name to a more friendly name i.e. Account Name
    translator = {'de': 'Platform Dev', 'pe': 'Platform Prod', 'pde': 'Products Dev', 'ppe': 'Products Prod'}
    for profile_name in profile_names:
        session = boto3.Session(profile_name=profile_name)
        ec2 = session.resource('ec2')

        (public_instances, instance_public_ips, instance_private_ips, instance_ident) = find_public_addresses(ec2)

        for instance in public_instances:
            for sg_id in public_instances[instance]:
                open_cidrs = inspect_security_group(ec2, sg_id)
                if open_cidrs: #only print if there are open cidrs
                    print "=================================="
                    print " %s, %s" % (instance, translator[profile_name])
                    print "=================================="
                    print "\tprivate ip: %s\n\tpublic ip: %s\n\t%s" % (instance_private_ips[instance], instance_public_ips[instance], instance_ident[instance])
                    print "\t=========================="
                    print "\t open ingress rules"
                    print "\t=========================="
                    for cidr in open_cidrs:
                        print "\t\t" + cidr

To run this you also need to setup your .aws/config and .aws/credentials file.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Email me tuxninja [at] tuxlabs.com if you have any issues.
Boto is awesome 🙂 Note so is the AWS CLI, but requires some shell scripting as opposed to Python to do cool stuff.

The github for this code here https://github.com/jasonriedel/AWS/blob/master/sg-audit.py

Enjoy !

Preventing (bind9) DNS Naughty-ness (named.conf & iptables/ufw) on Ubuntu

If you run a DNS server on the Internet with a default configuration many people/robots will take advantage of you. The same is true for Mail, but that is another article. Needless to say if you are running a service on the Internet, the naughty goblins will find you. To thwart these dirty criminals all that’s necessary is to configure your named.conf properly. However, since these robotos are being naughty there is a high degree of certainty they are infected endpoints, and as such I really don’t want them coming anywhere near me or my machines. After all for humanity sake we don’t want to be infected by the deadly plague ! This article is short and sweet, here is how to protect your DNS server & your server in one article using named.conf & ufw (iptables).

 

Named.conf.options

Now a days named.conf is really just a file that inherits 3 other files, named.conf.local, named.conf.options, and named.conf.default-zones. The one we are going to fix is named.conf.options. The configuration below should only be applied in a scenario where you want to run an authorative nameserver, and a caching name server, but the key is you only want to allow people to query the cache that ‘you know personally or are you’ vs. allowing the entire internet, because then bad things happen. If this is not the setup you are going for, don’t do this 🙂 But if it is follow along.

Add the following section with the proper IP’s to the top fo the file

acl "trusted" {
192.241.206.98;
localhost;
localnets;
};

Note you can also add a CIDR for a subnet like 192.168.0.0/16

After that’s done under the options {} section… make it look like this

        allow-query { any; };
        allow-recursion { trusted; };
        allow-query-cache { trusted; };
        allow-transfer { 202.157.182.142; };

Note, allow transfer is necessary if you have a secondary nameserver that needs to receive updates. Now restart bind9

tuxninja@tlprod1:/etc/bind$ sudo service bind9 restart

Ok now all querying including behavior from non-trusted people will not be allowed. If it is working check your /var/log/syslog and you will see some denies like this

Nov 11 16:00:31 tlprod1 named[952]: client 192.163.221.224#80 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied
Nov 11 16:00:31 tlprod1 named[952]: client 192.163.221.224#80 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied
Nov 11 16:00:31 tlprod1 named[952]: client 104.37.29.110#4761 (hehehey.ru): query (cache) 'hehehey.ru/ANY/IN' denied

Now the above is from my actual log file. I was quite annoyed that clients are basically abusing the hell out of hehehey.ru… so I decided I don’t want to talk to those people at all. To those people I should be a blackhole. To do this I used UFW which is short for uncomplicated firewall, which essentially makes dealing with Iptables much much nicer. It’s only my 2nd time using UFW, but I’ve been using Iptables for well over a decade. Anyway, here is my simple setup with UFW that I came up with.

tuxninja@tlprod1:/etc/bind$ sudo ufw default deny incoming
Default incoming policy changed to 'deny'
(be sure to update your rules accordingly)

tuxninja@tlprod1:/etc/bind$ sudo ufw default allow outgoing
Default outgoing policy changed to 'allow'
(be sure to update your rules accordingly)

tuxninja@tlprod1:/etc/bind$ sudo ufw allow ssh
Rules updated
Rules updated (v6)

tuxninja@tlprod1:/etc/bind$ sudo ufw allow 80
Rules updated
Rules updated (v6)

So we are configuring the default policy to deny all incoming traffic, allow outgoing, and then allow SSH & Apache/Web traffic basically. Next I created a script called block.sh to add ufw deny rules for bad actors I parsed out of my log, here’s what block.sh looks like

# cat block.sh 
#!/bin/bash

while read line; do
	ufw deny from $line
done

Don’t forget to chmod +x your shell script. Then I did this… blocking all bad actors…

root@tlprod1:~# cat /var/log/syslog | grep hehehey.ru | grep -v repeated | awk -F ' ' '{print $7}' | cut -d '#' -f 1 | ./block.sh

Note, use sudo if you don’t run this as root. This will go through my log and find all these bad requests, and block the requestor. It’s quite aggresive, so be careful, make sure you thoroughly limit your parsing with grep to only block things you really don’t want talking to your server, because this blocks ALL traffic from this requestor to your service, not just DNS.

Once that is complete you need to finally permit good DNS requests by running

ufw allow 53

And then finally enable your firewall

ufw enable

If you are successful you should see entries in your log that look like this

Nov 11 15:10:35 tlprod1 kernel: [1652178.544292] [UFW BLOCK] IN=eth0 OUT= MAC=04:01:63:57:8a:01:3c:8a:b0:0d:3f:f0:08:00 SRC=65.60.18.103 DST=192.241.206.198 LEN=72 TOS=0x00 PREC=0x00 TTL=247 ID=31303 PROTO=UDP SPT=20225 DPT=53 LEN=52

You can also view all your firewall rules by running

sudo ufw status numbered

Happy Blocking !

 

 

 

Introducing Vault

Vault

Vault is a command line utility for encrypting & decrypting things. Those things are stored on disk in hidden files, meaning in *nix they simply have a ‘.’ in front and don’t show up unless you type ls -la 😉 But anyway, who cares if someone can locate the files ! They are AES encrypted !

I wrote Vault for a couple of reasons.

  1. To learn how to use encryption in Python
  2. To store passwords for stuff
  3. To turn Vault into methods in my Utilities class that I use for my daily programming activities. First write a command line utility, then write a class 🙂

Vault Usage

I literally wrote vault earlier today and even though it is super simple, I still find it cool & useful so I had to share. Here’s a demo on it’s simple usage !

Keys

AES / Vault supports 16, 24, or 32 byte encryption keys. A byte is one character. Your key should be something ambiguous that you can remember and you should not use the ‘0123…’ example below. What would really be paranoid is to encrypt your keys in a separate vault somewhere 🙂 Note, you will need to use the same key to encrypt as decrypt for a given stored piece of encrypted data.

Encrypting

➜  vault git:(master) python store.py --name tuxlabs -t 'Well I could store a password or b00bs, cause everyone likes b00bs even the newbs' --key 01234567890987654321abcd
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w== was stored as tuxlabs
➜  vault git:(master) ✗

Can You Read The File Contents ?

➜  vault git:(master) ✗ cat .tuxlabs 
p/hvz1hf9RIyBeyMmgL2CILvlM20vU72E075K+32tysNU8dIJOcX/gVmRISYQTp0tHZ/W+qL2mCvMFrMP3rGAV2kCNNjGQNnbUSgPibPGiqfwMrQm3/EhH/f18dZofDGTwcMmHZ3LiERuIZt1toU0w==%                                                                                                                     ➜  vault git:(master) ✗

Decrypting

➜  vault git:(master) ✗ python retrieve.py --name tuxlabs --key 01234567890987654321abcd
Well I could store a password or b00bs, cause everyone likes b00bs even the newbs
➜  vault git:(master) ✗

Alternatively, you can hide the key from the command line, by not specifying it. 

➜  vault  python store.py --name JimmyJohns --text 'Really Fast'
Key:
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ= was stored as JimmyJohns
➜  vault  python retrieve.py --name JimmyJohns
Key:
Really Fast
➜  vault  cat .JimmyJohns 
1HHF5TPl0cklDU/TXjMQ8nFeqK4zULQ50dVpJ+apgzQ=%                                                                                                                                                                                                                                 ➜  vault

Clone Vault on github

Enjoy !
Jason Riedel