TuxLabs LLC

All things DevOps

Category: Security

How To: Launch A Jump Host In AWS Using Terraform

Published / by tuxninja / Leave a Comment

I have been a Hashicorp fan boy for a couple of years now. I am impressed, and happy with pretty much everything they have done from Vagrant to Consul and more. In short they make the DevOps world a better place. That being said this article is about the aptly named Terraform product. Here is how Hashicorp describes Terraform in their own words…

“Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.”

Interestingly, enough it doesn’t point out, but in a way implies it by omitting anything about providers that Terraform is multi-cloud (or cloud agnostic). Terraform works with AWS, GCP, Azure and Openstack. In this article we will be covering how to use Terraform with AWS.

Step 1, download Terraform, I am not going to cover that part 😉
https://www.terraform.io/downloads.html

Step 2, Configuration…

Configuration

Hashicorp uses their own configuration language for Terraform, it is fully JSON compatible, which is nice.. The details are covered here https://github.com/hashicorp/hcl.

After downloading and installing Terraform, its time to start generating the configs.

AWS IAM Keys

AWS keys are required to do anything with Terraform. You can read about how to generate an access key / secret key for a user here : http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey

Terraform Configuration Files Overview

When you execute the terraform commands, you should be within a directory containing terraform configuration files. Those files ending in a ‘.tf’ extension will be loaded by Terraform in alphabetical order.

Before we jump into our configuration files for our Jump box, it may be helpful to review a quick primer on the syntax here https://www.terraform.io/docs/configuration/syntax.html

& some more advanced features such as lookup here https://www.terraform.io/docs/configuration/interpolation.html

Most users of Terraform choose to make their configs modular resulting in multiple .tf files with names like main.tf, variables.tf and data.tf… This is not required…you can choose to put everything in one big Terraform file, but I caution you modularity/compartmentalization is always a better approach than one monolithic file. Let’s take a look at our main.tf

Main.tf

Typically if you only see one terraform config file, it is called main.tf, most commonly there will be at least one other file called variables.tf used specifically for providing values for variables used in other TF files such as main.tf. Let’s take a look at our main.tf file section by section.

Provider

The provider keyword is used to identify the platform (cloud) you will be talking to, whether it is AWS, or another cloud. In our case it is AWS, and define three configuration items, an access key, secret key, and a region all of which we are inserting variables for, which will later be looked up / translated into real values in variables.tf

Resource aws_instance

This section defines a resource, which is an “aws_instance” that we are calling “jump_box”. We define all configuration requirements for that instance. Substituting variable names where necessary, and in some cases we just hard code the value. Notice we are attaching two security groups to the instance to allow for ICMP & SSH. We are also tagging our instance, which is critical an AWS environment so your administrators/teammates have some idea about the machine that is spun up and what it is used for.

Btw, the this resource type is provided by an AWS module found here https://www.terraform.io/docs/providers/aws/r/instance.html

You have to download the module using terraform get (which can be done once you write some config files and type terraform get 🙂 ).

Also, note the usage of ‘user_data’ here to update the machines packages at boot time. This is an AWS feature that is exposed through the AWS module in terraform.

Resource aws_security_group

Next we define a new security group (vs attaching an existing one in the above section). We are creating this new security group for other VM’s in the environment to attach later, such that it can be used to allow SSH from the Jump host to the VM’s in the environment.

Also notice under cidr_blocks we define a single IP address a /32 of our jump host…but more important is to notice how we determine that jump hosts IP address. Using .private_ip to access the attribute of the “jump_box” aws_instance we are creating/just created in AWS. That is pretty cool.

Resource aws_route53_record

The last entry in our main.tf creates a DNS entry for our jump host in Route53. Again notice we are specifying a name of jump. has a prefix to an entry, but the remainder of the FQDN is figured out by the lookup command.  The lookup command is used to lookup values inside of a map. In this case the map is defined in our variables.tf that we will review next.

Variables.tf

I will attempt to match the section structure I used above for main.tf when explaining the variables in variables.tf though it is not really in as clear of a layout using sections.

Provider variables

When terraform is run it compiles all .tf files, and replaces any key that equals a variable, with the value it finds listed in the variables.tf file (in our case) with the variable keyword as a prefix. Notice that the first two variables are empty, they have no value defined. Why ? Terraform supports taking input at runtime, by leaving these values blank, Terraform will prompt us for the values. Region is pretty straight forward, default is the value returned and description in this case is really an unused value except as a comment.

I would like to demonstrate the behavior of Terraform as described above, when the variables are left empty

At this phase you would enter your AWS key info, and terraform would ‘plan’ out your deployment. Meaning it would run through your configs, and print to your screen it’s plan, but not actually change any state in AWS. That is the difference between Terraform plan & Terraform apply.

Resource aws_instance variables

Here we define the values of our AMI, SSH Key, Instance prefix for the name, Tags, security groups, and subnets. Again this should be pretty straight forward, no magic here, just the use of string variables & maps where necessary.

It’s important to note the variables above are also used in other sections as needed, such as the aws_security_group section in main.tf …

Resource aws_route53_record variables

Here we define the ID & Name that are used in the ‘lookup’ functionality from our main.tf Route53 section above.

It’s important to note Terraform or TF files do not care when or where things are loaded. All files are loaded and variables require no specific order consistent with any other part of the configuration. All that is required is that for each variable you try to insert a value for, it has a value listed via the variable keyword in a TF file somewhere.

Output.tf

Again, I want to remind folks you can put these terraform syntax in one file if you wanted to, but I choose to split things up for readability and simplicity. So we have an output.tf file specifically for the output command, there is only one command, which lists the results of our terraform configurations upon success.

Ok so let’s run this and see how it looks…First a reminder, to test your config you can run Terraform plan first..It will tell you the changes its going to make…example

If everything looks good & is green, you are ready to apply.

Congratulations, you now have a jump box in AWS using Terraform. Remember to attach the required security group to each machine you want to grant access to, and start locking down your jump box / bastion and VM’s.

Outro

Remember if you take the above config and try to run it, swapping out only the variables it will error something about a required module. Downloading the required modules is as simple as typing ‘terraform get‘ , which I believe the error message even tells you 🙂

So again this was a brief intro to Terraform it does a lot & is extremely powerful. One of the thing I did when setting up a Mongo cluster using Terraform, was to take advantage of a map to change node count per region. So if you wanted to deploy a different number of instances in different regions, your config might look something like…

main.tf

variables.tf

It also supports split if you want to multi-value a string variable.

Another couple things before I forget, Terraform apply, doesn’t just set up new infrastructure, it also can be used to modify existing infrastructure, which is a very powerful feature. So if you have something deployed and want to make a change, terraform apply is your friend.

And finally, when you are done with the infrastructure you spun up or it’s time to bring her down… ‘terraform destroy’

I hope this article helps.

Happy Terraforming 😉

 

Storing passwords securely using Pass (GPG)

Published / by tuxninja / Leave a Comment

Today we live in an endless sea of passwords, which are a very inefficient and ineffective means of securing our data & environments. Many companies are trying to solve this problem using a variety of techniques that all revolve around various forms of multi-factor authentication.

However, in the mean time were all screwed 😉

Just kidding. Quick PSA though, use two factor authentication at a minimum everywhere you can ESPECIALLY your email, since it’s used for password recovery on other sites. Ok then moving on…

There are many password managers like LastPass and 1Password, which do a fairly effective job at providing convenience and prevent you from scribbling down your passwords on paper (STOP IT !!!). However, I personally can’t get passed the whole ‘store all my passwords in one super secure vault on the Internet’ thing. To be fair some of these password managers can be downloaded on your machine and ran locally, but there are two other drawbacks to those I found.

  1. Some of them are not free and…
  2. Some of them have ugly and clunky UI’s

So what do I like/use then ? I use something called ‘pass’. Which is a command line utility that wraps GPG. The reason I use it is because…

  1. I love using command line utilities over GUI, I find it far more convenient and…
  2. I was going to write this exact utility (a GPG wrapper) until I found out someone else did and…
  3. Because I like GPG.

At most of the organizations I have worked at, password management was done poorly i.e. everyone used different approaches and there was no governance or oversight. I hope with this article to make folks aware of what I feel is a simple, effective method that every unix savvy administrator should use.

FYI Pass provides migration scripts from the most popular password manager tools on their website.

Introducing Pass

From the Pass site “Password management should be simple and follow Unix philosophy. With pass, each password lives inside of a gpg encrypted file whose filename is the title of the website or resource that requires the password. These encrypted files may be organized into meaningful folder hierarchies, copied from computer to computer, and, in general, manipulated using standard command line file management utilities.”

Where Can You Get or Learn More About Pass ? 

https://www.passwordstore.org/

Installing Pass

Depending on your operating system there are various ways to install

Ubuntu/Debian

Fedora / RHEL

Mac

Since I already installed pass on my Mac a while back I will be installing it on a Docker container with Ubuntu 16.04.

After pass successfully installs, try running it

Well that is pretty straight forward, it appears we need to initiliaze the db.

Looks like we need to provide ‘key’…can that be just anything?

Now our password store looks initialized ! Let’s try inserting a password into the DB !

Uh oh what happened ? Well remember I said it uses GPG, and we not only don’t have a gpg key setup in our Docker container, but we initialized our Pass DB without using a GPG Key (the whole point) !

To remedy this we need to create a GPG key

Creating your GPG Key

To view your GPG key run

Now we can see we have one GPG key, with the ID 5B2F89A5

Let’s try re-initializing Pass. 

But we have a problem, re-initializing Pass doesn’t get rid of our previous insert into the db. As you can see here our Pass DB is effectively corrupt.

Hmmm, what’s a guy to do….

Yes it really was that simple, and that is one more reason why I love pass.

You can also initialize your password store using git for version control, see the passwordstore.org website for more info !

Now let’s insert some good stuff.

Inserting A Password into Pass

That seems to have worked. Let’s try to retrieve the pass.

Retrieving A Password In Pass

Note, I retrieve the password twice using my GPG Passsword (You will be prompted through a curses interface to enter your passphrase). Then I run it again, because of the initial GPG migration messages just to show how it would normally work after you’ve used GPG once with Pass.

Now let’s say someone is standing over your shoulder, you want to access your passsword, but you don’t want them to see it. You can get it straight to your clipboard by using -c.

Copying Passwords To Your Clipboard

Docker Issue ?

Notice the prompt is not included in the above example ? That is cause it didn’t actually work. Apparently, it doesn’t work in Docker due to not having display dependencies installed/configured. So what I show above is the output from my mac…but my actual Docker related error was.

There might be an easy way to fix this (like install X), but I don’t usually use Docker for storing my passwords I just happen to be using it for this tutorial, so moving on !

Folders

It’s also important to note that Pass supports folder structures, as shown in my example I am creating a ‘Gmail’ folder and placing a password file called ‘myemail’ with my password in it. In reality I recommend not naming the file after your account/email and using the multiline version to encrypt those details as well. That way you can just stick to the site name for the name of the encrypted file in whatever folder or in the top level of Pass.

Multiline Encrypted Files with Pass

A common use case with Pass is adding an entire encrypted file so you can store more than just a password…

Again retrieving it is as easy as..

Finally if you no longer want the info to be stored in Pass…

If you want to copy you password to the clipboard from a multiline file, you must store your password on the first line of the file !

Deleting An Entry In Pass

Another thing, the output on my mac is much prettier than this `– thing I am getting in the Ubuntu Docker container… Not sure if that’s an Ubuntu issue or Docker, but on the Mac the output is much prettier, which can be seen on the passwordstore.org home page.

So that’s it, Pass is pretty straight forward, easy to work with, depends on GPG security and that is why I like it.

Stay secure, until next time !

 

How to use Boto to Audit your AWS EC2 instance security groups

Published / by tuxninja / Leave a Comment

Boto is a Software Development Kit for accessing the AWS API’s using Python.

https://github.com/boto/boto3

Recently, I needed to determine how many of my EC2 instances were spawned in a public subnet, that also had security groups with wide open access on any port via any protocol to the instances. Because I have an IGW (Internet Gateway) in my VPC’s and properly setup routing tables, instances launched in the public subnets with wide open security groups (allowing ingress traffic from any source) is a really bad thing 🙂

Here is the code I wrote to identify these naughty instances. It will require slight modifications in your own environment, to match your public subnet IP Ranges, EC2 Tags, and Account names.

To run this you also need to setup your .aws/config and .aws/credentials file.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files

Email me tuxninja [at] tuxlabs.com if you have any issues.
Boto is awesome 🙂 Note so is the AWS CLI, but requires some shell scripting as opposed to Python to do cool stuff.

The github for this code here https://github.com/jasonriedel/AWS/blob/master/sg-audit.py

Enjoy !

Preventing (bind9) DNS Naughty-ness (named.conf & iptables/ufw) on Ubuntu

Published / by tuxninja / Leave a Comment

If you run a DNS server on the Internet with a default configuration many people/robots will take advantage of you. The same is true for Mail, but that is another article. Needless to say if you are running a service on the Internet, the naughty goblins will find you. To thwart these dirty criminals all that’s necessary is to configure your named.conf properly. However, since these robotos are being naughty there is a high degree of certainty they are infected endpoints, and as such I really don’t want them coming anywhere near me or my machines. After all for humanity sake we don’t want to be infected by the deadly plague ! This article is short and sweet, here is how to protect your DNS server & your server in one article using named.conf & ufw (iptables).

 

Named.conf.options

Now a days named.conf is really just a file that inherits 3 other files, named.conf.local, named.conf.options, and named.conf.default-zones. The one we are going to fix is named.conf.options. The configuration below should only be applied in a scenario where you want to run an authorative nameserver, and a caching name server, but the key is you only want to allow people to query the cache that ‘you know personally or are you’ vs. allowing the entire internet, because then bad things happen. If this is not the setup you are going for, don’t do this 🙂 But if it is follow along.

Add the following section with the proper IP’s to the top fo the file

Note you can also add a CIDR for a subnet like 192.168.0.0/16

After that’s done under the options {} section… make it look like this

Note, allow transfer is necessary if you have a secondary nameserver that needs to receive updates. Now restart bind9

Ok now all querying including behavior from non-trusted people will not be allowed. If it is working check your /var/log/syslog and you will see some denies like this

Now the above is from my actual log file. I was quite annoyed that clients are basically abusing the hell out of hehehey.ru… so I decided I don’t want to talk to those people at all. To those people I should be a blackhole. To do this I used UFW which is short for uncomplicated firewall, which essentially makes dealing with Iptables much much nicer. It’s only my 2nd time using UFW, but I’ve been using Iptables for well over a decade. Anyway, here is my simple setup with UFW that I came up with.

So we are configuring the default policy to deny all incoming traffic, allow outgoing, and then allow SSH & Apache/Web traffic basically. Next I created a script called block.sh to add ufw deny rules for bad actors I parsed out of my log, here’s what block.sh looks like

Don’t forget to chmod +x your shell script. Then I did this… blocking all bad actors…

Note, use sudo if you don’t run this as root. This will go through my log and find all these bad requests, and block the requestor. It’s quite aggresive, so be careful, make sure you thoroughly limit your parsing with grep to only block things you really don’t want talking to your server, because this blocks ALL traffic from this requestor to your service, not just DNS.

Once that is complete you need to finally permit good DNS requests by running

And then finally enable your firewall

If you are successful you should see entries in your log that look like this

You can also view all your firewall rules by running

Happy Blocking !

 

 

 

Introducing Vault

Published / by tuxninja / Leave a Comment

Vault

Vault is a command line utility for encrypting & decrypting things. Those things are stored on disk in hidden files, meaning in *nix they simply have a ‘.’ in front and don’t show up unless you type ls -la 😉 But anyway, who cares if someone can locate the files ! They are AES encrypted !

I wrote Vault for a couple of reasons.

  1. To learn how to use encryption in Python
  2. To store passwords for stuff
  3. To turn Vault into methods in my Utilities class that I use for my daily programming activities. First write a command line utility, then write a class 🙂

Vault Usage

I literally wrote vault earlier today and even though it is super simple, I still find it cool & useful so I had to share. Here’s a demo on it’s simple usage !

Keys

AES / Vault supports 16, 24, or 32 byte encryption keys. A byte is one character. Your key should be something ambiguous that you can remember and you should not use the ‘0123…’ example below. What would really be paranoid is to encrypt your keys in a separate vault somewhere 🙂 Note, you will need to use the same key to encrypt as decrypt for a given stored piece of encrypted data.

Encrypting

Can You Read The File Contents ?

Decrypting

Alternatively, you can hide the key from the command line, by not specifying it. 

Clone Vault on github

Enjoy !
Jason Riedel