tuxninja

Tuxninja aka Jason Riedel has worked as a Systems & Network Administrator and Code Hacker since 1999. Since 2005 he has worked for PayPal with a focus on Operations Architecture. He is also CEO of Tuxlabs LLC where he dedicates his time to the experimentation and close study of new technologies and programming languages.

How To: Interact with AWS S3 Using the Go SDK and not lose your mind

After these messages we will carry on with our regularly scheduled programming…

Yesterday ( during the scribbling of this article ) AWS suffered one of it’s worst outages in history in the us-east-1 region. A reminder to us all to be multi-region and more importantly multi-cloud. Please see my other articles on HA deployments using AWS and my perspective & caution on the path to centralization or singularity we appear to be on (though the outage may help people wake up).

Now back to your regularly scheduled program…


My team and I are building a CMDB for AWS, which provides us with everything happening in our AWS environment + OS level metadata + change history. There will be a separate article on the CMDB journey, but today I want to focus on a specific service in AWS called S3, which is their object store. S3 is a bit of a special snowflake when it comes to AWS services and because of that I ran into challenges structuring my code, because up until S3 (which was the last service I wrote code for) everything had been very similar, and easily modularized. We will get to more detail, but let’s start this article by covering how to use the Go SDK for AWS.

Dependencies

This article assumes you already program in Go and have Go installed on your machine. To get started you will need a couple additional items.

  1. Download and install the SDK here : https://github.com/aws/aws-sdk-go 
  2. This is the documentation for the SDK, you will need it, bookmark it : http://docs.aws.amazon.com/sdk-for-go/api/
  3. It is extremely helpful when working with the API’s to have aws-shell installed : https://github.com/awslabs/aws-shell
    • This enables you to interact with AWS API’s on the fly so you can understand the output of commands as you are searching for what you are trying to accomplish.

The Collector Structure

The collector is the component in my CMDB architecture that does all the work of collecting the metadata that we shove into our CMDB. The collector is  heavily threaded using go routines for performance. The basic structure looks like this.

  • Call a go routine for each service you want to collect
    • //pass in all accounts, regions (from config-file) and pre-established awsSessions to each account you are collecting
    • Inside of a services go routine, loop overs accounts & regions
      • Launch a go routine for each account & region
        • Inside of those go routines make your AWS API call(s), example DescribeInstances
        • Store the response (I loop through mine and store it in a map using the resource-id as the key)
        • Finally, kick off another go routine to write to our API and store the data.

Ok, so hopefully that seems straight forward as a basic structure…let’s get to why S3 through me for a loop.

S3 Challenges

It will be best if I show you what I tried first, basically I tried to marry my existing pattern to S3 and that certainly was a bad idea from the start. Here was the structure of the S3 part of the code.

  • The S3 go routine gets called from main.go
  • //all accounts, regions and AWS Sessions are past into the next go routine
    • Inside of the S3 go routine, loop over accounts & regions
      • Launch a go routine for each account & region
        • Inside of those go routines List S3 Buckets
          • For each S3 buckets returned
            • Call additional API’s such as GetBucketTagging()

Ok so what happened ? I got a lot of errors that’s what 🙂 Ones like this….

BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
status code: 301, request id:

At first, I thought maybe my code wasn’t thread safe…but that didn’t make much sense given the other services had no issues like this.

So as I debugged my code, I began to realize the buckets list I was getting, wasn’t limited to the region I was passing in/ establishing a session for.

Naturally, I googled can I list buckets for a single region ?

https://github.com/aws/aws-sdk-java/issues/920 (even though this is the Java SDK it still applies)..

"spfink commented on Nov 16, 2016
It is not possible to list the buckets in a single region. Regardless of the endpoint or region that you set, when calling list buckets you will get buckets from all regions.

In order to determine the region of a bucket you can use getBucketLocation(String bucketName).

https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3.java#L1026”

Ah ok, the BucketList being returned on an AWS Session established with a specific account and region, ignores the region. Because S3 Buckets are global to an account, thus all buckets under an account are returned in the ListBuckets() call. I knew S3 buckets were global per account, but failed to expect a matching behavior/output when a specific region is passed into the SDK/API.

Ok so how then can I distinguish where a bucket actually lives?

As spfink says above, I needed to run GetBucketLocation() per bucket. Thus my code structure started to look like this…

  • For each account, region
    • ListBuckets
      • For each bucket returned in that account, region
        • GetBucketLocation
        • If a LocationConstraint (region) is returned, set the new region (otherwise if response is null, do nothing)
        • Get tags for the bucket in account, region

With this code I was still getting errors about region, but why ?

Well I made the mistake of thinking a ‘null’ response from the API for LocationConstraint had no meaning (or meant query it from any region), wrong (null actually means us-east-1 see from my google below) thus the IF condition evaluated false and the existing region from the outer loop was used because GetBucketLocation() returned null and this resulted in many errors.

Here’s what the google turned up..

https://github.com/aws/aws-cli/issues/564

"kyleknap commented on Mar 16, 2015
@xurume

For buckets located in US Standard, the location constraint will be null. For S3, here is the list of region names with their corresponding regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. Notice that the location constraint is none for US Standard.

The CLI uses the values in the region column for the --region parameter. So for S3's US Standard, you need to use us-east-1 as the region.”

So let’s clarify my mistakes…

  1. The S3 ListBuckets call returns all buckets under an account globally.
    • It does not abide by a region configured in an API Session
    • Thus I/you should not loop over regions from a config file for the S3 service.
    • Instead I/you need to find a buckets ‘real’ location using GetBucketLocation
    • Then set the region for actions other than ListBuckets (which is global per account and ignores region passed).
  2. GetBucketLocation returning null, doesn’t mean the bucket is global or that you can interact with the bucket from endpoint you please…it actually means us-east-1 http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

The Working Code

So in the end the working code for S3 looks like this…

  • collector/main.go fires off a bunch of go routines per service we are collecting for.
  • It passes in accounts, and regions from a config file.
  • For the S3 service/file under the ‘services’ package the entry point is a function called StoreS3Resources.

Everything in the code should be self explanatory from that point on. You will note a function call to ‘writeToCis’… CIS is the name of our internal CMDB project/service. Again, I will later be blogging about the entire system in detail once we open source the code. Please keep in mind this code is MVP, it will be changed a lot (optimization, modularized, bug fixes, etc) before & after we open source it, but for now he is the quick and dirty, but hopefully functional code 🙂 Use at your own risk !

package services

import (
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/aws"
	"sync"
	"fmt"
	"time"
	"encoding/json"
	"strings"
)

var wgS3BucketList sync.WaitGroup
var wgS3GetBucketDetails sync.WaitGroup
var accountRegionsMap = make(map[string]map[string][]string)
var accountToBuckets = make(map[string][]string)
var bucketToAccount = make(map[string]string)
var defaultRegion string = "us-east-1"

func writeS3ResourceToCis(resType string, resourceData map[string]interface{}, account string, region string){
	b, err := json.Marshal(resourceData)
	check(err)

	err, status, url := writeToCisBulk(resType, region, b)
	check(err)
	fmt.Printf("%s - %s - %s - %s - Bytes: %d\n", status, url, account, region, cap(b))
}

func StoreS3Resources(awsSessions map[string]*session.Session, accounts []string, configuredRegions []string) {
	s3Start := time.Now()

	wgS3BucketList.Add(1)
	go func () {
		defer wgS3BucketList.Done()
		for _, account := range accounts {
			awsSession := awsSessions[account]
			getS3AccountBucketList(awsSession, account)
		}
	}()
	wgS3BucketList.Wait()

	getS3BucketDetails(awsSessions, configuredRegions)

	s3Elapsed := time.Since(s3Start)
	fmt.Printf("S3 completed in: %s\n", s3Elapsed)
}

func getS3AccountBucketList(awsSession *session.Session, account string) {
	svcS3 := s3.New(awsSession, &aws.Config{Region: aws.String(defaultRegion)})

	//list returned is for all buckets in an account ( no regard for region )
	resp, err := svcS3.ListBuckets(nil)
	check(err)

	var buckets []string

	for _,bucket := range resp.Buckets {
		buckets = append(buckets, *bucket.Name)

		//reverse mapping needed for lookups in other funcs
		bucketToAccount[*bucket.Name] = account
	}

	//a list of buckets per account
	accountToBuckets[account] = buckets
}


func getS3BucketLocation(awsSession *session.Session, bucket string, bucketToRegion map[string]string, regionToBuckets map[string][]string)  {
	wgS3GetBucketDetails.Add(1)
	go func() {
		defer wgS3GetBucketDetails.Done()
		svcS3 := s3.New(awsSession, &aws.Config{Region: aws.String(defaultRegion)}) // default

		var requiredRegion string

		locationParams := &s3.GetBucketLocationInput{
			Bucket: aws.String(bucket),
		}
		respLocation, err := svcS3.GetBucketLocation(locationParams)
		check(err)

		//We must query the bucket based on the location constraint
		if strings.Contains(respLocation.String(), "LocationConstraint") {
			requiredRegion = *respLocation.LocationConstraint
		} else {
			//if getBucketLocation is null us-east-1 used
			//http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
			requiredRegion = "us-east-1"
		}

		bucketToRegion[bucket] = requiredRegion
		regionToBuckets[requiredRegion] = append(regionToBuckets[requiredRegion], bucket)
		accountRegionsMap[bucketToAccount[bucket]] = regionToBuckets
	}()
}

func getS3BucketsTags(awsSession *session.Session, buckets []string, account string, region string) {
	wgS3GetBucketDetails.Add(1)
	go func() {
		defer wgS3GetBucketDetails.Done()
		svcS3 := s3.New(awsSession, &aws.Config{Region: aws.String(region)})

		var resourceData = make(map[string]interface{})

		for _, bucket := range buckets {
			taggingParams := &s3.GetBucketTaggingInput{
				Bucket: aws.String(bucket),
			}
			respTags, err := svcS3.GetBucketTagging(taggingParams)
			check(err)

			resourceData[bucket] = respTags
		}
		writeS3ResourceToCis("buckets", resourceData, account, region)
	}()
}


func getS3BucketDetails(awsSessions map[string]*session.Session, configuredRegions []string) {

	for account, buckets := range accountToBuckets {
		//reset regions for each account
		var bucketToRegion = make(map[string]string)
		var regionToBuckets = make(map[string][]string)
		for _,bucket := range buckets {
			awsSession := awsSessions[account]
			getS3BucketLocation(awsSession, bucket, bucketToRegion, regionToBuckets)
		}
	}
	wgS3GetBucketDetails.Wait()

	//Preparing configured regions to make sure we only write to CIS for regions configured
	var configuredRegionsMap = make(map[string]bool)
	for _,region := range configuredRegions {
		configuredRegionsMap[region] = true
	}

	for account := range accountRegionsMap {
		awsSession := awsSessions[account]
		for region, buckets := range accountRegionsMap[account] {
			//Only proceed if it's a configuredRegion from the config file.
			if _, ok := configuredRegionsMap[region]; ok {
				fmt.Printf("%s %s has %d buckets\n", account, region, len(buckets))
				getS3BucketsTags(awsSession, buckets, account, region)
			} else {
				fmt.Printf("Skipping buckets in %s because is not a configured region\n", region)
			}
		}
	}
	wgS3GetBucketDetails.Wait()
}

How To: Interact with AWS S3 Using the Go SDK and not lose your mind Read More »

How To: Launch EC2 Instances In AWS Using The AWS CLI

 

It occurred to me recently that while I have written articles on Boto for AWS (the Python SDK) I have yet to write articles on how to use the AWS CLI, Terraform and the Go SDK. All of that will come in due time, for starters this article is going to be about the AWS CLI.

To start you will need to install the AWS CLI  following these links:
https://aws.amazon.com/cli/
https://github.com/aws/aws-cli

Note you will need to make sure you have an account with an access key and have setup the required credentials under ~/.aws/ for the CLI to work. How to do this is covered near the end of the second link above to the git repo.

After that is done you are ready to rock and roll. To test it out you can run…

aws ec2 describe-instances

Assuming your default region, and profile settings are correct it should output JSON.

Launching an EC2 instance

To launch an EC2 instance from the command line use the command below replacing the variables preceded with $ with their real values.

aws --profile $account --region $region ec2 run-instances --image-id $image_id --count $count --instance-type $instance_type --key-name $ssh_key_name --subnet-id $subnet_id

(Assuming you have setup the required dependencies like uploading your SSH key to AWS and specifying its name in the command above this should launch your VM).

It should be noted there is a lot more you can to to tweak your instance, such as changing the EBS volume size for your root disk that is launched or tagging. You will see examples of this in my shell script. The purpose of this article is to share a shell script I have written and use whenever I want to quickly launch a test VM (which is common). For more permanent things I use an infrastructure as code approach via Terraform. But the need for launching quick test VM’s never goes away, thus this shell script was born. You will notice my script auto-tags our VM’s…I do this because in our environment if you VM isn’t tagged appropriately it is deleted + it’s courtesy in an AWS environment to tag your resources, otherwise no one will ever what tree to bark up when there is a problem such as ‘are you still using this cause it looks idle?’ 🙂

My Shell Script for Launching EC2 VM’s

#!/bin/bash

# Global Settings
account="my-account"
region="us-east-1"

# Instance settings
image_id="ami-03ebd214" # ubuntu 14.04
ssh_key_name="my_ssh_key-rsa-2048"
instance_type="m4.xlarge"
subnet_id="subnet-b8214792"
root_vol_size=20
count=1

# Tags
tags_Name="my-test-instance"
tags_Owner="tuxninja"
tags_ApplicationRole="Testing"
tags_Cluster="Test Cluster"
tags_Environment="dev"
tags_OwnerEmail="tuxninja@tuxlabs.com"
tags_Project="Test"
tags_BusinessUnit="Cloud Platform Engineering"
tags_SupportEmail="tuxninja@tuxlabs.com"

echo 'creating instance...'
id=$(aws --profile $account --region $region ec2 run-instances --image-id $image_id --count $count --instance-type $instance_type --key-name $ssh_key_name --subnet-id $subnet_id --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": $root_vol_size } } ]" --query 'Instances[*].InstanceId' --output text)

echo "$id created"

# tag it

echo "tagging $id..."

aws --profile $account --region $region ec2 create-tags --resources $id --tags Key=Name,Value="$tags_Name" Key=Owner,Value="$tags_Owner"  Key=ApplicationRole,Value="$tags_ApplicationRole" Key=Cluster,Value="$tags_Cluster" Key=Environment,Value="$tags_Environment" Key=OwnerEmail,Value="$tags_OwnerEmail" Key=Project,Value="$tags_Project" Key=BusinessUnit,Value="$tags_BusinessUnit" Key=SupportEmail,Value="$tags_SupportEmail" Key=OwnerGroups,Value="$tags_OwnerGroups"

echo "storing instance details..."
# store the data
aws --profile $account --region $region ec2 describe-instances --instance-ids $id > instance-details.json

echo "create termination script"
echo "#!/bin/bash" > terminate-instance.sh
echo "aws --profile $account --region $region ec2 terminate-instances --instance-ids $id" >> terminate-instance.sh
chmod +x terminate-instance.sh

After substituting the required variables at the top with your real values you can run this script. Notice that after creating the VM I capture the instance details in a file & the ID in a variable so I can subsequently tag it, and then I create a termination script…this makes for very simple operations when you need to repeatedly start and then kill/destroy/delete a VM.

Using these scripts should come in quite handy. A copy of create-instance.sh can be found on my github here.

One other thing… I use the normal AWS CLI for automation as shown here…but for poking around interactively I use something called ‘aws-shell’ formerly ‘saw’. Check it out and you won’t be disappointed !

My next post will be on Terraform or the Go SDK…but both are coming soon!

How To: Launch EC2 Instances In AWS Using The AWS CLI Read More »

Storing passwords securely using Pass (GPG)

Today we live in an endless sea of passwords, which are a very inefficient and ineffective means of securing our data & environments. Many companies are trying to solve this problem using a variety of techniques that all revolve around various forms of multi-factor authentication.

However, in the mean time were all screwed 😉

Just kidding. Quick PSA though, use two factor authentication at a minimum everywhere you can ESPECIALLY your email, since it’s used for password recovery on other sites. Ok then moving on…

There are many password managers like LastPass and 1Password, which do a fairly effective job at providing convenience and prevent you from scribbling down your passwords on paper (STOP IT !!!). However, I personally can’t get passed the whole ‘store all my passwords in one super secure vault on the Internet’ thing. To be fair some of these password managers can be downloaded on your machine and ran locally, but there are two other drawbacks to those I found.

  1. Some of them are not free and…
  2. Some of them have ugly and clunky UI’s

So what do I like/use then ? I use something called ‘pass’. Which is a command line utility that wraps GPG. The reason I use it is because…

  1. I love using command line utilities over GUI, I find it far more convenient and…
  2. I was going to write this exact utility (a GPG wrapper) until I found out someone else did and…
  3. Because I like GPG.

At most of the organizations I have worked at, password management was done poorly i.e. everyone used different approaches and there was no governance or oversight. I hope with this article to make folks aware of what I feel is a simple, effective method that every unix savvy administrator should use.

FYI Pass provides migration scripts from the most popular password manager tools on their website.

Introducing Pass

From the Pass site “Password management should be simple and follow Unix philosophy. With pass, each password lives inside of a gpg encrypted file whose filename is the title of the website or resource that requires the password. These encrypted files may be organized into meaningful folder hierarchies, copied from computer to computer, and, in general, manipulated using standard command line file management utilities.”

Where Can You Get or Learn More About Pass ? 

https://www.passwordstore.org/

Installing Pass

Depending on your operating system there are various ways to install

Ubuntu/Debian

sudo apt-get install pass

Fedora / RHEL

sudo yum install pass

Mac

brew install pass
echo "source /usr/local/etc/bash_completion.d/password-store" >> ~/.bashrc

Since I already installed pass on my Mac a while back I will be installing it on a Docker container with Ubuntu 16.04.

root@0b415380eb80:/# apt-get install -y pass

After pass successfully installs, try running it

root@0b415380eb80:/# pass
Error: password store is empty. Try "pass init".
root@0b415380eb80:/#

Well that is pretty straight forward, it appears we need to initiliaze the db.

root@0b415380eb80:/# pass init
Usage: pass init [--path=subfolder,-p subfolder] gpg-id...
root@0b415380eb80:/#

Looks like we need to provide ‘key’…can that be just anything?

root@0b415380eb80:/# pass init "tuxlabs Password Key"
mkdir: created directory '/root/.password-store/'
Password store initialized for tuxlabs Password Key
root@0b415380eb80:/# pass
Password Store
root@0b415380eb80:/#

Now our password store looks initialized ! Let’s try inserting a password into the DB !

root@0b415380eb80:/# pass insert Gmail/myemail
mkdir: created directory '/root/.password-store/Gmail'
Enter password for Gmail/myemail:
Retype password for Gmail/myemail:
gpg: tuxlabs Password Key: skipped: No public key
gpg: [stdin]: encryption failed: No public key
root@0b415380eb80:/#

Uh oh what happened ? Well remember I said it uses GPG, and we not only don’t have a gpg key setup in our Docker container, but we initialized our Pass DB without using a GPG Key (the whole point) !

root@0b415380eb80:/# gpg --list-keys
root@0b415380eb80:/#

To remedy this we need to create a GPG key

Creating your GPG Key

root@0b415380eb80:/# gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Tuxninja
Email address: tuxninja@tuxlabs.com
Comment: TuxLabs
You selected this USER-ID:
    "Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: gpg-agent is not available in this session
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
............+++++
...................+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
...+++++
+++++
gpg: key 5B2F89A5 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/5B2F89A5 2016-12-14
      Key fingerprint = 5FF6 1717 4415 03FF D455  7516 CF8E 1BDC 5B2F 89A5
uid                  Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>
sub   4096R/EF0F232F 2016-12-14

root@0b415380eb80:/#

To view your GPG key run

root@0b415380eb80:/# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   4096R/5B2F89A5 2016-12-14
uid                  Tuxninja (TuxLabs) <tuxninja@tuxlabs.com>
sub   4096R/EF0F232F 2016-12-14

root@0b415380eb80:/#

Now we can see we have one GPG key, with the ID 5B2F89A5

Let’s try re-initializing Pass. 

root@0b415380eb80:/# pass init "5B2F89A5"
Password store initialized for 5B2F89A5
root@0b415380eb80:/#

But we have a problem, re-initializing Pass doesn’t get rid of our previous insert into the db. As you can see here our Pass DB is effectively corrupt.

root@0b415380eb80:~# pass
Password Store
`-- Gmail
root@0b415380eb80:~# pass rm Gmail
Are you sure you would like to delete Gmail? [y/N] y
rm: cannot remove '/root/.password-store/Gmail': Is a directory
root@0b415380eb80:~# pass rm Gmail/myemail
Error: Gmail/myemail is not in the password store.
root@0b415380eb80:~#

Hmmm, what’s a guy to do….

root@0b415380eb80:~# rm -rf .password-store/Gmail/
root@0b415380eb80:~# pass
Password Store
root@0b415380eb80:~#

Yes it really was that simple, and that is one more reason why I love pass.

You can also initialize your password store using git for version control, see the passwordstore.org website for more info !

Now let’s insert some good stuff.

Inserting A Password into Pass

root@0b415380eb80:~# pass insert Gmail/myemail
Enter password for Gmail/myemail:
Retype password for Gmail/myemail:
root@0b415380eb80:~# pass
Password Store
`-- Gmail
    `-- myemail
root@0b415380eb80:~#

That seems to have worked. Let’s try to retrieve the pass.

Retrieving A Password In Pass

root@0b415380eb80:~# pass Gmail/myemail
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/root/.gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
testpass
root@0b415380eb80:~# pass Gmail/myemail
testpass
root@0b415380eb80:~#

Note, I retrieve the password twice using my GPG Passsword (You will be prompted through a curses interface to enter your passphrase). Then I run it again, because of the initial GPG migration messages just to show how it would normally work after you’ve used GPG once with Pass.

Now let’s say someone is standing over your shoulder, you want to access your passsword, but you don’t want them to see it. You can get it straight to your clipboard by using -c.

Copying Passwords To Your Clipboard

pass -c Gmail/myemail
Copied Gmail/myemail to clipboard. Will clear in 45 seconds.

Docker Issue ?

Notice the prompt is not included in the above example ? That is cause it didn’t actually work. Apparently, it doesn’t work in Docker due to not having display dependencies installed/configured. So what I show above is the output from my mac…but my actual Docker related error was.

root@0b415380eb80:~# pass -c Gmail/myemail
Error: Can't open display: (null)
Error: Could not copy data to the clipboard
root@0b415380eb80:~#

There might be an easy way to fix this (like install X), but I don’t usually use Docker for storing my passwords I just happen to be using it for this tutorial, so moving on !

Folders

It’s also important to note that Pass supports folder structures, as shown in my example I am creating a ‘Gmail’ folder and placing a password file called ‘myemail’ with my password in it. In reality I recommend not naming the file after your account/email and using the multiline version to encrypt those details as well. That way you can just stick to the site name for the name of the encrypted file in whatever folder or in the top level of Pass.

Multiline Encrypted Files with Pass

A common use case with Pass is adding an entire encrypted file so you can store more than just a password…

root@0b415380eb80:~# pass insert -m tuxlabs/databases
mkdir: created directory '/root/.password-store/tuxlabs'
Enter contents of tuxlabs/databases and press Ctrl+D when finished:

this is an example of a multiline
encrypted file
this way you can store more than just a password you can store user/pass/url etc
root@0b415380eb80:~# pass
Password Store
|-- Gmail
|   `-- myemail
`-- tuxlabs
    `-- databases
root@0b415380eb80:~#

Again retrieving it is as easy as..

root@0b415380eb80:~# pass tuxlabs/databases
this is an example of a multiline
encrypted file
this way you can store more than just a password you can store user/pass/url etc
root@0b415380eb80:~#

Finally if you no longer want the info to be stored in Pass…

If you want to copy you password to the clipboard from a multiline file, you must store your password on the first line of the file !

Deleting An Entry In Pass

root@0b415380eb80:~# pass rm Gmail/myemail
Are you sure you would like to delete Gmail/myemail? [y/N] y
removed '/root/.password-store/Gmail/myemail.gpg'
root@0b415380eb80:~# pass rm tuxlabs/databases
Are you sure you would like to delete tuxlabs/databases? [y/N] y
removed '/root/.password-store/tuxlabs/databases.gpg'
root@0b415380eb80:~# pass
Password Store
root@0b415380eb80:~#

Another thing, the output on my mac is much prettier than this `– thing I am getting in the Ubuntu Docker container… Not sure if that’s an Ubuntu issue or Docker, but on the Mac the output is much prettier, which can be seen on the passwordstore.org home page.

So that’s it, Pass is pretty straight forward, easy to work with, depends on GPG security and that is why I like it.

Stay secure, until next time !

 

Storing passwords securely using Pass (GPG) Read More »

Setting up Netflix’s Edda (CMDB) in AWS on Ubuntu

If you are running any kind of environment with greater than 10 servers, than you need a CMDB (Configuration Management DataBase). CMDB’s are the brain of your fleet & it’s environment. You can store anything in a CMDB, but commonly the metadata in CMDB’s consists of any of the following physical & digital asset inventory, software licenses, software configuration data, policy information, relationships (I.E. This VM—> Compute –> Rack –> Availability Zone –> Datacenter), automation metadata, and more… they also commonly provide change history for changes in your environment.

In the world of infrastructure as code, CMDB is king.

CMDB’s enable endless automation possibilities, without them you are stuck gathering and collecting ‘current’ configuration state about your infrastructure every time you want perform an automated change or run an audit/report . In my career I have built or been a part of CMDB efforts at nearly every company I have worked for. They are simply necessary, and by their nature they tend to require the choice of ‘built by us’ vs ‘buy or run’.

However, if you have the luxury of only running in AWS, you are in luck, because Netflix (The AWS poster child)  open sourced Edda in 2012 for this purpose!

Rather than talk about the specific features of Edda refer to the blog post or documentation, I want to keep this article short and jump right into setting up Edda, which is a bit tricky, because the documentation is out of date!

Setting Up Edda (2016)

First, in AWS you need setup an EC2 VM that has at least.. 6G for OS + dependencies including Mongo, and then however much disk you need to store the metadata for your environment (keep in mind it keeps change history). Personally I just created a root partition with 100G to keep things simple. For instance type I used ‘m4.xlarge’ and the Ubuntu version is 14.04.

After booting the VM, SSH to it and create a directory wherever your storage is allocated partition wise to store Edda & it’s dependencies. I will be using /cmdb/ in my example.

Initial Install Steps

mkdir /cmdb
cd /cmdb
export JAVA_OPTS="-Xmx1g -XX:MaxPermSize=256M"
git clone https://github.com/Netflix/edda.git
sudo add-apt-repository -y ppa:webupd8team/java &> /dev/null
sudo apt-get update
sudo debconf-set-selections <<< 'oracle-java8-installer shared/accepted-oracle-license-v1-1 boolean true'
sudo apt-get install -y oracle-java8-installer
sudo apt-get install -y scala
sudo apt-get install make

cd /cmdb/edda
make build

For the record, the Edda Wiki has the build steps wrong, it appears they no long are using Gradle, but have switch to SBT… which reminds me be aware Edda is written in Scala, which isn’t as popular as Java, Python etc… in addition it’s functional programming, which I don’t personally know a lot about, but I hear it’s got quite the learning curve..so beware if you need to make custom code changes, I would not recommend it, unless you know Scala ! 🙂

After the build of Edda succeeds, install Mongo

apt-get install -y mongodb

That’s it for dependencies

Configuring Mongo

For Edda to use Mongo all we need to do is ‘use’ the database we want to use for Edda & create an associated user. (Mongo will auto-create DB’s upon insert).

mongo

> use edda
> db.addUser({user:'edda',pwd:'t00t0ri4l',Roles: { edda: ['readWrite']}, roles: []})

You can test the user is working by doing… 

$ mongo edda -u edda -p
MongoDB shell version: 2.4.9
Enter password:
connecting to: edda
Server has startup warnings:
Sat Dec 10 00:53:21.093 [initandlisten]
Sat Dec 10 00:53:21.094 [initandlisten] ** WARNING: You are running on a NUMA machine.
Sat Dec 10 00:53:21.094 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Sat Dec 10 00:53:21.094 [initandlisten] **              numactl --interleave=all mongod [other options]
Sat Dec 10 00:53:21.094 [initandlisten]
>

Configuring Edda

Under /cmdb/edda/src/main/resources we need to modify ‘edda.properties’ with valid config values for accounts, regions & mongo access.

Relevant Mongo Values

edda.mongo.address=127.0.0.1:27017
edda.mongo.database=edda
edda.mongo.user=edda
edda.mongo.password=t00t0ri4l

Account & Region Values 

edda.accounts=dev.us-east-1
edda.dev.us-east-1.region=us-east-1
edda.dev.us-east-1.aws.accessKey=fakeaccesskey
edda.dev.us-east-1.aws.secretKey=fakesecret

The above example is using one account and only one region. The Edda configuration uses generic labels, they are very flexible, but when using them you might be confused by the name of the label as it’s intent. Don’t fall into that trap, I did, and then I found this post on Google Groups… Check it out to gain more insight on how the configuration works and can be tweaked for  your needs. There is also the standard documentation, but it’s a little light IMO.

Running Edda

Congrats you made it, time to run Edda ! Again the documentation has this wrong (listed as gradle & Jetty)…instead were using SBT + Jetty…

$ cd /cmdb/edda/
$ ./project/sbt
> jetty:start

If everything goes smoothly you will start to see logs about crawling AWS API’s spewing to your screen 🙂 After about 2 minutes you should see data. You can check by doing a curl.

curl http://127.0.0.1:8080/api/v2/view/instances

This API URL should return a JSON object with instance ID’s for the account & region specified.

Additionally, Edda is listening on whatever private IP address you have setup, you will just need to modify the default security group to allow 8080 on your machine.

I get a bit frustrated with out of date documentation..so I hope this helps ! Happy automating !

Setting up Netflix’s Edda (CMDB) in AWS on Ubuntu Read More »

AWS, Google Cloud, Azure, and the centralized future of the Internet

I have just left AWS re-invent and I wanted to give my brief thoughts on the future of cloud computing. I believe in the next few years the shift we have been witnessing will be completed. That is to say that the thousands of enterprises and small businesses alike will finish their migrations to public clouds, simply because the benefits are far too great. Less people, less hardware, less glue code, more functionality, more value etc. AWS will be the dominant public cloud for the next couple of years minimum due to their first to market advantage and If you look at the announcements at re-invent 2016, you see a series of products that solve common problems. In fact a lot of the “innovations” AWS announced today, replace many SaaS solutions who ironically (or maybe not so much) are hosted on AWS. None of this concerns me, this is great disruption. AWS is teaching these businesses to move even further up and away from creating tooling for DevOps as products (they will take care of that) and focus on products that provide value differently, like sifting through massive amounts of data, increasing the quality and providing intelligence from that data. This is all great, and it’s definitely where things are headed, kudos to Amazon for guiding folks.

BUT here is what is disturbing to me, and it’s seems like no one talks about it. It’s as if they can’t see the elephant in the room.

The elephant in the room is that every company in the world is converging on a fewer number/types of physical devices, paths, datacenters etc. This means the global failure domains that should be distributed in nature are actually becoming more centralized therefore the risk of massive security or availability (outage) events is higher.

If you really think about it, Cloud was always the return of Utility computing (mainframes) etc. and as we go down this journey it’s becoming more evident it’s simply a more distributed version of mainframe, and in my opinion, at the moment that is giving people a false sense of comfort.

Early in my career almost 20 years ago, I was working at an ISP, and one of the core services for the Internet (DNS) was directly attacked. There was of course a widespread failure and what we soon realized was the Internet had more of a shared fate than most believed.

Fast forward to present day, and it just happened again with Dyn who hosted DNS for some very critical companies. This problem hasn’t been solved, it is getting worse.

This is the same problem we are going to have with AWS, Google Cloud and Azure.

As companies & governments converge on datacenters, and those datacenters connect to common interconnected fabrics (aka the Internet itself) and resources.

The Internet is becoming far more grouped…far more shared & central…and thus the shared fate of the Internet will lie solely on the shoulders of giants or as we like to call them in our industry monoliths.

Public cloud monoliths, monopolies…etc

Perhaps my worries will be mitigated by fantastic diversification and investment in truly distributed, distinct network paths, independent power plants, etc…BUT my fear is the convergence is happening so fast, the providers won’t be able to make that a reality fast enough and what’s the incentive for them ? They have to invest a tremendous amount of capital when they are already successful and this problem has not publicly and visibly humiliated us yet. But I fear that it will in the next few years…

So Godspeed to journey men of the cloud, as we enjoy the luxuries that AWS, Azure, & Google Cloud offer us. We are entering a beautiful and dangerous time. Beware, and hedge your company & product by distributing it as much as you can to avoid these central dependencies. Avoid these massive shared, global failure domains and ensure you diversified to avoid increased security risk.

AWS, Google Cloud, Azure, and the centralized future of the Internet Read More »