How To: Enable SSH On A Cisco 2950

First you have to make sure you are running a version of code that has encryption. See my previous article for instructions on how to upgrade the code. Once your code is upgraded, here are the steps to enable SSH on a Cisco 2950.

Generate An SSH Key

switch-2950-1.tuxlabs.com#config t
Enter configuration commands, one per line.  End with CNTL/Z.
switch-2950-1.tuxlabs(config)#crypto key generate rsa modulus 1024
The name for the keys will be: switch-2950-1.tuxlabs.com.tuxlabs.com

% The key modulus size is 1024 bits
Generating RSA keys ...
[OK]

switch-2950-1.tuxlabs(config)#exit
switch-2950-1.tuxlabs.com#

Verify your key like so

switch-2950-1.tuxlabs.com#show crypto key mypubkey rsa 
% Key pair was generated at: 00:10:35 UTC Mar 1 1993
Key name: switch-2950-1.tuxlabs.com.tuxlabs.com
 Usage: General Purpose Key
 Key Data:
  30819F30 0D06092A 864886F7 0D010101 05000381 8D003081 89028181 00E6AA25 
  8DB58145 F882CD0B C62F5123 AB0064A6 A09BD636 FA854D82 B1510A31 3A00606E 
  00F601F1 ECF64FCC 0F516E73 E80E0961 9CCCE91B 5C3D5919 4803B805 04AC2633 
  9D0A32E8 0196F572 5CE9FFF4 A5C27FC4 698DE75B F0573804 22D0CCFE 58936F4E 
  5BE394F4 3BDED1AC DC1BF1C9 5E71ABD5 34F1C21E CDA47B7E 72D40C34 6B020301 0001
% Key pair was generated at: 00:10:41 UTC Mar 1 1993
Key name: switch-2950-1.tuxlabs.com.tuxlabs.com.server
 Usage: Encryption Key
 Key Data:
  307C300D 06092A86 4886F70D 01010105 00036B00 30680261 009F3354 2ECB6FB6 
  7A19D04D 929FEB38 05145D39 C9DB6CAB 5AC1A262 14FEFFBC DE6E5FA9 8565BEA6 
  1A888A92 C7D1ED2E DB8D3894 D972C9AE 853DFB98 8261D518 0F8A994C 9293C49C 
  0E946A95 0F89EA08 45E4DCB7 74F5A23C CDC5938C CD01C6C1 4D020301 0001
switch-2950-1.tuxlabs.com#

Wow 1993, feels good to be a time machine 🙂

Configure the allowed number of retries

switch-2950-1.tuxlabs.com#config t                     
Enter configuration commands, one per line.  End with CNTL/Z.
switch-2950-1.tuxlabs(config)#ip ssh authentication-retries 5
switch-2950-1.tuxlabs(config)#exit
switch-2950-1.tuxlabs.com#

Enabling SSH on the VTYs

switch-2950-1.tuxlabs.com#config t           
Enter configuration commands, one per line.  End with CNTL/Z.
switch-2950-1.tuxlabs(config)#line vty 0 4
switch-2950-1.tu(config-line)#login local
switch-2950-1.tu(config-line)#transport input ssh
switch-2950-1.tu(config-line)#line vty 5 15
switch-2950-1.tu(config-line)#login local
switch-2950-1.tu(config-line)#transport input ssh
switch-2950-1.tu(config-line)#exit
switch-2950-1.tuxlabs(config)#exit
switch-2950-1.tuxlabs.com#wr mem
Building configuration...
[OK]
switch-2950-1.tuxlabs.com#

Configuring A Username

When you’re running telnet you don’t need a username. But when you are using SSH, you do.

switch-2950-1.tuxlabs.com#config t 
Enter configuration commands, one per line.  End with CNTL/Z.
switch-2950-1.tuxlabs(config)#username tuxninja privilege 15 password sup3rs3cr3t
switch-2950-1.tuxlabs(config)#exit
switch-2950-1.tuxlabs.com#wr mem
Building configuration...
[OK]
switch-2950-1.tuxlabs.com#

Now were golden, let’s test.

➜  ~  ssh tuxninja@switch-2950-1.tuxlabs.com
The authenticity of host 'switch-2950-1.tuxlabs.com (192.168.1.2)' can't be established.
RSA key fingerprint is 21:6b:44:bb:24:ff:ef:14:9d:f2:00:44:64:3d:3b:f8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'switch-2950-1.tuxlabs.com,192.168.1.2' (RSA) to the list of known hosts.
tuxninja@switch-2950-1.tuxlabs.com's password: 

switch-2950-1.tuxlabs.com#show privil
switch-2950-1.tuxlabs.com#show privilege 
Current privilege level is 15
switch-2950-1.tuxlabs.com#exit
Connection to switch-2950-1.tuxlabs.com closed.
➜  ~

Awesome ! That concludes this short tutorial.

How To: Upgrade IOS On A Cisco 2950

My cisco 2950 came with an older IOS version 12.1, but more importantly, one that does not support encryption, and thus I cannot use SSH. I need to upgrade the code aka IOS Image on this switch to enable SSH. So here we go, I’ll be referencing the following guide : http://kb.promise.com/KnowledgebaseArticle10139.aspx throughout this how to article. Note: You should be in enable/privilege 15 mode for the duration of this article.

Existing version info

switch-2950-1.tuxlabs.com#show version
Cisco Internetwork Operating System Software 
IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(19)EA1c, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2004 by cisco Systems, Inc.
Compiled Mon 02-Feb-04 23:29 by yenanh
Image text-base: 0x80010000, data-base: 0x8058A000

ROM: Bootstrap program is C2950 boot loader

switch-2950-1.tuxlabs.com uptime is 1 hour, 38 minutes
System returned to ROM by power-on
System image file is "flash:/c2950-i6q4l2-mz.121-19.EA1c.bin"

cisco WS-C2950T-24 (RC32300) processor (revision P0) with 20808K bytes of memory.
Processor board ID FOC0812T17M
Last reset from system-reset
Running Enhanced Image
24 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0F:8F:DB:4E:00
Motherboard assembly number: 73-6114-10
Power supply part number: 34-0965-01
Motherboard serial number: FOC0812243L
Power supply serial number: DAB080842YJ
Model revision number: P0
Motherboard revision number: A0
Model number: WS-C2950T-24
System serial number: FOC0812T17M
Configuration register is 0xF

switch-2950-1.tuxlabs.com#

Cisco Image & TFTP Server

I had to register for the Cisco site to download the latest image, with support for encryption. This is the image I will be installing : c2950-i6k2l2q4-mz.121-22.EA14.bin

After the image is downloaded, we need to configure the TFTP server. Mac OS X comes with tftpd automatically. You are going to want to place the imagine in /private/tftpboot.  After you have copied the image there, make absolutely sure you update the permissions. Otherwise, your tftp request will timeout from your device.

➜  ~  sudo chmod 766 /private/tftpboot/*
➜  ~  ls -l /private/tftpboot       
total 7272
-rwxrw-rw-@ 1 root  wheel  3722814 Sep  7 23:21 c2950-i6k2l2q4-mz.121-22.EA14.bin
➜  ~

After you update the permissions you are ready to start TFTP.

 

➜  ~  netstat -atp UDP | grep tftp                                     
➜  ~  sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist 
dyld: DYLD_ environment variables being ignored because main executable (/usr/bin/sudo) is setuid or setgid
➜  ~  netstat -atp UDP | grep tftp                                   
udp4       0      0  *.tftp                 *.*                               
udp6       0      0  *.tftp                 *.*                               
➜  ~

Great TFTP is running. Now we are ready to request it from the Cisco switch, aka the client in this scenario.

Copy TFTP Flash

switch-2950-1.tuxlabs.com#copy tftp flash
Address or name of remote host []? 192.168.1.123
Source filename []? c2950-i6k2l2q4-mz.121-22.EA14.bin
Destination filename [c2950-i6k2l2q4-mz.121-22.EA14.bin]? 
Accessing tftp://192.168.1.123/c2950-i6k2l2q4-mz.121-22.EA14.bin...
Loading c2950-i6k2l2q4-mz.121-22.EA14.bin from 192.168.1.123 (via Vlan1): !!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!O!OO!OO!OOOOO!OOOO!
%Error copying flash:/c2950-i6k2l2q4-mz.121-22.EA14.bin (No space left on device)
switch-2950-1.tuxlabs.com#

Uh oh shaggy ! I am out of space. After doing a ‘dir flash’ I saw that really I had no choice, but to delete my existing flash image to make room for the new one. Feels dangerous and scary, but luckily this is my lab environment 🙂

Deleting From Flash

switch-2950-1.tuxlabs.com#delete flash://c2950-i6q4l2-mz.121-19.EA1c.bin
Delete filename [c2950-i6q4l2-mz.121-19.EA1c.bin]? 
Delete flash:/c2950-i6q4l2-mz.121-19.EA1c.bin? [confirm]
switch-2950-1.tuxlabs.com#

Copy TFTP Flash Again (This time with our fingers crossed)

switch-2950-1.tuxlabs.com#copy tftp flash                               
Address or name of remote host [192.168.1.123]? 
Source filename [c2950-i6k2l2q4-mz.121-22.EA14.bin]? 
Destination filename [c2950-i6k2l2q4-mz.121-22.EA14.bin]? 
Accessing tftp://192.168.1.123/c2950-i6k2l2q4-mz.121-22.EA14.bin...
Loading c2950-i6k2l2q4-mz.121-22.EA14.bin from 192.168.1.123 (via Vlan1): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 3722814 bytes]

3722814 bytes copied in 143.080 secs (26019 bytes/sec)
switch-2950-1.tuxlabs.com#

Whew…close one 🙂 To be on the safe side we can verify our image like this.

switch-2950-1.tuxlabs.com#verify /md5 c2950-i6k2l2q4-mz.121-22.EA14.bin
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Done!
verify /md5 (flash:c2950-i6k2l2q4-mz.121-22.EA14.bin) = 8d3250ee253b81b7fe2762e281773fbc


switch-2950-1.tuxlabs.com#

Next we make our new flash image bootable.

switch-2950-1.tuxlabs.com#config t
switch-2950-1.tuxlabs(config)#boot system flash:c2950-i6k2l2q4-mz.121-22.EA14.bin
switch-2950-1.tuxlabs(config)#exit
switch-2950-1.tuxlabs.com#show boot
BOOT path-list:       flash:c2950-i6k2l2q4-mz.121-22.EA14.bin
Config file:          flash:/config.text
Private Config file:  flash:/private-config.text
Enable Break:         no
Manual Boot:          no
HELPER path-list:     
NVRAM/Config file
      buffer size:    32768
switch-2950-1.tuxlabs.com#wr mem
Building configuration...
[OK]
switch-2950-1.tuxlabs.com#

Great, that looks good, now we are ready to reload our switch !

switch-2950-1.tuxlabs.com#reload
Proceed with reload? [confirm]
Connection closed by foreign host.
➜  ~

Once the switch comes back to life, validate the version info.

switch-2950-1.tuxlabs.com#show version
Cisco Internetwork Operating System Software 
IOS (tm) C2950 Software (C2950-I6K2L2Q4-M), Version 12.1(22)EA14, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by cisco Systems, Inc.
Compiled Tue 26-Oct-10 10:35 by nburra
Image text-base: 0x80010000, data-base: 0x80680000

ROM: Bootstrap program is C2950 boot loader

switch-2950-1.tuxlabs.com uptime is 2 minutes
System returned to ROM by power-on
System image file is "flash:c2950-i6k2l2q4-mz.121-22.EA14.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2950T-24 (RC32300) processor (revision P0) with 19911K bytes of memory.
Processor board ID FOC0812T17M
Last reset from system-reset
Running Enhanced Image
24 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:0F:8F:DB:4E:00
Motherboard assembly number: 73-6114-10
Power supply part number: 34-0965-01
Motherboard serial number: FOC0812243L
Power supply serial number: DAB080842YJ
Model revision number: P0
Motherboard revision number: A0
Model number: WS-C2950T-24
System serial number: FOC0812T17M
Configuration register is 0xF

switch-2950-1.tuxlabs.com#

We went from 2004, to 2010…but 2010 is the latest image available for my ancient switch ! Awesome. Now we are ready to enable SSH in the next article !

Thanks for reading,
Jason Riedel

 

Runner Features Have Been Updated !

Runner Reminder

Runner is a command line tool for running commands on thousands of devices that support SSH. I wrote Runner and use it every single day, because unlike Ansible, Runner truly has no dependencies on the client or server side other than SSH. I have used Runner to build entire datacenters, so it is proven and tested and has a lot of well thought out features, which brings me to todays post. Since I initially debuted Runner I have added a lot of features, but I had yet to check them into github, until now. Here is a run down of Runner’s features.

Features

  • Runner takes your login credentials & doesn’t require you to setup SSH keys on the client machines/devices.
  • Runner can be used through a bastion/jump host via an SSH tunnel (see prunner.py)
  • Runner reads it’s main host list from a file ~/.runner/hosts/hosts-all
  • Runner can accept custom hosts lists via -f
  • -e can be used to echo a command before it is run, this is useful for running commands on F5 load balancers for example, when no output is returned on success.
  • -T will allow you to tune the number of threads, but be careful you can easily exhaust your system or site resources (I.E. do NOT DOS your LDAP authentication servers by trying to do hundreds of threads across thousands of machines, unless you know they can handle it 😉 ).
  • -s is for sudo for those users who have permissions in the sudoers file.
  • -1 reduces any host list down to one host per pool. It uses a regex, which you will likely have to modify for your own host / device naming standard.
  • -r can be used to supply a regular expression for matching hosts. Remember sometimes you have to quote the regex and/or escape the shell when using certain characters.
  • -c will run a single command on many hosts, but -cf will run a series of commands listed in a file on any hosts specified. This is particularly useful for automations. For example, I used it to build out load balancer virtuals and pools on an F5.
  • -p enables you to break apart the number of hosts to run at a time using a percentage. This is a handy & more humanized way to ensure you do not kill your machine or the infrastructure you are managing when you crank threads through the roof 😉

Now that I have taken the time to explain some of those cool features, here’s an example of what it looks like in action.

Runner Demo

Host List

➜  ~  runner -l -r tuxlabs                               
tuxlabs.com
old.tuxlabs.com

There were 2 hosts listed.
➜

Basic Run Using Only -c, -u and defaults

Note: User defaults to the user you are logged in as if you don’t specify -u . Since I am logged in as ‘jriedel’ I have specified the user tuxninja instead.

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),27(sudo)
old.tuxlabs.com: uid=1000(tuxninja) gid=1000(tuxninja) groups=1000(tuxninja),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),114(sambashare),1001(admin)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.01:59:59
➜  ~

The Same Run Using Sudo 

Note: I just realized if you do not prompt for a password for sudo it will fail, I will have to fix that ! Whoops ! P.S. You should always prompt for a password when using sudo !

➜  ~  runner -r tuxlabs -c 'id' -u tuxninja -s
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'tuxlabs'
RUNNER [INFO]: 2 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 10 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INFO]: SUDO IS ON
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)
old.tuxlabs.com: uid=0(root) gid=0(root) groups=0(root)

RUNNER [RESULT]: Successfully logged into 2/2 hosts and ran your command(s) in 0:00:03 second(s)
RUNNER [RESULT]: There were 0 login failures.


RUNNER [INFO]: Your logfile can be viewed @ /Users/jriedel/.runner/logs/runner.log.2015-08-25.02:10:27
➜  ~

Runner with a command file in super quiet mode  !

➜  ~  cat lets-run-these 
uptime
who
date
uptime -s
➜  ~  

➜  ~  runner -r tuxlabs -cf lets-run-these -T 2 -p 50 -qq -u tuxninja
RUNNER [INPUT]: Please Enter Site Pass: 
tuxlabs.com:  05:17:51 up 14 days,  4:45,  0 users,  load average: 0.00, 0.01, 0.05
old.tuxlabs.com:  03:17:52 up 80 days, 47 min,  1 user,  load average: 0.42, 0.69, 0.78
old.tuxlabs.com: root     pts/0        2015-08-25 02:23 (173.224.162.99)
tuxlabs.com: Tue Aug 25 05:17:52 EDT 2015
old.tuxlabs.com: Tue Aug 25 03:17:53 CST 2015
tuxlabs.com: 2015-08-11 00:32:18
old.tuxlabs.com: 2015-06-06 02:30:42
➜  ~

Example of a simple regex & a failure

➜  ~  runner -r old.* -l
zsh: no matches found: old.*
➜  ~  

➜  ~  runner -r 'old.*' -l
old.tuxlabs.com

There was 1 host listed.
➜  ~

I hope you enjoyed the overview and new features. You can clone Runner on github.

Enjoy,
Jason Riedel

SSH Tunneling

In my last post about Runner I briefly explained needing to modify your ~/.ssh/config to use a ProxyCommand to allow for automatic tunneling with SSH.

Host tlbastion
User tuxninja
ForwardAgent yes
HostName tlbastion.tuxlabs.com
DynamicForward 8081

Host *.tuxlabs.com
User tuxninja
ProxyCommand /usr/local/bin/sconnect -4 -w 4 -S localhost:8081 %h %p

What I didn’t explain is there is an alternative method that is arguably simpler. It requires creating three small shells scripts & placing them in your path or a common host path like /usr/local/bin/ with the chmod +x permission. Here is the script that sets up the ssh tunnel.

Script: starttunnel

$ cat /usr/local/bin/starttunnel 
ssh -o ServerAliveInterval=300 -CfgNTL -D 8081 tlbastion.tuxlabs.com
$

Running starttunnel, will connect you to your bastion/jump box and then background this connection with keep alives on. It will listen / dynamically forward ssh requests to 8081 through or to tlbastion.tuxlabs.com. Additionally, if you wanted to tunnel a web port specifically on a machine that sits within your network back to the machine you are tunneling from, you can add it to the script. Such that the required host/port always gets tunneled and is available on your machine when you run starttunnel. Example config would look like.

Script: starttunnel + forwarding http

 

$ cat /usr/local/bin/starttunnel 
ssh -o ServerAliveInterval=300 -CfgNTL 8080:tuxlabs1.tuxlabs.com:80 -D 8081 tlbastion.tuxlabs.com
$

Now that you have authenticated to your bastion and have a working tunnel you need to get ssh requests to go through this tunnel. However, if your like me you still want the ability to ssh to other stuff without going through that tunnel. So I created a new script called ‘sshp’. When I want to ssh through the tunnel / proxy I use ‘sshp’, when I want to ssh to somewhere else on the internet or another network I use plain old ‘ssh’. Here is my sshp script used to connect to machines behind the bastion.

Script: sshp

$ cat /usr/local/bin/sshp 
#!/bin/sh

ssh -o ConnectTimeout=3 -o StrictHostKeyChecking=no -o CheckHostIP=no -o ServerAliveInterval=300 -o "ProxyCommand /bin/nc -X 5 -x localhost:8081 %h %p" $1

$

Now, when you run sshp tuxlabs1@tuxlabs.com you will be connection through the tuxlabs bastion into tuxlabs1. Also notice in my previous post I used sconnect as the proxy command in this one we are using ‘nc’ aka netcat. I have found this method of tunneling to be the most simplistic and effective in my daily life. One more script you need is if you want to copy files you need to use scp. So you have to make a similar command ‘scpp’ for tunneling your copying of files. Here’s the script.

Script: scpp

$ cat /usr/local/bin/scpp 
#!/bin/sh

scp -pr -o ConnectTimeout=3 -o StrictHostKeyChecking=no -o CheckHostIP=no -o "ProxyCommand /bin/nc -x localhost:8081 %h %p" $1 $2
$

One final note…if you need use ‘*’ aka splat for copying many files you cannot use the script above, because the shell or script converts that incorrectly. Instead just use the full command yourself from the command line.

scp’ing with *

$ scp -pr -o ConnectTimeout=3 -o StrictHostKeyChecking=no -o CheckHostIP=no -o "ProxyCommand /bin/nc -x localhost:8081 %h %p" copy.all.* tuxninja@tlbastion.tuxlabs.com:

This would copy all files named ‘copy.all.<whatever>’ to the  bastion. Hope this hopes the folks out there feeling limited by bastions. They provide great security and are an absolute requirement in secure environments so learning tricks that make sure you only need to authenticate once for an extended period of time can come in real handy.

Enjoy,
Jason Riedel

 

 

Runner: Multi-threaded SSH with Sudo support using Python & Paramiko

Example of Runner

$ runner -r web1 -c "whoami" -s
RUNNER [INFO]: MATCHING HOSTNAMES WITH 'web1'
RUNNER: 1 HOSTS HAVE BEEN SELECTED
RUNNER [INFO]: LOGFILE SET - logs/runner.log.2015-01-17.03:10:00
RUNNER [INFO]: USER SET - tuxninja
RUNNER [INFO]: SSH CONNECT TIMEOUT is: 5 seconds
RUNNER [INFO]: THREADS SET - 20
RUNNER [INFO]: SUDO IS ON
RUNNER [INPUT]: Please Enter Site Pass: 
web1.tuxlabs.com: 
web1.tuxlabs.com: root
web1.tuxlabs.com: [tuxninja@web1 ~]$ 

RUNNER [RESULT]: Successfully logged into 1/1 hosts and ran your commands in 0:00:08 second(s)
RUNNER [RESULT]: There were 0 login failures.

Why Runner ?

I have been working as a Systems & Network Administrator since 1999. In that time I have repeatedly had the need for rapidly executing commands across thousands of servers. There are many applications out there that solve this problem in various ways…to name a few…pdsh, Ansible, Salt, Chef, Puppet (mcollective),  even Cfengine and more. Some require agents running on the machines, some use SSH, but require keys…or learning curves. Alternatively, you can write your own code to solve this problem, which is what I did mostly for fun. I don’t recommend re-inventing the wheel if you need this for your job, just use what is already out there, or download runner and hack it to your hearts content for your purposes.

Fabric vs. Paramiko

Because I use Python for most of my work code these days, I decided to write my multi-threaded SSH command runner in Python this way I can use Runner for parallel SSH transport & easily bolt on my other Python scripts for additional functionality. Python has fantastic support for SSH via two libraries Fabric & Paramiko. Fabric is built on top of Paramiko. Fabric provides a simpler interface than Paramiko does for doing just about anything you can think of. Create a fabfile run it, and wolla instant results from commands ran via SSH. Fabric is really great for running & re-running a set of commands to automate an install or reporting for example. All that being said I still chose to use Paramiko over Fabric for three reasons.

  1. I don’t like abstraction. Fabric hides the ugly-ness of Paramiko, which I prefer to understand better.
  2. Writing this using Paramiko lent itself better to a command line utility used for adhoc commands than Fabric did.
  3. I wasn’t sure if Fabric’s abstraction would limit me later based on needing custom functionality. So for Runner I chose Paramiko, but to be clear, 9 times out of 10 I think I would choose Fabric.

Bastions

A bastion or jump box is a machine that is used as the gatekeeper of access to the rest of the machines in your network. In secure environments where your Corp network is separate from your Production network, you will have to SSH into a bastion, which usually has some form of 2-factor authentication (at least it should !) and then from there you may SSH into other hosts. A bastion can throw a real wrench in trying to manage thousands of machines in seconds, because you would have to authenticate to the bastion 1000 times ! The way around this, is by setting up your SSH config to proxy commands.

ProxyCommand & Sconnect

Sconnect (or connect.c) is a binary that is most commonly used as the proxy command for SSH. You can download / read more about sconnect here : https://bitbucket.org/gotoh/connect/wiki/Home and it will also tell you how to setup your SSH config. Using a ProxyCommand with Runner is required, you can however use any ProxyCommand you would like. Really quickly here is what you basically need to do.

  1. Download / Compile connect.c
  2. Copy it to /usr/local/bin/sconnect and set executable permissions
  3. In your SSH Config (.ssh/config) add…
    1. Host <ssh-config-profile-name>
      User tuxninja
      ForwardAgent yes
      HostName <bastion_name>
      DynamicForward 8081 (any uncommon port is fine)
    2. Host *.tuxlabs.com
      User tuxninja
      ProxyCommand /usr/local/bin/sconnect -4 -w 4 -S localhost:8081 %h.tuxlabs.com %p

That is basically it. Then you should start a screen session so you can background the SSH session, since you will leave this open for other SSH sessions to proxy through so you don’t have to go through 2-factor authentication more than once. So something like…

screen -S sshsession
ssh <ssh-config-profile-name>

After you authenticate, detach yourself from the screen using CTRL A then D. Now you can ssh to anything @ domain name in my case tuxlab.com and it will forward through the bastion. At this point you still have to authenticate using a username / password, which is fine. Runner deals with this.

Hosts

Runner requires a hosts file to run. By default it is configured to look in hosts/hosts-all for a list of all hosts. I use a script called ‘update-runner-hosts.pl’, which is included in my github to gather hosts from a URL and update the required hosts file. Once you have populated hosts/hosts-all with the FQDN for your hosts, you are ready to use Runner.

Note: You can use ‘-f’ to provide a custom location for your hosts file.

Great Flags / Features

So some of the really great features of Runner are threading (-t), sudo (-s), list only mode (-l) and the regular expression (-r). -r is for pattern matching your hosts lists, which is incredibly handy and absolutely required in an environment with hundreds to thousands of hosts and you only want to select hosts with -r ‘web’ in them.

(-1) one host per pool mode is a great feature, however it is dependent on understanding your environments hostname pattern so you will have to modify the regular expression in the code to make sure it works for you. It is currently setup to identify hostnames in pools when the naming convention is something like apache1234.tuxlabs.com.

Ok I could go on and on about runner, but it’s better to just share the code at this point and let you go! Note the statically defined proxy_command in the code, you may need to change this if you didn’t use sconnect or the same port.

Note: by default runner uses the user you are logged in as to SSH, you can prompt input for a different user with ‘-u’.

All code and accessories are available for download on github : https://github.com/jasonriedel/tuxlabs/tree/master/runner

Email tuxninja@tuxlabs.com with any question ! Happy SSH’ing admins!

Note: In various versions of this code I had a ‘-h’ allowing you to pass a CSV list of hosts, somehow I let that drop out of this version, sorry ! Feel free to re-add it !

The Runner Code

#!/usr/bin/env python
#Author: Jason Riedel

import paramiko
import getpass
import Queue
import threading
import argparse
import os.path
import time
import logging
import re
import datetime

## SETUP AVAILABLE ARGUMENTS ##
parser = argparse.ArgumentParser()
parser.add_argument('-f', action="store", dest="file_path", required=False, help="Specify your own path to a hosts file")
parser.add_argument('-l', action="store_true", dest="list_only", required=False, help="List all known hosts")
parser.add_argument('-q', action="store_true", dest="quiet_mode", required=False, help="Quiet mode: turns off RUNNER INFO messages.")
parser.add_argument('-qq', action="store_true", dest="super_quiet_mode", required=False, help="Super Quiet mode: turns off ALL RUNNER messages except [INPUT].")
parser.add_argument('-r', action="store", dest="host_match", required=False, help="Select Hosts matching supplied pattern")
parser.add_argument('-c', action="store", dest="command_string", required=False, help="Command to run")
parser.add_argument('-s', action="store_true", dest="sudo", required=False, help="Run command inside root shell using sudo") 
parser.add_argument('-t', action="store", dest="connect_timeout", required=False, help="ssh timeout to hosts in seconds")
parser.add_argument('-T', action="store", dest="threads", required=False, help="# of threads to run (don't get crazy)")
parser.add_argument('-u', action="store", dest="site_user", required=False, help="Specify a username (by default I use who you are logged in as)")
parser.add_argument('-1', action="store_true", dest="host_per_pool", required=False, help="One host per pool")
args = parser.parse_args()

##GLOBAL##
logging.getLogger('paramiko.transport').addHandler(logging.NullHandler())

stime = time.time()

## SET TIMEOUT ##
connect_timeout = 5
if args.connect_timeout:
    connect_timeout = args.connect_timeout

## SET THREADS / WORKERS ##
workers = 20
if args.threads:
    workers = int(args.threads)

## SET USER / PASS ##
site_user = getpass.getuser()
site_passwd = ''
if args.site_user:
    site_user = args.site_user

failed_logins = []
successful_logins = []

tstamp = datetime.datetime.now().strftime("%Y-%m-%d.%H:%M:%S")
logfile_dir = 'logs'
if not os.path.exists(logfile_dir):
    os.makedirs(logfile_dir)
logfile_path = '%s/runner.log.%s' % (logfile_dir, tstamp)
logfile = open(logfile_path, 'w')

## END GLOBAL ##

def ssh_to_host(hosts, site_passwd):
    for i in range(workers):
        t = threading.Thread(target=worker, args=(site_user, site_passwd))
        t.daemon = True
        t.start()

    for hostname in hosts:
        hostname = hostname.rstrip()
        q.put(hostname)

    q.join()

def worker(site_user, site_passwd):
    while True:
        hostname = q.get()
        node_shell(hostname, site_user, site_passwd)
        q.task_done()


def node_shell(hostname, site_user, site_passwd):
    ssh = paramiko.SSHClient()
    proxy_command = "sconnect -4 -w 4 -S localhost:8081 %s %s" % (hostname,'22')
    proxy_sock = paramiko.ProxyCommand(proxy_command)
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    try:
        ssh.connect(hostname, username=site_user, password=site_passwd, timeout=connect_timeout, sock=proxy_sock)
        transport = ssh.get_transport()
        transport.set_keepalive(1)

        cmd = args.command_string
	if args.sudo: 
		try: 
			## have to use invoke shell for sudo due to ssh config on machines requirng a TTY
			channel = ssh.invoke_shell() 
			sudocmd = 'sudo ' + cmd

			channel.send(sudocmd + '\n') 

			buff = ''
			while not '[sudo] password' in buff: 
				resp = channel.recv(9999)
				buff += resp

			channel.send(site_passwd + '\n') 

			buff = ''
			while not buff.endswith('$ '):
				resp = channel.recv(9999)
				buff += resp

			for line in buff.split('\n'):
				log_and_print("%s: %s" % (hostname, line))

		except Exception as e:
			log_and_print("ERROR: Sudo failed: %s" % (e))  
  
	else: 
        	(stdin, stdout, stderr) = ssh.exec_command(cmd)

		## stdout 
        	for line in stdout.readlines():
            		line = line.rstrip()
            		log_and_print("%s: %s" % (hostname, line))
		## stderr
        	for line in stderr.readlines():
            		line = line.rstrip()
            		log_and_print("%s: %s" % (hostname, line))

        successful_logins.append(hostname)
        ssh.close()

    except Exception as e:
        log_and_print("%s: failed to login : %s" % (hostname, e))
        failed_logins.append(hostname)
        ssh.close()

def log_and_print(message):
    if args.super_quiet_mode or args.list_only:
        if "RUNNER [INPUT]" in message or "RUNNER [ERROR]" in message or "RUNNER" not in message:
            print message
            logfile.write(message + '\n')
    elif args.quiet_mode or args.list_only:
        if "RUNNER [INFO]" not in message:
            print message
            logfile.write(message + '\n')
    else:
        print message
        if not args.list_only:
            logfile.write(message + '\n')

def get_hosts(file_path):
    if os.path.exists(file_path):
        hosts = open(file_path)
        selected_hosts = []
        if not args.host_match:
            selected_hosts = list(hosts)
            log_and_print("RUNNER [INFO]: SELECTING ALL HOSTS")
        else:
            host_match = args.host_match
            for host in hosts:
                if re.search(host_match, host):
                    selected_hosts.append(host)
            log_and_print("RUNNER [INFO]: MATCHING HOSTNAMES WITH '%s'" % (host_match))
    else:
        log_and_print("RUNNER [ERROR]: %s does not exist ! Try running ./update-runner-hosts" % (file_path))
        exit()

    ## Select one host per pool
    if args.host_per_pool:
        seen = {}
        host_per_pool = []
        for host in selected_hosts:
	    # Here strip values that make hostnames unique like #'s
	    # That way the dict matches after 1 host per pool has been seen 
            nhost = re.sub("\d+?\.", ".", host) #Removing #'s in a hostname like host1234.tuxlabs.com
            if not nhost in seen:
                seen[nhost] = 1
                host_per_pool.append(host)
        selected_hosts = host_per_pool

    log_and_print("RUNNER: %s HOSTS HAVE BEEN SELECTED" % (len(selected_hosts)))
    return selected_hosts

if __name__ == "__main__":
    file_path = 'hosts/hosts-all' ## update-hosts-all creates the DIR 

    if args.file_path:
        file_path = args.file_path
        if '~' in file_path:
            print "RUNNER [ERROR]: -f does not support '~'"
            exit()

    if args.list_only or args.command_string:
        selected_hosts = get_hosts(file_path)
        if args.list_only:
            for host in selected_hosts:
                host = host.rstrip()
                log_and_print(host)
            log_and_print("\nThere were %s hosts listed." % (len(selected_hosts)))
            exit()

        else:
            log_and_print("RUNNER [INFO]: LOGFILE SET - %s" % (logfile_path))
            log_and_print("RUNNER [INFO]: USER SET - %s" % (site_user))
            log_and_print("RUNNER [INFO]: SSH CONNECT TIMEOUT is: %s seconds" % (connect_timeout))
            log_and_print("RUNNER [INFO]: THREADS SET - %s" % (workers))
	    if args.sudo:
		log_and_print("RUNNER [INFO]: SUDO IS ON") 

            site_passwd = getpass.getpass("RUNNER [INPUT]: Please Enter Site Pass: ")

            q = Queue.Queue()

            ssh_to_host(selected_hosts,site_passwd)

            etime=time.time()
            run_time = int(etime-stime)

            timestamp = str(datetime.timedelta(seconds=run_time))
            log_and_print("\nRUNNER [RESULT]: Successfully logged into %s/%s hosts and ran your commands in %s second(s)" % (len(successful_logins), len(selected_hosts), timestamp))
            log_and_print("RUNNER [RESULT]: There were %s login failures.\n" % (len(failed_logins)))
            if len(failed_logins) > 0:
                for failed_host in failed_logins:
                    log_and_print("RUNNER [RESULT]: Failed to login to: %s" % (failed_host))
    else:
        parser.print_help()
        output = "\nRUNNER [INFO]: Either -l (list hosts only) or -s (Run cmd string) is required.\n"
        log_and_print(output)