TuxLabs LLC

All things DevOps

Category: Python

Object-Oriented Programming With Python : Encapsulation (1/3)

Published / by tuxninja / Leave a Comment

Justification For Learning OOP

linux-python-logo1Since 1995, I have written some form of code, but over the years my career has never taken shape with a primary focus on development. Instead in my various roles of Systems, Network, Application and Datacenter engineering & architecture, I have found excuses to let the inner code hacker in me, out. Allowing me to consistently pursue my passion for coding and automation within the scope of my daily duties. However, this approach has never allowed me to work on a development team and thus the necessity for using object oriented programming rarely has occurred.

Sure I’ve written classes and modules for re-use, but I have never had to place a tremendous amount of forethought into the design for flexibility or inflexibility of the interfaces those classes provide because I was the only one using them. This hasn’t change, but as of late, I find myself realizing my career may be ready for a change. It is likely the next step for me in my career is a full blown software development role. This is partly because of boredom and mastery in my existing field after decades of experience, but mostly because of my own career planning and the apparent industry trends.

Simply put more jobs are moving to software development, and Systems Engineering is on a steady decline due to the advent of Cloud computing, Containers and more. Due to the amazing products, platforms and tools that are now available less and less Systems Engineers are required and I would say Systems Engineers that don’t code are already obsolete, whether they realize it yet or not.

In todays world if you are or are going to be a developer, knowing how to write object oriented code is a requirement and more importantly thinking ahead about who and how your code will be used becomes paramount when designing effective software. This article is intended to help you get a better understanding of how to write object oriented programming using Python.

Procedural Programming vs. Object Oriented Programming

Procedural programming  is a more antiquated paradigm for software design that can be described as:

  • Code that processes data
  • The data we process is stored in variables
  • Then we create functions to process the data in those variables

In the object oriented programming paradigm:

  • We organize data into Objects
  • Functions become Methods
  • And The design of these objects & methods is specified in a Class, which is essentially a blue print.

Object oriented programming is the primary software design paradigm in existence today because it provides programmers with a way to effectively organize and share code for re-use while to protecting the integrity of the existing code. To understand how this works we need to jump right into it with The Three Pillars of Object Oriented Programming.

The Three Pillars (Encapsulation, Inheritance, and Polymorphism)

Encapsulation

Encapsulation is about ensuring the safe storage of data as attributes in an instance.

Encapsulation tells us that :

  • Data should only be accessed through instance methods.
  • Data should always be correct based on the validation requirement set in the class methods.
  • And Data should be safe from changes by external processes.

Great so what does that mean 🙂 It means we should be using ‘setter’ and ‘getter’ methods to access object attribute values. Here’s an example of a class that uses a setter and getter method (note the name setter & getter is not actually required) to set and get the variable/attribute value.

The output of this code is: 

It’s important to recognize that encapsulation is not enforced by Python. So a programmer using a class is not required to access the data through the getter and setter methods. For example, in the class above someone could set website, without ever interacting with the defined setter and getter classes. This is called breaking encapsulation, and it’s bad form / practice for a programmer to this because the class author & maintainer can no longer validate the data that is being accessed and this can cause unforeseen problems with a program that is dependent on the class it is inheriting from. Here’s a modified example where we are breaking encapsulation. 

Here is the output of the above example: 

A number of things have changed.

  1. We added a variable website to make the example more clear and easy to follow.
  2. We added validation to our setter() method, it now checks to make sure ‘http’ is contained in the value that we are setting for website.
  3. If ‘http’ is not in website the setter() method will throw an error and will not set website!
  4. We added some additional formatting to our getter() method so we can tell when it is being used verses when encapsulation is being broken. getter() now pre-pends any website passed with ‘Website is now set to:’

So let’s walk thru how these changes effected the program and what actually happened. First, we tried to set website to ‘http://www.tuxlabs.com’ and then we changed it to ‘http://www.google.com’. In both of these examples we followed the rules of encapsulation, setting and getting the attribute value through methods in the class. These examples are correct and set website to the value passed and outputed them with the correct formatting we implemented in getter().

Next, we tried setting website to a really long string, without using setter() ! So we did tuxlabs.website = website directly on the instance of our class ! We also circumvented using getter() and just printed in exactly the same, broken way print tuxlabs.website. Unfortunately, as previously mentioned nothing requires the programmer to use our setter and getter methods directly, and thus website is not being checked for containing ‘http’ here and so it can printed our very long string (“I should not be accessing website in the class directly!, but since I am breaking encapsulation it does still work, it\’s just naughty!”).

The last and final example, is an example of encapsulations value proposition. We try to set website to something that does not contain ‘http’ and therefore is not a website. An error is thrown and website is not set, which is exactly what we want.

Encapsulation is  simple, easy to follow and powerful when your programming on or with a team of developers that are going to share code amongst each other. In these scenarios encapsulation is obviously a must promoting collaboration through protecting programmers from potentially mis-using each others code. It should be easy to see now that breaking encapsulation is bad and very poor practice because code that perfectly functions one day, could be broken the next without any changes from you, because you are violating the contract that encapsulation methods of the class provide.

This concludes our learning about encapsulation. Still to come, Inheritance, Polymorphism, and some real world examples!

Until next time…Keep learning,
Jason Riedel

Upgrading to Python 2.7 on CentOS 6.5

Published / by tuxninja / 1 Comment on Upgrading to Python 2.7 on CentOS 6.5

Hey Folks,

The systems running Tuxlabs are currently running CentOS 6.5 to emulate a production RHEL like setup for an Openstack Cloud. Running an operating system this old has it’s drawbacks such as dependencies. I was recently installing a well know Python framework and ran into compatibility issues. The framework required Python 2.7 and CentOS 6.5 comes with 2.6. The below is a step by step procedure for how to upgrade to Python 2.7 on CentOS 6.5 if  you ever should need it. However, as a reminder run a newer OS when possible and for god sakes if you don’t need Redhat support, run Ubuntu.

Step one, we verify we are indeed running Python 2.6

Ok then, let’s upgrade Python to 2.7. First let’s update all of our system applications, just in case for version dependencies and it’s good for security etc.

Next, we have to install Develop Tools, it is a required dependency to install Python.

Additionally, we will need these…

Now, let’s install Python 2.7

It is important to use ‘altinstall’ otherwise you will end up with two different versions of Python on your filesystem, both named ‘python’.

You can verify the install like so

That’s it ! Enjoy.

Runner: Multi-threaded SSH with Sudo support using Python & Paramiko

Published / by tuxninja / Leave a Comment

Example of Runner

 

Why Runner ?

I have been working as a Systems & Network Administrator since 1999. In that time I have repeatedly had the need for rapidly executing commands across thousands of servers. There are many applications out there that solve this problem in various ways…to name a few…pdsh, Ansible, Salt, Chef, Puppet (mcollective),  even Cfengine and more. Some require agents running on the machines, some use SSH, but require keys…or learning curves. Alternatively, you can write your own code to solve this problem, which is what I did mostly for fun. I don’t recommend re-inventing the wheel if you need this for your job, just use what is already out there, or download runner and hack it to your hearts content for your purposes.

Fabric vs. Paramiko

Because I use Python for most of my work code these days, I decided to write my multi-threaded SSH command runner in Python this way I can use Runner for parallel SSH transport & easily bolt on my other Python scripts for additional functionality. Python has fantastic support for SSH via two libraries Fabric & Paramiko. Fabric is built on top of Paramiko. Fabric provides a simpler interface than Paramiko does for doing just about anything you can think of. Create a fabfile run it, and wolla instant results from commands ran via SSH. Fabric is really great for running & re-running a set of commands to automate an install or reporting for example. All that being said I still chose to use Paramiko over Fabric for three reasons.

  1. I don’t like abstraction. Fabric hides the ugly-ness of Paramiko, which I prefer to understand better.
  2. Writing this using Paramiko lent itself better to a command line utility used for adhoc commands than Fabric did.
  3. I wasn’t sure if Fabric’s abstraction would limit me later based on needing custom functionality. So for Runner I chose Paramiko, but to be clear, 9 times out of 10 I think I would choose Fabric.

Bastions

A bastion or jump box is a machine that is used as the gatekeeper of access to the rest of the machines in your network. In secure environments where your Corp network is separate from your Production network, you will have to SSH into a bastion, which usually has some form of 2-factor authentication (at least it should !) and then from there you may SSH into other hosts. A bastion can throw a real wrench in trying to manage thousands of machines in seconds, because you would have to authenticate to the bastion 1000 times ! The way around this, is by setting up your SSH config to proxy commands.

ProxyCommand & Sconnect

Sconnect (or connect.c) is a binary that is most commonly used as the proxy command for SSH. You can download / read more about sconnect here : https://bitbucket.org/gotoh/connect/wiki/Home and it will also tell you how to setup your SSH config. Using a ProxyCommand with Runner is required, you can however use any ProxyCommand you would like. Really quickly here is what you basically need to do.

  1. Download / Compile connect.c
  2. Copy it to /usr/local/bin/sconnect and set executable permissions
  3. In your SSH Config (.ssh/config) add…
    1. Host <ssh-config-profile-name>
      User tuxninja
      ForwardAgent yes
      HostName <bastion_name>
      DynamicForward 8081 (any uncommon port is fine)
    2. Host *.tuxlabs.com
      User tuxninja
      ProxyCommand /usr/local/bin/sconnect -4 -w 4 -S localhost:8081 %h.tuxlabs.com %p

That is basically it. Then you should start a screen session so you can background the SSH session, since you will leave this open for other SSH sessions to proxy through so you don’t have to go through 2-factor authentication more than once. So something like…

After you authenticate, detach yourself from the screen using CTRL A then D. Now you can ssh to anything @ domain name in my case tuxlab.com and it will forward through the bastion. At this point you still have to authenticate using a username / password, which is fine. Runner deals with this.

Hosts

Runner requires a hosts file to run. By default it is configured to look in hosts/hosts-all for a list of all hosts. I use a script called ‘update-runner-hosts.pl’, which is included in my github to gather hosts from a URL and update the required hosts file. Once you have populated hosts/hosts-all with the FQDN for your hosts, you are ready to use Runner.

Note: You can use ‘-f’ to provide a custom location for your hosts file.

Great Flags / Features

So some of the really great features of Runner are threading (-t), sudo (-s), list only mode (-l) and the regular expression (-r). -r is for pattern matching your hosts lists, which is incredibly handy and absolutely required in an environment with hundreds to thousands of hosts and you only want to select hosts with -r ‘web’ in them.

(-1) one host per pool mode is a great feature, however it is dependent on understanding your environments hostname pattern so you will have to modify the regular expression in the code to make sure it works for you. It is currently setup to identify hostnames in pools when the naming convention is something like apache1234.tuxlabs.com.

Ok I could go on and on about runner, but it’s better to just share the code at this point and let you go! Note the statically defined proxy_command in the code, you may need to change this if you didn’t use sconnect or the same port.

Note: by default runner uses the user you are logged in as to SSH, you can prompt input for a different user with ‘-u’.

All code and accessories are available for download on github : https://github.com/tuxninja/tuxlabs-code/tree/master/runner

Email tuxninja@tuxlabs.com with any question ! Happy SSH’ing admins!

Note: In various versions of this code I had a ‘-h’ allowing you to pass a CSV list of hosts, somehow I let that drop out of this version, sorry ! Feel free to re-add it !

The Runner Code

 

 

 

 

Python & Fun with Checkio.org

Published / by tuxninja / Leave a Comment

My Brief Coding History

13 years ago, I fell in love with Perl. I liked Perl a lot better than my previous flings with HTML, Shell (Bash), PHP, ASP, Javascript, Sed, Awk and a few others, mainly because Perl was always willing to do what I wanted, when I wanted, making my life as a Sys Admin a lot easier and never asked much in return just to be loved. I loved Perl for years and years and in a way I will never stop loving Perl, after all Perl was my first love. But too often our first loves die hard no matter how hard we try to will their existence forward…

“I never thought I could love a language more than Perl, until I met Python”  (TuxNinja, 2012)

Enter Python

Old loves die hard.. I haven’t used Perl for about ~18 months, whatever the exact amount of time is I am not sure, but it matches the exact time at which I started learning Python.  I decided to start learning Python because it’s popularity seemed to be growing exponentially (and still does), once I learned basic syntax by reading and watching a couple of videos, I was up and running and it wasn’t long before I was solving real business critical problems with it.

[stextbox id=”grey” caption=”Best Way To Learn A New Language”]I have learned a lot of languages. By far the best way to learn a new language is to solve a real problem (preferably at your job) with it. The reason being is that inevitably when you solve the problem you will need to maintain the code (i.e. add features and fix bugs). This ‘recall’ you experience every time you re-visit and re-factor your code will burn into your brain valuable learning’s better than any amount of writing and/or reading alone. Suffices to say that troubleshooting is the best way to learn.[/stextbox]

Two Reasons

There were two major reasons once I started learning Python I decided it was my new #1.

  1. Python is the most readable language I have ever seen. It reads like pseudo code or plain english. If that isn’t important to you, go try to read someones competition winning obfuscated code ./facepalm. For the 13 years I coded Perl every time I took over a Perl script from someone else I re-wrote it. That is because Perl allows for coders to follow many different styles such as a C or Shell style of coding. It’s flexibility is it’s greatest enemy in my opinion, there are 1000 ways to do something in Perl. In Python there are 10’s of ways, but there is usually only 1 pythonic way. I prefer a language that has a clear best way to do something and that encourages readability. Of course there are still some assholes out there who prefer lambda() and map() over list comprehension and to them I say… read a book (err website)
  2. Community support. I forever loved Perl. Two of the reasons was the incredible support I found on www.perlmonks.com and because of CPAN and the plethra of modules that existed. Well Python wins again. It’s seems most former monks now spend their time correcting peoples Python on StackOverflow.com and PyPi > CPAN, sorry but it’s true !

All that being said, I encourage you to learn Python, Ruby, and Node.js… these 3 seem to be the front runners of that the next generation is learning and therefore will continue to be abound now and in the future. But since this is a Python article and the language I know best, learn Python first !

CheckIO

Finally, the purpose of this article! About 3 months into learning Python a good friend of mine who is a programmer for a living told me about CheckIO.org, which is a website, that provides a fun & free way to learn Python in the form of a game that reminds me of Wii’s Super Mario. Now when I initially signed up ~15 months ago I never actually ended up playing the game. Until a couple of days ago when I got an email from them asking how I liked CheckIO. Well that was all the reminder I needed, looking to continue in life long learning and specifically my Python knowledge I started playing the game. It’s been 48 hours and I am pretty hooked ! The benefits to learning using CheckIO are unmatched by any other learning mechanisms I have used to date here’s why.

How It Works

CheckIO starts you at your ‘home’ and presents you with challenges that it gives you points and badges for completing. Once you complete enough you move on to the next challenge and eventually unlock other adventures and your next set of challenges. Yea, yea ok that’s cool so what…well when you solve your challenge, you have the option of publishing your code and reviewing / voting on the best solution of all time for that problem. This is an amazing thing. It combines not only exercising to solve a problem yourself, but ultimately showing you the most clean, efficient and pythonic way (caveat: not always but mostly the social voting keeps this true). This is an incredible about of knowledge in one place, and the challenges you are solving are universally applicable to some of the most common problems with data structures, sorting and counting that you will run into. Admittedly, I am a level 4 currently and only about 60% of the way through and have already unlocked other adventures, but I was having so much fun and learning so much that I decided I’m going to complete 100% of an adventure before moving on.

CheckIO Teaches You To Be An Efficient Programmer

Feel free to read that again. Yes it does. My whole programming life I have been focused on function…function, function, function. Making my code functional was always #1 because after all you are solving a problem and making your code functional has to be the right first step anyway, you can’t work on making an application efficient if you haven’t figured out a way to solve the problem programmaticly. So the only time I went back and re-wrote an program to make it more efficient was when it was heavily used so incredibly inefficient that it demanded to be FIXED!

Let There Be Light

What CheckIO does is encourages you to compete with other programmers solving the same challenges in the game format, keeping it fun. So finally 6 challenges in, I started to get it… instead of just solving the challenge, why not try to solve the challenge in the most efficient & cleanest way possible and then publish my code for voting with the community. I did just that and the difference in my code was a difference of 47 lines.  The example I am showing below is a basic one, but the story is what’s crucial. If you solve the CheckIO problems and challenge yourself to write the best possible solution each time and then compare against your peers you will ultimately be a far better programmer then most people. Not to mention you will have a nice repository for re-usable code for common problems you will encounter as a programmer.

Ok enough…it’s show and tell time.

The Problem

Full Problem Description: http://www.checkio.org/mission/roman-numerals/
Short Version: For this task, you should return a roman numeral using the specified integer value ranging from 1 to 3999.

Input: An integer ranging from 1 to 3999.
Output: A string in the form of a Roman numeral.
Example:

Functional Solution

My Think Twice Solution

It’s amazing how quickly you can improve code once you understand how it needs to function. This improvement took me a total of 2 minutes to finish.

Published Solution

The above solutions were what I used to test and run before submitting, when you actually submit it has to be as a CheckIO function. The real code submitted can be found below here.