Ansible and Virtualenvs Part 1 - Why

Python is famous for its batteries included approach, a reference to the powerful standard library which comes with any Python install. However typically in environments of any complexity, for example writing APIs or using AWS, the basic install needs to be supplemented by some of the vast selection of external libraries such as flask or boto.

These libraries can be installed at a system-wide level, available to everyone or at a user level which also allows users to install libraries without root privilege. Alternatively, you can use Python virtualenvs. Virtualenvs are a simple way to created isolated python run time environments. Enabling simple control over your python version, libraries, and python executables. Virtualenv creation and configuration are covered in more depth in Part 2 and Part 3.

Before we look at why use virtualenvs it’s worth spending a few paragraphs on why we need this isolation and control with Ansible at all. It is not uncommon for newcomers to Ansible to assume that because it offers extensive support and modules for AWS, Windows, Azure, networking equipment etc that it comes ready to communicate with them all "out of the box". In other words that you can write your first ansible playbook for AWS and type ansible-playbook make-aws-cloud.yml and off it goes.

This will fail on an un-configured machine…​

Rather than the Ansible engineers duplicate effort in providing their own python libraries to communicate with AWS and many other platforms (clouds, hypervisors, operating systems like Windows, Network Equipment like F5s) Ansible leverages existing python libraries which are not part of the Python Standard Libraries. e.g.

The first gain, and key step towards IaC, from virtualenvs, is they allow you to simply and consistently isolate dependencies. When I teach Red Hat’s OpenShift Advanced Development class early on, we examine and discuss The Twelve-Factor App.

Factor number 2 is "Isolate Dependencies", one of the key building blocks to get away from "works on my machine" syndrome. This allows others, whether individual’s or automated deployers to pull in and recreate a virtualenv that exactly matches "my machine". Or for me as a developer to exactly match dev or prod etc.

We’ll look more at the "How" in Part 2 but briefly pip freeze, requirements.txt, and pip install -r requirements.txt are your friends in adopting this practice. Though you should, of course, automate capturing and deploying your virtualenvs. We’ll see in Part 2 how Ansible can automate this. However let’s take a quick look at my primary virtualenv "latest" (latest basically is my jumbo virtualenv which has the "latest and greatest" of everything I use frequently and my default virtualenv):

Isolate and store the output of pip freeze in each repo along with your pip and python versions. One key component not captured above is which version of python I’m using, not surprisingly in an virtualenv called _latestL it is python 3! We’ll also look at setting your virtualenv python interpreter in Part 3.

The vast bulk of our production deploys today occur on Ansible 2.6.4 but we are adding more and more environments, workshops, and labs from multiple different sources with differing requirements. One such example is the linklight workshop deployer from the Ansible team themselves. This forces Ansible 2.7 so to deploy from a production deployment host we wrap and deploy in an automatically deployed Ansible 2.7 based virtualenv.

Actually as of last week this is no longer true as we moved to Ansible 2.7.9, but you get the point

Whilst generally a good practice in Security circles to keep "everything" up to data occasionally we have use cases for working environments which we want to completely lock down to very specific versions.

For example for customer facing workshops, and in the more introductory level courses, we may want to create an environment that is extremely stable and that exactly matches the workshop notes and labs down to the exact version numbers etc.

Whilst maybe not a common use case for many of you one particular use case I have frequently as I teach Ansible is making sure that I capture all the steps students need to go through to configure their environments. Playbooks which might just run on my machine will fail on theirs as they lack libraries I’ve forgotten I’d installed.

For example if we again take that repo linklight on a vanilla machine with no external python libraries the first time you try to deploy you may well fail on boto not found. Once you’ve fixed that (pip install boto boto3) then next time it may well fail on passlib and so on.

The simple solution for me is to create and activate an empty virtualenv and run it from there isolating what external libraries I need. Then I can either provide a requirements.txt or instructions to install the necessary external libraries.

Note: By default Ansible executing in a virtualenv will not see external libraries but will see external executables. So if you have boto at a system wide level that will be invisible, but if you have pip install aws-cli system-wide then you will see that. More on that in Parts 2 and 3

With the End of Life (EOL) of Python 2 rapidly approaching virtualenvs give a simple and effective way of jump starting your testing of Python 3. I’ll cover this in Part 2 but if you are in a hurry, just use the -p or --python flag to create a virtualenv that defaults to python3. mkvirtualenv -p /usr/local/bin/python3 py3. If you already have python3 as your default interpreter then new virtualenvs will also default to python3.