Ansible and virtualenvs
What started out as a single post quickly became overly long and still didn’t touch on some of the more advanced topics, plus it was marked draft for way too long. Welcome instead to a series on effectively using Ansible and virtualenvs together. Much of this is relevant in a non-Ansible context, however, it is written from an Ansible perspective.
Why virtualenvs for Ansible Part 1(this article)
Ansible and Virtualenvs Part 2 - Creating and Using Introduction
Ansible and Virtualenvs Part 3 - Under the Covers
Ansible Tower and Virtualenvs Part 4 (future topic) In the meantime check out Ansiblejunky’s article.
Run ansible-playbook commands directly on Ansible Tower server
Pip and Virtualenv Cheatsheet
Python is famous for its batteries included approach, a reference to the powerful standard library which comes with any Python install. However typically in environments of any complexity, for example writing APIs or using AWS, the basic install needs to be supplemented by some of the vast selection of external libraries such as flask or boto.
These libraries can be installed at a system-wide level, available to everyone or at a user level which also allows users to install libraries without root privilege. Alternatively, you can use Python virtualenvs. Virtualenvs are a simple way to created isolated python run time environments. Enabling simple control over your python version, libraries, and python executables. Virtualenv creation and configuration are covered in more depth in Part 2 and Part 3.
Ansible leverages Python Libraries
Before we look at why use virtualenvs it’s worth spending a few paragraphs on why we need this isolation and control with Ansible at all. It is not uncommon for newcomers to Ansible to assume that because it offers extensive support and modules for AWS, Windows, Azure, networking equipment etc that it comes ready to communicate with them all "out of the box". In other words that you can write your first ansible playbook for AWS and type
ansible-playbook make-aws-cloud.yml and off it goes.
This will fail on an un-configured machine…
Rather than the Ansible engineers duplicate effort in providing their own python libraries to communicate with AWS and many other platforms (clouds, hypervisors, operating systems like Windows, Network Equipment like F5s) Ansible leverages existing python libraries which are not part of the Python Standard Libraries. e.g.
openstacksdk(used to use
shade) - OpenStack
winrmand also since Ansible
psrp- Microsoft Windows
Why use Virtualenvs - Infrastructure as Code (IaC)
I work in a global DevOps and Automation team at Red Hat where we actually mix both teaching Red Hat and Partner Consultants and Architects Ansible and OpenShift with also actually creating and running significant amounts of Cloud and OpenShift/Kubernetes infrastructure. Using Ansible we deploy hundreds of environments a day into production and have thousands of instances running concurrently. So we have direct hands on experience of running complex automation at scale.
There are plenty of good solid technical reasons for using
virtualenvs in such environments, and anywhere you have automated dev, test, and production needs. However the simplest and most compelling reason it is a foundational practice to allow you to practice Infrastructure as Code. Without IaC it is very hard to effectively move towards CI/CD at any scale.
Before we go any further I’m going to assume you are already practicing IaC, or moving towards, some form of Infrastructure as Code. ThoughtWorks have a nice introduction here, and not surprisingly their
example code snippet example is
Ansible:) (I always thought they where smart even though they once turned me down, very politely, for a job.) I recommend their writings and particularly the very readable and well thought out blog of their Chief Scientist Martin Fowler.
The first gain, and key step towards IaC, from virtualenvs, is they allow you to simply and consistently isolate dependencies. When I teach Red Hat’s OpenShift Advanced Development class early on, we examine and discuss The Twelve-Factor App.
Factor number 2 is "Isolate Dependencies", one of the key building blocks to get away from "works on my machine" syndrome. This allows others, whether individual’s or automated deployers to pull in and recreate a virtualenv that exactly matches "my machine". Or for me as a developer to exactly match dev or prod etc.
We’ll look more at the "How" in Part 2 but briefly
pip install -r requirements.txt are your friends in adopting this practice. Though you should, of course, automate capturing and deploying your virtualenvs. We’ll see in Part 2 how Ansible can automate this. However let’s take a quick look at my primary virtualenv "latest" (latest basically is my jumbo virtualenv which has the "latest and greatest" of everything I use frequently and my default virtualenv):
blog git:(master) ✗ workon latest (latest) ➜ public git:(master) ✗ pip freeze ansible==2.7.7 asn1crypto==0.24.0 bcrypt==3.1.6 boto==2.49.0 boto3==1.9.91 botocore==1.12.91 ... <snip> ... paramiko==2.4.2 passlib==1.7.1 pyasn1==0.4.5 pycparser==2.19 PyNaCl==1.3.0 python-dateutil==2.8.0 pywinrm==0.3.0 PyYAML==3.13 ... <snip> ...
Isolate and store the output of
pip freeze in each repo along with your pip
and python versions. One key component not captured above is which version of python I’m using, not surprisingly in an virtualenv called _latestL it is python 3! We’ll also look at setting your virtualenv python interpreter in Part 3.
Run Multiple Different Deployment Environments in Production
The vast bulk of our production deploys today occur on Ansible 2.6.4 but we are adding more and more environments, workshops, and labs from multiple different sources with differing requirements. One such example is the linklight workshop deployer from the Ansible team themselves. This forces Ansible 2.7 so to deploy from a production deployment host we wrap and deploy in an automatically deployed Ansible 2.7 based virtualenv.
Actually as of last week this is no longer true as we moved to Ansible
2.7.9, but you get the point
Lockdown known good configs
Whilst generally a good practice in Security circles to keep "everything" up to data occasionally we have use cases for working environments which we want to completely lock down to very specific versions.
For example for customer facing workshops, and in the more introductory level courses, we may want to create an environment that is extremely stable and that exactly matches the workshop notes and labs down to the exact version numbers etc.
Teaching (and Blogging!)
Whilst maybe not a common use case for many of you one particular use case I have frequently as I teach Ansible is making sure that I capture all the steps students need to go through to configure their environments. Playbooks which might just run on my machine will fail on theirs as they lack libraries I’ve forgotten I’d installed.
For example if we again take that repo linklight on a vanilla machine with no external python libraries the first time you try to deploy you may well fail on
boto not found. Once you’ve fixed that (
pip install boto boto3) then next time it may well fail on
passlib and so on.
The simple solution for me is to create and activate an empty virtualenv and run it from there isolating what external libraries I need. Then I can either provide a
requirements.txt or instructions to install the necessary external libraries.
Note: By default Ansible executing in a virtualenv will not see external libraries but will see external executables. So if you have
boto at a system wide level that will be invisible, but if you have
pip install aws-cli system-wide then you will see that. More on that in Parts 2 and 3
Finally - Python 2 is approaching EOL!
With the End of Life (EOL) of Python 2 rapidly approaching virtualenvs give a simple and effective way of jump starting your testing of Python 3. I’ll cover this in Part 2 but if you are in a hurry, just use the
--python flag to create a virtualenv that defaults to python3.
mkvirtualenv -p /usr/local/bin/python3 py3. If you already have python3 as your default interpreter then new virtualenvs will also default to python3.
Python’s virtualenv provides a simple yet powerful form of isolation for python applications and libraries. Used well it provides key building blocks allowing.
Infrastructure as Code
Isolation of Dependencies
Lockdown of known good environments