Ansible Performance

May 18, 2019 Ansible, Python

Setting up Virtualenvs

Welcome to the Second part in this series on Ansible and virtualenvs, to read Part 1 see Why and When. The below should work on both Linux and macOS though you’ll want pip installed. (I use brew on the Mac, and EPEL far more than I should on RHEL/CentOS/Fedora, I rarely use Debian/Ubuntu but if python and pip are installed the below should work.)

TL:DR

If you just want to jump in and just get started follow the steps immediately below. However if you want to read more about the use cases and a little of how virtualenvs work, read the full article below.

This is opinionated, for example many people don’t use virtualenvwrapper, but using virtualenvs daily for testing and production and this is what I use and I find that it streamlines my workflow efficiently, effectively.

Install virtualenv and virtualenvwrapper
```
$ pip install virtualenv virtualenvwrapper
```
Hopefully we are all be on Python 3 by now that is the case you can omit virtualenv as it is integrated as venv in Python 3.3 and later. In fact virtualenvs are a great way to test your playbooks etc on Python 3.
Setup some virtualenvwrapper environmental variables in your shell profile of choice (.profile, .bashrc, .zshrc etc)
```
export WORKON_HOME=$HOME/.virtualenvs           (1)
source /usr/local/bin/virtualenvwrapper.sh      (2)
```
(1) Where your virtualenvs will be created

(2) Load your virtualenvwrapper functions etc

Tip
On Centos, RHEL etc virtualenvwrapper.sh will probably be /bin/virtualenvwrapper.sh

Note
In Part 3 we’ll explore other configuration options and how to configure more richly. The above is the minimum to get up and running..
Source whatever file you put those variables above ( ~/.bashrc, ~/.zshrc, etc)
```
$ source ~/.bashrc   # In my case source ~/.zshrc, adapt for your toolchain
```
Make your first virtualenv
```
mkvirtualenv ansible-aws
```
It automatically becomes active, note your new prompt. Install a specfic version of ansible
```
( ansible-aws ) $ pip install ansible==2.7.10
```
"Deactivate it", your shell will still be looking at the "old" ansible
```
( ansible-2.6.1 ) $ deactivate
$
```

Activate it (workon, from virtualenvwrapper supports tab completion)

$  workon ansible-2.6.1
( ansible-2.6.1 ) $ echo $PATH
/Users/tok/Dropbox/virtual-envs/ansible-2.6.1/bin:/usr/local/bin:/bin:/usr/sbin
$ (ansible-2.6.1) $ virtual-envs ansible --version
ansible 2.6.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/tok/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/tok/Dropbox/virtual-envs/ansible-2.6.1/lib/python2.7/site-packages/ansible
  executable location = /Users/tok/Dropbox/virtual-envs/ansible-2.6.1/bin/ansible
  python version = 2.7.10 (default, Aug 17 2018, 17:41:52) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]

You should see from your path that ansible etc is now from inside your new “virtualenv”. A lot of the "magic" of virtualenvs is simply PATH manipulation. From now on when you execute pip it will install libraries within the virtualenv.

Introduction

There are many, genuine, technical reasons driving the need to run multiple different Ansible versions and supporting libraries. Python’s virtualenvs allow you to simply switch between different versions of both Ansible and underlying python Ansible modules.

It becomes simple to have Ansible 2.6.3, 2.7.0, 2.7.latest, etc., all installed. Or have test virtualenvs with ansible 2.next and say Python 3 as your interpreter. It allows you to create multiple different sandboxes with particular version of python libraries. Eg. Ansible 2.6.2 with boto x.x.x and another with Ansible 2.7.1 with boto x.x.y

As Ansible continues to evolve at a quick pace and add new features and modules the traditional software problem of latest and greatest versus known and stable raises its head.

Works on their Machine…

In our team we provide extensive Ansible playbooks and roles that run a large production environment which deploys and configures hundreds of lab, test, and demo environments onto cloud providers every week. At peak times over 2,000 instances can be spun up, and torn down, daily. Ansible is used to create and configure everything from Highly Available OpenShift Clusters, OpenStack, and JBoss environments to much simpler stand alone systems. So stability and reliability is key for these complex multi-stage builds.

Note	If you want to see, and better still contribute to AgnosticD, our core Cloud Agnostic Deployer Platform please take a look.

Recently we moved from Ansible 2.4.3 to 2.6.4 but for config development and local cloud deployments from a laptop I tend to work with not only whatever is latest and greatest in Ansible - currently 2.7.1 but also on Python 3.

In addition I’m lucky enough to get teach Ansible Advanced Bootcamps globally and frequently need to be on the latest and greatest (and beyond) yet be able to switch to older versions on demand. Also it is incredibly useful to be able to switch to a clean environment and follow through the same steps as a student.

Python Virtualenvs to the Rescue

Fortunately with Ansible being Python based this particular problem has a clean and elegant solution in the form of Python virtualenvs. In a nutshell a Python virtualenv creates isolated sandboxes for python applications and libraries where you have your own version of python, pip, libraries, etc., and of course your applications such as ansible.

Once “activated” your current shell/terminal session will resolve dependencies first from within the virtualenv. So for example if you don’t have ansible-lint in your virtualenv but do have it somewhere else e.g. /usr/local/bin it will find it after first looking within the virtualenv.

Effectively what happens is a sandbox directory structure is created per virtualenv with your interpreter, utilities (e.g. pip), and libraries. Once activated any new libraries etc are installed into this structure.

Being driven via Environmental variables coupled with its it is entirely feasible to have multiple terminal/shell sessions concurrently running and by default virtualenvs show your current working environnement in your prompt.

How virtualenvs work in short

A note on Python 3

A word about python3. Python 3.x (check this) comes with virtualenv support in the form of the venv module

Quick peep inside a virtualenv

Output of tree command Output of env command. Diff 2 env cmd before and after.

Python 3

Creating your first virtual I wa initially intimidated when I

alias v-an2.4.3='source ~/Dropbox/virtualenvs/v-an2.4.3/bin/activate' alias v-an2.4='source ~/Dropbox/virtualenvs/v-an2.4/bin/activate' alias v-an2.5='source ~/Dropbox/virtualenvs/v-an2.5/bin/activate' alias v-an2.6='source ~/google/courses/ansible/ansible/repo/ansible/hacking/env-setup'

Note: I actually put these on Dropbox and share them across multiple laptops but this is a use case I’ve not made much use of.

Ah but Ansible hardcodes the executable path?

Note:

Quick Workflow Cd into virtualenv home Create virtualenv Install ansible version of choice

Applications v. Libraries

Whilst if your virtualenv doesn’t have a library e.g. boto for AWS it will complain if it needs it, despite it existing outside the virtualenv and in other virtualenvs. Not so for applications. If you have not installed ansible-lint in your virtualenv but have it present elsewhere in your path it will execute if called.

This has pros and cons, for example you don’t need to -re-install every utility you use in every virtualenv. However be aware you executing python applications outside the virtualenv.

If you have installed a application at the OS level, e.g. /usr/local/bin and another version in your virtualenv you can of course use the full PATH:

(ansible-2.6.1) ➜  ~ ansible-lint --version
ansible-lint 3.5.1
(ansible-2.6.1) ➜  ~ /usr/local/bin/ansible-lint --version
ansible-lint 3.4.21

Visibility

The default behavior of a virtualenv is to hide all libra

So if you have a system with the

However it does not hide system wide executables, i.e. if you have a /usr/local/bin/ansible and it’s in your PATH then

==- pip v. yum

For Production use on supported Ansible Control Nodes

After installing an application e.g. ansible-lint deactivate and reactivate (workon) the virtualenv. Otherwise you may find your

Resources:

Offcial Documetation https://virtualenv.pypa.io/en/stable/

https://docs.python-guide.org/dev/virtualenvs/

https://virtualenvwrapper.readthedocs.io/en/latest/install.html

docs:

Cheatsheet of commands

Futures: One of the new features in Ansible Tower 3.3 is the ability to execute Jobs in virtualenvs - look out for an article on that topic in the future.

Acknowledgments and credits

http://mkelsey.com/2013/04/30/how-i-setup-virtualenv-and-virtualenvwrapper-on-my-mac/