Make Ansible faster with Mitogen

What is Mitogen? 

From Mitogen’s docs

Mitogen is a Python library for writing distributed self-replicating programs.

[Mitogen] updates Ansible’s slow and wasteful shell-centric implementation with pure-Python equivalents, invoked via highly efficient remote procedure calls to persistent interpreters tunneled over SSH.

No changes are required to target hosts. The extension is considered stable and real-world use is encouraged.

Mitogen claims to give a 1.25x to 7x speedup along with various other good things like reduced CPU usage and fewer writes to remote file systems.

How to use and download it? 

Enabling Mitogen for Ansible is as simple as downloading and extracting the plugin, then adding 2 lines to the [defaults] section of your ansible.cfg:

# ansible.cfg

[defaults]
strategy_plugins = /path/to/mitogen-0.2.5/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

How I measured

The best way I’ve found to time the execution of Ansible playbooks is by enabling the profile_tasks callback. This callback is included with Ansible and all you need to do to enable it is add callback_whitelist = profile_tasks to the [defaults] section of your ansible.cfg:

# ansible.cfg

[defaults]
callback_whitelist = profile_tasks

And also a nice sorted list of tasks at the end of the run:

Thursday 21 February 2019  14:20:43 +0800 (0:00:00.065)       0:00:16.918 ***** 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 3.73s
geerlingguy.java : Ensure Java is installed. ---------------------------- 1.10s
geerlingguy.docker : Install Docker. ------------------------------------ 0.82s
geerlingguy.nginx : Ensure nginx is installed. -------------------------- 0.76s
...

You can see a bit more on the documentation. See the video below for practice information about it. 

Mitogen for Ansible demonstration from David Wilson on Vimeo.

Extra: 

If you really want to increase the speed of your roles you also should check the Ansible pipeline option.

What is Pipeline? 

From Ansible Documentation: 

Pipelining, if supported by the connection plugin, reduces the number of network operations required to execute a module on the remote server, by executing many Ansible modules without actual file transfer. It can result in a very significant performance improvement when enabled. However this conflicts with privilege escalation (become). For example, when using ‘sudo:’ operations you must first disable ‘requiretty’ in /etc/sudoers on all managed hosts, which is why it is disabled by default. This setting will be disabled if ANSIBLE_KEEP_REMOTE_FILES is enabled.

By default, Ansible executes tasks by copying a file onto the remote host, executing it, and reporting the results back to the control machine. As I understand, with pipelining enabled, Ansible can send commands directly to STDIN through a persistent SSH connection, which is much faster than the default process.

How to enable pipelining

You can enable pipelining by simply adding pipelining = True to the [ssh_connection] area of your ansible.cfg or by using the ANSIBLE_PIPELINING and ANSIBLE_SSH_PIPELINING environment variables.

# ansible.cfg
...
[ssh_connection]
pipelining = True

You’ll also need to make sure that requiretty is disabled in /etc/sudoers on the remote host, or become won’t work with pipelining enabled.

Further reading

• Mitogen for Ansible Docs
• Pipelining in Ansible Docs