Nowadays infrastructure tools like Puppet, Chef, Salt and Ansible are widely used. Part of sysadmin work consist to code with these tools. I use the word "code" because it is more than just configuring a tool.
Before being a sysadmin, I am a true developer, I code in python and I love to work in Test driven development. When I work on infrastructure code, it is not easy to test, and when there are no tests the development becomes longer, there are regressions and anger.
So we use Vagrant to start and provision a test VM. The fact that the deployment runs without errors is already a good point. Then we test manually that the service is running and the configuration files are rendered correctly. To automate these test I have used Serverspec, you can read a good introduction about Serverspec here. I like the concept of Serverspec, but was disappointed by the implementation with Rspec and limited by my ruby skills.
So I wrote a Serverspec equivalent in python: Testinfra
Testinfra is based on the Pytest test framework, I just wrote functions to call shell commands and inspect the state of local or remote systems. All the other features, parametrization, parallelization, hooks and reporting are provided by Pytest.
Here are some of the main features of testinfra:
- Testing local, remote through ssh, docker containers, salt minions systems.
- Testinfra has modules to easily test files, users, services, packages, puppet resources, salt modules
- Support for several OS like Debian, Ubuntu, Centos, Fedora, RedHat, FreeBSD, OpenBSD, NetBSD.
- Easily extensible. (Create new modules and package them).
- Integration with Jenkins, Vagrant, Docker and Nagios
Ok, but do I REALLY need infrastructure testing ?
When I present testinfra, I hear things like:
- Why use a tool to test the state of my server when my deployment tool already do and even fix it ?
- I already have monitoring to test that services are running well
The purpose of writing tests is to see the failures on the developer machine or the CI server instead of the production one. A test that never fails is useless.
For instance if your have a complex template, it can be useful to test the rendering. More generally a complex code that triggers implicit actions is a risky code.
Other interesting tests are those to check that the service you are deploying is running well. During the development phase, you ensure not to break the service, then you can run the same tests under the production for monitoring. Testinfra can be used like a nagios plugin:
$ testinfra -qq --nagios test.py; echo $? TESTINFRA CRITICAL - 1 passed, 1 failed, 0 skipped in 2.24 seconds [Traceback that explain the failed test] 2
Another usage of testinfra could be for audits. You can easily extend Pytest to generate custom reports with the format of your choice (html, json, pdf...). For instance we could generate a human readable report based on the docstring of failed tests.
Testinfra is still a new project. I would like to release the first version soon and for this I need your feedback to know what you are thinking about infrastructure testing and what feature you would like to see in testinfra.
Also if you have any questions I would be very pleased to answer you.
Testinfra is obviously a free software, released under the Apache 2 license.