May 11, 2012
We, at Digital Science, are pleased to announce that we open sourced one of the key pieces of our technical infrastructure, baton, which is a general purpose tool to query and run commands on our servers. It provides a basic framework that is meant to be extended in order to fit specific needs such as deployment tasks, monitoring tasks, etc.
This is the first step Digital Science takes towards contributing to the open source community. We have always been trying new technologies and therefore we rely on many open source projects. Thus, contributing to the open source community is a natural step for our company and we very much hope this will be the first of many other projects that we will share.
We strongly believe that automating daily tasks can dramatically increase productivity. One of our goals was having one click deployment of code to development/production servers or quick service discovery. After trying existing tools, we found nothing that was good enough.
That’s when we decided to build baton.
Since then, we created a number of baton extentions that have been powering code deployments, service discoveries, handle alert messages, etc.
The initial configuration will setup an input exchange and an output exchange.
On the input exchange, baton will wait for meaningful messages to perform actions (described by each service) and it will output messages to the output exchange.
For example, one could send a message to discover services in production. Baton would receive that message on the input exchange, and every service that responds to that message would write an output message to the output exchange.
This is the entry point for input messages. One should extend the API class and add meaningful methods that ultimately use `publish` to publish messages to the input exchange. One example can be found here.
Any baton extension should have an executable script that will start the extension service. Here is an example.
The Service is the starting point of any baton extension. The idea of the service is to setup consumers for the input messages arriving from the API. By implementing `setup_consumers` one will allow the consumers to receive messages. Here is an example.
This class is an orchestration class that attaches observers to the consumers (like logger, etc), binds the input queues to the correct exchanges, dispatches the received messages to the consumers and updates the observers on changes. One doesn’t need to extend this class unless one wants to change its behaviour.
This class must be extended in order to process each received message. One should implement `process_message` at least, in order to give meaning to each received message. One can also override `routing_key` in order to listen to specific messages. Here is an example of an implementation.
Like the consumer manager, this class doesn’t need to be extended. It provides functionality to setup the exchanges and add consumers.
The observer class provides methods to notify observers. It is by default included in the consumers so that the output exchange (and possibly loggers, etc) receive the output messages.
baton-deploy, an extension
One of the most important extensions created so far was baton-deploy, which is a an extension that allows one to deploy code to our servers.
We have installed baton-deploy on each server (development environments included). Each baton-deploy deamon will read a configuration file which will provide information concerning which applications are expected to be running on that server.
Baton-deploy then creates queues with specific routing keys (e.g. such as application_name.current_environment). When the services start listening on those queues for incoming messages, we then use a web interface to interact with baton’s api. One could then deploy a given application to all production servers, all development servers, or ultimately target specific servers.
You can easily extend baton to perform your own tasks. baton-ping provides what we consider to be a minimal extension to baton. One should note that there is an extra class on `baton-ping` called monitor. This is a good example of what to do with output messages from a baton extension. Together with baton-ping-monitor, it provides a standard way of consuming output messages and do something relevant with them.
If you would like to create your own extension, simply install baton (`gem install baton`) and run the following command:
batonize gem GEMNAME -b
This will create a basic gem structure with the necessary files to create a minimum viable baton extension.