CFEngine and Docker – Ensure application availability and container integrity

Posted by:

22 Nov 2013

eystein.maloy.stenberg@cfengine.com

Product Management, CFEngine

Introduction

The beauty and power of Docker comes from its ability to containerize any application into a portable self-sufficient entity, one that can run anywhere.  This enables developers to deliver an application with all dependencies in a layered image structure.   As if that was not enough, the images can be shared in an image repository (known as the Docker index) and deployed to and run on any Docker host.

CFEngine, a pioneer in IT automation, enables organizations to become more agile by radically simplifying, automating and transforming the way they build, deliver and consume IT infrastructure and applications. With CFEngine, some of the largest IT organizations provision resources and deploy new applications orders of magnitude faster, while ensuring continuous availability, security and compliance in large-scale, very dynamic and highly complex environments.

After applications are deployed, maintenance tasks such as process management and signaling, log management, and managing run-time configuration consistency and compliance become important.

CFEngine is designed to be very lightweight, distributed and fault-tolerant.  These attributes make it very well suited for managing containerized Docker applications.  CFEngine is an extremely versatile technology and a good complement to Docker, which is very strong at getting code containerized, shipped and deployed.

 

Advancing Process Management in Docker

Docker monitors one process in each running container and the container lives or dies with that process.  By introducing CFEngine inside Docker containers, we can alleviate a few of the issues that may arise:

  • It is possible to easily start multiple processes within a container, all of which will be managed automatically.
  • If a managed process dies or crashes, CFEngine will start it again within 1 minute.
  • The container itself will live as long as the CFEngine scheduling daemon (cf-execd) lives. With CFEngine, we are able to decouple the life of the container from the uptime of the service it provides.

 

How it works

CFEngine, together with the cfe-docker integration policies, are installed as part of the Dockerfile. This essentially builds CFEngine into our Docker image.  The advantage is that the Dockerfile’s ‘ENTRYPOINT’ takes an arbitrary amount of commands (with any desired arguments) as parameters.  When we run the Docker container, the parameters get written to CFEngine policies and CFEngine takes over to ensure that the desired processes are running in the container.

CFEngine scans the process table for the basename of the commands given to the ENTRYPOINT and runs the command to start the process if the basename is not found.
For example, if one starts the container with

docker run "/path/to/my/application parameters"

CFEngine will look for a process named “application” and run the command.  Later, if an entry for ‘application’ is not found in the process table, CFEngine will execute “/path/to/my/application parameters” to re-start the application once again.  The check on the process table happens every minute.

 

Test it out!

To demonstrate the benefits of using CFEngine to manage processes in Docker containers, we will create a simple example that:

  • Starts both apache2 and sshd in a single container
  • Ensures that both these processes are continuously running

To start off, we construct a Dockerfile that includes CFEngine, the cfe-docker policies, as well as our apache2 and sshd packages (also available at  https://github.com/estenberg/cfe-docker#usage):

 

FROM ubuntu
MAINTAINER Eystein Måløy Stenberg <eytein.stenberg@gmail.com>
RUN apt-get -y install wget lsb-release unzip
# install latest CFEngine
RUN wget -qO- http://cfengine.com/pub/gpg.key | apt-key add -
RUN echo "deb http://cfengine.com/pub/apt $(lsb_release -cs) main" > /etc/apt/sources.list.d/cfengine-community.list
RUN apt-get update
RUN apt-get install cfengine-community
# install cfe-docker process management policy 
RUN wget --no-check-certificate https://github.com/estenberg/cfe-docker/archive/master.zip -P /tmp/ && unzip /tmp/master.zip -d /tmp/
RUN cp /tmp/cfe-docker-master/cfengine/bin/* /var/cfengine/bin/
RUN cp /tmp/cfe-docker-master/cfengine/inputs/* /var/cfengine/inputs/
RUN rm -rf /tmp/cfe-docker-master /tmp/master.zip

# apache2 and openssh is just for testing purposes 
RUN apt-get -y install openssh-server apache2
RUN mkdir -p /var/run/sshd
RUN echo "root:password" | chpasswd # need a password for ssh

ENTRYPOINT ["/var/cfengine/bin/docker_processes_run.sh"]

 

Build the container:

docker build -t managed_image .

Start the container with apache2 and sshd running and managed, forwarding a port to our SSH instance:

docker run -p 127.0.0.1:222:22 -d managed_image "/usr/sbin/sshd" "/etc/init.d/apache2 start"

We can now log in to our new container and see that both apache2 and sshd are running. We have set the root password to “password” in the Dockerfile above and can use that to log in with ssh:

ssh -p222 root@127.0.0.1    # ps -ef  UID        PID  PPID  C STIME TTY          TIME CMD  root         1     0  0 07:48 ?        00:00:00 /bin/bash /var/cfengine/bin/docker_processes_run.sh /usr/sbin/sshd /etc/init.d/apache2 start  root        18     1  0 07:48 ?        00:00:00 /var/cfengine/bin/cf-execd -F  root        20     1  0 07:48 ?        00:00:00 /usr/sbin/sshd  root        32     1  0 07:48 ?        00:00:00 /usr/sbin/apache2 -k start  www-data    34    32  0 07:48 ?        00:00:00 /usr/sbin/apache2 -k start  www-data    35    32  0 07:48 ?        00:00:00 /usr/sbin/apache2 -k start  www-data    36    32  0 07:48 ?        00:00:00 /usr/sbin/apache2 -k start  root        93    20  0 07:48 ?        00:00:00 sshd: root@pts/0   root       105    93  0 07:48 pts/0    00:00:00 -bash  root       112   105  0 07:49 pts/0    00:00:00 ps -ef

If we stop apache2, it will be started again within a minute by CFEngine:

# service apache2 status  Apache2 is running (pid 32).  # service apache2 stop   * Stopping web server apache2                                                                                                                         /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted)  apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName   ... waiting                                                                                                                                    [ OK ]  # service apache2 status  Apache2 is NOT running.  # ... wait up to 1 minute...  # service apache2 status  Apache2 is running (pid 173).


Adapting to your applications

To make sure your applications get managed in the same manner, there are just two things you need to adjust from the above example:

  • In the Dockerfile used above, install your applications instead of apache2 and sshd
  • When you start the container with docker run, specify the command line arguments to your applications rather than apache2 and sshd

That’s it!

 

Future extensibility

Since CFEngine is a very mature management tool, it can be leveraged to solve additional Docker use cases.  For example:

  • Secrets such as passwords should not be built into the Docker images, but can be discovered during run-time by CFEngine
  • Service dependencies such as database addresses can be discovered and configured in run-time by CFEngine

We hope this post is useful and you can start experimenting with your own applications, CFEngine and Docker.  We would love to hear back from you.  Please let us know if you have feedback or ideas for areas where you think CFEngine could make your containerized life easier!  For now ‘Dock On’ and rest assured CFEngine is right there helping you achieve your application goals!

 

Learn more about CFEngine by visiting our website

Evaluate CFEngine for free (upto 25 licenses, unlimited usage)

Eystein Maloy Stenberg