How to implement CFEngine custom promise types in Python

This tutorial focuses on how to write a promise module, implementing a new CFEngine promise type. It assumes you already know how to install promise modules and use custom promise types, as shown in the previous blog post.

Why Python?

Promise modules can be written in any programming language, but there are some advantages of using python:

Readable and beginner friendly language / syntax
Popular and familiar to a lot of people, also used in some CFEngine package modules
Big standard library, allowing you to reuse data structures, parsers, etc. without reinventing the wheel or adding dependencies
Official CFEngine promise module library
- Most of the code needed is already done (protocol, parsing, etc.)
- You can focus on only the business logic, what is unique to your new promise type

With that said, there are some reasons why you might not always want to use python:

1 extra dependency (Must install python on all hosts which use module)
Relatively slow, in some situations (slow to start, slow for CPU-bound workloads)

For these reasons we’ve ensured the implementation of custom promise types is language-agnostic, and we will post more tutorials for different programming languages soon.

The git promise type

As an example, we’ll start writing a promise type to manage git repos. It won’t be complete in a short tutorial, but the aim is to show you how to get started implementing your own promise types.

Doing nothing, successfully

Before we do anything else, we should make a module which speaks the protocol and doesn’t cause any errors, even if it doesn’t do anything. We’ll start by importing the necessary parts of the library:

import os
from cfengine import PromiseModule, ValidationError, Result

The 3 parts we need from the library are:

PromiseModule - The class which implements the module, we need to subclass this
ValidationError - Exception to raise when there are problems with validating policy
Result - Enumeration of the different results a promise evaulation can have

Now, we simply make a subclass of PromiseModule, give our module a name and version number, and specify what should be done on validate and evaluate requests:

class GitPromiseTypeModule(PromiseModule):
    def __init__(self):
        super().__init__("git_promise_module", "0.0.1")

    def validate_promise(self, promiser, attributes, metadata):
        pass

    def evaluate_promise(self, promiser, attributes, metadata):
        return Result.KEPT

This module is now ready to use, it’s called git_promise_module, version 0.0.1, it performs no validation, and always returns promise result kept (success). validate_promise should raise an exception if validation fails, for example if the policy is using an attribute name which does not exist. If no exception is raised, the promise is considered valid and ready to be evaluated. pass is a no-op in python, it does nothing except indicate that this method is empty. The evaluate method should return one of 3 promise results:

Result.KEPT - No changes were necessary, everything was already as promised
- There should be a verbose log message which explains why nothing was done
Result.REPAIRED - Changes were made to the system, and they were successful, the system state is now as promised
- There should be an info log message which explains what was changed on the system
Result.NOT_KEPT - Changes were necessary, but they failed
- There should be an error log message explaining the failure

The last thing we need to do in the python code is to start the module:

if __name__ == "__main__":
    GitPromiseTypeModule().start()

The full module now looks like this:

import os
from cfengine import PromiseModule, ValidationError, Result


class GitPromiseTypeModule(PromiseModule):
    def __init__(self):
        super().__init__("git_promise_module", "0.0.1")

    def validate_promise(self, promiser, attributes, metadata):
        pass

    def evaluate_promise(self, promiser, attributes, metadata):
        return Result.KEPT


if __name__ == "__main__":
    GitPromiseTypeModule().start()

As explained previously, the recommended way to install the module is to put it in masterfiles/modules/promises:

$ cp *.py /var/cfengine/masterfiles/modules/promises/ && cf-agent -KIf update.cf && cf-agent -KI

Assuming that you have cfengine.py and git.py in the current working directory. Remember to run this command again after making changes to the module.

It is now ready to be used from policy:

promise agent git
{
  path => "/var/cfengine/modules/promises/git.py";
  interpreter => "/usr/bin/python3";
}

bundle agent __main__
{
  git:
    "/opt/cfengine/masterfiles"
      repository => "https://github.com/cfengine/masterfiles";
}

If we run this policy, nothing happens, successfully:

$ cf-agent -KI ./test.cf
$ echo $?
0

We can see some interesting details from the debug logs:

$ cf-agent -K --debug ./test.cf | grep git
   debug: Received header from promise module: 'git_promise_module 0.0.1 v1 json_based'
   debug: Received line from module: '{"operation": "validate_promise", "promiser": "/opt/cfengine/masterfiles", "attributes": {"repo": "https://github.com/cfengine/masterfiles"}, "result": "valid"}'
   debug: Received line from module: '{"operation": "evaluate_promise", "promiser": "/opt/cfengine/masterfiles", "attributes": {"repo": "https://github.com/cfengine/masterfiles"}, "result": "kept"}'
 verbose: Promise with promiser '/opt/cfengine/masterfiles' was kept by promise module '/var/cfengine/modules/promises/git.py'
 verbose: P: END git promise (/opt/cfengine/masterfiles)

Minimum Viable Promise type

In the methods we receive promiser (string) and attributes (dict) as parameters. Adding some validation is quite straight forward:

def validate_promise(self, promiser, attributes, metadata):
    if not promiser.startswith("/"):
        raise ValidationError(f"File path '{promiser}' must be absolute")
    if "repository" not in attributes:
        raise ValidationError(f"Attribute 'repository' is required")

We can also make our module do something, like running git clone:

def evaluate_promise(self, promiser, attributes, metadata):
    url = attributes["repository"]
    folder = promiser

    if os.path.exists(folder):
        self.log_verbose(f"'{folder}' already exists, nothing to do")
        return Result.KEPT

    self.log_info(f"Cloning '{url}' -> '{folder}'...")
    os.system(f"git clone {url} {folder} 2>/dev/null")

    if os.path.exists(folder):
        self.log_info(f"Successfully cloned '{url}' -> '{folder}'")
        return Result.REPAIRED
    else:
        self.log_error(f"Failed to clone '{url}' -> '{folder}'")
        return Result.NOT_KEPT

Here we see that the promise evaluation has 3 possible outcomes, along with appropriate log messages:

Kept - The folder already existed, and nothing was done, this is explained in verbose output
Repaired - The git clone command was run, this is shown in info output, since it changes the system
Not kept - The git clone command was run, but it failed, this deserves an error message

This is a very simplified implementation. It assumes that if the folder exists, it is a correct clone, but we can improve it later. Here is the full module, again:

import os
from cfengine import PromiseModule, ValidationError, Result


class GitPromiseTypeModule(PromiseModule):
    def __init__(self):
        super().__init__("git_promise_module", "0.0.1")

    def validate_promise(self, promiser, attributes, metadata):
        if not promiser.startswith("/"):
            raise ValidationError(f"File path '{promiser}' must be absolute")
        if "repository" not in attributes:
            raise ValidationError(f"Attribute 'repository' is required")

    def evaluate_promise(self, promiser, attributes, metadata):
        url = attributes["repository"]
        folder = promiser

        if os.path.exists(folder):
            self.log_verbose(f"'{folder}' already exists, nothing to do")
            return Result.KEPT

        self.log_info(f"Cloning '{url}' -> '{folder}'...")
        os.system(f"git clone {url} {folder} 2>/dev/null")

        if os.path.exists(folder):
            self.log_info(f"Successfully cloned '{url}' -> '{folder}'")
            return Result.REPAIRED
        else:
            self.log_error(f"Failed to clone '{url}' -> '{folder}'")
            return Result.NOT_KEPT


if __name__ == "__main__":
    GitPromiseTypeModule().start()

Adding more useful functionality

There are some obvious areas of improvement:

We want to be able to specify a branch other than the default
We want the module to be able to git pull new commits
We want the module to clone recursively (submodules)

Adding these are left as an exercise for the reader.

Additional resources

If you’d like to learn more, here are a few useful resources: