How to implement CFEngine custom promise types in bash

This blog post will focus on the bash programming part of implementing a promise type. To understand what custom promise types are, and how to use them, you should read the introduction first.

To implement a custom promise type in CFEngine, you need a promise module. The module is an executable, and can be written in any language. It’s possible to write one from scratch, but to make it as easy as possible, we decided to provide libraries for common programming languages. In our previous blog post, we showed how to implement modules in Python. That’s great, it’s a powerful, expressive and readable programming language, however there is one drawback; installing python. Many systems don’t have python already, or have a version which is too old. So you will need to add policy to install / update Python, to make sure modules work correctly everywhere.

CFEngine runs on a wide variety of platforms; Linux, BSD, Windows, Mac, HP-UX, IBM AIX, Solaris, etc. Most of these platforms have some version of bash already installed (except for Windows). Thus, by writing promise modules in bash we can make them very portable, and easy to use everywhere without additional dependencies.

Using the CFEngine promise module bash library

The library implements the protocol and the main loop of the program, receiving and answering requests from cf-agent. The following examples will be based on it, but authors of shell-based custom promise modules are welcome to look through the source code and use it as they find fitting. To use the library, the module script should set environment variables and define functions, then source the library script and call module_main function with two arguments: The module name and version.

The most trivial example, which does nothing, looks like this:

do_evaluate() {
    response_result="kept"
}

. "$(dirname "$0")/cfengine.sh"
module_main "trivial_example_module" "1.0"

The function do_evaluate is called during policy evaluation. It must set response_result variable, otherwise policy evaluation will fail. After sourcing the library, module_main is called with module name and version as two arguments.

Note: Generally, you cannot add echo or printf statements like you would in a normal shell script, to print error messages or log what the module is doing. Your module code should be silent (not print to stdout). This is because the module is communicating with cf-agent on standard output, so its output needs to follow the protocol.

Logging

To send log messages to the agent (which will in the end be printed to the user), follow one of these examples:

log error "Could not create file 'blah'"
log info "Created file 'blah'"
log verbose "File 'blah' already exists, skipping"
log debug "Module 'create_file_module' received request to validate 'blah'"

Logging should only be added inside do_evaluate and do_validate functions. What to log and what the different levels mean is explained in the specification. If you need additional logging, which will not be printed to CFEngine users, write to a file instead:

echo "Example promise module starting" > example_module.log

Variables

Currently, there are three optional variables, which instruct the library to check validity of promise attributes.

required_attributes - space-separated list of attributes which must be presented in each promise. If provided - each promise is checked for presence of these attributes, and fails validation if at least one of them is not present. If not provided, no such check is performed.

optional_attributes - space-separated list of attributes which are allowed, but not required, in each promise. Example: optional_attributes="this that".

all_attributes_are_valid - yes or no (default: no). By default, any attribute which is not in list of required or optional attributes is considered invalid and renders promise invalid. If set to yes, then no extra check for optional attributes is performed - all attributes are considered valid.

Functions

Currently, there is only one required function - which performs evaluation (the step which makes changes to the system), and three optional ones: to validate and to do setup / teardown if needed.

do_evaluate - required, the function which does promise evaluation. As inputs it can use request*promiser variable and a set of request_attribute*\*variables - one for each specified attribute. As outputs, it must setresponse_resultvariable to one ofkept, not_kept, or repaired strings.

do_validate - optional, called to validate promise. Uses same inputs as do_evaluate, as output it can change response_result to valid or invalid. Note that the library itself checks presence of required_attributes and absence of invalid attributes (if requested), and sets response_result accordingly before calling this function. This function can change value of this variable. Also, you can check, for example, that request_promiser is a valid absolute pathname (that it starts with /).

do_initialize - called once at the very beginning of module lifecycle, before any other function. Can be used for initialization / set up, if needed.

do_terminate - called once at the very end of module lifecycle, after any other function. Can be used for cleaning up, if needed.

Copy promises

Let’s have a look at a less trivial example - copying files:

required_attributes="from"
optional_attributes=""
all_attributes_are_valid="no"

do_evaluate() {
    if diff -q "$request_attribute_from" "$request_promiser" 2>/dev/null; then
        response_result="kept"
        return 0
    fi
    if cp -a "$request_attribute_from" "$request_promiser"; then
        response_result="repaired"
    else
        response_result="not_kept"
    fi
}

. "$(dirname "$0")/cfengine.sh"
module_main "cp" "1.0"

This declares cp custom promise type, which has one required attribute: from, and no optional attributes. In do_evaluate, value of this attribute is available in request_attribute_from variable.

Also note that instead of overwriting the target file every time, this promise module first compares source and destination, and does the copying only if they differ. Promise result reflects that: the promise will be “kept” if source and destination were the same before policy evaluation, “repaired” if they were different, but cp command succeeded, and “not_kept” if cp command failed.

In the policy it can be used like this:

promise agent cp
{
    interpreter => "/bin/bash";
    path => "$(sys.inputdir)/cp.sh";
}
bundle agent main
{
    cp:
    "/tmp/dst"
        from => "/tmp/src";
}

When evaluating above policy, do_evaluate will be called with variables like this:

request_attribute_from="/tmp/src"
request_promiser="/tmp/dst"

Backup promises

As a final example, I’ve added my pet project, namely backup3, as a CFEngine custom promise module:

required_attributes="from"
optional_attributes="root"
all_attributes_are_valid="no"

do_evaluate() {
    # Prepare environment for wrapped script
    BACKUP_ROOT="$request_attribute_root"
    BACKUP_BIN="$(dirname "$0")/backup3"
    BACKUP_KEEP_TMP="1"
    run_this() {
        run_rsync always "$request_promiser" "$request_attribute_from"
    }

    # call the script
    . "$BACKUP_BIN/backup.sh"

    # Analyse its results
    if [ "$?" != 0 -o ! -f "$BACKUP_TMP".files ]; then
        response_result="not_kept"
        rm -f "$BACKUP_TMP".sql "$BACKUP_TMP".files
        return 0
    fi
    # Note: $BACKUP_TMP".files file contains lists of created and deleted files,
    # one file per line, separated by line saying "separator", and first line
    # being literally "first line". So if script succeeded, but there were no
    # changed files, then this file has only 2 lines.
    local filelist_lines="$(cat "$BACKUP_TMP".files | wc -l)"
    if [ "$filelist_lines" -gt 2 ]; then
        response_result="repaired"
    else
        response_result="kept"
    fi

    # Clean up
    rm -f "$BACKUP_TMP".sql "$BACKUP_TMP".files
}

. "$(dirname "$0")/cfengine.sh"
module_main "backup" "1.0"

From first three lines we see that promises for this module must always have a “from” attribute, and, optionally, a “root” attribute (if not provided, it’s expected that the backup3/backup.sh script called in do_evaluate will figure it out).

In the do_evaluate function we see a different order of operations compared to the “copy promise”: instead of checking if something needs to be done first, promise module always first calls the backup3/backup.sh script, and only then checks if it did anything. That’s due to how the backup3 is written: it always first runs rsync, and then checks if it did anything. The promise module script analyzes one of its temporary files and concludes if there were any changes or not, based on its contents.

Summary

That’s it. Whether you’re using bash, python, or another programming language, extending CFEngine policy language has never been easier. If you encounter any issues or have any questions when working with promise modules, we encourage you to post them here. We are excited to see what modules the community creates, and to share more than just example modules soon.

If you want to learn more, here are some useful resources: