How can I test CFEngine policy?

This question was covered in The agent is in, Episode 27 - CFEngine Q&A: Policy questions.

Testing is an important part of the software life-cycle. Writing tests for your CFEngine policy can help to bring improved assurance that your policy behaves as expected. Follow along and write your first test policy.

Test stages

When writing tests there are three or four basic stages that typically need to be handled.

Initialization - Set up the necessary conditions for the test, e.g. create some files to be edited.
Testing - Running the policy whose behavior you wish to test.
Checking - Inspecting the results of the test policy to see if they conform with expectations.
Cleanup - You might need to cleanup artifacts produced by the test if your testing system does not handle it for you.

These stages map well to a sequence of bundles. So, a simple test template could look like this:

bundle agent __main__
{
  methods:
    "init";
    "test";
    "check";
    "cleanup";
}
bundle agent init
{}
bundle agent test
{}
bundle agent check
{}
bundle agent cleanup
{}

Let’s take this template and start filling it in. First, we need some test case. Let’s do something simple, for example, making sure the reports promise type attribute report_to_file works as expected.

report_to_file simply emits a report to a file instead of to standard output. To initilize the test we might want to make sure the file we report to does not exist, then we would test reporting to the file. After reporting to the file we would inspect the file to make sure the content within matches our expectations before finally cleaning up that test file.

Let’s get going with bundle agent init.

bundle agent init

During test initialization let’s make sure the test file does not exist when we exercise report_to_file in the test bundle.

bundle agent init
# @brief Initialize the environment to prepare for the test
{
  vars:
    "testfile"
      string => "/tmp/test-reports-report_to_file.txt";

  files:
    "$(testfile)"
      delete => my_tidy,
      if => fileexists( $(testfile) );
}
body delete my_tidy
# @brief Duplicated from the stdlib
{
  dirlinks => "delete";
  rmdirs   => "true";
}

bundle agent test

In our test bundle we exercise report_to_file emitting our report to our test file.

bundle agent test
# @brief Test that report_to_file causes reports promises to be directed into a file.
{
  reports:
    "Hello World!"
      report_to_file => "$(init.testfile)";
}

bundle agent check

Next we need to check that the file does indeed contain the expected content and emit something that let us know if it passed or failed.

bundle agent check
# @brief Check that reports sent to a file are found as expected
{
  classes:
    "testfile_exists"
      expression => fileexists( "$(init.testfile)" );

  reports:
    testfile_exists::
      "$(with)"
        with => ifelse(
          regcmp( "Hello World!", readfile( "$(init.testfile)" ) ),
          "Pass",
          "FAIL"
        );

    !testfile_exists::
      "FAIL";
}

bundle agent cleanup

Let’s cleanup that test artifact. It’s perhaps duplicating efforts from init, but it’s best to not leave unwanted artifacts behind if there is no test runner processing the artifacts from each test.

bundle agent cleanup
# @brief Clean up after checking test results
{
  files:
    "$(init.testfile)"
      delete => my_tidy,
      if => fileexists( "$(init.testfile)" );
}

bundle agent main

Finally, let’s add bundle agent __main__ to drive this bundle sequence if the test is the policy entry.

bundle agent __main__
# @brief Clean up after checking test results
{
  methods:
    "init";
    "test";
    "check";
    "cleanup";
}

Full basic example test policy

All together in /tmp/example-test.cf we now have this policy.

bundle agent init
# @brief Initialize the environment to prepare for the test
{
  vars:
    "testfile"
      string => "/tmp/test-reports-report_to_file.txt";

  files:
    "$(testfile)"
      delete => my_tidy,
      if => fileexists( $(testfile) );
}
body delete my_tidy
# @brief Duplicated from the stdlib
{
  dirlinks => "delete";
  rmdirs   => "true";
}
bundle agent test
# @brief Test that report_to_file causes reports promises to be directed into a file.
{
  reports:
    "Hello World!"
      report_to_file => "$(init.testfile)";
}
bundle agent check
# @brief Check that reports sent to a file are found as expected
{
  classes:
    "testfile_exists"
      expression => fileexists( "$(init.testfile)" );

  reports:
    testfile_exists::
      "$(with)"
        with => ifelse(
          regcmp( "Hello World!", readfile( "$(init.testfile)" ) ),
          "Pass",
          "FAIL"
        );

    !testfile_exists::
      "FAIL";
}
bundle agent cleanup
# @brief Clean up after checking test results
{
  files:
    "$(init.testfile)"
      delete => my_tidy,
      if => fileexists( "$(init.testfile)" );
}
bundle agent __main__
# @brief Clean up after checking test results
{
  methods:
    "init";
    "test";
    "check";
    "cleanup";
}

Running the example test policy

Let’s run it and inspect the output:

command
cf-agent --no-lock --file /tmp/example-test.cf

output
R: Pass

Let’s synthesize a failure by blocking writes to the test file. chattr +i sets a file to be immutable. A file with the i attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file’s metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

command
touch /tmp/test-reports-report_to_file.txt
chattr +i /tmp/test-reports-report_to_file.txt
lsattr /tmp/test-reports-report_to_file.txt

output
----i---------e------- /tmp/test-reports-report_to_file.txt

command
cf-agent --no-lock --file /tmp/example-test.cf

output
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt'
   error: Method 'init' failed in some repairs
   error: Could not open log file '/tmp/test-reports-report_to_file.txt', message 'Hello World!'. (fopen: Operation not permitted)
R: FAIL
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt'
   error: Method 'cleanup' failed in some repairs

Great! Now we have a functioning test. Let’s clear our synthesized failure before proceeding (again, you will need elevated privileges).

command
chattr -i /tmp/test-reports-report_to_file.txt
lsattr /tmp/test-reports-report_to_file.txt
rm /tmp/test-reports-report_to_file.txt

output
--------------e------- /tmp/test-reports-report_to_file.txt

A next logical step would be to get this test result interpreted by something. The Masterfiles Policy Framework ships with a testing library (lib/testing.cf) to facilitate emitting output compatible for Test Anything Protocol (TAP) or JUnit. Let’s extend the existing test to output in JUnit.

Adding JUnit output support from lib/testing.cf

First, we need to include the library as part of our inputs.

body file control
# @brief Include necessary files from the standard library
{
  inputs => {
    "$(sys.libdir)/testing.cf",
    "$(sys.libdir)/reports.cf", # For printfile => default:cat()
  };
}

Refactoring bundle agent check

Next refactor bundle agent check to emit in the JUnit format to a file and print it out. Normally your testing tools (Jenkins, GitHub, etc …) would read that XML in to give you some nice visualization of the result.

bundle agent check
# @brief Check that reports sent to a file are found as expected
{
  classes:
    "testfile_exists"
      expression => fileexists( "$(init.testfile)" );

    testfile_exists::
      "expected_string_found_in_testfile"
        scope => "namespace",
        if => regcmp( "Hello World!",
                      readfile( "$(init.testfile)" ) );

  methods:
    "Register test case"
      usebundle => testing_ok_if(
        "expected_string_found_in_testfile",
        "Test that the expected string is found in file created by printfile => report_to_file",
        "error: printfile => report_to_file failed to create a file with the expected string",
        "trace",
        "jUnit"
      );
    "jUnit Report"
       usebundle => testing_junit_report( "/tmp/test-result.xml" );

  reports:
    inform_mode::
     "The full xml report:"
       printfile => default:cat( "/tmp/test-result.xml" ),
       if => fileexists( "/tmp/test-result.xml" );

      !expected_string_found_in_testfile::
        "$(init.testfile) did not have expected content. Contains:"
          printfile => default:cat( "$(init.testfile)" ),
          if => fileexists( "/tmp/test-result.xml" );

}

That gives us this full policy file:

body file control
# @brief Include necessary files from the standard library
{
  inputs => {
    "$(sys.libdir)/testing.cf",
    "$(sys.libdir)/reports.cf", # For printfile => default:cat()
  };
}
bundle agent init
# @brief Initialize the environment to prepare for the test
{
  vars:
    "testfile"
      string => "/tmp/test-reports-report_to_file.txt";

  files:
    "$(testfile)"
      delete => my_tidy,
      if => fileexists( $(testfile) );
}
body delete my_tidy
# @brief Duplicated from the stdlib
{
  dirlinks => "delete";
  rmdirs   => "true";
}
bundle agent test
# @brief Test that report_to_file causes reports promises to be directed into a file.
{
  reports:
    "Hello World!"
      report_to_file => "$(init.testfile)";
}
bundle agent check
# @brief Check that reports sent to a file are found as expected
{
  classes:
    "testfile_exists"
      expression => fileexists( "$(init.testfile)" );

    testfile_exists::
      "expected_string_found_in_testfile"
        scope => "namespace",
        if => regcmp(
          "Hello World!",
          readfile( "$(init.testfile)" ) );

  methods:
    "Register test case"
      usebundle => testing_ok_if(
        "expected_string_found_in_testfile",
        "Test that the expected string is found in file created by printfile => report_to_file",
        "error: printfile => report_to_file failed to create a file with the expected string",
        "trace",
        "jUnit");
    "jUnit Report"
       usebundle => testing_junit_report( "/tmp/test-result.xml" );

  reports:
    inform_mode::
     "The full xml report:"
       printfile => default:cat( "/tmp/test-result.xml" ),
       if => fileexists( "/tmp/test-result.xml" );

    !expected_string_found_in_testfile::
      "$(init.testfile) did not have expected content. Contains:"
        printfile => default:cat( "$(init.testfile)" ),
        if => fileexists( "/tmp/test-result.xml" );

}
bundle agent cleanup
# @brief Clean up after checking test results
{
  files:
    "$(init.testfile)"
      delete => my_tidy,
      if => fileexists( "$(init.testfile)" );
}
bundle agent __main__
# @brief Clean up after checking test results
{
  methods:
    "init";
    "test";
    "check";
    "cleanup";
}

Running the refactored example test

Let’s run it to see it pass.

command
cf-agent -KIf /tmp/example-test-with-junit.cf

output
R: testing_ok_if: adding testing report for class expected_string_found_in_testfile at position 1
    info: Updated rendering of '/tmp/test-result.xml' from mustache template '/home/nickanderson/.cfagent/inputs/lib/templates/junit.mustache'
R: testing_generic_report: report summary: counts = 1/1/0/0/0 failed = {  }, passed = { "testing_expected_string_found_in_testfile" }, skipped = {  }, todo = {  }, failed = {  }; tests = [{"tap_message":"ok Test that the expected string is found in file created by printfile => report_to_file","test_message":"Test that the expected string is found in file created by printfile => report_to_file","test_offset":1,"testcase":"expected_string_found_in_testfile"}]+[]+[]+[]
R: The full xml report:
R: <?xml version="1.0" encoding="UTF-8"?>
R: <testsuite tests="1" failures="0" timestamp="2023-07-27T14:52:13">
R:
R:   <testcase name="expected_string_found_in_testfile">Test that the expected string is found in file created by printfile =&gt; report_to_file</testcase>
R:
R:
R:
R:
R: </testsuite>
R:
R: <!-- not implemented (yet):
R: 1) errors: <error message="my error message">my crash report</error>
R: 2) STDOUT: <system-out>my STDOUT dump</system-out>
R: 3) STDERR: <system-err>my STDERR dump</system-err>
R:  -->
    info: Deleted file '/tmp/test-reports-report_to_file.txt'

Now, let’s synthesize failure again (you will need elevated privileges to use chattr).

command
touch /tmp/test-reports-report_to_file.txt
chattr +i /tmp/test-reports-report_to_file.txt
lsattr /tmp/test-reports-report_to_file.txt

output
----i---------e------- /tmp/test-reports-report_to_file.txt

Now, let’s run the policy to see what failure with JUnit output looks like.

command
cf-agent -KIf /tmp/example-test-with-junit.cf

output
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt'
   error: Method 'init' failed in some repairs
   error: Could not open log file '/tmp/test-reports-report_to_file.txt', message 'Hello World!'. (fopen: Operation not permitted)
R: testing_ok_if: adding testing report for class expected_string_found_in_testfile at position 1
    info: Updated rendering of '/tmp/test-result.xml' from mustache template '/home/nickanderson/.cfagent/inputs/lib/templates/junit.mustache'
R: testing_generic_report: report summary: counts = 1/0/0/0/1 failed = { "testing_expected_string_found_in_testfile_failed" }, passed = {  }, skipped = {  }, todo = {  }, failed = { "testing_expected_string_found_in_testfile_failed" }; tests = []+[]+[]+[{"fail_message":"error: printfile => report_to_file failed to create a file with the expected string","tap_message":"not ok Test that the expected string is found in file created by printfile => report_to_file","test_message":"Test that the expected string is found in file created by printfile => report_to_file","test_offset":1,"testcase":"expected_string_found_in_testfile_failed","trace_message":"trace"}]
R: The full xml report:
R: <?xml version="1.0" encoding="UTF-8"?>
R: <testsuite tests="1" failures="1" timestamp="2023-07-27T14:54:11">
R:
R:
R:   <testcase name="expected_string_found_in_testfile_failed">
R:     <failure message="error: printfile =&gt; report_to_file failed to create a file with the expected string">Test that the expected string is found in file created by printfile =&gt; report_to_filetrace</failure>
R:   </testcase>
R:
R:
R:
R: </testsuite>
R:
R: <!-- not implemented (yet):
R: 1) errors: <error message="my error message">my crash report</error>
R: 2) STDOUT: <system-out>my STDOUT dump</system-out>
R: 3) STDERR: <system-err>my STDERR dump</system-err>
R:  -->
R: /tmp/test-reports-report_to_file.txt did not have expected content. Contains:
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted)
   error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt'
   error: Method 'cleanup' failed in some repairs

Let’s clean up that immutable file.

command
chattr -i /tmp/test-reports-report_to_file.txt
rm /tmp/test-reports-report_to_file.txt