The CFEngine Test Suites

When developing a large and flexible software system like CFEngine, it is an unfortunate truth that developers do not always get everything right at the first attempt. The source code is also changing frequently due to new features, optimizations and bug-fixes. As you can see from the Ohloh online statistics, thousands of lines are added and removed every week.

All changes have the risk of not working in every scenario, or breaking existing functionality. In addition, build errors may introduce unexpected behaviour that you want to detect before deploying to production.

There is a clear need for a system to limit these risks. This is where the CFEngine test systems come in to play.

CFEngine has three extensive sets of tests, each covering a different purpose to ensure correct and performant operation.

Acceptance tests

As you might know, an acceptance test is a test where you know a certain input should generate a specific output. This is also known as black-box testing, because you do not consider the internals (the “contents of the box”) of the software, merely the resulting output.

In the CFEngine github source repository, you can find the acceptance tests under tests/acceptance.

These are CFEngine policies that have (at least) two bundles: test and check. The test bundle is the actual policy that we are testing for correctness, while the check bundle compares the actual against the expected result after the policy has been run. For example, we may try to create an empty file in the test bundle, and check if the file exists in the check bundle.

At the time of writing, there are 939 of these acceptance tests. They can be run by using the testall script:

~/core/tests/acceptance$ ./testall
======================================================================
Testsuite started at 2012-04-24 11:13:38
----------------------------------------------------------------------
Total tests: 939

./00_basics/01_compiler/001.cf Pass
./00_basics/01_compiler/002.x.cf Pass
....
....

======================================================================
Testsuite finished at 2012-04-24 11:01:21 (172 seconds)

Passed tests: 692
Failed tests: 0
Failed to crash tests: 0
Skipped tests: 247

Have a look in the README file for a more extensive description on how to use the suite and create new tests.

Load tests

Load tests are intended to measure the performance and correctness under a high load.

For example, imagine you have a policy server with thousands of clients. The code that cf-serverd executes on that policy server will have to perform very well, but correctness during concurrent (threaded) operations is equally important.

Code executed in scenarios like these are intended to be covered by the set of load tests.

At the moment, only a test for embedded database operation is covered by the load tests, but this will (like the other tests) expand over time. This test was used extensively by developers when vastly improving the database performance in the 3.3.0 release.

LinkedIn reported a performance improvement of 14 to 3 seconds during file transfers from their policy server. It all started with this simple load test.

Unit tests

Finally, unit tests are small programs that test very specific functionality within CFEngine. Manipulation of internal data-structures, such as the lists that holds all the policy variables, is a common test area for unit-tests. These are also known as white-box tests, because we care not only about the total input and output of CFEngine, but the internal workings.

To run them after you have cloned the core repository, you first haveto compile them individually.

~/core/tests/unit$ make alloc_test
  CC     alloc_test.o
  CC     cmockery.lo
  CC     test.lo
  CC     alloc.lo
  CCLD   libtest.la
  CCLD   alloc_test
~/core/tests/unit$ ./alloc_test
test_xasprintf: Starting test
test_xasprintf: Test completed successfully.
test_xvasprintf: Starting test
test_xvasprintf: Test completed successfully.
All 2 tests passed

These three sets of tests ensure the long-term correctness and performance of CFEngine. They are regularly run by the CFEngine build system, so that the developers will get feedback as soon as any do not pass anymore. This way, errors can be corrected promptly and before any users run the code.

Please consider submitting your own tests if you find issues or see areas that are not covered. This can be done through a github pull request.