CFEngine for IoT

Posted by Craig Comstock
October 19, 2021

CFEngine is well suited for use in IoT environments due to it’s portability, size, and performance. There already exists a meta layer for including the CFEngine community client and Masterfiles Policy Framework in Yocto Project builds. This enables developing policy to:

  • ensure a service stays running
  • track changes to important files
  • monitor a value over time for normalcy

Let’s walk through bringing up a qemu environment with CFEngine and ensure that a few basic things work: ensure the udev service stays running, tracking changes to important files like /etc/group and a look at monitoring capabilities.

For the next blog in this series we will show using the CFEngine Enterprise client which will enable us to collect reporting data including monitoring data. This will enable us to track the health of my co-worker Nick Anderson by monitoring the height of his desk. He should really be adjusting it about once an hour and is definitely willing to add some automation via CFEngine to optimize his ergonomics.

The Enterprise client is not yet available though we do hope to provide a binary yocto layer in the future.

Yocto Image with CFEngine Community

The Yocto project has a well documented Quick Build process to get up and running. I used Ubuntu 20 for a base OS since it seemed the most supported.

Here are the condensed setup steps to perform:

$ sudo apt-get install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm python3-subunit mesa-common-dev
$ git clone git://git.yoctoproject.org/poky -b hardknott
$ cd poky
$ source oe-init-build-env
Note that this will change your current working directory to `$HOME/poky/build`
$ bitbake core-image-minimal

I like to use core-image-minimal because it takes the least amount of time to iterate on changes and I don’t need any graphics environment to play around with CFEngine. This last step of calling bitbake will take QUITE a long time the first time you execute it on a machine.

On an old laptop with a 2 core Intel Core Duo at 2.80GHz it took about 4 and a half hours. On a newer laptop with a 4 core Intel Core i7-6500U @ 2.50GHz it took about 3 hours.

Once the image has been built, run the image with qemu with options serial nographic so you can stay in your terminal.

$ runqemu qemux86-64 serial nographic

Login as root with no password (the default)

... lots of kernel messages ...
INIT: Entering runlevel: 5
Configuring network interfaces... ip: RTNETLINK answers: File exists
Starting syslogd/klogd: done

Poky (Yocto Project Reference Distro) 3.3.3 qemux86-64 /dev/ttyS0

qemux86-64 login: root
root@qemux86-64:~# uname -a
Linux qemux86-64 5.10.63-yocto-standard #1 SMP PREEMPT Thu Sep 9 02:58:13 UTC 2021 x86_64 GNU/Linux

Use Ctrl-a x to exit qemu, which will shut down the instance running the image you built.

Add CFEngine Layer

We will use the meta-openembedded, meta-oe layer to add CFEngine. Clone their repository under the poky directory you cloned earlier and checkout the latest stable release: hardknott.

$ pwd
/home/user/poky
$ git clone https://github.com/openembedded/meta-openembedded.git
$ cd meta-openembedded
$ git checkout hardknott

Add the meta-oe layer to poky/build/conf/bblayers.conf so that the BBLAYERS section looks like the following:

BBLAYERS ?= " \
  /home/user/poky/meta \
  /home/user/poky/meta-poky \
  /home/user/poky/meta-yocto-bsp \
  /home/user/poky/meta-openembedded/meta-oe \
  "

Now append three lines to the end of poky/build/conf/local.conf.

  • procps is needed to enable CFEngine to properly operate processes
  • cfengine provides the core agent and other daemons
IMAGE_INSTALL_append = " procps"
IMAGE_INSTALL_append = " cfengine"
IMAGE_ROOTFS_EXTRA_SPACE_append = " + 10240"

Running the build again results in a much quicker build. Only about 5 minutes on my slower laptop.

$ time bitbake core-image-minimal

real    5m5.388s
user    0m1.331s
sys     0m0.200s

5 minutes isn’t bad compared to our initial build time of 4.5 hours!

Create Initial Policy

In a typical CFEngine environment we would bootstrap to a policy server that would provide a masterfiles policy set customized to our needs. If we were creating a production yocto image we would likely pre-bootstrap the device to our policy server so that when it first booted it would already be able to setup a trusted connection and pull down any updates to policy. But in this case we want a more self-contained example so will simply copy masterfiles to inputs and work from inputs. This is a common practice when iterating heavily on policy in a development environment.

Start qemu again with

$ runqemu qemux86-64 serial nographic

and confirm that CFEngine is available:

# cf-agent --version
CFEngine Core 3.15.0

And now we can create our first policy which will ensure that the udev service is restarted if it stops.

Create the following /var/libntech/inputs/promises.cf file. Due to CFE-3745 we must use /var/libntech instead of the normal /var/cfengine for the top-level directory. Both vi and cat are available by default in core-image-minimal.

bundle agent main
{
  processes:
    "udevd"
      restart_class => "udevd_not_running";

  commands:
    udevd_not_running::
      "/etc/init.d/udev start"
        classes => results( "bundle", "udevd_restarted" );

  reports:
    udevd_not_running::
      "udevd not running at ${sys.date}";
    udevd_restarted_not_kept::
      "failed to restart udevd at ${sys.date}";
    udevd_restarted_repaired::
      "restart of udevd succeeded at ${sys.date}";
}

body classes results(scope, class_prefix)
{
  scope => "${scope}";

  promise_repaired => { "${class_prefix}_repaired" };

  repair_failed => { "${class_prefix}_not_kept" };
  repair_denied => { "${class_prefix}_not_kept" };
  repair_timeout => { "${class_prefix}_not_kept" };
}

Next follow along with this process of running the agent, killing udevd and ensuring that the agent causes the service to be restarted properly.

root@qemux86-64:~# ps -efl | grep udev[d]
5 S root       115     1  0  80   0 -   869 do_epo 19:24 ?        00:00:00 /sbin/udevd -d
root@qemux86-64:~# cf-agent -KI
root@qemux86-64:~# 

Here we see no output from cf-agent because by default CFEngine is quiet when things are as expected.

Now kill udevd manually and run the agent again.

root@qemux86-64:~# pkill -9 udevd
root@qemux86-64:~# ps -efl | grep udev[d]
root@qemux86-64:~# 
root@qemux86-64:~# cf-agent -KI
    info: Executing 'no timeout' ... '/etc/init.d/udev start'
[  516.810936] udevd[307]: starting version 3.2.10
[  516.913521] udevd[308]: starting eudev-3.2.10
  notice: Q: "...nit.d/udev star": Starting udev
    info: Last 1 quoted lines were generated by promiser '/etc/init.d/udev start'
    info: Completed execution of '/etc/init.d/udev start'
R: udevd not running at Tue Oct 12 19:33:05 2021
R: restart of udevd succeeded at Tue Oct 12 19:33:05 2021

The service was restarted properly by running /etc/init.d/udev start.

The Masterfiles Policy Framework and CFEngine support managing many types of service promises. Unfortunately for our example the name of the service in /etc/init.d, udev, doesn’t match the daemon name udevd so we had to use a workaround in the policy above using processes and commands promises instead.

Track File Changes

Replace /var/libntech/inputs/promises.cf with the following policy:

bundle agent main
{
  vars:
    "track_files" slist => { "/etc/passwd", "/etc/group" };

  files:
    "${track_files}"
      changes => track_changes;

}

body changes track_changes
{
  hash => "sha256";
  report_changes => "all";
  update_hashes => "yes";
}

The first run of the agent after changing promises.cf should show that a new hash is stored for each file:

root@qemux86-64:~# cf-agent -KI
  notice: Storing sha256 hash for '/etc/passwd' (SHA=4db85105a3a3b6ba18f4d119076e0d9682be470b3b66f03d46c4d324cdc05668)
  notice: Storing sha256 hash for '/etc/group' (SHA=f7c9f78c9fb2a002256cb993649c0f691abff1b9594ad1e986acd611cfcc0d53)

Now if we touch or modify these files, agent runs will report the changes:

root@qemux86-64:~# touch /etc/group
root@qemux86-64:~# cf-agent -KI
  notice: Last modified time for '/etc/group' changed 'Fri Mar  9 12:34:56 2018' -> 'Tue Oct 12 18:20:31 2021'
root@qemux86-64:~# echo "junk" >> /etc/group
root@qemux86-64:~# cf-agent -KI
  notice: Hash 'sha256' for '/etc/group' changed!
  notice: Updating sha256 hash for '/etc/group' (SHA=7bf20f1cafa94cd6227e809e4f7e5a497b41027c3a3882a66c4bb9c348a900dd)
  notice: Last modified time for '/etc/group' changed 'Tue Oct 12 18:20:31 2021' -> 'Tue Oct 12 18:20:41 2021'

Monitoring in Community Edition

Here we will simply start the daemon cf-monitord so that some built in measurements are collected and then report with policy what has been collected. By default, cf-monitord stores information about the average value, standard deviation, and most recent value for each monitor.

It is possible to create custom measurements but we won’t touch on that here.

Run cf-monitord, wait for the log line info: Updated averages at ... and press Ctrl-c.

# cf-monitord --no-fork --inform
cf-monitord     info: Updated averages at 'Tue:Hr18:Min25_30'  
<Ctrl-c>

We can run cf-promises --show-vars to see the non-zero values collected so far:

cf-promises --show-vars=default:mon\. | grep -v 00 | grep -v \"\"

Variable name                            Variable value                                               Meta tags
default:mon.av_cpu                       5.04                                                         monitoring,source=environment
default:mon.av_cpu0                      5.04                                                         monitoring,source=environment
default:mon.av_diskfree                  13.80                                                        monitoring,source=environment
default:mon.av_loadavg                   0.08                                                         monitoring,source=environment
default:mon.av_mem_cached                7.68                                                         monitoring,source=environment
default:mon.av_mem_total                 136.13                                                       monitoring,source=environment
default:mon.av_messages                  303.60                                                       monitoring,source=environment
default:mon.av_rootprocs                 34.20                                                        monitoring,source=environment
default:mon.av_users                     0.60                                                         monitoring,source=environment
default:mon.dev_cpu                      8.22                                                         monitoring,source=environment
default:mon.dev_cpu0                     8.22                                                         monitoring,source=environment
default:mon.dev_diskfree                 22.54                                                        monitoring,source=environment
default:mon.dev_loadavg                  0.13                                                         monitoring,source=environment
default:mon.dev_mem_cached               12.55                                                        monitoring,source=environment
default:mon.dev_mem_free                 143.70                                                       monitoring,source=environment
default:mon.dev_mem_total                222.30                                                       monitoring,source=environment
default:mon.dev_messages                 495.78                                                       monitoring,source=environment
default:mon.dev_rootprocs                55.85                                                        monitoring,source=environment
default:mon.dev_users                    0.98                                                         monitoring,source=environment
default:mon.env_time                     Tue Oct 12 18:27:40 2021                                     time_based,source=agent
default:mon.value_cpu                    8.39                                                         monitoring,source=environment
default:mon.value_cpu0                   8.39                                                         monitoring,source=environment
default:mon.value_loadavg                0.13                                                         monitoring,source=environment
default:mon.value_mem_cached             12.80                                                        monitoring,source=environment
default:mon.value_mem_free               146.67                                                       monitoring,source=environment
default:mon.value_mem_total              226.88                                                       monitoring,source=environment

This monitoring data is presented to policy as variables which can then be used to take actions such as pushing data up to a server via a REST API or making needed configuration changes to adapt such as cleaning up disk space.

Extend and Revise our Remarks

We showed that it is easy to include CFEngine in a yocto image built for IoT devices and that there are many use cases which can provide value in this ecosystem.

Next time when we use the Enterprise client we will see how these measurements are collected automatically and can be viewed as charts and data in Mission Portal, our Enterprise Web Interface.

We have an epic in our Jira, CFE-3748, in which we are tracking issues. Feel free to check that out and help out!