CFEngine is well suited for use in IoT environments due to it’s portability, size, and performance. There already exists a meta layer for including the CFEngine community client and Masterfiles Policy Framework in Yocto Project builds. This enables developing policy to:
- ensure a service stays running
- track changes to important files
- monitor a value over time for normalcy
Let’s walk through bringing up a qemu environment with CFEngine and ensure that a few basic things work: ensure the udev service stays running, tracking changes to important files like /etc/group and a look at monitoring capabilities.
For the next blog in this series we will show using the CFEngine Enterprise client which will enable us to collect reporting data including monitoring data. This will enable us to track the health of my co-worker Nick Anderson by monitoring the height of his desk. He should really be adjusting it about once an hour and is definitely willing to add some automation via CFEngine to optimize his ergonomics.
The Enterprise client is not yet available though we do hope to provide a binary yocto layer in the future.
Yocto Image with CFEngine Community
The Yocto project has a well documented Quick Build process to get up and running. I used Ubuntu 20 for a base OS since it seemed the most supported.
Here are the condensed setup steps to perform:
$ sudo apt-get install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm python3-subunit mesa-common-dev
$ git clone git://git.yoctoproject.org/poky -b hardknott
$ cd poky
$ source oe-init-build-env
Note that this will change your current working directory to `$HOME/poky/build`
$ bitbake core-image-minimalI like to use core-image-minimal because it takes the least amount of time to iterate on changes and I don’t need any graphics environment to play around with CFEngine.
This last step of calling bitbake will take QUITE a long time the first time you execute it on a machine.
On an old laptop with a 2 core Intel Core Duo at 2.80GHz it took about 4 and a half hours. On a newer laptop with a 4 core Intel Core i7-6500U @ 2.50GHz it took about 3 hours.
Once the image has been built, run the image with qemu with options serial nographic so you can stay in your terminal.
$ runqemu qemux86-64 serial nographicLogin as root with no password (the default)
... lots of kernel messages ...
INIT: Entering runlevel: 5
Configuring network interfaces... ip: RTNETLINK answers: File exists
Starting syslogd/klogd: done
Poky (Yocto Project Reference Distro) 3.3.3 qemux86-64 /dev/ttyS0
qemux86-64 login: root
root@qemux86-64:~# uname -a
Linux qemux86-64 5.10.63-yocto-standard #1 SMP PREEMPT Thu Sep 9 02:58:13 UTC 2021 x86_64 GNU/LinuxUse Ctrl-a x to exit qemu, which will shut down the instance running the image you built.
Add CFEngine Layer
We will use the meta-openembedded, meta-oe layer to add CFEngine.
Clone their repository under the poky directory you cloned earlier and checkout the latest stable release: hardknott.
$ pwd
/home/user/poky
$ git clone https://github.com/openembedded/meta-openembedded.git
$ cd meta-openembedded
$ git checkout hardknottAdd the meta-oe layer to poky/build/conf/bblayers.conf so that the BBLAYERS section looks like the following:
BBLAYERS ?= " \
/home/user/poky/meta \
/home/user/poky/meta-poky \
/home/user/poky/meta-yocto-bsp \
/home/user/poky/meta-openembedded/meta-oe \
"Now append three lines to the end of poky/build/conf/local.conf.
procpsis needed to enable CFEngine to properly operateprocessescfengineprovides the core agent and other daemons
IMAGE_INSTALL_append = " procps"
IMAGE_INSTALL_append = " cfengine"
IMAGE_ROOTFS_EXTRA_SPACE_append = " + 10240"Running the build again results in a much quicker build. Only about 5 minutes on my slower laptop.
$ time bitbake core-image-minimal
real 5m5.388s
user 0m1.331s
sys 0m0.200s5 minutes isn’t bad compared to our initial build time of 4.5 hours!
Create Initial Policy
In a typical CFEngine environment we would bootstrap to a policy server that would provide a masterfiles policy set customized to our needs. If we were creating a production yocto image we would likely pre-bootstrap the device to our policy server so that when it first booted it would already be able to setup a trusted connection and pull down any updates to policy. But in this case we want a more self-contained example so will simply copy masterfiles to inputs and work from inputs. This is a common practice when iterating heavily on policy in a development environment.
Start qemu again with
$ runqemu qemux86-64 serial nographicand confirm that CFEngine is available:
# cf-agent --version
CFEngine Core 3.15.0And now we can create our first policy which will ensure that the udev service is restarted if it stops.
Create the following /var/libntech/inputs/promises.cf file.
Due to CFE-3745 we must use /var/libntech instead of the normal /var/cfengine for the top-level directory.
Both vi and cat are available by default in core-image-minimal.
bundle agent main
{
processes:
"udevd"
restart_class => "udevd_not_running";
commands:
udevd_not_running::
"/etc/init.d/udev start"
classes => results( "bundle", "udevd_restarted" );
reports:
udevd_not_running::
"udevd not running at ${sys.date}";
udevd_restarted_not_kept::
"failed to restart udevd at ${sys.date}";
udevd_restarted_repaired::
"restart of udevd succeeded at ${sys.date}";
}
body classes results(scope, class_prefix)
{
scope => "${scope}";
promise_repaired => { "${class_prefix}_repaired" };
repair_failed => { "${class_prefix}_not_kept" };
repair_denied => { "${class_prefix}_not_kept" };
repair_timeout => { "${class_prefix}_not_kept" };
}Next follow along with this process of running the agent, killing udevd and ensuring that the agent causes the service to be restarted properly.
root@qemux86-64:~# ps -efl | grep udev[d]
5 S root 115 1 0 80 0 - 869 do_epo 19:24 ? 00:00:00 /sbin/udevd -d
root@qemux86-64:~# cf-agent -KI
root@qemux86-64:~# Here we see no output from cf-agent because by default CFEngine is quiet when things are as expected.
Now kill udevd manually and run the agent again.
root@qemux86-64:~# pkill -9 udevd
root@qemux86-64:~# ps -efl | grep udev[d]
root@qemux86-64:~#
root@qemux86-64:~# cf-agent -KI
info: Executing 'no timeout' ... '/etc/init.d/udev start'
[ 516.810936] udevd[307]: starting version 3.2.10
[ 516.913521] udevd[308]: starting eudev-3.2.10
notice: Q: "...nit.d/udev star": Starting udev
info: Last 1 quoted lines were generated by promiser '/etc/init.d/udev start'
info: Completed execution of '/etc/init.d/udev start'
R: udevd not running at Tue Oct 12 19:33:05 2021
R: restart of udevd succeeded at Tue Oct 12 19:33:05 2021The service was restarted properly by running /etc/init.d/udev start.
The Masterfiles Policy Framework and CFEngine support managing many types of service promises.
Unfortunately for our example the name of the service in /etc/init.d, udev, doesn’t match the daemon name udevd so we had to use a workaround in the policy above using processes and commands promises instead.
Track File Changes
Replace /var/libntech/inputs/promises.cf with the following policy:
bundle agent main
{
vars:
"track_files" slist => { "/etc/passwd", "/etc/group" };
files:
"${track_files}"
changes => track_changes;
}
body changes track_changes
{
hash => "sha256";
report_changes => "all";
update_hashes => "yes";
}The first run of the agent after changing promises.cf should show that a new hash is stored for each file:
root@qemux86-64:~# cf-agent -KI
notice: Storing sha256 hash for '/etc/passwd' (SHA=4db85105a3a3b6ba18f4d119076e0d9682be470b3b66f03d46c4d324cdc05668)
notice: Storing sha256 hash for '/etc/group' (SHA=f7c9f78c9fb2a002256cb993649c0f691abff1b9594ad1e986acd611cfcc0d53)Now if we touch or modify these files, agent runs will report the changes:
root@qemux86-64:~# touch /etc/group
root@qemux86-64:~# cf-agent -KI
notice: Last modified time for '/etc/group' changed 'Fri Mar 9 12:34:56 2018' -> 'Tue Oct 12 18:20:31 2021'
root@qemux86-64:~# echo "junk" >> /etc/group
root@qemux86-64:~# cf-agent -KI
notice: Hash 'sha256' for '/etc/group' changed!
notice: Updating sha256 hash for '/etc/group' (SHA=7bf20f1cafa94cd6227e809e4f7e5a497b41027c3a3882a66c4bb9c348a900dd)
notice: Last modified time for '/etc/group' changed 'Tue Oct 12 18:20:31 2021' -> 'Tue Oct 12 18:20:41 2021'Monitoring in Community Edition
Here we will simply start the daemon cf-monitord so that some built in measurements are collected and
then report with policy what has been collected. By default, cf-monitord stores information about the average
value, standard deviation, and most recent value for each monitor.
It is possible to create custom measurements but we won’t touch on that here.
Run cf-monitord, wait for the log line info: Updated averages at ... and press Ctrl-c.
# cf-monitord --no-fork --inform
cf-monitord info: Updated averages at 'Tue:Hr18:Min25_30'
<Ctrl-c>We can run cf-promises --show-vars to see the non-zero values collected so far:
cf-promises --show-vars=default:mon\. | grep -v 00 | grep -v \"\"
Variable name Variable value Meta tags
default:mon.av_cpu 5.04 monitoring,source=environment
default:mon.av_cpu0 5.04 monitoring,source=environment
default:mon.av_diskfree 13.80 monitoring,source=environment
default:mon.av_loadavg 0.08 monitoring,source=environment
default:mon.av_mem_cached 7.68 monitoring,source=environment
default:mon.av_mem_total 136.13 monitoring,source=environment
default:mon.av_messages 303.60 monitoring,source=environment
default:mon.av_rootprocs 34.20 monitoring,source=environment
default:mon.av_users 0.60 monitoring,source=environment
default:mon.dev_cpu 8.22 monitoring,source=environment
default:mon.dev_cpu0 8.22 monitoring,source=environment
default:mon.dev_diskfree 22.54 monitoring,source=environment
default:mon.dev_loadavg 0.13 monitoring,source=environment
default:mon.dev_mem_cached 12.55 monitoring,source=environment
default:mon.dev_mem_free 143.70 monitoring,source=environment
default:mon.dev_mem_total 222.30 monitoring,source=environment
default:mon.dev_messages 495.78 monitoring,source=environment
default:mon.dev_rootprocs 55.85 monitoring,source=environment
default:mon.dev_users 0.98 monitoring,source=environment
default:mon.env_time Tue Oct 12 18:27:40 2021 time_based,source=agent
default:mon.value_cpu 8.39 monitoring,source=environment
default:mon.value_cpu0 8.39 monitoring,source=environment
default:mon.value_loadavg 0.13 monitoring,source=environment
default:mon.value_mem_cached 12.80 monitoring,source=environment
default:mon.value_mem_free 146.67 monitoring,source=environment
default:mon.value_mem_total 226.88 monitoring,source=environmentThis monitoring data is presented to policy as variables which can then be used to take actions such as pushing data up to a server via a REST API or making needed configuration changes to adapt such as cleaning up disk space.
Extend and Revise our Remarks
We showed that it is easy to include CFEngine in a yocto image built for IoT devices and that there are many use cases which can provide value in this ecosystem.
Next time when we use the Enterprise client we will see how these measurements are collected automatically and can be viewed as charts and data in Mission Portal, our Enterprise Web Interface.
We have an epic in our Jira, CFE-3748, in which we are tracking issues. Feel free to check that out and help out!
