At SURFsara we use CFEngine on our National Compute Cluster (LISA) and other systems as our configuration management tool. With the release of CFEngine 3.12 I want to highlight 2 new features, namely:
missing_okmultiple augments
We use these 2 new features heavily in our framework in combination with
my open source library
cf_surfsara_lib.
This library aims to be a central repository for configuring services,
eg: ssh. For configuring the services we use JSON as data format and it
is easily to override the default values via JSON. Pre CFEngine 3.12
there is only one strategy possible:
- def.json : Setting the initial value
- def.cf : Override the initial value
Due to this restriction we have a def.json for all hosts even when the difference between them are small. For minor tweaks we use def.cf with the aid of classes. As with CFEngine version 3.12 you can define multiple JSON files with the aid of the multiple augments feature. In our environment we have implemented the following strategy in def.json:
Listing 1: def.json
{
"augments": [
"$(sys.inputdir)/json.d/global.json",
"$(sys.inputdir)/json.d/os.json",
"$(sys.inputdir)/json.d/key.json",
"$(sys.inputdir)/json.d/ip.json"
]
}We only have these definitions in our def.json so we can easily create a local JSON file with local paths for testing and running automatic checks. Another issue is that all the JSON files must be present else an error is reported. This issue has been reported:
Due to this restriction we must have a default JSON file for all defined JSON files in def.json. In our cluster we have defined roles for this we use the CFEngine key, eg: all compute nodes in our cluster share the same CFEngine key. That is the reason why key.json is not the last JSON file being used in our def.json. For copying the JSON files we use variables that are set by CFEngine, namely:
- os.json : is copied via the CFEngine variable: sys.flavor
- key.json : is copied via the CFEngine variable: connecton.id
- ip.json : is copied via the CFEngine variable: connection.ip
We do not care if the os.json, key.json or ip.json files are present on
the policy server. For this I wrote a patch and introduced the
missing_ok
feature. When this is set then CFEngine will consider the promise kept
even if the file is missing. This is how we use this two new features in
CFengine at SURFsara. Below I will share the update.conf and
cfserverd.conf code. It would be nice for the feature if we can specify
another JSON merge option then override. The discussion for other merge
options is in ticket:
- CFE-2742 : Augments merge strategy
Another feature that I have proposed is CFE-2790 (auto-loading JSON file(s)). With this feature I can just drop JSON file(s) in a directory and it will be load just like the autorun bundle feature.
Listing 2: Update policy
bundle agent update_def_json()
{
methods:
any::
"" usebundle => update_json_file("global.json", "global_def_json", "global_def_json");
"" usebundle => update_json_file("os.json", "$(sys.flavor)_def_json", "default_os_def_json");
"" usebundle => update_json_file("key.json", "key_def_json", "default_key_def_json");
"" usebundle => update_json_file("ip.json", "ip_def_json", "default_ip_def_json");
}
bundle agent update_json_file(file, shortcut, failback_shortcut)
{
vars:
"json_file" string => "$(update_init.json_dir)/$(file)";
files:
any::
"$(json_file)"
comment => "def json file copy for $(shortcut), missing_ok: true",
copy_from => update_def_json("$(shortcut)", "true"),
perms => update_perms("600"),
move_obstructions => "true",
action => update_immediate;
"$(json_file)"
comment => "default def json file copy for $(failback_shortcut), missing_ok: false",
copy_from => update_def_json("$(failback_shortcut)", "false"),
perms => update_perms("600"),
move_obstructions => "true",
action => update_immediate,
classes => update_results("$(failback_shortcut)");
reports:
any::
"$(this.bundle): Failed to fetch: $(json_file)"
ifvarclass => or( canonify("$(failback_shortcut)_failed") );
}
body copy_from update_def_json(from, missing)
{
source => "$(from)";
compare => "hash";
copy_backup => "false";
missing_ok => "$(missing)";
protocol_version => "2";
servers => { @(update_init.servers) };
}Listing 3: cf-serverd policy
bundle server access_rules()
{
### Global.json
"/data/cfengine3/cmdb/global.json"
comment => "def.json for all hosts",
shortcut => "global_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
### Key def.json
"/data/cfengine3/cmdb/key/$(connection.key).json"
comment => "host key def.json",
shortcut => "key_def_json",
admit_keys => { "$(connection.key)" };
"/data/cfengine3/cmdb/key/default.json"
comment => "default host key def.json, failback",
shortcut => "default_key_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
### End Key def.json
### IP def.json
"/data/cfengine3/cmdb/ip/$(connection.ip).json"
comment => "host ip def.json",
shortcut => "ip_def_json",
admit_ips => { "$(connection.ip)" };
"/data/cfengine3/cmdb/ip/default.json"
comment => "default host ip def.json, failback",
shortcut => "default_ip_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
### End IP def.json
### OS def.json
"/data/cfengine3/cmdb/os/debian_9.json"
comment => "debian stretch def.json",
shortcut => "debian_9_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
"/data/cfengine3/cmdb/os/debian_8.json"
comment => "debian jessie def.json",
shortcut => "debian_9_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
"/data/cfengine3/cmdb/os/debian_7.json"
comment => "debian jessie def.json",
shortcut => "debian_9_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
"/data/cfengine3/cmdb/os/centos_6.json"
comment => "centos 6 def.json",
shortcut => "centos_6_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
"/data/cfengine3/cmdb/os/default.json"
comment => "default host os def.json, failback",
shortcut => "default_os_def_json",
admit_ips => { "@(g.acl_admit_ips)" };
### End OS def.json
}This post is syndicated with permission from Bas van der Vlies blog post CFEngine 3.12 New Features Missing OK and Multiple Augments.
