Ever want to manage subvalues in a configuration file? In episode 39 we talk about using field_edits
in edit_line
bundles.
Cody and Craig join Nick as he prototypes and iterates on some policy showing how to manage subvalues in KEY = quoted values. A question raised during one of our recent post show discussions.
Video
The video recording is available on YouTube:
At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. Join the next webinar to not miss this discussion.
Developing policy to edit /etc/default/grub
and similar files
Promise the initial content for a test file
To prototype the policy we first lay down a file with content that is representative of what we are trying to edit. This is very helpful in prototyping policy as it re-sets the initial state of the test file during each execution.
bundle agent __main__
{
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
R: GRUB_CMDLINE_LINUX=""
R:
Testing against GRUB_CMDLINE_LINUX_DEFAULT
When we went to make a promise about GRUB_CMDLINE_LINUX_DEFAULT
we encountered an error. The field_separator
in body edit_field quoted_var
is configured to look only for double quotes.
bundle agent __main__
{
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub;
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub
{
field_edits:
"GRUB_CMDLINE_LINUX=.*"
edit_field => quoted_var( "ipv6.disable=1", "append" );
"GRUB_CMDLINE_LINUX_DEFAULT=.*"
edit_field => quoted_var( "init_on_alloc=1", "append" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
error: Field edit, no fields found by promised pattern '"' in '/tmp/etc-default-grub'
error: Errors encountered when actuating fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*'
info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired
info: Edited file '/tmp/etc-default-grub'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
R: GRUB_CMDLINE_LINUX="ipv6.disable=1"
R:
Refactor to handle single or double quotes
Since body edit_field quoted_var
from the standard library does not handle single quotes we define our own edit_field
body, single_or_double_quoted_subvalue
(based on quoted_var
) which supports single or double quoted values.
bundle agent __main__
{
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub;
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub
{
field_edits:
"GRUB_CMDLINE_LINUX=.*"
edit_field => quoted_var( "ipv6.disable=1", "append" );
"GRUB_CMDLINE_LINUX_DEFAULT=.*"
edit_field => single_or_double_quoted_subvalue( "init_on_alloc=1", "append" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired
info: Appended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Edited file '/tmp/etc-default-grub'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash init_on_alloc=1'
R: GRUB_CMDLINE_LINUX="ipv6.disable=1"
R:
Make more re-usable (but less efficient)
Next we made the policy more re-usable by exposing parameters to the edit_line
bundle. This change however is less efficient as each files promise results in a separate file operation.
bundle agent __main__
{
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub("GRUB_CMDLINE_LINUX",
"ipv6.disable=1",
"append" );
"/tmp/etc-default-grub"
edit_line => etc_default_grub("GRUB_CMDLINE_LINUX_DEFAULT",
"init_on_alloc=1",
"append" );
"/tmp/etc-default-grub"
edit_line => etc_default_grub("GRUB_CMDLINE_LINUX_DEFAULT",
"quiet",
"delete" );
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub(key, entry, method)
{
field_edits:
"$(key)=.*"
edit_field => single_or_double_quoted_subvalue( "$(entry)", "$(method)" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired
info: Edited file '/tmp/etc-default-grub'
info: Appended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Edited file '/tmp/etc-default-grub'
info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Edited file '/tmp/etc-default-grub'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash init_on_alloc=1'
R: GRUB_CMDLINE_LINUX="ipv6.disable=1"
R:
From the execution output above we can clearly see that the file is actually modified three separate times.
Making it a single edit – broken – need coffee
During the show we started to reign in the file operations while maintaining the ability to reuse the policy in different contexts but Nick hadn’t yet had his coffee and chose poorly when he defined the data structure.
bundle agent __main__
{
vars:
"conf[GRUB_CMDLINE_LINUX][ipv6.disable=1]" string => "append";
"conf[GRUB_CMDLINE_LINUX_DEFAULT][init_on_alloc=1]" string => "append";
"conf[GRUB_CMDLINE_LINUX_DEFAULT][quiet]" string => "delete";
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub( "$(this.namespace):$(this.bundle).conf" );
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub(d)
{
vars:
"s" data => mergedata( d );
"keys" slist => getindices( s );
field_edits:
"$(s[$(keys)])=.*"
edit_field => single_or_double_quoted_subvalue( "$(s[$(keys)][1])",
"$(s[$(keys)][2])" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
R: GRUB_CMDLINE_LINUX=""
R:
The fix for a single edit
The solution came to Nick after the post show discussion when he walked away to get coffee.
He needed to change the way he was structuring the data. Here we see he switched to a data
type variable and changed the way that the data was passed to the edit_line
bundle.
bundle agent __main__
{
vars:
"conf" data => '
{
"GRUB_CMDLINE_LINUX": {
"append": [ "ipv6.disable=1" ],
},
"GRUB_CMDLINE_LINUX_DEFAULT": {
"prepend": [ "init_on_alloc=1" ],
"delete": [ "quiet" ],
}
}';
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub( "@($(this.namespace):$(this.bundle).conf)" );
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub(conf_map)
{
vars:
"keys" slist => getindices( conf_map );
"methods[$(keys)]" slist => getindices( "conf_map[$(keys)]" );
field_edits:
"$(keys)=.*"
edit_field => single_or_double_quoted_subvalue( "$(conf_map[$(keys)][$(methods[$(keys)])])",
"$(methods[$(keys)])" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired
info: Prepended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Edited file '/tmp/etc-default-grub'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='init_on_alloc=1 cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
R: GRUB_CMDLINE_LINUX="ipv6.disable=1"
R:
From the above output we can see that all the changes were made as expected and only a single file change was made.
How do I use the same data structure with associative arrays?
In addressing the issue Nick switched to using a data
variable instead of an associative array which is a collection of individual variables in CFEngine.
Here is an iteration showing use with associative arrays. Note that in addition to changing the way the data structure is defined, we also changed how it was passed back to passing the name of the variable rather than the data structure as a whole.
bundle agent __main__
{
vars:
"conf[GRUB_CMDLINE_LINUX][append]" slist => { "ipv6.disable=1" };
"conf[GRUB_CMDLINE_LINUX_DEFAULT][prepend]" slist => { "init_on_alloc=1" };
"conf[GRUB_CMDLINE_LINUX_DEFAULT][delete]" slist => { "quiet" };
files:
"/tmp/etc-default-grub"
content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
`;
"/tmp/etc-default-grub"
edit_line => etc_default_grub( "$(this.namespace):$(this.bundle).conf" );
reports:
"/tmp/etc-default-grub"
printfile => cat( "$(this.promiser)");
}
bundle edit_line etc_default_grub(conf_var)
{
vars:
"conf_map" data => mergedata( "$(conf_var)" );
"keys" slist => getindices( conf_map );
"methods[$(keys)]" slist => getindices( "conf_map[$(keys)]" );
field_edits:
"$(keys)=.*"
edit_field => single_or_double_quoted_subvalue( "$(conf_map[$(keys)][$(methods[$(keys)])])",
"$(methods[$(keys)])" );
}
body edit_field single_or_double_quoted_subvalue( entry, method)
{
inherit_from => quoted_var( "$(entry)", "$(method)" );
field_separator => "(\"|')";
}
cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf
info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
GRUB_CMDLINE_LINUX=""
'
info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Prepended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired
info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub'
info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired
info: Edited file '/tmp/etc-default-grub'
R: /tmp/etc-default-grub
R: GRUB_CMDLINE_LINUX_DEFAULT='init_on_alloc=1 cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash'
R: GRUB_CMDLINE_LINUX="ipv6.disable=1"
R:
Post show discussion
- Did you know that CFEngine has 3 quote characters? Single quote (
'
), double quote ("
) and back tick (`
). This can allow you to avoid having to escape inner quotes.
Questions
How to influence order of operations (Normal Order)
Using handle
and depends_on
bundle agent __main__
[file=/tmp/example-depends-on.cf]
{
commands:
"/bin/echo RED"
handle => "red",
depends_on => { "yellow" };
"/bin/echo YELLOW"
handle => "yellow";
methods:
"help"
depends_on => { "red" };
}
bundle agent help
{
commands: "/bin/echo Byeee";
}
cf-agent --no-lock --log-level info --file /tmp/example-depends-on.cf
info: Executing 'no timeout' ... '/bin/echo YELLOW'
notice: Q: ".../bin/echo YELLO": YELLOW
info: Last 1 quoted lines were generated by promiser '/bin/echo YELLOW'
info: Completed execution of '/bin/echo YELLOW'
info: Executing 'no timeout' ... '/bin/echo RED'
notice: Q: ".../bin/echo RED": RED
info: Last 1 quoted lines were generated by promiser '/bin/echo RED'
info: Completed execution of '/bin/echo RED'
info: Executing 'no timeout' ... '/bin/echo Byeee'
notice: Q: ".../bin/echo Byeee": Byeee
info: Last 1 quoted lines were generated by promiser '/bin/echo Byeee'
info: Completed execution of '/bin/echo Byeee'
Use methods to get a more strict order
bundle agent __main__
{
methods:
"stage1";
"stage2";
"stage3";
}
Verify configuration before restarting the service
You may want to verify a configuration file has valid syntax before restarting a service. Consider rendering your configuration to a staging location that is verified and copied to final destination after syntax check. This way you can avoid issues that could be cause by some random reboot where you have a bad config in place but the service wasn’t restarted since it wasn’t valid.
How can I access another bundles classes?
You can not. Classes are either namespace
scope (visible by any other bundle), or they are bundle
scoped.
- Classes defined from classes promises in common bundles are
namespace
scoped by default. - Classes defined from classes promises in agent bundles are
bundle
scoped by default. - You can specify override the default scope using the
scope
attribute. - Bundle scoped classes can be inherited by another bundle
- Information from a bundle can be passed to an associative array in the parent bundle via the
bundle_return_value_index
attribute ofreports
promises if the parent called the bundle withuseresult
.
Links
- Connect on LinkedIn w/ Cody, Craig, or Nick
- All Episodes
- Post show discussion
- Influence order with depends_on
- Methods can inherit the callers bundle scoped classes
- Allow called bundle to return information to the calling bundle via useresult
- Pass data to calling bundle using bundle_return_value_index