Migrating to cfbs

Traditionally, CFEngine policy sets are managed as a whole. When upgrading the Masterfiles Policy Framework (MPF)¹ users must download the new version of the policy framework and integrate it into the existing policy set, carefully diffing the vendored policy files against their currently integrated policy. Updates to policy authored by others must be sought out and similarly integrated. The burden is on the user to maintain the knowledge of where policy is sourced, if updates are available, and how it is integrated into the policy set as a whole.

cfbs ²the command line tool for the CFEngine Build System³ streamlines management of a CFEngine policy set. It facilitates upgrades of the Masterfiles Policy Framework (MPF), consumption of modules (inventory and management policy, custom promise types, compliance reports, etc …) from build.cfengine.com and other sources.

In this blog post we will cover the process of initializing a traditionally managed policy set integrating custom policy in a couple of typical ways and then migrating to a policy set managed by cfbs.

Initalizing a traditionally managed policy set

First, we need to initialize a traditional policy set. Let’s start with masterfiles from the 3.18.3 release. This way we can go through the process of migrating to the same version with cfbs. We need to download it and unpack it into the root of our repository.

MPF_TARBALL_URL=$(cf-remote --version 3.18.3 list masterfiles | tail -n 1)
MPF_TARBALL_FILENAME=$(basename $MPF_TARBALL_URL)
curl -s -O $MPF_TARBALL_URL
tar -zxf $MPF_TARBALL_FILENAME --strip-components=2
rm -f $MPF_TARBALL_FILENAME
ls

We can see it’s a standard stock policy set.

cfe_internal
controls
inventory
lib
modules
promises.cf
services
standalone_self_upgrade.cf
templates
update.cf

Keep generated files from getting committed to the policy set

Next we need to lay down the typical .gitignore for a policy set.

printf "# These files are generated and should not become part of the policy set\n" > .gitignore
printf "cf_promises_validated\n"  >> .gitignore
printf "cf_promises_release_id\n" >> .gitignore

Let’s go ahead and initialize the git repository and commit the current changes.

git init
git add .
git commit -m "In the beginning there was darkness, then Nick said let there be a stock 3.18.3 policy set" | grep -v "create mode"

hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: 	git config --global init.defaultBranch <name>
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 	git branch -m <name>
Initialized empty Git repository in /tmp/migrating-to-cfbs/.git/
[master (root-commit) fba5378] In the beginning there was darkness, then Nick said let there be a stock 3.18.3 policy set
 131 files changed, 27107 insertions(+)

Now we have a stock 3.18.3 policy set.

Integrating custom policy

Next we need to add some custom policy. Let’s integrate policy in a few different ways to be representative of what you might find in a policy set that has been around for some time.

Long, long ago it was standard practice to modify the default policy to add custom policy to inputs and the bundlesequence. Let’s create a custom policy file and add it to inputs and bundlesequence.

cat << EOF > custom-policy-1.cf
bundle agent custom_policy_1
{
  reports: 'Policy from \$(this.bundle)';
}
EOF
sed -i '/^\s*"services\/main.cf",/a "custom-policy-1.cf",' promises.cf
sed -i '/^\s*@(def.bundlesequence_end),/a custom_policy_1,' promises.cf

These days we have Augments (def.json) that allow us to define variables and classes very early during the bundlesequence and the MPF is instrumented with many variables which can be leveraged to influence the behavior of the policy. Let’s add a couple more policies integrated explicitly via augments as well as via autorun.

cat << EOF > services/custom-policy-2.cf
bundle agent custom_policy_2
{
   reports: 'Hello from \$(this.bundle) \$(with)'
             with => join( ",", callstack_promisers() );
}
EOF
cat << EOF > services/autorun/custom-policy-3.cf
bundle agent custom_policy_3
{
   meta: "tags" slist => { "autorun" };
  reports: 'Hello from \$(this.bundle)';
}
EOF

cat << EOF > def.json
{
  "inputs": [ "services/custom-policy-2.cf" ],

  "classes": {
    "services_autorun":  [ "any::" ]
  },

  "vars": {
    "control_common_bundlesequence_end": [ "custom_policy_2" ]
  }
}
EOF
#git commit -m "Added custom policy integrated via Augments"

services/main.cf also commonly contains custom policy, typically methods type promises to call other custom policy.

sed -i '/^\s*# Activate your custom policies here/a "Calling custom_policy_2 from main" usebundle => custom_policy_2;' services/main.cf

With custom policy in place, let’s commit.

git add .
git commit -m "Added custom policy integrated in various ways"

[master e390bb0] Added custom policy integrated in various ways
 6 files changed, 28 insertions(+)
 create mode 100644 custom-policy-1.cf
 create mode 100644 def.json
 create mode 100644 services/autorun/custom-policy-3.cf
 create mode 100644 services/custom-policy-2.cf

Initialize the cfbs project

Now, let’s initialize this repository as a cfbs project.

The easiest way to get a cfbs project started is to use cfbs init.

cfbs init --non-interactive

[master f58dc2f] Initialized a new CFEngine Build project
 1 file changed, 7 insertions(+)
 create mode 100644 cfbs.json
[master 72ce205] Added module 'masterfiles'
 1 file changed, 13 insertions(+), 1 deletion(-)


Initialized an empty project called 'Example project' in 'cfbs.json'
Added module: masterfiles

By default it will use the most recent version of masterfiles, but we want to first get cfbs replicating our existing policy set so we will remove masterfiles.

cfbs remove masterfiles --non-interactive

[master c9aeb2a] Removed module 'masterfiles'
 1 file changed, 7 insertions(+), 19 deletions(-)
 rewrite cfbs.json (80%)
Removing module 'masterfiles'

And add it back at a specific version.

cfbs add masterfiles@3.18.3 --non-interactive

[master efc200f] Added module 'masterfiles'
 1 file changed, 20 insertions(+), 1 deletion(-)
Added module: masterfiles

At this point we can cfbs build.

cfbs build


001 masterfiles @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded)

001 masterfiles : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh'
001 masterfiles : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y'
001 masterfiles : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g''
001 masterfiles : run 'find . -name '*.bak' | xargs rm'
001 masterfiles : copy './' 'masterfiles/'

Generating tarball...

Build complete, ready to deploy 🐿
 -> Directory: out/masterfiles
 -> Tarball:   out/masterfiles.tgz

To install on this machine: sudo cfbs install
To deploy on remote hub(s): cf-remote deploy

Find the differences between cfbs built policy and traditionally integrated policy set

Now we can see how our cfbs built 3.18.3 policy differs from our current policy set. We recursively diff the root of our policy set against the cfbs build result in out/masterfiles.

diff --exclude=out --recursive --unified . out/masterfiles

Only in .: cfbs.json
Only in out/masterfiles/cfe_internal/core/watchdog: README.md
Only in out/masterfiles/cfe_internal/enterprise/ha: ha_info.json
Only in .: custom-policy-1.cf
Only in .: def.json
Only in .: .git
Only in out/masterfiles: .github
Only in .: .gitignore
Only in out/masterfiles/inventory: README.md
Only in out/masterfiles/lib: README.md
Only in out/masterfiles: LICENSE
Only in out/masterfiles/modules: promises
Only in out/masterfiles: .no-distrib
diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf
--- ./promises.cf	2023-06-06 08:09:17.303386890 -0500
+++ out/masterfiles/promises.cf	2023-06-06 08:11:05.094842107 -0500
@@ -59,7 +59,6 @@
main,
@(cfengine_enterprise_hub_ha.management_bundles),
@(def.bundlesequence_end),
-custom_policy_1,

};

@@ -86,7 +85,6 @@
@(services_autorun.inputs),

"services/main.cf",
-"custom-policy-1.cf",
};

version => "CFEngine Promises.cf 3.18.3";
Only in ./services/autorun: custom-policy-3.cf
Only in out/masterfiles/services/autorun: README.md
Only in ./services: custom-policy-2.cf
diff '--exclude=out' --recursive --unified ./services/main.cf out/masterfiles/services/main.cf
--- ./services/main.cf	2023-06-06 08:09:25.299346646 -0500
+++ out/masterfiles/services/main.cf	2023-06-06 08:11:05.098842087 -0500
@@ -10,6 +10,5 @@
{
# Activate your custom policies here
-"Calling custom_policy_2 from main" usebundle => custom_policy_2;

}
Only in out/masterfiles/templates: README.md

Integrate custom policy into cfbs project

We can see there are a few differences, but first let’s focus on the files that are missing from the cfbs built policy.

diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \."

Only in .: cfbs.json
Only in .: custom-policy-1.cf
Only in .: def.json
Only in .: .git
Only in .: .gitignore
Only in ./services/autorun: custom-policy-3.cf
Only in ./services: custom-policy-2.cf

We can ignore the .git, .gitignore, and cfbs.json.

diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json"

Only in .: custom-policy-1.cf
Only in .: def.json
Only in ./services/autorun: custom-policy-3.cf
Only in ./services: custom-policy-2.cf

custom-policy-1.cf

Now we can easily see that we need to get custom-policy-1.cf, def.json, services/autorun/custom-policy-3.cf, and services/custom-policy-2.cf into the cfbs built policy set. Let’s cfbs add them.

cfbs add custom-policy-1.cf --non-interactive

[master 05ee342] Added module './custom-policy-1.cf'
 1 file changed, 11 insertions(+)
Added module: ./custom-policy-1.cf

Let’s take a look at what cfbs add did.

git log -p -n 1

commit 05ee342d1c58ea62b449614f622cd234f60a413a
Author: Nick Anderson <nick@cmdln.org>
Date:   Tue Jun 6 08:11:40 2023 -0500

    Added module './custom-policy-1.cf'

diff --git a/cfbs.json b/cfbs.json
index 885c056..5a30ddc 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -20,6 +20,17 @@
         "run find . -name '*.bak' | xargs rm",
         "copy ./ ./"
       ]
+    },
+    {
+      "name": "./custom-policy-1.cf",
+      "description": "Local policy file added using cfbs command line",
+      "tags": ["local"],
+      "added_by": "cfbs add",
+      "steps": [
+        "copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf",
+        "policy_files services/cfbs/custom-policy-1.cf",
+        "bundles custom_policy_1"
+      ]
     }
   ],
   "git": true

copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf: This copies custom-policy-1.cf to services/cfbs/custom-policy-1.cf relative to the root of the built policy.
policy_files services/cfbs/custom-policy-1.cf: This adds services/cfbs/custom-policy-1.cf to inputs in def.json so that the policy file is parsed.
bundles custom_policy_1: This adds custom_policy_1 to default:def.control_common_bundlesequence_end in def.json so that the bundle will be run as part of the bundlesequence.

At least for now, we are looking to replicate the existing policy set while trying to avoid modifications to the MPF, so let’s edit cfbs.json so that custom-policy-1.cf is placed in the root of the built policy.

sed -i 's|services/cfbs/custom-policy-1.cf|custom-policy-1.cf|g' cfbs.json

Let’s check that the build steps for custom-policy-1.cf are as desired.

git diff

diff --git a/cfbs.json b/cfbs.json
index 5a30ddc..9429cf3 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -27,8 +27,8 @@
       "tags": ["local"],
       "added_by": "cfbs add",
       "steps": [
-        "copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf",
-        "policy_files services/cfbs/custom-policy-1.cf",
+        "copy ./custom-policy-1.cf custom-policy-1.cf",
+        "policy_files custom-policy-1.cf",
         "bundles custom_policy_1"
       ]
     }

That looks correct, so we can commit those changes to the cfbs project.

git add cfbs.json
git commit -m "Fixed custom-policy-1.cf build integration target location, inputs and bundlesequence"

[master c36c917] Fixed custom-policy-1.cf build integration target location, inputs and bundlesequence
 1 file changed, 2 insertions(+), 2 deletions(-)

custom-policy-2.cf

Next we do similar with the other custom policy files, first services/custom-policy-2.cf.

First we add the local policy to the project.

cfbs add services/custom-policy-2.cf --non-interactive

Which bundle should be evaluated (added to bundle sequence)?
 1. ./services/custom-policy-2.cf:custom_policy_2 (default)
 2. (None)
 [1/2]: Added module: ./services/custom-policy-2.cf
The default commit message is 'Added module './services/custom-policy-2.cf'' - edit it? [yes/y/NO/n] [master 497fdbb] Added module './services/custom-policy-2.cf'
 1 file changed, 11 insertions(+)

We see that it’s target location is not as desired.

git log -p -n 1

commit 497fdbb63f2ce48f2c11616049168835221aed37
Author: Nick Anderson <nick@cmdln.org>
Date:   Tue Jun 6 08:12:07 2023 -0500

    Added module './services/custom-policy-2.cf'

diff --git a/cfbs.json b/cfbs.json
index 9429cf3..f075886 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -31,6 +31,17 @@
         "policy_files custom-policy-1.cf",
         "bundles custom_policy_1"
       ]
+    },
+    {
+      "name": "./services/custom-policy-2.cf",
+      "description": "Local policy file added using cfbs command line",
+      "tags": ["local"],
+      "added_by": "cfbs add",
+      "steps": [
+        "copy ./services/custom-policy-2.cf services/cfbs/services/custom-policy-2.cf",
+        "policy_files services/cfbs/services/custom-policy-2.cf",
+        "bundles custom_policy_2"
+      ]
     }
   ],
   "git": true

We correct the target location.

sed -i 's|services/cfbs/services/custom-policy-2.cf|services/custom-policy-2.cf|g' cfbs.json

We inspect our change.

git diff

diff --git a/cfbs.json b/cfbs.json
index f075886..e5811ea 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -38,8 +38,8 @@
       "tags": ["local"],
       "added_by": "cfbs add",
       "steps": [
-        "copy ./services/custom-policy-2.cf services/cfbs/services/custom-policy-2.cf",
-        "policy_files services/cfbs/services/custom-policy-2.cf",
+        "copy ./services/custom-policy-2.cf services/custom-policy-2.cf",
+        "policy_files services/custom-policy-2.cf",
         "bundles custom_policy_2"
       ]
     }

We commit our change fixing the target location for custom-policy-2.cf

git add cfbs.json
git commit -m "Fixed custom-policy-2.cf build integration target location, inputs and bundlesequence"

[master 4868aeb] Fixed custom-policy-2.cf build integration target location, inputs and bundlesequence
 1 file changed, 2 insertions(+), 2 deletions(-)

custom-policy-3.cf

Finally, we start integrating the missing custom-policy-3.cf.

cfbs add services/autorun/custom-policy-3.cf --non-interactive

WARNING: Found bundle tagged with autorun in local policy file './services/autorun/custom-policy-3.cf': Note that the autorun tag is ignored when adding local policy files or subdirectories.
[master 4894b2f] Added module './services/autorun/custom-policy-3.cf'
 1 file changed, 11 insertions(+)
Added module: ./services/autorun/custom-policy-3.cf

We again see the target is not as desired, but also we see that we don’t need to explicitly include this policy file in inputs or the bundlesequence as it leverages autorun for inclusion.

git log -p -n 1

commit 4894b2fd4c47746e4bc734da13d79a6c881b8858
Author: Nick Anderson <nick@cmdln.org>
Date:   Tue Jun 6 08:13:04 2023 -0500

    Added module './services/autorun/custom-policy-3.cf'

diff --git a/cfbs.json b/cfbs.json
index e5811ea..f57ad96 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -42,6 +42,17 @@
         "policy_files services/custom-policy-2.cf",
         "bundles custom_policy_2"
       ]
+    },
+    {
+      "name": "./services/autorun/custom-policy-3.cf",
+      "description": "Local policy file added using cfbs command line",
+      "tags": ["local"],
+      "added_by": "cfbs add",
+      "steps": [
+        "copy ./services/autorun/custom-policy-3.cf services/cfbs/services/autorun/custom-policy-3.cf",
+        "policy_files services/cfbs/services/autorun/custom-policy-3.cf",
+        "bundles custom_policy_3"
+      ]
     }
   ],
   "git": true

We adjust the target path and commit the changes to the cfbs project, and we redact the addition of the file to inputs (since it’s in the services/autorun directory as well as it’s inclusion to the top level bundlesequence.

# Fix  the target path for custom-policy-3
sed -i 's|services/cfbs/services/autorun/custom-policy-3.cf|services/autorun/custom-policy-3.cf|g' cfbs.json
# Remove custom-policy-3.cf from inputs and bundlesequnce
sed -i '/.*policy_files services\/autorun\/custom-policy-3.cf.*/d' cfbs.json
sed -i '/.*bundles custom_policy_3.*/d' cfbs.json
# Now we need to correct the JSON since the copy is the only build step and the
# last entry of a JSON array must not contain a comma
sed -ri 's/("copy.*)custom-policy-3.cf",/\1custom-policy-3.cf"/g' cfbs.json

git diff

diff --git a/cfbs.json b/cfbs.json
index f57ad96..bae7213 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -49,9 +49,7 @@
       "tags": ["local"],
       "added_by": "cfbs add",
       "steps": [
-        "copy ./services/autorun/custom-policy-3.cf services/cfbs/services/autorun/custom-policy-3.cf",
-        "policy_files services/cfbs/services/autorun/custom-policy-3.cf",
-        "bundles custom_policy_3"
+        "copy ./services/autorun/custom-policy-3.cf services/autorun/custom-policy-3.cf"
       ]
     }
   ],

git add cfbs.json
git commit -m "Fixed custom-policy-3.cf build integration target location, redacted inputs and bundlesequence"

[master ae38672] Fixed custom-policy-3.cf build integration target location, redacted inputs and bundlesequence
 1 file changed, 1 insertion(+), 3 deletions(-)

Finally we can see that the policy set resulting from cfbs build is not missing anything.

cfbs build
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json"


001 masterfiles                           @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded)
002 ./custom-policy-1.cf                  @ local                                    (Copied)
003 ./services/custom-policy-2.cf         @ local                                    (Copied)
004 ./services/autorun/custom-policy-3.cf @ local                                    (Copied)

001 masterfiles                           : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh'
001 masterfiles                           : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y'
001 masterfiles                           : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g''
001 masterfiles                           : run 'find . -name '*.bak' | xargs rm'
001 masterfiles                           : copy './' 'masterfiles/'
002 ./custom-policy-1.cf                  : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf'
002 ./custom-policy-1.cf                  : policy_files 'custom-policy-1.cf'
002 ./custom-policy-1.cf                  : bundles 'custom_policy_1'
003 ./services/custom-policy-2.cf         : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : policy_files 'services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : bundles 'custom_policy_2'
004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf'

Generating tarball...

Build complete, ready to deploy 🐿
-> Directory: out/masterfiles
-> Tarball:   out/masterfiles.tgz

To install on this machine: sudo cfbs install
To deploy on remote hub(s): cf-remote deploy

Check for files in cfbs build result that are not present in tarball initalized policy set

Let’s look at the files in the cfbs built result that are not in the current policy.

diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out"

Only in out/masterfiles/cfe_internal/core/watchdog: README.md
Only in out/masterfiles/cfe_internal/enterprise/ha: ha_info.json
Only in out/masterfiles: .github
Only in out/masterfiles/inventory: README.md
Only in out/masterfiles/lib: README.md
Only in out/masterfiles: LICENSE
Only in out/masterfiles/modules: promises
Only in out/masterfiles: .no-distrib
Only in out/masterfiles/services/autorun: README.md
Only in out/masterfiles/templates: README.md

We have some extra files, let’s take care of those by editing cfbs.json and deleting them in the build steps of the masterfiles module.

# Modify masterfiles build steps to remove files not delivered by release tarball
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/core\/watchdog\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/enterprise\/ha\/ha_info.json",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .github",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete inventory\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete lib\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete LICENSE",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete modules\/promises",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .no-distrib",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete services\/autorun\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete templates\/README.md",\n\1\2/' cfbs.json

After making the edits to the masterfiles module build steps we can review our changes.

git diff

diff --git a/cfbs.json b/cfbs.json
index bae7213..947f990 100644
--- a/cfbs.json
+++ b/cfbs.json
@@ -18,6 +18,16 @@
         "run EXPLICIT_RELEASE=2 ./prepare.sh -y",
         "run grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\\.18\\.3a.*\\b/3.18.3/g'",
         "run find . -name '*.bak' | xargs rm",
+        "delete cfe_internal/core/watchdog/README.md",
+        "delete cfe_internal/enterprise/ha/ha_info.json",
+        "delete .github",
+        "delete inventory/README.md",
+        "delete lib/README.md",
+        "delete LICENSE",
+        "delete modules/promises",
+        "delete .no-distrib",
+        "delete services/autorun/README.md",
+        "delete templates/README.md",
         "copy ./ ./"
       ]
     },

And we can make sure that the result of cfbs build is not missing any files.

cfbs build
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out"


001 masterfiles                           @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded)
002 ./custom-policy-1.cf                  @ local                                    (Copied)
003 ./services/custom-policy-2.cf         @ local                                    (Copied)
004 ./services/autorun/custom-policy-3.cf @ local                                    (Copied)

001 masterfiles                           : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh'
001 masterfiles                           : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y'
001 masterfiles                           : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g''
001 masterfiles                           : run 'find . -name '*.bak' | xargs rm'
001 masterfiles                           : delete 'cfe_internal/core/watchdog/README.md'
001 masterfiles                           : delete 'cfe_internal/enterprise/ha/ha_info.json'
001 masterfiles                           : delete '.github'
001 masterfiles                           : delete 'inventory/README.md'
001 masterfiles                           : delete 'lib/README.md'
001 masterfiles                           : delete 'LICENSE'
001 masterfiles                           : delete 'modules/promises'
001 masterfiles                           : delete '.no-distrib'
001 masterfiles                           : delete 'services/autorun/README.md'
001 masterfiles                           : delete 'templates/README.md'
001 masterfiles                           : copy './' 'masterfiles/'
002 ./custom-policy-1.cf                  : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf'
002 ./custom-policy-1.cf                  : policy_files 'custom-policy-1.cf'
002 ./custom-policy-1.cf                  : bundles 'custom_policy_1'
003 ./services/custom-policy-2.cf         : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : policy_files 'services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : bundles 'custom_policy_2'
004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf'

Generating tarball...

Build complete, ready to deploy 🐿
 -> Directory: out/masterfiles
 -> Tarball:   out/masterfiles.tgz

To install on this machine: sudo cfbs install
To deploy on remote hub(s): cf-remote deploy

Review and resolve remaining differences

Now all that remains is reviewing any other differences between our traditionally managed policy with the result of cfbs build.

diff --exclude=out --recursive --unified . out/masterfiles

Only in .: cfbs.json
diff '--exclude=out' --recursive --unified ./def.json out/masterfiles/def.json
--- ./def.json	2023-06-06 08:09:22.719359634 -0500
+++ out/masterfiles/def.json	2023-06-06 08:13:48.942006067 -0500
@@ -1,11 +1,6 @@
 {
-  "inputs": [ "services/custom-policy-2.cf" ],
-
-  "classes": {
-    "services_autorun":  [ "any::" ]
-  },
-
+  "inputs": ["custom-policy-1.cf", "services/custom-policy-2.cf"],
   "vars": {
-    "control_common_bundlesequence_end": [ "custom_policy_2" ]
+    "control_common_bundlesequence_end": ["custom_policy_1", "custom_policy_2"]
   }
 }
Only in .: .git
Only in .: .gitignore
diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf
--- ./promises.cf	2023-06-06 08:09:17.303386890 -0500
+++ out/masterfiles/promises.cf	2023-06-06 08:13:48.738007113 -0500
@@ -59,7 +59,6 @@
                           main,
                           @(cfengine_enterprise_hub_ha.management_bundles),
                           @(def.bundlesequence_end),
-custom_policy_1,

       };

@@ -86,7 +85,6 @@
                   @(services_autorun.inputs),

                   "services/main.cf",
-"custom-policy-1.cf",
       };

       version => "CFEngine Promises.cf 3.18.3";
diff '--exclude=out' --recursive --unified ./services/main.cf out/masterfiles/services/main.cf
--- ./services/main.cf	2023-06-06 08:09:25.299346646 -0500
+++ out/masterfiles/services/main.cf	2023-06-06 08:13:48.742007092 -0500
@@ -10,6 +10,5 @@
 {
     # Activate your custom policies here
-"Calling custom_policy_2 from main" usebundle => custom_policy_2;

 }

The diff above highlights differences in def.json, .git, .gitignore, promises.cf, and services/main.cf.

The difference in the inputs key and the control_common_bundlesequence_end array in the vars key is expected as we migrated a modification including custom-policy-1.cf in inputs and bundlesquence from promises.cf to def.json via the build steps for that module, removing an unnecessary modification to the vendored files which is also reflected in the differences of promises.cf
We are lacking the class definition for services_autorun to enable the autorun service.
We are lacking the modification to services/main.cf explicitly running the custom_policy_2 bundle.

We can address services/main.cf as we have addressed other files deployed from our traditionally managed policy set, first adding the file.

cfbs add services/main.cf --non-interactive

[master 482f67c] Added module './services/main.cf'
 1 file changed, 21 insertions(+)
Added module: ./services/main.cf

Then adjusting the build steps as necessary.

# Overwrite the vendored services/main
sed -i 's|services/cfbs/services/main.cf|services/main.cf|g' cfbs.json
# Remove build steps for services/main.cf for inputs and bundlesequnce as it's
# already inclided by default in the framework as a vendored file
sed -i '/.*policy_files services\/main.*/d' cfbs.json
sed -i '/.*bundles main.*/d' cfbs.json
# Now we need to correct the JSON since the copy is the only build step and the
# last entry of a JSON array must not contain a comma
sed -ri 's/("copy.*)main.cf",/\1main.cf"/g' cfbs.json

We can address enabling autorun by simply adding in the autorun module.

cfbs add autorun --non-interactive

Added module: autorun
The default commit message is 'Added module 'autorun'' - edit it? [yes/y/NO/n] [master e5f9c9a] Added module 'autorun'
 1 file changed, 13 insertions(+), 5 deletions(-)

This leaves only the expected differences between the traditionally manged policy set and the cfbs built policy set.

cfbs build
diff --exclude=out --recursive --unified . out/masterfiles


001 masterfiles                           @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded)
002 ./custom-policy-1.cf                  @ local                                    (Copied)
003 ./services/custom-policy-2.cf         @ local                                    (Copied)
004 ./services/autorun/custom-policy-3.cf @ local                                    (Copied)
005 ./services/main.cf                    @ local                                    (Copied)
006 autorun                               @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded)

001 masterfiles                           : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh'
001 masterfiles                           : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y'
001 masterfiles                           : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g''
001 masterfiles                           : run 'find . -name '*.bak' | xargs rm'
001 masterfiles                           : delete 'cfe_internal/core/watchdog/README.md'
001 masterfiles                           : delete 'cfe_internal/enterprise/ha/ha_info.json'
001 masterfiles                           : delete '.github'
001 masterfiles                           : delete 'inventory/README.md'
001 masterfiles                           : delete 'lib/README.md'
001 masterfiles                           : delete 'LICENSE'
001 masterfiles                           : delete 'modules/promises'
001 masterfiles                           : delete '.no-distrib'
001 masterfiles                           : delete 'services/autorun/README.md'
001 masterfiles                           : delete 'templates/README.md'
001 masterfiles                           : copy './' 'masterfiles/'
002 ./custom-policy-1.cf                  : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf'
002 ./custom-policy-1.cf                  : policy_files 'custom-policy-1.cf'
002 ./custom-policy-1.cf                  : bundles 'custom_policy_1'
003 ./services/custom-policy-2.cf         : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : policy_files 'services/custom-policy-2.cf'
003 ./services/custom-policy-2.cf         : bundles 'custom_policy_2'
004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf'
005 ./services/main.cf                    : copy './services/main.cf' 'masterfiles/services/main.cf'
006 autorun                               : json 'def.json' 'masterfiles/def.json'

Generating tarball...

Build complete, ready to deploy 🐿
 -> Directory: out/masterfiles
 -> Tarball:   out/masterfiles.tgz

To install on this machine: sudo cfbs install
To deploy on remote hub(s): cf-remote deploy
Only in .: cfbs.json
diff '--exclude=out' --recursive --unified ./def.json out/masterfiles/def.json
--- ./def.json	2023-06-06 08:09:22.719359634 -0500
+++ out/masterfiles/def.json	2023-06-06 08:14:27.481808290 -0500
@@ -1,11 +1,7 @@
 {
-  "inputs": [ "services/custom-policy-2.cf" ],
-
-  "classes": {
-    "services_autorun":  [ "any::" ]
-  },
-
+  "inputs": ["custom-policy-1.cf", "services/custom-policy-2.cf"],
   "vars": {
-    "control_common_bundlesequence_end": [ "custom_policy_2" ]
-  }
+    "control_common_bundlesequence_end": ["custom_policy_1", "custom_policy_2"]
+  },
+  "classes": { "services_autorun": ["any"] }
 }
Only in .: .git
Only in .: .gitignore
diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf
--- ./promises.cf	2023-06-06 08:09:17.303386890 -0500
+++ out/masterfiles/promises.cf	2023-06-06 08:14:27.221809626 -0500
@@ -59,7 +59,6 @@
                           main,
                           @(cfengine_enterprise_hub_ha.management_bundles),
                           @(def.bundlesequence_end),
-custom_policy_1,

       };

@@ -86,7 +85,6 @@
                   @(services_autorun.inputs),

                   "services/main.cf",
-"custom-policy-1.cf",
       };

       version => "CFEngine Promises.cf 3.18.3";

Moving ahead

At this point we have completed the process of bringing our policy set under cfbs management. Look around build.cfengine.com for modules that interest you, try adding and removing modules, you can even try cfbs update masterfiles to see how the policy framework upgrade has been simplified.

If you have ideas for improving the feature set or behavior of cfbs please open a ticket on the issue tracker.

The Masterfiles Policy Framework (MPF) is the default policy set which contains the standard library and provides a framework for custom policy sets to be built on top. https://github.com/cfengine/masterfiles ↩︎
cfbs is the command line tool used to manage a policy set and interact with the CFEngine Build System (build.cfengine.com). It’s available via the Python Package Index (PyPi). https://pypi.org/project/cfbs/ ↩︎
The CFEngine Build System provides tooling and a catalog of modules to simplify the process of building and maintaining an infrastructure with CFEngine. https://build.cfengine.com ↩︎