Change in behavior: findfiles() and directory trailing slashes

CFEngine 3.24.4+, 3.27.1+, and 3.28.0+ include a change to how findfiles() handles trailing slashes on directory paths. This change restores trailing slashes to directory results, but with improved consistency compared to earlier versions. The new behavior ensures that directory paths always include a trailing slash, making them reliably distinguishable from file paths regardless of the glob pattern used.

The behavior changes

CFEngine 3.23.0 and earlier: Pattern-dependent behavior

The presence of a trailing slash in the returned paths depended on whether the glob pattern itself included a trailing slash. If you use findfiles("/path/*/") (with trailing slash in pattern), the results include trailing slashes. If you use findfiles("/path/*") (without trailing slash in pattern), the results do not include trailing slashes.

bundle agent main
{
  vars:
    "testdir" string => "/tmp/testdir";

    # Create a mix of directories and files
    "empty_dirs" slist => { "empty_dir" };
    "dirs_with_files" slist => { "dir_with_files" };
    "nested_dirs" slist => { "nested/level1/level2" };
    "files" slist => { "file1.txt", "file2.txt" };

  files:
    # Create empty directory
    "$(testdir)/$(empty_dirs)/."
      create => "true";

    # Create directory with files
    "$(testdir)/$(dirs_with_files)/."
      create => "true";
    "$(testdir)/$(dirs_with_files)/$(files)"
      create => "true";

    # Create nested directory structure
    "$(testdir)/$(nested_dirs)/."
      create => "true";
    "$(testdir)/$(nested_dirs)/deep_file.txt"
      create => "true";

    # Create some files in the root test directory
    "$(testdir)/root_file.txt"
      create => "true";

  methods:
    "test";
}

bundle agent test
{
  vars:
    # Test two patterns: with and without trailing slash
    "with_slash" slist => findfiles("$(main.testdir)/*/");
    "without_slash" slist => findfiles("$(main.testdir)/*");

  reports:
    "CFEngine: $(sys.cf_version)";
    "=== Pattern: $(main.testdir)/*/ (with trailing slash) ===";
    "  with_slash: $(with_slash)";
    "=== Pattern: $(main.testdir)/* (without trailing slash) ===";
    "  without_slash: $(without_slash)";
}

Running this policy on CFEngine 3.21.8 demonstrates the pattern-dependent behavior:

podman run --rm \
  -v /tmp/test-findfiles-dirs-with-slash.cf:/test.cf:ro,Z \
  cfengine-test:3.21.8 \
  /var/cfengine/bin/cf-agent -KIf /test.cf 2>&1 | grep "^R:"

R: CFEngine: 3.21.8
R: === Pattern: /tmp/testdir/*/ (with trailing slash) ===
R:   with_slash: /tmp/testdir/dir_with_files/
R:   with_slash: /tmp/testdir/empty_dir/
R:   with_slash: /tmp/testdir/nested/
R: === Pattern: /tmp/testdir/* (without trailing slash) ===
R:   without_slash: /tmp/testdir/dir_with_files
R:   without_slash: /tmp/testdir/empty_dir
R:   without_slash: /tmp/testdir/nested
R:   without_slash: /tmp/testdir/root_file.txt

CFEngine 3.24.0 through 3.24.3, and 3.27.0: No trailing slashes

The trailing slash is no longer included in the returned paths, regardless of whether the glob pattern includes a trailing slash. Additionally, both patterns (with and without trailing slash) return identical results - they match both directories and files, but strip trailing slashes from all paths. This breaks policies that depended on the trailing slash being present when using patterns like findfiles("/path/*/"), and also breaks the distinction between patterns with and without trailing slashes.

podman run --rm \
  -v /tmp/test-findfiles-dirs-with-slash.cf:/test.cf:ro,Z \
  cfengine-test:3.24.0 \
  /var/cfengine/bin/cf-agent -KIf /test.cf 2>&1 | grep "^R:"

R: CFEngine: 3.24.0
R: === Pattern: /tmp/testdir/*/ (with trailing slash) ===
R:   with_slash: /tmp/testdir/dir_with_files
R:   with_slash: /tmp/testdir/empty_dir
R:   with_slash: /tmp/testdir/nested
R:   with_slash: /tmp/testdir/root_file.txt
R: === Pattern: /tmp/testdir/* (without trailing slash) ===
R:   without_slash: /tmp/testdir/dir_with_files
R:   without_slash: /tmp/testdir/empty_dir
R:   without_slash: /tmp/testdir/nested
R:   without_slash: /tmp/testdir/root_file.txt

CFEngine 3.24.4+, 3.27.1+, and 3.28.0+: Always trailing slashes for directories

The unintentional regression (tracked as https://northerntech.atlassian.net/browse/CFE-4623) has been fixed in master and backported to the 3.24.x and 3.27.x LTS releases.

However, the new behavior is not identical to pre-3.24.0: instead of the trailing slash depending on the glob pattern, findfiles() now always returns directory paths with trailing slashes and file paths without trailing slashes, regardless of the glob pattern used. Both patterns (with and without trailing slash) now return identical results.

This is arguably more consistent and predictable than the original pattern-dependent behavior, as it makes the presence of a trailing slash a reliable indicator of whether a path is a directory or a file, rather than depending on which glob pattern was used.

# Test with master using a container (adjust based on available nightly builds)
podman run -d --name cfengine-master-test ubuntu:24.04 sleep infinity >/dev/null 2>&1
podman cp /tmp/cfengine-nova_3.28.0a.e56ed82f7~195.ubuntu24_amd64.deb cfengine-master-test:/tmp/ 2>/dev/null
podman exec cfengine-master-test apt-get update -qq >/dev/null 2>&1
podman exec cfengine-master-test apt-get install -y -qq /tmp/cfengine-nova_3.28.0a.e56ed82f7~195.ubuntu24_amd64.deb >/dev/null 2>&1
podman cp /tmp/test-findfiles-dirs-with-slash.cf cfengine-master-test:/tmp/test.cf 2>/dev/null
podman exec cfengine-master-test /var/cfengine/bin/cf-agent -KIf /tmp/test.cf 2>&1 | grep "^R:"
podman rm -f cfengine-master-test >/dev/null 2>&1

R: CFEngine: 3.28.0a.e56ed82f7
R: === Pattern: /tmp/testdir/*/ (with trailing slash) ===
R:   with_slash: /tmp/testdir/dir_with_files/
R:   with_slash: /tmp/testdir/empty_dir/
R:   with_slash: /tmp/testdir/nested/
R:   with_slash: /tmp/testdir/root_file.txt
R: === Pattern: /tmp/testdir/* (without trailing slash) ===
R:   without_slash: /tmp/testdir/dir_with_files/
R:   without_slash: /tmp/testdir/empty_dir/
R:   without_slash: /tmp/testdir/nested/
R:   without_slash: /tmp/testdir/root_file.txt

Impact on policy

The inconsistent handling of trailing slashes across versions creates challenges for policies that rely on string matching or filtering directory paths. This is particularly problematic when using filter() to exclude specific directories.

Consider this example from the MPF modules_presence bundle:

bundle agent modules_presence
{
  vars:
    "_override_dir" string => "$(this.promise_dirname)/../../modules/packages/";
    "_package_paths_tmp" slist => findfiles("$(_override_dir)*");
    "_package_paths" slist => filter("$(_override_dir)vendored", _package_paths_tmp, "false", "true", 999);
}

The filter is designed to exclude the vendored subdirectory from the list of package paths. However, the behavior varies across CFEngine versions:

3.23.0 and earlier: When findfiles() used a pattern with a trailing slash, it returned /path/to/modules/packages/vendored/. The filter pattern $(_override_dir)vendored matched and excluded it correctly.
3.24.0 through 3.27.0: Returns /path/to/modules/packages/vendored without a trailing slash, breaking the filter match. This caused the vendored directory to be included, leading to recursion errors when CFEngine attempted to copy overlapping source and destination directories.
3.24.4+, 3.27.1+, 3.28.0+: Returns /path/to/modules/packages/vendored/ with a trailing slash for directories, restoring correct filtering behavior.

Example version-aware workaround

We we ultimately fixed this differently in the MPF, but wanted to share a pattern leveraging cf_version* functions to support both old and new behavior across CFEngine versions, use version detection to conditionally add the directory separator:

vars:
  "_package_paths"
    with => ifelse(
      or(
        cf_version_between("3.24.0", "3.24.3"),
        cf_version_between("3.25.0", "3.27.0")
      ),
      "$(const.dirsep)",  # Add separator for versions without trailing slash
      ""                   # No separator for versions with trailing slash
    ),
    slist => filter("$(_override_dir)vendored$(with)", _package_paths_tmp, "false", "true", 999);

This pattern uses cf_version_between() to detect versions affected by the behavior change (3.24.0 through 3.27.x) and conditionally adds $(const.dirsep) to the filter pattern only when needed. This ensures the filter works correctly regardless of which CFEngine version is running the policy.

The fix for CFE-4623 has been backported and released in 3.24.4+, 3.27.1+, and 3.28.0+. This workaround continues to work correctly as it only adds the separator for the specific versions that need it (3.24.0-3.24.3, 3.26.0, 3.27.0).

If you have questions or need help, reach out on the mailing list or GitHub discussions. If you have a support contract, feel free to open a ticket in our support system.