3.11.0 Beta ready for testing!

We’re happy to announce that CFEngine 3.11.0 Beta (non-LTS) is now ready. Thanks to everyone for all of the contributions! Please test extensively and submit bug reports. 3.11.x introduces some new features and deprecates some underutilized functionality. Please note that 3.11.0 will be a non-LTS release, which means that it will be maintained only for 6 months from the release date and not supported for CFEngine Enterprise customers (but Enterprise packages will be available). Looking at the CFEngine release schedule, we can see:

• 3.7 LTS is maintained (and supported for Enterprise customers) until July 17th 2018
• 3.10 LTS is maintained (and supported for Enterprise customers) until December 27th 2019
• 3.11 non-LTS will be maintained for 6 months after the release of 3.11.0.

The guest_environments promise type has been removed and the associated libvirt dependency has been dropped. Support for postgres in database type promises has been removed and the associated agent dependency has also been removed. Additionally please note that upgrades are unsupported for the BETA release. The new with attribute that allows the use of function calls on a promiser (CFE-1092). This feature can help in simplifying policy by removing the need for intermediary variables.

Listing 1: Example usage of with attribute.

bundle agent main
{
  vars:
      "todo" slist => { "a 1", "b 2", "c 3" };
      # Here, `with` is the canonified version of $(todo), letting us avoid an

      # intermediate canonification array.
      "$(with)" string => "$(todo)", with => canonify($(todo));

      "complex" data => '
{
  "x": 200,
  "y": [ 1, 2, null, true, false ]
}
';

  reports:
      "For iterable '$(todo)' we created variable '$(with)' and its value is '$(todo)'"
        with => canonify($(todo));

      "We can print a data container compactly without creating a temporary variable: $(with)"
        with => format("%S", complex);

      "We can print a data container fully without creating a temporary variable: $(with)"
        with => storejson(complex);
}

R: For iterable 'a 1' we created variable 'a_1' and its value is 'a 1'
R: For iterable 'b 2' we created variable 'b_2' and its value is 'b 2'
R: For iterable 'c 3' we created variable 'c_3' and its value is 'c 3'
R: We can print a data container compactly without creating a temporary variable: {"x":200,"y":[1,2,null,true,false]}
R: We can print a data container fully without creating a temporary variable: {
  "x": 200,
  "y": [
    1,
    2,
    null,
    true,
    false
  ]
}

System discovery now leverages /etc/os-release and populates sys.os_release with the parsed data. Amazon Linux and CoreOS are now automatically detected during system discovery. Now, when running on Amazon Linux the AmazonLinux class will be defined and sys.flavour will be set to AmazonLinux and when running on CoreOS the coreos class is defined and sys.flavor is set to coreos_VERSION where VERSION is derived from /etc/os-release. For example coreos_1185.3.0 (ENT-3046, CFE-1881). The new functions sysctlvalue() and data_sysctlvalues() for extracting data from /proc/sys are available on Linux systems (CFE-2513). The new readenvfile() function parses environment files ( KEY = VALUE, or KEY = "VALUE" ). The readdata() function can now parse ENV files now detects files ending in .yml as YAML and files ending in .env as ENV (CFE-1881). cf-agent gained the --show-evaluated-vars and --show-evaluated-classes options which works similarly to cf-promises --show-vars and cf-promises --show-classes. The difference being that it is displayed at the end of the agent run and thus has access to classes and variables that were defined during full policy evaluation instead of only those available after system discovery and pre-evaluation. Additionally cf-promises --show-vars and show-classes options can now take an optional regular expression used to filter the output. A new tool cf-net has been added. It is a CLI tool targeted at testing and debugging the network protocol (CFE-2493). In the MPF the default update policy has been made more resilient against partial or corrupt updates. You can now enable additional verification on files transferred during policy updates by defining the cfengine_internal_verify_update_transfers class. Additionally if there is a problem transferring any policy file the cf_promises_validated marker is cleared in order to ensure a full update scan on the next agent execution (CFE-2587). The self upgrade policy has been re-factored into a standalone policy entry (standalone_self_upgrade.cf). Agents run this standalone policy as a sub agent. On systemd managed hosts the run is executed via systemd-run in order to allow for the package upgrade to not be terminated early when the service unit shuts down (ENT-3155). The files_single_copy agent control can now be specified via augments (CFE-2458). The bundlesequence for the main policy (promises.cf) can now be augmented by defining the variable def.control_common_bundlesequence_end. The apt_get package_module now works with both python 2 and 3 (CFE-2445) and is now version aware. For apt-get versions prior to 1.1 --force-yes is used and in versions 1.1 and later is replaced by --allow-downgrades, --allow=remove-essential, and --allow-change-held-packages (CFE-2360). Additionally a bug preventing package updates when multiple sources are available (CFE-2605). Fix apt_get package_module incorrectly using interactive mode when a configuration file is created before a package is installed by always using the --force-confold and --force-confdef options Fix bug which prevented updates from being picked up if there was more than one source listed in the apt upgrade output, without a comma in between (CFE-2605). oslevel inventory has been added for AIX hosts (ENT-3117) and package_module based package inventory is disabled on old distributions with unsupported python versions (e.g. redhat_4) (CFE-2602). A new class client_initated_reporting_enable (CFE-2623) introduced along with the new variable def.mpf_access_rules_collect_calls_admit_ips (CFE-2624), and def.control_hub_exclude_hosts controlling the networks for which the hub should not initiate report collection (CFE-2622) in order to make it easier to enable and configure client initiated reporting in CFEngine Enterprise without modifying policy. New variables def.default_data_select_host_monitoring_include and def.default_data_select_policy_hub_monitoring_include (CFE-2505) were introduced to make it easier to specify which measurements taken by cf-monitord should be collected for central reporting. The default_repository can now be configured by defining the variable def.control_agent_default_repository when the mpf_control_agent_default_repository class is defined. The new variable def.max_client_history_size was introduced to allow controlling the amount of client side reports waiting to be collected into the CFEngine Enterprise hub that are allowed to build up before being purged. Additionally previous_state and untracked_reports are now included in the purge when def.max_client_history_size threshold is reached (ENT-3161). Promises with handles matching the regular expression noreport_.* are no longer collected for central reporting in CFEngine Enterprise by default (ENT-3061). The prunetree agent bundle has been added to the standard library (CFE-2608) complementing the existing prunedir bundle. The ability to select an INI region containing meta characters has been fixed (CFE-2519). For example this fixes the ability to manage hidden smb shares like [hidden$]. select_region INI_section has also been fixed so that the end of a file is sufficient to match the end of the section (CFE-2519). A new shortcut for templates was added (CFE-2582). The set_config_values() and set_line_based() edit_line bundles behavior changed. Now when inserting a configuration option with an empty value a trailing space is no longer inserted (CFE-2466). systemd services are now only “re-started” if the service is already running (CFE-2541). If the service is NOT running, it will simply be started. The paths bundle in the standard library now uses the path to POSIX compliant tools by default. The behavior can be disabled by defining the class disable_mpf_stdlib_use_posix_utils (CFE-2616). Directories for ubuntu_16 and centos_7 are now automatically created on Enterprise hubs (ENT-3136). The default files for which detailed change monitoring is tracked are now only monitored if the files actually exist (ENT-3040). cf-serverd now listens on both IPv4 and IPv6 interfaces by default (CFE-528). A new package_module for zypper was introduced (CFE-2533). In addition to the improvements in core CFEngine Enterprise has seen the addition of a global search box allowing for hosts to be easily located by hostname or IP address (ENT-3059). Host info pages now include a button to copy the url for a direct link to the currently viewed host (ENT-3096) and hostnames shown the host tree summary reports now link to the host specific info page (ENT-3094). The Changes report now defaults to “todays” changes instead of changes from the last 7 days (ENT-2840). Exporting of CSV reports and access to reports shared with users having multiple roles has been fixed (ENT-3099). The query API has been optimized (ENT-2829) and the sorting of columns from an exported inventory report are now retained (ENT-614). A new binary tarball package was introduced making it easier to test and use cfengine on platforms without traditional native package managers. Mission Portal’s application logs have been relocated from /var/cfengine/httpd/htdocs/application/logs to /var/cfengine/httpd/logs/application (ENT-2758). The following dependencies have been upgraded:

• Update redis from 3.2.6 to 3.2.8
• Update git from 2.10.2 to 2.13.0
• Update autoconf from 2.60 to 2.69
• Update postgresql for hub from 9.6.1 to 9.6.2
• Update curl from 7.50.3 to 7.53.1
• Update libiconv from 1.14 to 1.15
• Update pcre from 8.39 to 8.40
• Upgrade to LMDB 0.9.19
• Update postgresql from 9.0.22 to 9.0.23
• Update libcurl to 7.54.0
• Update php from 5.6.29 to 5.6.30
• Update lcov from 1.10 to 1.13