# CFEngine > CFEngine is an open source configuration management and automation framework for managing large-scale IT infrastructure. CFEngine Enterprise adds a web UI, reporting, compliance and support. -------------------------------------------------------------------------------- title: "CFEngine" url: /index.md --------------------------------------------------------------------------------
Trusted by

CFEngine AI: Faster and easier

Ask the agent about your infrastructure and get insights and ideas quickly without writing any code

Choose your model provider

CFEngine AI is configurable, supporting the model / provider you want to use, including the ability to set it up locally / on-prem.

Transparency built in

When the agent shows you data, it's based on generated SQL queries run on your database. You can easily inspect and verify the SQL in the UI. Only the database schema is shared with the AI provider, not the data itself.

Save as report

CFEngine's UI provides a single source of truth to see the most critical infrastructure information at a glance. When interacting with the AI agent, you can easily save the tables you see as reports.

Automate your infrastructure
  • Configuration management
  • CI/CD integration
Learn more
Automate your security
  • Patch management
  • Infrastructure hardening
Learn more
Automate your compliance
  • Prevent configuration drift
  • Compliance reporting
Learn more

Gain control & visibility of your infrastructure.

CFEngine provides DevSecOps teams with a versatile solution to automate day-to-day tasks - freeing up time to focus on more important business priorities. Regardless of scale and complexity, CFEngine continuously keeps your infrastructure secure, compliant & up to date.
Read an overview
A list of items with green checkmarks on a clipboard. Illustration.

CFEngine is available in both open source and commercial options

Find the offering that fits your needs

Community Edition
  • Open Source license
  • Runs on Linux
  • Trusted by the world's largest infrastructures
  • Community support
GitHub
Enterprise Edition
  • Commercial license
  • Runs on Windows & Linux
  • Inventory & compliance reports
  • APIs & web interface
  • Alerts & dashboards
  • Enhanced platform support
Try for free  
View full comparison
Headshot of Mike Svoboda
"Our operations team will make around 5-10 production changes per day. CFEngine provides the automation framework & gives us the ability to continue to scale operationally."
Mike Svoboda
Systems & Automation Engineer, LinkedIn
Read more use cases
One paper airplane highlighted in orange, ahead of others. Illustration.

The leader in infrastructure automation & security.

CFEngine pioneered and revolutionized the way organizations manage their infrastructure. As these environments have matured over the years, CFEngine continues to support the world's most critical infrastructures through improved automation, as well as setting the standard for security & compliance.

About us
-------------------------------------------------------------------------------- title: "CFEngine Champions" url: /cfengine-champions/index.md --------------------------------------------------------------------------------

The CFEngine Champion Program

The CFEngine Champion program rewards the voluntary efforts of individuals who have significantly enhanced the CFEngine Community by promoting CFEngine and its use. The contributions of the CFEngine Community are a vital part of our company's ecosystem.

When deciding who should be named CFEngine Champions we take the following into account:

  • Frequent posts on the CFEngine mailing lists and forums that help users learn and use CFEngine
  • Discussing CFEngine on popular SysAdmin and DevOps websites
  • Writing about the technical and business benefits of CFEngine in blogs and publications
  • Presentations and talks at industry events supporting CFEngine
  • Encouraging users to upgrade to the latest versions of CFEngine
  • Any additional documented activities that support the CFEngine Community

Nominate a CFEngine Champion:

Recognize a CFEngineer by nominating him or her as a Champion. You may nominate as many people as you like using this form.

Previous CFEngine Champions:

Aleksey Tsalolikhin

Aleksey Tsalolikhin (@atsaloli) has been using CFEngine since 2006. He has helped many operations engineers (including Nick Anderson) get started with CFEngine 3 through webinars, articles, and countless user group and conference appearances. Aleksey founded Vertical Sysadmin, Inc. to deliver on-site CFEngine training and consulting. He lives with his wife and daughter in Burbank, California, where, on his balcony, he tends the world's smallest garden.

Nick Anderson

Nick (@cmdln_) lives and works in Lawrence KS. He has been a Systems Administrator since 2002. He has worked in High Performance Computing, E-Commerce, Data Center, and Office environments. Nick is an advocate of open source software and shared knowledge. He helped form the LOPSA Mentorship program in 2010 and has been a contributor and supporter of the CFEngine Design Center. Nick has been using CFEngine 3 since 2010. He chose CFEngine for its strong scientific background, its simple design, small footprint, and distributed nature. When not in front of the computer Nick enjoys photography and spending time with his wife, two children and dog.

Dimitrios Apostolou

Jimis has been contributing to CFEngine core since 2013. He was named a CFEngine champion in 2020 for his voluminous contributions including open-sourcing custom measurement promises. He originally hails from Athens Greece but now lives and works in Oslo, Norway. He enjoys low level systems programming and slurping Ramen.


Brian Bennett

Brian (@bahamat) has been a long time user of CFEngine and can easily be found on the CFEngine mailing list. He is also very active within the Debian team. This year Brian gave us his "Case Study in CFEngine Layout" and his mini zero-to-hero introduction that one of our previous champions Aleksey Tsalolikhin delivered as a training session at LOPSA East. Brian currently works for Amazon.

Jon Henrik Bjørnstad

Jon Henrik co-founded CFEngine in 2008 and worked as Infrastructure Engineer and Product Manager until he left the company to pursue other opportunities in 2013. He holds a MSc. in Network and System Administration and has 10 years of professional experience. Apart from his employment in CFEngine he has been working mostly as a System Administrator within Finance, Oil & Gas and manufacturing. He also co-founded www.cfengineers.net in 2013 which mainly contributes to the community by providing CFEngine packages and tools. He lives in Oslo, Norway with his girlfriend and 2 kids.

Nicolas Charles

Nicolas (@nico_charles) co-founded Normation to develop Rudder, a configuration management tool built on top of cfengine.


Jonathan Clarke

Jonathan (@jooooooon42) is Normation's CTO, and an open source believer. He has been working on Open Source infrastructures since his beginnings as a system administrator. His work is now focused around configuration management tools, in particular Rudder and CFEngine. He is also a contributor to several open source projects including CFEngine, OpenLDAP, LSC and LTB. He lives in and loves Paris, where he enjoys good food, cinema and cycling around the city.

Seva Gluschenko

Seva (@my2coins): is a long time cfengine user and can be frequently found helping people on the mailing list.

Klaus Kämpf

Klaus (@kkaempf) is an active and long-term contributor to various open source projects, from GNU tools to CFEngine. His main professional interests are systems and configuration management and agile principles in software development and project management.

Klaus is currently working as a senior project manager at SUSE Linux, where he packages and maintains CFEngine, among other distribution packages. He still considers himself a CFEngine rookie which helps him keep a fresh look at this tool. His interest in CFEngine is more from a technology integration point of view, rather than a systems administrator pov. Being a father of three boys, Klaus spends his rare spare time with coding, electronics, running, cycling, scuba diving, underwater rugby, and teaching his kids where to grab a hot soldering iron.

Marco Marongiu

Marco (@brontolinux) became passionate about computers as a kid, as soon as he could put his hands on his cousin's Commodore VIC20. He fell in love with cfengine in the early 2000's, after reading about it in a book. He started his career as a System Administrator in a Research Center in 1999, moved to an Italian ISP in 2003, and a Regional Government Agency in 2008. Finally, he left his beloved island of Sardinia, and moved to Norway in 2010, where he currently works as a Senior System Administrator for Opera Software ASA in Oslo. He co-founded one of the first LUGs in Italy (the GULCh), and is the author of several technical articles, a Free Software advocate, a Perl and cfengine enthusiast, and many other things that it would be too much to mention. He has a degree in Mathematics that he keeps hidden in a secret place.

William Orr

Will (@worr) is a relatively new contributor to CFEngine's core, and a drive-by contributor to many other open source projects. He's known within CFEngine largely for taking care of many of the 4k problems, adding global keys to bodies and hanging out on the dev mailing list.

Will works as a Senior Systems Engineer at LinkedIn, largely responsible for developing tooling and automation. At work, he evangelizes use of and contribution back to open source projects. Since starting to work on CFEngine, another LinkedIn employee has contributed to CFEngine and another is getting bootstrapped. Will graduated from Rochester Institute of Technology with a BS in Applied Networking and Systems Administration in 2013. In his spare time, he works on various FOSS projects, hacks on kernels and makes dumb computer jokes on Twitter.

Benoit Peccatte

Benoit (@peckpeck) lives in Paris, he has worked as a sysadmin, developer, and security engineer for several years where he had the opportunity to work with CFEngine v2 and v3. He returned to his preferred field of system administration, where he has been almost exclusively using and hacking open source tools for the past 8 years. He is now employed by Normation to improve Rudder by combining these experiences.

Loïc Pefferkorn

Loic (@LoicPefferkorn) has started using Linux as his main platform in 1999 and quickly became an avid open source contributor, including to Ubuntu for two years as a MOTU. He experienced the pressures of a fast paced, high availability environment and appreciates the peace of mind CFEngine allows him in handling several thousands of servers. Loic has been an invaluable community member. In the last year he has given us cfe-profiler and blog posts that explore the scalability and limitations in CFEngine. In addition to his tooling and writing he can also be found lurking on the cfengine-help mailing list helping other community members. In his words, he is also responsible for "spreading good CFEngine vibes". He likes jogging, reading books, mind hacks and enjoying the pleasant southern France region.

Martin Simons

Martin (@Webhuis) initiated the Debian-team in June 2013 and as a result we are now having an up to date and state of the art Debian Standards compliant CFEngine3 3.6.2. in the Debian Jessie repository.

He did the successful CFEngine Enterprise implementation at DirecTV in the beginning of 2014. He now is the driving force behind The CFEngine Roadshow, the open source show case of a complete CFEngine solution. He is a member of the International CFEngine Community support team. He is happily living with his wife being a proud father of three daughters who left home to study at universities all over the country. The opportunity is to be able to spend more CFEngine time!

Mike Svoboda

Mike (@mikesvoboda): I currently work for LinkedIn as a Senior Unix Engineer in System Operations. Since joining LinkedIn two years ago, we have implemented the Cfengine community edition and currently have deployed version 3.3.1 to production. Our operations team will make around 5-10 production changes a day, but we also have automation within Cfengine that makes hundreds of changes to thousands of machines without any human intervention. We have better control of our infrastructure with Cfengine executing on thousands of machines than when we were administrating just a few servers by hand. LinkedIn hosts the largest professional social network in the world. We are currently the 10th largest website in the US by traffic. Cfengine provides the automation framework and gives us the ability to allow LinkedIn to continue to operationally scale.

Bas van der Vlies

Bas (@tvbas) is passionate about IT automation and he is active on many open source projects such as sali, pxeconfig, pbs_python, and email2trac. He has embraced CFEngine as early as 1996 and, among other things, has contributed to the CFEngine parser. Last year the CFEngine team has invited Bas to Oslo to improve the reporting of syntax errors in the configuration files. He has also been a tremendous help within the Debian team to ensure that the Debian repository gets up to date CFEngine builds. Bas lives in Amsterdam and besides computers he likes traveling, music, sports (cycling, squash, snowboarding and speed skating) and walking on the beach with his dogs.

Neil Watson

Neil (@neil_h_watson) is a long time cfengine user. You may have read his cfengine 2 cookbook or seen him helping other users in IRC or on the mailing list.


Diego Zamboni

Diego (@zzamboni) wrote the Learning CFEngine 3 book and can be frequently found on the mailing list helping others.

Ted Zlatanov

Ever since graduating from Boston University in 1999 majoring in Computer Engineering, Ted (@tzlatanov) has been hands on with CFEngine as a programmer, a system administrator, and lately as a CFEngine employee. Ted is frequently found supporting community users on the help mailing list, and championing fixes for bugs and new features on the bug tracker. In the past year he has contributed many new functions including, reverse, shuffle, sort, strftime, unique, upcase and downcase just to name a few. Emacs lovers can thank Ted for his continued maintenance of the cfengine3-mode. His other interests include travel, soccer, reading voraciously, and most importantly his family. Ted lives in Quincy, Massachusetts.
-------------------------------------------------------------------------------- title: "services-terms-1 - CFEngine" url: /services-terms-1/index.md date: "2021-09-13" --------------------------------------------------------------------------------

NORTHERN.TECH INC
SERVICES AGREEMENT

This Services Agreement for Community Edition (“Agreement”) by and between Customer (as identified in the order Form “Order Form”) and Northern.tech, Inc, (“CFEngine”) is effective as of the Effective Date (as identified on the Order Form). This Agreement consists of the Services Terms and Conditions set forth below and any order form or statement of work (“SOW”) which references this Agreement. This Agreement shall govern Customer’s initial purchase on the Effective Date as well as any future purchases made by Customer which reference this Agreement.

This Agreement permits Customer to purchase any one or more of the following services (“Services”) from CFEngine:


SERVICES TERMS AND CONDITIONS

1. Community Edition Support. If Customer purchases Community Edition Support, CFEngine shall, subject to the terms and conditions of this Agreement, provide to Customer the support services at the level and for the period specified in the applicable Order Form in accordance with the then-current Software Support Policy as defined by Exhibit A at https://cfengine.com/msla/#support, with the modification that the support applies to CFEngine Community Edition, not its Enterprise Edition.

2. Training Courses. If Customer purchases Training Courses, the following terms shall apply:
a. General. CFEngine shall, subject to the terms and conditions of this Agreement, provide to Customer the Training Courses subject to the then-current version of the CFEngine Training Policy as provided to Customer and any applicable CFEngine training materials (“Training Materials”).

b. Use of Materials. Subject to the terms and conditions of this Agreement, CFEngine grants Customer a limited, non-exclusive, non-transferable, non-sublicensable license to use a single copy of the Training Materials for internal educational and training purposes only for each paid Training Course attendee, unless otherwise specified. Customer shall not: (i) copy the Training Materials or any portion thereof; (ii) rent, sublicense or transfer any copies of the Training Materials, or portions thereof, to a third party or allow a third party to use the Training Materials; (iii) modify, decompile, disassemble or reverse engineer any sample object code provided with the Training Materials; (iv) use the Training Materials to develop services or products for sale or include any components of the Training Materials in any product; (v) use any portion of the Training Materials to create a competitive service, product or technology; or (vi) export the Training Materials from the United States.

3. Professional Services. If Customer purchases Professional Services, the following terms shall apply:
a. Professional Services. CFEngine shall provide the number of person-days or hours of the Professional Services described in the applicable Order Form or SOW. Any deliverables provided to Customer as part of the Professional Services shall be deemed “Deliverables”. Customer agrees to provide CFEngine with reasonable access to Customer materials, resources, personnel, equipment or facilities to the extent such access is necessary for the performance of Professional Services. To the extent that Customer does not timely provide the foregoing access required for CFEngine to perform the Professional Services, CFEngine shall be excused from performance until such items are provided. Any references to project timeframes or delivery dates for Deliverables in any Order Form or SOW are good faith estimates only; Customer’s obligation to fees is based solely on number of days of Professional Services delivered.

b. Changes to Scope of Professional Services. If Customer desires to change an Order Form or SOW for Professional Services, Customer will submit a written request to CFEngine detailing the proposed changes. If CFEngine has the resources available to accommodate such changes, CFEngine shall prepare an amendment to the Order Form or SOW detailing the changes, any fee adjustments required as a result of such changes, any adjustments to the delivery schedule required as a result of such changes, and any other necessary adjustments. If the Order Form amendment or SOW amendment is agreeable to Customer, both parties will execute such amendment. If Customer and CFEngine are not able to agree to an adjustment to the Order Form or SOW, it will remain unchanged.

c. Customer Content. Customer hereby grants CFEngine a limited right to use any Customer data or other materials provided to CFEngine in connection with the Professional Services (the “Customer Content”) solely for the purpose of performing the Professional Services for Customer. Customer owns and will retain ownership (including all intellectual property rights) in the Customer Content.

d. Deliverables. Subject to the terms and conditions of this Agreement, Customer shall have a perpetual right to use and distribute any Deliverables for any internal business purpose, but may not distribute the Deliverables to third parties as part of a product or service. Customer’s use of the Deliverables is subject to any additional restrictions or limitations set forth in the applicable Order Form or SOW.

e. Other Agreements. For the avoidance of doubt, CFEngine and Customer may have entered into a separate agreement under which CFEngine licenses certain commercial software, including updates and enhancements to that software, to Customer. The delivery and use of any such software is governed by the terms of such other agreement and not the terms of this Agreement, and even if listed in an Order Form or SOW such software shall not be considered Deliverables hereunder.

4. Special Terms Regarding Third Party Open Source. As may be specified in the applicable policy or Order Form, certain Services may cover third party open source products (“Third Party Open Source”). All obligations of CFEngine with respect to Third Party Open Source shall be on a “reasonable commercial efforts” basis. As a general policy, CFEngine shall make available any bug fixes, error corrections or modifications which CFEngine creates to any Third Party Open Source under the version of the open source license under which such corrected code was initially licensed. For the avoidance of doubt, Customer is solely responsible for determining its right to use Third Party Open Source and for compliance with the related applicable license terms. ANY AND ALL BUG FIXES, ERROR CORRECTIONS OR MODIFICATIONS TO THIRD PARTY OPEN SOURCE CREATED BY CFENGINE ARE PROVIDED “AS IS”. CFENGINE MAKES NO WARRANTIES, CONDITIONS OR UNDERTAKINGS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NONINFRINGEMENT. CFENGINE SHALL HAVE NO LIABILITY WHATSOEVER WITH RESPECT TO THIRD PARTY OPEN SOURCE OR ANY BUG FIXES, ERROR CORRECTIONS OR MODIFICATIONS THERETO PROVIDED BY CFENGINE.

5. Ownership of Materials. Notwithstanding anything to the contrary contained herein, except for the limited license rights expressly provided herein, CFEngine and its suppliers have and will retain all right, title and interest (including, without limitation, all patent, copyright, trademark, trade secret and other intellectual property rights) in and to the Training Materials, Deliverables (excluding any Customer Content as incorporated therein) and any other materials provided by CFEngine to Customer in connection with the Services (collectively, the “Materials”). Customer acknowledges that it is obtaining only a limited license right to the Materials in connection with the Service being provided and that irrespective of any use of the words “purchase,” “sale” or like terms hereunder no ownership rights are being conveyed to Customer under this Agreement or otherwise. All copies, improvements, updates, modifications or enhancements of the Materials (including any modifications to sample files) shall remain the property of CFEngine (including any changes which incorporate any ideas, feedback or suggestions of Customer). All rights not granted by CFEngine are reserved.

6. Payment. Customer will pay CFEngine the amounts and at the times set forth on the applicable Order Form or SOW. Customer agrees to reimburse CFEngine for all reasonable travel, lodging and meal expenses incurred in the course of performing the Services at any location other than CFEngine’s site. CFEngine will invoice Customer for expenses incurred and, at Customer’s request and expense, CFEngine will provide copies of receipts for which charges are incurred. All payments are non-refundable and shall be made in U.S. dollars within thirty (30) days from the date of CFEngine’s invoice in U.S. Dollars. Customer will be responsible for all taxes, withholdings, duties and levies in connection with the Services (excluding taxes based on the net income of CFEngine). Any late payments shall be subject to a service charge equal to 1.5% per month of the amount due or the maximum amount allowed by law, whichever is less.

7. Confidential Information. In the course of their relationship, the parties may disclose to each other information identified at the time of disclosure as confidential (“Confidential Information”), which may include information concerning their respective businesses and technology. The Materials (including Customer feedback regarding the Materials) are Confidential Information of CFEngine. All Confidential Information shall remain the property of the disclosing party, and the receiving party shall have no interest in or rights in such except as expressly set forth in this Agreement. Each party agrees to maintain all Confidential Information of the other party in confidence and to take all reasonable precautions to prevent any unauthorized disclosure of such information. These restrictions on disclosure will not apply to any information which (a) becomes generally known or publicly available through no act or failure to act on the part of the receiving party; (b) is furnished to others by the disclosing party without restriction on disclosure; (c) is known by the receiving party at the time of receiving such information as evidenced by its records; or (d) is furnished to the receiving party by a third party, as a matter of right and without restriction on disclosure.

8. No Warranty. THE SERVICES AND MATERIALS ARE PROVIDED “AS IS,” WITHOUT WARRANTY OF ANY KIND, EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. CUSTOMER MAY HAVE OTHER STATUTORY RIGHTS; HOWEVER, TO THE FULL EXTENT PERMITTED BY LAW, THE DURATION OF STATUTORILY REQUIRED WARRANTIES, IF ANY, SHALL BE LIMITED TO THE MAXIMUM EXTENT PERMITTED BY LAW.

9. Limitation of Liability. IN NO EVENT SHALL CFENGINE BE LIABLE FOR ANY LOSS OF USE, LOST DATA, FAILURE OF SECURITY MECHANISMS, INTERRUPTION OF BUSINESS, OR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING LOST PROFITS), REGARDLESS OF THE FORM OF ACTION, WHETHER IN CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF INFORMED OF THE POSSIBILITY OF SUCH DAMAGES IN ADVANCE. NOTWITHSTANDING ANY OTHER PROVISION OF THIS AGREEMENT, CFENGINE’S ENTIRE LIABILITY TO CUSTOMER UNDER THIS AGREEMENT SHALL NOT EXCEED THE AMOUNT ACTUALLY PAID BY CUSTOMER TO CFENGINE UNDER THIS AGREEMENT. THIS SECTION 9 SHALL NOT APPLY TO CUSTOMER WITH RESPECT TO ANY CLAIM ARISING UNDER SECTION 2(b) (USE OF MATERIALS), SECTION 3(d) (DELIVERABLES) OR SECTION 7 (CONFIDENTIAL INFORMATION). THE PARTIES AGREE THAT THE LIMITATIONS SPECIFIED IN THIS SECTION 9 SHALL SURVIVE AND APPLY EVEN IF ANY LIMITED REMEDY SPECIFIED IN THIS AGREEMENT IS FOUND TO HAVE FAILED ITS ESSENTIAL PURPOSE.

10. Term and Termination. This Agreement is effective as of the Effective Date and shall remain in effect until the later to occur of one (1) year from the Effective Date or the completion of all Services under all Order Forms and SOWs. Either party may terminate this Agreement (including all related Order Forms and SOWs) if the other party: (a) fails to cure any material breach of this Agreement within thirty (30) days after written notice of such breach; (b) ceases operation without a successor; or (c) seeks protection under any bankruptcy, receivership, trust deed, creditors arrangement, composition or comparable proceeding, or if any such proceeding is instituted against such party (and not dismissed within sixty (60) days thereafter). Termination is not an exclusive remedy and the exercise by either party of any remedy under this Agreement will be without prejudice to any other remedies it may have under this Agreement, by law, or otherwise. Sections 5 (Ownership of Materials), 6 (Payment), 7 (Confidential Information), 8 (No Warranty), 9 (Limitation of Liability), 11 (Export Compliance) and 12 (General) will survive any expiration or termination of this Agreement.

11. Export Compliance. Customer acknowledges that the Materials are subject to export restrictions by the United States government and import restrictions by certain foreign governments. Customer shall not and shall not allow any third-party to remove or export from the United States or allow the export or re-export of any part of the Materials or any direct product thereof: (a) into (or to a national or resident of) any embargoed or terrorist-supporting country; (b) to anyone on the U.S. Commerce Department’s Table of Denial Orders or U.S. Treasury Department’s list of Specially Designated Nationals; (c) to any country to which such export or re-export is restricted or prohibited, or as to which the United States government or any agency thereof requires an export license or other governmental approval at the time of export or re-export without first obtaining such license or approval; or (d) otherwise in violation of any export or import restrictions, laws or regulations of any United States or foreign agency or authority. Customer agrees to the foregoing and warrants that it is not located in, under the control of, or a national or resident of any such prohibited country or on any such prohibited party list. The Materials are further restricted from being used for the design or development of nuclear, chemical, or biological weapons or missile technology, or for terrorist activity, without the prior permission of the United States government.

12. General. This Agreement shall be governed by and construed under the laws of the State of California. Any suit or proceeding arising out of or relating to this Agreement shall be commenced exclusively in the state or federal courts in Santa Clara County, California, and each party irrevocably submits to the exclusive jurisdiction and venue of such courts. The United Nations Convention on Contracts for the International Sale of Goods is specifically disclaimed. Customer acknowledges that CFEngine would not make available the Materials except on all the terms included herein, and accordingly CFEngine may terminate this Agreement and seek injunctive relief upon any breach. The parties are independent contractors and no employment, agency, or joint venture is created hereunder. All notices, requests and other communications under this Agreement must be in writing, and must be mailed by registered or certified mail, postage prepaid and return receipt requested, or delivered by hand to the party to whom such notice is required or permitted to be given. This Agreement will bind and inure to the benefit of each party’s permitted successors and assigns. CFEngine may assign this Agreement to any affiliate or in connection with a merger, reorganization, acquisition or other transfer of all or substantially all of CFEngine’s assets or voting securities. Customer may not assign or transfer this Agreement, in whole or in part, without CFEngine’s prior written consent. Any attempt by Customer to transfer or assign this Agreement without such written consent will be null and void. If any provision of this Agreement is held by a court of competent jurisdiction to be unenforceable for any reason, the remaining provisions hereof shall be unaffected and remain in full force and effect. This Agreement is the final, complete and exclusive agreement between the parties relating to the subject matter hereof, and supersedes all prior or contemporaneous understandings and agreements relating to such subject matter, whether oral or written.

-------------------------------------------------------------------------------- title: "Extended Support - CFEngine" url: /extended-support/index.md date: "2021-03-30" --------------------------------------------------------------------------------

Extended Support

Extended Support is offered to CFEngine enterprise customers.

Prior to version 3.7, the support policy covers the prior version of software for 12 months after a major release is made by CFEngine. With the release of 3.7, Long-Term-Support (LTS) was introduced. LTS versions will be released every 18 month and supported for 36 months. Non-LTS versions are released every 6 months. Extended support is offered on 3.6 and LTS CFEngine Enterprise versions. Extended support is only offered on CFEngine Enterprise.

Version Released End of support
3.5 June 12th 2013 June 18th 2015
3.6 June 18th 2014 July 17th 2016
3.7 LTS July 17th 2015 August 31st 2018
3.10 LTS December 28th 2016 December 28th 2019
3.12 LTS June 28th 2018 August 31st 2021
3.15 LTS December 18th 2019 December 31st 2022
3.18 LTS June 24th 2021 August 31st 2024
3.21 LTS December 22nd 2022 December 31st 2025
3.24 LTS July 22nd 2024 July 31st 2027
3.27 LTS January 14th 2026 January 31st 2029

Extended support ensures continued support under current master software license agreement, but with some limitations; Support is given on existing installations, not new. If CFEngine is used in a non-standardized way there might be limited depth-analysis of policies. There will be no new security or bug fixes, and no root-cause analysis of issues. However, under certain circumstances and at an additional cost, patches can be produced. Support will assist with upgrading to new releases, but there is no guarantee on upgrade compatibility. Professional services might be needed for upgrades.

We encourage all customers to stay on LTS-releases, and if you need our assistance, please contact us.

-------------------------------------------------------------------------------- title: "MSLA - CFEngine" url: /msla/index.md date: "2021-03-23" --------------------------------------------------------------------------------

Master Subscription License Agreement

Your use of the CFEngine 3 Enterprise software is subject to the terms and conditions of the CFEngine Master Subscription License Agreement (the "MSLA"). Please review the terms of this MSLA carefully and sign the order form to indicate your acceptance of the MSLA before you start using the software.

1. License Subscriptions.

1.1 Grant of License. Subject to all of the terms and conditions of this Agreement, Northern.tech ("NT") grants to Customer a non-transferable, non-sublicensable, non-exclusive license during the applicable Subscription Term (as defined below) to use the object code form of the software products specified in an Order Form ("Software") internally, but only in accordance with (a) the technical specification documentation generally made available by NT to its customers with regard to the specific Software ("Documentation"), (b) this Agreement and (c) any volume, user, field of use, configuration or other restrictions set forth in the applicable Order Form or this Agreement. "Software" shall also include any Documentation and any Support and Maintenance releases of the same specific Software product provided to Customer under this Agreement.

1.2 Agents. Customer may install Agents on its computers in the number specified in the applicable Order Form. For clarity, usage of Agents in excess of such number shall not be deemed a breach of this Agreement provided that Customer pays the charges for such excess usage described in Section 3.3. "Agent" means each agent of the Software installed on a physical or virtual computer (server or client) of Customer.

1.3 Subscription Terms and Renewals. The Software is licensed under each Order Form on a subscription basis ("Subscription"). The term of any Subscription shall be twelve (12) months commencing on the start date specified on the applicable Order Form (unless otherwise designated in the Order Form) ("Subscription Term"). If no Subscription start date is specified on the applicable Order Form, the start date shall be the date when NT delivers to Customer the license key for the Software. Unless terminated earlier in accordance with Section 4, each Subscription Term will automatically renew upon expiration of the initial Subscription Term for additional successive one (1) year terms unless either party gives the other prior written notice of cancellation at least thirty (30) days prior to expiration of the then-current term. Unless otherwise specified on the Order Form, the rates for any Subscription Term renewals shall be NT's then-current Subscription rates.

1.4 Installation and Copies. Customer may copy and install on Customer's computers for use only by Customer's employees, Affiliates and Contractors (each as defined in Section 1.5 below) as many copies of the Software as is designated on the applicable Order Form. Customer may also make a reasonable number of copies of the Software for back-up and archival purposes.

1.5 Use by Affiliates and Contractors. Subject to the terms and conditions of this Agreement, Customer's Affiliates and Contractors may use the licenses granted to Customer, provided that (a) such use is only for Customer's or such Affiliate's benefit, (b) Customer agrees to remain responsible for each such Affiliate's and Contractor's compliance with the terms and conditions of this Agreement and (c) upon request Customer will identify each such Affiliate and Contractor. Use of the Software by the Affiliates, Contractors and Customer in the aggregate must be within the restrictions in the applicable Order Form. "Affiliate" means any entity under the control of Customer where "control" means ownership of or the right to control greater than 50% of the voting securities of such entity. "Contractor" means any third party engaged by Customer to perform services on behalf of Customer.

1.6 License Restrictions. Customer shall not (and shall not allow any third party to):
(a) decompile, disassemble, or otherwise reverse engineer the Software or attempt to reconstruct or discover any source code, underlying ideas, algorithms, file formats or programming interfaces of the Software by any means whatsoever (except and only to the extent that applicable law prohibits or restricts reverse engineering restrictions, and then only with prior written notice to NT);
(b) distribute, sell, sublicense, rent, lease or use the Software (or any portion thereof) for time sharing, hosting, service provider or like purposes;
(c) remove any product identification, proprietary, copyright or other notices contained in the Software;
(d) modify any part of the Software, create a derivative work of any part of the Software, or incorporate the Software into or with other software, except to the extent expressly authorized in writing by NT; or
(e) publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software.

2. Ownership.

Notwithstanding anything to the contrary contained herein, except for the limited license rights expressly provided herein, NT and its suppliers have and will retain all rights, title and interest in and to the Software (including, without limitation, all patent, copyright, trademark, trade secret and other intellectual property rights) and all copies, modifications and derivative works thereof. Customer acknowledges that it is obtaining only a limited license right to the Software and that irrespective of any use of the words "purchase", "sale" or like terms hereunder no ownership rights are being conveyed to Customer under this Agreement or otherwise.

3. Payment and Delivery.

3.1 Payment. All payments are non-refundable (except as expressly set forth in this Agreement) and shall be made in U.S. dollars. Unless otherwise specified on the applicable Order Form, all Professional Services fees, training fees, and setup fees are due within thirty (30) days of NT's invoice. Subscription fees are payable as follows: (a) for each initial Subscription Term under an Order Form, within thirty (30) days of the effective date of such Order Form, unless otherwise specified therein and (b) for each renewal Subscription Term, within thirty (30) days after NT's invoice for the renewal Subscription fees. NT will send such renewal invoices no sooner than thirty (30) days prior to the expiration of the then-current Subscription Term. Customer shall be responsible for all taxes, withholdings, duties and levies arising from the order (excluding taxes based on the net income of NT). Any late payments shall be subject to a service charge equal to 1.5% per month of the amount due or the maximum amount allowed by law, whichever is less.

3.2 Procurement Agent. Customer hereby notifies NT that Customer's Procurement Agent is authorized to place Order Forms, receive invoices and make payments to NT on Customer's behalf. NT is entitled to rely and act upon any Order Form submitted by the Procurement Agent and Customer shall remain liable for any invoices that remain unpaid by its Procurement Agent or which are paid late. If any refunds are required to be provided to Customer under this Agreement, NT will refund amounts paid by the Procurement Agent to the Procurement Agent, and the Procurement Agent will be responsible for refunding the corresponding amount to Customer. Customer acknowledges and agrees that it is granted permission to use its Procurement Agent under this Agreement solely as a matter of convenience in connection with Customer's order management process. NT shall have no liability or obligation to the Procurement Agent and the Procurement Agent obtains no rights under or in connection with this Agreement, including, without limitation, any license or other right to use the Software.

3.3 Excess Usage of Agents. For each Subscription Term, on the first business day following (a) the anniversary of the start date of the Subscription Term and (b) the date that is six months after such anniversary, Customer shall run a License Status Report using the Software and shall promptly (but no less than 14 days after such business day) email the results to NT. If the report reveals that Customer has exceeded the number of Agents set forth on the Order Form during the report period based on NT's standard policies for calculating Agent usage, then Customer shall pay for the excess usage and for any ongoing excess usage in the next reporting period, as invoiced by NT on a subsequent invoice.

3.4 Delivery. All Software and Documentation shall be delivered by electronic means unless otherwise specified on the applicable Order Form.

4. Term of Agreement.

4.1 Term. This Agreement is effective as of the Effective Date and expires on the day that the Subscription Terms for all Software licensed hereunder have expired. Either party may terminate this Agreement (including all related Order Forms) if the other party: (a) fails to cure any material breach of this Agreement within thirty (30) days after written notice of such breach; (b) ceases operation without a successor; or (c) seeks protection under any bankruptcy, receivership, trust deed, creditors arrangement, composition or comparable proceeding, or if any such proceeding is instituted against such party (and not dismissed within sixty (60) days thereafter). Termination is not an exclusive remedy and the exercise by either party of any remedy under this Agreement will be without prejudice to any other remedies it may have under this Agreement, by law, or otherwise.

4.2 Termination. Upon any expiration or termination of this Agreement, Customer shall cease any and all use of any Software and destroy all copies thereof and so certify to NT in writing.

4.3 Survival. Sections 1.6 (License Restrictions), 2 (Ownership), 3 (Payment and Delivery), 4 (Term of Agreement), 5.3 (Disclaimer), 8 (Limitation of Remedies and Damages), 10 (Confidential Information), 12 (General), and Customer's right to Work Product and ownership of Customer Content described in Section 7 shall survive any termination or expiration of this Agreement.

5. Limited Warranty and Disclaimer.

5.1 Limited Warranty. NT warrants to Customer that for a period of ninety (90) days from the Effective Date (the "Warranty Period"), the Software shall operate in substantial conformity with the Documentation. NT does not warrant that Customer's use of the Software will be uninterrupted or error-free, will not result in data loss, or that any security mechanisms implemented by the Software will not have inherent limitations. NT's sole liability (and Customer's exclusive remedy) for any breach of this warranty shall be, in NT's sole discretion, to use commercially reasonable efforts to provide Customer with an error-correction or work-around which corrects the reported non-conformity, to replace the non-conforming Software with conforming Software, or if NT determines such remedies to be impracticable within a reasonable period of time, to terminate the applicable Subscription Term and refund the Subscription fee paid for the non-conforming Software. NT shall have no obligation with respect to a warranty claim unless notified of such claim within the Warranty Period.

5.2 Exclusions. The above warranty shall not apply: (a) if the Software is used with hardware or software not specified in the Documentation; (b) if any modifications are made to the Software by Customer or any third party; (c) to defects in the Software due to accident, abuse or improper use by Customer; or (d) to items provided on a no charge or evaluation basis.

5.3 Disclaimer. THIS SECTION 5 CONTAINS A LIMITED WARRANTY AND EXCEPT AS EXPRESSLY SET FORTH IN THIS SECTION 5 THE SOFTWARE AND ALL SERVICES ARE PROVIDED "AS IS". NEITHER NT NOR ANY OF ITS SUPPLIERS MAKES ANY OTHER WARRANTIES, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, TITLE, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. CUSTOMER MAY HAVE OTHER STATUTORY RIGHTS. HOWEVER, TO THE FULL EXTENT PERMITTED BY LAW, THE DURATION OF STATUTORILY REQUIRED WARRANTIES, IF ANY, SHALL BE LIMITED TO THE LIMITED WARRANTY PERIOD.

6. Support & Maintenance.

NT shall provide the support and maintenance services set forth on Exhibit A ("Support and Maintenance") for each Subscription Term.

7. Professional Services.

7.1 Professional Services. NT shall provide the number of person-days of professional consulting services ("Professional Services") purchased in the applicable Order Form. The parties acknowledge that the scope of the Professional Services provided hereunder consists solely of either or both of: (a) assistance with Software installation, deployment, and usage; or (b) development or delivery of additional related NT copyrighted software or code. NT shall retain all right, title and interest in and to any such work product, code or software and any derivative, enhancement or modification thereof created by NT (or its agents) ("Work Product"). Unless otherwise specified in the applicable SOW, Customer shall have a perpetual right to use and distribute any Work Product delivered as part of the Professional Services for any internal business purpose, but may not distribute the Work Product to third parties as part of a product or service. Professional Services may be ordered by Customer pursuant to a Statement of Work ("SOW") describing the work to be performed, fees and any applicable milestones, dependencies and other technical specifications or related information. Each SOW must be signed by both parties before NT shall commence work under such SOW. If the parties do not execute a separate Statement of Work, the Services shall be provided as stated on the Order Form. Customer will reimburse NT for reasonable travel and lodging expenses as incurred.

7.2 Customer Content. Customer hereby grants NT a limited right to use any Customer data or other materials provided to NT in connection with the Professional Services (the "Customer Content") solely for the purpose of performing the Professional Services for Customer. Customer owns and will retain ownership (including all intellectual property rights) in the Customer Content.

8. Limitation of Remedies and Damages.

8.1 NEITHER PARTY SHALL BE LIABLE FOR ANY LOSS OF USE, LOST DATA, FAILURE OF SECURITY MECHANISMS, INTERRUPTION OF BUSINESS, OR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING LOST PROFITS), REGARDLESS OF THE FORM OF ACTION, WHETHER IN CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF INFORMED OF THE POSSIBILITY OF SUCH DAMAGES IN ADVANCE.

8.2 NOTWITHSTANDING ANY OTHER PROVISION OF THIS AGREEMENT, NT'S AND ITS SUPPLIERS' ENTIRE LIABILITY TO CUSTOMER SHALL NOT EXCEED THE AMOUNT ACTUALLY PAID BY CUSTOMER TO NT UNDER THIS AGREEMENT.

8.3 THIS SECTION 8 SHALL NOT APPLY TO CUSTOMER WITH RESPECT TO ANY CLAIM ARISING UNDER THE SECTIONS TITLED "GRANT OF LICENSE," "LICENSE RESTRICTIONS" OR "CONFIDENTIAL INFORMATION".

8.4 The parties agree that the limitations specified in this Section 8 will survive and apply even if any limited remedy specified in this Agreement is found to have failed of its essential purpose.

9. Indemnification.

NT shall defend, indemnify and hold harmless Customer from and against any claim of infringement of a U.S. patent, U.S. copyright, or U.S. trademark asserted against Customer by a third party based upon Customer's use of the Software in accordance with the terms of this Agreement, provided that NT shall have received from Customer: (i) prompt written notice of such claim (but in any event notice in sufficient time for NT to respond without prejudice); (ii) the exclusive right to control and direct the investigation, defense, and settlement (if applicable) of such claim; and (iii) all reasonable necessary cooperation of Customer. If Customer's use of any of the Software is, or in NT's opinion is likely to be, enjoined due to the type of infringement specified above, or if required by settlement, NT may, in its sole discretion: (a) substitute for the Software substantially functionally similar programs and documentation; (b) procure for Customer the right to continue using the Software; or if (a) and (b) are commercially impracticable, (c) terminate the Agreement and refund to Customer the pro-rata portion of the Subscription fees paid by Customer allocable to the unused remainder of the Subscription Term. The foregoing indemnification obligation of NT shall not apply: (1) if the Software is modified by any person other than NT, but solely to the extent the alleged infringement is caused by such modification; (2) if the Software is combined with other non-NT products or process not authorized by NT, but solely to the extent the alleged infringement is caused by such combination; (3) to any unauthorized use of the Software; (4) to any unsupported release of the Software; or (5) to any Open Source Software or other third-party code contained within the Software. THIS SECTION 9 SETS FORTH NT'S AND ITS SUPPLIERS' SOLE LIABILITY AND CUSTOMER'S SOLE AND EXCLUSIVE REMEDY WITH RESPECT TO ANY CLAIM OF INTELLECTUAL PROPERTY INFRINGEMENT.

10. Confidential Information.

Each party agrees that all code, inventions, know-how, business, technical and financial information it obtains ("Receiving Party") from the disclosing party ("Disclosing Party") constitute the confidential property of the Disclosing Party ("Confidential Information"), provided that it is identified as confidential at the time of disclosure or should be reasonably known by the Receiving Party to be Confidential Information due to the nature of the information disclosed and the circumstances surrounding the disclosure. Any software, documentation or technical information provided by NT (or its agents), performance information relating to the Software, and the terms of this Agreement shall be deemed Confidential Information of NT without any marking or further designation. Except as expressly authorized herein, the Receiving Party will hold in confidence and not use or disclose any Confidential Information. Without limiting the foregoing, Customer shall ensure that its Procurement Agent is subject to written confidentiality terms that are no less protective of NT than the terms of this Agreement. The Receiving Party's nondisclosure obligation shall not apply to information which the Receiving Party can document: (a) was rightfully in its possession or known to it prior to receipt of the Confidential Information; (b) is or has become public knowledge through no fault of the Receiving Party; (c) is rightfully obtained by the Receiving Party from a third party without breach of any confidentiality obligation; (d) is independently developed by employees of the Receiving Party who had no access to such information; or (e) is required to be disclosed pursuant to a regulation, law or court order (but only to the minimum extent required to comply with such regulation or order and with advance notice to the Disclosing Party). The Receiving Party acknowledges that disclosure of Confidential Information would cause substantial harm for which damages alone would not be a sufficient remedy, and therefore that upon any such disclosure by the Receiving Party the Disclosing Party shall be entitled to appropriate equitable relief in addition to whatever other remedies it might have at law. For clarity, the above confidentiality restrictions do not apply to Customer's use of Work Product.

11. Customer Acknowledgement.

Customer agrees that NT may publicly disclose Customer as a customer of NT.

12. General.

12.1 Assignment. This Agreement will bind and inure to the benefit of each party's permitted successors and assigns. Neither party shall assign this Agreement (or any part thereof) without the advance written consent of the other party, except that either party may assign this Agreement in connection with a merger, reorganization, acquisition or other transfer of all or substantially all of such party's assets or voting securities. Any attempt to transfer or assign this Agreement except as expressly authorized under this Section 12.1 will be null and void.

12.2 Severability. If any provision of this Agreement shall be adjudged by any court of competent jurisdiction to be unenforceable or invalid, that provision shall be limited to the minimum extent necessary so that this Agreement shall otherwise remain in effect.

12.3 Governing Law; Jurisdiction and Venue. This Agreement shall be governed by the laws of the State of California and the United States without regard to conflicts of laws provisions thereof, and without regard to the United Nations Convention on the International Sale of Goods. The jurisdiction and venue for actions related to the subject matter hereof shall be the California state and United States federal courts located in Santa Clara County California, and both parties hereby submit to the personal jurisdiction of such courts.

12.4 Attorneys' Fees and Costs. The prevailing party in any action to enforce this Agreement will be entitled to recover its attorneys' fees and costs in connection with such action.

12.5 Notices and Reports. Any notice or report hereunder shall be in writing to the notice address set forth above and shall be deemed given: (a) upon receipt if by personal delivery; (b) upon receipt if sent by certified or registered U.S. mail (return receipt requested); or (c) one day after it is sent if by next day delivery by a major commercial delivery service.

12.6 Amendments; Waivers. No supplement, modification, or amendment of this Agreement shall be binding, unless executed in writing by a duly authorized representative of each party to this Agreement. No waiver will be implied from conduct or failure to enforce or exercise rights under this Agreement, nor will any waiver be effective unless in a writing signed by a duly authorized representative on behalf of the party claimed to have waived. No provision of any purchase order or other business form employed by Customer will supersede the terms and conditions of this Agreement, and any such document relating to this Agreement shall be for administrative purposes only and shall have no legal effect.

12.7 Entire Agreement. This Agreement is the complete and exclusive statement of the mutual understanding of the parties and supersedes and cancels all previous written and oral agreements and communications relating to the subject matter of this Agreement. Customer acknowledges that NT offers subscription-based products and that, in order to provide improved customer experience, NT may make changes to NT type products (including the CFEngine Software) or Documentation. In such event, NT will update the Documentation accordingly.

12.8 Audit Rights. Upon NT's written request, Customer shall certify in a signed writing that Customer's use of the Software is in full compliance with the terms of this Agreement (including any copy and user limitations). With prior reasonable notice of at least 10 days, NT may audit the copies of the Software in use by Customer provided such audit is during regular business hours; Customer is responsible for such audit costs only in the event the audit reveals that Customer's use is not in accordance with the licensed scope of use.

12.9 Independent Contractors. The parties to this Agreement are independent contractors. There is no relationship of partnership, joint venture, employment, franchise or agency created hereby between the parties. Neither party will have the power to bind the other or incur obligations on the other party's behalf without the other party's prior written consent.

12.10 Force Majeure. Neither party shall be liable to the other for any delay or failure to perform any obligation under this Agreement (except for a failure to pay fees) if the delay or failure is due to events which are beyond the reasonable control of such party, including but not limited to any strike, blockade, war, act of terrorism, riot, natural disaster, failure or diminishment of power or of telecommunications or data networks or services, or refusal of approval or a license by a government agency.

12.11 Government End-Users. The Software is commercial computer software. If the user or licensee of the Software is an agency, department, or other entity of the United States Government, the use, duplication, reproduction, release, modification, disclosure, or transfer of the Software, or any related documentation of any kind, including technical data and manuals, is restricted by a license agreement or by the terms of this Agreement in accordance with Federal Acquisition Regulation 12.212 for civilian purposes and Defense Federal Acquisition Regulation Supplement 227.7202 for military purposes. The Software was developed fully at private expense. All other use is prohibited.

12.12 Export Compliance. Customer acknowledges that the Software is subject to export restrictions by the United States government and import restrictions by certain foreign governments. Customer shall not and shall not allow any third-party to remove or export from the United States or allow the export or re-export of any part of the Software or any direct product thereof: (a) into (or to a national or resident of) any embargoed or terrorist-supporting country; (b) to anyone on the U.S. Commerce Department's Table of Denial Orders or U.S. Treasury Department's list of Specially Designated Nationals; (c) to any country to which such export or re-export is restricted or prohibited, or as to which the United States government or any agency thereof requires an export license or other governmental approval at the time of export or re-export without first obtaining such license or approval; or (d) otherwise in violation of any export or import restrictions, laws or regulations of any United States or foreign agency or authority. Customer agrees to the foregoing and warrants that it is not located in, under the control of, or a national or resident of any such prohibited country or on any such prohibited party list. The Software is further restricted from being used for the design or development of nuclear, chemical, or biological weapons or missile technology, or for terrorist activity, without the prior permission of the United States government.

12.13 Third-Party Code. The Software may contain or be provided with components subject to the terms and conditions of third party "open source" software licenses ("Open Source Software"). Open Source Software may be identified in the Documentation, or NT shall provide a list of the Open Source Software for a particular version of the Software to Customer upon Customer's written request. To the extent required by the license that accompanies the Open Source Software, the terms of such license will apply in lieu of the terms of this Agreement with respect to such Open Source Software, including, without limitation, any provisions governing access to source code, modification or reverse engineering. Customer agrees that they have read and understood the Third Party Open Source Software disclosures in the "Legal and licenses" segment of CFEngine's documentation, located at: https://docs.cfengine.com/licenses.

12.14 Community Edition. The Software includes certain code that is the same as code in the CFEngine Community Edition Software ("CCES"), which NT also licenses separately under the GNU General Public License ("GPL"). Notwithstanding Section 12.13 (Third-Party-Code), for purposes of this Agreement, any such code in the Software is licensed under the terms and conditions contained herein, not the GPL.

EXHIBIT A

COMMERCIAL SOFTWARE SUPPORT POLICY

I. Overview

This Commercial Software Support Policy ("Support Policy") describes the policies and procedures under which NT provides support and maintenance services ("Support") for its proprietary commercial software product CFEngine ("Software").
Support is provided for the Software pursuant to the separate Master Software License Subscription Agreement under which Customer has purchased Support and is subject to the terms and conditions of that separate agreement and the terms of this Support Policy. Support is provided for the Subscription Term specified in the Order Form or Master Software License Subscription Agreement, or for the period otherwise specified in the Master Software License Subscription Agreement.
Support is provided through Software's online web-based support portal (the "Support Portal"), and if indicated on the Support Matrix (see Schedule 1, attached), also by telephone.

This Support Policy sets forth expectations for Support between the Customer organization and NT's Customer Support organization, including:
a. who is authorized to submit issues
b. how to submit issues
c. what types of issues are supported
d. how and when NT resolves and closes reported issues.

II. Scope of Support.

What Support Includes.
NT shall provide Customer with Support consisting of the following: (a) the number of web-based Incident (as defined below) submissions up to the number of designated Contacts (as defined below) of Customer as specified on the Support Matrix; (b) LTS Releases (as defined below) of the Software; and (c) the provision of expert level guidance and troubleshooting to Customer in connection with questions and issues arising from the following Customer activities with respect to the Software:
Installation and Downloads: Support for installation includes providing guidance and troubleshooting in connection with Customer's downloading and installing of the Software.
Basic Configuration Issues: Support for configuration includes troubleshooting Customer's configuration settings for existing installations on Supported Platforms (as defined below) to ensure proper operation and connectivity.
Usage Issues: Software experts will answer your "how to" questions related to standard and intended product usage.
Assistance with Policies. Software experts will assist you with a specific issue that you have encountered when developing policies to be used in the Software. General policy development issues fall outside the scope of Support.
Assistance with New Releases. Software experts will assist you with specific issues encountered when you update from one supported release of the Software to another supported release. Customer should update to the latest release of the Software to ensure that they have the latest patches and features. Assistance with updates from unsupported releases falls outside the scope of Support.

What Support Excludes.
The following are excluded from NT's Support obligations: (a) Software that is used on or in conjunction with hardware or software other than as specified in the applicable Documentation; (b) altered or modified Software, unless altered or modified by NT; (c) defects in the Software due to accident, hardware malfunction, abuse or improper use; (d) Support for unsupported releases or updates from unsupported releases; (e) "beta" releases, evaluation software or other software provided at no charge; (f) any Software sold separately by NT, including, without limitation, consulting code, unless generally made available to NT's subscription customers at no additional charge for the Software; (g) training, customization, integration and any issues arising from non-standard usage of the Software; (h) any on-site services or remote access services (unless NT requests remote access to assist NT in understanding an issue); (i) problems caused by version mismatch (as described in Section II.E below); (j) problems with customers' existing CFEngine policy syntax due to the change of software instruction related to the version upgrade (as described in Section II.F below); and (k) problems with customers' reporting methods under earlier versions (as described in Section II.G below.). NT separately offers a range of fee-based professional services to address issues related to:
a. Product training
b. Product customizations
c. Custom integrations
d. Performance tuning
e. Custom build strategies
f. Process improvements
g. Policy development issues that fall outside the scope of Support
h. Any other Incidents that we advise you fall outside the scope of Support

Software Versions Covered.
Supported Versions: LTS (abbreviation for "Long Term Support")

Releases: means generally any release announced on Software's website which is designated as a LTS and can include commercially released major new releases, modifications or enhancements to the same Software. The Support Policy covers an LTS release for 36 months. Any new version of the same Software as designated by a change in the number(s) to the far right decimal in version number is considered part of the LTS release. For instance if version 3.7.0 is released July 1st 2015, it will be supported until July 1st 2018, and any new versions like 3.7.1, 3.7.2, etc, will be supported until this July 1st 2018 date.

Non-LTS releases: means commercially released major new releases, modifications or enhancements to the same Software which are not declared as a LTS release on 's webpage. Non-LTS releases are not covered by "The Support Policy."
Platforms Supported: NT supports use of the Software only on the platforms specified in the Support Portal or in the Release Notes of a given release (the "Supported Platforms"). NT may make changes to the list of Supported Platforms from time to time when making new releases of the Software available.
Version Mismatch.

To avoid problems with the Software, the CFEngine hub and the Agents connecting to it must be running on the same version. Support is not provided for problems caused by version mismatch. Keep in mind that during upgrades, hubs are upgraded before Agents, so there may be a temporary version mismatch while the Software is being upgraded. During that time, the full features of the Software may not be available.

Changes of CFEngine policy syntax.
LTS releases of Software may create changes to the CFEngine policy syntax to agents, which will impact how customers' existing CFEngine policies are supported. The Software Support Policy will not apply to changes required in customers' CFEngine policy caused by modifications in CFEngine policy syntax based on major revisions. Customers are responsible for their CFEngine policy.
Changes to Reporting.

LTS releases of Software may impact customer's preexisting reporting methods, queries and calls. These potential changes are not covered by the Support Policy.

III. Incident Submission and Resolution

Customer shall obtain Support by reporting individual issues to NT. Each individual issue reported to NT shall be tracked from initial report through final resolution (each such issue, an "Incident"). Customer is entitled to the number of Incidents specified in the Support Matrix.
Submitting Incidents

Who May Submit Incidents.
Support is intended to provide assistance to individuals for issues and questions beyond what is covered in documentation and introductory material. Customers are expected to make every effort to ensure that the individuals that are designated as authorized contacts are qualified to support the Customer teams internally. To be qualified, these individuals should know the internal build systems, tools, policies, and practices in use by the Customer, and they should also be proficient users of the Software. Each such qualified contact is a "Contact".

Customer shall be entitled to designate the number of Contacts specified in the Support Matrix as authorized to submit support Incidents. Customer is responsible for designating at least one authorized Contact at time of purchase. That individual may submit change requests to the list of authorized support Contacts in writing through the channel(s) specified for the plan selected.

How to Submit Incidents.
Incidents are to be submitted to NT by a Contact through the communication channel(s) (Support Portal and where applicable, by phone) specified for the applicable Support plan in the Support Matrix. The Support Matrix specifies which communication channels are available for each plan offered.

How to Report an Incident.
In order to expedite the resolution of Incidents, NT expects that Customer will make every attempt possible to:
a. Verify that the Incident is reproducible on the Supported Platforms for the Software (as applicable).
b. Provide information necessary to help NT track, prioritize, reproduce or investigate the Incident, such as: Customer name and organization.
c. A full description of the issue and expected results.
d. Category of issues: general question, defect, enhancement request, etc.
e. Steps to reproduce the issue and relevant data.
f. Any applicable log files or console output.
g. Exact wording of all issue related error messages.
h. Any special circumstances surrounding the discovery of the issue, i.e. first occurrence or occurred after what specific event, Customer's business impact of problem and suggested priority for resolution.
i. Identifying issue number in any ongoing communications with NT on an existing issue.

Support Response and Incident Resolution
Software Incident Response.
For each Incident reported by Customer in accordance with these procedures, NT shall:
a. Confirm receipt of the reported Incident within the acknowledgement time specified in the Support Matrix.
b. Set a Priority Level for the Incident in accordance with the terms below.
c. Begin responding to the Incident within the response time specified in the Support Matrix.
d. Analyze the Incident and, as applicable, verify the existence of the problem(s) resulting in the Incident, which may include requesting that Customer provide additional information, logs and re-execution of commands to help identify the root cause and dependencies of the reported issue.
e. Give Customer direction and assistance in resolving the Incident.
f. Keep a record of ongoing communications with Customer.
g. Use reasonable commercial efforts to resolve the Incident in accordance with the target resolution times set forth in the

Support Matrix.

Priority Levels.
NT will prioritize Incidents according to the following criteria:
P1 = Issues which make the Software completely inaccessible or the majority of its functionality unusable for Customer.
P2= means an issue that significantly degrades performance of the Software or materially restricts Customer's use of or the functionality of the Software.
P3= means an issue that causes only a minor impact on Customer's use of the Software.
P4= means any other request for guidance or information.

Resolution and Closure of Incidents.
Incidents shall be closed in the following manner:

For solvable issues, depending on the nature of the issue, the resolution may take the form of an explanation, recommendation, usage instructions, workaround instructions, or advising Customer of an available software fix.

For unsolvable issues, NT undertakes to provide an explanation why the issue is not solvable, and provide the reporting Customer suggestions for a work-around or other ways to mitigate the issue.

In the event that custom or unsupported plug-ins or modules are used, NT may ask, in the course of attempting to resolve the issue, that the Customer remove any unsupported plug-ins or modules. If the problem disappears upon removal of an unsupported plug-in or module, then NT may consider the issue to be resolved.

For issues outside of scope of Support, NT may also close issues by identifying the Incident as outside the scope of the Support or arising from a version, platform or usage case which is excluded from the Support Policy.

Schedule 1

Software Support Matrix

POLICY TERMS
Coverage Hours Business Hours*
Supported Channels Support Portal
Supported Contacts 1
Max. # of Incidents 15
TARGET RESPONSE TIME
Acknowledgement Time 2 Hours for all Incidents
Response Time P1 - 12 hours
P2 - 48 hours
P3 - 72 hours
Target Resolution Time P1 - 48 hours
P2 - 1 week
P3 - Next LTS Release

SUBSCRIPTION TERM
Term The same period as the then-current Software Subscription Term

*Notes:

(1) For customers who have an agreement with NT. NT business hours are 9 a.m. to 5p.m. Eastern Standard Time Monday thru Friday except NT holidays. For customers who have an agreement with NT.tech AS, NT business hours are 9 a.m. to 5p.m. Central European Time Monday thru Friday except NT holidays. NT holidays are posted on the Support Portal.

(2) All times are relative to the times when the issues are first submitted to NT. All times and days are relative to NT business hours/days.

(3) A PDF of this entire document can be downloaded from here.

(4) This MSLA applies to version 3.7 and above.

(5) Previous MSLA (version 3.5 to 3.7) can be found here.

(6) Previous MSLA (version 3.7 to 3.10 – per July 1st 2017) can be found here

-------------------------------------------------------------------------------- title: "Developers - Join the community - CFEngine" url: /developers/index.md date: "2021-03-11" --------------------------------------------------------------------------------

Community resources for developers, by developers

People holding tablets and phones, talking. Photograph.

CFEngine Build logo

CFEngine Build is a catalogue of policy and modules created by CFEngine, our partner and community that helps DevSecOps teams to simplify the automation process. With CFEngine Build you can automate more with less effort.

Visit CFEngine build

GitHub octocat icon Contribute

We love pull requests - whether you're a C developer or a Policy writer, your contributions are welcome! You can help maintain and improve CFEngine and make your changes available to the wider community.

CFEngine on GitHub

Collaborate

There are other ways to help improve CFEngine. Report a bug, or help with documentation - your feedback, questions, and edits all go towards making CFEngine better for you and for others.

Report a bug
CFEngine documentation on GitHub

Discuss

Discuss CFEngine with a supportive and friendly community of users. Ask or answer questions and discuss a wide range of topics. Whether you are a beginner or a contributor, you're welcome to join the conversation!

GitHub discussions
Matrix chat
Join the mailing list

Nightly build packages

Help test new features and the latest bug fixes with these bleeding edge development builds.

Download nightly build packages

Become a CFEngine Champion

Each year the CFEngine community nominates community members who have made significant contributions to the CFEngine community. By participating, contributing and advocating you are well on your way to becoming a CFEngine Champion.

Learn about the CFEngine Champion program and past winners
-------------------------------------------------------------------------------- title: "Contact us - CFEngine" url: /contact/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Contact us

If you have any commercial questions or comments, please get in touch with us by filling in this form - we'll get back to you as soon as we can.

For technical questions see the Enterprise Support Portal and our Developer Resources for info about our help mailing list/forum, GitHub Discussions, Matrix channel and other community resources.

USA NORWAY
470 Ramona Street Gaustadalleen 21,
Palo Alto, CA 94301 N-0349 Oslo, Norway
{{< recaptacha contact >}} -------------------------------------------------------------------------------- title: "Download CFEngine for free" url: /download/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Try CFEngine Enterprise for free

Contact us if you would like a quote, or have any questions about licenses or upgrading from Community Edition.

{{< recaptacha free-25 >}} -------------------------------------------------------------------------------- title: "Product - Community vs Enterprise - CFEngine" url: /community-vs-enterprise-comparison/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Find the offering that fits your needs

CFEngine is available as open source or commercial editions - Community and Enterprise.

Community edition is free to use and licensed under GNU General Public License GPL, while Enterprise adds a comprehensive set of features and SLA support under a commercial license.

Community Enterprise
Technical support Included
Prioritized feature requests Included
Linux support Included Included
Windows, AIX, Solaris & HP-UX support Included
Configuration management Included Included
Autonomous agents Included Included
Reporting hub with SQL database Included
Mission Portal web interface Included
REST APIs Included
Compliance reports Included
Policy analyzer Included
Alerts & triggered actions Included
Inventory reporting Included
Change reporting Included
File integrity monitoring Included
Performance monitoring Included
Download Try for free Ask for a quote

Product FAQ

How will upgrading to Enterprise help me?

Enterprise is aimed to help businesses take their CFEngine implementation to the next level. Mission Portal gives users greater transparency into their infrastructure to review compliance, asset & software inventory, performance metrics, and more. Enterprise also provides wider platform support & technical support from the same engineers who maintain and develop the software.

What is the best way to get started?

To begin your journey try CFEngine Enterprise for free, on up to 25 hosts, without restrictions. Don't forget to join our GitHub Discussions community to help you along the way.

How do I upgrade from CFEngine Community to Enterprise?

Every CFEngine implementation is different, therefore the upgrade path will be slightly different. We offer Community users a free assessment that will provide you with best practices & recommendations when upgrading. Contact us if you are interested in this.

What CFEngine learning resources are available to me?

The best place to start your CFEngine education is to visit our documentation. We also provide in-person or virtual courses for more specialized training. Contact us to learn more.

How is CFEngine Enterprise priced?

Our goal is to make working with CFEngine easy. As always, our Community version is open source and free. For Enterprise, all features are included in a single product at a single price per license. No add-ons, plugins or additional costs for extra functionality.

What professional services does CFEngine offer?

CFEngine offers a range of professional services to help you get the most out of your implementation. This includes consulting, implementation support, performance optimization, custom/sponsored development, training, and more. Our team of experts is here to ensure your CFEngine deployment is successful and tailored to your specific needs. Contact us if you are interested in this.

What is custom/sponsored development?

Custom/sponsored development is a service where CFEngine's development team creates tailored solutions to meet your specific requirements. Whether it's adding new features, integrating with other tools, or developing custom modules, or getting a bug prioritized (Community edition users), we can build solutions that enhance the functionality of CFEngine for your organization.

Upgrade your existing CFEngine Community installation to Enterprise.

Already running CFEngine Community Edition? Take advantage of the Enterprise feature set and improve your visibility, security & compliance.

Contact us for information Learn about our support & services
-------------------------------------------------------------------------------- title: "Product - Overview - CFEngine" url: /product-overview/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Enterprise grade automation

Whether your IT infrastructure consists of private or public cloud servers, desktops or IoT devices, CFEngine provides powerful automation and unparalleled visibility to keep infrastructure secure & compliant.

Improve security

Minimize potential vulnerabilities due to configuration drift. Quickly roll out updates and changes to infrastructure in seconds.

Ensure compliance

Leverage automation to continuously enforce compliance and minimize manual intervention. As compliance requirements evolve, CFEngine's flexibility & customization helps you quickly adapt.

Enhance visibility

Through Mission Portal, CFEngine provides a single source of truth to see the most critical infrastructure information at a glance. Use pre-built or custom reports to gain deeper insights.

View features

How CFEngine works

Illustration of how CFEngine works. The UI talks to the hub, which has a database, and talks to servers, desktops, laptops or IoT devices, all running the CFEngine agent. Illustration of how CFEngine works. The UI talks to the hub, which has a database, and talks to servers, desktops, laptops or IoT devices, all running the CFEngine agent.

Mission Portal

Mission Portal provides visibility into potential configuration issues, compliance information, and infrastructure inventory data with easy-to-use dashboards and reports.

CFEngine hub

The CFEngine hub acts as both the repository for storing your configuration settings as well as the access point for Mission Portal.

CFEngine agent

The CFEngine agent runs on each device under management and automatically connects to the CFEngine hub, by default, every 5 minutes to ensure configuration compliance.

Department of Energy
LinkedIn
Novartis
Pfizer
FIS

For any organization, no matter how large or complex.

CFEngine runs on the smallest embedded devices, on servers, in the cloud, and on mainframes, easily handling tens or hundreds of thousands of nodes.

It's battle tested with the Enterprise, but built for everyone.

Read our case studies

Solutions

Infrastructure automation

Learn how CFEngine can help your team do more with less by automating day-to-day administrative tasks.

Learn more

Solutions

Infrastructure compliance

Continuously prevent configuration drift to harden your infrastructure and meet compliance goals & requirements.

Learn more
-------------------------------------------------------------------------------- title: "Support & services - CFEngine" url: /support-services/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Support & services

CFEngine helps organizations through each step of their IT infrastructure automation journey. We provide industry leading support & professional services based on experience from the world's largest and most complex IT infrastructures to help you take your implementation to the next level.

Technical support

CFEngine Enterprise offers a dedicated support team available to answer questions, help you troubleshoot, recommend best practices, and even prioritize feature requests in future releases.

For pricing and more information. Contact us

Training

Whether you are a novice or expert, CFEngine offers remote or in-person training sessions & team workshops tailored to your specific needs. Contact us for pricing and more information.

To book in-person or virtual training. Contact us

Professional services

Need help with your CFEngine implementation? Our team offers professional services to do the heavy lifting for you.

  • Upgrading to Enterprise or the latest version.
  • Writing Policy to solve specific business needs & use-cases.
  • Integrating CFEngine with 3rd party systems to ensure end-to-end automation.
  • Make organizations more data-driven in their automation.

Pricing and more information about these services. Contact us

Upgrading to Enterprise

If you're already running CFEngine Community, we can help you upgrade to take advantage of the Enterprise features set and improve your visibility, security & compliance

For free consultation & policy review. Contact us
-------------------------------------------------------------------------------- title: "Configuring CFEngine Linux Package Repositories" url: /cfengine-linux-distros/index.md date: "2020-12-10" --------------------------------------------------------------------------------

Configuring CFEngine Linux Package Repositories

CFEngine GPG Key Fingerprint:

F2EE 2A0E 76A6 D469 8AE5  301A 7420 C142 9B92 0A5E

To install CFEngine using these repositories, as root follow the instructions below for your platform.

For Debian-based systems (apt):

Please note: You may need to install apt-transport-https in order to be able to use HTTPS sources. (apt-get install apt-transport-https). For very old distributions (Debian 4) you may have to change the URL from HTTPS to HTTP.

Complete community installation documentation.

For RPM-based systems with yum:

Complete community installation documentation.

For RPM-based systems with zypper:

# rpm --import https://cfengine-package-repos.s3.amazonaws.com/pub/gpg.key

For 64 bit systems:

# zypper addrepo -t YUM https://cfengine-package-repos.s3.amazonaws.com/pub/yum/x86_64 cfengine-repository

For 32 bit systems:

# zypper addrepo -t YUM https://cfengine-package-repos.s3.amazonaws.com/pub/yum/i386 cfengine-repository

To verify that the repository was added:

# zypper se cfengine-community

Output should look like this:

S | Name               | Summary                                                 | Type
--+------------------------------------------------------------------------------+-----------
i | cfengine-community | The CFEngine Configuration System                       | package

To install cfengine community:

# zypper -n install cfengine-community

Complete community installation documentation.

-------------------------------------------------------------------------------- title: "Product - Features - CFEngine" url: /features/index.md date: "2020-12-10" --------------------------------------------------------------------------------

Advanced features to help you know more & react faster.

Bring together your organization's DevSecOps with enterprise-level security, stability and scalability.

Dashboards

Providing comprehensive, infrastructure visibility at-a-glance.

  • Real-time compliance levels
  • Performance monitoring
  • Custom alerts & actions
  • Customizable & shareable dashboards
Widget with pie chart of Operating Systems and widget with graph of changes over time
UI with report showing ports listening and hostnames in a table.

Inventory reporting

  • Detailed reporting across your entire infrastructure.
  • With wide platform support you can collect all the information you need from any device in your infrastructure - whether it's a bare metal server, a VM, a cloud instance or IoT device.

Compliance reporting

  • Powerful and customizable compliance reports to ensure you meet your internal and external requirements at all times.
  • Configure different reports for various stakeholders. Keep your entire organization informed.
Bar graph of Compliance by category. Each category, like Security, has a percentage and a color coded bar.
UI showing code in CFEngine policy language. There are line numbers and some lines are highlighted in green.

Policy analyzer

  • Define system states using CFEngine policy.
  • Policy Analyzer gives you real time information about how each line of policy is performing on servers in your infrastructure - helping you troubleshoot without guesswork.

Alerts & triggered actions

Create custom alerts to notify you on specific systems events or infrastructure changes. Set up custom actions to trigger based on alerts or easily integrate them with your existing tools and technologies.

System health
Widget showing a green checkmark for policy compliance.
Policy compliance
Widget showing 98% policy compliance on 277 hosts.
HPUX, AIX, Linux, Oracle Solaris, Windows.

Wide platform support

CFEngine supports major platforms like RHEL, Ubuntu, Debian, Windows Server, and more.

Through custom-tailored support, our experienced team can also bring CFEngine to your specific environment, including OpenBSD, IBM AIX, macOS, Solaris, and more.

Contact us to discuss your platform needs  

Compare Enterprise & Community editions

View full comparison

Enterprise customer support

There's nothing quite like having an expert on your side. With CFEngine Enterprise, you have a dedicated support team that is available to answer questions, recommend best practices, and even prioritize development of the features you need.

Contact us
-------------------------------------------------------------------------------- title: "Supported versions - CFEngine" url: /supported-versions/index.md date: "2020-12-10" --------------------------------------------------------------------------------

Supported versions

Commercial support is exclusively provided for Enterprise LTS versions. Community editions and interim non-LTS releases do not include commercial support.

To inquire about additional support or a custom service agreement, please contact us

.

Release schedule for CFEngine Enterprise & Community

CFEngine releases adhere to a time-based release schedule, in addition to following a practice of Long Term Support (LTS).

This model offers stability and predictability, with the compromise of being able to introduce features more quickly in the non-LTS releases. LTS releases are scheduled every 18 months and are supported and maintained for 3 years, in comparison to non-LTS releases which are scheduled every 6 months and maintained for a further 6 months. Note that only LTS releases are supported as part of the CFEngine Enterprise customer support contract.

Timeline showing when releases are supported. 3.15 LTS: 2018-2021. 3.18 LTS: 2021-2022. 3.21 LTS: 2023-2024. 3.24 LTS: 2024-2025 3.27 LTS: 2026-2028

Time-based releases means that we will not delay a release in order to wait for features that are in development, but will release at regular intervals with the features that are available at the time. The advantage of this is that releases are predictable and users will know when the next release containing their fix will be. It is a great way to bring features out faster, both from us and from our community contributors.

Find out more details in the full blog post.

Supported operating systems, platforms and architectures

CFEngine supports a wide range of operating systems like Ubuntu, Redhat, Windows, AIX, Solaris, etc. on various architectures.

To provide predictable platform support in CFEngine, our policy is that CFEngine supports operating system versions as long as they are supported by their vendor, and up to one year after the vendor drops support for that platform.

For a list of supported operating systems platforms and architectures for the latest LTS version, click here

-------------------------------------------------------------------------------- title: "The CFEngine Blog" url: /blog/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Dispatching Ansible playbooks with CFEngine" url: /blog/2026/dispatching-ansible-playbooks-with-cfengine/index.md date: "2026-10-21" -------------------------------------------------------------------------------- In large organizations and infrastructures, it's fairly common to use multiple configuration management tools. There are many reasons for this. Different teams may have different preferences, some tools suit certain use cases better than others, and existing systems can come with their own limitations. Whatever the reason, the tools don't have to work in isolation. In this post, we'll show how to run Ansible playbooks on CFEngine-managed hosts using Mission Portal. We'll install Ansible with the `install-ansible` Build module, add a playbook with the `ansible-playbook-dispatcher` module, and verify that it actually runs. ## Installing Ansible To run Ansible playbooks on our CFEngine-managed hosts, we first need to install Ansible on them. The [`install-ansible`](https://build.cfengine.com/modules/install-ansible/) build module, written by [Craig Comstock](https://github.com/craigcomstock), makes this easy. {{< figure src="01-install-ansible.png" >}} In Mission Portal, open the **Build** app from the lower-left corner. Search for `install-ansible` and press **Enter**. The module should appear as the first result. Click it, then click **Add module**, followed by **Push and deploy**. The `install-ansible` build module is configured through CFEngine classes. For running playbooks, the minimal install is enough, and we enable it by defining the class `data:ansible_minimal_install`. {{< figure src="02-groups-app.png" >}} One way to define the class is through the **Groups** app in Mission Portal. Select an existing group or create a new one. Here, we'll select the **All** group so that Ansible is installed on every host. {{< figure src="03-define-class.png" >}} Click the **Data** tab, then click **Add**. Choose **Class** as the type and enter `data:ansible_minimal_install` as the name. Optionally, add a suitable description and tags. Once saved, Ansible will be installed on all hosts in the group during their next agent run. ## Running a playbook We're now ready to run Ansible playbooks. Note that the `ansible-playbook-dispatcher` module requires cfbs 5.8.0 or newer, which is bundled with CFEngine 3.29 and later. If you're on an older version, you can use the [`run-ansible-playbooks`](https://build.cfengine.com/modules/run-ansible-playbooks/) module instead. However, it doesn't distribute the playbooks to your hosts, so you'll need to do that another way, for example with CFEngine policy. {{< figure src="04-ansible-playbook-dispatcher.png" >}} Go back to the **Build** app, search for `ansible-playbook-dispatcher`, and select the first result. {{< figure src="05-input-playbook.png" >}} Under **Input data**, click **Edit**. You can either upload a playbook file or paste its contents directly. Here, we'll choose **Paste content**. Paste your playbook under **Playbook** and give it a filename. Use **Condition** to set when the playbook should run based on a CFEngine class expression. Use **ifelapsed** to set the minimum time between runs, in minutes. When you're done, click **Save**. On the next agent run, CFEngine will copy the playbook to every host, but only run it where the condition is true. ## Verifying the results {{< figure src="06-reports-app.png" >}} To verify that the playbook is running, open the **Reports** app and click **Changes**. You should see entries from `ansible-playbook-dispatcher`. If other changes make these hard to find, add a filter where **Promise type** is `ansible`. ## Wrapping up That's all it takes to run Ansible playbooks on CFEngine-managed hosts. With the `install-ansible` and `ansible-playbook-dispatcher` Build modules, CFEngine distributes your playbooks, uses classes to decide when they run, and reports the results in Mission Portal. This lets teams keep using the tools they prefer, while managing everything from one place. If you try this out, we'd love to hear how it goes on [GitHub Discussions](https://github.com/cfengine/core/discussions). -------------------------------------------------------------------------------- title: "Antivirus scanning with CFEngine and ClamAV" url: /blog/2026/clamav-module/index.md date: "2026-10-19" -------------------------------------------------------------------------------- Is the antivirus on your servers working correctly right now? Installing ClamAV takes a single command. The hard part is keeping it running on all servers, since freshclam fails without indicating a problem, cron jobs disappear, and no one notices until an audit. The new [ClamAV](https://build.cfengine.com/modules/clamav/) module deals with this by ensuring that ClamAV remains operational on all hosts and also indicates to you in the Mission Portal when it isn't. ## What the module does - **Installs ClamAV** from the upstream `.deb`/`.rpm` packages published on clamav.net, on Debian/Ubuntu and RHEL-family hosts (x86_64 and aarch64). - **Keeps signatures fresh** by running `freshclam` every 2 days. - **Scans on a schedule** by running `clamscan` over `/home`, `/root`, `/tmp` and `/var/tmp` when the last report is older than 3 days. - **Reports the results** as CFEngine Enterprise inventory: engine version, signature count, scanned and infected files, data scanned, last scan time, signature freshness and the names of detected threats. - **Alerts on infections** on every agent run until they are dealt with. All the intervals and paths can be configured. ## Getting started [Add the module](https://docs.cfengine.com/docs/lts/getting-started/03-adding-modules/) to your project and build, then deploy the result to your hub the way you normally do: ```command cfbs add clamav cfbs build ``` CFEngine Enterprise users can also add the module from Mission Portal, using the Build application in the left menu: {{< wideimg "/blog/2026/images/clamav-module/add-via-mission-portal-build-application.png" "Adding the clamav module in the Build application in Mission Portal" >}} To change the defaults, run `cfbs input clamav`. It asks about the scan interval, the signature update interval, the directories to scan and the directories to skip. You can also override any setting through augments without touching the module: ```json { "variables": { "clamav:globals.max_age_days": "1", "clamav:globals.signature_max_age_days": "1", "clamav:globals.scan_target": ["/home", "/srv"], "clamav:globals.exclude_dirs": ["/proc", "/sys", "/dev", "/run", "/srv/backup"] } } ``` If you don't want to wait for the next scheduled run, for example after cleaning up an infection, trigger a scan or a signature update straight away: ```command cf-agent -KI --define clamav:want_scan_now cf-agent -KI --define clamav:want_freshen_now ``` ### Updates and scans run only when needed CFEngine [runs every few minutes](https://docs.cfengine.com/docs/lts/web-ui/hub_administration/adjusting-schedules/#set-cf-execd-agent-execution-schedule), but signature updates and scans don't. The module starts them only once the configured interval has passed. If an update fails, for example because the mirror is down, the module waits for the next interval instead of retrying on every run. The exception is a host with no signatures at all, which keeps retrying until the first update succeeds. A scan that finds infected files still counts as a completed scan, and the findings show up in inventory. ## Seeing the results Once hosts report in, the ClamAV attributes show up in Mission Portal inventory like any other attribute. You can filter on them, add them to reports and export them. {{< wideimg "/blog/2026/images/clamav-module/inventory.png" "ClamAV inventory in Mission Portal" >}} The module also ships a compliance report, deployed automatically through the `compliance-report-imports` dependency. It checks three conditions: | Condition | Severity | Passes when | |-------------------------------------------|----------|--------------------------------------------------------| | ClamAV: no infected files found | high | The latest scan reported results with 0 infected files | | ClamAV: scan ran within the last week | medium | A scan completed within the last 7 days | | ClamAV: signatures updated within 2 weeks | low | The signature database is less than 14 days old | {{< wideimg "/blog/2026/images/clamav-module/compliance-report-mission-portal.png" "ClamAV compliance report in Mission Portal" >}} Hosts that report no ClamAV data at all fail all three conditions. A host where the module was never deployed, or where ClamAV failed to install, shows up as non-compliant instead of quietly dropping out of the report. ## Getting notified Reports are useful for audits, but when a scan finds something you want to know right away. Add an [alert](https://docs.cfengine.com/docs/lts/web-ui/alerts-and-notifications/) widget on the Mission Portal dashboard and create a new condition of type **Inventory** with two rules: - **ClamAV Infected file is reported** - **ClamAV Infected files is not 0** The first rule limits the alert to hosts that have scan results. Without it, hosts that never reported a scan would trigger the alert too. Those hosts are already caught by the compliance report. Then tick **Email notifications** and enter the address to notify. The next time a scan finds an infected file, you get an email naming the affected host. {{< wideimg "/blog/2026/images/clamav-module/setup-alert-and-email-notification.png" "Setting up an alert for ClamAV infected files with email notifications in Mission Portal" >}} ## Things to know - **Memory.** `clamscan` and `freshclam` load the entire signature database into memory, so hosts need about 2 GB of available RAM. On smaller hosts, add a swap file with the [manage-swap](https://build.cfengine.com/modules/manage-swap/) module. - **The freshclam daemon on Debian/Ubuntu.** If the distro's `clamav-freshclam` service is running, it locks the signature database and the module's updates fail. Disable it with `systemctl disable --now clamav-freshclam` so CFEngine is the only updater. - **Choose scan targets carefully.** Scanning the whole filesystem is slow and rarely useful. Start with the defaults, add directories where untrusted content arrives (uploads, shared storage, home directories), and exclude large backup or data directories. ## What's next The module is tagged experimental, and a few improvements are planned: - **Windows support.** - **Automatic ClamAV upgrades.** Feedback, bug reports and pull requests are welcome on [GitHub](https://github.com/aleksandrychev/cfengine-clamav). -------------------------------------------------------------------------------- title: "CFEngine 3.24.5 and 3.27.2 released" url: /blog/2026/cfengine-3-24-5-and-3-27-2-released/index.md date: "2026-10-16" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.24.5 and 3.27.2\! These patch releases contain bug fixes and dependency updates. ## Changes As these are patch releases for long term supported (LTS) branches of CFEngine, there are no new major features included. ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.27.2 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.27/release-notes/whatsnew/changelog-core/) - [3.27.2 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.27/release-notes/whatsnew/changelog-enterprise/) - [3.27.2 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.27/release-notes/whatsnew/changelog-masterfiles-policy-framework/) - [3.24.5 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes/whatsnew/changelog-core/) - [3.24.5 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes/whatsnew/changelog-enterprise/) - [3.24.5 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes/whatsnew/changelog-masterfiles-policy-framework/) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades We have updated the following dependencies: | CFEngine version | 3.24.4 | **3.24.5** | 3.27.1 | **3.27.2** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.66 | **2.4.69** | 2.4.66 | **2.4.69** | | [APR](https://apr.apache.org/) | 1.7.6 | 1.7.6 | 1.7.6 | 1.7.6 | | [apr-util](https://apr.apache.org/) | 1.6.3 | **1.6.5** | 1.6.3 | **1.6.5** | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.12 | 3.12 | 3.12 | 3.12 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.54.0 | **2.56.0** | 2.54.0 | **2.56.0** | | [leech](https://github.com/larsewi/leech/releases) | - | - | 0.2.0 | 0.2.0 | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.2 | **2.4.0** | 2.3.2 | **2.4.0** | | [libattr](https://download.savannah.gnu.org/releases/attr) | 2.5.2 | **2.6.0** | 2.5.2 | **2.6.0** | | [libcurl](https://curl.se/download.html) | 8.17.0 | **8.22.0** | 8.17.0 | **8.22.0** | | [libcurl-hub](https://curl.se/download.html) | 8.17.0 | **8.22.0** | 8.17.0 | **8.22.0** | | [libexpat](https://libexpat.github.io/) | 2.8.0 | **2.9.0** | 2.8.0 | **2.9.0** | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | 2.5.1 | 2.5.1 | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.19 | 1.19 | 1.19 | 1.19 | | [librsync](https://github.com/librsync/librsync/releases) | - | - | 2.3.4 | 2.3.4 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.15.3 | **2.15.4** | 2.15.3 | **2.15.4** | | [LibYAML](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | 0.2.5 | 0.2.5 | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.35 | **0.9.36** | 0.9.35 | **0.9.36** | | [nghttp2](https://nghttp2.org/) | - | - | 1.69.0 | **1.70.0** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.13 | **2.7.1** | 2.6.13 | **2.7.1** | | [OpenSSL](https://openssl.org/) | 3.6.2 | **3.6.5** | 3.6.2 | **3.6.5** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | 10.47 | **10.49** | 10.47 | **10.49** | | [PHP](https://php.net/) | 8.3.30 | **8.3.35** | 8.5.5 | **8.5.11** | | [PostgreSQL](https://www.postgresql.org/) | 16.13 | **16.15** | 18.3 | **18.6** | | [pthreads-w32](https://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | 2-9-1 | 2-9-1 | | [rsync](https://download.samba.org/pub/rsync/) | 3.4.2 | **3.5.1** | 3.4.2 | **3.5.1** | | [SASL2](https://www.cyrusimap.org/sasl/) | 2.1.28 | 2.1.28 | 2.1.28 | 2.1.28 | | [zlib](https://www.zlib.net/) | 1.3.2 | 1.3.2 | 1.3.2 | 1.3.2 | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/lts/release-notes/legal-and-licenses/). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.27.2 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.24.5 list ``` ```command cf-remote --version 3.27.2 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 65 - Dispatching Ansible with CFEngine" url: /blog/2026/the-agent-is-in-episode-65/index.md date: "2026-09-24" -------------------------------------------------------------------------------- We enjoyed Lars so much [last time](https://youtu.be/XH82LNIoQyk) that we asked him to come back and show off some Ansible work he and Craig have been doing. Lars demoed a hybrid CFEngine + Ansible workflow. Starting with two fresh hosts, he used an [CFEngine Ansible Galaxy collection](https://galaxy.ansible.com/ui/repo/published/cfengine/cfengine/) to install and bootstrap CFEngine with a playbook. Then he set up CFEngine Build with Craig's [install-ansible](https://build.cfengine.com/modules/install-ansible/) module to ensure Ansible is available everywhere via pipx, and then his new [ansible-playbook-dispatcher](https://build.cfengine.com/modules/ansible-playbook-dispatcher/) module to upload, distribute, and locally execute Ansible playbooks across the fleet. ## Why dispatch Ansible with CFEngine? {#why-dispatch-ansible-with-cfengine} If you started with Ansible and your fleet grew, you may have hit the point where central push-based execution takes too long or becomes fragile over unreliable networks. A hybrid setup lets CFEngine handle what it's good at (continuous local enforcement, convergence, reporting) while still running your existing Ansible playbooks. The key advantage is that once the playbooks are distributed, the agent runs them locally even if the network goes down. If your hosts live on a ship, in a vehicle, or behind intermittent connectivity, they keep converging. ## The demo {#the-demo} Lars walked through: 1. **Bootstrapping CFEngine with Ansible** -- Install the `cfengine.cfengine` collection from Ansible Galaxy and run a playbook that installs and bootstraps both a hub and a client. A `wait_for` task avoids the race condition where the client tries to bootstrap before `cf-serverd` is ready on the hub. 2. **Set up CFEngine Build** -- Create a `cfbs` project and add two modules: - [install-ansible](https://build.cfengine.com/modules/install-ansible/) ensures Ansible is installed on all managed hosts via pipx. - [ansible-playbook-dispatcher](https://build.cfengine.com/modules/ansible-playbook-dispatcher/) (Lars' new module) lets you upload a playbook through the Build UI. It gets distributed and run locally on each host. 3. **Deploy and verify** -- After deploying the build project, running the update and agent on a client installs Ansible via pipx, creates a local inventory file, and executes the uploaded playbook. ## Two ways to dispatch playbooks {#two-ways-to-dispatch-playbooks} The [ansible-playbook-dispatcher](https://build.cfengine.com/modules/ansible-playbook-dispatcher/) module lets you upload playbook files directly through Mission Portal's Build interface. The playbook content gets embedded in the build project and distributed to all hosts. The [run-ansible-playbooks](https://build.cfengine.com/modules/run-ansible-playbooks/) module takes a different approach: instead of uploading files, you point it at a path where your playbooks already exist. It uses the same underlying [ansible promise type](https://build.cfengine.com/modules/promise-type-ansible/) but doesn't prescribe how playbooks are distributed. Both modules support a `condition` field for targeting specific hosts and an `ifelapsed` field to control how often playbooks run. ## Video {#video} The video recording is available on YouTube: {{< youtube LkLkAOjs6xE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Lars](https://www.linkedin.com/in/larsewi), [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [cfengine.cfengine Ansible Galaxy collection](https://galaxy.ansible.com/ui/repo/published/cfengine/cfengine/) - [install-ansible module (CFEngine Build)](https://build.cfengine.com/modules/install-ansible/) - [ansible-playbook-dispatcher module (CFEngine Build)](https://build.cfengine.com/modules/ansible-playbook-dispatcher/) - [run-ansible-playbooks module (CFEngine Build)](https://build.cfengine.com/modules/run-ansible-playbooks/) - [ansible promise type (CFEngine Build)](https://build.cfengine.com/modules/promise-type-ansible/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Installing CFEngine with Ansible" url: /blog/2026/install-cfengine-with-ansible/index.md date: "2026-09-14" -------------------------------------------------------------------------------- You started with Ansible, and for a long time it was the only thing you needed. With a handful of playbooks and an inventory file, you got the job done. However, as the fleet grew, runs went from taking minutes to hours, and parts of the inventory were unreachable at any given moment. This is not an Ansible flaw. Push-based configuration and continuous state enforcement are simply two different jobs. This blog post is all about getting you started with a hybrid system. A system where you can leverage the best of both worlds by using CFEngine alongside Ansible. The first step is to install and bootstrap CFEngine on all your hosts. And since Ansible already has a working path to the hosts, it is the obvious tool to get it done. For illustrative purposes, I've set up a small inventory with one host (the hub) being responsible for distributing CFEngine policies and Ansible playbooks to the other hosts (the clients). This post only covers the installation. See the follow-up reading at the end for how to distribute playbooks with CFEngine. ```ini {file="ansible/inventory.ini"} [hub] localhost ansible_connection=local [clients] 172.31.0.20 ansible_user=admin 172.31.2.234 ansible_user=ec2-user [all:vars] ansible_python_interpreter=/usr/bin/python3 ``` Before proceeding with the CFEngine installation, I find it useful to ping all the hosts to make sure they are all reachable. ```command ansible all -i ansible/inventory.ini -m ping ``` ```output localhost | SUCCESS => { "changed": false, "ping": "pong" } 172.31.0.20 | SUCCESS => { "changed": false, "ping": "pong" } 172.31.2.234 | SUCCESS => { "changed": false, "ping": "pong" } ``` To download and install the correct packages, as well as bootstrap all the hosts, there is a neat [CFEngine Ansible Collection](https://galaxy.ansible.com/ui/repo/published/cfengine/cfengine/) that can help us out _(written by [Vratislav Podzimek](https://github.com/vpodzime))_. ```command ansible-galaxy collection install cfengine.cfengine ``` ```output Starting galaxy collection install process Process install dependency map Starting collection install process Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/cfengine-cfengine-1.1.0.tar.gz to /home/ubuntu/.ansible/tmp/ansible-local-2452gwlze5dl/tmpqsw5ntqd/cfengine-cfengine-1.1.0-cfdnq5iz Installing 'cfengine.cfengine:1.1.0' to '/home/ubuntu/.ansible/collections/ansible_collections/cfengine/cfengine' cfengine.cfengine:1.1.0 was installed successfully ``` Next we'll need to write the actual playbook that does the work. It's fairly straightforward. One thing to watch out for, though, is that we need to make sure we wait for [`cf-serverd`](https://docs.cfengine.com/docs/master/reference/components/cf-serverd/) on the hub to start listening on port 5308 before trying to bootstrap any of the other hosts. This can easily be achieved with the [`ansible.builtin.wait_for`](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/wait_for_module.html) module. In the playbook below, `172.31.10.16` is the hub's own address. This is the address the clients use to reach it. ```yaml {file="ansible/playbook.yaml"} - name: Bootstrap the CFEngine hub hosts: hub become: true tasks: - name: Install and bootstrap the hub cfengine.cfengine.cfengine: policy_server: 172.31.10.16 - name: Bootstrap the CFEngine clients hosts: clients become: true tasks: - name: Wait for cf-serverd on the hub ansible.builtin.wait_for: host: 172.31.10.16 port: 5308 timeout: 120 - name: Install and bootstrap the clients cfengine.cfengine.cfengine: policy_server: 172.31.10.16 ``` The only thing that remains now is to run the playbook. ```command ansible-playbook -i ansible/inventory.ini ansible/playbook.yaml ``` ```output PLAY [Bootstrap the CFEngine hub] ******************************************************************************** TASK [Gathering Facts] ******************************************************************************************* ok: [localhost] TASK [Install and bootstrap the hub] ***************************************************************************** changed: [localhost] PLAY [Bootstrap the CFEngine clients] **************************************************************************** TASK [Gathering Facts] ******************************************************************************************* ok: [172.31.0.20] ok: [172.31.2.234] TASK [Wait for cf-serverd on the hub] **************************************************************************** ok: [172.31.0.20] ok: [172.31.2.234] TASK [Install and bootstrap the clients] ************************************************************************* changed: [172.31.2.234] changed: [172.31.0.20] PLAY RECAP ******************************************************************************************************* 172.31.0.20 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 172.31.2.234 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` It all looks good. Let's verify. ```command ansible all -i ansible/inventory.ini -b -m command -a "cf-agent -V" ``` ```output localhost | CHANGED | rc=0 >> CFEngine Core 3.27.1 CFEngine Enterprise 3.27.1 172.31.0.20 | CHANGED | rc=0 >> CFEngine Core 3.27.1 CFEngine Enterprise 3.27.1 172.31.2.234 | CHANGED | rc=0 >> CFEngine Core 3.27.1 CFEngine Enterprise 3.27.1 ``` Notice that the output says Enterprise. That is what the collection installs by default. CFEngine Enterprise is free for up to 25 hosts. If you'd rather run Community, set the `edition` option: ```yaml {file="ansible/playbook.yaml"} ... - name: Install and bootstrap the clients cfengine.cfengine.cfengine: policy_server: 172.31.10.16 edition: community ... ``` And that concludes this blog post. Where to go next? Check out [Getting started with Ansible playbooks in CFEngine](/blog/2026/ansible-cfengine/), which takes the easy route. Or check out [Use Ansible playbooks in CFEngine policy with promise-type-ansible module](https://cfengine.com/blog/2024/promise-type-ansible/) for something more production ready. If you have any questions about the CFEngine Ansible collection, let us know at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear from you. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 64 - Ship the change, not the table, with leech2" url: /blog/2026/the-agent-is-in-episode-64/index.md date: "2026-08-27" -------------------------------------------------------------------------------- For episode 64 Lars demoed [leech2](https://github.com/larsewi/leech2) live: the library used to ship container data from CFEngine agents to Mission Portal. The [slides](the-agent-is-in-episode-64/leech2.pdf) he presented from are available as a PDF. Lars showed how you can interact with leech2 through the CLI tool `lch(1)`. The CLI tool is bundled in CFEngine >= 3.28 packages. Later, he showed how the algorithm for block consolidation works. After the show there was a nice discussion about how leech2 can be used for other purposes. We talked about extending the CFEngine reporting system with [leech2 drop-in fragments](https://github.com/larsewi/leech2#drop-in-fragments), interacting with leech2 through the [C FFI API](https://github.com/larsewi/leech2/blob/master/include/leech2.h), and how it would be nice to add [JSON support](https://github.com/larsewi/leech2/issues/215). ## Video {#video} The video recording is available on YouTube: {{< youtube XH82LNIoQyk >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Lars](https://www.linkedin.com/in/larsewi/) - [CFEngine 3.28 release (changelog)](/blog/2026/cfengine-3-28-released-introspection/) - [leech2](https://github.com/larsewi/leech2/) - [leech2 -- C FFI API](https://github.com/larsewi/leech2/blob/master/include/leech2.h) - [leech2 -- slides (PDF)](the-agent-is-in-episode-64/leech2.pdf) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Change in behavior: Storage promise" url: /blog/2026/change_in_behavior_storage_promises/index.md date: "2026-08-13" -------------------------------------------------------------------------------- CFEngine 3.29.0 changes how `storage` promises treat a filesystem that is already mounted. A promise for an unmounted filesystem now mounts only that filesystem instead of all unmounted filesystems, `edit_fstab` actively maintains the file system table entry to keep it aligned with the promise even when the filesystem is already mounted, and an `unmount` promise acts only on the explicitly promised filesystem. Additionally there are is new functionality to remount a filesystem automatically to correct its state. ## Only the promised filesystem is mounted Previously, a promise for a filesystem that was not mounted did not mount that filesystem directly, rather it was seeing that a mount was needed which would cause `mount -a` to run at the end of the agent run (mounting any unmounted entries). A single promise could therefore bring up unrelated filesystems as a side effect. Now only the promised filesystem is mounted ([CFE-1863](https://northerntech.atlassian.net/browse/CFE-1863)). If you want every entry in the file system table mounted, use [`mountfilesystems` in `body agent control`](https://docs.cfengine.com/docs/lts/reference/components/cf-agent/#mountfilesystems). ## The file system table entry is maintained for mounted filesystems `edit_fstab` used to write the entry as part of the initial mount. If the filesystem was already mounted there was nothing to do, so a missing or drifted entry was not altered as long as the mount point had something mounted. With 3.29.0 the fstab entry is managed independently, regardless of whether the filesystem is mounted ([CFE-1539](https://northerntech.atlassian.net/browse/CFE-1539)). A missing entry is restored, and an entry whose options have drifted is rewritten. When `mount_options` is not specified the entry is written with the platform default options -- `defaults` on Linux, `bg,hard,intr` on AIX, HP-UX and Solaris, `-i,-b` on the BSDs and macOS. ## unmount acts only on the filesystem it names Promising that a filesystem is not mounted keys off of `mount_source` and `mount_server` instead of the promised mountpoint ([CFE-2350](https://northerntech.atlassian.net/browse/CFE-2350)). ```cfengine3 bundle agent example_unmount { storage: "/mnt/-export-home" mount => decomissioned; } body mount decommissioned { mount_type => "nfs"; unmount => "true"; mount_source => "/export/home"; mount_server => "decommissioned_host.example.org"; edit_fstab => "true"; } ``` If a filesystem _other_ than the promised one is mounted at the promiser, it is neither unmounted nor removed from the file system table. When the promised filesystem _is_ mounted there it is unmounted and its entry removed; when nothing is mounted there the entry is removed anyway. ## New: reconciling the options of a running mount By default `mount_options` still affect only the initial mount and, with `edit_fstab`, the file system table entry. A filesystem that is already mounted with different options is left as it is. When `remount` is enabled the promised options are compared against the running mount and reconciled when they differ. ```cfengine3 body mount readonly_data { mount_type => "nfs"; mount_source => "/export/data"; mount_server => "fileserver.example.com"; mount_options => { "ro", "nosuid" }; edit_fstab => "true"; remount => "true"; } ``` `remount` also decides what happens when the mount point holds a filesystem _different_ from the promised one. Without it the promise would be reported as not kept. With `remount` enabled cf-agent will attempt correction. ### Mount reconciliation `remount_methods` is an ordered list of mechanisms attempted sequentially, waiting for `remount_timeout` (60 second default) between each. - `remount` -- remount in place (`mount -o remount,...`). Applies generic mount flags such as `ro` / `rw` and the `atime` options, but cannot change NFS-negotiated options such as `vers`, `proto` or `sec`. - `unmount_mount` -- unmount and mount again. Applies any option change and can correct a wrong mount source or server, but is disruptive and fails if the filesystem is busy. The default is `{ "remount" }`. ```cfengine3 body mount reconcile_hard { remount => "true"; # try an in-place remount first, then unmount + mount for options a # remount cannot change, or a wrong source or server remount_methods => { "remount", "unmount_mount" }; } ``` ## How promised fstab options are compared Only the fstab options promised are enforced. Kernel-added options like `vers=`, `rsize=`, `wsize=`, `timeo=`, `addr=` and anything else the promise does not mention are not enforced. CFEngine expands `defaults` to `rw`, `suid`, `dev`, `exec` and `async`, then checks each of those against the running mount. ## Let us know how it goes! If you have questions or want to share your experience with the new behavior, reach out on [GitHub discussions](https://github.com/cfengine/core/discussions), the [mailing list](https://groups.google.com/g/help-cfengine), or [Matrix](https://matrix.to/#/%23CFEngine:matrix.org). To report a bug or request a feature, open an issue on the [issue tracker](https://northerntech.atlassian.net/projects/CFE). If you have a support contract, feel free to open a ticket (sooner is better than later) in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Getting started with Ansible playbooks in CFEngine" url: /blog/2026/ansible-cfengine/index.md date: "2026-08-05" -------------------------------------------------------------------------------- Recently, I've been playing around with the [ansible promise type](https://build.cfengine.com/modules/promise-type-ansible/) by [Fabio Tranchitella](https://github.com/tranchitella). It's amazing how easy it is to leverage the [benefits of agents](https://cfengine.com/blog/2024/benefits-of-agents/) with all the existing automation from the ansible community. Hence, I wanted to share it in a blog post, with an emphasis on **easy**. Let's start off by creating a new [cfbs](https://github.com/cfengine/cfbs) project in a `cfengine-ansible` directory. Make sure to answer **yes** on the prompt to build on top of the default policy set. You may answer the remaining prompts as you see fit. ```command cfbs init ``` ```output Please enter the name of this CFEngine Build project [Example project]: cfengine-ansible Please enter the description of this CFEngine Build project [Example description]: Example of running Ansible playbooks with CFEngine Do you want cfbs to initialize a git repository and make commits to it? [YES/y/no/n] no Initialized an empty project called 'cfengine-ansible' in 'cfbs.json' Do you wish to build on top of the default policy set, masterfiles? (Recommended) [YES/y/no/n]: yes Added module: masterfiles ``` Since ansible is not a built-in promise type in CFEngine, we'll need to add it to our project from [CFEngine Build](https://build.cfengine.com/). ```command cfbs add promise-type-ansible ``` ```output Added module: library-for-promise-types-in-python (Dependency of promise-type-ansible) Added module: promise-type-ansible ``` The addition of the custom promise type doesn't pull in Ansible on its own. Hence, we'll need make sure it's installed on all the hosts. This can easily be achieved with a simple packages promise. Create the file `ansible/ansible.cf` within the cfbs project and add the following bundle. ```cf3 bundle agent ansible_install { packages: "ansible"; } ``` Next we can create our first playbook in `ansible/playbook.yaml` that simply touches a file. ```yaml - name: My first play hosts: localhost tasks: - name: Making changes ansible.builtin.file: path: /tmp/cfengine-ansible-probe state: touch ``` Unfortunately, or rather thankfully, the playbook won't run on its own. We'll have to explicitly tell CFEngine to run it using the ansible promise type (that we added earlier). ```cf3 bundle agent ansible_playbook { ansible: "$(this.promise_dirname)/playbook.yaml"; } ``` For the sake of convenience, let's create one more bundle as a single entry point for calling the other bundles. ```cf3 bundle agent ansible_main { methods: "ansible_install"; "ansible_playbook"; } ``` If we now add our `ansible` directory to our cfbs project, we'll ensure that all of its contents will be copied into the build. Furthermore, cfbs will automatically detect the policy file and the bundles within it. When prompted, select `ansible_main` as the bundle to evaluate. ```command cfbs add ./ansible/ ``` ```output Which bundle should be evaluated (added to bundle sequence)? 1. ./ansible/ansible.cf:ansible_install (default) 2. ./ansible/ansible.cf:ansible_playbook 3. ./ansible/ansible.cf:ansible_main 4. (None) [1/2/3/4]: 3 Added module: ./ansible/ ``` Now let's build and hope for the happy chipmunk. ```command cfbs build ``` ```output ---snip--- Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` After installing / deploying the project according to the hints provided in the build output, you should be able to run the agent and see that the task successfully changed. ```command sudo cf-agent --no-lock --inform ``` ```output [WARNING]: No inventory was parsed, only implicit localhost is available info: Task 'Making changes' successfully changed ``` Furthermore, we can stat the file to make sure we've been talking to our beloved cf-agent and not a hallucinating AI agent. ```command stat /tmp/cfengine-ansible-probe ``` ```output File: /tmp/cfengine-ansible-probe Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 202,1 Inode: 2955 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-31 14:14:05.346516492 +0000 Modify: 2026-07-31 14:14:05.346516492 +0000 Change: 2026-07-31 14:14:05.346516492 +0000 Birth: 2026-07-31 12:32:03.662445650 +0000 ``` Indeed, everything works as expected. However, there was that annoying warning from Ansible complaining about a missing inventory file. Let's get rid of that by providing an inventory file using the following bundle to render it. ```cf3 bundle agent ansible_inventory { files: "$(sys.workdir)/ansible/inventory.ini" content => "[local] localhost ansible_connection=local"; } ``` Next, we need to tell the ansible promise type where the inventory file is. Update the `ansible_playbook` bundle with the inventory attribute. ```cf3 bundle agent ansible_playbook { ansible: "$(this.promise_dirname)/playbook.yaml" inventory => "$(sys.workdir)/ansible/inventory.ini"; } ``` Last, but not least, add the new inventory bundle to our main bundle. ```cf3 bundle agent ansible_main { methods: "ansible_install"; "ansible_inventory"; "ansible_playbook"; } ``` If we build and deploy again, we can observe that the warning is gone on the next agent run. ```command sudo cf-agent -KI ``` ```output info: Task 'Making changes' successfully changed ``` And that concludes this blog post. However, if you need something more production ready, you should probably go ahead with a similar approach to what [Craig Comstock](https://github.com/craigcomstock) did in his [blog post on ansible](https://cfengine.com/blog/2024/promise-type-ansible/). If you have any questions about the ansible promise type, let us know at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear from you. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 63 - Demo of CFEngine 3.28" url: /blog/2026/the-agent-is-in-episode-63/index.md date: "2026-07-30" -------------------------------------------------------------------------------- For episode `0b111111` Nick and Craig demo CFEngine 3.28 live: container data in Mission Portal, and classes promises gain more flexibility. In this episode, after learning that 63 is the ASCII code for `?`, the atomic number for Europium, and the country code for the Philippines Craig followed up on [Episode 61](https://cfengine.com/blog/2026/the-agent-is-in-episode-61/) (libntech and json) showing container inventory flowing into Mission Portal surfacing Docker and Podman containers and images. Then Nick showed new low level policy features: optional arguments on [getusers](https://docs.cfengine.com/docs/3.28/reference/functions/getusers/)/[getgroups](https://docs.cfengine.com/docs/3.28/reference/functions/getgroups/), a single-value [expandrange](https://docs.cfengine.com/docs/3.28/reference/functions/expandrange/#top) fix from Normation, and two new classes promise attributes ([cancel](https://docs.cfengine.com/docs/3.28/reference/promise-types/classes/#cancel) and [timer_policy](https://docs.cfengine.com/docs/3.28/reference/promise-types/classes/#timer_policy)) that give you more direct control over classes and when they should be defined. There was also a fun discussion with Martin about variables and how bundles aren't functions (though several features like the [bundle_return_value_index](https://docs.cfengine.com/docs/master/reference/promise-types/reports/#bundle_return_value_index) of reports promises and how classic [associative arrays are cleared at the start of each bundle](https://docs.cfengine.com/docs/master/reference/language-concepts/bundles/) actuation are very useful if you are going to abuse them). After the show there was nice discussion about [Promise Theory](https://www.youtube.com/c/PromiseTheoryandApplications), where CFEngine fits more or less naturally in the infrastructure and one user learned that [packages promises](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/) can be made about packages on disk (you don't have to run a command to install a local package, just promise it's path) and that you generally are not required to specify the `package_module` you wish to use if that's also the platforms default package manager.. ```cfengine3 bundle agent packages_example { packages: "/opt/emacs/my-emacs.deb" package_policy => "present"; } ``` ## Container inventory lands in Mission Portal {#container-inventory-lands-in-mission-portal} Earlier episodes showed how CFEngine can **collect** container and image data from hosts, that data is now flowing into Mission Portal's UI. The new **Containers** tab in the host detail view shows a summary (total images, sizes, volumes, running engines) and a full table of containers with sortable columns. Images and containers from both engines appear in a single unified view, just like a `docker ps` but with historical timestamps and host-side context. The collection happens automatically during the reporting phase of the agent run, no inventory policy needed. ## New & changed functions {#new-and-changed-functions} ### `getusers()` and `getgroups()` arguments now optional {#getusers-and-getgroups-arguments-now-optional} Both functions now return the full list if called with no arguments. Previously you had to pass two empty strings to get all users/groups. ```cfengine3 bundle agent main { reports: 'getusers() -> $(length(getusers())) = $(with)'; } ``` ```text R: getusers() -> 26 ``` ### `expandrange()` can now emit a single value {#expandrange-can-now-emit-a-single-value} The agent no longer errors when the expanded range produces a single element — a fix contributed by Normation. ```cfengine3 bundle agent main { vars: "hosts" slist => expandrange("host[1-1]", 3); reports: "$(hosts)"; } ``` ```text R: host1 ``` ## Classes promises {#classes-promises} ### Cancel / undefine a class directly {#cancel-undefine-a-class-directly} Classes promises can now **un-set** a class. Previously you had to piggyback on a `commands` or `files` outcome using `cancel_kept` / `cancel_repaired` in a `body classes`: ```cfengine3 bundle agent __main__ { classes: "My_Class" expression => "any", comment => "Simulate a feature flag that is currently on"; commands: "/bin/true" classes => if_ok_cancel("My_Class"); reports: My_Class:: "My_Class is defined"; !My_Class:: "My_Class is not defined"; } body classes if_ok(x) # @brief Define the class `x` if the promise is kept or repaired # @param x The name of the class that should be defined { promise_repaired => { "$(x)" }; promise_kept => { "$(x)" }; } ``` In 3.28 you can undefine a class **directly** from a `classes` promise with the new `cancel` attribute (a boolean). The class-context guard controls **when** it fires: ```cfengine3 bundle agent main { classes: "My_Class" expression => "any"; "My_Class" cancel => "true"; } ``` ### Extend a persistent class timer {#extend-a-persistent-class-timer} The new `timer_policy` attribute for classes promises controls whether a persistent class keeps its original expiry ("absolute") or restarts the countdown each time the promise is evaluated ("reset"). In 3.28.0 the default is "reset" — a persistent class that is still in context gets its timer extended on every agent run. The attribute was backported to 3.24.5 and 3.27.2 with a default of "absolute", preserving the pre-existing behavior. ```cfengine3 bundle agent main { classes: "cached_class" expression => "any", persistence => "120", timer_policy => "reset"; } ``` This is the kind of change that silently affects anyone already using persistent classes — if you're currently relying on the absolute expiry behavior and haven't pinned a version, the default will now extend instead. ## Video {#video} The video recording is available on YouTube: {{< youtube boNz1LgXhvY >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [CFEngine 3.28 release (changelog)](/blog/2026/cfengine-3-28-release/) - [Episode 61 - libnetchart and json (previous container inventory episode)](/blog/2026/the-agent-is-in-episode-061/) - [`getusers()` function docs](https://docs.cfengine.com/docs/3.28/reference/functions/getusers/) - [`getgroups()` function docs](https://docs.cfengine.com/docs/3.28/reference/functions/getgroups/) - [`expandrange()` docs](https://docs.cfengine.com/docs/3.28/reference/functions/expandrange/) - [`classes` promise type docs](https://docs.cfengine.com/docs/3.28/reference/promise-types/classes/) - [`packages` promise type docs](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.28 released - Introspection" url: /blog/2026/cfengine-3-28-released-introspection/index.md date: "2026-07-10" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.28.0\! The code word for this release is _introspection_. This is a non-LTS, not supported, feature release, with features which will become part of the next LTS scheduled for summer 2027. ## What's new ### Container introspection In the CFEngine Enterprise Web UI, users can now see information about containers: {{< wideimg "images/containers.png" "Screenshot of container information in the host info page, with a table of running containers and images. ">}} For this first iteration, we support `docker` and `podman` as the 2 engines / CLIs where we get information about containers. Customers who are using something else can open a support ticket / feature request and we can add more in future versions. ### Breaking changes From time to time, we need to change the behavior of certain features of CFEngine in a way that is not completely backwards compatible. This is usually to address security concerns, bugs causing issues for our users, or features behaving in really unexpected or unintended ways. In such cases, we try to communicate the changes with users, along with what is needed from them (such as adjusting their policy, reports, or API usage). Please take a look at the recent `change-in-behavior` blog posts, and stay up to date on these via our blog: https://cfengine.com/tags/change-in-behavior/ ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.28.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.28/release-notes-whatsnew-changelog-core.html) - [3.28.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.28/release-notes-whatsnew-changelog-enterprise.html) - [3.28.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.28/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to Enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates We update dependencies to ensure we have the latest security fixes in the libraries and tools used in CFEngine. The table below shows our dependencies and their versions, with version 3.26.0 for comparison: | CFEngine version | 3.27.0 | **3.28.0** | | :-------------------------------------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.66 | **2.4.68** | | [APR](https://apr.apache.org/) | 1.7.6 | 1.7.6 | | [apr-util](https://apr.apache.org/) | 1.6.3 | 1.6.3 | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.12 | 3.12 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.52.0 | **2.54.0** | | [leech](https://github.com/larsewi/leech/releases) | 0.2.0 | **-** | | [leech2](https://github.com/larsewi/leech2/releases) | - | **5.2.0** | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.2 | 2.3.2 | | [libattr](https://download.savannah.gnu.org/releases/attr) | 2.5.2 | 2.5.2 | | [libcurl](https://curl.se/download.html) | 8.17.0 | **8.20.0** | | [libcurl-hub](https://curl.se/download.html) | 8.17.0 | **8.20.0** | | [libexpat](https://libexpat.github.io/) | 2.7.3 | **2.8.1** | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.18 | **1.19** | | [librsync](https://github.com/librsync/librsync/releases) | 2.3.4 | 2.3.4 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.15.1 | **2.15.3** | | [LibYAML](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.33 | **0.9.35** | | [nghttp2](https://nghttp2.org/) | 1.68.0 | **1.69.0** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.10 | **2.6.13** | | [OpenSSL](https://openssl.org/) | 3.6.0 | **4.0.1** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | 10.47 | 10.47 | | [PHP](https://php.net/) | 8.5.1 | **8.5.7** | | [PostgreSQL](https://www.postgresql.org/) | 18.1 | **18.4** | | [pthreads-w32](https://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | | [rsync](https://download.samba.org/pub/rsync/) | 3.4.1 | **3.4.4** | | [SASL2](https://www.cyrusimap.org/sasl/) | 2.1.28 | 2.1.28 | | [zlib](https://www.zlib.net/) | 1.3.1 | **1.3.2** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, you can specify this version using the `--version` command line flag: ```command cf-remote --version 3.28.0 download ``` (Since this is a non-LTS release, it will not be the default in `cf-remote` and other places, even though it is the newest). ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 62 - An MCP for Mission Portal?" url: /blog/2026/the-agent-is-in-episode-62/index.md date: "2026-06-25" -------------------------------------------------------------------------------- Can an AI agent investigate a compromised host well enough to spot a fake user named Palpatine? Watch me find out the hard way. In this episode, Nick demonstrates a Mission Portal MCP server he's been building with AI — a real-time demo of using an AI-powered agent to investigate a compromised host that went dark after a simulated attack. The episode walks through fleet health monitoring, forensic timeline investigation, and deploying security audit modules. The discussion wraps up with practical patterns for using AI-assisted policy authoring with syntax validation and local documentation to avoid hallucinated function names. ## Video The video recording is available on YouTube: {{< youtube jfteuikh8PQ >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Change in behavior: Classes promises can reset the timer for currently defined persistent classes" url: /blog/2026/change_in_behavior_classes_promises_can_reset_the_timer_for_currently_defined_persistent_classes/index.md date: "2026-06-16" -------------------------------------------------------------------------------- CFEngine is changing how the persistence timer of a class defined by a `classes` promise is handled when that class is rediscovered on a later run. We consider this a bug fix: `classes` promises should always have been able to reset the timer, just like `classes` bodies can. A new `timer_policy` attribute on the `classes` promise type lets you choose whether the timer is reset on each evaluation (`reset`) or left to count down from its original expiry (`absolute`). In CFEngine 3.28.0 the default becomes `reset`, aligning `classes` promises with `classes` bodies. The attribute is also being backported to the 3.27.x and 3.24.x LTS streams (3.27.2 and 3.24.5), where the default stays `absolute` so that existing behavior is unchanged and the new behavior is strictly opt-in. ## Persistent classes and their timers A _persistent_ class survives between agent runs for a set number of minutes, recorded in a local database (`cf_state.lmdb`). It is pre-defined before policy evaluation begins, so it can influence behavior very early in a run. There are two ways to define a persistent class, and historically they did not behave the same way when the class was rediscovered on a later run: - The `classes` promise type, using the `persistence` attribute (minutes). - A `classes` body attached to another promise (for example `if_repaired`), using `persist_time` together with `timer_policy`. The `classes` body has carried a `timer_policy` attribute for a long time, defaulting to `reset`: each time the owning promise repairs, the timer starts over. The `classes` promise type had no such control -- it only had `persistence`, and the timer always behaved as `absolute`. That inconsistency is the bug this change addresses (tracked as [CFE-4681](https://northerntech.atlassian.net/browse/CFE-4681)). ## The previous behavior Previously, a `classes` promise whose class was already defined (loaded from the persistent store) was simply skipped. The timer counted down from the original expiry and the class disappeared after the configured minutes, regardless of how many agent runs happened in between. There was no way to extend the window directly (only as the result of some other promise using a `classes` _body_). ## The new behavior: `timer_policy` The `classes` promise type now accepts a `timer_policy` attribute: - `absolute` -- preserve the original expiry (the old behavior). - `reset` -- refresh the timer on each evaluation, even when the class is already defined. With `reset`, the effect is a _sliding window_: the class stays defined as long as the expression keeps evaluating true, expiring after the configured minutes from the _last_ run rather than the first. Try it -- run this policy twice: ```cfengine3 body common control { bundlesequence => { "demo" }; } bundle agent demo { classes: "my_class" expression => "any", persistence => "120", timer_policy => "reset"; reports: "my_class is defined"; } ``` ```bash # Requires a cf-agent build that includes CFE-4681. # Clear any prior state so the first run is a clean "Creating". rm -f /var/cfengine/state/cf_state.lmdb* echo "=== Run 1 ===" cf-agent -Kvf /tmp/timer_policy_demo.cf 2>&1 | grep -i "persistent class" echo "=== Run 2 ===" cf-agent -Kvf /tmp/timer_policy_demo.cf 2>&1 | grep -i "persistent class" ``` Run 1 logs `Creating persistent class ... policy reset` and run 2 logs `Resetting persistent class ... timer to 120 minutes`. ## Defaults and availability The `timer_policy` attribute for `classes` promises arrives in three releases, but the _default_ differs: | Release | `timer_policy` available | Default for `classes` promises | |---------|--------------------------|--------------------------------| | 3.28.0 | yes | `reset` | | 3.27.2 | yes (backport) | `absolute` | | 3.24.5 | yes (backport) | `absolute` | In 3.28.0 the default flips to `reset` to align `classes` promises with `classes` bodies and match what most users expect: a persistent class that keeps being discovered should keep persisting. The backports ship the attribute on the LTS streams while keeping the `absolute` default so nothing changes unless you ask for it. If you depend on a specific behavior, set `timer_policy` explicitly rather than relying on the default. That makes the policy behave the same on 3.24.5, 3.27.2, and 3.28.0 alike. If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "CVE-2026-33553 - Cross-site scripting in Mission Portal" url: /blog/2026/cve-2026-33553-xss-in-mission-portal/index.md date: "2026-06-01" -------------------------------------------------------------------------------- We are writing to inform you of a recently discovered issue in CFEngine Enterprise. This issue was responsibly disclosed through our HackerOne Bug Bounty program, discovered during the re-testing / follow-ups from some of the issues we announced earlier this year; https://cfengine.com/blog/2026/cve-2026-24710-and-cve-2026-24711-and-cve-2026-24712/ We have no indications of this issue being exploited or known outside of the company and the user who reported it. Thank you to [Tahmid Akbar Omim](https://hackerone.com/imperial_coder) for discovering and responsibly disclosing this issue. In order to exploit this vulnerability, someone would need access to a low privilege Mission Portal account, and interaction from the victim (admin) is also required (they need to click a link or similar). ## Description Mission Portal is susceptible to cross-site scripting (XSS), due to a mistake in the content-type HTTP header, returned by some of the API endpoints. This header is used by browsers to determine what to do with the returned content, for example whether to just treat it as data and show it, or to render it as a web page and run the included JavaScript, etc. The latter is problematic, since in this case, some of the data could be controlled by another lower-privilege user. ## Impact An authenticated low-privilege Mission Portal user could put malicious JavaScript code into certain fields in Mission Portal, and get an admin to open a page which would cause the code to run using their access allowing them to escalate their own privileges. They would need to send the admin a specific link (API URL) and have the admin click that link to trigger the issue, it does not normally happen through navigating the UI. If they gain admin access, they would have access to take over the hub and the infrastructure under management from that hub. ## Remediation CFEngine Enterprise 3.24.3, 3.27.0, and earlier versions are affected. We recommend upgrading to 3.24.4, 3.27.1, or later versions to fix the issue. For more information on how to upgrade, please see our documentation: https://docs.cfengine.com/docs/lts/getting-started/01-installation/upgrading ## Contact For help with upgrading or additional questions, please contact support at: https://support.northern.tech -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 61 - Use our lightweight C library libntech to make a handy json tool" url: /blog/2026/the-agent-is-in-episode-61/index.md date: "2026-05-28" -------------------------------------------------------------------------------- We thought we would mix things up a bit here and highlight our open source C library: [libntech](https://github.com/NorthernTechHQ/libntech). I wanted to use libntech's [JSON](https://www.json.org/json-en.html) code to write a minimal tool to parse, check and query JSON data. Using the various library functions related to files, json, strings, sequences and regexes helps make writing a C program a breeze and the result easy to read. ## A beginning (of sorts) This library was created in August 2019 with this commit: ``` Author: CFEngine Contributors <> Date: Wed Aug 14 12:41:41 2019 +0200 Forked cfengine/core libntech was forked from CFEngine community at this commit SHA: https://github.com/cfengine/core/commit/3b6af9850007fe9339db7d7fcc69f490acea5867 To see git history, blame, etc. before this point, checkout the commit in cfengine/core above. This was done because the cfengine repository already had 15 000 commits, takes quite some time to clone and most of the history is not related to the contents of libntech (only a small part of the CFEngine codebase). This will also give the repo a "fresh" start in git and github statistics. Over time these will show contributors to the new libntech library, not to the rest of the cfengine codebase. Thank you to everyone who contributed to CFEngine in the past: https://github.com/cfengine/core/blob/3b6af9850007fe9339db7d7fcc69f490acea5867/AUTHORS Signed-off-by: Ole Herman Schumacher Elgesem ``` ## How to use libntech There is a [template repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository) in github [libntech-example](https://github.com/NorthernTechHQ/libntech-example) which can be used to create a new repository. The advantage over cloning libntech is that you get a nice clean commit history and a pre-canned auto tools setup with libntech as a git submodule. First, visit [libntech-example](https://github.com/NorthernTechHQ/libntech-example) and click on Use this template and Create a new repository. Once that is done simply follow the directions in the README.md for cloning, initialization and building. ## Make a new command Adding a new binary program to build in automake is fairly straight forward and easier still because we already have an example to follow in the sample command `argv_printer`. Here we add a new `json` program to the top-level `Makefile.am`: ``` --- a/Makefile.am +++ b/Makefile.am @@ -17,9 +17,16 @@ ACLOCAL_AMFLAGS = -I m4 --install EXTRA_DIST = m4/NOTES bin_PROGRAMS = \ + json \ argv_printer_zero \ argv_printer_libs # Equal length file name for fair comparison +json_SOURCES = json.c +json_LDADD = libntech/libutils/.libs/libutils.la +json_CPPFLAGS = -I libntech/libutils/ -I libntech/libcompat/ +json_CFLAGS = -Os -fdata-sections -ffunction-sections +json_LDFLAGS = $(STRIP_LDFLAGS) ``` To test that this works we can add a minimal C file: `json.c`: ```c int main(int argc, char *argv[]) { return 0; } ``` And indeed, making this program and running it works. ```sh $ make json gcc -DHAVE_CONFIG_H -I. -I libntech/libutils/ -I libntech/libcompat/ -g -O2 -MT json-json.o -MD -MP -MF .deps/json-json.Tpo -c -o json-json.o `test -f 'json.c' || echo './'`json.c mv -f .deps/json-json.Tpo .deps/json-json.Po /bin/sh ./libtool --tag=CC --mode=link gcc -g -O2 -o json json-json.o libntech/libutils/.libs/libutils.la libtool: link: gcc -g -O2 -o json json-json.o libntech/libutils/.libs/libutils.a -lpcre2-8 -lssl -lcrypto -lyaml -ldl -lrt -lm $ ./json $ echo $? 0 ``` ## Speak JSON? Yes. Next up, let's use libntech libraries to read a JSON file or stdin and parse it. From `file_lib.h` we use `safe_open()`, `PathExists()`, `FileReadFromFd()` to read from stdin or a path. These functions definitely save us from having to do far more low-level programming and error condition checking. From `json.h` we use `JsonParse()` to parse the content and return a non-zero exit code if that fails. ```sh $ ./json /dev/null ID "fcc93df9746b831206140f1c4a3fa0bd25cbc389a6aa1de40353024c7240eae1" ID "3552cd1635e85bc278358b12050887f5b28893d956f87744ef47a731ef0c32e8" ++ ./json q '*/Id' podman.ps.json 2>/dev/null Id "ece583449836c061d7f97b75f142a6a0e38a4ea4583227222921cb378859df83" ++ ./json q '*/Names' podman.ps.json 2>/dev/null Names ["agitated_aryabhata"] ``` ## Slight refinement Notice above that the `Names` value is an array of strings. I am going to assume that generally a container only has one name so not sure why this is an array but let's give our tool the ability to grab the first item in the array and show that instead for ease of use later. I thought using the subscript notation from C using `[n]` and also from the Plan 9 shell [rc](http://doc.cat-v.org/plan_9/4th_edition/papers/rc) using `(n)` would be a nice combination as both seem reasonable to me. To bring this in we can use `regex.h` from libntech and the fantastically helpful and code-reducing `StringMatchCaptures()` function. Wow! I don't really want to learn [PCRE2](https://pcre2project.github.io/pcre2/) today, maybe another day. And with this addition feature we can get the name of the container out in a much more usable form without the surrounding square brackets: ```sh ++ ./json q '*/Names[0)' podman.ps.json 2>/dev/null Names "agitated_aryabhata" ``` ## UNIX philosophy One final note: I tried to make this tool usable with other tools by outputing the results of the `query` command with a tab delimiter so our quoted values work with the `cut` command `:)`. ```sh $ ./json q docker.ps.json '*/Status' 2>/dev/null Status "Up 13 days" Status "Up 3 weeks" [libre - cfengine-agent-is-in-2026-05-28-libntech-and-json-command] $ ./json q docker.ps.json '*/Status' 2>/dev/null | cut -f2 "Up 13 days" "Up 3 weeks" ``` ## Video The video recording is available on YouTube: {{< youtube KwCMURzl3O8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [libntech repository](https://github.com/NorthernTechHQ/libntech) - [json-cmd repository](https://github.com/craigcomstock/json-cmd) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.24.4 and 3.27.1 released" url: /blog/2026/cfengine-3-24-4-and-3-27-1-released/index.md date: "2026-05-08" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.24.4 and 3.27.1\! These patch releases contain bug fixes and dependency updates. ## Changes As these are patch releases for long term supported (LTS) branches of CFEngine, there are no new major features included. ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.27.1 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-core.html) - [3.27.1 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-enterprise.html) - [3.27.1 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.24.4 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-core.html) - [3.24.4 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-enterprise.html) - [3.24.4 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades We have updated the following dependencies: | CFEngine version | 3.24.3 | **3.24.4** | 3.27.0 | **3.27.1** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.66 | 2.4.66 | 2.4.66 | 2.4.66 | | [APR](https://apr.apache.org/) | 1.7.6 | 1.7.6 | 1.7.6 | 1.7.6 | | [apr-util](https://apr.apache.org/) | 1.6.3 | 1.6.3 | 1.6.3 | 1.6.3 | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.12 | 3.12 | 3.12 | 3.12 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.52.0 | **2.54.0** | 2.52.0 | **2.54.0** | | [leech](https://github.com/larsewi/leech/releases) | - | - | 0.2.0 | 0.2.0 | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.2 | 2.3.2 | 2.3.2 | 2.3.2 | | [libattr](https://download.savannah.gnu.org/releases/attr) | 2.5.2 | 2.5.2 | 2.5.2 | 2.5.2 | | [libcurl](https://curl.se/download.html) | 8.17.0 | 8.17.0 | 8.17.0 | 8.17.0 | | [libcurl-hub](https://curl.se/download.html) | 8.17.0 | 8.17.0 | 8.17.0 | 8.17.0 | | [libexpat](https://libexpat.github.io/) | 2.7.3 | **2.8.0** | 2.7.3 | **2.8.0** | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | 2.5.1 | 2.5.1 | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.18 | **1.19** | 1.18 | **1.19** | | [librsync](https://github.com/librsync/librsync/releases) | - | - | 2.3.4 | 2.3.4 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.15.1 | **2.15.3** | 2.15.1 | **2.15.3** | | [LibYAML](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | 0.2.5 | 0.2.5 | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.33 | **0.9.35** | 0.9.33 | **0.9.35** | | [nghttp2](https://nghttp2.org/) | - | - | 1.68.0 | **1.69.0** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.10 | **2.6.13** | 2.6.10 | **2.6.13** | | [OpenSSL](https://openssl.org/) | 3.6.0 | **3.6.2** | 3.6.0 | **3.6.2** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | 10.47 | 10.47 | 10.47 | 10.47 | | [PHP](https://php.net/) | 8.3.28 | **8.3.30** | 8.5.1 | **8.5.5** | | [PostgreSQL](https://www.postgresql.org/) | 16.11 | **16.13** | 18.1 | **18.3** | | [pthreads-w32](https://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | 2-9-1 | 2-9-1 | | [rsync](https://download.samba.org/pub/rsync/) | 3.4.1 | **3.4.2** | 3.4.1 | **3.4.2** | | [SASL2](https://www.cyrusimap.org/sasl/) | 2.1.28 | 2.1.28 | 2.1.28 | 2.1.28 | | [zlib](https://www.zlib.net/) | 1.3.1 | **1.3.2** | 1.3.1 | **1.3.2** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.24/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.27.1 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.24.4 list ``` ```command cf-remote --version 3.27.1 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.21 LTS Please note that support for CFEngine 3.21 LTS series ended in January. If you are on 3.21 or an older unsupported version, please [contact us](/contact) for help with upgrading or to discuss an extended support contract. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "May the 4th be with you and your data" url: /blog/2026/smartctl-inventory/index.md date: "2026-05-04" -------------------------------------------------------------------------------- Drive failures are a matter of _when_, not _if_. The good news is that most modern drives warn you before they fail, using [S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology)](https://en.wikipedia.org/wiki/Self-Monitoring,_Analysis_and_Reporting_Technology). The challenge is collecting that data across a fleet and making it actionable. The new [inventory-smartctl module](https://build.cfengine.com/modules/inventory-smartctl/) makes this straightforward with a single `cfbs add`. Once installed, the module auto-detects all storage devices, caches their SMART data, and exposes it as inventory attributes in Mission Portal. ## What gets inventoried The module exposes several inventory attributes, broken into categories. ### Overall status The `SMART status` attribute provides a rolled-up health indicator for each host: - `OK` - all detected drives report PASSED - `DEGRADED` - one or more drives report FAILED - `SMARTCTL_MISSING` - smartctl is not installed Use this attribute to filter on in Mission Portal when looking for hosts that need attention. A host with `DEGRADED` status means at least one drive is predicting imminent failure. ### Per-drive attributes Each detected drive gets individual entries for: - **SMART drive health** -- `PASSED` or `FAILED` per device (e.g. `/dev/sda: PASSED`) - **SMART drive model** -- drive model identifier (e.g. `/dev/sda: Samsung SSD 870 EVO`) - **SMART drive temperatures (C)** -- current temperature in Celsius - **SMART drive power-on hours** -- cumulative runtime, useful for tracking drive age and warranty For NVMe devices, you also get: - **SMART NVMe available spare** -- remaining spare blocks as a percentage (low values indicate approaching end of life) - **SMART NVMe percentage used** -- drive life consumed based on the manufacturer's endurance rating - **SMART NVMe media errors** -- uncorrectable media error count (any non-zero value warrants investigation) ### Alerts A special `SMART failed drives` attribute is only present on hosts where at least one drive has a FAILED health status. This makes it easy to build alerts or compliance reports that target only the hosts that actually need attention. Try out the module and let us know what you think on [Github Discussions](https://github.com/cfengine/core/discussions) or on the [help mailing list](https://groups.google.com/g/help-cfengine). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 60 - Improved package management on Enterprise Linux" url: /blog/2026/the-agent-is-in-episode-60/index.md date: "2026-04-30" -------------------------------------------------------------------------------- > We’re 60 episodes in, and today we’re getting back to one of the most fundamental tasks in systems management, package management. In this episode we look at three new dnf-related improvements for managing packages on Enterprise Linux (Red Hat, Rocky Linux, AlmaLinux). We walk through the new `dnf` and `dnf_group` package modules, and the `appstreams` custom promise type. ## Why new package modules? The existing `yum` package module works, but it shells out to run commands. The new `dnf` module takes a different approach: it uses the dnf and rpm Python libraries directly. This matters for security, reliability, and performance on modern Enterprise Linux where dnf has replaced yum as the native package manager. Both `dnf` and `dnf_group` are merged into the maintained branches (3.27.x and 3.24.x) but not yet the default for any platform, you will need to explicitly specify `package_module => dnf` or `package_module => dnf_group`. The `appstreams` custom promise type is available from [CFEngine Build](https://build.cfengine.com). ## The `dnf` package module The `dnf` module handles individual packages: ```cfengine3 bundle agent dnf_examples { packages: "vim-enhanced" policy => "present", package_module => dnf, version => "8.2.2637-23.el9_7.1"; "httpd" policy => "present", version => "latest", package_module => dnf; "emacs-nox" policy => "present", package_module => dnf; } ``` To integrate it before the next release, you need the [`dnf.mustache` module](https://github.com/cfengine/masterfiles/blob/master/modules/packages/vendored/dnf.mustache) and the [associated body definition](https://github.com/search?q=repo%3Acfengine%2Fmasterfiles%20path%3Alib%2Fpackages.cf%20%2Fpackage_module%20dnf%5Cb%2F&type=code). ## The `dnf_group` package module The `dnf_group` package module lets you manage a curated set of packages as a single unit. ```cfengine3 bundle agent dnf_group_example { packages: "development" policy => "present", package_module => dnf_group, options => { "group_package_types=mandatory,default,optional", "install_weak_deps=false" }, version => "latest"; } ``` The `options` attribute controls which package types within the group get installed (mandatory, default, optional) and whether weak dependencies are pulled in. This gives you fine-grained control over what a group actually installs on your systems. ## AppStreams promise type AppStreams help to solve the issue where you need PHP 8.3 on one set of hosts and PHP 8.1 on another, from the same OS repositories. AppStreams let multiple versions coexist and you pick which stream to enable. The `appstreams` promise type is available via CFEngine Build: ```bash cfbs add promise-type-appstreams ``` Or using the shorter alias: ```bash cfbs add appstreams ``` Then in policy: ```cfengine3 bundle agent appstreams_example { appstreams: "php" state => "installed", stream => "8.3", profile => "minimal", options => {"install_weak_deps=false"}; } ``` This ensures PHP is installed from the 8.3 stream using the minimal profile. Unlike the package modules, the `appstreams` promise type is a custom module distributed through [CFEngine Build](https://build.cfengine.com/modules/promise-type-appstreams/), not vendored in masterfiles. ## Video The video recording is available on YouTube: {{< youtube Xpe2NPjI_Zw >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [Improved dnf support (blog post)](/blog/2026/improved_dnf_related_functionality/) - [`dnf` package module documentation](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/#dnf) - [`dnf_group` package module documentation](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/#dnf_group) - [`appstreams` promise type on CFEngine Build](https://build.cfengine.com/modules/promise-type-appstreams/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Improved dnf support" url: /blog/2026/improved_dnf_related_functionality/index.md date: "2026-04-23" -------------------------------------------------------------------------------- Three notable dnf related improvements making it easier to manage packages on modern Enterprise Linux based systems (Red Hat, Rocky Linux, Alma Linux, etc ...) have been merged recently. - [`dnf` package module](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/#dnf) - Manage packages using dnf - [`dnf_group` package module](https://docs.cfengine.com/docs/lts/reference/promise-types/packages/#dnf_group) - Manage package groups using dnf - [`appstreams` promise type](https://build.cfengine.com/modules/promise-type-appstreams/) - Manage application stream modules and profiles ## `dnf` package module The new `dnf` package module unlike the existing yum module does not perform any shell operations, instead it leverages only the dnf and rpm python libraries for querying and modifying the system. Note that `dnf` is not yet the _default_ package module for any platform it must be explicitly specified as seen in these examples. ```cfengine3 bundle agent dnf_examples { packages: "vim-enhanced" policy => "present", package_module => dnf, version => "8.2.2637-23.el9_7.1"; "httpd" policy => "present", version => "latest", package_module => dnf; "emacs-nox" policy => "present", package_module => dnf; } ``` To integrate the unreleased dnf package module into your policy set you will need both the [`dnf.mustache` module](https://github.com/cfengine/masterfiles/blob/master/modules/packages/vendored/dnf.mustache) and its [associated `body package_module dnf`](https://github.com/search?q=repo%3Acfengine%2Fmasterfiles%20path%3Alib%2Fpackages.cf%20%2Fpackage_module%20dnf%5Cb%2F&type=code). ```cfengine3 body package_module dnf # @brief Define details used when interfacing with dnf { query_installed_ifelapsed => "$(package_module_knowledge.query_installed_ifelapsed)"; query_updates_ifelapsed => "$(package_module_knowledge.query_updates_ifelapsed)"; interpreter => "$(sys.bindir)/cfengine-selected-python"; } ``` ## `dnf_group` package module {#dnf-group-package-module} The new `dnf_group` module introduces package group management. Like the `dnf` package module, utilization of the `dnf_group` package module requires explicit specification. ```cfengine3 bundle agent dnf_group_example { packages: "development" policy => "present", package_module => dnf_group, options => { "group_package_types=mandatory,default,optional", "install_weak_deps=false" }, version => "latest"; } ``` To integrate the unreleased `dnf_group` package module into your policy set you will need both the [`dnf_group.mustache` module](https://github.com/cfengine/masterfiles/blob/master/modules/packages/vendored/dnf_group.mustache) and its [associated `body package_module dnf_group`](https://github.com/search?q=repo%3Acfengine%2Fmasterfiles+path%3Alib%2Fpackages.cf+dnf_group&type=code). ```cfengine3 body package_module dnf_group { query_installed_ifelapsed => "$(package_module_knowledge.query_installed_ifelapsed)"; query_updates_ifelapsed => "$(package_module_knowledge.query_updates_ifelapsed)"; interpreter => "$(sys.bindir)/cfengine-selected-python"; } ``` ## `appstreams` promise type AppStreams allow multiple versions of software components to coexist on the same system. For example, you might have Node.js 12, 14, and 16 available and you can enable the specific stream you need. The `appstreams` promise type ([promise-type-appstreams](https://build.cfengine.com/modules/promise-type-appstreams/)) is available as a custom module via CFEngine Build (cfbs). Add it to your cfbs project by running `cfbs add promise-type-appstreams`. This example ensures php is installed from the 8.3 stream using the minimal profile and not installing weak dependencies. ```cfengine3 bundle agent appstreams_example { appstreams: "php" state => "installed", stream => "8.3", # 8.1, 8.2 profile => "minimal", options => {"install_weak_deps=false"}; } ``` -------------------------------------------------------------------------------- title: "Change in behavior: findfiles() and directory trailing slashes" url: /blog/2026/change_in_behavior_findfiles_trailing_slash/index.md date: "2026-04-15" -------------------------------------------------------------------------------- CFEngine 3.24.4+, 3.27.1+, and 3.28.0+ include a change to how `findfiles()` handles trailing slashes on directory paths. This change restores trailing slashes to directory results, but with improved consistency compared to earlier versions. The new behavior ensures that directory paths always include a trailing slash, making them reliably distinguishable from file paths regardless of the glob pattern used. ## The behavior changes ### CFEngine 3.23.0 and earlier: Pattern-dependent behavior The presence of a trailing slash in the returned paths depended on whether the **glob pattern itself** included a trailing slash. If you use `findfiles("/path/*/")` (with trailing slash in pattern), the results include trailing slashes. If you use `findfiles("/path/*")` (without trailing slash in pattern), the results do not include trailing slashes. ```cfengine3 bundle agent main { vars: "testdir" string => "/tmp/testdir"; # Create a mix of directories and files "empty_dirs" slist => { "empty_dir" }; "dirs_with_files" slist => { "dir_with_files" }; "nested_dirs" slist => { "nested/level1/level2" }; "files" slist => { "file1.txt", "file2.txt" }; files: # Create empty directory "$(testdir)/$(empty_dirs)/." create => "true"; # Create directory with files "$(testdir)/$(dirs_with_files)/." create => "true"; "$(testdir)/$(dirs_with_files)/$(files)" create => "true"; # Create nested directory structure "$(testdir)/$(nested_dirs)/." create => "true"; "$(testdir)/$(nested_dirs)/deep_file.txt" create => "true"; # Create some files in the root test directory "$(testdir)/root_file.txt" create => "true"; methods: "test"; } bundle agent test { vars: # Test two patterns: with and without trailing slash "with_slash" slist => findfiles("$(main.testdir)/*/"); "without_slash" slist => findfiles("$(main.testdir)/*"); reports: "CFEngine: $(sys.cf_version)"; "=== Pattern: $(main.testdir)/*/ (with trailing slash) ==="; " with_slash: $(with_slash)"; "=== Pattern: $(main.testdir)/* (without trailing slash) ==="; " without_slash: $(without_slash)"; } ``` Running this policy on CFEngine 3.21.8 demonstrates the pattern-dependent behavior: ```bash podman run --rm \ -v /tmp/test-findfiles-dirs-with-slash.cf:/test.cf:ro,Z \ cfengine-test:3.21.8 \ /var/cfengine/bin/cf-agent -KIf /test.cf 2>&1 | grep "^R:" ``` ```text R: CFEngine: 3.21.8 R: === Pattern: /tmp/testdir/*/ (with trailing slash) === R: with_slash: /tmp/testdir/dir_with_files/ R: with_slash: /tmp/testdir/empty_dir/ R: with_slash: /tmp/testdir/nested/ R: === Pattern: /tmp/testdir/* (without trailing slash) === R: without_slash: /tmp/testdir/dir_with_files R: without_slash: /tmp/testdir/empty_dir R: without_slash: /tmp/testdir/nested R: without_slash: /tmp/testdir/root_file.txt ``` ### CFEngine 3.24.0 through 3.24.3, and 3.27.0: No trailing slashes The trailing slash is no longer included in the returned paths, **regardless of whether the glob pattern includes a trailing slash**. Additionally, both patterns (with and without trailing slash) return identical results - they match both directories and files, but strip trailing slashes from all paths. This breaks policies that depended on the trailing slash being present when using patterns like `findfiles("/path/*/")`, and also breaks the distinction between patterns with and without trailing slashes. ```bash podman run --rm \ -v /tmp/test-findfiles-dirs-with-slash.cf:/test.cf:ro,Z \ cfengine-test:3.24.0 \ /var/cfengine/bin/cf-agent -KIf /test.cf 2>&1 | grep "^R:" ``` ```text R: CFEngine: 3.24.0 R: === Pattern: /tmp/testdir/*/ (with trailing slash) === R: with_slash: /tmp/testdir/dir_with_files R: with_slash: /tmp/testdir/empty_dir R: with_slash: /tmp/testdir/nested R: with_slash: /tmp/testdir/root_file.txt R: === Pattern: /tmp/testdir/* (without trailing slash) === R: without_slash: /tmp/testdir/dir_with_files R: without_slash: /tmp/testdir/empty_dir R: without_slash: /tmp/testdir/nested R: without_slash: /tmp/testdir/root_file.txt ``` ### CFEngine 3.24.4+, 3.27.1+, and 3.28.0+: Always trailing slashes for directories The unintentional regression (tracked as ) has been fixed in master and backported to the 3.24.x and 3.27.x LTS releases. However, the new behavior is **not identical** to pre-3.24.0: instead of the trailing slash depending on the glob pattern, `findfiles()` now **always** returns directory paths with trailing slashes and file paths without trailing slashes, **regardless of the glob pattern used**. Both patterns (with and without trailing slash) now return identical results. This is arguably more consistent and predictable than the original pattern-dependent behavior, as it makes the presence of a trailing slash a reliable indicator of whether a path is a directory or a file, rather than depending on which glob pattern was used. ```bash # Test with master using a container (adjust based on available nightly builds) podman run -d --name cfengine-master-test ubuntu:24.04 sleep infinity >/dev/null 2>&1 podman cp /tmp/cfengine-nova_3.28.0a.e56ed82f7~195.ubuntu24_amd64.deb cfengine-master-test:/tmp/ 2>/dev/null podman exec cfengine-master-test apt-get update -qq >/dev/null 2>&1 podman exec cfengine-master-test apt-get install -y -qq /tmp/cfengine-nova_3.28.0a.e56ed82f7~195.ubuntu24_amd64.deb >/dev/null 2>&1 podman cp /tmp/test-findfiles-dirs-with-slash.cf cfengine-master-test:/tmp/test.cf 2>/dev/null podman exec cfengine-master-test /var/cfengine/bin/cf-agent -KIf /tmp/test.cf 2>&1 | grep "^R:" podman rm -f cfengine-master-test >/dev/null 2>&1 ``` ```text R: CFEngine: 3.28.0a.e56ed82f7 R: === Pattern: /tmp/testdir/*/ (with trailing slash) === R: with_slash: /tmp/testdir/dir_with_files/ R: with_slash: /tmp/testdir/empty_dir/ R: with_slash: /tmp/testdir/nested/ R: with_slash: /tmp/testdir/root_file.txt R: === Pattern: /tmp/testdir/* (without trailing slash) === R: without_slash: /tmp/testdir/dir_with_files/ R: without_slash: /tmp/testdir/empty_dir/ R: without_slash: /tmp/testdir/nested/ R: without_slash: /tmp/testdir/root_file.txt ``` ## Impact on policy The inconsistent handling of trailing slashes across versions creates challenges for policies that rely on string matching or filtering directory paths. This is particularly problematic when using `filter()` to exclude specific directories. Consider this example from the MPF `modules_presence` bundle: ```cfengine3 bundle agent modules_presence { vars: "_override_dir" string => "$(this.promise_dirname)/../../modules/packages/"; "_package_paths_tmp" slist => findfiles("$(_override_dir)*"); "_package_paths" slist => filter("$(_override_dir)vendored", _package_paths_tmp, "false", "true", 999); } ``` The filter is designed to exclude the `vendored` subdirectory from the list of package paths. However, the behavior varies across CFEngine versions: - ****3.23.0 and earlier****: When `findfiles()` used a pattern with a trailing slash, it returned `/path/to/modules/packages/vendored/`. The filter pattern `$(_override_dir)vendored` matched and excluded it correctly. - ****3.24.0 through 3.27.0****: Returns `/path/to/modules/packages/vendored` without a trailing slash, breaking the filter match. This caused the `vendored` directory to be included, leading to recursion errors when CFEngine attempted to copy overlapping source and destination directories. - ****3.24.4+, 3.27.1+, 3.28.0+****: Returns `/path/to/modules/packages/vendored/` with a trailing slash for directories, restoring correct filtering behavior. ## Example version-aware workaround We ultimately fixed this differently in the MPF, but wanted to share a pattern leveraging `cf_version*` functions to support both old and new behavior across CFEngine versions, use version detection to conditionally add the directory separator: ```cfengine3 vars: "_package_paths" with => ifelse( or( cf_version_between("3.24.0", "3.24.3"), cf_version_between("3.25.0", "3.27.0") ), "$(const.dirsep)", # Add separator for versions without trailing slash "" # No separator for versions with trailing slash ), slist => filter("$(_override_dir)vendored$(with)", _package_paths_tmp, "false", "true", 999); ``` This pattern uses `cf_version_between()` to detect versions affected by the behavior change (3.24.0 through 3.27.x) and conditionally adds `$(const.dirsep)` to the filter pattern only when needed. This ensures the filter works correctly regardless of which CFEngine version is running the policy. The fix for CFE-4623 has been backported and released in 3.24.4+, 3.27.1+, and 3.28.0+. This workaround continues to work correctly as it only adds the separator for the specific versions that need it (3.24.0-3.24.3, 3.26.0, 3.27.0). If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 59 - Extending update policy for greater efficiency and autonomy" url: /blog/2026/the-agent-is-in-episode-59/index.md date: "2026-03-26" -------------------------------------------------------------------------------- When using CFEngine with the [Masterfiles Policy Framework](https://docs.cfengine.com/docs/lts/reference/masterfiles-policy-framework/) there are two standard "stages" involved in periodic maintenance: update the policy (update.cf) and evaluated the policy (promises.cf). In a standard install the [cf-execd](https://docs.cfengine.com/docs/lts/reference/components/cf-execd/) component periodically runs first the update policy and then the policy proper. We have talked in the past about [Extending the CFEngine Policy Update Procedure](blog/2016/extending-the-cfengine-policy-update-procedure/) as well as [Writing a cfbs module for your custom policy update](blog/2022/example-custom-policy-update-cfbs-module/). While both of these previous strategies are very useful I have a couple of different itches to scratch this time: - only copy files to certain clients depending on what role they play - copy files which don't match the `input_name_patterns` or that have no extension as part of their name - what happens if the policy server is not available? For these three purposes it is most easy to write a new bundle and have it evaluated after the standard update policy is run. This easily be done using an [Augment](https://docs.cfengine.com/docs/lts/reference/language-concepts/augments/#top) available with the MPF: [Evaluate additional bundles during update](https://docs.cfengine.com/docs/lts/reference/masterfiles-policy-framework/#evaluate-additional-bundles-during-update-bundlesequence). Specifying a bundle name in this augment allows us to append additional bundles to evaluate at the end of standard update policy. ## How does update policy copy files to clients? The update policy only copies certain files from the policy server to each client. It decides which files based on an internal variable called `input_name_patterns`. We have augments to replace or extend this variable: [Override files considered for copy during policy updates](https://docs.cfengine.com/docs/lts/reference/masterfiles-policy-framework/#override-files-considered-for-copy-during-policy-updates) and [Extend files considered for copy during policy updates](https://docs.cfengine.com/docs/lts/reference/masterfiles-policy-framework/#extend-files-considered-for-copy-during-policy-updates) And while we could add new extensions to this list that strategy would not solve our two issues: files without extensions and files that should only be distributed to certain clients. ## Custom update policy I think the easiest way to accomplish this task is to use [CFEngine Build](https://docs.cfengine.com/docs/master/getting-started/02-modules-from-cfengine-build/), add a json file with our augment and a policy file as well as some of those tasty special files to transfer to special hosts. Adding this to an existing project is easy. First create a sub-folder for this work ```sh mkdir custom-update mkdir -p files/special files/common echo "only tell special clients" > files/special/this-is-kind-of-secret echo "yeah, I am an open book (file)" > files/common/this-file-has-no-secrets ``` We add both the policy file to `update_inputs` as well as the bundle name to `control_common_update_bundlesequence_end` in custom-update/def.json: ```json { "variables": { "default:def.update_inputs": { "value": [ "services/cfbs/custom-update/custom_update_bundle.cf" ] }, "default:def.control_common_update_bundlesequence_end": { "value": [ "custom_update_bundle" ] } } } ``` And finally some policy. One important thing to remember is that the update policy does NOT use the [standard library](https://docs.cfengine.com/docs/lts/reference/masterfiles-policy-framework/lib/) so if you want to leverage some helpful bundles and bodies you will have to dig a bit in the [internals](https://github.com/cfengine/masterfiles/blob/master/cfe_internal/update). Most common items can be prefixed with a `u_` e.g. `u_results`, `u_mog`, etc. ```cf3 bundle agent custom_update_bundle { classes: "special_host" expression => fileexists("/etc/cfengine-special-host.flag"); vars: "files_dir" string => "${sys.masterdir}/services/cfbs/custom-update/files"; files: special_host:: "${sys.inputdir}/files/special" perms => u_mog("0644", "root", "root" ), copy_from => u_remote_dcp_missing_ok("${files_dir}/special", "${sys.policy_hub}"), depth_search => u_recurse("inf"); any:: "${sys.inputdir}/files/common" perms => u_mog("0644", "root", "root" ), copy_from => u_remote_dcp_missing_ok("${files_dir}/common", "${sys.policy_hub}"), depth_search => u_recurse("inf"); } ``` At this point, just add the `custom-update` folder as a module being careful to NOT accept the default of adding `custom_update_bundle` bundle to the (default policy) bundle sequence. ```sh $ cfbs add custom-update/ Which bundle should be evaluated (added to bundle sequence)? 1. ./custom-update/custom_update_bundle.cf:custom_update_bundle (default) 2. (None) [1/2]: 2 ``` Adding the `custom-update` directory as a module has the benefit of copying ALL of the files inside to the masterfiles policy set which includes our odd special and common files with no extensions. Now we can configure a policy server to use this policy and bootstrap our special and common clients and see the results! ## Setup an environment I like to use lower-resource hardware so setting up an environment with 3 hosts: 1 policy server and two clients calls for the use of containers. A community member contributed instructions for [Installing Community Using Containers](https://docs.cfengine.com/docs/lts/getting-started/01-installation/general-installation/installation-community-containerized/#top). I had to adjust slightly to use redhat 10 instead of redhat 9 as our latest LTS: 3.27.0 requires a newer selinux policy version to install. (I have a pending [pull request](https://github.com/cfengine/documentation/pull/3612) to fix that detail.) ```sh torizon@verdin-imx8mp-06849243:~$ docker-compose -f cfengine-community-3.yaml up -d [+] Running 4/4 ✔ Network cfengine-demo_control-plane Created 0.1s ✔ Container cfengine-policy-server Healthy 0.1s ✔ Container cfengine-demo-client-1-1 Started 0.1s ✔ Container cfengine-demo-client-2-1 Started 0.1s ``` ## Install and configure masterfiles-stage Since I am using a Build project and community edition I need to install and setup [masterfiles-stage](https://github.com/cfengine/core/tree/master/contrib/masterfiles-stage) manually. In order to synchronize and build a Build project from a git repository we must install `git` and `rsync` on the policy server. I did this manually with: ```sh sudo dnf install -y rsync git ``` Following the instructions I download the install script and run it: ```sh [root@ba1a7e623d16 ~]# ./install-masterfiles-stage.sh Now, edit /opt/cfengine/dc-scripts/params.sh to conigure your upstream repository. Then, run '/var/cfengine/httpd/htdocs/api/dc-scripts/masterfiles-stage.sh --DEBUG' to test deployment ``` In GitHub I add a fine-grained access token. I use this token value as the `GIT_PASSWORD` variable in `params.sh` mentioned above. `GIT_URL` is set to `https://github.com/cfengine/play-cfbs` and `GIT_REFSPEC` is set to `extend-update-policy` Finally `VCS_TYPE` is set to `GIT_CFBS` so that masterfiles-stage knows to grab a git repo as well as run `cfbs build` on the repo to deploy the policy. After modifying `params.sh` and running the masterfiles-stage script again my policy is in place and the clients will pick up the policy. ## Video {#video} The video recording is available on YouTube: {{< youtube OZgKpRVnZdc >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Post-recording discussion about AI One of us wondered if others felt awkward often talking about AI and we agreed. We also agreed that a lot is going on in the AI/LLM realm and that learning, considering and reflecting about this topic is worth the effort. Some recent articles we found helpful are: - https://www.media.mit.edu/projects/your-brain-on-chatgpt/overview/ - https://shumer.dev/something-big-is-happening - https://www.cato.org/commentary/something-big-happening-ai-thats-only-thing-matt-shumer-got-right ## Links {#links} - Build project for this episode: [extend-update-policy](https://github.com/cfengine/play-cfbs/tree/extend-update-policy) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 58 - From monolith to modules with cfbs" url: /blog/2026/the-agent-is-in-episode-58/index.md date: "2026-02-26" -------------------------------------------------------------------------------- > The standard process for managing that monolithic set, it is a fair amount of > git diffing. It's not hard once you get used to it, but it's still a lot to do > and read. In this episode we take a monolithic CFEngine policy set, the kind most of us have been managing for years in production, and turn it into `cfbs`-managed project using `cfbs convert`. We start with `cfbs analyze` to see what we're working with, walk through the interactive conversion, and finish with running `cfbs update` to jump from masterfiles 3.24.0 to 3.27 in seconds. ## Show Notes {#show-notes} ### The problem with monolithic policy sets {#the-problem-with-monolithic-policy-sets} If you've been running CFEngine for a while, your masterfiles directory probably started as a clean copy of the Masterfiles Policy Framework, then someone added custom policy files. Someone else modified `def.cf` to change the agent run schedule. Now try upgrading. You're diffing files, hoping you don't clobber a local tweak or miss an upstream fix. The [upgrade documentation](https://docs.cfengine.com/docs/lts/getting-started/01-installation/upgrading/) is thorough but it's a lot to work through. ### Look before you leap {#look-before-you-leap} `cfbs analyze` gives you a picture of what you're working with before you commit to anything. Running it against our policy set, it told us: - Based on **masterfiles 3.24.0** - `controls/def.cf` has been **modified** (the schedule change) - **Stale files from old CFEngine versions** show leftover version-split library directories from the 3.6/3.7 era - Several **custom** files that aren't part of upstream at all This is the same analysis that `cfbs convert` uses under the hood to figure out what to keep, what to prune, and what to turn into patches. ### cfbs convert {#cfbs-convert} One thing to know about `cfbs convert`: it's picky about the starting layout. It wants a directory containing a single subdirectory named `masterfiles-`. That directory becomes the root of your new cfbs project. So we copied our policy set into a fresh directory as `masterfiles-acme` and ran `cfbs convert` from the parent. 1. It found `promises.cf` and detected **masterfiles 3.24.0** 2. It initialized a git repo and created a `cfbs.json` with 3.24.0 as the base 3. It picked up our custom policy files 4. It **pruned all the unmodified vendored files**, stripping the monolith down to just our stuff 5. For `controls/def.cf`, it showed the diff: someone had changed the agent run schedule to a 10-minute interval. We chose **"keep as patch file"**, which generated a `.patch` file and wired it up as a `custom-masterfiles-patches` module What's left is a clean `cfbs.json` that pulls in masterfiles 3.24.0, layers our custom policy on top, and applies the schedule patch. Everything else we never touched and it's now gone from our repo because cfbs manages those now. ### cfbs update {#cfbs-update} We ran `cfbs build` and got a working policy set with our patches applied. Good, but best part was: ```bash cfbs update ``` Masterfiles jumped from 3.24.0 to 3.27. It noticed the build steps had changed between versions, asked if we wanted the new ones, and after a quick `cfbs build` we were running 3.27 with our schedule patch still cleanly applied. That's the whole point. An upgrade that used to mean a git diff marathon, done in one command. ### Trailing commas in CFEngine lists {#trailing-commas-in-cfengine-lists} An audience question about a trailing comma in the schedule list is a useful syntax reminder. In CFEngine policy, trailing commas in lists are perfectly fine: ```cfengine3 vars: "schedule" slist => { "Min00_05", "Min15_20", "Min30_35", "Min45_50", # <-- valid CFEngine syntax }; ``` Just don't do this in JSON, your `def.json` augments files won't tolerate it. ## Video {#video} The video recording is available on YouTube: {{< youtube liZVS6Ervyc >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [cfbs on GitHub](https://github.com/cfengine/cfbs) - [Upgrading CFEngine and the MPF](https://docs.cfengine.com/docs/master/guide-upgrading.html) - [cfbs documentation](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-cfbs.html) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CVE-2026-24710, CVE-2026-24711 & CVE-2026-24712 - Injection & broken access control" url: /blog/2026/cve-2026-24710-and-cve-2026-24711-and-cve-2026-24712/index.md date: "2026-02-09" -------------------------------------------------------------------------------- We have recently discovered and fixed multiple security issues in CFEngine. These discoveries were made by ethical hackers as a part of our HackerOne bug bounty program. All of these issues can be remediated by upgrading to CFEngine 3.27.0, 3.24.3, 3.21.8, or later versions. We have no indications of these issues being exploited or known outside of the company and the security researchers that reported them. All of these issues require authentication / some initial level of access - they cannot be exploited on their own, without first compromising an account, or similar. ## CVE-2026-24710 - Missing input sanitization / injection in Mission Portal Multiple instances were identified where Mission Portal was susceptible to injection (JavaScript XSS, SQL injection, OS command injection), through not correctly sanitizing user input and/or not correctly escaping it before presenting it. Thank you to [Tahsin Akbar Ohi](https://hackerone.com/royal_coder) and [i-forgot-it](https://hackerone.com/i-forgot-it) for discovering and responsibly disclosing these issues. ### Impact These issues could enable an attacker to inject malicious code into Mission Portal and: - Run JavaScript when another user, for example admin, visits a page (XSS). - Run shell commands on the hub (Command injection). - Extract information through timing based methods (blind SQL injection). ### Remediation [CVE-2026-24710](https://www.cve.org/CVERecord?id=CVE-2026-24710) affects CFEngine Enterprise 3.26.0, 3.24.2, 3.21.7, and earlier versions. The issues have been fixed in 3.27.0, 3.24.3, 3.21.8, and later versions. If you are using one of the affected versions, we recommend upgrading. Our documentation has details on how to upgrade CFEngine: https://docs.cfengine.com/docs/lts/getting-started/01-installation/upgrading ## CVE-2026-24711 - Broken access control in Mission Portal Issues around access control could enable an attacker with access to a low privilege user account to get access to more information than they should have, and in the most severe case enable them to escalate their privileges to become an admin. Thank you to [Tahmid Akbar Omim](https://hackerone.com/imperial_coder) for discovering and responsibly disclosing these issues. ### Impact The vulnerability requires authentication, it cannot be exploited without first compromising a user account. An attacker with a compromised low-privilege account could use this to bypass access control, escalate their own privileges, and take over the hub and ultimately the whole infrastructure managed by that hub. ### Remediation [CVE-2026-24711](https://www.cve.org/CVERecord?id=CVE-2026-24711) affects CFEngine Enterprise 3.26.0, 3.24.2, 3.21.7, and earlier versions. The issues have been fixed in 3.27.0, 3.24.3, 3.21.8, and later versions. If you are using one of the affected versions, we recommend upgrading. Our documentation has details on how to upgrade CFEngine: https://docs.cfengine.com/docs/lts/getting-started/01-installation/upgrading ## CVE-2026-24712 - Missing input sanitization / command injection in CFEngine policy language In CFEngine policy language, you can write policy to manage files, users, packages, groups, and so on. Each of these resources are managed by their respective promise types. You'd expect that `files` promises can only be used to manage files, `users` only to manage users and so on. Normally, the risk of someone "injecting" commands into policy is not a concern, it's a programming language and there is a `commands` promise type for running arbitrary commands. However, it can matter in some situations - if the policy uses data from an external source, and you expect it to only do one thing. The external source might have some access control mechanisms and assumptions, for example giving someone access to _only_ manage users. Thus, to account for the possibility of such cases (depending on what is in your custom policy) we have identified and fixed cases where it was possible to put (inject) shell commands into places where it should not be possible. Thank you to [Dipesh Thakur](https://hackerone.com/bughunter0xff) for discovering and responsibly disclosing these issues. ### Impact This issue is not exploitable in the default configuration, the main concern comes with custom policy which processes data from external sources and use it to manage users, packages, and similar. In such cases, you might have a system where you expect a user to only be able to delete users (for example), but through external data, an attacker could also run (inject) arbitrary shell commands. ### Remediation [CVE-2026-24712](https://www.cve.org/CVERecord?id=CVE-2026-24712) affects CFEngine 3.26.0, 3.24.2, 3.21.7, and earlier versions. Both CFEngine Enterprise and CFEngine Community are affected. It has been fixed in 3.27.0, 3.24.3, 3.21.8, and later versions. If you are using one of the affected versions, we recommend upgrading. Our documentation has details on how to upgrade CFEngine: https://docs.cfengine.com/docs/lts/getting-started/01-installation/upgrading ## Contact For help with upgrading or additional questions, please contact support at: https://support.northern.tech -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 57 - using ansible custom promise type to manage firewalld" url: /blog/2026/the-agent-is-in-episode-57/index.md date: "2026-01-29" -------------------------------------------------------------------------------- We here at CFEngine have seen the collaboration possibilities with Ansible for a long time. See our many [ansible related blog posts](/tags/ansible) including previously where I discussed our [promise-type-ansible module](/blog/2024/promise-type-ansible/) which enables you to run ansible playbooks from CFEngine policy. *You might ask why you would want to do such a thing?* ### We came up with one possible answer: what happens if you block ssh access to a host? Now you can certainly setup [ansible-pull](https://docs.ansible.com/projects/ansible/latest/cli/ansible-pull.html) but that requires configuring credentials and access to a repository. If you are already or interested in using both [CFEngine and Ansible together](/blog/2021/comparing-ansible-and-cfengine/) then this episode will interest you. ## Notes Previously we used the [promise-type-ansible](https://github.com/cfengine/play-cfbs/tree/promise-type-ansible) to patch our systems. This time we will leverage CFEngine's [Host specific data](https://docs.cfengine.com/docs/lts/web-ui/hosts/#host-specific-data) to easily manage whether to "lockdown" certain hosts or not. ## Copy/Paste/Modify Let's take the previous blog's framework of a `main.cf` and `site.yml` and adjust it to our purpose: locking down `sshd` with the `firewalld` ansible module. Previously I had worked with a Debian system but this time I will use [Rocky Linux](https://rockylinux.org/) so we must adjust the packages needed on each client: ``` cf3 vars: redhat:: "ansible_package" string => "ansible-core"; debian:: "ansible_package" string => "ansible"; packages: "${ansible_package}" classes => if_ok("ansible_package_installed"); ``` I wanted to drive the lock or not lock state from host specific data. By default the namespace for classes entered into the UI is `data` so a bit of mangling is needed to convert the `data:lockdown` class into something simple to use in policy: ```cf3 vars: # here we have to do some mangling to support CMDB since those classes are prefixed with the namespace "data:" "ansible_role_classes" slist => classesmatching(".*", "ansible_role"); "ansible_roles_normalized[${ansible_role_classes}]" string => string_replace("${ansible_role_classes}", "data:", ""); "ansible_roles" slist => getvalues("ansible_roles_normalized"); ``` When paired with some policy to render the inventory: ```cf3 files: "$(localhost_inventory)" create => "true", content => string_mustache("[all:vars] ansible_connection=local {{#-top-}}[{{.}}] localhost {{/-top-}}", "ansible_roles" ), classes => if_ok("localhost_inventory_created"); ``` We will arrive at inventory something like this for our rocky VM: ```sh [root@localhost vagrant]# cat /var/cfengine/state/localhost_inventory.ini [all:vars] ansible_connection=local [managed] localhost [lockdown] localhost ``` Finally, let's craft our `site.yml` to implement our lockdown (or not). ```yaml - hosts: lockdown roles: - lockdown - hosts: all:!lockdown roles: - notlockdown ``` So here I am leveraging the default role of `all` and the expression `!lockdown` to implement enabling sshd when the `lockdown` class is NOT defined. And for our roles we have: ```yaml # roles/lockdown/tasks/main.yml --- - name: lockdown firewalld: service: 'ssh' state: disabled permanent: true immediate: true offline: true ``` and ```yaml # roles/notlockdown/tasks/main.yml --- - name: notlockdown firewalld: service: 'ssh' state: enabled permanent: true immediate: true offline: true ``` Notice the difference is `state: enabled` or `state: disabled`. I certainly could have used various other ways to set the state either way but this way seemed a nice balance of simple and working. ## To Lock or not to Lock So now we configure our hub to use this build project, bootstrap our rocky-10 VM and sit back and enjoy the show. ```sh watch -n 5 'ssh vagrant@192.168.1.117 date' ``` At first all is well and we can SSH: ```sh Mon Jan 26 05:51:34 PM UTC 2026 ``` But then... we set the class in host info page to `lockdown` {{< img "images/lockdown-class-in-host-specific-data.png" "Host specific data on host info page where we add a lockdown class for the rocky-10 linux host">}} After a few minutes (CFEngineers can take their time usually) we see our rocky VM is unavailable. ```sh ssh: connect to host 192.168.1.117 port 22: Host is unreachable ``` And because we are using CFEngine which defaults to a "pull" architecture for updating policy even if sshd is locked down we can update the host info page and open up sshd again. :) To achieve that we can either remove the `lockdown` class or rename it to something easy to remember like `lockdownnope`. ## To Do I had to manually install a `ansible.posix` collection with ```sh ansible-galaxy collection install ansible.posix ``` We should create a package method and module for this. For a package method, something like: ```cf3 body package_method ansible_galaxy_collection { package_changes => "bulk"; package_name_convention => "$(name)"; package_delete_convention => "$(name)"; package_installed_regex => ".*"; package_list_name_regex => "^([a-z\.]*)\s.*"; package_list_version_regex => "^[a-z\.]*\s(.*)"; package_list_command => "ansible-galaxy collection list"; package_delete_command => 'echo "Deleting an Ansible Galaxy Collection is not supported"'; package_add_command => "ansible-galaxy collection install "; } ``` And usable like: ```cf3 packages: "ansible.posix" package_method => ansible_galaxy_collection, package_policy => "add"; ``` ## QED That's it! We showed how you can leverage the benefits of both CFEngine (default pull architecture) and Ansible (lots of modules supporting many things to manage). Thanks for your time. Be well! {{< youtube Fgmz2L-iEfs >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Build project for this episode: [ansible-firewalld-lockdown](https://github.com/cfengine/play-cfbs/tree/ansible-firewalld-lockdown) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.27 LTS released - Exploration" url: /blog/2026/cfengine-3-27-lts-released-exploration/index.md date: "2026-01-09" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.27.0\! The code word for this release is _exploration_. This release also marks an important event, the beginning of the 3.27 LTS series, which will be supported for 3 years. Several new features have been added since the release of CFEngine 3.24 LTS, in the form of non-LTS releases. In this blog post we'll highlight the most important features since the previous LTS release, even though some of them technically landed in intermediate non-supported releases. ## What's new ### 3.27 sneak peek webinar episode In the Halloween special of our monthly webinar, we looked at some of the new features coming in 3.27: {{< youtube IJs2t8JhP_8 >}} ### Explore your infrastructure with the CFEngine AI agent Inside Mission Portal, CFEngine users can now ask questions about their infrastructure and get answers based on the reporting data. This works by adding an integration with one of the supported AI providers. We currently support OpenAI, Anthropic and Google Gemini. Local / on-prem LLMs are also supported, via either Ollama or software providing an OpenAI compatible API. {{< img "images/ai-agent.png" "AI agent chat with question about number of hosts and operating systems.">}} Read more about this feature in our documentation: https://docs.cfengine.com/docs/3.27/web-ui/ai-agent/ ### CMDB redesign The CMDB has been redesigned to provide a more intuitive and user-friendly experience. The UI is now more oriented around what the user enters and wants to achieve, rather than the underlying data structure. {{< wideimg "images/cmdb-redesign.png" "The CMDB UI, showing a variable, a class, and an inventory attribute.">}} ### cfbs analyze and convert Although `cfbs` is a separate tool, with its own versioning, we thought it would make sense to mention some new features here. - `cfbs analyze` - Analyze your policy set, showing which version of the default policy it's based on, what files have custom modifications, and whether there are some unexpected mismatches (for example individual files from an older version). - `cfbs convert` - Convert a traditionally managed policy set to a CFEngine Build project. Traditionally, policy sets have been managed as "forks" of the default policy, with custom modifications on top. By converting to a CFEngine Build project, you can take advantage of CFEngine Build and its tooling. This allows you to easily upgrade the base version with one simple command, to add modules made by the community, and to minimize the amount of customizations you have to maintain. These new commands are available in `cfbs` version 5.5.0. ### Mission Portal first time setup CFEngine's web UI, Mission Portal, no longer has a default admin user and password. During installation, you will receive a 6 digit setup code to use when logging in for the first time. With this code, you will create the first admin user and specify its password. The username does not have to be `admin`, it can be any valid username, we recommend creating individual accounts for each administrator. {{< img "/blog/2025/images/mp-first-time-setup-1.png" "Screenshot of first time setup screen prompting the user to enter a setup code">}} ### Audit logs Our new audit logs show security-relevant events. In short, they summarize who changed what and when. This information is especially useful for auditing or during incident response. {{< wideimg "/blog/2025/images/audit-log.png" "Screenshot of the audit log, showing actions of users creating, updating, and deleting groups.">}} ### New variables and functions available in policy language For policy writers, we have added some new functions and variables for your convenience: - [`classfilterdata(data_container, data_structure, key_or_index)`](https://docs.cfengine.com/docs/master/reference/functions/classfilterdata) - Filter a data container (JSON) using class expressions. - [`findlocalgroups(filter)`](https://docs.cfengine.com/docs/master/reference/functions/findlocalgroups) - Returns a data container of all local groups with their attributes that are matching a filter. If no filter is specified, it will return all the local groups. - [`findlocalusers(filter)`](https://docs.cfengine.com/docs/master/reference/functions/findlocalusers) - Returns a data container of all local users with their attributes that are matching a filter. If no filter is specified, it will return all the local users. - [`getacls(path, type)`](https://docs.cfengine.com/docs/master/reference/functions/getacls) - Return the Access Control List (ACL) for a given file- or directory path. - [`getbundlemetatags(bundlename, optional_tag)`](https://docs.cfengine.com/docs/master/reference/functions/getbundlemetatags) - Returns the list of [meta](https://docs.cfengine.com/docs/master/reference/promise-types/#meta) tags for bundle bundlename. - [`getgroupinfo(optional_gidorname)`](https://docs.cfengine.com/docs/master/reference/functions/getgroupinfo) - Returns information about the current group or any other, looked up by group ID (GID) or group name. - [`getgroups(exclude_names, exclude_ids)`](https://docs.cfengine.com/docs/master/reference/functions/getgroups) - Returns a string list of group names on the system, with options to exclude specific names or IDs. - [`hostswithgroup(group_name, field)`](https://docs.cfengine.com/docs/master/reference/functions/hostswithgroup) - Returns a list from the CFEngine Database with the information field of all hosts sharing the group group\_name. - [`is_type(var, type)`](https://docs.cfengine.com/docs/master/reference/functions/is_type) - Returns whether a variable var is of type [type](https://docs.cfengine.com/docs/master/reference/functions/type/). - [`isconnectable(hostname, port, timeout)`](https://docs.cfengine.com/docs/master/reference/functions/isconnectable) - Checks whether a port is connectable. - [`isnewerthantime(file, time)`](https://docs.cfengine.com/docs/master/reference/functions/isnewerthantime) - Returns whether the file file is newer (modified later) than the time time. - [`useringroup(user)`](https://docs.cfengine.com/docs/master/reference/functions/useringroup) - Return whether a user is a member of a given group. ### Breaking changes From time to time, we need to change the behavior of certain features of CFEngine in a way that is not completely backwards compatible. This is usually to address security concerns, bugs causing issues for our users, or features behaving in really unexpected or unintended ways. In such cases, we try to communicate the changes with users, along with what is needed from them (such as adjusting their policy, reports, or API usage). Please take a look at the recent `change-in-behavior` blog posts, and stay up to date on these via our blog: https://cfengine.com/tags/change-in-behavior/ ### Platform support In order to focus on the platforms most widely used by our users, these platforms will no longer be supported in 3.27 and future versions: - Solaris - IBM AIX - HP-UX - SUSE - Debian 11 If you need support for any of the mentioned platforms, this is available as part of an _Extended platform support agreement_. [Contact us](/contact) for more details. We are also dropping hub support for the following platforms: - RHEL 7 - Ubuntu 20.04 This means you can still run CFEngine on machines with RHEL 7 and Ubuntu 20.04, but you will need a newer operating system on your hub(s). ### Release announcements for non-LTS releases There are other big changes introduced in the past 1.5 years, but not shown here. We encourage you to read our previous release blog posts to see more of these changes: - [CFEngine 3.25 - Auditability](/blog/2025/cfengine-3-25-released-auditability/) - [CFEngine 3.26 - Admin](/blog/2025/cfengine-3-26-released-admin/) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.27.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-core.html) - [3.27.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-enterprise.html) - [3.27.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.27/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) If you are upgrading from the 3.21 LTS series, scroll down in the changelog to find changes from the earlier 3.22 and 3.23 releases. Please note that the Enterprise changelogs contain only changes specific to Enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates We update dependencies to ensure we have the latest security fixes in the libraries and tools used in CFEngine. The table below shows our dependencies and their versions, with version 3.26.0 for comparison: | CFEngine version | 3.26.0 | **3.27.0** | | :-------------------------------------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.63 | **2.4.66** | | [APR](https://apr.apache.org/) | 1.7.5 | **1.7.6** | | [apr-util](https://apr.apache.org/) | 1.6.3 | 1.6.3 | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.12 | 3.12 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.49.0 | **2.52.0** | | [leech](https://github.com/larsewi/leech/releases) | 0.1.24 | **0.2.0** | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.2 | 2.3.2 | | [libattr](https://download.savannah.gnu.org/releases/attr) | 2.5.2 | 2.5.2 | | [libcurl](https://curl.se/download.html) | 8.13.0 | **8.17.0** | | [libcurl-hub](https://curl.se/download.html) | 8.13.0 | **8.17.0** | | [libexpat](https://libexpat.github.io/) | 2.7.1 | **2.7.3** | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.18 | 1.18 | | [librsync](https://github.com/librsync/librsync/releases) | 2.3.4 | 2.3.4 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.14.3 | **2.15.1** | | [LibYAML](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.33 | 0.9.33 | | [nghttp2](https://nghttp2.org/) | 1.65.0 | **1.68.0** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.9 | **2.6.10** | | [OpenSSL](https://openssl.org/) | 3.5.0 | **3.6.0** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | 10.45 | **10.47** | | [PHP](https://php.net/) | 8.4.7 | **8.5.1** | | [PostgreSQL](https://www.postgresql.org/) | 17.5 | **18.1** | | [pthreads-w32](https://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | | [rsync](https://download.samba.org/pub/rsync/) | 3.4.1 | 3.4.1 | | [SASL2](https://www.cyrusimap.org/sasl/) | 2.1.28 | 2.1.28 | | [zlib](https://www.zlib.net/) | 1.3.1 | 1.3.1 | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, it will now default to 3.27.0, since this is the latest LTS release available. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 56 - Profiling cf-agent" url: /blog/2025/the-agent-is-in-episode-56/index.md date: "2025-12-25" -------------------------------------------------------------------------------- > Take a fast thing and make it faster. {{< figure src="./2025-12-24_14-48-32_screenshot.png" >}} In this Christmas special, Nick and Herman chat about the new built-in profiling support for `cf-agent` in the upcoming release. The upcoming 3.27 LTS release introduces a first-class profiling capability directly into `cf-agent`. Unlike previous solutions (like `cf-profile` or the Perl-based profiler) which often required real-time analysis or significant logging overhead, this new approach decouples collection from analysis. To profile a run, you simply add the `--profile` option, redirect the output to a file for later analysis. ```sh cf-agent --no-lock --profile > callstack.json ``` This generates a JSON file containing the call stack and timing information for every promise evaluation. Nick demonstrated a new, un-released tool, [cfengine-cli](https://github.com/cfengine/cfengine-cli), to digest this JSON data. You can see which bundles, promises, or functions are consuming the most time: ```sh cfengine profile --top 15 --bundles --promises --functions ./callstack.json ``` ## Video {#video} The video recording is available on YouTube: {{< youtube q6hrwto91JE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.21.8 and 3.24.3 released" url: /blog/2025/cfengine-3-21-8-and-3-24-3-released/index.md date: "2025-12-12" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.21.8 and 3.24.3\! These patch releases contain bug fixes and dependency updates. ## Changes As these are patch releases for long term supported (LTS) branches of CFEngine, there are no new major features included. ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.24.3 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-core.html) - [3.24.3 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-enterprise.html) - [3.24.3 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.8 changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.8 changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.8 changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades Compared to the previous releases, CFEngine [3.21.7 and 3.24.2](/blog/2025/cfengine-3-21-7-and-3-24-2-released/), the following dependencies were updated: | CFEngine version | 3.21.7 | **3.21.8** | 3.24.2 | **3.24.3** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.33 | 0.9.33 | 0.9.33 | 0.9.33 | | [Apache](https://httpd.apache.org/) | 2.4.63 | **2.4.66** | 2.4.63 | **2.4.66** | | [APR](https://apr.apache.org/) | 1.7.5 | **1.7.6** | 1.7.5 | **1.7.6** | | [apr-util](https://apr.apache.org/) | 1.6.3 | 1.6.3 | 1.6.3 | 1.6.3 | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.12 | 3.12 | 3.12 | 3.12 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.49.0 | **2.52.0** | 2.49.0 | **2.52.0** | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.2 | 2.3.2 | 2.3.2 | 2.3.2 | | [libattr](https://download.savannah.gnu.org/releases/attr) | 2.5.2 | 2.5.2 | 2.5.2 | 2.5.2 | | [libcurl](https://curl.se/download.html) | 8.13.0 | **8.17.0** | 8.13.0 | **8.17.0** | | [libcurl-hub](https://curl.se/download.html) | 8.13.0 | **8.17.0** | 8.13.0 | **8.17.0** | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | 2.5.1 | 2.5.1 | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.18 | 1.18 | 1.18 | 1.18 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.14.2 | **2.15.1** | 2.14.2 | **2.15.1** | | [LibYAML](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | 0.2.5 | 0.2.5 | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.9 | **2.6.10** | 2.6.9 | **2.6.10** | | [OpenSSL](https://openssl.org/) | 3.0.16 | **3.0.18** | 3.5.0 | **3.6.0** | | [PCRE](https://www.pcre.org/) | 8.45 | 8.45 | - | - | | [PHP](https://php.net/) | 8.3.21 | **8.3.28** | 8.3.21 | **8.3.28** | | [PostgreSQL](https://www.postgresql.org/) | 15.12 | **15.15** | 16.8 | **16.11** | | [pthreads-w32](https://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | 2-9-1 | 2-9-1 | | [rsync](https://download.samba.org/pub/rsync/) | 3.4.1 | 3.4.1 | 3.4.1 | 3.4.1 | | [SASL2](https://www.cyrusimap.org/sasl/) | 2.1.28 | 2.1.28 | 2.1.28 | 2.1.28 | | [zlib](https://www.zlib.net/) | 1.3.1 | 1.3.1 | 1.3.1 | 1.3.1 | | [libexpat](https://libexpat.github.io/) | - | - | 2.7.1 | **2.7.3** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | - | - | 10.45 | **10.47** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.24/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.24.3 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.21.8 list ``` ```command cf-remote --version 3.24.3 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.21 LTS After CFEngine 3.21.8 there will be no more releases for the CFEngine 3.21 LTS series. Please note that support for the 3.21 LTS series is now ending - After this month, on January 1st 2026, CFEngine 3.21 will no longer be supported. If you are on 3.21 or an older unsupported version, please [contact us](/contact) for help with upgrading or to discuss an extended support contract. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 55 - evaluation_order in body file control" url: /blog/2025/the-agent-is-in-episode-55/index.md date: "2025-11-27" -------------------------------------------------------------------------------- > When you first told me that this change was coming I was astonished because I know that normal order, the normal ordering is very intentional like a lot of thought went into it right and it's not configurable, again on purpose, right!? {{< figure src="2025-11-26_12-23-53_evaluation_order-in-body_file_control.png" >}} In this episode, Nick is joined by long-time CFEngine user and trainer, Aleksey Tsalolikhin. It was a conversation with Aleksey at a LISA conference in 2010 that set Nick on his CFEngine journey, asking, "What do you want from your configuration management tooling?". Nick knew immediately that the tool he was using, while great, didn't fit the characteristics he was looking for. Today, they reunite to discuss a significant new feature in CFEngine 3.27: file-specific evaluation order. For years, CFEngine's "normal" evaluation order has been a cornerstone of its predictable, convergent behavior. But what happens when you need a more procedural, top-down approach for a specific part of your policy? Now, with `evaluation_order => "top_down";` in `body file control`, you can have the best of both worlds. We explore how this feature works, why it's useful for logging and for simplifying policy for new users, and how to use it safely alongside your existing policy. ## Notes {#notes} One of the most discussed aspects of learning CFEngine is its evaluation order. The "normal" order is carefully designed for convergence, ensuring that dependencies are typically met before they are needed (e.g., files are promised before packages that might own them). However, this can sometimes be counter-intuitive for those accustomed to procedural scripting. With CFEngine 3.27, you can now control this behavior on a per-file basis using `body file control`. ### Normal (Classic) Evaluation Order {#normal--classic--evaluation-order} By default, CFEngine evaluates promises in a specific order, regardless of where they appear in the bundle. For example, `commands` promises are evaluated before `reports` promises. ```cfengine3 bundle agent reports_normal { reports: "Hello world:"; commands: "/bin/echo hi"; reports: "bye:"; } bundle agent __main__ { methods: "reports_normal"; } ``` Running this bundle results in the command executing first: ```text notice: Q: ".../bin/echo hi": hi R: Hello world: R: bye: ``` ### Top-Down Evaluation Order {#top-down-evaluation-order} By adding a `body file control` with `evaluation_order ==> "top_down";`, you can force CFEngine to evaluate the promises in the order they are written in the file. ```cfengine3 body file control { evaluation_order => "top_down"; } bundle agent reports_top_down { reports: "Hello world:"; commands: "/bin/echo hi"; reports: "bye:"; } bundle agent __main__ { methods: "reports_top_down"; } ``` Now, the output respects the written order: ```text R: Hello world: notice: Q: ".../bin/echo hi": hi R: bye: ``` This is particularly useful for: - **Logging and Debugging:** Easily print messages at the beginning and end of a bundle. - **On-boarding New Policy Writers:** Provides a more familiar, procedural execution model for specific tasks. - **Complex Sequences:** When you need to enforce a strict, linear sequence of operations without relying on `depends_on` or multiple bundle calls. You can safely mix and match evaluation orders by using the `inputs` key in `body file control`, allowing you to adopt this new feature gradually and only where it makes sense. ## Video {#video} The video recording is available on YouTube: {{< youtube bmppbSdlpRE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 54 - Sneak peek of CFEngine 3.27" url: /blog/2025/the-agent-is-in-episode-54/index.md date: "2025-10-30" -------------------------------------------------------------------------------- "I came here to chew bubble gum and demo the next LTS. And I'm all out of chewing gum." {{< figure src="./2025-10-30_13-10-30_IMG_6618.png" >}} Herman joined Cody, Craig and Nick to give a sneak peek of the upcoming CFEngine 3.27 LTS release. Herman shows off some of the new features, including: - An AI agent to query your infrastructure using natural language. - Audit log improvements. - A new first-time setup experience for Mission Portal. - An enhanced Mission Portal UI. - New policy language functions like `getgroups()`, `getgroupinfo()`, `findlocalgroups()`, `getacls()`, `isconnectable()`, `isnewerthantime()`, and `classfilterdata()` (which we covered in [The agent is in - Episode 51 - Data-driven configuration with classfilterdata()](https://cfengine.com/blog/2025/the-agent-is-in-episode-51/)). - A new `evaluation_order` control to run promises in a top-down order. We also answer some great questions from the audience about custom promise modules, nested groups, and AI model integration. ## Video {#video} The video recording is available on YouTube: {{< youtube IJs2t8JhP_8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Post show discussion {#post-show-discussion} ### Custom promise types: Global vs. per-promise configuration? {#custom-promise-types-global-vs-dot-per-promise-configuration} A question arose regarding custom promise types and whether their interpreter and path configurations could be set globally rather than per promise. Nick and Herman clarified that the convention is to declare custom promise types in `init.cf`. Once instantiated there, they should be globally known and not require redefinition in every promise. ### AI in restrictive environments: On-premise models? {#ai-in-restrictive-environments-on-premise-models} A user inquired about using the AI agent with in-house, on-premise AI models due to security requirements, rather than public services like Google Gemini or Copilot. Herman confirmed that CFEngine currently supports two options for local/on-premise models: - ****OpenAI Like:**** This option allows connecting to any model that speaks the OpenAI API. - ****Ollama:**** Direct support for Ollama-compatible models. He encouraged users to provide feedback if their specific in-house models don't fit these categories. ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - New Function Documentation - [`get_groups()`](https://docs.cfengine.com/docs/master/reference/functions/getgroups/) - [`get_group_info()`](https://docs.cfengine.com/docs/master/reference/functions/getgroupinfo/) - [`find_local_groups()`](https://docs.cfengine.com/docs/master/reference/functions/findlocalgroups/) - [`get_acls()`](https://docs.cfengine.com/docs/master/reference/functions/getacls/) - [`is_connectable()`](https://docs.cfengine.com/docs/master/reference/functions/isconnectable/) - [`isnewerthantime()`](https://docs.cfengine.com/docs/master/reference/functions/isnewerthantime/) - [`classfilterdata()`](https://docs.cfengine.com/docs/master/reference/functions/classfilterdata/) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 53 - CFEngine and decomputing" url: /blog/2025/the-agent-is-in-episode-53/index.md date: "2025-09-25" -------------------------------------------------------------------------------- In these times dominated by ever growing amounts of technology, have you ever considered the power of _less_? In this very special episode, we celebrate Craig Comstock's 8-year anniversary with the company! Craig takes us on a journey into the concept of "decomputing" - the idea of using computer systems less and within limits. {{< figure src="8-year-craig-celebration.png" >}} Join us as Craig explores how CFEngine's philosophy and design principles, such as being lightweight and having few dependencies, align with the decomputing mindset. He shares some book recommendations (The Mechanic and the Luddite, Resist AI), and examples of fun, interesting, old, and resource constrained platforms where he exercises CFEngine regularly. ## Video {#video} The video recording is available on YouTube: {{< youtube nc9qv0VzB_k >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Post show discussion {#post-show-discussion} After the show there was some brief discussion about [access promise types](https://docs.cfengine.com/docs/3.24/reference-promise-types-access.html). ### How to integrate custom access promises with cfbs? {#how-to-integrate-custom-access-promises-with-cfbs} > In the old days we would edit `cf_serverd.cf` and add new access promises next to the ones that ship in the Masterfiles Policy Framework. How should we do this with cfbs? While there is absolutely nothing wrong with modifying the Masterfiles Policy Framework, reducing modifications to the vendored files can significantly benefit maintenance. Rather than editing an existing server bundle like [mpf_default_access_rules](https://docs.cfengine.com/docs/3.24/reference-masterfiles-policy-framework-controls-cf_serverd.html#mpf_default_access_rules), introduce your own server bundle. ```cfengine3 bundle server custom_access_promises { access: "/srv/customers/" handle => "customer_files_access", admit => { "0.0.0.0/0" }; } ``` Then to integrate this into the policy with `cfbs`, run `cfbs add` and provide the path to the policy file. Once the policy is in inputs on the host `cf-serverd` should notice and reload the policy, but it may require a service restart if the data is separate from the policy as we typically recommend. ### How does the policy get run when not in the bundlesequence? {#how-does-the-policy-get-run-when-not-in-the-bundlesequence} You might notice that server bundles aren't listed in `body common control` `bundlesequence`. `cf-serverd` and `cf-monitord` don't use bundlesequence. Instead they process `server` and `monitor` bundles as they are found during policy parsing. ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [Dan McQuillan - Resist AI](https://bristoluniversitypress.co.uk/resisting-ai) - [Jathan Sadowski - The Mechanic and the Luddite](https://www.ucpress.edu/books/the-mechanic-and-the-luddite/paper) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Introducing classfilterdata() policy function" url: /blog/2025/classfilterdata/index.md date: "2025-09-01" -------------------------------------------------------------------------------- We recently introduced a new policy function [`classfilterdata()`](https://docs.cfengine.com/docs/master/reference/functions/classfilterdata/), which will be available in the next LTS release of CFEngine, version 3.27. If you can't wait for the release, feel free to grab the latest `master non-LTS` from our [nightly packages](https://cfengine.com/downloads/cfengine-community-nightly/). In this blog post, we'll illustrate how the `classfilterdata()` policy function works. However, if you want a more real-world example, you should check out [The agent is in - Episode 51 - Data-Driven Configuration with classfilterdata()](https://cfengine.com/blog/2025/the-agent-is-in-episode-51/) by [Jay Goldberg](https://www.linkedin.com/in/jay-goldberg-73734/) from [Two Sigma](https://www.twosigma.com/). The `classfilterdata()` policy function takes 2 to 3 arguments. The first being a reference to a data container or inline JSON, followed by the type of data structure, and the key or index of the class expression within the data structure. Here is a tarball containing the files used in this blog post [classfilterdata.tar.gz](classfilterdata/classfilterdata.tar.gz). ## Array of arrays Let's start off with the first data structure, `"array_of_arrays"`, which is reminiscent of its precursor [`classfiltercsv()`](https://docs.cfengine.com/docs/master/reference/functions/classfiltercsv/). The major difference is that `classfilterdata()` takes a data container as its first argument. This is in contrast to the path of a CSV file. Having a data container as the first argument gives you the flexibility to parse the data container from various formats using [`readjson()`](https://docs.cfengine.com/docs/master/reference/functions/readjson/), [`readyaml()`](https://docs.cfengine.com/docs/master/reference/functions/readyaml/), [`readcsv()`](https://docs.cfengine.com/docs/master/reference/functions/readcsv/), [`readenvfile()`](https://docs.cfengine.com/docs/master/reference/functions/readenvfile/), [`readdata()`](https://docs.cfengine.com/docs/master/reference/functions/readdata/), etc. Given the following JSON file containing the inventory of a company, let's say you want to filter it based on the department. ```json {file="array_of_arrays/my_data.json"} [ ["ID", "Name", "Decription", "Quantity", "Department"], [1, "Laptop", "MacBook Pro", 13, "sales"], [2, "Phone", "The Fairphone (Gen. 6)", 5, "workshop"], [3, "Monitor", "DELL P2419HC", 7, "sales"], [4, "Printer", "HP Printer", 1, "marketing"] ] ``` With policy, you can simply parse the JSON file using the `readjson()` policy function, followed by e.g. filtering it based on the defined `sales` class using `classfilterdata()`. The index `”4”` tells the policy function which column to filter on. In our case, the 4th column, counting from 0. ```cf3 {file="array_of_arrays/my_policy.cf"} bundle agent array_of_arrays { classes: "sales"; vars: "raw" data => readjson("$(this.promise_dirname)/my_data.json"); "filtered" data => classfilterdata("@(raw)", "array_of_arrays", "4"); reports: "Inventory for sales $(with)" with => storejson("@(filtered)"); } ``` In the end, we convert the data container back to a JSON string and print it in a reports promise. From the output, you can see that only entries from the sales department are present. ``` R: Inventory for sales [ [ 1, "Laptop", "MacBook Pro", 13, "sales" ], [ 3, "Monitor", "DELL P2419HC", 7, "sales" ] ] ``` ## Array of objects The ability to support more complex data structures, such as `"array_of_objects"`, is what makes `classfilterdata()` so much more powerful than its predecessor (i.e., `classfiltercsv()`). Imagine that you are the system administrator of developer machines, where you are responsible for configuring and installing the default editor for each developer. You're handed a JSON file containing their usernames and preferred editor. ```json {file="array_of_objects/my_data.json"} [ { "username": "larsewi", "editor": "vim" }, { "username": "nickanderson", "editor": "emacs" }, { "username": "olehermanse", "editor": "vim" }, { "username": "craigcomstock", "editor": "ed" } ] ``` The policy for filtering out the correct editors based on the users (defined as classes) can be written as follows. Notice that we no longer use an index to specify the class expression. The key difference (pun intended) is that we need a key to access the child element rather than an index. The reason is that the ordering of key-value pairs in a JSON object is arbitrary. Hence, we specify the key of the desired class expression. Which, in our case, is the username. ```cf3 {file="array_of_objects/my_policy.cf"} bundle agent array_of_objects { classes: "nickanderson"; vars: "raw" data => readjson("$(this.promise_dirname)/my_data.json"); "filtered" data => classfilterdata("@(raw)", "array_of_objects", "username"); reports: "Required editors $(with)" with => storejson("@(filtered)"); } ``` In this example, the username is hardcoded, but it could, for example, be parsed from /etc/passwd. The output would be: ``` R: Required editors [ { "editor": "emacs", "username": "nickanderson" } ] ``` ## Object of arrays So far, we've only seen arrays as the root element in the data structure. However, an object can also be used. Using an object as the root element comes with an extra perk. You can either specify the key or an index of the child element as the third argument, or you can simply omit it, causing the key of the child element itself to be the class expression. Let's illustrate `"object_of_arrays"` with the following data: ```json {file="object_of_arrays/my_data.json"} { "debian": ["bison", "flex", "libacl1"], "debian.dev": ["libbison-dev", "libacl1-dev"], "redhat": ["byacc", "flex"], "redhat.dev": ["flex-devel"] } ``` The drawback of using the key to the child element as the class expression is that you can never list the exact same class expression twice, since a key has to be unique. ```cf3 {file="object_of_arrays/my_policy.cf"} bundle agent object_of_arrays { vars: "raw" data => readjson("$(this.promise_dirname)/my_data.json"); "filtered" data => classfilterdata("@(raw)", "object_of_arrays"); reports: "Packages to be installed $(with)" with => storejson("@(filtered)"); } ``` If we run the policy on an Ubuntu machine, we would get the following output: ``` R: Packages to be installed { "debian": [ "bison", "flex", "libacl1" ] } ``` For more information on classes and expressions check out the following resources: - [Making decisions based on classes](https://docs.cfengine.com/docs/master/reference/language-concepts/classes/#making-decisions-based-on-classes) - [Class expressions](https://docs.cfengine.com/docs/master/reference/promise-types/classes/#expression) ## Object of objects The last supported data structure is `"object_of_objects"`. Also, here, you can specify a key in the child object, or omit it to use the key of the child object itself. Let's demonstrate the former with the following YAML file: ```yaml # Files and permissions /etc/chrony.conf: mode: 644 owner: root group: root if: redhat /etc/apt/apt.conf: mode: 644 owner: root group: root if: debian|ubuntu ``` Now, let's filter the data based on the value of the `"if"` key: ```cf3 bundle agent object_of_objects { vars: "raw" data => readyaml("$(this.promise_dirname)/my_data.yaml"); "filtered" data => classfilterdata("@(raw)", "object_of_objects", "if"); reports: "File permissions $(with)" with => storejson("@(filtered)"); } ``` Unfortunately, there is no policy function to convert data containers back to YAML, so we'll use JSON instead. ``` R: File permissions { "/etc/apt/apt.conf": { "group": "root", "if": "debian|ubuntu", "mode": 644, "owner": "root" } } ``` ## Auto If you're lazy, or you're, e.g., writing a more generic bundle, you can tell CFEngine to automatically detect the data structure by using `"auto"` instead of explicitly specifying the data structure. Furthermore, `"auto"` lets you interchange the different supported data structures. Take this data as an example: ```json { "array": ["foo", "bar", "baz"], "object": { "0": "bogus", "1": "any", "2": "doofus" } } ``` Here we have an object of arrays and objects. Any of the "more specific" data structures from before would cause the policy function to fail. However, for `"auto"`, the interpretation is less strict. Notice how the keys in the child object must be in a way that they can be interpreted as both keys in the objects or indices in the arrays. ```cf3 bundle agent auto { classes: "bar"; vars: "raw" data => readjson("$(this.promise_dirname)/my_data.json"); "filtered" data => classfilterdata("@(raw)", "auto", "1"); reports: "Filtered $(with)" with => storejson("@(filtered)"); } ``` The filtered output from the hybrid data structure would be as follows: ``` R: Filtered { "array": [ "foo", "bar", "baz" ], "object": { "0": "bogus", "1": "any", "2": "doofus" } } ``` ## Share your thoughts You are now an expert user of the `classfilterdata()`. If you have any cool use cases for this policy function or you have any questions about the CFEngine in general, let us know at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear from you. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 52 - body fsattrs and the immutable bit" url: /blog/2025/the-agent-is-in-episode-52/index.md date: "2025-08-28" -------------------------------------------------------------------------------- Is 5 minutes of drift too long? Winter is coming, and so is CFEngine support for dealing with immutable files. {{< figure src="2025-08-28_12-47-48_changine-permissions-of-file-managed-by-cfengine.gif" >}} CFEngine has always been a powerful tool for enforcing configuration and preventing drift. It’s the very essence of its mission to ensure the state of a system remains in line with its policy. For years, that periodic enforcement with the speed and frequency of execution has been the core of CFEngine’s value. But what if you need to add another layer of protection? What if you want to make files on a system even more resistant to change, whether from a well-meaning administrator, an errant process, or even a malicious actor but not necessarily hamper CFEngine? In this episode Nick showed off support for the immutable bit coming later this year in 3.27.0. While the new `fsattrs` body could eventually support additional file system attributes the immutable bit is the first target. At the kernel level, the immutable bit is an extended file attribute (i) that, when set on a file, prevents any modification, including deletion, renaming, or changes to permissions, **even by the root user**. The only way to modify the file is to first remove the immutable bit using the `chattr` command (`chattr -i filename`). Previously, if a CFEngine promise attempted to manage an immutable file, it would simply fail and report a promise not kept, as it lacked the ability to temporarily remove this bit. The new `fsattrs` body changes this dynamic. CFEngine can now actively manage this bit as part of a files promise transparently adding and removing the bit as necessary automatically when needed to facilitate change. How CFEngine interacts with files based on the immutable bit setting in a promise: {{< figure src="2025-08-28_13-09-03_screenshot.png" >}} - When a promise has an `fsattrs` body attached which contains `immutable => "true"`: CFEngine will ensure that the file is in an immutable state at the conclusion of the promise actuation. If the file needs to be modified for instance, if its content, permissions, or ownership needs to be updated CFEngine will transparently and temporarily remove the immutable bit, make the required changes, and then re-apply the immutable bit. This allows a single CFEngine promise to manage both the content and the immutability of a file without causing a promise failure. - When a promise has an `fsattrs` body attached which contains `immutable => "false"`: CFEngine will ensure that the file is in a mutable state. If it encounters a file that has the immutable bit set, it will remove the bit, leaving the file in a state that can be freely modified by other processes or administrators with proper permission. - When a promise does not contain a body `fsattrs` or if `immutable` is not set in an attached `fsattrs` body: CFEngine will revert to its legacy behavior. If it attempts to modify a file that has the immutable bit set, the operation will fail with a "permission denied" error, and the promise will be reported as not kept. There was discussion about the benefits of this functionality: - Should CFEngine be able to do this? - Some people use reporting about the presence of files with the immutable bit and presence or absence of promises not kept. These reports help operators determine what might be the cause of an issue. In this context, CFEngine transparently handling and modifying immutable files would be undesirable. - Examples of files on which people consider that setting the immutable bit could be useful include `/etc/resolv.conf`, and `ssh_known_hosts`. If you have more thoughts and use cases please share them with the team. - Managed Service Providers (MSPs) were thought to be an area that might make more use of functionality like this. ## Video {#video} The video recording is available on YouTube: {{< youtube y5OkNRu7m2M >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). This time discussion included: - Advice to use the immutable bit as a warning signal to identify policy gaps and correlate them with promise failures. - Commiserating about the struggle in managing complex permissions and, as a result, often disables SELinux. - The importance of being able to track why a file was changed or flagged, highlighting the potential for embedding context via extended file attributes. - Desire for new CFEngine functionality regarding SELinux. ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 51 - Data-Driven Configuration with classfilterdata()" url: /blog/2025/the-agent-is-in-episode-51/index.md date: "2025-07-31" -------------------------------------------------------------------------------- They say data is the new oil, that makes `classfilterdata()` the new rig. Check out how Jay Goldberg pumps the Two Sigma data lake with CFEngine. {{< figure src="2025-07-31_12-24-00_classfilterdata-context-filter-rig.jpg" >}} In Episode 51 special guest Jay Goldberg of [Two Sigma](https://www.twosigma.com/) joined the CFEngine team to discuss data-driven configuration, other examples of data driven policy ( [EFL](https://github.com/neilhwatson/evolve_cfengine_freelib), [SCL](https://github.com/basvandervlies/cf_surfsara_lib), [sysops-api](https://github.com/linkedin/sysops-api), [CFEngine-Roadshow](https://github.com/Webhuis/CFEngine-Roadshow), and other un-named organizations), the provenance of `classfiltercsv()`, and the upcoming `classfilterdata()` function (already in nightly packages near you). He explained how Two Sigma organizes their external data making it easy for engineers to reason with and how leveraging external data democratized the ability for other engineers to leverage the power of CFEngine without having to look at or write any policy. `classfilterdata()` helps filter data containers by class expression, returning only data that is relevant for the current context. This approach allows for policies to be managed with configuration files rather than being hard-coded, reducing 110 policies down to 10 for common building blocks like files, packages, and services. Jay highlighted real-world benefits, including safer changes, less code, and significantly less effort, demonstrating a 93% reduction in code and over 30 times faster execution, transforming processes that took over 60 seconds to less than 2 seconds. ## Video {#video} The video recording is available on YouTube: {{< youtube BKZuPh8mgzw >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Jay](https://www.linkedin.com/in/jay-goldberg-73734/), [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [A brief history of configuration Managing sprawl](https://www.youtube.com/watch?v=Aorwdg2dRm0) - [Mark Burgess](https://www.linkedin.com/in/markburgessoslo/) - [classfiltercsv()](https://docs.cfengine.com/docs/master/reference/functions/classfiltercsv/) - [classfilterdata()](https://docs.cfengine.com/docs/master/reference/functions/classfilterdata/) - [Neil Watson](https://www.linkedin.com/in/neilhwatson/) - [EFL (Evolve Thinking's free CFEngine Library)](https://github.com/neilhwatson/evolve_cfengine_freelib) - [Bas van der Vlies](https://www.linkedin.com/in/basvandervlies/) - [SURF CFEngine Library (SCL)](https://github.com/basvandervlies/cf_surfsara_lib) - [Martin Simons](https://www.linkedin.com/in/martinsimons1/) - [CFEngine-Roadshow](https://github.com/Webhuis/CFEngine-Roadshow) (internal, variable based data) - [Episode 51 resources](https://gitlab.com/nickanderson/the-agent-is-in/-/tree/main/Episodes/51) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 50 - AI Roundtable" url: /blog/2025/the-agent-is-in-episode-50/index.md date: "2025-06-26" -------------------------------------------------------------------------------- "For some reason they're willing to read a 20-page essay from a chatbot when they would never imagine doing it from a colleague." For episode 50 Cody, Craig, and Nick are joined by special guests Mark Burgess, progenitor and original creator of CFEngine, Ted Zlatanov, configuration management expert and CFEngine Champion and Igor Aleksandrychev, CFEngine Enterprise developer to talk about AI. The guests touched on the rapidly evolving landscape of AI code generation, with Ted sharing insights on Glean and Claude, noting impressive speed but requiring substantial rework for production. Mark offered a historical perspective, highlighting his work with AI within his Semantic Space Storytime project [SSTorytime](https://github.com/markburgess/SSTorytime). Igor demonstrated a natural language query demo for CFEngine Enterprise. Craig recounted his experience with LLM summarization, admitting to being occasionally overwhelmed by the sheer volume of output. Nick raised concerns about the computational cost and the challenge of balancing efficiency with quality. Post-show discussion explored applying data to dynamic host groups and successful integration of CFEngine with interactive user acceptance on GUI workstations. ## Video {#video} The video recording is available on YouTube: {{< youtube 1BWnZ31ydc0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Mark](https://www.linkedin.com/in/markburgessoslo/), [Ted](https://www.linkedin.com/in/teodor-zlatanov-081b531a0/), [Igor](https://www.linkedin.com/in/igor-aleksandrychev-5b626526/), [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Change in behavior: Policy function findfiles" url: /blog/2025/change-in-behavior-findfiles/index.md date: "2025-06-12" -------------------------------------------------------------------------------- *_Note: this blog post was updated January 29th, 2026_* Here comes a profoundly belated blog post on a behavior change. Better late than never. Due to various bugs with the glob engine on Windows, we decided to rewrite it in CFEngine 3.24.0. Not only does the new glob engine resolve these bugs on Windows, but it also adds support for brace expansion on all platforms. E.g. ```cf3 {file="findfiles.cf"} bundle agent main { vars: "matches" slist => findfiles("C:/{foo,bar}.txt"); reports: "$(matches)"; } ``` ```command & 'C:\Program Files\Cfengine\bin\cf-agent.exe' -Kf C:\findfiles.cf ``` ```output R: C:\bar.txt R: C:\foo.txt ``` Users may experience issues due to the fact that the new glob engine outputs the matched paths with the system separator (i.e., `$(const.dirsep)`). E.g., given the following policy, you can see how the output with the matched files changed from having forward slashes in CFEngine 3.21 to having backslashes in CFEngine 3.24 on Windows. ```cf3 {file="findfiles.cf"} bundle agent main { vars: "matches" slist => findfiles("C:/Users/Administrator/Downloads/*.msi"); reports: "CFEngine $(sys.cf_version) matches:"; "$(matches)"; } ``` ```command & 'C:\Program Files\Cfengine\bin\cf-agent.exe' -Kf C:\findfiles.cf ``` ```output R: CFEngine 3.21.7 matches: R: C:/Users/Administrator/Downloads/cfengine-nova-3.21.7-1-x86_64.msi R: C:/Users/Administrator/Downloads/cfengine-nova-3.24.2-1-x86_64.msi ``` ```command & 'C:\Program Files\Cfengine\bin\cf-agent.exe' -Kf C:\findfiles.cf ``` ```output R: CFEngine 3.24.2 matches: R: C:\Users\Administrator\Downloads\cfengine-nova-3.21.7-1-x86_64.msi R: C:\Users\Administrator\Downloads\cfengine-nova-3.24.2-1-x86_64.msi ``` Some users may experience issues where they use the returned file paths of the `findfiles()` function in regular expressions. In this case, the back slashes must either be escaped with an additional backslash, or replaced with forward slashes. Otherwise, the regex engine will interpret them as control characters. A typical error message you would encounter in this case is: ``` error: Regular expression error: 'unknown property after \P or \p' in expression 'C:\Program Files\foo.json' (offset: 5) ``` Hence, if your policy breaks due to the fact that it depends on the assumption that the matched paths contain forward slashes, then you know why. If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 49 - Demo of CFEngine 3.26.0" url: /blog/2025/the-agent-is-in-episode-49/index.md date: "2025-05-29" -------------------------------------------------------------------------------- No more default passwords! CFEngine 3.26.0 (“admin”) forces you to create an admin user from scratch. Cody, Craig, and Nick take a look at the latest CFEngine release, 3.26.0, the "admin" release. After showing the new setup process to create an initial privileged user and highlighting `cfbs` and `cf-remote` updates and separate release schedule the majority of the conversation centered around new functions that were introduced. - [findlocalusers()](https://docs.cfengine.com/docs/3.26/reference-functions-findlocalusers.html) - Stop parsing `/etc/passwd` in policy, let the C do it. - [getbundlemetatatags()](https://docs.cfengine.com/docs/3.26/reference-functions-getbundlemetatags.html) - More introspection capabilities from within policy. - [hostswithgroup()](https://docs.cfengine.com/docs/3.26/reference-functions-hostswithgroup.html) - Generate lists of hosts that are in a group from policy on the hub (**Enterprise Hub only**). - [is_type()](https://docs.cfengine.com/docs/3.26/reference-functions-is_type.html) - Ensure the data you are looking at is what you expect it to be. - [isconnectable()](https://docs.cfengine.com/docs/3.26/reference-functions-isconnectable.html) - Speed up policy by probing a port to see if it's even connectable. - [useringroup()](https://docs.cfengine.com/docs/3.26/reference-functions-useringroup.html) - Stop parsing `/etc/group` in policy, let the C do it. The audience chimed in with some ideas for new and existing functions: - A function like [classfiltercsv()](https://docs.cfengine.com/docs/3.24/reference-functions-classfiltercsv.html#top) that operates on data containers. - The addition of a `timeout` option to [readtcp()](https://docs.cfengine.com/docs/3.24/reference-functions-readtcp.html#top). - Extend [version_compare()](https://docs.cfengine.com/docs/3.24/reference-functions-version_compare.html) to handle additional versioning information, perhaps leveraging version sorting in GNU Core utils. Citing inconsistencies in various versioning strategies making that potentially challenging, the team also highlighted `dpkg --compare-versions` and `rpmvercmp` (which ships in CFEngine Enterprise Linux packages) which can be used to compare versions for deb and rpm based systems. - A function to watch a file for change that could trigger execution of some promise(s). On this Nick mentioned potential of a new agent for "event driven policy". The episode wrapped up with some commentary on how to influence promise order with [depends_on](https://docs.cfengine.com/docs/3.24/reference-promise-types.html#depends_on) and how you can abort execution of a bundle or the entire agent using [abortbundleclasses](https://docs.cfengine.com/docs/3.24/reference-components-cf-agent.html#abortbundleclasses) and [abortclasses](https://docs.cfengine.com/docs/3.24/reference-components-cf-agent.html#abortclasses). ## Video {#video} The video recording is available on YouTube: {{< youtube Txc6zLopoD0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [Version sort in GNU Coreutils 9.7](https://www.gnu.org/software/coreutils/manual/html_node/Version-sort-overview.html) - [Differences between Version Sort in GNU Coreutils and Debian](https://www.gnu.org/software/coreutils/manual/html_node/Differences-from-Debian-version-sort.html) - [dpkg --compare-versions](https://www.gnu.org/software/coreutils/manual/html_node/Comparing-two-strings-using-Debian_0027s-algorithm.html) -------------------------------------------------------------------------------- title: "CFEngine 3.26 released - Admin" url: /blog/2025/cfengine-3-26-released-admin/index.md date: "2025-05-20" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.26.0\! Being a non-LTS (not supported) release, this release allows users to test the new functionality we've been working on before it arrives in an LTS release later this year. The codename for this release is a bit different, as it is named after a new feature introduced, and what it eliminates - the default `admin` user. ## What's new In recent releases, we've made important security improvements like stricter password policies, 2FA support, audit logs, and more. We're continuing down this track of improving the overall account and login security of CFEngine Enterprise by eliminating the default `admin` user and introducing a much more secure first time setup feature to create your own user with administrative rights. ### Mission Portal first time setup When you set up a CFEngine hub for the first time, you'll now see some new screens to securely set up the initial user. {{< img "images/mp-first-time-setup-1.png" "Screenshot of first time setup screen prompting the user to enter a setup code">}} You'll need the setup code printed by the installer to set up for the first time. If you don't see the output / code, or it expires, you can use the `cf-hub` program to generate a new one; ```command cf-hub --new-setup-code ``` After entering the correct setup code, you'll create the first user: {{< img "images/mp-first-time-setup-2.png" "Screenshot of the second screen prompting the user to enter a username, password, and email address for the admin account.">}} Both username and password of the administrative user can be customized and there will no longer be an initial `admin` user with a default password. This provides much better secure defaults and makes it significantly less likely that attackers can get control of admin accounts in certain scenarios: - They can no longer assume there is an account named `admin`. - There is no longer a (short) time window during setup where you can compromise a hub with default credentials if it's exposed on the network. - It's no longer possible to forget / leave Mission Portal in a bad and insecure state, with `admin` as username and `admin` as password (misconfiguration). In the future, we will expand on this, so upgrading users are also able to get rid of the default `admin` user. ### New functions in policy language We're adding more utility functions to make policy language easier to use. Here are the 6 new ones added in 3.26.0: - [`findlocalusers()`](https://docs.cfengine.com/docs/3.26/reference-functions-findlocalusers.html) - [`getbundlemetatags()`](https://docs.cfengine.com/docs/3.26/reference-functions-getbundlemetatags.html) - [`hostswithgroup()`](https://docs.cfengine.com/docs/3.26/reference-functions-hostswithgroup.html) - [`is_type()`](https://docs.cfengine.com/docs/3.26/reference-functions-is_type.html) - [`isconnectable()`](https://docs.cfengine.com/docs/3.26/reference-functions-isconnectable.html) - [`useringroup()`](https://docs.cfengine.com/docs/3.26/reference-functions-useringroup.html) Each of these were added based on feedback from users, to make the lives of policy writers better. If you have a suggestion for a new policy function, feel free to create [an _idea_ in GitHub Discussion](https://github.com/cfengine/core/discussions/categories/ideas), or [a ticket in our bug tracker](https://northerntech.atlassian.net). ### Analyze your current policy set and get help with upgrading We're making several improvements to our command line tools for installing CFEngine remotely (`cf-remote`) and managing CFEngine policy (`cfbs`). One big new feature is the ability to analyze your current policy set, using `cfbs analyze`: ```command cfbs analyze ``` ```output Policy set path: . Reference version: 3.21.5 Files missing from the version: └── controls/cf_monitord.cf Files from the version but with modifications: └── promises.cf No files are from a different version. Files not from any version (with both custom content and path): └── services/my_policy.cf ``` This command highlights which version you're on, what custom edits you've made and whether there are some files (outliers) not matching your version, indicating mistakes in previous upgrades. With this information, it's easier to upgrade to a new version of the masterfiles policy framework or migrate to a CFEngine Build project where masterfiles is a separate module from your custom policy. Next, we're working on functionality to automatically achieve this, using the data provided by `cfbs analyze`. Read more about the new command in this blog post: https://cfengine.com/blog/2025/introducing-cfbs-analyze-command/ ### Quickly connect with SSH to hosts saved in cf-remote If you have a host named `hub` in `cf-remote` (a group named `hub` with one host) you can now easily `connect` to it with SSH: ```command cf-remote connect -H hub ``` ```output Opening a SSH command shell... The authenticity of host '54.216.94.247 (54.216.94.247)' can't be established. ED25519 key fingerprint is SHA256:xGx3IvoReyyIhz63OAHj3uhF/O+YVEaszgBpE2v4GOg. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '54.216.94.247' (ED25519) to the list of known hosts. Register this system with Red Hat Insights: insights-client --register Create an account or view all your systems at https://red.ht/insights-dashboard [ec2-user@ip-172-31-44-225 ~]$ ``` There's no longer a need for copy-pasting usernames and IP addresses, `cf-remote` can keep track of this for you. (Host first needs to be spawned using `cf-remote spawn` or saved using `cf-remote save`). ### Frequent releases to command line tools We release new versions of these convenience tools much more frequently than the CFEngine 6-month release cycle. You can stay up to date with these projects and new features in each version on GitHub: https://github.com/cfengine/cfbs/releases https://github.com/cfengine/cf-remote/releases ### Features from 3.25.0 The upcoming 3.27.0 LTS release will include all features from 3.25.0 and 3.26.0. See the blog post below, if you haven't already checked out audit logs and the other features in CFEngine 3.25.0: https://cfengine.com/blog/2025/cfengine-3-25-released-auditability/ ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.26.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.26/release-notes-whatsnew-changelog-core.html) - [3.26.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.26/release-notes-whatsnew-changelog-enterprise.html) - [3.26.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.26/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to [3.25.0](/blog/2025/cfengine-3-25-released-auditability/), these dependencies have been updated: | CFEngine version | 3.25.0 | **3.26.0** | | :------------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.62 | **2.4.63** | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.10 | **3.12** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.47.1 | **2.49.0** | | [libcurl](https://curl.se/download.html) | 8.11.1 | **8.13.0** | | [libcurl-hub](https://curl.se/download.html) | 8.11.1 | **8.13.0** | | [libexpat](https://libexpat.github.io/) | 2.6.3 | **2.7.1** | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.17 | **1.18** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.13.5 | **2.14.3** | | [OpenSSL](https://openssl.org/) | 3.4.0 | **3.5.0** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | 10.44 | **10.45** | | [PHP](https://php.net/) | 8.3.15 | **8.4.7** | | [PostgreSQL](https://www.postgresql.org/) | 17.2 | **17.5** | | [rsync](https://download.samba.org/pub/rsync/) | 3.3.0 | **3.4.1** | | [leech](https://github.com/larsewi/leech/releases) | - | **0.1.24** | | [nghttp2](https://nghttp2.org/) | - | **1.65.0** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download) to go to the download pages with new packages. If you are using `cf-remote`, you can specify version to install with the `--version` option: ```command cf-remote --version 3.26.0 install --hub hub --bootstrap hub ``` ## Security vulnerabilities, breaking changes, and known issues We'd like to highlight a few useful resources to all our users. When we discover and fix security issues in CFEngine, these are published as CVEs. Use the _cve_ tag on our blog to stay up to date with these announcements: https://cfengine.com/tags/cve/ Similarly, when we have to change the behavior of CFEngine and its policy language in ways that could be considered breaking, we publish a _change-in-behavior_ blog post: https://cfengine.com/tags/change-in-behavior/ (This is generally something we avoid, and so the examples of that are usually around unintended / accidental behavior, or making more explicit / strict safe defaults for things which could relate to security or safe / predictable behavior). Lastly, our documentation has a known issues page: https://docs.cfengine.com/docs/master/release-notes-known-issues.html ([Here](https://docs.cfengine.com/docs/3.26/release-notes-known-issues.html) is a permalink to the 3.25 version of that page). At the time of writing, there is one known issue for 3.26.0, preventing the hub package to install successfully on Ubuntu hosts, when using `cf-remote`. We recommend our users to use these resources to stay up to date on notable changes and issues. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net) -------------------------------------------------------------------------------- title: "CFEngine 3.21.7 and 3.24.2 released" url: /blog/2025/cfengine-3-21-7-and-3-24-2-released/index.md date: "2025-05-15" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.21.7 and 3.24.2\! These patch releases contain bug fixes and dependency updates. ## Changes As these are patch releases for long term supported (LTS) branches of CFEngine, there are no new major features included. ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.24.2 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-core.html) - [3.24.2 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-enterprise.html) - [3.24.2 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.7 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.7 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.7 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades Compared to the previous releases, CFEngine [3.21.6 and 3.24.1](/blog/2024/cfengine-3-21-6-and-3-24-1-released/), the following dependencies were updated: | CFEngine version | 3.21.6 | **3.21.7** | 3.24.1 | **3.24.2** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.62 | **2.4.63** | 2.4.62 | **2.4.63** | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.10 | **3.12** | 3.10 | **3.12** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.47.0 | **2.49.0** | 2.47.0 | **2.49.0** | | [libcurl](https://curl.se/download.html) | 8.10.1 | **8.13.0** | 8.10.1 | **8.13.0** | | [libcurl-hub](https://curl.se/download.html) | 8.10.1 | **8.13.0** | 8.10.1 | **8.13.0** | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.17 | **1.18** | 1.17 | **1.18** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.13.4 | **2.14.2** | 2.13.4 | **2.14.2** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.8 | **2.6.9** | 2.6.8 | **2.6.9** | | [OpenSSL](https://openssl.org/) | 3.0.15 | **3.0.16** | 3.4.0 | **3.5.0** | | [PHP](https://php.net/) | 8.3.13 | **8.3.21** | 8.3.13 | **8.3.21** | | [PostgreSQL](https://www.postgresql.org/) | 15.8 | **15.12** | 16.4 | **16.8** | | [rsync](https://download.samba.org/pub/rsync/) | 3.3.0 | **3.4.1** | 3.3.0 | **3.4.1** | | [libexpat](https://libexpat.github.io/) | - | - | 2.6.3 | **2.7.1** | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | - | - | 10.44 | **10.45** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.24/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.24.2 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.21.7 list ``` ```command cf-remote --version 3.24.2 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.21 LTS Please note that support for the 3.21 LTS series is ending later this year, in December. If you are on 3.21 or an older unsupported version, please [contact us](/contact) for help with upgrading or to discuss an extended support contract. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Package signing and verification" url: /blog/2025/package-signing/index.md date: "2025-05-12" -------------------------------------------------------------------------------- _Disclaimer: This post focuses on Debian-based and Fedora/RHEL-based distributions and packaging._ Everybody using a GNU/Linux distribution most likely knows that packages used by the given distribution are somehow signed and such signatures are somehow verified. Usually, this knowledge comes with the first requirement to import some key when an extra package repository is being added to the system (the standard repositories of a distribution use keys that are present and trusted by default). While users don't usually pay much attention to the key import process and the particular key used, these keys and signatures are actually parts of the critical security mechanisms in their systems. ## Purpose Why are these signatures a critical security mechanism? And what are they exactly? Package signatures are _digital signatures_[^1] and as such, they prove two things: - _authenticity_, i.e. that the signed data is coming from the particular originator, and - _integrity_, i.e. that the signed data was not tampered with. In the case of package signing, they can be used to verify that the software (or data, in general) contained in the package comes from the given _entity_ and that the contents are the same as when the signature was created, i.e. without extra content added later and without any modifications, introducing security issues, for example. The _entity_ can be a particular human (sometimes required), a company, or a service. However, a part of best practices is that the signatures are not created automatically as part of a build process, but rather as an explicit step done by a human, only on packages that are properly tested and approved for release. This is a practice the CFEngine team has been following as well. ## The key To be more precise, a successful verification proves that the given signature was created with a particular _key_. Whoever can access the key and the data can create a valid signature. As mentioned above, it can be a human, a build system, or a package repository system. A particular key can represent a person, a company, or a service (or a particular user using the service). In all cases, **it is the key that is trusted**. This means that it is important to pay enough attention to making sure that it is the particular key desired to be trusted. Ideally, the key should be coming from a different place and channel than the signed data. Such a setup requires an attacker to control two sources/channels if they want to provide fraud packages. In all cases, the key's fingerprint and metadata should be verified before it is imported and thus trusted. For example, when installing with `yum`, after adding the CFEngine Community Edition repository and pointing it to the particular key, you get a prompt to confirm the fingerprint and import the key: ```command sudo yum install cfengine-community ``` ```output Importing GPG key 0x9B920A5E: Userid : "CFEngine Packages (Key for signing releases) " Fingerprint: F2EE 2A0E 76A6 D469 8AE5 301A 7420 C142 9B92 0A5E From : https://cfengine-package-repos.s3.amazonaws.com/pub/gpg.key Is this ok [y/N]: ``` and if we check the [instructions for using the repository](https://cfengine.com/cfengine-linux-distros/), it says that the fingerprint of the key used for CFEngine Community packages is ``` F2EE 2A0E 76A6 D469 8AE5 301A 7420 C142 9B92 0A5E ``` Cross-checking the fingerprint in the `yum` prompt and on the website allows the user to verify that the key associated with the repository is actually the expected (and trusted) one. Once accepted, the key is imported and marked as trusted in the system and `yum` then verifies the digital signature of the downloaded package before installing it. ## What is being signed In the above command, `yum` fetches the repository metadata, determines which packages it needs to download to install `cfengine-community`, downloads the package(s), downloads the particular key, and shows the prompt. If rejected or if `yum` doesn't know which key to import, the package installation fails with an error. However, if we try a similar thing on a Debian-based system and **skip the step to import the key**, this is the error we get from: ```command sudo apt-get update ``` ```output Err:6 https://cfengine-package-repos.s3.amazonaws.com/pub/apt/packages stable InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7420C1429B920A5E E: The repository 'https://cfengine-package-repos.s3.amazonaws.com/pub/apt/packages stable InRelease' is not signed. N: Updating from such a repository can't be done securely, and is therefore disabled by default. ``` We can see a significant difference in the above. While `yum` checks the signature of the downloaded package, `apt-get` checks the signature of the repository metadata. From a user perspective, as long as everything works, it is all the same -- import a key and have signatures verified. However, the big difference is that while in the case of Fedora/RHEL-based distributions using RPM packages, individual packages each have a signature, on Debian-based distributions, it's the repository metadata that is signed. In particular, the `Release` file in the repository is signed, which contains a checksum of the `Packages` file, which provides information about packages present in the repository together with checksums of the individual packages. The trust and verification mechanisms are different than in the case of RPM-based distributions. However, even in the case of Debian-based distributions each package is verified by comparing the checksum of the downloaded package with the checksum declared in the metadata (`Packages`) and the checksum of this piece of metadata is compared to the checksum in the `Release` file. And since this piece of metadata is trusted as long as the key it is signed with is trusted, there exists a _chain of trust_ from the key to every individual package in the repository. ## Standalone packages As written above, when using a repository, there's no functional difference between the approach to signatures in the Fedora/RHEL-based and Debian-based distributions. However, what happens when a standalone package, i.e. an `.rpm` (RPM) or `.deb` (DEB) file is being installed? There is no repository and thus on a Debian-based distribution, there is no way to verify the particular package (there is no signature). One would expect that the result must be that `dpkg -i ./some_package.deb` (or `apt install ./some_package.deb`) installs the file just fine and `rpm -i ./some_package.rpm` (or `yum install ./some_package.rpm`) reject the file because the file is signed and the key is not available. However, it's not so easy. **By default**, the result is the same, the package is installed in both cases. `rpm -i` only gives a warning like this: ```command sudo rpm -i ./cfengine-nova-3.24.1-1.el9.x86_64.rpm ``` ```output warning: ./cfengine-nova-3.24.1-1.el9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY ``` but the package is installed. The [recommended approach](https://docs.cfengine.com/docs/master/getting-started-installation-pre-installation-checklist-verify-signatures.html#validating-signature-of-rpm) is to use `rpm -K` to verify the signature of the file before installing it.[^2] ## Beyond default (practices) The above describes the default behavior of the tools and, indirectly, the default (packaging) practices. Yet, the area of packaging, even when limited to Fedora/RHEL-based and Debian-based distributions, is wide, with many alternatives. ### Forcing rpm to do package verification For example, to follow up on the last case mentioned above, there is a way to force `rpm` to actually reject such packages: ```command sudo echo '%_pkgverify_level all' >> /etc/rpm/macros sudo rpm -i ./cfengine-nova-3.24.1-1.el9.x86_64.rpm ``` ```output warning: ./cfengine-nova-3.24.1-1.el9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY package cfengine-nova-3.24.1-1.el9.x86_64 does not verify: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY ``` The same warning is printed, but this time it is followed by a statement that the package does not verify and the package is **not** installed. `yum` (or `dnf`) has the `localpkg_gpgcheck` configuration option to enforce the same behavior[^3] ### DEB package signatures The [Securing Debian Manual](https://www.debian.org/doc/manuals/securing-debian-manual/deb-pack-sign.en.html) explains how the standard scheme for signatures works, the way described above. At the end of the page, however, it also describes an _Alternative per-package signing scheme_ in which individual packages are signed, similarly to how the default RPM package signing schema works. In fact, this is the approach the CFEngine team is using to allow [verification of individual DEB packages](https://docs.cfengine.com/docs/master/getting-started-installation-pre-installation-checklist-verify-signatures.html#validating-signature-of-deb) before installation, which is a non-trivial process requiring extra tools and a custom XML config file. ### RPM repo signatures For completeness, we need to mention that `yum` and `dnf` have the `repo_gpgcheck` configuration option which, when enabled, will trigger signature verification of repository metadata, similarly to how `apt` and `apt-get` do it in Debian-based distributions. ### Avoiding signatures As described in this blog post, there are multiple different approaches to how signatures can be used to verify packages. Or more precisely, how to verify their authenticity and integrity. They have their pros and cons and limitations and all require a key to be verified and imported/trusted. To avoid their complexities and to require the user to carefully check and accept a key and to provide an easy verification of standalone packages, a commonly used alternative is to provide digests (fingerprints) of strong cryptographic checksums of the published packages. A strong cryptographic checksum (hash) ensures that the package contents were not tampered with. And if the source of the checksum digests can be trusted, ideally being a separate channel from the one used when downloading the package, comparing the checksum digest (fingerprint) of a downloaded package with the expected and trusted value can work as both integrity and authenticity verification. CFEngine release announcement emails use this practice to give users an alternative to signature-based package verification. ## Summary This blog post provides an overview of how package verification using digital signatures works in Fedora/RHEL-based and Debian-based GNU/Linux distributions. The schemes used are surprisingly different and the default practices and behavior of the default and commonly used tools can be surprising too. Thus the text also provides suggestions for how to enforce verification of packages in all scenarios and what to pay attention to in order to avoid undermining this critical part of the security mechanisms in modern GNU/Linux systems. [^1]: Not to be confused with _electronic signatures_ which also cover scans or pictures of human-written signatures and similar things. [^2]: You will most likely not find `-K/--checksig` in the rpm's man page, it is actually a separate command `rpmkeys` nowadays. [^3]: The enforcing policy for `rpm` applies to `yum/dnf` as well, if used. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 48 - Using CFEngine for Android application inventory & management" url: /blog/2025/the-agent-is-in-episode-48/index.md date: "2025-04-24" -------------------------------------------------------------------------------- Manage your phone like a server. In this episode, Craig brings CFEngine to Android with custom package modules and creative hacks. Cody and Herman join Craig while he shares his journey of integrating CFEngine into a Fairphone 4 running /e/OS (a privacy-focused Android fork), using Termux to enable a Unix-like environment, and extending CFEngine's capabilities to manage and inventory Android applications. He talks about running Termux on Android to create a CFEngine-compatible environment, a custom package module to install APKs from GitHub, enabling inventory collection and version tracking via ADB. Craig showed how the collected data is displayed in Mission Portal, offering visibility into Android environments. Craig also talks about other practical use cases, such as using CFEngine to purge large log files on a smartwatch to keep it functioning smoothly. ## Video {#video} The video recording is available on YouTube: {{< youtube vO0lL2obYc8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Herman](https://www.linkedin.com/in/olehermanse) and [Craig](/author/craig-comstock/). - - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Introducing cfbs 4.4.0 and the analyze command" url: /blog/2025/introducing-cfbs-analyze-command/index.md date: "2025-04-08" -------------------------------------------------------------------------------- The latest release of `cfbs` ([4.4.0](https://github.com/cfengine/cfbs/releases/tag/4.4.0) released April 4th, 2025) introduces the `analyze` command. Last time I used this ([Show notes: The agent is in - Episode 47 - Preview of cfbs analyze](/blog/2025/the-agent-is-in-episode-47/)) I had installed it from a git clone, now I want to go back to regular install ```command pipx uninstall cfbs pipx install cfbs ``` ```output uninstalled cfbs! ✨ 🌟 ✨ installed package cfbs 4.4.0, installed using Python 3.12.3 These apps are now globally available - cfbs ``` Now, `cfbs help` should have our new cfbs analyze option: ```command cfbs help ``` ```output usage: cfbs [-h] [--loglevel LOGLEVEL] [-M] [--version] [--force] [--non-interactive] [--index INDEX] [--check] [--checksum CHECKSUM] [--keep-order] [--git {yes,no}] [--git-user-name GIT_USER_NAME] [--git-user-email GIT_USER_EMAIL] [--git-commit-message GIT_COMMIT_MESSAGE] [--ignore-versions-json] [--omit-download] [--check-against-git] [--from MINIMUM_VERSION] [--to-json [TO_JSON]] [--reference-version REFERENCE_VERSION] [--masterfiles-dir MASTERFILES_DIR] [--ignored-path-components [IGNORED_PATH_COMPONENTS ...]] [--offline] [--masterfiles MASTERFILES] [cmd] [args ...] CFEngine Build System. positional arguments: cmd The command to perform (pretty, init, status, search, add, remove, clean, update, validate, download, build, install, help, info, show, analyse, analyze, input, set-input, get-input, generate-release-information) args Command arguments options: -h, --help show this help message and exit --loglevel LOGLEVEL, -l LOGLEVEL Set log level for more/less detailed output -M, --manual Print manual page --version, -V Print version number --force Force rebuild / redownload --non-interactive Don't prompt, use defaults (only for testing) --index INDEX Specify alternate index --check Check if file(s) would be reformatted --checksum CHECKSUM Expected checksum of the downloaded file --keep-order Keep order of items in the JSON in 'cfbs pretty' --git {yes,no} Override git option in cfbs.json --git-user-name GIT_USER_NAME Specify git user name --git-user-email GIT_USER_EMAIL Specify git user email --git-commit-message GIT_COMMIT_MESSAGE Specify git commit message --ignore-versions-json Ignore versions.json. Necessary in case of a custom index or testing changes to the default index. --omit-download Use existing masterfiles instead of downloading in 'cfbs generate-release-information' --check-against-git Check whether masterfiles from cfengine.com and github.com match in 'cfbs generate-release- information' --from MINIMUM_VERSION Specify minimum version in 'cfbs generate-release- information' --to-json [TO_JSON] Output 'cfbs analyze' results to a JSON file; optionally specify the JSON's filename --reference-version REFERENCE_VERSION Specify version to compare against for 'cfbs analyze' --masterfiles-dir MASTERFILES_DIR If the path given to 'cfbs analyze' contains a masterfiles subdirectory, specify the subdirectory's name --ignored-path-components [IGNORED_PATH_COMPONENTS ...] Specify path components which should be ignored during 'cfbs analyze' (the components should be passed separately, delimited by spaces) --offline Do not connect to the Internet to download the latest version of MPF release information during 'cfbs analyze' --masterfiles MASTERFILES Add masterfiles on cfbs init choose between ``` Let's grab oldest version of the Masterfiles Policy Framework that `cf-remote` knows about and test it out. ```command OLDEST_MPF_IN_CFREMOTE="$(cf-remote list | head -n1 | awk '{print $NF}')" URL_OLDEST_MPF_IN_CFREMOTE="$(cf-remote --version ${OLDEST_MPF_IN_CFREMOTE} list masterfiles | tail -n1)" TARBALL_FILENAME_OLDEST_MPF_IN_CFREMOTE="$(basename ${URL_OLDEST_MPF_IN_CFREMOTE})" cf-remote --version 3.21.0 download masterfiles --output-dir . tar --no-same-permissions --no-overwrite-dir -zxf ${TARBALL_FILENAME_OLDEST_MPF_IN_CFREMOTE} 2> /dev/null mv masterfiles "masterfiles-${OLDEST_MPF_IN_CFREMOTE}" ``` ```output Available releases: master, 3.25.0, 3.24.x, 3.24.1, 3.24.0, 3.21.x, 3.21.6, 3.21.5, 3.21.4, 3.21.3, 3.21.2, 3.21.1, 3.21.0 Using 3.21.0 LTS: Downloading package: '/home/nickanderson/.cfengine/cf-remote/packages/cfengine-masterfiles-3.21.0-1.pkg.tar.gz' Copied to '/tmp/cfengine-masterfiles-3.21.0-1.pkg.tar.gz' (Checksum OK). ``` If we analyze it: ```command cfbs analyze "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}" ``` We can see that it appears to be completely un-modified, as expected. ```output Policy set path: ./masterfiles-3.21.0 Reference version: 3.21.0 No files are missing from the version. No files of the version are modified. No files are from a different version. No files are not from any version. ``` Now, let's make some representative modifications that are typically found in policy sets: Let's remove one of the vendored files: ```command # Trigger Files missing from the version: rm -f "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}/services/autorun/hello.cf" ``` Let's customize a vendored policy file: ```command # Trigger Files from the version but with modifications: echo 'bundle agent mpf_main { reports: "Customized services/main.cf"; } bundle agent hello_world { reports: "Hello World!"; }' > "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}/services/main.cf" ``` Let's introduce a new file: ```command # Trigger Files not from any version (with both custom content and path): echo '{ "variables": { "default:def.bundlesequence_end": [ "hello_world" ] } }' > "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}/def.json" ``` And, let's introduce a file (but with different content) that would have been present in an older release of the MPF: ```command # Files from a different version, with modifications: mkdir -p "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}/lib/3.7/" echo "# Oops, we neglected to clean up this file from an old MPF version. " > "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}/lib/3.7/services.cf" ``` Now let's analyze it: ```command cfbs analyze "./masterfiles-${OLDEST_MPF_IN_CFREMOTE}" ``` Looking at the result we see the files indicated in the appropriate section. For `lib/3.7/services.cf` notice that 4 different versions of the MPF were known to contain this file ('3.7.8', '3.7.7', '3.7.6', '3.7.5'), and it highlights the fact that the file still differed from what was included with those releases. ```output Policy set path: ./masterfiles-3.21.0 Reference version: 3.21.0 Files missing from the version: └── services/autorun/hello.cf Files from the version but with modifications: └── services/main.cf Files from a different version, with modifications: └── ('lib/3.7/services.cf', ['3.7.8', '3.7.7', '3.7.6', '3.7.5']) Files not from any version (with both custom content and path): └── def.json ``` We aim to add additional functionality to this tooling, but we would love to hear your feedback on [GitHub discussions](https://github.com/cfengine/core/discussions) or our [mailing list](https://groups.google.com/g/help-cfengine). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 47 - Preview of cfbs analyze" url: /blog/2025/the-agent-is-in-episode-47/index.md date: "2025-03-27" -------------------------------------------------------------------------------- Is your CFEngine policy set a Frankensteinian monster of outdated files and custom tweaks? Take a look at tooling we have in development (`cfbs analyze`) to help shed light on the makeup of your policy set. In this episode the team showcases a new tool for analyzing CFEngine policy sets. Learn how to identify modifications, missing files, and stale policies from old MPF versions and hear about additional features we can look forward to when it's fully released. ## Video {#video} The video recording is available on YouTube: {{< youtube 5no9iBiEQG8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Take it for a spin {#take-it-for-a-spin} First, we need to get `cfbs` installed. Remember this is a work in progress, we need to grab it from a GitHub pull request to the cfbs repo, [#218](https://github.com/cfengine/cfbs/pull/218) specifically. Before I install from this branch I first want to remove any pre-existing install of `cfbs`. ```command pipx uninstall cfbs ``` ```output uninstalled cfbs! ✨ 🌟 ✨ ``` Next we install from the checked out branch: ```command pipx install . ``` ```output installed package cfbs 2.4.3+351.git.g41c9be7, installed using Python 3.12.3 These apps are now globally available - cfbs ``` Now that we have the development version of `cfbs` installed lets grab the oldest version of the MPF we can easily get with cfbs (`3.21.0` ). ```command OLDEST_MPF_IN_CFBS="$(cf-remote list | head -n1 | awk '{print $NF}')" URL_OLDEST_MPF_IN_CFBS="$(cf-remote --version ${OLDEST_MPF_IN_CFBS} list masterfiles | tail -n1)" TARBALL_FILENAME_OLDEST_MPF_IN_CFBS="$(basename ${URL_OLDEST_MPF_IN_CFBS})" wget ${URL_OLDEST_MPF_IN_CFBS} tar --no-same-permissions --no-overwrite-dir -mzxf ${TARBALL_FILENAME_OLDEST_MPF_IN_CFBS} mv masterfiles "masterfiles-${OLDEST_MPF_IN_CFBS}" ``` ```output --2025-03-27 12:25:02-- https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.21.0/misc/cfengine-masterfiles-3.21.0-1.pkg.tar.gz Resolving cfengine-package-repos.s3.amazonaws.com (cfengine-package-repos.s3.amazonaws.com)... 52.216.216.81, 3.5.25.199, 52.216.219.9, ... Connecting to cfengine-package-repos.s3.amazonaws.com (cfengine-package-repos.s3.amazonaws.com)|52.216.216.81|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 212907 (208K) [application/x-tar] Saving to: ‘cfengine-masterfiles-3.21.0-1.pkg.tar.gz’ cfengine- 0%[ ] 0 --.-KB/s cfengine-masterfile 100%[===================>] 207.92K --.-KB/s in 0.1s 2025-03-27 12:25:03 (1.58 MB/s) - ‘cfengine-masterfiles-3.21.0-1.pkg.tar.gz’ saved [212907/212907] ``` Let's analyze it. ```command cfbs analyze "./masterfiles-${OLDEST_MPF_IN_CFBS}" ``` ```output Policy set path: ./masterfiles-3.21.0 Same filepath versions distribution: [('3.21.0', 130), ('3.21.1', 127), ('3.21.2', 111), ('3.22.0', 110), ('3.21.3', 107), ('3.18.4', 106), ('3.18.3', 106), ('3.20.0', 104), ('3.18.5', 97), ('3.23.0', 96), ('3.21.4', 96), ('3.18.6', 96), ('3.18.2', 94), ('3.21.5', 89), ('3.21.6', 86), ('3.18.7', 85), ('3.24.0', 81), ('3.18.8', 80), ('3.19.0', 79), ('3.25.0', 78), ('3.24.1', 78), ('3.18.1', 78), ('3.18.0', 71), ('3.17.0', 55), ('3.15.7', 54), ('3.15.6', 54), ('3.15.5', 49), ('3.16.0', 36), ('3.15.4', 35), ('3.15.3', 34), ('3.15.2', 33), ('3.15.1', 31), ('3.15.0', 30), ('3.12.7', 29), ('3.12.6', 28), ('3.12.5', 28), ('3.12.4', 28), ('3.12.3', 26), ('3.15.0b1', 25), ('3.14.0', 23), ('3.12.2', 21), ('3.13.0', 20), ('3.12.1', 20), ('3.12.0', 19), ('3.12.0b1', 19), ('3.10.7', 15), ('3.10.6', 15), ('3.10.5', 15), ('3.10.4', 15), ('3.10.3', 14), ('3.11.0', 11), ('3.11.0b1', 11), ('3.10.2', 11), ('3.10.1', 11), ('3.10.0', 11), ('3.10.0b1', 11), ('3.9.2', 9), ('3.7.8', 9), ('3.7.7', 8), ('3.7.6', 7), ('3.7.5', 7)] Same filepath highest versions distribution: [('3.25.0', 78), ('3.21.1', 16), ('3.23.0', 15), ('3.22.0', 14), ('3.24.0', 3), ('3.21.0', 3), ('3.21.4', 1)] Reference version: 3.21.0 No files are missing from the version. No files of the version are modified. No files are from a different version. No files are not from any version. ``` Now let's seed some typical modifications to the base. ```command # Trigger Files missing from the version: rm -f "./masterfiles-${OLDEST_MPF_IN_CFBS}/services/autorun/hello.cf" # Trigger Files from the version but with modifications: echo 'bundle agent mpf_main { reports: "Customized services/main.cf"; } bundle agent hello_world { reports: "Hello World!"; }' > "./masterfiles-${OLDEST_MPF_IN_CFBS}/services/main.cf" # Trigger Files not from any version (with both custom content and path): echo '{ "variables": { "default:def.bundlesequence_end": [ "hello_world" ] } }' > "./masterfiles-${OLDEST_MPF_IN_CFBS}/def.json" # Files from a different version, with modifications: mkdir -p "./masterfiles-${OLDEST_MPF_IN_CFBS}/lib/3.7/" echo "# Oops, we neglected to clean up this file from an old MPF version. " > "./masterfiles-${OLDEST_MPF_IN_CFBS}/lib/3.7/services.cf" ``` If we analyze it again: ```command cfbs analyze "./masterfiles-${OLDEST_MPF_IN_CFBS}" ``` We see that it correctly identified that `services/autorun/hello.cf` was missing, `services/main.cf` had been modified, `lib/3.7/services.cf` seemed to be a modified version from an old MPF version, and `def.json` was a custom addition. ```output Policy set path: ./masterfiles-3.21.0 Same filepath versions distribution: [('3.21.0', 128), ('3.21.1', 125), ('3.21.2', 109), ('3.22.0', 108), ('3.21.3', 105), ('3.18.4', 105), ('3.18.3', 105), ('3.20.0', 102), ('3.18.5', 96), ('3.18.6', 95), ('3.23.0', 94), ('3.21.4', 94), ('3.18.2', 93), ('3.21.5', 87), ('3.21.6', 84), ('3.18.7', 84), ('3.24.0', 79), ('3.18.8', 79), ('3.19.0', 78), ('3.18.1', 77), ('3.25.0', 76), ('3.24.1', 76), ('3.18.0', 70), ('3.17.0', 54), ('3.15.7', 53), ('3.15.6', 53), ('3.15.5', 48), ('3.16.0', 35), ('3.15.4', 34), ('3.15.3', 33), ('3.15.2', 32), ('3.15.1', 30), ('3.15.0', 29), ('3.12.7', 28), ('3.12.6', 27), ('3.12.5', 27), ('3.12.4', 27), ('3.12.3', 25), ('3.15.0b1', 24), ('3.14.0', 22), ('3.12.2', 20), ('3.13.0', 19), ('3.12.1', 19), ('3.12.0', 18), ('3.12.0b1', 18), ('3.10.7', 14), ('3.10.6', 14), ('3.10.5', 14), ('3.10.4', 14), ('3.10.3', 13), ('3.11.0', 10), ('3.11.0b1', 10), ('3.10.2', 10), ('3.10.1', 10), ('3.10.0', 10), ('3.10.0b1', 10), ('3.9.2', 8), ('3.7.8', 8), ('3.7.7', 7), ('3.7.6', 6), ('3.7.5', 6)] Same filepath highest versions distribution: [('3.25.0', 76), ('3.21.1', 16), ('3.23.0', 15), ('3.22.0', 14), ('3.24.0', 3), ('3.21.0', 3), ('3.21.4', 1)] Reference version: 3.21.0 Files missing from the version: └── services/autorun/hello.cf Files from the version but with modifications: └── services/main.cf Files from a different version, with modifications: └── ('lib/3.7/services.cf', ['3.7.8', '3.7.7', '3.7.6', '3.7.5']) Files not from any version (with both custom content and path): └── def.json ``` ## Post Show Discussion {#post-show-discussion} ### How to suppress errors from promises not kept? {#how-to-suppress-errors-from-promises-not-kept} For example, a commands promise to add a user to a group. It's expected that it may fail. 1. Generically, avoid making the promise when it's not necessary. CFEngine does this internally but when running commands for things it's the users responsibility to properly contextualize promises. E.g. first probe to find out if the group already exists to determine if it actually needs to be added or not. 2. Consider overriding the interpretation of return codes for that specific promise. 3. Consider using the [groups custom promise type](https://build.cfengine.com/modules/promise-type-groups/). It's written in Python, so much more accessible if the implementation needs to be tweaked to support your specific case. Also, keep an eye out for a new function `findlocalusers()` [on the horizon](https://github.com/cfengine/core/pull/5714). ### Various topics {#various-topics} * **Fairphone Love:** Craig shared his enthusiasm for Fairphone, a repairable and customizable phone. They even discussed installing alternative operating systems on it. Craig may or may not be a Fairphone sponsor. The jury is still out. * **Hardware Woes:** There were tales of hardware failures, including a dead Ryzen processor. * **Android devices:** Craig is investigating a Google Play packages promise implementation. * **Dumber Smart Phones:** [Cat S22 flip phone](https://www.amazon.com/Cat-S22-Flip-Touchscreen-Resistant/dp/B0BTTQGVVW) for minimizing social media exposure. But is there something similar that works well with Android Auto? ## Links {#links} * Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) * [cfbs analyze in development](https://github.com/cfengine/cfbs/pull/218) * [groups custom promise type](https://build.cfengine.com/modules/promise-type-groups/) * [Fairphone 4](https://shop.fairphone.com/fairphone-4) * [Cat S22 flip phone](https://www.amazon.com/Cat-S22-Flip-Touchscreen-Resistant/dp/B0BTTQGVVW) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Efficient file streams with librsync" url: /blog/2025/efficient-file-streams-with-librsync/index.md date: "2025-03-06" -------------------------------------------------------------------------------- In the latest version of the [CFEngine](https://cfengine.com/) network protocol (filestream - v4), we leveraged [`librsync`](https://github.com/librsync/librsync) for efficient file copying using their [Streaming API](https://github.com/librsync/librsync/blob/master/doc/streaming.md). While implementing the file streaming in CFEngine, I found that the documentation on the Streaming API was a bit unclear. Thus, I created two example programs to experiment with how it works. I thought I'd share them in this blog post as a tutorial to help other developers get up to speed faster. The complete files after following this tutorial should look more or less like these: - [server.c](efficient-file-streams-with-librsync/server.c) - [client.c](efficient-file-streams-with-librsync/client.c) - [common.h](efficient-file-streams-with-librsync/common.h) - [Makefile](efficient-file-streams-with-librsync/Makefile) ## Implementing a server To implement efficient file streaming, we'll first need a client and a server to connect and communicate through sockets. Let's start with implementing the server, given that a client cannot function without a server. We'll start by creating a file `server.c` with the entry point `main()` calling a function `accept_connection()` (I also included all the header files we'll need throughout this blog post). ```C {file="server.c"} #include #include #include #include #include #include #include #include #include #include "common.h" static int accept_connection(void); int main(int argc, char *argv[]) { puts("Waiting for connection..."); int sock = accept_connection(); if (sock == -1) { return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } static int accept_connection(void) { /* Create socket */ int sock = socket(AF_INET, SOCK_STREAM, 0); if (sock == -1) { perror("Failed to create socket"); return -1; } /* Enable reuse address (to avoid "Address already in use" errors */ int opt = 1; setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); /* Assign IP address and port */ struct sockaddr_in server_addr = { 0 }; server_addr.sin_family = AF_INET; server_addr.sin_addr.s_addr = htons(INADDR_ANY); /* Use local IP */ server_addr.sin_port = htons(PORT); /* Bind socket to given IP address */ int ret = bind(sock, (struct sockaddr *)&server_addr, sizeof(server_addr)); if (ret == -1) { perror("Failed to bind socket"); close(sock); return -1; } /* Listen for incoming connections. In a real-world application you should have a larger "connection request" queue. */ ret = listen(sock, 1); if (ret == -1) { perror("Failed to listen"); close(sock); return -1; } /* Accept incoming connection */ struct sockaddr_in client_addr; socklen_t addr_len; int conn = accept(sock, (struct sockaddr *)&client_addr, &addr_len); /* We don't expect any more connections in this example. In a real-world application you should keep this socket open to accept more connections. */ close(sock); if (conn == -1) { perror("Failed to accept"); return -1; } return conn; } ``` I won't go into great detail about network programming. If there is something you don't understand, I would highly recommend reading [Beej's Guide to Network Programming](https://beej.us/guide/bgnet/). It's free and written in a language mortal programmers can understand. Nonetheless, here is a short description of what the code does. We create an endpoint for communication through TCP/IP using the "system call" `socket()` which provides a file descriptor for reading and writing. Next comes a little hack that can save you some time and frustration. Soon, we'll bind our newly created socket to a port on our machine. However, when rerunning the server program, one might experience errors such as "Address already in use." This is because parts of the socket may still linger in the kernel after exiting our program. We can either wait for the socket to clear or tell the kernel to allow the port to be reused. Finally, we bind the server to a local IPv4 address and a port (that we will later specify using a macro definition), followed by listening for incoming connections. In the call to `listen()`, I specified `1` as the backlog argument. You'd probably want a grander connection request queue in a real-world application. Otherwise, the server would accept the first connection and refuse all others while processing the first one. The program blocks until a connection request is received. At this point, we will have to decide whether to accept. Of course - we'll give the nod - because "Happiness can exist only in acceptance" (George Orwell). The call to `accept()` returns a new connected socket for talking to the client. Given that we don't expect any more connections in our example server program, we will close our listening socket and return the connected socket. We are almost ready to compile our server program. However, we still need to define the `PORT` macro. We will add the definition in a shared header file, `common.h`. This way, the client program will also be able to access it. I took the liberty to include all of the header files that we'll need throughout this blog post here as well. ```C {file="common.h"} #ifndef COMMON_H #define COMMON_H #include #include #include #include #include #include /** The port the server will be listening on / connecting to */ #define PORT 5612 #endif /* COMMON_H */ ``` We should now be able to compile the server program using the following command. ```command gcc -g -o server server.c ``` Run it with the following command. ```command ./server ``` ```output Waiting for connection... ^C ``` However, no clients are connecting to it. Thus, the program is blocked forever. Type CTRL+C to stop it. ## Implementing a client A server without any clients ain't no fun. Thus, let's create a new file `client.c` - also with the entry point `main()` calling the function `connect_to_server()`. As always, I included all the header files we'll need throughout. ```C {file="client.c"} #include #include #include #include #include #include #include #include #include #include #include #include "common.h" #define IP_ADDRESS "127.0.0.1" static int connect_to_server(const char *ip_addr); int main(int argc, char *argv[]) { puts("Connecting to server..."); int sock = connect_to_server(IP_ADDRESS); if (sock == -1) { return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } static int connect_to_server(const char *ip_addr) { /* Create socket */ int sock = socket(AF_INET, SOCK_STREAM, 0); if (sock == -1) { perror("Failed to create socket"); return -1; } /* Assign IP address and port */ struct sockaddr_in addr = { 0 }; addr.sin_family = AF_INET; addr.sin_addr.s_addr = inet_addr(ip_addr); addr.sin_port = htons(PORT); /* Connect to server */ int ret = connect(sock, (struct sockaddr *)&addr, sizeof(addr)); if (ret == -1) { perror("Failed to connect"); close(sock); return -1; } return sock; } ``` We create a new socket, just like in the server program. However, for the client, we call the `connect()` function using the loop-back address `127.0.0.1`. There is nothing more to it. It's pretty straightforward. By now, we are ready to compile the client. However, I cannot talk on behalf of everyone else, but I, for one, am tired of typing these `gcc` commands. Instead, let's create a new file, `Makefile`, to take care of this once and for all. ```Makefile {file="Makefile"} CFLAGS = -g -Wall -Wextra -Wconversion LDLIBS = -lrsync .PHONY: all clean all: server client server: server.c common.h client: client.c common.h clean: rm -f server rm -f client rm -f *.o ``` Perfect! Now, we compile both our programs by simply running this one command: ```command make ``` ```output cc -g -Wall -Wextra -Wconversion server.c common.h -lrsync -o server ---snip--- cc -g -Wall -Wextra -Wconversion client.c common.h -lrsync -o client ---snip--- ``` Don't worry about any warnings related to unused arguments. We will take care of these later. Now, let's open two terminal sessions. In the first one, we run the server: ```command ./server ``` ```output Waiting for connection... ``` In the second one, we run the client: ```command ./client ``` ```output Connecting to server... ``` It works! Although, there's not much to it yet. The client connects to the server. Then, both ends close the connection and exit. ## Implementing a communication protocol By now, we should be able to send and receive messages. However, knowing when the other end is finished sending and ready to receive remains an issue for us to solve. We can solve it by building an application layer network protocol on top of TCP/IP. Most importantly, we need a cool name, like SNP (short for Simple Network Protocol.) Yup, that name is definitely original and has never been used before. Trust me! Network protocols usually send data in chunks called PDUs (Protocol Data Units). Furthermore, PDUs are built from a protocol header and a payload, often called an SDU (Service Data Unit). The SDU usually contains the actual data that you want to transmit. While the protocol header provides information such as the payload size and whether to expect more data (among other things). We should aim for restricted bandwidth consumption and forward compatibility when designing the protocol. In other words, we want to pack the required information into as few bits as possible while reserving some for the future. Hence, I suggest a protocol header consisting of 16 bits distributed as follows: | SDU Len | Reserved | EoF Flag | |---------|----------|----------| | 12 bits | 3 bits | 1 bit | - **SDU Len:** 12 bits are allocated to indicate the payload length within the datagram. Providing a total length of 4095 bytes. - **Reserved:** 3 bits are reserved for future use. - **EoF Flag:** 1 bit determines whether the receiver should expect to receive more PDUs. Now that we have a bulletproof design, we can implement two utility functions for sending and receiving messages. Add the following code to `common.h`: ```C {file="common.h"} /** It's the largest value we can represent with the 12 bits in the SDU Length * header field (2^12-1 = 4095). Hence, it's the largest allowed payload. */ #define BUFFER_SIZE 4095 static int send_message(int sock, const char *msg, size_t len, int eof) { assert(len <= BUFFER_SIZE); uint16_t header = (uint16_t)len << 4; /* Set EoF flag */ if (eof != 0) { header |= 1; } /* Send header */ header = htons(header); size_t n_bytes = 0; do { ssize_t ret = write(sock, &header + n_bytes, sizeof(header) - n_bytes); if (ret < 0) { if (errno == EINTR) { /* The call was interrupted by a signal before any data was written. It is safe to continue. */ continue; } perror("Failed to send message header"); return -1; } n_bytes += (size_t)ret; } while (n_bytes < sizeof(header)); if (len > 0) { /* Send payload */ n_bytes = 0; do { ssize_t ret = write(sock, msg + n_bytes, len - n_bytes); if (ret < 0) { if (errno == EINTR) { /* The call was interrupted by a signal before any data was written. It is safe to continue. */ continue; } perror("Failed to send message payload"); return -1; } n_bytes += (size_t)ret; } while (n_bytes < len); } return 0; } static int recv_message(int sock, char *msg, size_t *len, int *eof) { /* Receive header */ uint16_t header; size_t n_bytes = 0; do { ssize_t ret = read(sock, &header + n_bytes, sizeof(header) - n_bytes); if (ret < 0) { if (errno == EINTR) { /* The call was interrupted by a signal before any data was read. It is safe to continue. */ continue; } perror("Failed to receive message header"); return -1; } n_bytes += (size_t)ret; } while (n_bytes < sizeof(header)); header = ntohs(header); /* Extract EOF flag */ *eof = header & 1; /* Extract message length */ *len = header >> 4; if (*len > 0) { /* Read payload */ n_bytes = 0; do { ssize_t ret = read(sock, msg + n_bytes, *len - n_bytes); if (ret < 0) { if (errno == EINTR) { /* The call was interrupted by a signal before any data was read. It is safe to continue. */ continue; } perror("Failed to receive message payload"); return -1; } n_bytes += (size_t)ret; } while (n_bytes < *len); } return 0; } ``` The function `send_message()` marshals and sends the protocol header, followed by the payload. The `recv_message()` function reads and unmarshals the protocol header first. Based on the header fields, it determines how many bytes to read to get the entire payload. Furthermore, it determines whether to expect more PDUs. Different computer architectures can have different byte orders. Hence, we must convert the 16-bit header from host to network byte order and back. Otherwise, the header can be misinterpreted. Please note the use of `htons()` and `ntohs()` in `send_message()` and `recv_message()`. Let's test the protocol by adding two calls to `send_message()` in `client.c`. In the second call to `send_message()`, we set the End-of-File flag. ```C {file="client.c"} int main(int argc, char *argv[]) { ... const char *first_msg = "Hello"; if (send_message(sock, first_msg, strlen(first_msg), 0) == -1) { close(sock); return EXIT_FAILURE; } const char *second_msg = " from client\n"; if (send_message(sock, second_msg, strlen(second_msg), 1) == -1) { close(sock); return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } ``` Similarly, we repeat calling the `recv_message()` function in `server.c` and print the contents received in the buffer until the End-of-File flag gets set. ```C {file="server.c"} int main(int argc, char *argv[]) { ... int eof; size_t len; char buffer[BUFFER_SIZE + 1 /* 1 extra to add terminating null-byte */]; do { int ret = recv_message(sock, buffer, &len, &eof); if (ret == -1) { close(sock); return EXIT_FAILURE; } buffer[len] = '\0'; printf("%s", buffer); } while (!eof); close(sock); return EXIT_SUCCESS; } ``` Again, we can build the program with a simple command thanks to our `Makefile`: ```command make ``` ```output cc -g -Wall -Wextra -Wconversion server.c common.h -lrsync -o server ---snip--- cc -g -Wall -Wextra -Wconversion client.c common.h -lrsync -o client ---snip--- ``` Now, we can run the server and the client in two terminal sessions and see that messages are sent and received. Furthermore, we can see that the server knows when to exit, thanks to the End-of-File flag. ```command ./server ``` ```output Waiting for connection... Hello from client ``` ```command ./client ``` ```output Connecting to server... ``` With a fully functional and battle-tested network protocol, we are ready to start implementing efficient file streams. N.B. Remember to remove the temporary calls to `send_message()` and `recv_message()` before proceeding with the tutorial. ## Computing & sending the signature The first step in implementing efficient file streams using the rsync protocol involves computing a signature of the basis file and sending this to the server. The basis file is basically the client's copy of the file. Furthermore, the signature lets the server determine which parts of the client's copy are up-to-date. In the `main()` function of `client.c`, we'll add some code to extract the path of the basis file from the command-line arguments. We will assume that the user wants to use the IO streams if no arguments are passed. I.e., we will read from `stdin` and write to `stdout`. ```C {file="client.c"} static int send_signature(int sock, const char *fname); int main(int argc, char *argv[]) { /* Parse arguments (use stdin and stdout if no argument) */ const char *fname = (argc >= 2) ? argv[1] : NULL; puts("Connecting to server..."); int sock = connect_to_server(IP_ADDRESS); if (sock == -1) { return EXIT_FAILURE; } puts("Sending signature..."); int ret = send_signature(sock, fname); if (ret == -1) { close(sock); return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } ``` Furthermore, we'll declare a function, `send_signature(),` in `client.c` and call it from `main()`. Now, let's define it: ```C {file="client.c"} static int send_signature(int sock, const char *fname) { /* Make sure the basis file exists, unless it is stdin */ const int use_io_stream = (fname == NULL) || (strcmp(fname, "-") == 0); if (!use_io_stream) { int fd = open(fname, O_RDONLY | O_CREAT); if (fd != -1) { close(fd); } } /* Open basis file */ FILE *file = rs_file_open(fname, "rb", 0); assert(file != NULL); rs_file_close(file); return 0; } ``` To compute the signature of the basis file, we'll first need to open it. Unfortunately, to my knowledge, there is no good way of opening a file in read-only mode and creating it if it does not exist. I.e., when using the `fopen(3)` API. Hence, we will have to do this in two steps. First, we open it with the `O_CREAT` flag (unless it's `stdin`) to ensure the file exists while avoiding truncation. Next, we open it in read-only mode. The library `librsync` provides handy utility functions for opening and closing files. The function `rs_file_open()` provides a platform-independent way to open large binary files. Furthermore, it uses `stdin` for reading and `stdout` for writing if `NULL` or `"-"` is passed as the filename argument. The function `rs_file_close()` provides special handling to avoid actually closing `stdin` and `stdout`. The Streaming API in `librsync` is designed around starting jobs, filling the input buffer, iterating the job, and draining the output buffer. Hence, we'll need to allocate two buffers for our jobs. Add the following line just before the function declarations in `client.c`: ```C {file="client.c"} static char in_buf[BUFFER_SIZE * 2], out_buf[BUFFER_SIZE]; ``` Given that our network protocol can send chunks of `BUFFER_SIZE` bytes. And that there may still be leftovers in the buffers after iterating a job. We need a `BUFFER_SIZE * 2` length input buffer to contain a new message in addition to leftovers. The file stream can be optimized through parameters given to the signature job. Thankfully, we don't have to worry too much about them. The reason is that `librsync` provides a helper function `rs_sig_args()` that can be used to get the recommended arguments based on the input file's size. The documentation explicitly says to use `-1` in the input argument of `rs_sig_args()` if the file size is unknown, as it would be if we read from `stdin`. Once again, `librsync` provides a utility function `rs_file_size()` to get the file size in a platform-independent way and return `-1` if the size cannot be determined due to not being a regular file. ```C {file="client.c"} static int send_signature(int sock, const char *fname) { ... /* Get file size */ rs_long_t fsize = rs_file_size(file); /* Get recommended arguments */ rs_magic_number sig_magic = 0; size_t block_len = 0, strong_len = 0; rs_result res = rs_sig_args(fsize, &sig_magic, &block_len, &strong_len); if (res != RS_DONE) { rs_file_close(file); return -1; } rs_file_close(file); return 0; } ``` With what is hopefully close to optimal parameters, we can start the signature job with a call to `rs_sig_begin()`. Furthermore, we'll set up a `rs_buffers_t` struct, pointing the `next_out` attribute to the output buffer. The `avail_out` attribute specifies the remaining free space in the output buffer before iterating the job. We'll set that to `BUFFER_SIZE,` given the limitations of our network protocol. Bear in mind that this attribute will be updated to hold the size of the unused buffer after iterating the job. ```C {file="client.c"} static int send_signature(int sock, const char *fname) { ... /* Start generating signature */ rs_job_t *job = rs_sig_begin(block_len, strong_len, sig_magic); assert(job != NULL); /* Setup buffers */ rs_buffers_t bufs = { 0 }; bufs.next_out = out_buf; bufs.avail_out = BUFFER_SIZE; /* We cannot send more in one message */ rs_job_free(job); rs_file_close(file); return 0; } ``` Now comes the hard part. We will iterate the job in a do-while loop until `rs_job_iter()` returns `RS_DONE`. Unless End-of-File has been reached or the input buffer is already full, we try to fill it. There is a potential for leftover input data after the last job iteration. If this is the case, we must move it to the front of the buffer. Finally, we fill the input buffer by appending to the input buffer with a call to `fread().` Furthermore, we check if the End-of-File is reached and update the `eof_in` attribute in the `rs_buffers_t` struct to inform the rsync algorithm. Also, we set the `next_in` attribute to point to the start of the input buffer, in addition to updating `avail_in` with the number of bytes read. ```C {file="client.c"} static int send_signature(int sock, const char *fname) { ... /* Generate signature */ do { if ((bufs.eof_in == 0) && (bufs.avail_in < sizeof(in_buf))) { if (bufs.avail_in > 0) { /* Leftover tail data, move it to front */ memmove(in_buf, bufs.next_in, bufs.avail_in); } /* Fill input buffer */ size_t n_bytes = fread(in_buf + bufs.avail_in, 1, sizeof(in_buf) - bufs.avail_in, file); if (n_bytes == 0) { if (ferror(file)) { perror("Failed to read file"); rs_file_close(file); rs_job_free(job); return -1; } /* End-of-File reached */ bufs.eof_in = feof(file); assert(bufs.eof_in != 0); } bufs.next_in = in_buf; bufs.avail_in += n_bytes; } } while (res != RS_DONE); rs_job_free(job); rs_file_close(file); return 0; } ``` With the input buffer filled, we can execute `rs_job_iter()` to run the rsync algorithm. The algorithm may drain the input buffer and fill the output buffer. However, this is not always the case. Sending a message without a payload does not make sense unless it is to signal the End-of-File. Hence, we check for available data before draining the output buffer and updating the `next_out` and `avail_out` attributes. ```C {file="client.c"} static int send_signature(int sock, const char *fname) { ... /* Generate signature */ do { ... /* Iterate job */ res = rs_job_iter(job, &bufs); if (res != RS_DONE && res != RS_BLOCKED) { rs_file_close(file); rs_job_free(job); return -1; } assert(bufs.next_out >= out_buf); size_t present = (size_t)(bufs.next_out - out_buf); if (present > 0 || res == RS_DONE) { /* Drain output buffer */ assert(present <= BUFFER_SIZE); int ret = send_message(sock, out_buf, present, (res == RS_DONE) ? 1 : 0); if (ret == -1) { rs_file_close(file); rs_job_free(job); return -1; } bufs.next_out = out_buf; bufs.avail_out = BUFFER_SIZE; } } while (res != RS_DONE); rs_job_free(job); rs_file_close(file); return 0; } ``` That's it for sending signatures. Next, we'll look at receiving signatures on the server side. ## Receiving the signature Currently, the client computes and sends the signature to the server, but the server does not even bother to receive it. Let's change that by declaring a new function `recv_signature()` in `server.c` and calling it from `main().` The `recv_signature()` function will take an additional output parameter to hold the received signature. Don't forget to allocate the input and output buffers. ```C {file="server.c"} static char in_buf[BUFFER_SIZE * 2], out_buf[BUFFER_SIZE]; static int accept_connection(void); static int recv_signature(int sock, rs_signature_t **sig); int main(int argc, char *argv[]) { ... puts("Receiving signature..."); rs_signature_t *sig; int ret = recv_signature(sock, &sig); if (ret == -1) { close(sock); return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } ``` There isn't much that needs to be explained from the definition of `receive_signature().` One notable difference is that the job started by the call to `rs_loadsig_begin()` completely handles the draining of the output buffer for us. Ergo, one less worry on our minds. A more subtle difference is that the input buffer is filled with messages from a client instead of file contents. Thus, we ensure the buffer has at least `BUFFER_SIZE` remaining space. ```C {file="server.c"} static int recv_signature(int sock, rs_signature_t **sig) { rs_job_t *job = rs_loadsig_begin(sig); assert(job != NULL); /* Setup buffers */ rs_buffers_t bufs = { 0 }; rs_result res; do { if ((bufs.eof_in == 0) && (bufs.avail_in <= BUFFER_SIZE)) { if (bufs.avail_in > 0) { /* Leftover tail data, move it to front */ memmove(in_buf, bufs.next_in, bufs.avail_in); } size_t n_bytes; int ret = recv_message(sock, in_buf + bufs.avail_in, &n_bytes, &bufs.eof_in); if (ret == -1) { rs_job_free(job); return -1; } bufs.next_in = in_buf; bufs.avail_in += n_bytes; } /* Iterate job */ res = rs_job_iter(job, &bufs); if (res != RS_DONE && res != RS_BLOCKED) { rs_job_free(job); return -1; } /* The job should take care of draining the buffers */ } while (res != RS_DONE); rs_job_free(job); return 0; } ``` If you want to test the code thus far. You can compile it with `make` and run the server as follows: ```command ./server ``` ```output Waiting for connection... Receiving signature... ``` For the client, you can either use `stdin` as illustrated below or provide a file path as a command-line argument: ```command echo "foo bar baz" | ./client ``` ```output Connecting to server... Sending signature... ``` Now, let's move on to generating deltas. ## Generating & sending the delta We can now compute a delta for the client to patch the basis file from its signature and the up-to-date file (available on the server). You know the drill. We declare a new function, `send_delta()`, and call it from `main()`. Furthermore, we get the filename of the up-to-date file from the command-line arguments. This should probably be sent as an argument from the client through the network protocol. But, hey! We want to keep it simple. ```C {file="server.c"} static int accept_connection(void); static int recv_signature(int sock, rs_signature_t **sig); static int send_delta(int sock, rs_signature_t *sig, const char *fname); int main(int argc, char *argv[]) { /* Parse arguments (use stdin if no argument) */ const char *fname = (argc >= 2) ? argv[1] : NULL; ... puts("Sending delta..."); ret = send_delta(sock, sig, fname); rs_free_sumset(sig); if (ret == -1) { close(sock); return EXIT_FAILURE; } close(sock); puts("Success!"); return EXIT_SUCCESS; } ``` We need to initialize the hash table with a call to `rs_build_hash_table()` before using the signature to calculate the delta. Furthermore, we'll need to open the "up-to-date" file to fill the input buffer with its contents. The job is started with `rs_delta_begin()`. The remaining code is practically the same as what we've seen thus far. ```C {file="server.c"} static int send_delta(int sock, rs_signature_t *sig, const char *fname) { /* Build hash table */ rs_result res = rs_build_hash_table(sig); if (res != RS_DONE) { return -1; } /* Open file up-to-date file */ FILE *file = rs_file_open(fname, "rb", 0); assert(file != NULL); /* Start generating delta */ rs_job_t *job = rs_delta_begin(sig); assert(job != NULL); /* Setup buffers */ rs_buffers_t bufs = { 0 }; bufs.next_out = out_buf; bufs.avail_out = BUFFER_SIZE; /* We cannot send more in one message */ do { if ((bufs.eof_in == 0) && (bufs.avail_in < sizeof(in_buf))) { if (bufs.avail_in > 0) { /* Left over tail data, move to front */ memmove(in_buf, bufs.next_in, bufs.avail_in); } /* Fill input buffer */ size_t n_bytes = fread(in_buf + bufs.avail_in, 1, sizeof(in_buf) - bufs.avail_in, file); if (n_bytes == 0) { if (ferror(file)) { perror("Failed to read file"); rs_file_close(file); rs_job_free(job); return -1; } bufs.eof_in = feof(file); } bufs.next_in = in_buf; bufs.avail_in += n_bytes; } res = rs_job_iter(job, &bufs); if (res != RS_DONE && res != RS_BLOCKED) { rs_file_close(file); rs_job_free(job); return -1; } /* Drain output buffer, if there is data */ assert(bufs.next_out >= out_buf); size_t present = (size_t)(bufs.next_out - out_buf); if (present > 0 || res == RS_DONE) { assert(present <= BUFFER_SIZE); int ret = send_message(sock, out_buf, present, (res == RS_DONE) ? 1 : 0); if (ret == -1) { rs_file_close(file); rs_job_free(job); return -1; } bufs.next_out = out_buf; bufs.avail_out = BUFFER_SIZE; } } while (res != RS_DONE); rs_file_close(file); rs_job_free(job); return 0; } ``` ## Receiving the delta & patching the file Dare I say it? We declare a new function, `recv_delta_and_patch_file()`, and call it from `main()`. Shocking indeed! ```C {file="client.c"} static int connect_to_server(const char *ip_addr); static int send_signature(int sock, const char *fname); static int recv_delta_and_patch_file(int sock, const char *fname); int main(int argc, char *argv[]) { ... puts("Receiving delta and patching file..."); ret = recv_delta_and_patch_file(sock, fname); if (ret == -1) { close(sock); return EXIT_FAILURE; } close(sock); return EXIT_SUCCESS; } ``` The definition of `recv_delta_and_patch_file()` has some new nuts and bolts. It's worth noticing that `rs_patch_begin()` takes a callback function as an argument. This callback is supposed to retrieve content from the basis file at specific offsets during the job iterations. As we've seen multiple times throughout this tutorial, `librsync` provides a helper function, `rs_file_copy_cb`, to take care of this. ```C {file="client.c"} static int recv_delta_and_patch_file(int sock, const char *fname_old) { const int use_io_stream = (fname_old == NULL) || (strcmp(fname_old, "-") == 0); const char *fname_new = NULL; char path[PATH_MAX]; if (!use_io_stream) { int ret = snprintf(path, sizeof(path), "%s.new", fname_old); if (ret < 0 || (size_t)ret >= sizeof(path)) { fputs("Filename too long\n", stderr); return -1; } fname_new = path; } /* Open new file */ FILE *new = rs_file_open(fname_new, "wb", 1); assert(new != NULL); /* Open basis file */ FILE *old = old = rs_file_open(fname_old, "rb", 0); assert(old != NULL); rs_job_t *job = rs_patch_begin(rs_file_copy_cb, old); assert(job != NULL); /* Setup RSYNC buffers */ rs_buffers_t bufs = { 0 }; bufs.next_out = out_buf; bufs.avail_out = sizeof(out_buf); rs_result res; do { if ((bufs.eof_in == 0) && (bufs.avail_in < BUFFER_SIZE)) { if (bufs.avail_in > 0) { /* Left over tail data, move to front */ memmove(in_buf, bufs.next_in, bufs.avail_in); } size_t n_bytes; int ret = recv_message(sock, in_buf + bufs.avail_in, &n_bytes, &bufs.eof_in); if (ret == -1) { rs_file_close(new); rs_file_close(old); rs_job_free(job); return -1; } bufs.next_in = in_buf; bufs.avail_in += n_bytes; } res = rs_job_iter(job, &bufs); if (res != RS_DONE && res != RS_BLOCKED) { rs_file_close(new); rs_file_close(old); rs_job_free(job); return -1; } /* Drain output buffer, if there is data */ assert(bufs.next_out >= out_buf); size_t present = (size_t)(bufs.next_out - out_buf); if (present > 0) { size_t n_bytes = fwrite(out_buf, 1, present, new); if (n_bytes == 0) { perror("Failed to write to file"); rs_file_close(new); rs_file_close(old); rs_job_free(job); return -1; } bufs.next_out = out_buf; bufs.avail_out = sizeof(out_buf); } } while (res != RS_DONE); rs_file_close(new); rs_file_close(old); rs_job_free(job); if (!use_io_stream) { int ret = rename(fname_new, fname_old); if (ret == -1) { perror("Failed to swap basis file with patched file"); return -1; } } return 0; } ``` While running the job, we must be careful not to modify the basis file. Instead, we'll write the patched content to a new file, `.new`, and replace it with the basis file once the job is complete. We'll make an exception when using file streams. In this case, we are already reading and writing from different files (`stdin` and `stdout`). Furthermore, attempting to replace `stdout` does not sound like a good idea. Now, let's compile our programs with `make`: ```command make ``` ```output cc -g -Wall -Wextra -Wconversion server.c common.h -lrsync -o server cc -g -Wall -Wextra -Wconversion client.c common.h -lrsync -o client ``` By now, all the warnings should have cleared. Let's run the server by piping some text into `stdin`: ```command echo "foo bar baz" | ./server ``` ```output Waiting for connection... Receiving signature... Sending delta... ``` Finally, let's run the client by piping some slightly different text into `stdin`: ```command echo "foo baz bar" | ./client ``` ```output Connecting to server... Sending signature... Receiving delta and patching file... foo bar baz ``` Voila, the patched content is written to `stdout`. ## Trial by fire: Let's see if it holds up We have proven that we can send a stream of bytes from one side to the other. However, is it efficient? To find out, let's first add some code to print the number of bytes sent and received by the client. Declare a variable to hold the total number of bytes sent just before the do-while loop in `send_signature()` in `client.c`: ```C {file="client.c"} size_t tot_bytes_sent = 0; ``` Next, add the number of bytes sent to the total just after the call to `send_message()` toward the end of the do-while loop: ```C {file="client.c"} tot_bytes_sent += present; ``` Finally, print the result just before returning from `send_signature()`: ```C {file="client.c"} printf("Sent %zu bytes\n", tot_bytes_sent); ``` Similarly, in `recv_delta_and_patch_file(),` declare a variable to hold the total number of bytes received just before the do-while loop: ```C {file="client.c"} size_t tot_bytes_received = 0; ``` Next, add the number of bytes received to the total just after the call to `recv_message()` toward the start of the do-while loop: ```C {file="client.c"} tot_bytes_received += n_bytes; ``` And finally, print the result just before returning from `recv_delta_and_patch_file()`: ```C {file="client.c"} printf("Received %zu bytes\n", tot_bytes_received); ``` Now let's create the basis file by dumping 1 GB worth of random data into it: ```command head -c 1G /dev/urandom > bar.txt ``` Next, we create the up-to-date file by concatenating 500 MB to the basis file: ```command head -c 500M /dev/urandom | cat bar.txt - > foo.txt ``` Finally, compile the program with `make` and run the server with `foo.txt` as the argument: ```command ./server foo.txt ``` ```output Waiting for connection... Receiving signature... Sending delta... ``` In another shell, run the client with `bar.txt` as an argument: ```command ./client bar.txt ``` ```output Connecting to server... Sending signature... Sent 1179660 bytes Receiving delta and patching file... Received 524312014 bytes ``` From the client's output, we can see that the client sent approximately 1.2 MB and received around 524.3 MB, which adds up to 525.5 MB. This is considerably less than having to transmit the entire 1,5 GB file. Now, let's make sure the files are actually identical: ```command diff -s foo.txt bar.txt ``` ```output Files foo.txt and bar.txt are identical ``` Which they indeed are! ## Conclusion Congratulations! You have successfully implemented efficient file streaming. One can argue that the "trial by fire" is synthetic and does not adhere to the real world. However, in a future blog post, we plan to show the performance benefits of updating the [CFEngine MPF (Masterfiles Policy Framework)](https://docs.cfengine.com/docs/3.24/reference-masterfiles-policy-framework.html) from one version to another using CFEngine. CFEngine leverages the `librsync` Stream API in the latest network protocol, and we saw a performance benefit of over 80%. If you cannot wait for the next blog post, this experiment is also featured in [The agent is in - Episode 45](https://www.youtube.com/watch?v=1wvFuBrHseI&t). ## Share your thoughts You are now an expert user of the `librsync` Streaming API. If you choose to use this code as a launch pad for an upcoming project or have any questions about the code, let us know at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear from you. -------------------------------------------------------------------------------- title: "Masterfiles Policy Framework: the 'module' that started it all" url: /blog/2025/masterfiles-module/index.md date: "2025-03-03" -------------------------------------------------------------------------------- The MPF or Masterfiles Policy Framework is intended to provide a stable base policy for installations and upgrades, and is used by both CFEngine Enterprise and CFEngine community. When you create a new cfbs project with `cfbs init` one of the questions is related to the MPF: ```sh Do you wish to build on top of the default policy set, masterfiles? (Recommended) [YES/y/no/n]: Added module: masterfiles The default commit message is 'Added module 'masterfiles'' - edit it? [yes/y/NO/n] Committing using git: [main f84d0d4] Added module 'masterfiles' 1 file changed, 16 insertions(+), 1 deletion(-) ``` Of particular interest to policy writers is the `lib` sub-directory: > This directory contains the standard library aka COPBL or the Community Open Promise Body Library. The bodies and bundles found here are contributed and maintained by the CFEngine community. They codify many common and useful patterns. Let's look through some of the helpful bits you can re-use in your policy! ## cronjob Maybe you want to easily add a job to a user's cronjob? Use the [cronjob agent bundle](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-bundles.html#cronjob). ```cf3 methods: "cron" usebundle => cronjob("/bin/ls","mark","*","5,10"); ``` ## in_shell By default [commands promises](https://docs.cfengine.com/docs/3.24/reference-promise-types-commands.html) do not execute a shell due to resource and security concerns. Often though, we need to setup a pipeline of commands to achieve our goals. In this case we have to instruct CFEngine to use a shell with the contain body `useshell` attribute. Here we use a helper [`in_shell`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-commands.html#in_shell) that saves us from writing the contain body ourselves. ```cf3 commands: "/bin/pwd | /bin/cat" contain => in_shell; ``` ## warn_only A wise strategy when writing policy is to not make any changes but just see how things are currently. The `action` attribute provides a way to achieve this. The MPF has many helper action bodies. Here we use [`warn_only`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-common.html#warn_only) which warns once an hour if the promise needs to be repaired. ```cf3 commands: "something --destructive" action => warn_only; ``` ## classes_generic It is often helpful to use the result of one promise to decide to activate another promise. There are many classes bodies in the MPF. The [`classes_generic`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-common.html#classes_generic) body defines many classes depending on the outcome of the promise. ```cf3 commands: "/bin/true" classes => classes_generic("true"); ``` Would define `promise_repaired_true` and `promise_kept_true`. ## and so much more! There are many other bundles and bodies for all the various [promise types](https://docs.cfengine.com/docs/master/reference-promise-types.html). Take a look through the [documentation](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework.html) and find helpers for [commands](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-commands.html), [files](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-files.html), [paths](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-paths.html) and more. ## Contact For help with upgrading or additional questions, please contact support at: https://support.northern.tech -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 46 - Classification using genders" url: /blog/2025/the-agent-is-in-episode-46/index.md date: "2025-02-27" -------------------------------------------------------------------------------- Ever tried to wrangle a fleet of servers with just a text file? Nick shows how CFEngine can take advantage of genders for classification. In this episode, Nick dives into the configuration file, `/etc/genders`. Originally developed by Lawrence Livermore National Laboratory and currently maintained by the Chaos development team, [genders](https://github.com/chaos/genders) often seen in use in High-Performance Computing (HPC) environments. Nick presents two practical examples demonstrating policy implementations, using genders for inventory reporting and grouping hosts. ## Video {#video} The video recording is available on YouTube: {{< youtube rHyQHEjsxCI >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Examples {#examples} ### Example illustrating parsing /etc/genders from CFEngine policy {#example-illustrating-parsing-etc-genders-from-cfengine-policy} This policy defines `g_` prefixed `namespace` scoped classes for all attributes (with and without values) and defines variables for attributes with values both tagged for reporting to an Enterprise Hub but only the variables tagged for inventory. ```cfengine3 bundle agent __main__ { methods: "genders_classification_by_util"; } bundle agent genders_classification_by_util # I am assuming that this classification generally only relavent for cf-agent. # E.g. we won't use one of these classes to guard an access promise sharing some # directory. # This policy doesn't handle the installation of the packages necessary to provide nodeattr { vars: # Define a list of all the node attributes for the executing host Including those with values "my_attrs_list" slist => string_split( execresult( "nodeattr -l $(sys.uqhost)", useshell, both), "\n", inf ), if => not( isvariable( $(this.promiser) ) ); # Then we get the list of attributes that don't have a value "my_attrs_noval" slist => filter( ".*=.*", my_attrs_list, "true", "true", inf ); # Then we get the list of attributes that do have a value "my_attrs_wval" slist => filter( ".*=.*", my_attrs_list, "true", "false", inf ); "map_attrs_w_val[$(with)]" string => nth( string_split( "$(my_attrs_wval)", "=", 2 ), 1 ), meta => { "derived-from=/etc/genders", "report", "inventory", "attribute_name=/etc/genders[$(with)]" }, with => nth( string_split( "$(my_attrs_wval)", "=", 2 ), 0 ); "_map_attrs_w_val_list" slist => getindices( "map_attrs_w_val" ); classes: "g_$(my_attrs_list)" meta => { "derived-from=/etc/genders", "report" }, scope => "namespace"; reports: "Classes defined derived from /etc/genders: $(with)" with => join( ", ", classesmatching( ".*", "derived-from=/etc/genders" )); "$(_map_attrs_w_val_list) = $(map_attrs_w_val[$(_map_attrs_w_val_list)])"; } ``` ```text R: Classes defined derived from /etc/genders: g_linux, g_nickanderson, g_something_value R: something = value ``` ### Example classifying from /etc/genders using utilities (nodeattr) {#example-classifying-from-etc-genders-using-utilities--nodeattr} This policy defines `g_` prefixed `namespace` scoped classes for all attributes (with and without values) and defines variables for attributes with values both tagged for reporting to an Enterprise Hub but only the variables tagged for inventory. ```cfengine3 bundle agent __main__ { methods: "genders_classification_by_util"; } bundle agent genders_classification_by_util # I am assuming that this classification generally only relavent for cf-agent. # E.g. we won't use one of these classes to guard an access promise sharing some # directory. # This policy doesn't handle the installation of the packages necessary to provide nodeattr { vars: # Define a list of all the node attributes for the executing host Including those with values "my_attrs_list" slist => string_split( execresult( "nodeattr -l $(sys.uqhost)", useshell, both), "\n", inf ), if => not( isvariable( $(this.promiser) ) ); # Then we get the list of attributes that don't have a value "my_attrs_noval" slist => filter( ".*=.*", my_attrs_list, "true", "true", inf ); # Then we get the list of attributes that do have a value "my_attrs_wval" slist => filter( ".*=.*", my_attrs_list, "true", "false", inf ); "map_attrs_w_val[$(with)]" string => nth( string_split( "$(my_attrs_wval)", "=", 2 ), 1 ), meta => { "derived-from=/etc/genders", "report", "inventory", "attribute_name=/etc/genders[$(with)]" }, with => nth( string_split( "$(my_attrs_wval)", "=", 2 ), 0 ); "_map_attrs_w_val_list" slist => getindices( "map_attrs_w_val" ); classes: "g_$(my_attrs_list)" meta => { "derived-from=/etc/genders", "report" }, scope => "namespace"; reports: "Classes defined derived from /etc/genders: $(with)" with => join( ", ", classesmatching( ".*", "derived-from=/etc/genders" )); "$(_map_attrs_w_val_list) = $(map_attrs_w_val[$(_map_attrs_w_val_list)])"; } ``` ```text R: Classes defined derived from /etc/genders: g_linux, g_nickanderson, g_something_value R: something = value ``` ## Links {#links} * Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) * [Genders](https://man.cx/genders(3)) * [nodeattr](https://man.cx/nodeattr) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Enable automatic running of policies with autorun module" url: /blog/2025/autorun-module/index.md date: "2025-02-03" -------------------------------------------------------------------------------- When writing CFEngine policy we create files ending in the `.cf` extension but this alone won't cause the policy to be parsed and evaluated. By default [cf-agent](https://docs.cfengine.com/docs/3.24/reference-components-cf-agent.html) runs [`${sys.inputdir}/promises.cf`](https://docs.cfengine.com/docs/3.24/reference-masterfiles-policy-framework-promises.html). For a non-privileged user running `cf-agent` this will be in their `$HOME` directory: ```command cf-promises --show-vars=sys.inputdir ``` ```output Variable name Variable value Meta tags Comment default:sys.inputdir /home/craig/.cfagent/inputs source=agent ``` Usually though, CFEngine is run as a privileged user so the more common value is: ```command cf-promises --show-vars=sys.inputdir ``` ```output Variable name Variable value Meta tags Comment default:sys.inputdir /var/cfengine/inputs source=agent ``` ## MPF (Masterfiles Policy Framework) Typically users will use the [Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/reference-masterfiles-policy-framework.html) which is our body of standard policy that provides a nice standard library of helpers as well as many tunables to our standard policy set. Using the MPF you must include additional policy files in `inputs`. This is possible in many ways such as in policy with the [body file control attribute](https://docs.cfengine.com/docs/3.24/reference-components-file_control_promises.html#inputs) or with [augments](https://docs.cfengine.com/docs/3.24/reference-language-concepts-augments.html#inputs) as well as with the MPF [autorun feature](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html). In addition to enabling this feature with augments you can enable it with the [autorun module](https://build.cfengine.com/modules/autorun/). ## The build module way To get started from scratch you would download `cfbs` and create a new project: ```command pipx install cfbs && mkdir my-cfbs-project && cd my-cfbs-project && cfbs init ``` Then add the autorun module to your build project: ```command cfbs add autorun ``` Then add policy files with `cfbs add ./file.cf` and in this case `cfbs` will add the file to list of inputs so we can skip the placing of files in `services/autorun`. The files will be generated by default to `services/cfbs/(filename)` when `cfbs build` is run. ```cf3 {file="my-cfbs-project/my-autorun.cf"} bundle agent my_autorun_bundle { meta: "tags" slist => { "autorun" }; reports: "hello from my autorun bundle!"; } ``` And add it to your project. When cfbs asks which bundle should be evaluated answer None because the autorun module will take care of running it as it has that meta tag `autorun` and cfbs is handling the parsing of the file by adding it to inputs in def.json. ```command cfbs add ./my-autorun.cf ``` ```output Which bundle should be evaluated (added to bundle sequence)? 1. ./my-autorun.cf:my_autorun_bundle (default) 2. (None) [1/2]: 2 Added module: ./my-autorun.cf The default commit message is 'Added module './my-autorun.cf'' - edit it? [yes/y/NO/n] Committing using git: [main 9dadfc3] Added module './my-autorun.cf' 2 files changed, 17 insertions(+) create mode 100644 my-autorun.cf ``` Let's see the build result: ```command cfbs build && cat out/masterfiles/def.json ``` ```output [...] { "classes": { "services_autorun": ["any"] }, "inputs": ["services/cfbs/my-autorun.cf"] } ``` Notice that our file is added to `inputs` and the `services_autorun` class will be defined for all hosts because the special class `any` is a [hard class](https://docs.cfengine.com/docs/3.24/reference-language-concepts-classes.html#hard-classes) which is always set. And the output when we run policy? ```sh R: hello from my autorun bundle! ``` ## Autorun feature options There are two classes which give you more control over this feature: [`services_autorun_inputs`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html#automatically-add-policy-files-to-inputs) which allows you to control when files in `services/autorun` are parsed and [`services_autorun_bundles`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html#automatically-run-bundles-tagged-autorun) which allows you to specify when `autorun` tagged bundles are evaluated. There is also variable that can be defined in augments to extend which directories are searched for `.cf` files: [`def.mpf_extra_autorun_inputs`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html#additional-automatically-loaded-inputs). For a more advanced look at the autorun feature, how to extend it and how to implement your own see the [Extending autorun](https://cfengine.com/blog/2022/extending-autorun/) blog post ## Contact For help with upgrading or additional questions, please contact support at: https://support.northern.tech -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 45 - Deduping the Internet with CFEngine" url: /blog/2025/the-agent-is-in-episode-45/index.md date: "2025-01-30" -------------------------------------------------------------------------------- Did you notice that CFEngine 3.25.0 shipped with librsync as a new dependency? Lars joins Cody, Craig, Herman and Nick to show and tell the massive savings we can expect to see from librsync for remote file copies. Spoiler alert, Lars sees > 80% savings from a simple policy update between 3.24.0 and 3.24.1! From the audience, Jay mentioned that they saw similar rates of savings from their use of rsync with savings accumulating to over 16TB a day. Lars mentioned that he has plans for a couple of follow-up blog posts, so keep an eye on for those to appear. We look forward to hearing feedback from users the sorts of improvements they see so be sure to [let us know](https://github.com/cfengine/core/discussions)! ## Video {#video} The video recording is available on YouTube: {{< youtube 1wvFuBrHseI >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} * Connect w/ [Lars](https://www.linkedin.com/in/larsewi/), [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) * [librsync](https://librsync.github.io/) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CVE-2024-55958 - Cross-site scripting (XSS) / missing input validation in CFEngine Enterprise Mission Portal" url: /blog/2025/cve-2024-55958/index.md date: "2025-01-20" -------------------------------------------------------------------------------- We are writing to inform you of a recently discovered security issue in the CFEngine Enterprise web UI, Mission Portal. The issue has been fixed in the recently released 3.21.6 and 3.24.1 versions. Prior versions (3.24.0, 3.21.5, and below) are affected. We have no indications of this issue being exploited or known outside of the company. The issue was discovered thanks to the vulnerability scanning software _[Acunetix by Invicti](https://www.acunetix.com/)_. ## Description On the affected versions, some fields lack input validation, allowing an authenticated user with administrator-level privileges to enter javascript into input text fields, which will be evaluated by other users of the system who open up the same form. In addition to fixing this specific issue of confirmed XSS, we also added much more strict input validation to many other fields in Mission Portal, to prevent similar issues, even though we were not able to find something exploitable in those cases. ## Impact The impact of this issue is quite limited, the issue exists inside of settings, in a form that a normal (low privilege) user can neither edit nor open. So in the recommended setup where most users use lower-privileged roles, this issue can only be used to perform cross-site scripting between 2 administrator accounts, and it requires the victim to perform a quite specific action. ## Remediation To remediate the issue, please upgrade to CFEngine Enterprise 3.21.6, 3.24.1, or later versions. We recommend upgrading as soon as possible not just to fix this specific issue, but also to benefit from the other bugfixes and security improvements we put into every new release. Alternately, you can mitigate the risk of this vulnerability by following a [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) approach, ensuring that only a few users have administrator access to settings, user management, etc. while other users only have access to what they need (for example reports, CMDB, dashboards, etc.). ## Contact For help with upgrading or additional questions, please contact support at: https://support.northern.tech -------------------------------------------------------------------------------- title: "CFEngine 3.25 released - Auditability" url: /blog/2025/cfengine-3-25-released-auditability/index.md date: "2025-01-06" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.25.0\! The code word for this release is _auditability_. Being a non-LTS (not supported) release, this release allows users to test the new functionality we've been working on before it arrives in an LTS release ~1 year from now. ## What's new ### The audit log CFEngine Mission Portal now logs user actions in a structured audit log. This means you can go back and see who edited group data, who deleted a host, who created a user, etc. The audit log can be filtered by time and date, resource type, who performed the action, and what was affected. {{< wideimg "images/audit-log.png" "Screenshot of the audit log, showing actions of users creating, updating, and deleting groups.">}} To make it easy for users to download, process, back up, or ingest logs into other systems, there is a button to download in CSV format, and a [REST API for retrieving entries programmatically](https://docs.cfengine.com/docs/3.25/api-enterprise-api-ref-audit-logs-api.html). You can also click **View details** for individual entries to get more detailed information about exactly what was changed. {{< img "images/audit-log-view-details.png" "Screenshot of the details of one audit log event, showing the example variable which was changed.">}} In order to reduce noise and ensure the audit log has the most valuable information possible, it focuses on _changes made by users_ through the Mission Portal API and UI. ### 3.25 sneak peek webinar episode In our latest episode of "The agent is in" webinar series, we took a sneak peek at 3.25.0 and the audit log: {{< youtube 2YJiZSfKW_8 >}} ### 2-factor authentication and password security improvements As mentioned in the recent [3.24.1 release blog](/blog/2024/cfengine-3-21-6-and-3-24-1-released/), we're making several improvements to the authentication security of Mission Portal, including: - 2-factor authentication via time-based one time password (TOTP) app - Password complexity measurement and stricter requirements / defaults - More customizability and settings to enforce the use of 2FA, strong passwords, etc. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.25.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.25/release-notes-whatsnew-changelog-core.html) - [3.25.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.25/release-notes-whatsnew-changelog-enterprise.html) - [3.25.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.25/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to [3.24.0](/blog/2024/cfengine-3-24-lts-released-consistency/), these dependencies have been updated: | CFEngine version | 3.24.0 | **3.25.0** | | :-------------------------------------------------------------------------------- | :----- | :--------- | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.45.2 | **2.47.1** | | [libcurl](https://curl.haxx.se/download.html) | 8.8.0 | **8.11.1** | | [libexpat](https://libexpat.github.io/) | 2.5.0 | **2.6.3** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.13.1 | **2.13.5** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.8 | **2.6.9** | | [OpenSSL](https://openssl.org/) | 3.3.1 | **3.4.0** | | [PHP](https://php.net/) | 8.3.8 | **8.3.15** | | [PostgreSQL](https://www.postgresql.org/) | 16.3 | **17.2** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download) to go to the download pages with new packages. If you are using `cf-remote`, you can specify version to install with the `--version` option: ```command cf-remote --version 3.25.0 install --hub hub --bootstrap hub ``` ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net) -------------------------------------------------------------------------------- title: "See what services are running on your hosts" url: /blog/2024/inventory-running-services/index.md date: "2025-01-06" -------------------------------------------------------------------------------- This series of blogs, [Monthly Module Mondays](/tags/monthly-module-monday/) , started on April Fool's Day 2024 discussing how to [Inventory and remediate Red Hat Enterprise Linux with Security Technical Implementation Guides (STIGs)](/blog/2024/stig-rhel-7/) has now reached the 10th installment showcasing a couple of modules to take stock of what services are running on your systems. ## Command line tools are useful! Thanks to both [systemd](https://systemd.io/) and [Windows Services](https://learn.microsoft.com/en-us/windows/win32/services/services) command line tools [systemctl](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html#) and Powershell's [Get-Service](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-service?view=powershell-7.4) supporting output of json format lists of running services it is easy to bring together the power of CFEngine to [run commands](https://docs.cfengine.com/docs/3.24/reference-promise-types-commands.html), [read json files](https://docs.cfengine.com/docs/3.24/reference-functions-readjson.html#top) and [collect inventory](https://docs.cfengine.com/docs/3.24/examples-tutorials-custom_inventory.html#top) which can be used for many purposes such as: [reporting](https://docs.cfengine.com/docs/3.24/web-ui-enterprise-reporting.html), [alert widgets](https://docs.cfengine.com/docs/3.24/web-ui.html#alert-widgets), [compliance reports](https://docs.cfengine.com/docs/3.24/getting-started-reporting-and-web-ui.html#compliance-reports) as well as our new [Groups](https://cfengine.com/blog/2024/cfengine-3-24-lts-released-consistency/) feature. ## Add some modules To add this inventory simply add two modules: [inventory-systemd](https://build.cfengine.com/modules/inventory-systemd/) and [inventory-windows-services](https://build.cfengine.com/modules/inventory-windows-services/) to your Build project either via the command line [`cfbs`](https://build.cfengine.com/modules/inventory-systemd/) tool or in Mission Portal with the Build app. {{< wideimg "images/adding-inventory-systemd-module-in-build-app.png" "adding inventory-systemd module in Mission Portal Build app">}} ## Inventory report with services running information After the project is pushed and deployed to hosts and the hosts have a chance to collect the information and report back we can run the Inventory report and add two columns: Systemd service units running and Windows services running. {{< img "images/add-services-running-columns-to-inventory-report.png" "Adding services running columns to inventory report" >}} We now see a list of services running on each system. {{< wideimg "images/inventory-report-with-services-running-columns.png" "Inventory report with services running columns added and data showing" >}} From here we can use this information. ## Services running compliance report For example, to create a compliance report that states that CFEngine service must be running on every host regardless of Linux or Windows. Creating a new compliance report and adding two checks, we use the inventory we have gathered and check Systemd service units running that it matches `cf-execd.service` for hosts that match the `linux` class. and Windows services running that it matches `Cfengine Nova Executor` for hosts that match the `windows` class. {{< wideimg "images/create-cfengine-services-running-on-windows-compliance-check.png" "Create CFEngine services running on windows compliance check" >}} And voila! I have 100% compliance! {{< wideimg "images/cfengine-services-running-compliance-100-percent.png" "CFEngine services running is 100% compliant" >}} If I visit one host, `rocky-9` and run `systemctl stop cf-execd; systemctl disable cf-execd` and use the Host Info page to collect fresh reports my compliance changes to 75%. {{< wideimg "images/cfengine-services-running-compliance-at-75-percent.png" "Compliance at 75% after stopping cf-execd on one host" >}} ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "CFEngine 2024 retrospective" url: /blog/2024/2024-retrospective/index.md date: "2024-12-31" -------------------------------------------------------------------------------- With another year behind us, we take a look back at what happened with CFEngine in 2024. ## Webinars: 'The agent is in' We are 3 years in to our webinar series, 'The agent is in', and the show had, as expected, 12 new episodes. You can find all the recordings, and the registration link, at [cfengine.com/webinars](/webinars). Additional show notes are also available [on our blog](/tags/webinars). ## Module Mondays and Feature Fridays Nick and Craig have been writing regularly throughout the year about their favorite features and modules. You can find these posts on our blog: [Module Mondays](/tags/monthly-module-monday) [Feature Fridays](/tags/feature-friday) ## New feature: Audit log The new audit log feature helps security and compliance teams review who made changes, what the changes were, and when they were performed: {{< wideimg "/blog/2025/images/audit-log.png" "Screenshot of the audit log, showing actions of users creating, updating, and deleting groups.">}} The audit log will arrive in a supported release by the end of 2025 with CFEngine 3.27 LTS. ## Notable changes in behavior Occasionally, we have to make changes to CFEngine which might impact existing users relying on the current behavior. Most commonly, we do this, when the current behavior contradicts what is documented, what users expect, and/or if it has negative security implications to keep it this way. When we do this, and we see that users might be relying on the old behavior (i.e. it's not a simple bugfix which improves the situation for everyone without any manual changes). The line between bug fix and change in behavior is sometimes blurry, but we try our best to communicate these kinds of changes very clearly, through [_change-in-behavior_ blog posts](/tags/change-in-behavior), as well as warning log messages when appropriate. In 2024, there were 3 such cases: 1. [The arglist attribute now preserves spaces](/blog/2024/change-in-behavior-arglist) 2. [`depth_search` can now be used (but warns) with an individual file as source](/blog/2024/change-in-behavior-depth-search-can-now-be-used-but-warns-with-an-individual-file-as-source) 3. [CFEngine roles inventory attribute](/blog/2024/change-in-behavior-cfengine-roles) ## Feature release: 3.24.0 LTS In July, we released [CFEngine 3.24.0 LTS](/blog/2024/cfengine-3-24-lts-released-consistency). Being an LTS release, 3.24 brought previously released features such as host groups and group data into a supported release; {{< wideimg "/blog/2023/images/groups.png" "Groups application showing a group of Ubuntu hosts across different versions.">}} ## Patch releases Multiple patches have been released throughout 2024 for the LTS versions: * In January, [3.18.7 and 3.21.4](/blog/2024/cfengine-3-18-7-and-3-21-4-released) were released. * In June, [3.18.8 and 3.21.5](/blog/2024/cfengine-3-18-8-and-3-21-5-released) were released. * In December, [3.21.6 and 3.24.1](/blog/2024/cfengine-3-21-6-and-3-24-1-released) were released. We encourage all of our users to stay up to date with the latest patch releases, to benefit from bug fixes, security patches, and dependency updates. Please note that the 3.18.x LTS series is no longer supported, and the 3.21.x series is approaching end of life (December 2025). If you need assistance upgrading to one of our supported versions, or you would like to purchase extended support, please [contact us](/contact). For more information on previous and future 'end of support' dates, please visit [here](/extended-support). ## What's next? Next winter, by the end of 2025, we will launch CFEngine 3.27 LTS, bringing new functionality, including the audit log, to a stable, supported LTS release. If you're interested in what we're focusing on for CFEngine now, and going forward, it can be summarized by these three areas: * Modern infrastructures - Cloud, hybrid cloud, container, and Kubernetes-based environments. * Security and compliance - Recently we added audit logs, and before that, 2FA, a lynis security hardening module, and compliance reports. * Overall quality improvements - UI/UX, ease of use, scalability, performance, bandwidth. -------------------------------------------------------------------------------- title: "Feature Friday #42: ob-cfengine3" url: /blog/2024/feature-friday-42-ob-cfengine3/index.md date: "2024-12-27" -------------------------------------------------------------------------------- For the final post in the Feature Friday series I am here to tell you about something I use nearly hourly, [ob-cfengine3](https://github.com/nickanderson/ob-cfengine3) which extends Emacs Org Babel for executing CFEngine policy. ob-cfengine3 has been around for a little over seven years now and it has saved me countless hours, seconds at a time. At its core it lets you type a snippet of policy and execute it directly in your document, sort of like [Jupyter](https://jupyter.org). {{< figure src="2024-12-18_15-04-54_ob-cfengine3-demo-2019-08-01.gif" >}} By default it tries to include the standard library during execution so that you don't have to manually in-line the common things for small examples. For this to work you will want to be sure and setup CFEngine for unprivileged execution which I talked about in [Feature Friday #25: Unprivileged execution](/blog/2024/feature-friday-25-unprivileged-execution/). It's easy to ... ## Specify the bundlesequence {#specify-the-bundlesequence} ```text #+BEGIN_SRC cfengine3 :bundlesequence one,two bundle agent one { reports: "Running bundle $(this.bundle)"; } bundle agent two { reports: "Running bundle $(this.bundle)"; } #+END_SRC #+RESULTS: : R: Running bundle one : R: Running bundle two ``` ## Define additional classes {#define-additional-classes} ```text #+BEGIN_SRC cfengine3 :define EXTRA,MORE bundle agent main { reports: EXTRA:: "EXTRA Class defined"; MORE:: "EVEN MORE"; } #+END_SRC #+RESULTS: : R: EXTRA Class defined : R: EVEN MORE ``` ## Specify additional arbitrary options to cf-agent {#specify-additional-arbitrary-options-to-cf-agent} ```text #+BEGIN_SRC cfengine3 :extra-opts --show-evaluated-vars=default:main bundle agent main { vars: "test" string => "test string"; } #+END_SRC #+RESULTS: : Variable name Variable value Meta tags : default:main.test test string source=promise : default:maintain_key_values_meta.tags {"deprecated=3.6.0","deprecation-reason=Generic reimplementation","replaced-by=set_line_based"} source=promise ``` ## Change the output log level {#change-the-output-log-level} ```text #+BEGIN_SRC cfengine3 :log-level info bundle agent example { commands: "/bin/echo Hello World"; } bundle agent __main__ { methods: "example"; } #+END_SRC #+RESULTS: : info: Executing 'no timeout' ... '/bin/echo Hello World' : notice: Q: ".../bin/echo Hello": Hello World : info: Last 1 quoted lines were generated by promiser '/bin/echo Hello World' : info: Completed execution of '/bin/echo Hello World' ``` ## Use a custom command for snippet execution {#use-a-custom-command-for-snippet-execution} Here we are executing cf-agent from a docker image, this can be very useful if you are wanting to run a snippet against differing versions of CFEngine. ```text #+begin_src cfengine3 :command "docker run -v /tmp:/tmp zzamboni/cf-agent" :tmpdir /tmp bundle agent main { reports: "My hostname: $(sys.fqhost)"; } #+end_src #+RESULTS: : R: My hostname: 5cd98f9265a8 ``` ## Include command in the output {#include-command-in-the-output} Especially when showing examples or working on documentation showing the command executed can be quite nice. ```text #+begin_src cfengine3 :command "docker run -v /tmp:/tmp zzamboni/cf-agent" :tmpdir /tmp :command-in-result t bundle agent main { reports: "Hello world!"; } #+end_src #+RESULTS: : # docker run -v /tmp:/tmp zzamboni/cf-agent --no-lock --file /tmp/cfengine3-b7caCd : R: Hello world! ``` Fun fact: This is how examples are handled in [Learning CFEngine](https://leanpub.com/learning-cfengine). ## Execute partial snippets {#execute-partial-snippets} ```text #+begin_src cfengine3 :run-with-main yes reports: "Hello world!"; #+end_src #+RESULTS: : R: Hello world! ``` ## Getting it {#getting-it} In order to use this wonderful tool you will have to use Emacs. Learning Emacs might seem as steep a curve as CFEngine but I found Spacemacs got me going quickly and I didn't have to leave any of my vi muscle memory behind. In fact, [The agent is in - Episode 12 - Spacemacs for CFEngine](/blog/2022/the-agent-is-in-episode-12-spacemacs-for-cfengine/) covered getting set up. ## Thanks {#thanks} Thanks for coming along on this 42 part series as I celebrated CFEngine's 30th anniversary, I hope you found some useful tidbits along the way. One last time ... Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 44 - Cody and Nick's Christmas Special" url: /blog/2024/the-agent-is-in-episode-44/index.md date: "2024-12-26" -------------------------------------------------------------------------------- Join Cody and Nick for a Christmas Special showcasing the new Audit Log in Mission Portal for CFEngine 3.25. Nick walked through the new Audit Log demonstrating how actions in Mission Portal are tracked and available for review. He also took a quick look at changes to the global search and taking some questions off air from a few attendees. ## Video {#video} The video recording is available on YouTube: {{< youtube 2YJiZSfKW_8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Post Show Questions {#post-show-questions} ### Does CFEngine keep track of when the list of software updates change? {#does-cfengine-keep-track-of-when-the-list-of-software-updates-change} In Mission Portal the software updates available report shows a `Discovered at` timestamp: {{< wideimg "/blog/2024/the-agent-is-in-episode-44/2024-12-26_11-32-51_Screenshot_20241226_113238.png" "Software updates">}} This is the time when the set of software updates have changed, you will notice that all the packages share the same times in this report. ### Can I emit reports about the version a piece of software transitioned from and to? {#can-i-emit-reports-about-the-version-a-piece-of-software-transitioned-from-and-to} e.g. can I emit updated Firefox from version 12 to 13.2? Most of the information that CFEngine makes readily available has to do with current state. I can imagine ways to record histories of this locally on each client for each package but I think the value compared to the work necessary is quite low. Probably this would need a couple of separate bundles to be responsible for recording the current state early and late during the policy run and then comparing the states before emitting reports. ```cfengine3 bundle agent __main__ { reports: firefox_reparied:: "Firefox updated $(with)" with => concat( "from '", readfile( "$(sys.statedir)/software-versions/firefox.txt", "inf"), "' to ", $(CURRNENT VERSION) ); } ``` > The value is in being able to point to a nice timeline of this is when we updated application X on your hosts. Using Mission Portal in CFEngine Enterprise is probably a better way to get this information centrally. For example, we can use custom SQL to get the two most recent records for `libsqlite3-0` on `hub.example.com`: ```sql WITH SoftwareUpdates AS ( SELECT sl.hostkey, sl.changetimestamp, sl.changeoperation, sl.softwarename, sl.softwareversion, ROW_NUMBER() OVER (PARTITION BY sl.hostkey, sl.softwarename ORDER BY sl.changetimestamp DESC) AS rn FROM softwarelog sl ) SELECT h.hostname, su.changetimestamp, su.changeoperation, su.softwarename, su.softwareversion FROM SoftwareUpdates su JOIN hosts h ON su.hostkey = h.hostkey WHERE su.rn <= 2 AND su.softwarename = 'libsqlite3-0' AND h.hostname = 'hub.example.com' ORDER BY su.changetimestamp DESC; ``` This shows us that the most recent change for package `libsqlite3-0` which was an upgrade from `3.40.1-2` to `3.40.1-2+deb12u1`: ```text hostname | changetimestamp | changeoperation | softwarename | softwareversion -----------------+------------------------+-----------------+--------------+------------------ hub.example.com | 2024-12-23 22:57:34+00 | REMOVE | libsqlite3-0 | 3.40.1-2 hub.example.com | 2024-12-23 22:57:34+00 | ADD | libsqlite3-0 | 3.40.1-2+deb12u1 (2 rows) ``` ### Does CFEngine Inventory hosts that need to be re-started after software update? {#does-cfengine-inventory-hosts-that-need-to-be-re-started-after-software-update} No, at least not by default however it would be pretty easy to write some policy for this. For example we could [check if a reboot is required](https://megamorf.gitlab.io/2019/06/10/check-if-reboot-is-required-after-installing-linux-updates/) by inventorying the presence of `/var/run/reboot-required` on Debian derivatives or the return code and or output from `needs-restarting` on Red Hat derivatives. This would be a great candidate for a module on [build.cfengine.com](https://build.cfengine.com). ```cfengine3 bundle agent inventory_host_needs_reboot { classes: debian|ubuntu:: "host_needs_reboot" expression => fileexists( "/var/run/reboot-required" ), scope => "namespace"; redhat|centos|rocky_linux:: "host_needs_reboot" expression => not( returnzero( "needs-restarting", "useshell" ) ), scope => "namespace"; vars: "status" string => ifelse( "host_needs_reboot", "Yes", "No" ), meta => { "inventory", "attribute_name=Needs reboot" }; } ``` ## Links {#links} * Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/) or [Nick](https://www.linkedin.com/in/hithisisnick/) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Feature Friday #41: How can I quote thee, let me count the ways" url: /blog/2024/feature-friday-41-how-can-i-quote-thee/index.md date: "2024-12-20" -------------------------------------------------------------------------------- Do you enjoy escaping quotes inside strings? I sure don't, and I really appreciate the flexibility CFEngine provides with 3 different quoting characters (`"`, `'`, `` ` `` ). Let's take a look.
This came up in the post show discussion for [The agent is in, episode 39](https://cfengine.com/blog/2024/the-agent-is-in-episode-39/).
If you have a string that contains double quotes you might see it written with escaped quotes like this: ```cfengine3 bundle agent __main__ { vars: "my_string_contains_escaped_double_quotes" string => "I \"hate\" escaping quotes"; reports: "$(my_string_contains_escaped_double_quotes)"; } ``` ```text # cf-agent --no-lock --file /tmp/feature-friday-41-0.cf R: I "hate" escaping quotes ``` CFEngine also supports single quotes (`'`) which allows you to avoid escaping: ```cfengine3 bundle agent __main__ { vars: "my_string_contains_double_quotes" string => 'I "hate" escaping quotes'; reports: "$(my_string_contains_double_quotes)"; } ``` ```text # cf-agent --no-lock --file /tmp/feature-friday-41-1.cf R: I "hate" escaping quotes ``` There is however the occasion where you need both single and double quotes within a string. When that is the case it's a really good time to remember that CFEngine also recognizes backticks (`` ` ``) as a quoting symbol. ```cfengine3 bundle agent __main__ { vars: "my_string_contains_single_and_double_quotes" string => `'I "hate" escaping quotes'`; reports: "I said $(my_string_contains_single_and_double_quotes)!"; } ``` ```text # cf-agent --no-lock --file /tmp/feature-friday-41-2.cf R: I said 'I "hate" escaping quotes'! ``` Of course you can change the order around for your needs as we see in this next example: ```cfengine3 bundle agent __main__ { vars: "my_string_contains_single_and_double_quotes" string => `"I 'hate' escaping quotes"`; "my_string_contains_backticks_and_double_quotes" string => '"I `hate` escaping quotes"'; reports: "Have I mentioned $(my_string_contains_single_and_double_quotes)?"; "Yes, really, $(my_string_contains_backticks_and_double_quotes)."; } ``` ```text # cf-agent --no-lock --file /tmp/feature-friday-41-3.cf R: Have I mentioned "I 'hate' escaping quotes"? R: Yes, really, "I `hate` escaping quotes". ``` I hope you enjoyed this tip more than you enjoy escaping quotes. Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #40: What would CFEngine do?" url: /blog/2024/feature-friday-40-what-would-cfengine-do/index.md date: "2024-12-13" -------------------------------------------------------------------------------- CFEngine works by defining a desired state for a given context and converging towards that goal. Given there is no fixed starting point and that the current context might change wildly it can be challenging to succinctly answer the question "What would CFEngine do?". In [Feature Friday #22: Don't fix, just warn](/blog/2024/feature-friday-22-dont-fix-just-warn/) we saw how an individual promise could be made to warn instead of trying to automatically converge towards the desired state, a granular `--dry-run` mode. This time, let's take a look at the `--simulate` option of `cf-agent`. The `--simulate` option added back in November 2020 as part of [CFEngine 3.17.0](/blog/2020/cfengine-3-17-released-flexibility/) expands on the notion of simply warning that a change would be desired as is the case with `--dry-run` or `-n` (which set `action_policy` to `warn` for the entire policy run). As with the `--dry-run` option most promises ( `commands`, `packages`, `users` etc ... ) are not actuated. `--simulate` differs in the case of `vars` and `classes` promises and the use of functions that are considered _unsafe_ ( [execresult()](https://docs.cfengine.com/docs/3.24/reference-functions-execresult.html), [execresult_as_data()](https://docs.cfengine.com/docs/3.24/reference-functions-execresult_as_data.html), [returnszero()](https://docs.cfengine.com/docs/3.24/reference-functions-returnszero.html), and [usemodule()](https://docs.cfengine.com/docs/3.24/reference-functions-usemodule.html) ). With `--dry-run` those promises would be evaluated (as there is a general expectation that they are not used to _modify_ system state ) while with the `--simulate` option `vars` and `classes` promises leveraging those functions would only actuate if they are tagged with `simulate_safe`. Let's take a look at a small example: In this policy we have two classes `class_defined_by_returnszero_not_marked_safe` and `class_defined_by_returnszero_marked_safe` with the latter tagged as safe for simulation and two reports to be emitted based on the definition of the classes. ```cfengine3 {file="/tmp/feature-friday-40-0.cf"} bundle agent __main__ { classes: "class_defined_by_returnszero_not_marked_safe" expression => returnszero( "/bin/true", noshell ); "class_defined_by_returnszero_marked_safe" expression => returnszero( "/bin/true", noshell ), meta => { "simulate_safe" }; reports: "class_defined_by_returnszero_marked_safe is defined" if => "class_defined_by_returnszero_not_marked_safe"; "class_defined_by_returnszero_marked_safe is defined" if => "class_defined_by_returnszero_marked_safe"; } ``` Running the example with `--dry-run` we see warnings that indicate a desire to emit the reports "class_defined_by_returnszero_marked_safe is defined" and "class_defined_by_returnszero_marked_safe is defined". ```command cf-agent --no-lock --dry-run --file /tmp/feature-friday-40-0.cf ``` ```output warning: Need to repair reports promise: class_defined_by_returnszero_marked_safe is defined warning: Need to repair reports promise: class_defined_by_returnszero_marked_safe is defined ``` The reports were not emitted because the policy was executed with `--dry-run`, but we can infer that both classes were defined as part of the policy evaluation. Next, let's run the same policy with `--simulate=diff`: ```command cf-agent --no-lock --simulate=diff --file /tmp/feature-friday-40-0.cf ``` ```output warning: All changes in files will be made in the '/home/nickanderson/.cfagent/state/99979.changes' chroot warning: Not calling unsafe function 'returnszero' in simulate mode warning: Not calling unsafe function 'returnszero' in simulate mode warning: Not calling unsafe function 'returnszero' in simulate mode warning: Need to repair reports promise: class_defined_by_returnszero_marked_safe is defined warning: Not calling unsafe function 'returnszero' in simulate mode warning: Not calling unsafe function 'returnszero' in simulate mode warning: Not calling unsafe function 'returnszero' in simulate mode warning: Not calling unsafe function 'returnszero' in simulate mode ``` As you might have noticed from the above output, `--simulate` also differs from `--dry-run` in that `files` promises operate within a `change-root` so that we can safely observe the effects. This time we see that only a single warning about a desire to report is emitted, the report which we expect to see if the class `class_defined_by_returnszero_marked_safe` defined by the promise tagged `simulate_safe`. We see many cases where `returnszero()` would be executed but was avoided because it was not tagged `simulate_safe` and we can infer that the other class was not defined by the lack of a warning about a desire to report. Let's try a policy that makes a promise about file content. ```cfengine3 {file="/tmp/feature-friday-40-1.cf"} bundle agent __main__ { files: "/tmp/feature-friday-40.txt" content => "Happy Feature Friday #40!"; } ``` First, let's see what `--dry-run` will do: ```command cf-agent --no-lock --dry-run --file /tmp/feature-friday-40-0.cf ``` We can see from the output that the file would be created: ```output warning: Should create file '/tmp/feature-friday-40.txt', mode '0600' warning: Warnings encountered when actuating files promise '/tmp/feature-friday-40.txt' ``` Next, let's try `--simulate`: ```command cf-agent --no-lock --simulate=diff --file /tmp/feature-friday-40-0.cf ``` Here we can see that the proposed changes are made within a chroot: ```output warning: All changes in files will be made in the '/home/nickanderson/.cfagent/state/105101.changes' chroot =========================================================================== '/tmp/feature-friday-40.txt' is a regular file Size: 25 Access: (0600/rw-------) Uid: (1000/nickanderson) Gid: (1000/nickanderson) Access: 2024-10-21 16:04:55 -0500 Modify: 2024-10-21 16:04:55 -0500 Change: 2024-10-21 16:04:55 -0500 Contents of the file: Happy Feature Friday #40! \no newline at the end of file ``` And let's also take a look at the output in the case where the promised file (`/tmp/feature-friday-40.txt`) exists on the host system. ```command echo "Hello World!" > /tmp/feature-friday-40.txt;\ cf-agent --no-lock --simulate=diff --file /tmp/feature-friday-40-0.cf ``` ```output warning: All changes in files will be made in the '/home/nickanderson/.cfagent/state/105943.changes' chroot =========================================================================== --- original /tmp/feature-friday-40.txt +++ changed /tmp/feature-friday-40.txt @@ -1 +1 @@ -Hello World! +Happy Feature Friday #40! \ No newline at end of file ``` Until now we have only looked at the output of the `--simulate` option when set to `diff`. Let's briefly look at the output from the other two options `manifest` and `manifest-full`. The manifest modes do not output unified diff and instead are intended to help you see the full desired end state. Let's adjust the policy a bit more and take a look. ```cfengine3 {file="/tmp/feature-friday-40-2.cf"} bundle agent __main__ { files: "/tmp/feature-friday-40.txt" content => "Happy Feature Friday #40!"; "/tmp/feature-friday-40-Hello-World.txt" content => "Hello World!"; } ``` Now we are making two `files` promises, let's make the state of the host system match the second promise: Now when we run the policy with `--simulate=manifest` we see the end state that would be achieved had the run not been executed with `--simulate`: ```command cf-agent --no-lock --simulate=manifest --file /tmp/feature-friday-40-2.cf ``` ```output warning: All changes in files will be made in the '/home/nickanderson/.cfagent/state/110986.changes' chroot =========================================================================== '/tmp/feature-friday-40.txt' is a regular file Size: 25 Access: (0664/rw-rw-r--) Uid: (1000/nickanderson) Gid: (1000/nickanderson) Access: 2024-10-21 16:20:40 -0500 Modify: 2024-10-21 16:20:40 -0500 Change: 2024-10-21 16:20:40 -0500 Contents of the file: Happy Feature Friday #40! \no newline at the end of file ``` Notice that there was no mention of `/tmp/feature-friday-40-Hello-World.txt` since the execution of the agent was not going to result in a change to that file. To see a more full end state of what CFEngine manages we can run with `--simulate=manifest-full`. ```command cf-agent --no-lock --simulate=manifest-full --file /tmp/feature-friday-40-2.cf ``` ```output warning: All changes in files will be made in the '/home/nickanderson/.cfagent/state/111360.changes' chroot =========================================================================== '/tmp/feature-friday-40.txt' is a regular file Size: 25 Access: (0664/rw-rw-r--) Uid: (1000/nickanderson) Gid: (1000/nickanderson) Access: 2024-10-21 16:22:40 -0500 Modify: 2024-10-21 16:22:40 -0500 Change: 2024-10-21 16:22:40 -0500 Contents of the file: Happy Feature Friday #40! \no newline at the end of file =========================================================================== '/tmp/feature-friday-40-Hello-World.txt' is a regular file Size: 12 Access: (0664/rw-rw-r--) Uid: (1000/nickanderson) Gid: (1000/nickanderson) Access: 2024-10-21 16:22:40 -0500 Modify: 2024-10-21 16:22:40 -0500 Change: 2024-10-21 16:22:40 -0500 Contents of the file: Hello World! \no newline at the end of file ``` As we can see from the above output the state of `/tmp/feature-friday-40-Hello-World.txt` is included even though it already matches the desired end state. I hope you found this quick tour of the `--simulate` option stimulating. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #39: The power of lists and implicit iteration" url: /blog/2024/feature-friday-39-the-power-of-lists-and-implicit-iteration/index.md date: "2024-12-06" -------------------------------------------------------------------------------- Implicit list iteration in CFEngine is quite a unique and novel feature. Today we look at a practice example showing how lists can improve the readability and maintainability of your policy. A novel feature in CFEngine is how a list variable is iterated when referenced as a scalar (`$(variable)`). Let's take a look at a contrived example. Here we see a list of strings (`slist`) defined as `toys` and we have a single `reports` promise to emit toys we want to play with. ```cfengine3 {file="/tmp/feature-friday-39-toys.cf"} bundle agent __main__ { vars: "toys" slist => { "Legos", "Dolls", "Action figures", "Play-Doh" }; reports: "Let's play with $(toys)"; } ``` Running the example we see that the report is emitted for each toy in our list as a separate line. This is effectively making 4 individual promises, expressed in a single statement. ```command # cf-agent --no-lock --log-level info \ --file /tmp/feature-friday-39-toys.cf ``` ```output R: Let's play with Legos R: Let's play with Dolls R: Let's play with Action figures R: Let's play with Play-Doh ``` Now, let's move on to a more practical example. Let's say that you want to inventory the amount of free disk space available for various paths. Here we have several variables defining the amount of free disk space for each and then we have several more variables that do some math to make the data more user friendly which is actually tagged for inventory. ```cfengine3 {file="/tmp/feature-friday-39-no-iteration.cf"} bundle agent __main__ { vars: "boot_disk_free" int => diskfree("/boot"); "root_disk_free" int => diskfree("/"); "usr_disk_free" int => diskfree("/usr"); "var_disk_free" int => diskfree("/var"); "boot_disk_free_mb" string => eval("$(boot_disk_free) / 1024)","math", "infix"), meta => { "inventory", "attribute_name=Disk free boot (MB)" }; "root_disk_free_mb" string => eval("$(root_disk_free) / 1024)", "math", "infix"), meta => { "inventory", "attribute_name=Disk free root (MB)" }; "usr_disk_free_mb" string => eval("$(usr_disk_free) / 1024)", "math", "infix"), meta => { "inventory", "attribute_name=Disk free usr (MB)" }; "var_disk_free_mb" string => eval("$(var_disk_free) / 1024)", "math", "infix"), meta => { "inventory", "attribute_name=Disk free var (MB)" }; reports: "/boot has $(boot_disk_free_mb) MB of free disk space."; "/ has $(root_disk_free_mb) MB of free disk space."; "/usr has $(usr_disk_free_mb) MB of free disk space."; "/var has $(var_disk_free_mb) MB of free disk space."; } ``` Running the policy we see it works without issue: ```command # cf-agent --no-lock --log-level info \ --file /tmp/feature-friday-39-no-iteration.cf ``` ```output R: /boot has 3454.355469 MB of free disk space. R: / has 1486371.406250 MB of free disk space. R: /usr has 1486371.406250 MB of free disk space. R: /var has 1486371.406250 MB of free disk space. ``` But, using a list can significantly improve this policy. Let's refactor it so that we define a list of disks. Then we can iterate over that list of disks to get the free disk space for each while calculating the more human friendly value. You can see the policy required is significantly less. ```cfengine3 {file="/tmp/feature-friday-39-iteration.cf"} bundle agent __main__ { vars: "disks" slist => { "/boot", "/", "/usr", "/var" }; "disk_free_mb[$(disks)]" string => eval("$(with) / 1024", "math", "infix"), meta => { "inventory", "attribute_name=Disk free $(disks) (MB)" }, with => diskfree("$(disks)"); reports: "$(disks) has $(disk_free_mb[$(disks)]) MB of free disk space."; } ``` Executing the policy we can see that the results are the same. ```command # cf-agent --no-lock --log-level info \ --file /tmp/feature-friday-39-iteration.cf ``` ```output R: /boot has 3454.355469 MB of free disk space. R: / has 1486371.210938 MB of free disk space. R: /usr has 1486371.210938 MB of free disk space. R: /var has 1486371.210938 MB of free disk space. ``` We can even take this a step further. Maybe `3454.355469 MB` isn't so friendly after all. Instead we would prefer to see just a whole number. We can achieve this with a very minor refactor by wrapping the `eval()` with `format("%d")`, or as shown here, switching the type to `int` and wrapping it with `int()` instead. ```cfengine3 {file="/tmp/feature-friday-39-iteration-formatted.cf"} bundle agent __main__ { vars: "disks" slist => { "/boot", "/", "/usr", "/var" }; "disk_free_mb[$(disks)]" int => int( eval("$(with) / 1024", "math", "infix") ), meta => { "inventory", "attribute_name=Disk free $(disks) (MB)" }, with => diskfree("$(disks)"); reports: "$(disks) has $(disk_free_mb[$(disks)]) MB of free disk space."; } ``` Executing the policy we see a much better final result. ```command # cf-agent --no-lock --log-level info \ --file /tmp/feature-friday-39-iteration-formatted.cf ``` ```output R: /boot has 3454 MB of free disk space. R: / has 1486369 MB of free disk space. R: /usr has 1486369 MB of free disk space. R: /var has 1486369 MB of free disk space. ``` I hope this real-world example helps you see the great power that comes with CFEngine's implicit list iteration. Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "CFEngine 3.21.6 and 3.24.1 released" url: /blog/2024/cfengine-3-21-6-and-3-24-1-released/index.md date: "2024-12-04" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.21.6 and 3.24.1\! These patch releases contain bug fixes and dependency updates. ## Changes In 3.24.1, Mission Portal has one new feature, requested by our users, which we'd like to highlight - When logging in to the CFEngine Enterprise web UI, Mission Portal, we now support 2FA using a time-based one time password (TOTP). Aside from this, these are patch releases which mainly focus on bugfixes, not new features. ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.24.1 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-core.html) - [3.24.1 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-enterprise.html) - [3.24.1 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.6 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.6 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.6 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades Compared to the previous releases, CFEngine [3.21.5](https://cfengine.com/blog/2024/cfengine-3-18-8-and-3-21-5-released/) and [3.24.0](https://cfengine.com/blog/2024/cfengine-3-24-lts-released-consistency/), the following dependencies were updated for the client packages: | CFEngine version | 3.21.5 | **3.21.6** | 3.24.0 | **3.24.1** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [libcurl](https://curl.haxx.se/download.html) | 8.7.1 | **8.10.1** | 8.8.0 | **8.10.1** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.12.6 | **2.13.4** | 2.13.1 | **2.13.4** | | [OpenSSL](https://openssl.org/) | 3.0.13 | **3.0.15** | 3.3.1 | **3.4.0** | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.32 | **0.9.33** | 0.9.33 | 0.9.33 | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.7 | **2.6.8** | 2.6.7 | **2.6.8** | And for the hub packages, these additional dependencies were also upgraded: | CFEngine version | 3.21.5 | **3.21.6** | 3.24.0 | **3.24.1** | | :-------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.59 | **2.4.62** | 2.4.59 | **2.4.62** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.45.1 | **2.47.0** | 2.45.2 | **2.47.0** | | [PHP](https://php.net/) | 8.2.19 | **8.3.13** | 8.3.8 | **8.3.13** | | [PostgreSQL](https://www.postgresql.org/) | 15.6 | **15.8** | 16.3 | **17.0** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.21/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.24.1 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.21.6 list ``` ```command cf-remote --version 3.24.1 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.18 LTS Please note that support for the 3.18 LTS series has ended (August 2024). If you are on an old and unsupported version, please [contact us](/contact) for help with upgrading or to discuss an extended support contract. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Handle the state of containers in a Docker compose configuration" url: /blog/2024/promise-type-docker-compose/index.md date: "2024-12-02" -------------------------------------------------------------------------------- Recently we had a [Fireside Chat](the-agent-is-in-episode-42/) with long-time contributor and CFEngine Champion Bas Van der Vlies. During that talk he mentioned a Build module he developed: [promise-type-docker-compose](https://build.cfengine.com/modules/promise-type-docker-compose/). For this month's Module Monday post I thought I would take this promise type for a spin alongside the [Docker Compose Quickstart](https://docs.docker.com/compose/gettingstarted/) tutorial. # Setup For this blog I brought up a libvirt vagrant VM with Debian 12 and installed the latest LTS (3.24.0) with [cf-remote](https://github.com/cfengine/cf-remote). To install docker I follow the instructions at [Install Docker Engine](https://docs.docker.com/engine/install/). I was using [Debian 12 "bookworm"](https://www.debian.org/releases/bookworm/) and found that the default packages `docker.io` and `docker-compose` are rather old and were not compatible with the promise-type-docker-compose module. Additionally, the module relies on the `jq` utility, so I install that as well. I may write policy some time to take care of these dependencies but especially the Docker Engine install involves quite a few steps so will leave that for another time. # Build project After installing those dependencies I create a blank [CFBS](https://github.com/cfengine/cfbs) project. ```sh $ pipx install cfbs $ mkdir play-cfbs $ cd play-cfbs $ cfbs init # then answer some questions... ``` Then add the promise type module: ```sh $ cfbs add promise-type-docker-compose Added module: library-for-promise-types-in-bash (Dependency of promise-type-docker-compose) Added module: promise-type-docker-compose The default commit message is: Added 2 modules - Added module 'promise-type-docker-compose' - Added module 'library-for-promise-types-in-bash' Edit it? [yes/y/NO/n] Committing using git: [promise-type-docker-compose bcbffee] Added 2 modules 1 file changed, 28 insertions(+) ``` I copy/paste the various files from the tutorial. ```sh $ mkdir composetest $ cat > composetest/app.py $ cat > composetest/requirements.txt $ cat > composetest/Dockerfile $ cat > composetest/compose.yaml ``` # Policy I use the example policy from the module README to bring up the app. ```cf3 {file="main.cf"} bundle agent composetest { docker_compose: "${this.promise_dirname}/compose.yaml" state => "start"; } ``` I add this `composetest` directory to my cfbs project with the `cfbs add ` command. Adding a directory with cfbs causes all of the files in that directory [to be copied](https://github.com/cfengine/cfbs/blob/master/JSON.md#all-available-build-steps) into the built policy as well as specific files in the directory being handled in a reasonable way: - Copy any .cf policy files recursively and add their paths to def.json's inputs. - Enable services_autorun_bundles class in def.json. - Merge any def.json recursively into out/masterfiles/def.json. - Copy any other files with their existing directory structure to destination. ```sh $ cfbs add ./composetest Which bundle should be evaluated (added to bundle sequence)? 1. ./composetest/main.cf:composetest (default) 2. (None) [1/2]: Added module: ./composetest/ The default commit message is 'Added module './composetest/'' - edit it? [yes/y/NO/n] Committing using git: [promise-type-docker-compose f439e31] Added module './composetest/' 1 file changed, 11 insertions(+) ``` Notice that `cfbs add` detects that my `main.cf` policy file has a bundle named `composetest` which I agree to add as a bundle that should be included in my policy evaluation. One little snag we hit here is in the way that policy is copied and then where it is used by the default [Masterfiles Policy Framework](https://github.com/cfengine/masterfiles) update policy. The files in our `composetest` are indeed part of the built policy but only certain files are copied into `/var/cfengine/inputs` where `cf-execd` and `cf-agent` periodically run the policy from. Because of this we need to add an extra pattern of what files to copy to inputs. Luckily there is an MPF extension available for just this purpose: [Extend files considered for copy during policy updates](https://docs.cfengine.com/docs/3.24/reference-masterfiles-policy-framework.html#extend-files-considered-for-copy-during-policy-updates). As mentioned before we can add a `def.json` to our `composetest` directory and the cfbs directory command will add any augments in that file to the main `def.json` in the resulting policy. ```json {file="./composetest/def.json"} { "variables": { "default:update_def.input_name_patterns_extra": { "value": [ ".*\\Dockerfile" ], "comment": "Dockerfile or *.Dockerfile should be copied so that promise-type-docker-compose works when referencing this.promise_dirname/*.yaml files." } } } ``` I push my changes up to a branch on a git repo. Then I visit [Mission Portal](https://docs.cfengine.com/docs/3.24/web-ui.html) and configure a Build project to use this repository. After clicking on "Push and Deploy" I wait a few agent cycles (5-15 minutes) and check that the app is running. And indeed it is! {{< wideimg "images/docker-compose-tutorial-app-working.png" "browser showing docker compose tutorial app working">}} The complete Build project for this blog is available in the [promise-type-docker-module](https://github.com/cfengine/play-cfbs/tree/promise-type-docker-compose) branch of our example projects repo: [play-cfbs](https://github.com/cfengine/play-cfbs). Take a look at [all the branches](https://github.com/cfengine/play-cfbs/branches/all) for other example projects. ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #38: Developing modules that take input" url: /blog/2024/feature-friday-38_developing_modules_that_take_input/index.md date: "2024-11-29" -------------------------------------------------------------------------------- CFEngine build modules are great for quickly integrating 3rd party policy into your policy set. Module input (not to be confused with `inputs` in `body common control` or `body file control` which are the list of policy files to load) allows you to define values that apply for a particular module as it's integrated into your policy set. Let's take a look at a case of extending a module to support input. ## Extending the module to support input {#extending-the-module-to-support-input} The `lynis` module (), as of version `3.1.1` provides some variables that can be used to configure the version to be installed, and the hash of that version's archive. Each of those variables supports overriding from Augments. The module is hard-coded to download the specified version directly from CISOfy (`https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz`). These are all great candidates to expose as module input, allowing users to specify a version, hash, and url to download from, let's take a look at what is needed to add support. First, the `tar_url` variable needs to be instrumented to allow for the value in policy to be overridden from Augments. ```cfengine3 "tar_url" string => "https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz"; ``` To do that we can adjust the policy to define `tar_url` only if it's not already defined (which is the case if the value is defined via Augments, prior to policy evaluation). ```cfengine3 "tar_url" string => "https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz", if => not( isvariable( "tar_url" ) ); ``` Next, we can also allow `hash_type` which defaults to `sha256` to be overridden via augments. ```cfengine3 "hash_type" string => "sha256", if => not( isvariable( "hash_type" ) ); ``` With these two simple changes the policy is ready and we can look to instrument the `cfbs.json` file with module input. You can find the CFEngine Build JSON file format [here in JSON.md](https://github.com/cfengine/cfbs/blob/master/JSON.md) of the cfbs repository. Checkout the section on [Modules with input](https://github.com/cfengine/cfbs/blob/master/JSON.md#modules-with-input). In summary, module input is defined as an array of objects under the `input` key for the module. Here is the specification that covers exposing the version of lynis, the url to retrieve the archive from, the archive hash, and archive hash type with default values that match the policy defaults. ```json "input": [ { "type": "string", "namespace": "lynis", "bundle": "globals", "variable": "version", "label": "Lynis version", "question": "Where should clients download lynis from (https)", "default": "3.1.1" }, { "type": "string", "namespace": "lynis", "bundle": "globals", "variable": "tar_url", "label": "Tarball url", "question": "Where should clients download lynis from (http/https)", "default": "https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz", "comment": "Some may want to self host the tarball within their own infrastructure. This provides an easy way to do that." }, { "type": "string", "namespace": "lynis", "bundle": "globals", "variable": "archive_hash", "label": "Hash of the tarball", "question": "What is the hash of the tarball?", "default": "d72f4ee7325816bb8dbfcf31eb104207b9fe58a2493c2a875373746a71284cc3" }, { "type": "string", "namespace": "lynis", "bundle": "globals", "variable": "hash_type", "label": "Hash type to verify", "question": "What hashing algorithm should be used to verify the Lynis script?", "default": "sha256" } ] ``` We also need to add `input` to the build steps: ```json "steps": [ "copy policy/main.cf services/lynis/main.cf", "json cfbs/def.json def.json", "input ./input.json def.json" ], ``` Next we can take a look at the effect of adding support for module input. ## Reviewing the effects of module input {#reviewing-the-effects-of-module-input} Adding the module with this input via `cfbs` on the command line results in a session that looks like this: ```text $ cfbs add lynis Added module: lynis The added module 'lynis' accepts user input. Do you want to add it now? [yes/y/NO/n] y Collecting input for module 'lynis' What version of Lynis should be used? [3.1.1] Where should clients download lynis from (http/https) [https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz]: What is the hash of the tarball? [d72f4ee7325816bb8dbfcf31eb104207b9fe58a2493c2a875373746a71284cc3] What hashing algorithm should be used to verify the Lynis script? [sha256] The default commit message is: Added module 'lynis' - Added input for module 'lynis' Edit it? [yes/y/NO/n] Committing using git: [main 85dcb1d] Added module 'lynis' 2 files changed, 97 insertions(+) create mode 100644 lynis/input.json ``` We could of course override the module input defaults: ```text $ cfbs add lynis Added module: lynis The added module 'lynis' accepts user input. Do you want to add it now? [yes/y/NO/n] y Collecting input for module 'lynis' What version of Lynis should be used? [3.1.1] 3.1.0 Where should clients download lynis from (http/https) [https://downloads.cisofy.com/lynis/lynis-$(version).tar.gz]: What is the hash of the tarball? [d72f4ee7325816bb8dbfcf31eb104207b9fe58a2493c2a875373746a71284cc3] ca192ac67411b07ec8421d579b1f16c038299ff727a53d739403b729817bc2e7 What hashing algorithm should be used to verify the Lynis script? [sha256] The default commit message is: Added module 'lynis' - Added input for module 'lynis' Edit it? [yes/y/NO/n] Committing using git: [main 331b23d] Added module 'lynis' 2 files changed, 97 insertions(+) create mode 100644 lynis/input.json ``` For Enterprise users, modules installed from the build index input can be specified from within Mission Portal. {{< figure src="Reviewing_the_effects_of_module_input/2024-09-05_14-27-19_screenshot.png" >}} Running the policy, we can see that the module input takes effect, here causing the 3.1.0 package to be downloaded, installed and run: ```text [root@hub ~]# cf-agent -KI info: Created directory '/opt/lynis/lynis-3.1.0/.', mode 0700 info: Executing 'no timeout' ... '/usr/bin/curl --silent https://downloads.cisofy.com/lynis/lynis-3.1.0.tar.gz -L --output /opt/lynis/lynis-3.1.0.tar.gz' info: Completed execution of '/usr/bin/curl --silent https://downloads.cisofy.com/lynis/lynis-3.1.0.tar.gz -L --output /opt/lynis/lynis-3.1.0.tar.gz' info: Executing 'no timeout' ... '/bin/tar -zxf /opt/lynis/lynis-3.1.0.tar.gz -C /opt/lynis/lynis-3.1.0 --strip-components=1' info: Completed execution of '/bin/tar -zxf /opt/lynis/lynis-3.1.0.tar.gz -C /opt/lynis/lynis-3.1.0 --strip-components=1' info: Executing 'no timeout' ... '/opt/lynis/lynis-3.1.0/lynis audit system --quiet --auditor 'CFEngine cf-agent'' notice: Q: "....1.0/lynis audi": /opt/lynis/lynis-3.1.0/include/tests_mac_frameworks: line 161: permissive: command not found Q: "....1.0/lynis audi": /opt/lynis/lynis-3.1.0/include/tests_mac_frameworks: line 162: permissive: command not found info: Last 2 quoted lines were generated by promiser '/opt/lynis/lynis-3.1.0/lynis audit system --quiet --auditor 'CFEngine cf-agent'' info: Completed execution of '/opt/lynis/lynis-3.1.0/lynis audit system --quiet --auditor 'CFEngine cf-agent'' ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 43 - Nick & Cody's Thanksgiving Day Special" url: /blog/2024/the-agent-is-in-episode-43/index.md date: "2024-11-28" -------------------------------------------------------------------------------- Nick and Cody celebrate Thanksgiving with a holiday special reviewing some policy related questions Nick recently received. ### Get the list of all network interfaces present {#get-the-list-of-all-network-interfaces-present} [default:sys.interfaces](https://docs.cfengine.com/docs/3.24/reference-special-variables-sys.html#sys-interfaces) only contains configured interfaces, not configured interfaces. [default:sys.interfaces_data](https://docs.cfengine.com/docs/3.24/reference-special-variables-sys.html#sys-interfaces_data) at least for Linux systems, this variable contains much more information. ```cfengine3 {file="/tmp/getindices-sys.interfaces_data.cf"} bundle agent __main__ { vars: "sys_interfaces_data_keys" slist => getindices( "sys.interfaces_data" ); reports: "$(sys_interfaces_data_keys)"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/getindices-sys.interfaces_data.cf ``` ```output R: lo R: enx0892048803e7 R: enx5cff35c6864b R: wlp0s20f3 R: virbr0 R: docker0 R: br-a7d465b9949b R: vboxnet0 R: tun0 ``` [cf-promises](https://docs.cfengine.com/docs/3.24/reference-components-cf-promises.html#top) can be very useful for reviewing the first order variables that are defined using the `--show-vars` option. ```command cf-promises --show-vars=sys\.interfaces ``` ```output Variable name Variable value Meta tags Comment default:sys.interfaces {"enx0892048803e7","wlp0s20f3","virbr0","docker0","br-a7d465b9949b","vboxnet0","tun0"} inventory,source=agent,attribute_name=Interfaces default:sys.interfaces_data {"br-a7d465b9949b":{"device":"br-a7d465b9949b","receive_bytes":"0","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"0","transmit_bytes":"0","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"0"},"docker0":{"device":"docker0","receive_bytes":"0","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"0","transmit_bytes":"0","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"0"},"enx0892048803e7":{"device":"enx0892048803e7","receive_bytes":"6362565862","receive_compressed":"0","receive_drop":"10728","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"8119215","transmit_bytes":"2704360457","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"5210158"},"enx5cff35c6864b":{"device":"enx5cff35c6864b","receive_bytes":"0","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"0","transmit_bytes":"0","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"0"},"lo":{"device":"lo","receive_bytes":"30236494","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"331169","transmit_bytes":"30236494","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"331169"},"tun0":{"device":"tun0","receive_bytes":"1224387","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"6628","transmit_bytes":"1478565","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"8042"},"vboxnet0":{"device":"vboxnet0","receive_bytes":"0","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"0","transmit_bytes":"2473935","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"3790"},"virbr0":{"device":"virbr0","receive_bytes":"0","receive_compressed":"0","receive_drop":"0","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"0","transmit_bytes":"0","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"0"},"wlp0s20f3":{"device":"wlp0s20f3","receive_bytes":"29822471","receive_compressed":"0","receive_drop":"9015","receive_errors":"0","receive_fifo":"0","receive_frame":"0","receive_multicast":"0","receive_packets":"150090","transmit_bytes":"3287961","transmit_compressed":"0","transmit_drop":"0","transmit_errors":"0","transmit_fifo":"0","transmit_frame":"0","transmit_multicast":"0","transmit_packets":"9598"}} networking,/proc,source=agent,procfs ``` [cf-agent](https://docs.cfengine.com/docs/3.24/reference-components-cf-agent.html#top) can show the state of variables defined at the end of execution using the `--show-evaluated-vars` option. ### Matching an IP address with a regular expression {#matching-an-ip-address-with-a-regular-expression} Why was the regular expression `172\.16\.1\.*` **not** matching `172.16.1.10`? ```cfengine3 {file="/tmp/regex-ip.cf"} bundle agent __main__ { vars: "ipregex" string => "172\.16\.1\.*"; classes: "found" expression => regcmp("$(ipregex)","172.16.1.10"); reports: found:: "Found the ip"; !found:: "Did not find the ip"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/regex-ip.cf ``` ```output R: Did not find the ip ``` Because there was a bug in the regex, a missing `.` after the escaped one: ```cfengine3 {file="/tmp/regex-ip-fixed.cf"} bundle agent __main__ { vars: "ipregex" string => "172\.16\.1\..*"; classes: "found" expression => regcmp("$(ipregex)","172.16.1.10"); reports: found:: "Found the ip"; !found:: "Did not find the ip"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/regex-ip-fixed.cf ``` ```output R: Found the ip ``` Use the builtin functions for [iprange()](https://docs.cfengine.com/docs/3.24/reference-functions-iprange.html) and [isipinsubnet()](https://docs.cfengine.com/docs/3.24/reference-functions-isipinsubnet.html) for improved readability. #### Checking IP validity {#checking-ip-validity} Regular expressions can still be useful for validating input from external data sources to avoid application mis-configuration. For example, [every()](https://docs.cfengine.com/docs/3.24/reference-functions-every.html) can be used to validate a whole list of ip addresses. Remember, regular expressions can be multi-line which can improve readability. ```cfengine3 {file="/tmp/ipv4-validity.cf"} bundle agent __main__ { vars: "ip_list1" slist => { "192.168.56.2", "1.1.1.1" }; "ip_list2" slist => { "2.1.1.356", "192.168.1.1" }; "ipv4_regex" # https://watson-wilson.ca/blog/2015/08/20/build-better-regular-expressions-in-cfengine/ string => "(?xms) # First octet (?: 25[0-5] # Remember ips go as high as 255. | # or 2[0-4][0-9] # Between 200 and 249. | # or [0-1]?[0-9]{1,2} # Less than 200. ) [.] # Decimal point # Second octet (?: 25[0-5] # Remember ips go as high as 255. | # or 2[0-4][0-9] # Between 200 and 249. | # or [0-1]?[0-9]{1,2} # Less than 200. ) [.] # Decimal point # Third octet (?: 25[0-5] # Remember ips go as high as 255. | # or 2[0-4][0-9] # Between 200 and 249. | # or [0-1]?[0-9]{1,2} # Less than 200. ) [.] # Decimal point # Forth octet (?: 25[0-5] # Remember ips go as high as 255. | # or 2[0-4][0-9] # Between 200 and 249. | # or [0-1]?[0-9]{1,2} # Less than 200. ) "; classes: "ip_list1_all_valid_ipv4" expression => every( "$(ipv4_regex)", @(ip_list1) ); "ip_list2_all_valid_ipv4" expression => every( "$(ipv4_regex)", @(ip_list2) ); reports: ip_list1_all_valid_ipv4:: "All elements of ip_list1 are valid IPv4 addresses"; ip_list2_all_valid_ipv4:: "All elements of ip_list2 are valid IPv4 addresses"; !ip_list1_all_valid_ipv4:: "Not all elements of ip_list1 are valid IPv4 addresses"; !ip_list2_all_valid_ipv4:: "Not all elements of ip_list2 are valid IPv4 addresses"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/ipv4-validity.cf ``` ```output R: All elements of ip_list1 are valid IPv4 addresses R: Not all elements of ip_list2 are valid IPv4 addresses ``` ##### Validating IPv4 addresses with isipinsubnet() {#checking-ip-validity-isipinsubnet} You can use `isipinsubnet()` to check that an IP address is valid (Nick was trying to use `iprange()` incorrectly for this during the recording): ```cfengine3 {file="/tmp/validate-ip-isipinsubnet.cf"} bundle agent __main__ { vars: "ip" string => "172.16.1.10"; "range" string => "0.0.0.0/0"; classes: "valid_ipv4" expression => isipinsubnet( "$(range)", "$(ip)" ); reports: valid_ipv4:: "The IP $(ip) is part of the range $(range)"; !valid_ipv4:: "The IP $(ip) is NOT part of the range $(range)"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/validate-ip-isipinsubnet.cf ``` ```output : R: The IP 172.16.1.10 is part of the range 0.0.0.0/0 ``` ### Limiting messages to the system log {#limiting-messages-to-the-system-log} [system_log_level in body common control control](https://docs.cfengine.com/docs/3.24/reference-components.html#system_log_level) controls the _minimum_ log level for messages to be sent to the system log. Newly exposed (master/3.25.0) to be [configurable via Augments](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework.html#configure-the-minimum-log-level-for-system-log). ```json { "variables": { "default:def.control_common_system_log_level": { "value": "info", "comment": "We want syslog to recieve messages tha are level info and above." } } } ``` ### Is it possible to increase log verbosity for bundles called via usebundle? {#is-it-possible-to-increase-log-verbosity-for-bundles-called-via-usebundle} Yes, it's possible to increase the verbosity of specific promises, but not for all promises in a specific bundle. [log_level in action body](https://docs.cfengine.com/docs/3.24/reference-promise-types.html#log_level) allows you to increase the verbosity of a specific promise, but it does not apply to all promises of a bundle at once (nor would it be inherited by subsequent methods promises). ### Is it possible to change the order of promises {#is-it-possible-to-change-the-order-of-promises} [Normal ordering](https://docs.cfengine.com/docs/3.24/reference-language-concepts-normal-ordering.html#top) (`vars` before `classes`, `files` before `packages`) can not be changed but promise execution can be influenced using classes and/or more weakly with the `depends_on` attribute. ```cfengine3 {file="/tmp/depends_on.cf"} bundle agent __main__ { reports: "one" handle => "one", depends_on => { "two"}; "two" handle => "two"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/depends_on.cf ``` ```output R: two R: one ``` Maybe someday, an option to resolve using written order vs normal order. ## Video {#video} The video recording is available on YouTube: {{< youtube jjn6h5OxnGU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Feature Friday #37: Decisions based on arbitrary semantic versions" url: /blog/2024/feature-friday-37-decisions-based-on-arbitrary-semantic-versions/index.md date: "2024-11-22" -------------------------------------------------------------------------------- Ever need to make a decision based on the version of something? The `version_compare()` function might be useful for you.[^fn:1] Over time, software changes and features are added and removed. Sometimes, we need to make a decision based on versions. For example, the `Include` directive in `ssh_config` was introduced in OpenSSH 7.3.[^fn:2] Let's take a look at how we could possibly use it. This example illustrates the basic use of `version_compare()`: ```cfengine3 {file="/tmp/feature-friday-37-0.cf"} bundle agent __main__ { vars: "version_1" string => "7.3.0"; "version_2" string => "7.2.0"; "comparisons" slist => { "=", "==", "!=", ">", "<", ">=", "<=" }; reports: "$(version_1) is $(comparisons) $(version_2)" if => version_compare( "$(version_1)", "$(comparisons)", "$(version_2)" ); } ``` The example above compares `version_1` against `version_2` using each of the supported comparison operators, and it emits a report when the comparison evaluates to true. Running the example, we see the expected reports: ```command cf-agent -Kf /tmp/feature-friday-37-0.cf ``` ```output R: 7.3.0 is != 7.2.0 R: 7.3.0 is > 7.2.0 R: 7.3.0 is >= 7.2.0 ``` For something a bit more realistic, let's extend the example to test against the currently installed version of `openssh-client`: ```cfengine3 {file="/tmp/feature-friday-37-1.cf"} bundle agent __main__ { vars: "installed_version_data" data => packagesmatching("openssh-client", ".*", ".*", ".*"); "ssh_conf[HashKnownHosts]" string => "yes"; "ssh_conf[Include]" string => "/etc/ssh/ssh_config.d/*.conf", if => version_compare( # Debian version numbers may contain a leading epoch:string that is unrelated to the upstream package version # Here, if it exists we strip it. regex_replace( "$(installed_version_data[0][version])","^\d+:(.*)", "$1", "" ), ">=", "7.3.0" ); reports: "ssh_conf for openssh-client '$(installed_version_data[0][version])' should include: $(with)" with => join( ", ", getindices( ssh_conf ) ); } ``` Remember that `version_compare()` compares semantic versions. But not all versions you encounter are semantic. For example, Debian package versions _may_ include a leading epoch followed by a colon, resulting in a version that looks like this `1:8.9p1-3ubuntu0.6`. In that case, you must extract the "real" version you want to compare, which we did in the example using `regex_replace()`. Here `ssh-conf[Include]` is defined as `/etc/ssh/ssh_config.d/*.conf` if the version of the installed package matching `openssh-client` with the epoch stripped if it's present is greater than or equal to `7.3.0`. ```cfengine3 "ssh_conf[Include]" string => "/etc/ssh/ssh_config.d/*.conf", if => version_compare( # Debian version numbers may contain a leading epoch: string that is unrelated to the upstream package version # Here, if it exists we strip it. regex_replace( "$(installed_version_data[0][version])", "^\d+:(.*)", "$1", "" ), ">=", "7.3.0" ); ``` Running the policy we can see that configuration for both `HashKnownHosts` and `Include` should be included: ```command cf-agent -Kf /tmp/feature-friday-37-1.cf ``` ```output R: ssh_conf for openssh-client '1:8.9p1-3ubuntu0.6' should include: HashKnownHosts, Include ``` If you want to play around with this you can simply define a data container matching the results of the `packagesmatching()` function call you expect. For example, here we can see the result if the version of the package installed was `6.9p1-3ubuntu0.6`: ```cfengine3 {file="/tmp/feature-friday-37-2.cf"} bundle agent __main__ { vars: # "installed_version_data" data => packagesmatching( "openssh-client", # ".*", ".*", ".*" ); "installed_version_data" data => '[ { "arch": "amd64", "method": "apt_get", "name": "openssh-client", "version": "6.9p1-3ubuntu0.6" } ]'; "ssh_conf[HashKnownHosts]" string => "yes"; "ssh_conf[Include]" string => "/etc/ssh/ssh_config.d/*.conf", if => version_compare( # Debian version numbers may contain a leading epoch: string that is unrelated to the upstream package version # Here, if it exists we strip it. regex_replace( "$(installed_version_data[0][version])", "^\d+:(.*)", "$1", "" ), ">=", "7.3.0" ); reports: "ssh_conf for openssh-client '$(installed_version_data[0][version])' should include: $(with)" with => join( ", ", getindices( ssh_conf ) ); } ``` Executing the policy we can see that only configuration for `HashKnownHosts` should be included since `6.9p1-3ubuntu0.6` is not greater than or equal to `7.3.0`. ```command cf-agent -Kf /tmp/feature-friday-37-2.cf ``` ```output R: ssh_conf for openssh-client '6.9p1-3ubuntu0.6' should include: HashKnownHosts ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: The `version_compare()` function was introduced in CFEngine 3.23.0, find the documentation for it here: [^fn:2]: The OpenSSH 7.3 release notes indicate addition of `Include` directive for `ssh_config` files: -------------------------------------------------------------------------------- title: "Feature Friday #36: Formatting policy with cffmt" url: /blog/2024/feature-friday-36-formatting-policy-with-cffmt/index.md date: "2024-11-15" -------------------------------------------------------------------------------- Looking for a tool to help you format your CFEngine policy? Have you heard of `cffmt`? You might recall that we had a chat with the author, Miek Gieben in [The agent is in - Episode 24](/blog/2023/the-agent-is-in-episode-24/).[^fn:1] In case you missed it, `cffmt` is a command line tool for formatting CFEngine policy files, like `gofmt` for `.cf` files.[^fn:2] Let's take a look. Consistent formatting can really ease reading of policy, but sometimes editors are configured differently and you can end up with inconsistently formatted policy. For example, here is a contrived policy file with some irregular formatting: ```cfengine3 bundle agent __main__ { vars: "String1" string => "Hello String1"; "list" slist => { "one", "two", "three", "four", "five" }; files: "/tmp/hello-world.txt" content => "$(String1)", perms => mog( "600", "root", "root"), comment => "Important stuff"; classes: "Some_class" expression => "any"; reports: Some_class:: "Some_class is defined."; @if minium_version(3.21) any:: "Running 3.21 or later $(sys.cf_version)"; @endif } ``` Let's use `cffmt` to clean it up. First, we need to install `cffmt` which we can do using `go install`: ```sh go install github.com/miekg/cf/cmd/cffmt@main ``` ```text go: downloading github.com/miekg/cf v0.1.25 go: downloading github.com/shivamMg/ppds v0.0.1 go: downloading github.com/alecthomas/chroma/v2 v2.12.0 go: downloading github.com/dlclark/regexp2 v1.10.0 ``` Once installed you should find `cffmt` in `~/go/bin/`. Simply run `cffmt` with the path to the policy file as an argument. ```sh ~/go/bin/cffmt /tmp/feature-friday-36-0.cf ``` We can see it emits a much more nicely formatted policy file: ```cfengine3 bundle agent __main__ { vars: "String1" string => "Hello String1"; "list" slist => { "one", "two", "three", "four", "five" }; files: "/tmp/hello-world.txt" content => "$(String1)", perms => mog("600", "root", "root"), comment => "Important stuff"; classes: "Some_class" expression => "any"; reports: Some_class:: "Some_class is defined."; @if minium_version(3.21) any:: "Running 3.21 or later $(sys.cf_version)"; @endif } ``` While `cffmt` is quite opinionated, it does provide the capability to skip formatting a file if there is a top-level comment `# cffmt:no` which is useful if you are auto-formatting many policy files. You can also instruct it to format `slist` vars with an entry on each line by placing a `# cffmt:list` comment preceding the definition. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: Find Miek Gieben on [Mastodon](https://mastodon.cloud/@miek), [LinkedIn](https://www.linkedin.com/in/miek-gieben-932aaa20a/), and [GitHub](https://github.com/miekg). [^fn:2]: Find documentation and source for `cffmt` on Github: -------------------------------------------------------------------------------- title: "Feature Friday #35: Groups in Mission Portal" url: /blog/2024/feature-friday-35-groups-in-mission-portal/index.md date: "2024-11-08" -------------------------------------------------------------------------------- Have you seen the new Groups feature in CFEngine Enterprise Mission Portal? It was first released in 3.23.0 and it's part of the 3.24 LTS series released earlier this year, let's check it out. {{< figure src="2024-03-15_14-54-05_Screenshot_2024-03-15_140131.png" >}} Groups in Mission Portal can be based on any host reported data. They can be dynamic (hosts can come and go from a group) or they can be static and tied to specific hosts by hostname, mac address, IP or CFEngine's public key. By default groups are not shared with other users and they can be used any place where you would use a filter constraining the scope of hosts based on various conditions related to the hosts reported data. If a group is shared it can have _data_ associated with it. For example, here we adjust the execution schedule for all hosts to be on a 10 minute interval with a 9 minute splay time. {{< figure src="2024-03-15_15-18-14_Screenshot_2024-03-15_151631.png" >}} Shared group data is merged based on a group's priority along with any host specific data that has been applied and ultimately delivered to the host via `host_specific.json`.[^fn:1] Looking at `host_specific.json` on a host we see the data applied above has been delivered to the host. ```sh jq --monochrome-output < /var/cfengine/data/host_specific.json ``` ```json { "classes": {}, "variables": { "default:def.control_executor_schedule": { "comment": "", "name": "default:def.control_executor_schedule", "tags": [ "report" ], "type": "Array", "value": [ "Min00", "Min10", "Min20", "Min30", "Min40", "Min50" ] }, "default:def.control_executor_splaytime": { "comment": "", "name": "default:def.control_executor_splaytime", "tags": [ "report" ], "type": "String", "value": "9" } } } ``` We can apply host specific data to the same host to return the schedule to a 5 minute interval: {{< figure src="2024-03-15_15-31-51_Screenshot_2024-03-15_153147.png" >}} And (after some time) we can see how it took effect: ```sh jq --monochrome-output < /var/cfengine/data/host_specific.json ``` ```json { "classes": {}, "variables": { "default:def.control_executor_schedule": { "comment": "", "name": "default:def.control_executor_schedule", "tags": [ "report" ], "type": "Array", "value": [ "Min00", "Min10", "Min20", "Min30", "Min40", "Min50", "Min00", "Min05", "Min10", "Min15", "Min20", "Min25", "Min30", "Min35", "Min40", "Min45", "Min50", "Min55" ] }, "default:def.control_executor_splaytime": { "comment": "", "name": "default:def.control_executor_splaytime", "tags": [ "report" ], "type": "String", "value": "9" } } } ``` Beware the results of a merge between groups. Lists and arrays are extended, keys in data are overridden by a group of higher priority. Here we have defined a data container and a list for the Linux group: {{< figure src="2024-03-15_15-43-59_Screenshot_2024-03-15_154315.png" >}} Defining similar data for the RHEL group (which has higher priority than Linux): {{< figure src="2024-03-15_15-46-49_Screenshot_2024-03-15_154610.png" >}} Looking into the resulting `host_specific.json` we can see that lists/arrays are extended while strings and objects are overwritten by higher priority groups. ```sh jq --monochrome-output '.variables.data.value, .variables.list.value' < /var/cfengine/data/host_specific.json ``` ```json { "array1": [ "1", "2", "3", "one", "two", "three" ], "key1": "RHEL", "key2": "Also RHEL" } [ "1", "2", "3", "one", "two", "three" ] ``` I hope you find the new groups in Mission Portal useful. Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: We talked about `host_specific.json` in [Feature Friday #24: Augments - host_specific.json](/blog/2024/feature-friday-24-augments-host-specific-json). -------------------------------------------------------------------------------- title: "How to use a different ports for Mission Portal" url: /blog/2024/use-a-different-port-for-mission-portal/index.md date: "2024-11-07" -------------------------------------------------------------------------------- Out of the box, Mission Portal uses port 80 for HTTP and 443 for HTTPS traffic. These ports are widely used for web services, which makes the initial setup straightforward. However, there are several reasons why you might want to change these default ports: * Port conflicts: other services on your system might already be using ports 80 or 443. * Security concerns: well-known ports are common targets for attacks. * Compliance requirements: your organization's policies might restrict the use of standard web ports. ## Changing default ports **_Note:_** The custom port configuration feature described below is available in CFEngine Enterprise master (nightlies), 3.25.0 (to be released this winter) or later. ### From the Mission Portal First, you need to open a Hub info page. To do this, you can click on the Hub menu in the top right corner menu. ![open_hub_info_page.png](images/open_hub_info_page.png) Once you're on the Hub info page, scroll down until you see the "Host specific data" section. This is where the magic happens! ![cmdb_ui.png](images/cmdb_ui.png) Now, let's add those custom port settings: * For HTTPS: * Click the "Add variable" button * In the name field enter `default:cfe_internal_hub_vars.https_port` * In the value field, enter your desired HTTPS port number.   * For HTTP: * Repeat the process, but use `default:cfe_internal_hub_vars.http_port` as the variable name * Enter your desired HTTP port number as the value ![configured_ports_in_cmdb_ui.png](images/configured_ports_in_cmdb_ui.png) To apply changes, you can trigger an agent run in the top right corner or just wait until it happens automatically. By default, the agent run interval is 5 minutes. ![trigger_an_agent_run.png](images/trigger_an_agent_run.png) 🎉 And finally, try accessing Mission Portal using the new port in your browser: `https://cfengine.hub:8443` (Replace 8443 with your actual HTTPS port number if it's different) ### From CLI Define needed variables in the augment file `def.json` at the root of your policy set. ```json { "variables": { "default:cfe_internal_hub_vars.http_port": { "value": "8080", "comment": "Alternate port for Mission Portal HTTP traffic." }, "default:cfe_internal_hub_vars.https_port": { "value": "8443", "comment": "Alternate port for Mission Portal HTTPs traffic." } } } ``` More about augment files you can read [here](https://docs.cfengine.com/docs/master/reference-language-concepts-augments.html) Run `cf-agent -KIf update.cf && cf-agent -KI` to apply the new configuration. **_Note:_** If you have configured [Federated reporting](https://docs.cfengine.com/docs/master/web-ui-federated-reporting.html) and have changed the port number for the feeder, remember to update the feeder's URL accordingly. Otherwise, it will result in incorrect or failed reporting, as the system will still try to connect using the old port. -------------------------------------------------------------------------------- title: "Git promise type for private repositories" url: /blog/2024/promise-type-git-with-credentials/index.md date: "2024-11-04" -------------------------------------------------------------------------------- Over three years ago we introduced [git and systemd custom promise types](http://localhost:1313/blog/2021/git-and-systemd-promises/). While these are quite functional I recently needed to manage a git repository that was private and so needed a way to authenticate to a git server, in this case gitlab. Gitlab has [project access tokens](https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html) but we didn't directly support any sort of authentication attributes on the promise type. I ran across [git credential](https://git-scm.com/docs/git-credential) as a way to provide authentication and thought it was a good fit. I developed some policy which kept the token secure via [cf-secret](https://docs.cfengine.com/docs/master/reference-components-cf-secret.html) and setup the needed dot files in `/root` to setup git credentials. ```cf3 bundle agent manage_private_repo { vars: "source_directory" string => "/var/cfengine/masterfiles/files/buildhost"; "keys" slist => { "username", "password" }; "secret_path[${keys}]" string => "${sys.statedir}/private-repo-deploy-token-${keys}"; "credentials_final" string => "/root/.git-credentials"; "credentials_staged" string => "${credentials_final}.staged"; packages: linux:: "git"; git: credentials_ok:: "private-repo" repository => "https://gitlab.com/organization/private-repo.git", version => "master", destination => "/home/user/private-repo"; classes: "git_credentials_exists" expression => fileexists("${credentials_final}"); "username_newer_than_credentials" expression => isnewerthan("${secret_path[username]}", "${credentials_final}"); "password_newer_than_credentials" expression => isnewerthan("${secret_path[password]}", "${credentials_final}"); "update_git_stuff" expression => or(not(git_credentials_exists), username_newer_than_credentials, password_newer_than_credentials); commands: "{ echo -n 'https://'; cf-secret decrypt ${secret_path[username]} -o - | tr -d '\n' | sed 's/+/%2b/'; echo -n ':'; cf-secret decrypt ${secret_path[password]} -o - | tr -d '\n'; echo '@gitlab.com'; } > ${credentials_staged}" contain => in_shell, handle => "staged_credentials_rendered"; files: "${credentials_final}" copy_from => local_dcp("${credentials_staged}"), depends_on => { "staged_credentials_rendered" }, classes => scoped_classes_generic("bundle", "credentials"); "${sys.statedir}/private-repo-deploy-token-${keys}" copy_from => remote_dcp("${source_directory}/user-vms-deploy-token-${keys}", "${sys.policy_hub}"); "/root/.gitconfig" content => " [credential] helper = store [safe] directory = /home/user/private-repo "; "/home/user/private-repo/." depth_search => recurse_with_base(inf), perms => owner("user"); reports: DEBUG:: "secret_paths are ${secret_path[username]} and ${secret_path[password]}"; DEBUG.git_credentials_exists:: "git_credentials_exists is defined"; DEBUG.username_newer_than_credentials:: "username_newer_than_credentials"; DEBUG.password_newer_than_credentials:: "password_newer_than_credentials"; } ``` I can encrypt to one or more hosts with the `cf-secret` command and save the encrypted files in my policy set. Check out [more in this series](/tags/monthly-module-monday/) of monthly module blogs and consider what you might contribute! ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #34: Self organizing groups with select_class" url: /blog/2024/feature-friday-34-self-organizing-groups-with-select-class/index.md date: "2024-11-01" -------------------------------------------------------------------------------- Did you know CFEngine can self-organize hosts into different groups? Say you have a few hosts that you want to reboot once a month. You don't care when, but you want the hosts to self-organize and pick a date. The `select_class` attribute for `classes` type promises might be what you're looking for. Let's take a look. We'll keep things simple, so we want each host to self-select a day of the month (1-28). ```cfengine3 {file="/tmp/feature-friday.cf"} bundle agent reboot_day { vars: # one option for each of 28 days in a month. "options" slist => { expandrange( "reboot_day_[1-28]", 1 ) }; selected_option:: "selected" string => "$(with)", if => "$(options)", with => regex_replace( "$(options)", "reboot_day_(.*)", "$1", "" ), meta => { "inventory", "attribute_name=Reboot Day" }; classes: "selected_option" select_class => { @(options) }; reports: selected_option:: "I selected option $(selected)"; } bundle agent __main__ { methods: "reboot_day"; } ``` In the above policy, we have the `options` string list, a list of 28 elements, `reboot_day_1` through `reboot_day_28`. This list of options is fed to the `select_class` attribute of our `selected_option` classes promise.[^fn:1] When a choice is made both `selected_class` and `reboot_day_N` classes will be defined. We use the knowledge of this selection and `regex_replace()` to set the `selected` variable to the selected day of the month. The variable is tagged for inventory, for greater visibility from Mission Portal in CFEngine Enterprise.[^fn:2] Running this on a host you will find that the same selection is always made: ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday.cf ``` ```output R: I selected option 15 ``` Through Mission Portal, it's easy to see the distribution of the selected day: {{< figure src="inventory-select_class-reboot-day.png" >}} Looking at a Pie chart we can see a fairly even distribution: {{< figure src="inventory-select_class-reboot-day-pie-chart.png" >}} Hosts will continue to select the same day as long as the options to select from remain consistent (in order and number of options). If we adjust the policy to select a day in the first 5 days of the month: ```cfengine3 {file="/tmp/feature-friday.cf"} bundle agent reboot_day { vars: # one option for each of the first 5 days in a month. "options" slist => { expandrange( "reboot_day_[1-5]", 1 ) }; selected_option:: "selected" string => "$(with)", if => "$(options)", with => regex_replace( "$(options)", "reboot_day_(.*)", "$1", "" ), meta => { "inventory", "attribute_name=Reboot Day" }; classes: "selected_option" select_class => { @(options) }; reports: selected_option:: "I selected option $(selected)"; } bundle agent __main__ { methods: "reboot_day"; } ``` We see hosts make new selections: {{< figure src="inventory-select_class-reboot-day-5-table.png" >}} Not an absolutely even distribution, but still reasonable: {{< figure src="inventory-select_class-reboot-day-5-pie-chart.png" >}} Go forth and let your groups select themselves! Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: Documentation for `select_class`: [^fn:2]: Documentation for `regex_replace()`: -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 42 - Fireside Chat w/ Bas Van der Vlies" url: /blog/2024/the-agent-is-in-episode-42/index.md date: "2024-10-31" -------------------------------------------------------------------------------- The prolific contributor and CFEngine Champion Bas Van der Vlies joins the team for a chat about his history and experience with CFEngine dating back to version 1.0.3. For this special Halloween edition Bas joined Cody, Craig and Nick to reflect on his journey getting into information technology beginning with chemical engineering through joining the Dutch National Compute Center where he was managing many SGI and AIX machines and where he discovered CFEngine and how it could help him manage all the machines. Bas went on to highlight some of his favorite features and discussed how CFEngine has evolved over the years. His advice for both new and old users is to check out some of the newer tooling like `cf-remote` and `cfbs` for the CFEngine Build system as they can dramatically help to simplify policy maintenance. ## Video {#video} The video recording is available on YouTube: {{< youtube lCxg5Ynm-V0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Bas](https://www.linkedin.com/in/basvandervlies/) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [CFEngine blog posts from Bas](https://cfengine.com/author/bas-van-der-vlies/) - [CFEngine build module for SURFsara CFEngine Library](https://build.cfengine.com/modules/surf-cfengine-library/) - [CFEngine build module for Docker Compose](https://build.cfengine.com/modules/promise-type-docker-compose/) - [Evolve Thinking CFEngine Library (EFL)](https://github.com/neilhwatson/evolve_cfengine_freelib) - [NCF](https://github.com/Normation/ncf) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Feature Friday #33: Why associative arrays when data containers exist?" url: /blog/2024/feature-friday-33-why-associative-arrays-when-data-containers-exist/index.md date: "2024-10-25" -------------------------------------------------------------------------------- What's the difference between an associative array and a data container in CFEngine? CFEngine has two ways in which structured data can be used, associative arrays (sometimes called classic arrays) and data containers. Let's take a look at a simple data structure. Here we have two data structures, `a_email` an associative array and `d_email` a data container. The policy emits the JSON representation of each. ```cfengine3 bundle agent __main__ { vars: "a_email[john@example.com][FirstName]" string => "John"; "a_email[john@example.com][LastName]" string => "Doe"; "d_email" data => '{ "john@example.com": { "FirstName": "John", "LastName": "Doe" } }'; reports: "JSON representation of a_email (associateve array):$(const.n)$(with)" with => storejson( a_email ); "JSON representation of d_email (data container):$(const.n)$(with)" with => storejson( d_email ); } ``` Looking at the output, they are identical: ```text R: JSON representation of a_email (associateve array): { "john@example.com": { "FirstName": "John", "LastName": "Doe" } } R: JSON representation of d_email (data container): { "john@example.com": { "FirstName": "John", "LastName": "Doe" } } ``` While their JSON representations are identical, they actually differ in a significant way. Notice that the associative array is defined key value pair by key value pair: ```cfengine3 vars: "a_email[john@example.com][FirstName]" string => "John"; "a_email[john@example.com][LastName]" string => "Doe"; ``` Indeed, associative arrays in CFEngine are a collection of individual variables that are associated by their name, while a data container is an entire single unit. ```cfengine3 "d_email" data => '{ "john@example.com": { "FirstName": "John", "LastName": "Doe" } }'; ``` Both can be used and referenced similarly, but associative arrays being a collection of individual variables can be constructed dynamically. This can be a very useful capability, especially combined with CFEngine's implicit list iteration. Say we want to have a data structure representing files and we want to have a key with the files mtime, something like this: ```json { "/tmp/file-1.txt": { "mtime": "1234" }, "/tmp/file-2.txt": { "mtime": "4321" } } ``` This is trivial to achieve leveraging an associative array: ```cfengine3 bundle agent __main__ { vars: "files" slist => { "/tmp/file-1.txt", "/tmp/file-2.txt" }; "file[$(files)][mtime]" string => filestat( "$(files)", "mtime" ); reports: "$(with)" with => storejson( file ); } ``` ```text R: { "/tmp/file-1.txt": { "mtime": "1709940438" }, "/tmp/file-2.txt": { "mtime": "1709940820" } } ``` But that is much more challenging to do using data containers since the entire data container must be defined at once. Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Benefits of agent-based configuration management" url: /blog/2024/benefits-of-agents/index.md date: "2024-10-21" -------------------------------------------------------------------------------- Within the configuration management space, people often distinguish between _agent-based_ and _agent-less_ approaches. In short, an agent-based solution means that you install a software agent to run in the background / periodically on the system. That software agent then makes changes to the system as desired, and also commonly communicates over the network to send and receive updates, policy, commands, scripts, data, etc. On the other hand, an agentless system does _not involve installing something new_, they instead rely on some software which is (presumed) already installed, like the SSH server, which can be used to access and make changes to the system. There are also more hybrid approaches (e.g. a system which involves using SSH in some cases, and installing an agent in other cases). In the real world, in large and complex infrastructures, organizations typically rely on [a combination of tools](https://cfengine.com/blog/2021/comparing-ansible-and-cfengine/) for different purposes. Nevertheless, in this post, let's take a look at the benefits of agent-based configuration management solutions, such as CFEngine. ## Bandwidth - Limiting the amount of data transferred When the software, data, and policy are stored on each host, we can track changes to them and be efficient to minimize bandwidth consumption. When there is new policy, we transfer it, but only the files / parts needed, not the whole directory. The same applies to data, both to and from the host. Most of the time, for most files and data, there is no change, and thus no reason to re-transmit the data. By default, CFEngine reports useful information about a host to the hub, things like OS, CFEngine version, IP address, MAC address etc. In total there are around 200 of these values (variables, classes, inventory attributes) on a newly set up (stock) system, and users add more through the modules they use and policy they write. The difference between transferring just a few of these values, occasionally, when they change, compared to all of them every single time, is significant. ## Decoupling - Doing different tasks at different frequencies When you want to make changes to a remote system you need 3 things to happen: 1. **Update:** Transfer your desire to the remote system (can be a bash command, policy file, python script, JSON data, or something else) 2. **Enforce:** Have that remote system actually enforce your desired state (run a command, evaluate policy, delete a file, etc.) 3. **Report:** Transfer some information back to you about the result, whether it was successful, what was changed, etc. {{< wideimg "images/define-enforce-verify.png" "3 steps above visualized in infographic">}} These 3 things do not have to happen with the same frequency, and at scale this matters. When and how often you run these things depends on your requirements, how busy each host is, and how much network bandwidth you have available. Let's take an example: - You only make changes to policy a couple of times per day. - **Updating** could happen once per hour. CFEngine would spread your host updates out evenly throughout the hour. For 10 000 hosts that's still an average of around 167 hosts updating every minute. This means that you'd have several hosts updating immediately after your change, and all of them would be done after 1 hour. - Your policy is enforcing important security-related configuration like file permissions and local user accounts: - Since **enforcing** does not rely on the network, it can happen much more frequently without consuming excessive bandwidth. By default, CFEngine runs things every 5 minutes, if your policy is small / fast, you could turn it down to 1 minute, if your policy is large or involves slower operations, you can increase it to for example 10 or 15 minutes. - You want to have as fresh reporting information as possible, but the network / bandwidth is the bottleneck. - Sending **reporting** data every 10 minutes could be a reasonable schedule. For the case of 10 000 hosts, this averages out to around 17 hosts sending reporting data per second. The conditions for when to do these steps do not have to be simple intervals as above. If desirable, you can set different intervals based on time of day, day of the week, month, or in general the state of the system. Updates, or parts of policy enforcement, could also be postponed until a condition is met. ## Robustness - Autonomy and decentralization CFEngine was designed to be decentralized. Each host runs an agent, makes decisions, can serve and fetch policy and data from other hosts (not just the hub), if desirable. However, even in CFEngine, some things are typically centralized; the policy server and reporting database typically reside on one host called _the hub_. To mitigate the single point of failure scenario when the hub goes offline, one can set up alternative hubs, for example using the High Availability setup (HA). But even if all your hubs go offline (or your only hub), the agents on each host are _still_ autonomous and decentralized - they will keep enforcing the policy they have, making changes according to your intentions, even if there is no central hub to tell them what to do and when to do it. Relating to the example above - Step 2 will keep working, even if step 1 and 3 are not currently possible. {{< wideimg "images/autonomy.png" "Visualizations of 5 decentralized agents / servers which keep working even if the Hub in the center is offline.">}} If you have policy to uninstall unwanted packages like telnet, to limit who has superuser / sudo privileges, and to rotate log files when they grow to a certain size, those things will keep happening even if your centralized hub or network is having issues. Once the issues are resolved, the hosts will seamlessly start communicating with the hub again, and you get reporting data about what has happened on those hosts. ## Consistency - Preventing configuration drift Agentless systems allow you to quickly deploy a change or piece of software. They're often used in this way: > Deploy this software to all my hosts now. However, that sentence doesn't really describe what should happen in the future: - What about hosts which are currently unavailable / offline? - Should there be some retries? - How many retries? - What if they're offline for a long time (hours, days, weeks)? - What about hosts which don't exist yet, but will be spawned in the future? - Your automation should probably be a part of some provisioning step? - What about when a user / admin comes along and uninstalls the software? - Maybe run your automation as often as possible? (as bandwidth / resources allow) If the software relates to your security or reliability, it's important to consider these. It'll be appropriate to set up some periodic jobs, which run the automation, at some schedule, with retries. At that point, scalability gets more important, you want a configuration management system which is able to enforce a lot of _rules_ (we call them promises), frequently, on a lot of hosts. Additionally, everyone who uses the configuration management system and automates things with it, will need to take into account the considerations above and ensure their desired state is consistently enforced. We've seen that all of this poses a challenge for agentless systems in the real world [^1]. In practice, they're sometimes used in a deploy once / fire and forget way, and there is a problem with _configuration drift_ - outliers and subsets of the infrastructure are missing certain files / software, or are otherwise not configured correctly. In CFEngine, the default approach is to add the things you want into your policy set, which describes your desired state for the entire infrastructure (and each host in it). As mentioned above, CFEngine, on all the hosts where it is installed, will periodically pull and enforce this policy. This mitigates a lot of problems - offline hosts will fetch and enforce the policy when they come online, same for hosts spawned in the future, and all of your rules are enforced continuously (until you remove them). ## Flexibility & security - Programmable decisions at the edge An agent-based solution like CFEngine allows you to do distributed programming and make decentralized decisions in a very real and practical way. In many cases there is no need, or you don't want to wait for a centralized server to give you directions about what to do. Here are some useful ways to leverage this, based on stories from CFEngine users: **Cowboy mode:** When administrators log into a machine and want to make manual changes, they've written policy so that they can temporarily tell CFEngine to "back off". They can create a file running `touch /var/cfengine/cowboy`, and CFEngine goes into "warnings-only" mode. If the admin forgets the file, there is a report to show hosts which have this mode enabled, and a timeout causing the file to be cleared if it's been enabled for too long. **Flag files:** Similarly, you can make your policy react to users creating other flag files, for example to increase logging, to flag the host as problematic in an automated report, to temporarily allow a support team to SSH into the machine, or anything else you'd like to enable local admins to do. **Peer-to-peer distribution of files:** Hosts can communicate with each other, not just the hub. After establishing trust (exchanging cryptographic keys) hosts can fetch policy and data files from each other. This can be used to ease network load, or as a backup / fallback mechanism in case you cannot fetch updated policy from the main policy server. Depending on your configuration, any CFEngine host can act as a policy server, or file server more generally. **Peer-based anomaly detection of file changes:** If you have some files which are expected to be the same across your hosts, you can divide hosts into groups and have them check each other's files. Outliers, where the file differs from the rest of the group, could be highlighted in a report, or have some other automatic remediation associated with them. Again, if your hosts are communicating with each other, and are relatively close in the network topology, you can reduce the load on your hub and network, and your hosts can react to changes and anomalies much more quickly than every host having to check with a centralized server. **Decentralized decision on role and firewall rules:** Policy can look at local files, installed packages, databases, and more, to make a decision about what role the host should have. CFEngine could see in a local database that this host should act as a webserver, and based on this information perform a number of steps, such as; install nginx and certbot, uninstall things you don't want on your webservers (like compilers and development tools), configure the firewall to allow traffic on ports 443 and 80, render the appropriate configuration file to set up redirect from HTTP to HTTPS and to run certbot to get and renew the TLS certificate, etc. **Maintenance windows:** Some users have special time windows when they want things to happen. Especially in finance / trading, where performance is really important, you might choose to have CFEngine not do anything during business hours on weekdays, and do your maintenance, updates and automation only at night or during weekends. **Skip parts of policy based on load:** If the system is really busy (experiencing high load), you can check for this and choose to postpone some parts of your policy until later. You might want to prioritize fast and security-relevant changes, and postpone other slower and less critical tasks. **Fix SSH:** If you are having problems with SSH or other remote access software, you can use CFEngine (which does not need SSH to communicate) to fix the SSH config. Taking this a step further, you could detect if both SSH is not working, and the CFEngine policy server is unreachable, and activate some kind of emergency remote access mode, for example enabling a 3rd option for remote access. **Configure DB memory based on host specifications:** Databases often have configuration values which can be tuned for better performance. When using CFEngine to render these configuration files, you can customize the database configuration based on the state and specs of the system. A high performance system with a lot of memory can naturally use more memory for the database and in general have "higher" tunable values. The config can also be adjusted based on other factors, such as day / time, if you need extra performance during business hours, for example. (And want to use those same resources for something else at other times). ## Redundancy - Alternate channel for remote access to a host If SSH is down, you can fix it with CFEngine. If you have problems with NAT, DNS, or other networking issues which prevent you from opening an SSH connection from your machine to the desired host, there is a good chance that you can use CFEngine to access the host and/or to resolve the issues, since CFEngine opens a connection in the other direction, from the host to the hub, to fetch policy. All you'd have to do is put your fixes into the policy and wait for the hosts to fetch the new policy and resolve the issues. This means that if you are using something like SSH, you have CFEngine as a redundant and different option for remotely accessing your hosts, which is not affected by the same failure cases as SSH. ## Conclusion At scale, there are significant benefits to the agent-based, optimized, decentralized, and declarative approach CFEngine uses. As discussed above, these can impact security, robustness to failures, performance, bandwidth, and efficiency of staff making changes and observing the results. [Contact us](/contact) if you're interested in discussing how to best use CFEngine or how CFEngine can help in your environment. [^1]: Our customers often use both CFEngine and other tools at the same time. -------------------------------------------------------------------------------- title: "Feature Friday #32: Doing math in policy with eval()" url: /blog/2024/feature-friday-32-doing-math-in-policy-with-eval/index.md date: "2024-10-18" -------------------------------------------------------------------------------- Ever need to do some math during policy evaluation? Sometimes configuration settings are based on available resources. For example, what if you want to calculate the size of shared buffers to be 25% of your available memory? Let's write some policy. First, we need to figure out how much memory we have. Let's parse this out from `/proc/meminfo`: ```cfengine3 bundle agent __main__ { vars: "d_meminfo" data => data_readstringarray( "/proc/meminfo", "", "(:|\s+)", inf, inf); reports: "$(with)" with => storejson( "d_meminfo[MemTotal]" ); } ``` ```text R: [ "", "65505464", "kB" ] ``` So, we have `65505464` kB of memory in total. Knowing that we can use [eval()](https://docs.cfengine.com/docs/3.23/reference-functions-eval.html#top) to calculate what 25% is. ```cfengine3 bundle agent __main__ { vars: "d_meminfo" data => data_readstringarray( "/proc/meminfo", "", "(:|\s+)", inf, inf); "shared_buffer_percent" string => "25"; "calculated_shared_buffers_kB" string => format( "%d", eval( "$(d_meminfo[MemTotal][1]) * ($(shared_buffer_percent)/100)", "math", "infix")); reports: "I calculated $(calculated_shared_buffers_kB)kB should be used for shared bufferes"; } ``` ```text R: I calculated 16376366kB should be used for shared bufferes ``` `eval()` can also be used to test truthfulness. ```cfengine3 bundle agent __main__ { vars: "d_meminfo" data => data_readstringarray( "/proc/meminfo", "", "(:|\s+)", inf, inf); "shared_buffer_percent" string => "25"; "calculated_shared_buffers_kB" string => format( "%d", eval( "$(d_meminfo[MemTotal][1]) * ($(shared_buffer_percent)/100)", "math", "infix")); reports: "I calculated $(calculated_shared_buffers_kB)kB should be used for shared bufferes"; "$(calculated_shared_buffers_kB) is < 50% of $(d_meminfo[MemTotal][1])" if => eval( "$(calculated_shared_buffers_kB) < ($(d_meminfo[MemTotal][1])/2)", "class" ); "$(calculated_shared_buffers_kB) is 25% of $(d_meminfo[MemTotal][1])" if => eval( "$(calculated_shared_buffers_kB) == ($(d_meminfo[MemTotal][1])/4)", "class" ); } ``` ```text R: I calculated 16376366kB should be used for shared bufferes R: 16376366 is < 50% of 65505464 R: 16376366 is 25% of 65505464 ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #31: Seeing a data structure with storejson()" url: /blog/2024/feature-friday-31-seeing-a-data-structure-with-storejson/index.md date: "2024-10-11" -------------------------------------------------------------------------------- Ever need to visualize the data you're working with? [storejson()](https://docs.cfengine.com/docs/3.23/reference-functions-storejson.html) to the rescue! Let's re-visit our example for `sys.os_release` from [Feature Friday #12: Special variables](/blog/2024/feature-friday-12-special-variables): ```cfengine3 bundle agent __main__ { reports: "My custom key 'NORTHERN_TECH_OWNER' contains $(sys.os_release[NORTHERN_TECH_OWNER])"; } ``` ```text R: My custom key 'NORTHERN_TECH_OWNER' contains Nick Anderson ``` So, we saw the value of a single key, but if we don't know what keys are available it can be useful to render the JSON representation. The `with` attribute in combination with `storejson()` provides a convenient way to visualize the JSON representation of structured data in CFEngine. Let's adjust the policy: ```cfengine3 bundle agent __main__ { reports: "My custom key 'NORTHERN_TECH_OWNER' contains $(sys.os_release[NORTHERN_TECH_OWNER])"; "The JSON representation of sys.os_release:$(const.n)$(with)" with => storejson( "sys.os_release" ); } ``` Running the policy we can see all of the keys and values: ```text R: My custom key 'NORTHERN_TECH_OWNER' contains Nick Anderson R: The JSON representation of sys.os_release: { "BUG_REPORT_URL": "https://bugs.launchpad.net/ubuntu/", "HOME_URL": "https://www.ubuntu.com/", "ID": "ubuntu", "ID_LIKE": "debian", "NAME": "Ubuntu", "NORTHERN_TECH_OWNER": "Nick Anderson", "PRETTY_NAME": "Ubuntu 22.04.4 LTS", "PRIVACY_POLICY_URL": "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy", "SUPPORT_URL": "https://help.ubuntu.com/", "UBUNTU_CODENAME": "jammy", "VERSION": "22.04.4 LTS (Jammy Jellyfish)", "VERSION_CODENAME": "jammy", "VERSION_ID": "22.04" } ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "More Windows build modules: Capabilities, Optional Features and OpenSSH Server" url: /blog/2024/more-windows-modules/index.md date: "2024-10-07" -------------------------------------------------------------------------------- I promised more [Build](https://build.cfengine.com) modules in my previous monthly Monday module blog post: [package-method-winget](package-method-winget/). And here they are: [windows-capability](https://build.cfengine.com/modules/windows-capability) and [windows-optional-feature](https://build.cfengine.com/modules/windows-optional-feature). ## Inventory Both of these modules use similar usage details to control whether to inventory and to promise the state for specific capabilities and optional features. By default inventory is taken in the form of a classic array which ends up in the Mission Portal as a comma separated list: ```text OpenSSH.Server~~~~0.0.1.0:Installed, OpenSSH.Client~~~~0.0.1.0:Installed, etc. ``` To disable this inventory, define the class `disable_windows_capability_inventory` or `disable_windows_optional_feature_inventory` in the `data` namespace. The `data` namespace is the default if you use [Host specific data](https://docs.cfengine.com/docs/3.24/web-ui-hosts.html#host-specific-data) or [Group data](https://cfengine.com/blog/2024/cfengine-3-24-lts-released-consistency/). If you want to set these in augments you will need to specify the namespace explicitly like this: ```json "classes": { "data:disable_windows_capability_inventory": [ "windows" ] } ``` ## Compliance report Now that we have inventory we can make a compliance report based on this information. In my organization I expect OpenSSH server to be installed: {{< wideimg "images/windows-openssh-server-compliance-check.png" "compliance check for openssh server installed on windows">}} The result of the report (after I include the policy example later in this post): {{< wideimg "images/windows-openssh-server-compliance-report.png" "compliance report showing that all 1 windows host has openssh server installed">}} ## Bundles to promise state - `windows_capability_installed(capability_name)` - `windows_capability_notpresent(capability_name)` - `windows_optional_feature_enabled(feature_name)` - `windows_optional_feature_disabled(feature_name)` And now let's use these bundles in some example policy and utilize Host specific data to request the installs: {{< wideimg "images/windows-openssh-containers-host-specific-install-classes.png" "host specific data classes requesting install of openssh, steam and windows containers">}} ## Windows OpenSSH Installed Having these two modules enables me to more easily install OpenSSH as a server on my Windows machines with the following policy: ```cf3 bundle agent openssh_server { methods: data:openssh_server_installed:: "openssh_server_installed"; } bundle agent openssh_server_installed { methods: windows:: "Add OpenSSH.Server Capability" usebundle => windows_capability_installed("OpenSSH.Server~~~~0.0.1.0"), classes => classes_generic("openssh"); services: windows.openssh_ok:: "sshd" service_policy => "start", service_method => winmethod; } body service_method winmethod { service_type => "windows"; service_autostart_policy => "boot_time"; } ``` This policy utilizes the built-in [services](https://docs.cfengine.com/docs/3.24/reference-promise-types-services.html) promise type which in our [Enterprise](https://cfengine.com/community-vs-enterprise-comparison/) version supports the Windows platform. ## Windows Containers with Docker Desktop Installed I also wanted to get [Windows Containers](https://learn.microsoft.com/en-us/virtualization/windowscontainers/) up and working. There are many ways to use containers on Windows: [Docker Desktop](https://www.docker.com/products/docker-desktop/), [Docker Engine](https://docs.docker.com/engine/install/binaries/#install-server-and-client-binaries-on-windows), [containerd](https://containerd.io/), [BuildKit](https://github.com/moby/buildkit) (uses containerd) and [Stevedore](https://github.com/slonopotamus/stevedore) to name several. ```cf3 bundle agent windows_containers { methods: data:windows_containers_installed:: "windows_containers_installed"; } bundle agent windows_containers_installed { methods: windows:: "windows_containers_dependencies"; packages: windows:: "Docker.DockerDesktop" package_method => winget, package_policy => "add"; } bundle agent windows_containers_dependencies { methods: windows:: "enable-containers" usebundle => windows_optional_feature_enabled("containers"); "enable-hyperv" usebundle => windows_optional_feature_enabled("Microsoft-Hyper-V"); } ``` ## Some manual steps :'( This gets me part of the way there. I haven't quite figured out how to get the Docker services started and switching to Windows Containers in an automated way just yet. After some manual steps I can pull and run and exec in a Windows Container: ```powershell docker pull mcr.microsoft.com/windows/nanoserver:10.0.19042.1889-amd64 docker run -d --name test mcr.microsoft.com/windows/nanoserver:10.0.19042.1889-amd64 ping -t localhost docker exec -it test cmd ``` And I get a Windows Cmd prompt! ```text Microsoft Windows [Version 10.0.19042.1889] (c) Microsoft Corporation. All rights reserved. C:\> ``` Note: I had to do some searching to find an image version that was compatible with my older version of Windows 10. That's where the "10.0.19042.1889" came from. Searching the [Microsoft Artifact Registry](https://mcr.microsoft.com/). ## Next time Maybe that policy will be my next blog: even-more-windows or automated-windows-docker-install. ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #30: Agent svg" url: /blog/2024/feature-friday-30-agent-svg/index.md date: "2024-10-04" -------------------------------------------------------------------------------- Ever want a custom CFEngine Agent logo? Check out `agentsvg`. You can find `agentsvg` in [core/contrib/](https://github.com/cfengine/core/tree/master/contrib/agentsvg) it's a python script that can generate CFEngine agent logos. ```sh python3 ./agentsvg.py > agent.svg ``` {{< figure src="agent.svg" >}} ```sh python3 ./agentsvg.py --body="#f5821f" --head="#052569" > agent-body-head-colors.svg ``` {{< figure src="agent-body-head-colors.svg" >}} In addition to customizing the head and body colors you can customize the arm positions to be `up`, `down`, `out` or `angled` and the legs can be `straight` or `out` ```sh python3 ./agentsvg.py --arms angled --legs out > agent-arms-legs.svg ``` {{< figure src="agent-arms-legs.svg" >}} Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #29: Variable class expressions" url: /blog/2024/feature-friday-29-variable-class-expressions/index.md date: "2024-09-27" -------------------------------------------------------------------------------- Did you know you can use variables in class expressions? If you are reading this, you probably are already familiar with the ability to use class expressions to restrict the context of multiple promises. For example, here we have three `reports` type promises, all guarded by the class expression `linux::`. ```cfengine3 bundle agent __main__ { reports: linux:: "Only hosts with the linux class"; "Will have these promises"; "In context"; } ``` And, if you are tracking this series, you know that you can restrict the context of a single promise using `if` or `unless`.[^fn:1] However, you can also use variables in class expressions. Let's take a look. Let's modify the previous policy to use a variable class expression. ```cfengine3 bundle agent __main__ { vars: "state" string => "disabled"; classes: "state_enabled" expression => "any"; reports: "linux.state_$(state)":: "Only hosts with the linux class"; "Will have these promises"; "In context"; } ``` Now we have a variable `state`, which is currently set to `disabled`. Furthermore, a class `state_enabled` which is defined in any context. Lastly, the class expression guarding our reports has been quoted and includes a variable expansion. No output is generated if you run this policy since all the reports are out of context. However, if you change `state` to `enabled`: ```cfengine3 {file="/tmp/feature-friday-29.cf"} bundle agent __main__ { vars: "state" string => "enabled"; classes: "state_enabled" expression => "any"; reports: "linux.state_$(state)":: "Only hosts with the linux class"; "Will have these promises"; "In context"; } ``` ```command cf-agent -Kf /tmp/feature-friday-29.cf ``` ```output R: Only hosts with the linux class R: Will have these promises R: In context ``` Running the policy on a Linux host, we see that all three reports are emitted. Hence, if you have multiple promises that need to be guarded based on some variable class name, then this can be easier to maintain than explicitly constraining each promise with `if` attributes. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: In case you missed it, Feature Friday #28: Restricting individual promises using if and unless. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 41 - Policy language support in Zed, and other editors" url: /blog/2024/the-agent-is-in-episode-41/index.md date: "2024-09-26" -------------------------------------------------------------------------------- Get CFEngine support in Zed, "The editor for what's next". "Zed is a next-generation code editor designed for high-performance collaboration with humans and AI." Cody and Nick are joined by Herman for an introduction to tree-sitter grammars, language servers and a quick demo of progress being made to add CFEngine support to the Zed editor. After reviewing the progress made and future work Nick shows some features of Emacs (Spacemacs) and Org-mode with CFEngine. ## Video {#video} The video recording is available on YouTube: {{< youtube 88jc0dRQIBo >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [Zed](https://zed.dev/) - [zed-cfengine extension](https://github.com/olehermanse/zed-cfengine) - [tree-sitter-cfengine](https://github.com/olehermanse/tree-sitter-cfengine) - [Spacemacs](https://www.spacemacs.org/) - [CFEngine layer](https://develop.spacemacs.org/layers/+tools/cfengine/README.html) - [Org-mode](https://orgmode.org/) - [ob-cfengine3](https://github.com/nickanderson/ob-cfengine3) -------------------------------------------------------------------------------- title: "Feature Friday #28: Restricting individual promises using if and unless" url: /blog/2024/feature-friday-28-restricting-individual-promises-using-if-and-unless/index.md date: "2024-09-20" -------------------------------------------------------------------------------- Class expressions are powerful. They let you restrict the context for multiple promises in a single statement. What if you want to further control the context of a **specific** promise? Let's take a look at a contrived example: ```cfengine3 {file="/tmp/feature-friday-28-0.cf"} bundle agent __main__ { reports: "I am running $(sys.os_release[PRETTY_NAME])"; linux:: "I love Linux!"; linux.ubuntu:: "Especially Ubuntu."; linux.redhat:: "Especially RedHat."; linux.!(ubuntu|redhat):: "But not RedHat or Ubuntu."; } ``` ```command cf-agent -Kf /tmp/feature-friday-28-0.cf ``` ```output R: I am running Ubuntu 22.04.4 LTS R: I love Linux! R: Especially Ubuntu. ``` Here, we have a report showing the distribution we're running through class expressions protecting the individual promises. We would see `I love Linux!` on Linux hosts. Depending on the specific distribution running we would see `Especially Ubuntu.` or `Especially RedHat.`. If we are running something other than those two distributions `But not RedHat or Ubuntu.` would be reported. We can make this policy a bit less repetitive by leveraging `if` and `unless` to restrict the individual promises that are all available when `linux` is defined: ```cfengine3 {file="/tmp/feature-friday-28-1.cf"} bundle agent __main__ { reports: "I am running $(sys.os_release[PRETTY_NAME])"; linux:: "I love Linux!"; "Especially Ubuntu." if => "ubuntu"; "Especially RedHat." if => "redhat"; "But not RedHat or Ubuntu." unless => or( "ubuntu", "redhat" ); } ``` By executing this policy, we see that it returns the exact same output: ```command cf-agent -Kf /tmp/feature-friday-28-1.cf ``` ```output R: I am running Ubuntu 22.04.4 LTS R: I love Linux! R: Especially Ubuntu. ``` The `if` and `unless` attributes also let us use functions which can be useful for cases where you only use the restriction once, without defining a class separately. ```cfengine3 {file="/tmp/feature-friday-28-2.cf"} bundle agent __main__ { reports: "I am running $(sys.os_release[PRETTY_NAME])"; linux:: "I love Linux!"; "Especially $(with)." with => "$(sys.os_release[NAME])", # Case insensitive `(?i)` match against classes contining ubuntu or redhat if => regcmp( "(?i).*(ubuntu|redhat).*", "$(sys.os_release[PRETTY_NAME])" ); "But not RedHat or Ubuntu." unless => or( "ubuntu", "redhat" ); } ``` Again, we can see executing this policy results in the same output. ```command cf-agent -Kf /tmp/feature-friday-28-2.cf ``` ```text R: I am running Ubuntu 22.04.4 LTS R: I love Linux! R: Especially Ubuntu. ``` To a large extent, `if` and `unless` are opposites, and `if => not()` should be equivalent to `unless`. Ignoring the `not()` function for a second, `if` and `unless` are exact opposites, in any case where `if` would skip a promise, `unless` would evaluate it, and vice versa. This holds true even for the more weird situations (edge cases), such as when the right-hand side is unresolved. That's not the case for the `not()` function. There are cases (unresolved variables and function calls) where `if => x` and `if => not(x)` behave the same when the right-hand side is unresolved. In both of these cases, the promise would be skipped. ```cfengine3 bundle agent main { classes: "a" if => "$(no_such_var)"; # Will be skipped "b" if => not("$(no_such_var)"); # Will be skipped } ``` In other words, unresolved variables in function calls cause skipping. Similarly, `if` defaults to skipping, while `unless` defaults to evaluating. Strictly speaking, `if` and `unless` are true opposites, while `if` and `if => not()` are not. So, if you are interested in what should be the default behavior when variables/function calls are not resolved. Then use `if => not()` when you want to default to skipping, and `unless` when you want to default to evaluating. For more info, see the docs for [`if`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#if) and [`unless`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#unless). Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #27: Multiple outcomes" url: /blog/2024/feature-friday-27-multiple-outcomes/index.md date: "2024-09-13" -------------------------------------------------------------------------------- When promises are actuated, a class can be defined based on its result. For example, if a promise modifies a file's content, you could define a class that indicates it has been repaired. However, did you know that promises can have multiple outcomes concurrently? That's right! Native promises (but not custom promises) can have multiple outcomes. For example, a promise can be both kept and repaired at the same time. Let's take a look. First, let's create a policy `/tmp/feature-friday-27.cf` to ensure that a file exists with some specific content and use `body classes results` from the standard library. ```cfengine3 {file="/tmp/feature-friday-27.cf"} body file control { inputs => { "$(sys.libdir)/stdlib.cf" }; } bundle agent __main__ { files: "/tmp/feature-friday-27.txt" create => "true", touch => "true", classes => results( "bundle", "feature_friday_27" ); reports: "Feature Friday Classes: $(with)" with => join( ", ", classesmatching( "feature_friday_27.*" ) ); } ``` Notice how I've used `body file control` to include the standard library since I am not running this example as part of a larger policy set.[^fn:1], [^fn:2] Next, we have our files promise that explicitly specifies the file should exist with `create => "true"`. Furthermore, it specifies that we want to update the "last modified" timestamp for the file using `touch => "true"`, and that bundle scoped classes prefixed with `feature_friday_27` should be defined.[^fn:3] Finally, we emit a report listing the classes starting with `feature_friday_27`. Running this policy the first time: ```command rm -f /tmp/feature-friday-27.txt cf-agent -KIf /tmp/feature-friday-27.cf ``` ```output info: Created file '/tmp/feature-friday-27.txt', mode 0600 info: Touched (updated time stamps) for path '/tmp/feature-friday-27.txt' R: Feature Friday Classes: feature_friday_27_repaired, feature_friday_27_reached ``` We can see that two classes were defined, `feature_friday_27_repaired` and `feature_friday_27_reached`. Running the policy again: ```command cf-agent -KIf /tmp/feature-friday-27.cf ``` ```output info: Touched (updated time stamps) for path '/tmp/feature-friday-27.txt' R: Feature Friday Classes: feature_friday_27_repaired, feature_friday_27_kept, feature_friday_27_reached ``` We see that this time, three classes were defined, `feature_friday_27_repaired` (because the file was touched), `feature_friday_27_kept` (because the file already existed), and `feature_friday_27_reached` (which is always defined by the `results` classes body, no matter the promise-outcome). It's even possible to have a promise that is kept, repaired, and not-kept concurrently. For example, we added ownership management: ```cfengine3 {file="/tmp/feature-friday-27.cf"} body file control { inputs => { "$(sys.libdir)/stdlib.cf" }; } bundle agent __main__ { files: "/tmp/feature-friday-27.txt" create => "true", touch => "true", perms => owner( "root" ), classes => results( "bundle", "feature_friday_27" ); reports: "Feature Friday classes: $(with)" with => join( ", ", classesmatching( "feature_friday_27.*" ) ); } ``` But, running unprivileged, `cf-agent` fails to change the ownership. As a result, we end up with classes for kept, repaired, and not kept: ```command cf-agent -KIf /tmp/feature-friday-27.cf ``` ```output info: Touched (updated time stamps) for path '/tmp/feature-friday-27.txt' error: Cannot set ownership on file '/tmp/feature-friday-27.txt'. (chown: Operation not permitted) info: Updated timestamps on '/tmp/feature-friday-27.txt' info: Touched (updated time stamps) for path '/tmp/feature-friday-27.txt' error: Cannot set ownership on file '/tmp/feature-friday-27.txt'. (chown: Operation not permitted) info: Updated timestamps on '/tmp/feature-friday-27.txt' error: Errors encountered when actuating files promise '/tmp/feature-friday-27.txt' R: Feature Friday classes: feature_friday_27_denied, feature_friday_27_not_kept, feature_friday_27_error, feature_friday_27_repaired, feature_friday_27_kept, feature_friday_27_reached ``` In practice, this detail is often not that important. But it is good to understand in cases where you want to construct a class expression that takes these eventualities into consideration. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: In case you missed it, check out [Feature Friday #09: body file control - inputs](/blog/2024/feature-friday-9-body-file-control-inputs). [^fn:2]: In case you missed it, check out [Feature Friday #25: Unprivileged execution](/blog/2024/feature-friday-25-unprivileged-execution). [^fn:3]: Find the implementation and documentation of the results classes body [here](https://docs.cfengine.com/docs/3.21/reference-masterfiles-policy-framework-lib-common.html#results). -------------------------------------------------------------------------------- title: "Find us on Matrix" url: /blog/2024/find-us-on-matrix/index.md date: "2024-09-12" -------------------------------------------------------------------------------- Looking for a place to chat about CFEngine? Historically CFEngineers could be found in `#cfengine` on `irc.freenode.net` but we moved to `#cfengine` on `irc.libera.chat` in 2021. At that time we introduced some chat bridges linking various channels together. The bridge between Matrix and Libera was [temporarily shut down](https://matrix.org/blog/2023/08/libera-bridge-disabled/) last August but that temporary measure eventually became a [permanent situation](https://matrix.org/blog/2023/11/28/shutting-down-bridge-to-libera-chat/) leaving our channels split. Since then we have decided to remain on Matrix and have established a new channel [CFEngine:matrix.org](https://matrix.to/#/%23CFEngine:matrix.org), we hope to see you there! -------------------------------------------------------------------------------- title: "Feature Friday #26: Groups custom promise type" url: /blog/2024/feature-friday-26-groups-custom-promise-type/index.md date: "2024-09-06" -------------------------------------------------------------------------------- There's a `users` promise type for managing local users. However, did you know there is also a custom one for managing local _groups_? You might have seen it mentioned in the [CFEngine Build announcement](/blog/2021/cfengine-build-launched), the blog post on [Managing local groups](/blog/2021/managing-local-groups), or in the [announcement supporting custom bodies](/blog/2022/custom-bodies) post. But let's take another look. The easiest way to integrate the `groups` custom promise type is by using `cfbs`, simply `cfbs add promise-type-groups` in your project. Next, we need some policy that leverages the `groups` promise type. Let's create `groups.cf` in the project's root directory and add it to the project with `cfbs add ./groups.cf`, selecting the option to add the `groups` bundle to the bundlesequence. ```cfengine3 {file="groups.cf"} bundle agent groups { vars: "characters" slist => { "yoda", "obi-wan", "luke", "dooku", "anakin" }; users: "$(characters)" policy => "present"; groups: "jedi" policy => "present", members => jedi; "sith" policy => "present", members => sith; } body members jedi { include => { "yoda", "obi-wan", "luke" }; exclude => { "dooku", "anakin" }; } body members sith { exclude => { "yoda", "obi-wan", "luke" }; include => { "dooku", "anakin" }; } ``` After running `cfbs build` and `cfbs install`, we can see it in action when we run the policy: ```command cf-agent -KIf update.cf && cf-agent -KI ``` ```output info: Created user 'yoda' info: Created user 'obi-wan' info: Created user 'luke' info: Created user 'dooku' info: Created user 'anakin' info: Created group 'jedi' info: Added user 'yoda' to group 'jedi' info: Added user 'obi-wan' to group 'jedi' info: Added user 'luke' to group 'jedi' info: Created group 'sith' info: Added user 'dooku' to group 'sith' info: Added user 'anakin' to group 'sith' ``` It created the groups and users as expected. And if we add `anakin` to the `jedi` group manually, we see that it gets removed on the subsequent execution: ```text [root@hub cfbs]# usermod --groups jedi --append anakin [root@hub cfbs]# getent group jedi jedi:x:1006:yoda,obi-wan,luke,anakin [root@hub cfbs]# cf-agent -KI info: Removed user 'anakin' from group 'jedi' [root@hub cfbs]# getent group jedi jedi:x:1006:yoda,obi-wan,luke ``` Unfortunately, bodies in custom promise types cannot currently accept a list directly as a parameter, so if you try to abstract the membership like this: ```cfengine3 bundle agent groups { vars: "jedi" slist => { "yoda", "obi-wan", "luke" }; "sith" slist => { "dooku", "anakin" }; "characters" slist => { @(jedi), @(sith) }; users: "$(characters)" policy => "present"; groups: "jedi" policy => "present", members => members( @(jedi), @(sith) ) ; "sith" policy => "present", members => members( @(sith), @(jedi) ) ; } body members members(include, exclude) { include => { "@(include)" }; exclude => { "@(exclude)" }; } ``` It won't work, you will see errors about undefined variables: ```text error: groups promise with promiser 'jedi' has unresolved/unexpanded variables error: groups promise with promiser 'sith' has unresolved/unexpanded variables error: groups promise with promiser 'jedi' has unresolved/unexpanded variables error: groups promise with promiser 'sith' has unresolved/unexpanded variables ``` But that can be worked around by simply passing the name of the variable to the members' body and expanding it there like this: ```cfengine3 bundle agent groups { vars: "jedi" slist => { "yoda", "obi-wan", "luke" }; "sith" slist => { "dooku", "anakin" }; "characters" slist => { @(jedi), @(sith) }; users: "$(characters)" policy => "present"; groups: "jedi" policy => "present", members => members( "groups.jedi", "groups.sith" ) ; "sith" policy => "present", members => members( "groups.sith", "groups.jedi" ) ; } body members members(include, exclude) { include => { "@($(include))" }; exclude => { "@($(exclude))" }; } ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Manage software on Windows with the package-method-winget module" url: /blog/2024/package-method-winget/index.md date: "2024-09-02" -------------------------------------------------------------------------------- As a developer and user of CFEngine I want to use policy to manage the software on my systems so that I can switch operating systems, distributions, computers and have all my normal tools available wherever I go. Towards this end I searched for a Windows package manager and found one in [`winget`](https://learn.microsoft.com/en-us/windows/package-manager/winget/). I showed a prototype in [Agent Is In - Episode 37 - Windows package management](the-agent-is-in-episode-37/) as well as refined the whole process in [Agent Is In - Episode 40 - Windows module workshop](the-agent-is-in-episode-40/). ## Caveat In working with `winget` I found that it is primarily supported on desktop systems. I will be researching this more in the future to come up with ways to manage software on Windows server systems with CFEngine policy. From the `winget` documentation: > The WinGet command line tool enables users to discover, install, upgrade, remove and configure applications on Windows 10 and Windows 11 computers. This tool is the client interface to the Windows Package Manager service. ## Add the module Here I will show how to add this new [package-method-winget](https://build.cfengine.com/modules/package-method-winget/) module via the [cfbs](https://github.com/cfengine/cfbs) command line client. It is also possible to add it with the Build app in Mission Portal. ```sh $ cfbs add package-method-winget The added module 'package-method-winget' accepts user input. Do you want to add it now? [yes/y/NO/n] y Collecting input for module 'package-method-winget' Would you like to accept source agreements for winget packages promises? [yes|no]: y Would you like to accept package agreements for winget packages promises? [yes|no]: y Would you like to allow this module to change PowerShell Execution Policy to LocalMachine:Unrestricted? [yes|no]: y The default commit message is: Added 2 modules - Added module 'powershell-execution-policy' - Added module 'package-method-winget' - Added input for module 'package-method-winget' Edit it? [yes/y/NO/n] Committing using git: [main 8d1b96e] Added 2 modules 2 files changed, 77 insertions(+) create mode 100644 package-method-winget/input.json ``` This module has some important inputs which require answers in order to enable the package method to function properly. ```console Would you like to accept source agreements for winget packages promises? [yes|no]: y Would you like to accept package agreements for winget packages promises? [yes|no]: y ``` These two opt-in questions relate to ensuring that the package method can operate automatically without causing install commands to wait for user input. ```text Would you like to allow this module to change PowerShell Execution Policy to LocalMachine:Unrestricted? [yes|no]: y ``` This opt-in is required if `winget` and the [`Microsoft.WinGet.Client`](https://www.powershellgallery.com/packages/Microsoft.WinGet.Client) PowerShell module is not installed. The install can be automated by the policy by it running two PowerShell scripts but if the `LocalMachine` `Execution Policy` is set to the default of `Undefined` CFEngine will not be able to run these scripts and install the needed software. ## Usage At this point you can write some packages promises: ```cf3 packages: "Microsoft.WindowsTerminal" package_method => winget, package_policy => "add"; ``` ## More on the way Look for more Windows related modules in the upcoming month. During the module workshop mentioned above I also showed progress on modules for - Inventorying and managing Windows Capabilities - Inventorying and managing Windows Optional Features Let us know how you use CFEngine with Windows and how we can help you do more! ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #25: Unprivileged execution" url: /blog/2024/feature-friday-25-unprivileged-execution/index.md date: "2024-08-30" -------------------------------------------------------------------------------- Generally, `cf-agent` runs as a privileged user. But did you know that you can also run as an unprivileged user? A major benefit of running `cf-agent` unprivileged is the ability to prototype policies during development. However, attempting to execute `cf-agent` as an unprivileged user without proper configuration will result in errors. Let's create `/tmp/feature-friday-25.cf` with the following content: ```cfengine3 {file="/tmp/feature-friday-25.cf"} bundle agent main { reports: "Happy Friday!"; } ``` Now, let's try running that policy with `cf-agent` as an unprivileged user: ```command cf-agent -Kf /tmp/feature-friday-25.cf ``` ```output error: cf-promises needs to be installed in /home/nick/.cfagent/bin for pre-validation of full configuration error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution. error: Error reading CFEngine policy. Exiting... ``` When `cf-agent` executes, it wants to use `cf-promises` from `~/.cfagent/bin`, which is empty: ```command ls -al ~/.cfagent/bin/ ``` ```output total 8 drwx------ 2 nick nick 4096 Mar 8 09:53 . drwx------ 11 nick nick 4096 Mar 8 09:53 .. ``` So that the user's binaries stay in sync with the package on the system, it's best to create a symbolic link from `~/.cfagent/bin` to `/var/cfengine/bin`: ```command rm -rf ~/.cfagent/bin && ln -s /var/cfengine/bin ~/.cfagent/bin ls ~/.cfagent/bin/cf-promises ``` ```output /home/nick/.cfagent/bin/cf-promises ``` If you run the policy again, you will get an error if the policy file is writable by others: ```command cf-agent -Kf /tmp/feature-friday-25.cf ``` ```output error: File /tmp/feature-friday-25.cf (owner 1001) is writable by others (security exception) ``` However, after adjusting the permissions - so that only the owner can write to the file - it runs without any issues: ```command chmod 600 /tmp/feature-friday-25.cf cf-agent -Kf /tmp/feature-friday-25.cf ``` ```output R: Happy Friday! ``` Furthermore, seeding the standard library in `~/.cfagent/inputs` would be convenient. Let's use `cf-remote` to place the standard library for the version of CFEngine that is currently installed: ```command MPF_TARBALL_URL=$(cf-remote --version $(cf-agent --version | head -n 1 | cut -d' ' -f3) list masterfiles | tail -n 1) MPF_TARBALL_FILENAME=$(basename "$MPF_TARBALL_URL") TMPDIR=$(mktemp -d --suffix=-seed-mpf-stdlib) curl "$MPF_TARBALL_URL" --output-dir "$TMPDIR" --output "$MPF_TARBALL_FILENAME" --silent tar --directory "$HOME/.cfagent/inputs" --gunzip --extract --file $TMPDIR/$MPF_TARBALL_FILENAME ./masterfiles/lib --strip-components 2 rm -rf "$TMPDIR" ``` Now, we can adjust our sample policy to include the standard library and use something from it: ```cfengine3 {file="/tmp/feature-friday-25.cf"} body file control { inputs => { "$(sys.libdir)/stdlib.cf" }; } bundle agent main { reports: "Happy Friday!"; "This policy contains:" printfile => cat( "$(this.promise_filename)" ); } ``` ```command cf-agent -Kf /tmp/feature-friday-25.cf ``` ```output R: Happy Friday! R: This policy contains: R: body file control R: { R: inputs => { "$(sys.libdir)/stdlib.cf" }; R: } R: bundle agent main R: { R: reports: R: R: "Happy Friday!"; R: "This policy contains:" printfile => cat( "$(this.promise_filename)" ); R: } ``` However, there are some limitations. Anything that requires elevated privileges will not work since `cf-agent` won't elevate privileges to take action. For example, here we adjusted the policy to set the ownership of the policy file to `root`: ```cfengine3 {file="/tmp/feature-friday-25.cf"} body file control { inputs => { "$(sys.libdir)/stdlib.cf" }; } bundle agent main { files: "$(this.promise_filename)" perms => owner( "root" ); reports: "Happy Friday!"; "This policy contains:" printfile => cat( "$(this.promise_filename)" ); } ``` Your OS says "Get out of here!" And CFEngine emits errors. ```command cf-agent -Kf /tmp/feature-friday-25.cf ``` ```output error: Cannot set ownership on file '/tmp/feature-friday-25.cf'. (chown: Operation not permitted) error: Cannot set ownership on file '/tmp/feature-friday-25.cf'. (chown: Operation not permitted) error: Errors encountered when actuating files promise '/tmp/feature-friday-25.cf' R: Happy Friday! R: This policy contains: R: body file control R: { R: inputs => { "$(sys.libdir)/stdlib.cf" }; R: } R: bundle agent main R: { R: files: R: "$(this.promise_filename)" R: perms => owner( "root" ); R: R: reports: R: R: "Happy Friday!"; R: "This policy contains:" printfile => cat( "$(this.promise_filename)" ); R: } ``` So, while there is no capability to leverage unprivileged execution to manage resources that require elevated privileges. It's undoubtedly handy for prototyping policy. And it might save you from some future unintended regrets. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 40 - Windows module workshop with CFEngine" url: /blog/2024/the-agent-is-in-episode-40/index.md date: "2024-08-29" -------------------------------------------------------------------------------- Ready for more CFEngine on Windows? Join Cody, Craig and Nick for a walk through of some windows related build modules and policy that Craig has been working on. Craig talks about powershell, cmdlets, winget and docker and shows progress towards some new modules for [CFEngine Build](https://build.cfengine.com). ## Video {#video} The video recording is available on YouTube: {{< youtube ERemN60zb_k >}} ## Post show discussion {#post-show-discussion} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ### Can CFEngine run on Guix? {#can-cfengine-run-on-guix} It certainly can, it's quite portable but managing it may be very different given the system methodology. We are not aware of any existing package definitions for CFEngine on [Guix](https://guix.gnu.org/), but we would love to hear about your endeavor on the [help mailing list](https://groups.google.com/g/help-cfengine) and or [github discussions](https://github.com/cfengine/core/discussions). ### Do we still monitor IRC? {#do-we-still-monitor-irc} Long story short, checkout the Matrix channel: ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Feature Friday #24: Augments - host_specific.json" url: /blog/2024/feature-friday-24-augments-host-specific-json/index.md date: "2024-08-23" -------------------------------------------------------------------------------- You probably know about the `def.json` Augments file. However, are you familiar with `host_specific.json`? The `def.json` Augments file is read, if it's adjacent to the policy entry. As such, this file is generally distributed as part of the policy set. Its settings apply to all hosts that receive and run the policy. The `host_specific.json` Augments file, is on the other hand loaded from the `$(sys.workdir)/data/` directory. And it is expected to be independent from the policy. `host_specific.json` has some notable differences from `def.json`: - `$(sys.workdir/data/host_specific.json` is loaded first, if it exists. - Variables and classes, when processed, will have a `source=cmdb` meta tag instead of `source=augments_file`. - Variables defined with the same name in `host_specific.json` have precedence over variables defined in `def.json`. - Variables are defined in the `variables` bundle scope of the `data` namespace instead of the `def` bundle of the `default` namespace by default. - Classes are defined in the `data` namespace instead of the `default` namespace by default. - Classes are defined without constraint, you can not specify a list of regular expressions or class expressions to determine if the class is defined. Let's take a look at some of this behavior: First we have a `def.json` file with the following contents: ```json { "variables": { "var1": { "value": "default:def.var1 - Defined in def.json", "comment": "A similar, but different variable data:variables.var1 is defined in def.json." }, "data:variables.var2": { "value": "data.variables.var2 - Defined in def.json", "comment": "This same variable is defined in host_specific.json with a different value." }, "var3": { "value": "default:def.var3 - Defined in def.json", "comment": "This same variable is defined in host_specific.json with a different value." } }, "classes": { "from_def_json_testing": { "class_expressions": [ "any::" ] } } } ``` Next, we have a `host_specific.json` file with the following contents: ```json { "variables": { "var1": { "value": "data:variables.var1 - Defined in host_specific.json", "comment": "A similar, but different variable default:def.var1 is defined in def.json." }, "var2": { "value": "data:variables.var2 - Defined in host_specific.json", "comment": "This same variable is defined in def.json with a different value." }, "default:def.var3": { "value": "default:def.var3 - Defined in host_specific.json", "comment": "This same variable is defined in def.json with a different value." } }, "classes": { "from_host_specific_data_testing": { "class_expressions": [] } } } ``` Looking at the variables and classes defined that end in `testing` we see that the namespace differs based on the source of the definition and that when the same variable, in this case `var2` and `var3`, is defined in both `def.json` and `host_specific.json` the value from `host_specific.json` wins: ```text [root@hub data]# cf-promises --show-vars=var\\d$ --show-classes=testing$ Class name Meta tags Comment data:from_host_specific_data_testing source=cmdb from_def_json_testing source=augments_file Variable name Variable value Meta tags Comment data:variables.var1 data:variables.var1 - Defined in host_specific.json source=cmdb A similar, but different variable default:def.var1 is defined in def.json. data:variables.var2 data:variables.var2 - Defined in host_specific.json source=cmdb This same variable is defined in def.json with a different value. default:def.var1 default:def.var1 - Defined in def.json source=augments_file A similar, but different variable data:variables.var1 is defined in def.json. default:def.var3 default:def.var3 - Defined in host_specific.json source=cmdb This same variable is defined in def.json with a different value. ``` Remember, a variable defined by either `def.json` or `host_specific.json` may be overwritten by the policy. If you have a promise to define a variable but you want to respect the value set from Augments you need to adjust the promise to the appropriate constraints, perhaps using [isvariable()](https://docs.cfengine.com/latest/reference-functions-isvariable.html#top), and/or [getvariablemetatags()](https://docs.cfengine.com/latest/reference-functions-getvariablemetatags.html#top). In CFEngine Enterprise, this file is generated on the Hub, and stored in a directory named after the host's public key digest. Furthermore, the file is copied into place by the `update.cf` policy. This provides an easy path for changing data related to a host without necessarily having access to or modifying the policy itself. While community agents can leverage the file, there is no prescription for managing or distributing the files. However, we encourage you to consider how you can leverage this functionality to better integrate with your own existing data sources. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #23: Agent say!" url: /blog/2024/feature-friday-23-agent-say/index.md date: "2024-08-16" -------------------------------------------------------------------------------- You have probably heard of `cowsay`, but have you heard of `agentsay`? Just in case you haven't seen the greatness of `cowsay`, here is an example: ```command cowsay "Gee, I wish I was a cf-agent!" ``` ```output _______________________________ < Gee, I wish I was a cf-agent! > ------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || ``` If you look in [core/contrib](https://github.com/cfengine/core/tree/master/contrib) you will find [agentsay](https://github.com/cfengine/core/tree/master/contrib/agentsay) among other goodies. It's a Python script. You can use it to make an agent say something. ```command python3 agentsay.py "Happy Friday!" ``` ```output /-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\ | | | HAPPY FRIDAY! | | | | ● ● ● | | ● ● ● | | | | ● ● ● ● ● | | ● ● ● ● ● | | ● ● ● ● ● | | ● ● | | ● ● ● | | ● ● | | ● ● | | ● ● | | | \-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-/ ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #22: Don't fix, just warn" url: /blog/2024/feature-friday-22-dont-fix-just-warn/index.md date: "2024-08-09" -------------------------------------------------------------------------------- Did you know that CFEngine can simply _warn_ about something not being in the desired state? Traditionally with CFEngine, you define your desired state and CFEngine works towards making that happen. Sometimes you might not want CFEngine to take action and instead warn that a given promise wants to change something. Let's take a look at a contrived example. Say we want the file `/tmp/feature-friday-22.txt` to exist, we might write a policy that looks like this: ```cfengine3 bundle agent __main__ { files: "/tmp/feature-friday-22.txt" create => "true"; } ``` But, we don't actually want the file to exist, rather we want a _warning_. Well, you could implement that using a reports-type promise: ```cfengine3 bundle agent __main__ { reports: "warning: /tmp/feature-friday-22.txt doesn't exist!" unless => fileexists("/tmp/feature-friday-22.txt"); } ``` However, there is a slight difference. A `reports` promise will be seen as a promise `kept` - yes it will be in your logs - but it's a bit less visible in general. Instead, we can attach an `action` body with the `action_policy` attribute set to `warn`. Let's adjust our first example: ```cfengine3 {file="/tmp/feature-friday-22-0.cf"} bundle agent __main__ { files: "/tmp/feature-friday-22.txt" create => "true", action => warn_dont_fix, classes => results( "bundle", "example_warn" ); reports: example_warn_not_kept:: "My promise to ensure /tmp/feature-friday-22.txt was not kept"; } body action warn_dont_fix { action_policy => "warn"; } ``` Running the policy we can see that warnings were emitted by the agent and that the `files` promise itself was not kept. We even get a bit more detail about the described state of the file, like the fact that it would be created with permissions only allowing it to be read by the owner. ```command cf-agent -Kf /tmp/feature-friday-22-0.cf ``` ```output warning: Should create file '/tmp/feature-friday-22.txt', mode '0600', but only warning promised warning: Warnings encountered when actuating files promise '/tmp/feature-friday-22.txt' R: My promise to ensure /tmp/feature-friday-22.txt was not kept ``` Let's take it a bit further, what if we only want to warn when a specific context is true. For example, let's warn except on the day of the publication date of this post. Here the `action_policy` is guarded by a class expression targeting any day except the publication date of this post. Unless it's the publication date, the policy will just warn, otherwise it will have the default behavior and fix the state. ```cfengine3 {file="/tmp/feature-friday-22-1.cf"} bundle agent __main__ { files: "/tmp/feature-friday-22.txt" create => "true", action => warn_dont_fix_except_publication, classes => results( "bundle", "example_warn" ); reports: example_warn_not_kept:: "My promise to ensure /tmp/feature-friday-22.txt was not kept"; example_warn_repaired:: "My promise to ensure /tmp/feature-friday-22.txt was repaired"; example_warn_kept:: "My promise to ensure /tmp/feature-friday-22.txt needed no action and was kept"; } body action warn_dont_fix_except_publication { action_policy => "warn"; !(Yr2024.August.Day09):: action_policy => "fix"; } ``` Running the policy, on any other day than the publication of the post, causes it to fix the state as desired: ```command cf-agent -Kf /tmp/feature-friday-22-1.cf ``` ```output R: My promise to ensure /tmp/feature-friday-22.txt was repaired ``` This sort of logic can be tied to maintenance windows or other classes, allowing you to automatically adjust when a promise warns or takes action. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Inventory and mitigate OpenSSH vulnerability CVE-2024-6387" url: /blog/2024/inventory-and-mitigate-openssh-vulnerability-cve-2024-6387/index.md date: "2024-08-05" -------------------------------------------------------------------------------- The rather serious recent [OpenSSH vulnerability](https://blog.qualys.com/vulnerabilities-threat-research/2024/07/01/regresshion-remote-unauthenticated-code-execution-vulnerability-in-openssh-server) [CVE-2024-6387](https://nvd.nist.gov/vuln/detail/CVE-2024-6387) could affect as many as 14 million server instances exposed on the internet. Let's make it easy to examine your infrastructure and see if you need to do any upgrades or mitigations. On the back of my CFEngine T-shirt it says: ## Know more, React faster When I have a problem to solve in CFEngine I look for an easy and correct solution. [CFEngine Build](https://build.cfengine.com/) is a good first place to look. Two modules stand out as possibly useful; 1. We can use [inventory-openssl-versions](https://build.cfengine.com/modules/inventory-openssl-versions/) as a template to inventory OpenSSH version. This is the "Know more" step. 2. For the "React faster" we may need to change something. In case we can't upgrade OpenSSH we will need to mitigate the vulnerability by modifying `sshd` config. For this purpose we can leverage [library-sshd-config](https://build.cfengine.com/modules/library-sshd-config/). This module has the advantage of preparing a staged configuration as well as restarting the service in case of a change. ## Adding modules Previously we have used the Build app in [Mission Portal](https://docs.cfengine.com/docs/master/web-ui.html). This time let's use the [cfbs](https://github.com/cfengine/cfbs) command line tool. ```command cfbs add inventory-openssl-versions ``` ```output Added module: inventory-openssl-versions The default commit message is 'Added module 'inventory-openssl-versions'' - edit it? [yes/y/NO/n] Committing using git: [home 5964af0] Added module 'inventory-openssl-versions' 1 file changed, 15 insertions(+) ``` ```command cfbs add library-sshd-config ``` ```output Added module: library-sshd-config The default commit message is 'Added module 'library-sshd-config'' - edit it? [yes/y/NO/n] Committing using git: [home 0da07db] Added module 'library-sshd-config' 1 file changed, 15 insertions(+) ``` So let's "copy-paste-modify" the [policy from `inventory-openssl-versions`](https://build.cfengine.com/modules/inventory-openssl-versions/) to inventory OpenSSH aka `sshd` version. ## OpenSSH version information in inventory For simplicity's sake we will rely only on the `sshd` command only instead of also checking package managers. ```cfengine3 body file control { namespace => "inventory_sshd_version"; } bundle agent inventory_sshd_version { vars: !windows:: "_sshd_path" string => "/usr/sbin/sshd"; classes: linux:: "sshd_binary_works" if => returnszero("${_sshd_path} -t 1>/dev/null 2>&1", "useshell"); # -t option is Test mode, should return zero vars: sshd_binary_works:: "sshd_output" string => nth( splitstring( execresult("${_sshd_path} -nosuchargument", "useshell"), # Sadly sshd has no -V|--version so instead we use # -nosuchargument which causes sshd to output a full version # like OpenSSH_8.5p1 "\\n", 3), 1); classes: sshd_binary_works:: "found_sshd_version" expression => regextract( "^.*OpenSSH_([0-9\.p]+).*", "${sshd_output}", "captures" ); vars: found_sshd_version:: # OpenSSH versions are in the format (major).(minor)p(patch) # so we replace the "p" with "." to conform more to semantic # version standard used by version_compare() function. "sshd_version" string => string_replace( "${captures[1]}", "p", "."), meta => {"inventory", "attribute_name=OpenSSH version"}; reports: DEBUG|DEBUG_INVENTORY_OPENSSH_VERSION:: "OpenSSH version: $(sshd_version)"; } ``` After a few agent runs we can check inventory and see if we have any troubles. {{< wideimg "images/openssh-openssl-version-inventory.png" "inventory including OpenSSH version and OpenSSL versions">}} ## React faster Two of my systems have an affected version! If a system was affected and I couldn't upgrade the OpenSSH version, I would create policy to enact the mitigation procedure as recommended in the [technical details](https://www.qualys.com/2024/07/01/cve-2024-6387/regresshion.txt): > Finally, if `sshd` cannot be updated or recompiled, this signal handler > race condition can be fixed by simply setting LoginGraceTime to 0 in the > configuration file. This makes `sshd` vulnerable to a denial of service > (the exhaustion of all MaxStartups connections), but it makes it safe > from the remote code execution presented in this advisory. So let's write some policy. The first trick is to figure out if the OpenSSH version is affected. Let's express the logic of whether the version is OK in a simple expression: ```text (version >= 4.4p1 && version < 8.5p1) || version > 9.8p1 ``` Thankfully, we have a helper in a new function available in our [latest LTS](https://cfengine.com/blog/2024/cfengine-3-24-lts-released-consistency/), 3.24: [`version_compare()`](https://docs.cfengine.com/docs/3.24/reference-functions-version_compare.html). With it, we can define a class `sshd_version_ok` using this policy: ```cfengine3 classes: "sshd_version_ok" expression => or( and( version_compare("${sshd_version}", ">=", "4.4.1"), version_compare("${sshd_version}", "<", "8.5.1") ), version_compare("${sshd_version}", ">", "9.8.1") ); ``` But this could cause `sshd_version_ok` to not be defined in case the `sshd_version` is not defined. This is often an issue in CFEngine where [negative knowledge](https://docs.cfengine.com/docs/master/reference-language-concepts-classes.html#negative-knowledge) can't really be certain. We should rewrite this as "positive knowledge" aka something we determine explicitly. ```cfengine3 "sshd_version_cve" expression => and( isvariable("sshd_version"), or( version_compare("${sshd_version}", "<", "4.4.1"), and( version_compare("${sshd_version}", ">=", "8.5.1"), version_compare("${sshd_version}", "<=", "9.8.1") ) ) ); ``` And now we can write our mitigation policy: ```cfengine3 bundle agent openssh_mitigation_CVE_2006_5051_CVE_2008_4109 { vars: "sshd_version" string => "${inventory_openssh_version:inventory_openssh_version.sshd_version}"; classes: "sshd_version_cve" expression => and( isvariable("sshd_version"), or( version_compare("${sshd_version}", "<", "4.4.1"), and( version_compare("${sshd_version}", ">=", "8.5.1"), version_compare("${sshd_version}", "<=", "9.8.1") ) ) ); vars: "sshd_config[LoginGraceTime]" string => "0"; methods: sshd_version_cve:: "sshd config mitigation for CVE-2006-5051 and CVE-2008-4109" usebundle => lib_sshd_config:global_key_values( "$(this.namespace):$(this.bundle).sshd_config"); } ``` And with that, we know more: what versions of OpenSSH are present and have reacted faster: mitigated `sshd` config if needed. ## TODO - Both `inventory_openssh_version` and `openssh_mitigation_CVE_2006_5051_CVE_2008_4109` could be made into [CFEngine Build](https://build.cfengine.com/) modules. - We could make more generic modules to inventory various softwares and mitigate various vulnerabilities. ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #21: Promisees or stakeholders" url: /blog/2024/feature-friday-21-promisees-or-stakeholders/index.md date: "2024-08-02" -------------------------------------------------------------------------------- Who cares about that promise? Today, I want to highlight one of the lightweight knowledge management features in CFEngine. That is, Promisees, also known as Stakeholders. Promisees are references to things that might care about a specific promise. And they can be attached to any promise. Let's take a look at a contrived example: ```cfengine3 bundle agent __main__ { methods: "example_promisees" -> { "Feature Friday #21" }; } bundle agent example_promisees { reports: "Happy Friday!"; } ``` From the example above, we can see that the `methods` promise - promising to run the `example_promisees` bundle - has `Feature Friday #21` defined as the only promisee. It provides us with a hint that Feature Friday #21 cares that the bundle will be run. Promisees have no effect on the execution of the policy. However, they provide breadcrumbs that can be useful when the policy is re-visited. In CFEngine Enterprise, promisees are reported back to the hub. Hence, you can run queries against them. In practice, they often contain IDs for ticketing systems, for example: ```cfengine3 "total" -> { "CFE-2797", "CFE-2803" } string => execresult("/usr/bin/lparstat -i | awk '/Online Memory/ { print $4 }'", "useshell"), meta => { "inventory", "attribute_name=Memory size (MB)" }; ``` From this snippet from the Masterfiles Policy Framework, we can see the `total` variable is defined with a reference to [CFE-2797](https://northerntech.atlassian.net/browse/CFE-2797) and [CFE-2803](https://northerntech.atlassian.net/browse/CFE-2803). This gives us a quick way to find further information regarding its presence. In this case, we seem to be working around an issue where `cf-monitord` does not collect memory information used for inventory of `Memory size (MB)` on AIX. It's a free-form field, you can put any information that might be necessary to look for further information about a promise including names, URLs, and phone numbers. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday Feature Friday #20: Macros" url: /blog/2024/feature-friday-feature-friday-20-macros/index.md date: "2024-07-26" -------------------------------------------------------------------------------- Did you know CFEngine has Macros? They were first introduced in CFEngine 3.7 (back in 2015), and more have been introduced since then. Macros are convenient for preventing the parsing of a section of the policy. It is handy for protecting older binaries from getting tripped up on newer syntax the agent does not understand. Let's take a look. Currently there are 8 macros. - [minimum_version](https://docs.cfengine.com/docs/3.21/reference-macros.html#minimum-version) - Prevent the section of policy from being parsed unless the agent meets a minimum version. - [maximum_version](https://docs.cfengine.com/docs/3.21/reference-macros.html#maximum-version) - Prevent the section of policy from being parsed when the agent exceeds a maximum version. - [at_version](https://docs.cfengine.com/docs/3.21/reference-macros.html#at-version) - Prevents the section of policy from being parsed unless the agent is of a specific version. - [between_versions](https://docs.cfengine.com/docs/3.21/reference-macros.html#between-versions) - Prevents a section of policy from being parsed unless the agent is between (inclusive) a minimum and maximum version. - [before_version](https://docs.cfengine.com/docs/3.21/reference-macros.html#before-version) - Prevents a section of policy from being parsed unless the agent is below a specified version (not inclusive). - [after_version](https://docs.cfengine.com/docs/3.21/reference-macros.html#after-version) - Prevents a section of policy from being parsed unless the agent is above a specified version (not inclusive). - [else](https://docs.cfengine.com/docs/3.21/reference-macros.html#else) - Allows the agent to parse a section of policy only if the preceding macro is not applicable. - [feature](https://docs.cfengine.com/docs/3.21/reference-macros.html#features) - Prevents a section of policy from being parsed based on feature availability. You can find examples of use within the Masterfiles Policy Framework. For example, `body action fresh_systemd_state` uses the `minimum_version` macro to control the setting of `ifelapsed => "0"` to versions 3.18.1 and higher since versions below might produce a warning. ```cfengine3 body action fresh_systemd_state # @brief An 'action' body ensuring the state information for a systemd service is always fresh # # This 'action' body disables caching for functions, in particular the # execresult*() family of functions. # # Although it's now the same as the 'immediate' action body, this may change in # the future. { cfengine:: # ^Needed for versions 3.15.x and older that did not # support empty bodies. When 3.15.x is no longer # supported, this can be removed. @if minimum_version(3.18.1) # Beginning with 3.18.1 ifelapsed being set to 0 results in bypassing of # function caching. In versions prior to 3.18, if an action body with # ifelapsed set to 0 was used in a vars type promise a warning was # emitted. This is guarded to suppress that warning in older versions # where this setting would not change the behavior. ifelapsed => "0"; @endif } ``` I think that is a lot more manageable as opposed to using class expressions, which would get quite lengthy if we wanted to extend it to cover even older versions: ```cfengine3 body action fresh_systemd_state # @brief An 'action' body ensuring the state information for a systemd service is always fresh # # This 'action' body disables caching for functions, in particular the # execresult*() family of functions. # # Although it's now the same as the 'immediate' action body, this may change in # the future. { cfengine:: # ^Needed for versions 3.15.x and older that did not # support empty bodies. When 3.15.x is no longer # supported, this can be removed. !(cfengine_3_18_0|cfengine_3_17|cfengine_3_16|cfengine_3_15):: # Beginning with 3.18.1 ifelapsed being set to 0 results in bypassing of # function caching. In versions prior to 3.18, if an action body with # ifelapsed set to 0 was used in a vars type promise a warning was # emitted. This is guarded to suppress that warning in older versions # where this setting would not change the behavior. ifelapsed => "0"; } ``` The feature macro can conveniently be abused. For example, you could include content you don't want CFEngine to parse, perhaps inline documentation or examples. ````cfengine3 bundle agent __main__ { @if feature(documentation_markdown) # My embedded Markdown * list item ```sh echo hello world! ``` @endif vars: @if feature(example__etc_os_release) PRETTY_NAME="postmarketOS v23.12" NAME="postmarketOS" VERSION_ID="v23.12" VERSION="v23.12" ID="postmarketos" ID_LIKE="alpine" HOME_URL="https://www.postmarketos.org/" SUPPORT_URL="https://gitlab.com/postmarketOS" BUG_REPORT_URL="https://gitlab.com/postmarketOS/pmaports/issues" LOGO="postmarketos-logo" @endif "os_release" data => readenvfile( "/etc/os-release" ); } ```` As long as the parameter to the feature macro is not a feature class provided by the agent, the content won't be parsed by CFEngine. Hence, you don't have to prefix lines with hash comments. Accordingly, the copying/pasting of inner content is more practical. I hope you can find some good uses for macros. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 39 - Editing /etc/default/grub and similar files" url: /blog/2024/the-agent-is-in-episode-39/index.md date: "2024-07-25" -------------------------------------------------------------------------------- Ever want to manage subvalues in a configuration file? In episode 39 we talk about using `field_edits` in `edit_line` bundles. Cody and Craig join Nick as he prototypes and iterates on some policy showing how to manage subvalues in KEY = quoted values. A question raised during one of our recent post show discussions. ## Video {#video} The video recording is available on YouTube: {{< youtube jrFHUBZbI0o >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Developing policy to edit `/etc/default/grub` and similar files {#developing-policy-to-edit-etc-default-grub-and-similar-files} ### Promise the initial content for a test file {#promise-the-initial-content-for-a-test-file} To prototype the policy we first lay down a file with content that is representative of what we are trying to edit. This is very helpful in prototyping policy as it re-sets the initial state of the test file during each execution. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' R: GRUB_CMDLINE_LINUX="" R: ``` ### Testing against `GRUB_CMDLINE_LINUX_DEFAULT` {#testing-against-grub-cmdline-linux-default} When we went to make a promise about `GRUB_CMDLINE_LINUX_DEFAULT` we encountered an error. The `field_separator` in `body edit_field quoted_var` is configured to look only for double quotes. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub; reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub { field_edits: "GRUB_CMDLINE_LINUX=.*" edit_field => quoted_var( "ipv6.disable=1", "append" ); "GRUB_CMDLINE_LINUX_DEFAULT=.*" edit_field => quoted_var( "init_on_alloc=1", "append" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output error: Field edit, no fields found by promised pattern '"' in '/tmp/etc-default-grub' error: Errors encountered when actuating fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired info: Edited file '/tmp/etc-default-grub' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' R: GRUB_CMDLINE_LINUX="ipv6.disable=1" R: ``` ### Refactor to handle single or double quotes {#refactor-to-handle-single-or-double-quotes} Since `body edit_field quoted_var` from the standard library does not handle single quotes we define our own `edit_field` body, `single_or_double_quoted_subvalue` (based on `quoted_var`) which supports single or double quoted values. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub; reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub { field_edits: "GRUB_CMDLINE_LINUX=.*" edit_field => quoted_var( "ipv6.disable=1", "append" ); "GRUB_CMDLINE_LINUX_DEFAULT=.*" edit_field => single_or_double_quoted_subvalue( "init_on_alloc=1", "append" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired info: Appended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Edited file '/tmp/etc-default-grub' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash init_on_alloc=1' R: GRUB_CMDLINE_LINUX="ipv6.disable=1" R: ``` ### Make more re-usable (but less efficient) {#make-more-re-usable--but-less-efficient} Next we made the policy more re-usable by exposing parameters to the `edit_line` bundle. This change however is less efficient as each files promise results in a separate file operation. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub("GRUB_CMDLINE_LINUX", "ipv6.disable=1", "append" ); "/tmp/etc-default-grub" edit_line => etc_default_grub("GRUB_CMDLINE_LINUX_DEFAULT", "init_on_alloc=1", "append" ); "/tmp/etc-default-grub" edit_line => etc_default_grub("GRUB_CMDLINE_LINUX_DEFAULT", "quiet", "delete" ); reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub(key, entry, method) { field_edits: "$(key)=.*" edit_field => single_or_double_quoted_subvalue( "$(entry)", "$(method)" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired info: Edited file '/tmp/etc-default-grub' info: Appended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Edited file '/tmp/etc-default-grub' info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Edited file '/tmp/etc-default-grub' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash init_on_alloc=1' R: GRUB_CMDLINE_LINUX="ipv6.disable=1" R: ``` From the execution output above we can clearly see that the file is actually modified three separate times. ### Making it a single edit -- broken -- need coffee {#making-it-a-single-edit-broken-need-coffee} During the show we started to reign in the file operations while maintaining the ability to reuse the policy in different contexts but Nick hadn't yet had his coffee and chose poorly when he defined the data structure. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { vars: "conf[GRUB_CMDLINE_LINUX][ipv6.disable=1]" string => "append"; "conf[GRUB_CMDLINE_LINUX_DEFAULT][init_on_alloc=1]" string => "append"; "conf[GRUB_CMDLINE_LINUX_DEFAULT][quiet]" string => "delete"; files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub( "$(this.namespace):$(this.bundle).conf" ); reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub(d) { vars: "s" data => mergedata( d ); "keys" slist => getindices( s ); field_edits: "$(s[$(keys)])=.*" edit_field => single_or_double_quoted_subvalue( "$(s[$(keys)][1])", "$(s[$(keys)][2])" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' R: GRUB_CMDLINE_LINUX="" R: ``` ### The fix for a single edit {#the-fix-for-a-single-edit} The solution came to Nick after the post show discussion when he walked away to get coffee. He needed to change the way he was structuring the data. Here we see he switched to a `data` type variable and changed the way that the data was passed to the `edit_line` bundle. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { vars: "conf" data => ' { "GRUB_CMDLINE_LINUX": { "append": [ "ipv6.disable=1" ], }, "GRUB_CMDLINE_LINUX_DEFAULT": { "prepend": [ "init_on_alloc=1" ], "delete": [ "quiet" ], } }'; files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub( "@($(this.namespace):$(this.bundle).conf)" ); reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub(conf_map) { vars: "keys" slist => getindices( conf_map ); "methods[$(keys)]" slist => getindices( "conf_map[$(keys)]" ); field_edits: "$(keys)=.*" edit_field => single_or_double_quoted_subvalue( "$(conf_map[$(keys)][$(methods[$(keys)])])", "$(methods[$(keys)])" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired info: Prepended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Edited file '/tmp/etc-default-grub' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='init_on_alloc=1 cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' R: GRUB_CMDLINE_LINUX="ipv6.disable=1" R: ``` From the above output we can see that all the changes were made as expected and only a single file change was made. #### How do I use the same data structure with associative arrays? {#how-do-i-use-the-same-data-structure-with-associative-arrays} In addressing the issue Nick switched to using a `data` variable instead of an associative array which is a collection of individual variables in CFEngine. Here is an iteration showing use with associative arrays. Note that in addition to changing the way the data structure is defined, we also changed how it was passed back to passing the name of the variable rather than the data structure as a whole. ```cfengine3 {file="/tmp/example-edit-etc-grub-default.cf"} bundle agent __main__ { vars: "conf[GRUB_CMDLINE_LINUX][append]" slist => { "ipv6.disable=1" }; "conf[GRUB_CMDLINE_LINUX_DEFAULT][prepend]" slist => { "init_on_alloc=1" }; "conf[GRUB_CMDLINE_LINUX_DEFAULT][delete]" slist => { "quiet" }; files: "/tmp/etc-default-grub" content => `GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" `; "/tmp/etc-default-grub" edit_line => etc_default_grub( "$(this.namespace):$(this.bundle).conf" ); reports: "/tmp/etc-default-grub" printfile => cat( "$(this.promiser)"); } bundle edit_line etc_default_grub(conf_var) { vars: "conf_map" data => mergedata( "$(conf_var)" ); "keys" slist => getindices( conf_map ); "methods[$(keys)]" slist => getindices( "conf_map[$(keys)]" ); field_edits: "$(keys)=.*" edit_field => single_or_double_quoted_subvalue( "$(conf_map[$(keys)][$(methods[$(keys)])])", "$(methods[$(keys)])" ); } body edit_field single_or_double_quoted_subvalue( entry, method) { inherit_from => quoted_var( "$(entry)", "$(method)" ); field_separator => "(\"|')"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-edit-etc-grub-default.cf ``` ```output info: Updated file '/tmp/etc-default-grub' with content 'GRUB_CMDLINE_LINUX_DEFAULT='quiet cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' GRUB_CMDLINE_LINUX="" ' info: Deleted column field sub-value 'quiet' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Prepended field sub-value 'init_on_alloc=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX_DEFAULT=.*' repaired info: Appended field sub-value 'ipv6.disable=1' in '/tmp/etc-default-grub' info: fields_edit promise 'GRUB_CMDLINE_LINUX=.*' repaired info: Edited file '/tmp/etc-default-grub' R: /tmp/etc-default-grub R: GRUB_CMDLINE_LINUX_DEFAULT='init_on_alloc=1 cryptdevice=UUID=4c56337e-878a-48c7-bca3-ed6fa50cf017:luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 root=/dev/mapper/luks-4c56337e-878a-48c7-bca3-ed6fa50cf017 splash' R: GRUB_CMDLINE_LINUX="ipv6.disable=1" R: ``` ## Post show discussion {#post-show-discussion} - Did you know that CFEngine has 3 quote characters? Single quote ( `'` ), double quote ( `"` ) and back tick **( `` ` `` )**. This can allow you to avoid having to escape inner quotes. ### Questions {#questions} #### How to influence order of operations (Normal Order) {#how-to-influence-order-of-operations--normal-order} ##### Using `handle` and `depends_on` {#using-handle-and-depends-on} ```cfengine3 bundle agent __main__ [file=/tmp/example-depends-on.cf] { commands: "/bin/echo RED" handle => "red", depends_on => { "yellow" }; "/bin/echo YELLOW" handle => "yellow"; methods: "help" depends_on => { "red" }; } bundle agent help { commands: "/bin/echo Byeee"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/example-depends-on.cf ``` ```output info: Executing 'no timeout' ... '/bin/echo YELLOW' notice: Q: ".../bin/echo YELLO": YELLOW info: Last 1 quoted lines were generated by promiser '/bin/echo YELLOW' info: Completed execution of '/bin/echo YELLOW' info: Executing 'no timeout' ... '/bin/echo RED' notice: Q: ".../bin/echo RED": RED info: Last 1 quoted lines were generated by promiser '/bin/echo RED' info: Completed execution of '/bin/echo RED' info: Executing 'no timeout' ... '/bin/echo Byeee' notice: Q: ".../bin/echo Byeee": Byeee info: Last 1 quoted lines were generated by promiser '/bin/echo Byeee' info: Completed execution of '/bin/echo Byeee' ``` ##### Use methods to get a more strict order {#use-methods-to-get-a-more-strict-order} ```cfengine3 bundle agent __main__ { methods: "stage1"; "stage2"; "stage3"; } ``` ##### Verify configuration before restarting the service {#verify-configuration-before-restarting-the-service} You may want to verify a configuration file has valid syntax before restarting a service. Consider rendering your configuration to a staging location that is verified and copied to final destination after syntax check. This way you can avoid issues that could be caused by some random reboot where you have a bad config in place but the service wasn't restarted since it wasn't valid. #### How can I access another bundles classes? {#how-can-i-access-another-bundles-classes} You can not. Classes are either `namespace` scope (visible by any other bundle), or they are `bundle` scoped. - Classes defined from classes promises in _common_ bundles are `namespace` scoped by default. - Classes defined from classes promises in _agent_ bundles are `bundle` scoped by default. - You can specify override the default scope using the `scope` attribute. - Bundle scoped classes can be inherited by another bundle - Information from a bundle can be passed to an associative array in the parent bundle via the `bundle_return_value_index` attribute of `reports` promises if the parent called the bundle with `useresult`. ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - Post show discussion - Influence order with [depends_on](https://docs.cfengine.com/docs/3.24/reference-promise-types.html#depends_on) - Methods can [inherit](https://docs.cfengine.com/docs/3.24/reference-promise-types-methods.html#inherit) the callers bundle scoped classes - Allow called bundle to return information to the calling bundle via [useresult](https://docs.cfengine.com/docs/3.24/reference-promise-types-methods.html#useresult) - Pass data to calling bundle using [bundle_return_value_index](https://docs.cfengine.com/docs/3.24/reference-promise-types-reports.html#bundle_return_value_index) -------------------------------------------------------------------------------- title: "Change in behavior: CFEngine roles inventory attribute" url: /blog/2024/change-in-behavior-cfengine-roles/index.md date: "2024-07-23" -------------------------------------------------------------------------------- This is a heads up to anyone upgrading to CFEngine 3.24.0 or newer versions, about a small change that can be considered a breaking change. In CFEngine versions prior to 3.24.0, the **CFEngine roles** inventory attribute showed up in Mission Portal with the value `policy_server` for the hub, and as _(Not reported)_ for all the other hosts (clients). The technical reason for this was that the `policy_server` class was tagged with `inventory,attribute_name=CFEngine roles`. Now, (in 3.24.0+) the **CFEngine roles** attribute will show the roles **Client**, **Policy server**, and **Reporting hub**, as shown and explained below. ## Before and after Before we get into the details, a couple of pictures should help visualize the change. **Inventory report in 3.21.5 (latest LTS before 3.24.0), before the change:** {{< img "images/roles-before.png" "Inventory report showing two hosts with roles: Not reported, and policy_server">}} **Inventory report in 3.24.0, after the change:** {{< img "images/roles-after.png" "Inventory report showing two hosts with roles: Client, and Reporting hub, Policy Server">}} ## Rationale The previous behavior was unexpected and not very useful for a number of reasons: * Inventory attributes usually have a more human readable form, so **Policy server** would be more expected. * _(Not reported)_ could either mean a client, or a hub which has not reported it _yet_ - there is no good way to distinguish the two. * Should **policy_server** imply that it's an Enterprise reporting hub? Intuitively we know that in a Community setup, there is a policy server which is not a reporting hub. ## Specifics changed To address these issues, we have made some changes to this inventory attribute, and the underlying classes / variables it's based on: * The **CFEngine roles** inventory attribute is now based on a new string list policy variable: `sys.cfengine_roles`. * Includes `Policy server`, if the `policy_server` class is set (host bootstrapped to itself). * Includes `Reporting hub`, if the `cf-hub` binary exists in the expected location (CFEngine Enterprise hub package was installed). * Otherwise, the list has one element: `Client`. * All hosts report this new variable / inventory attribute, so you can now distinguish between the different options and not reported yet. * The `policy_server` class is defined as before (except for its inventory tags) - policy which uses the `policy_server` class still works as expected. * In order to make decisions in policy based on the CFEngine role, two classes were added: * `cfengine_reporting_hub` is defined when `Reporting hub` is in `sys.cfengine_roles`. * `cfengine_client` is defined when `Client` is in `sys.cfengine_roles`. Thus, if you use the **CFEngine roles** inventory attribute in Mission Portal (for example for filtering hosts or compliance report conditions), you will need to adjust those conditions. ## Detection Running this query on the hub can help you find inventory reports which use the **CFEngine roles** attribute and thus might need to be updated: ```command sudo -u cfpostgres /var/cfengine/bin/psql cfmp -c "SELECT id, label, username FROM report WHERE advancedreportsdata::text ~* 'filters.*CFEngine roles.*columns';" ``` ```output id | label | username ----+--------------+---------- 22 | report title | admin (1 row) ``` Similarly, to find other conditions which use the same attribute (in compliance reports or alerts), this query can be used: ```command sudo -u cfpostgres /var/cfengine/bin/psql cfmp -c "SELECT id, name, username FROM dashboard_rules WHERE inventoryconditions::text ~* 'filters.*CFEngine roles.*columns';" ``` ```output id | name | username ----+------+---------- 13 | test | admin (1 row) ``` If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "CFEngine 3.24 LTS released - Consistency" url: /blog/2024/cfengine-3-24-lts-released-consistency/index.md date: "2024-07-22" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.24.0\! The code word for this release is _consistency_. This release also marks an important event, the beginning of the 3.24 LTS series, which will be supported for 3 years. Several new features have been added since the release of CFEngine 3.21 LTS, in the form of non-LTS releases. In this blog post we'll highlight the most important features since the previous LTS release, even though some of them technically landed in intermediate non-supported releases. ## What's new ### 3.24 sneak peek webinar episode In our latest episode of "The agent is in" webinar series, we looked at some of the new features coming in 3.24: {{< youtube nfZcVKF1zS0 >}} ### Groups in Mission Portal In [3.22](/blog/2023/cfengine-3-22-released-coordination), we introduced the groups feature in Mission Portal. This feature allows you to split up your infrastructure into groups based on operating system, CFEngine version, environment, purpose, or any other information you include in the inventoried information about each machine. By default, Mission Portal comes with some groups like the **Ubuntu** group of all hosts running Ubuntu operating systems; {{< wideimg "/blog/2023/images/groups.png" "Groups application showing a group of Ubuntu hosts across different versions.">}} ### CMDB and group data The group feature extends the CMDB functionality, letting you assign variables and classes to groups of hosts, rather than just individual hosts; {{< wideimg "/blog/2023/images/group-data.png" "UI for editing group data, showing the Production group with the class to enable autorun and a list of files to delete as a variable.">}} ### Quality of life and requested improvements to groups Since [its introduction in 2022](/blog/2022/cfengine-3-22-released-coordination), we've had customers testing and giving us feedback about the groups feature, and made several improvements since then, including: - From the inventory report, you can now select a group to filter by, or save your current filter as a group. - The host info page now shows a list of shared groups the host is a member of, with links to jump to those groups. - The host info page also has a new **Show JSON** button for showing the merged JSON file (which combines group data and host specific data), giving you a complete view of exactly which variables and classes end up on the host. - There is now drag and drop support for reordering the list of shared groups (which affects their priority when merged). ### Compliance report scalability Compliance reports with many checks previously took a long time to load, as the checks were run almost one after another (very limited parallelism). We've added more parallelism and caching to compliance reports, allowing you to build bigger reports without being slowed down by long loading times. The [`compliance-report-lynis` module from CFEngine Build](https://build.cfengine.com/modules/compliance-report-lynis/) will benefit greatly from this, along with any similar compliance report modules we make in the future. When testing the changes, we saw a **~5x performance increase** (same report loaded in 1/5 of the time). ### Breaking changes From time to time, we need to change the behavior of certain features of CFEngine in a way that is not completely backwards compatible. This is usually to address security concerns, bugs causing issues for our users, or features behaving in really unexpected or unintended ways. In such cases, we try to communicate the changes with users, along with what is needed from them (such as adjusting their policy, reports, or API usage). Please take a look at the recent `change-in-behavior` blog posts, and stay up to date on these via our blog: https://cfengine.com/tags/change-in-behavior/ ### Other changes There are other big changes introduced in the past 1.5 years, but not shown here. We encourage you to read our previous release blog posts to see more of these changes: - [CFEngine 3.22 - Collaboration](/blog/2023/cfengine-3-22-released-coordination/) - [CFEngine 3.23 - Anniversary](/blog/2022/cfengine-3-23-released-anniversary/) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.24.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-core.html) - [3.24.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-enterprise.html) - [3.24.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.24/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) If you are upgrading from the 3.21 LTS series, scroll down in the changelog to find changes from the earlier 3.22 and 3.23 releases. Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the recently released [3.21.5](/blog/2024/cfengine-3-18-8-and-3-21-5-released.md), these dependencies have been updated: | CFEngine version | 3.21.5 | 3.24.0 | | --------------------------------------------------------------------------------- | ------ | ------ | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.45.1 | 2.45.2 | | [libcurl](https://curl.haxx.se/download.html) | 8.7.1 | 8.8.0 | | [libexpat](https://libexpat.github.io/) | - | 2.5.0 | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.12.6 | 2.13.1 | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.32 | 0.9.33 | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.7 | 2.6.8 | | [OpenSSL](https://openssl.org/) | 3.0.13 | 3.3.1 | | [PCRE2](https://github.com/PCRE2Project/pcre2/releases/) | - | 10.44 | | [PCRE](https://www.pcre.org/) | 8.45 | - | | [PHP](https://php.net/) | 8.2.19 | 8.3.8 | | [PostgreSQL](https://www.postgresql.org/) | 15.6 | 16.3 | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Platform support As mentioned when we announced the latest patch releases, support for Ubuntu 24, both ARM64 and x86-64, has arrived. This naturally applies to 3.24.0, as well. In order to allow us to focus our resources on the platforms most widely used by our users, we are dropping support for older operating systems which are not actively supported by their vendors (outside of paid extended support programs): - Debian 9 (Ended July 1st 2022) - Debian 10 (Ended June 30th 2024) - Ubuntu 16 (Ended April 2nd 2021) - Ubuntu 18 (Ended May 31st 2023) - Windows Server 2008 (Ended January 14th 2020) - CoreOS (Discontinued May 26th 2020) As always, if you need support for other operating systems than what we currently support (including older operating systems), please [contact us](/contact) and we can work on this for you. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, it will now default to 3.24.0, since this is the latest LTS release available. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "Feature Friday #19: What variables and classes are defined?" url: /blog/2024/feature-friday-19-what-variables-and-classes-are-defined/index.md date: "2024-07-19" -------------------------------------------------------------------------------- Do you know how to quickly see what variables and classes are defined? Often, while developing CFEngine policy it's convenient to emit a variable value or a report based on the presence of a class. For example: ```cfengine3 bundle agent main { reports: "Unqualified hostname = '$(sys.uqhost)'"; linux:: "I am running on linux"; } ``` In some cases, this is because you are exploring what classes are available. In other cases, it might be DEBUG-related reports helping you understand how a variable is resolved during policy evaluation. Often, you can get this information without writing CFEngine policy to emit it by using the `--show-vars` or `--show-classes` options to `cf-promises`. You can even filter the output. For example, here we show vars matching `sys.uqhost` and classes matching `linux`. ```command cf-promises --show-vars=sys.uqhost --show-classes=linux ``` ```output Class name Meta tags Comment compiled_on_linux_gnu source=agent,hardclass linux inventory,attribute_name=none,source=agent,derived-from=sys.class,hardclass linux_6_1_72_060172_generic inventory,attribute_name=none,source=agent,derived-from=sys.sysname,derived-from=sys.release,hardclass linux_x86_64 source=agent,derived-from=sys.sysname,derived-from=sys.machine,hardclass linux_x86_64_6_1_72_060172_generic source=agent,derived-from=sys.sysname,derived-from=sys.machine,derived-from=sys.release,hardclass linux_x86_64_6_1_72_060172_generic__202401101633_SMP_PREEMPT_DYNAMIC_Wed_Jan_10_16_59_46_UTC_2024 source=agent,derived-from=sys.long_arch,hardclass specific_linux_os source=promise Variable name Variable value Meta tags Comment default:sys.uqhost precision-5570 inventory,source=agent,attribute_name=none ``` Now, `cf-promises` does not perform a "full" policy execution. Hence, variables or classes getting defined during the policy evaluation, will not get resolved. In those cases, you might prefer to use the `--show-evaluated-vars` and `--show-evaluated-classes` options to `cf-agent`. It works similarly to the related options to `cf-promises` but the variables and classes are emitted after the agent completes policy execution. Also, note that only `namespace` (aka. globally scoped) classes are visible in this output. Ephemeral `bundle` scoped classes will not be shown. This example illustrates that: ```cfengine3 {file="/tmp/feature-friday-19.cf"} bundle agent main { vars: "MyString" string => "MyValue"; "MyList" slist => { "one", "two" }; classes: "bundle_scoped_class" expression => "any", scope => "bundle"; "namespace_scoped_class" expression => "any", scope => "namespace"; } ``` Running it, we filter for variables starting with `My` in the `main` bundle of the `default` namespace and classes ending in `_scoped_class`: ```command cf-agent -Kf /tmp/feature-friday-19.cf \ --show-evaluated-vars=default:main\.My \ --show-evaluated-classes=.*_scoped_class ``` ```output Class name Meta tags Comment namespace_scoped_class source=promise Variable name Variable value Meta tags Comment default:main.MyList {"one","two"} source=promise default:main.MyString MyValue source=promise ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #18: Augments - def.json" url: /blog/2024/feature-friday-18-augments-def-json/index.md date: "2024-07-12" -------------------------------------------------------------------------------- Ever want to get some data into CFEngine? Have you heard about the `def.json` Augments file?[^fn:1] Augments are JSON data files that allow you to define classes and variables very early during agent initialization, before policy. Augments are practical in a variety of ways. Perhaps most notably for overriding policy defaults. Let's look at a simple example. Here we have a policy `/tmp/feature-friday/18-0.cf` ```cfengine3 {file="/tmp/feature-friday/18-0.cf"} bundle agent main { reports: "MyVariable $(with) defined '$(MyVariable)'" with => ifelse( isvariable( MyVariable ), "is", "is not" ); } ``` Running it, we can see that `MyVariable` isn't defined ```command cf-agent -Kf /tmp/feature-friday/18-0.cf ``` ```output R: MyVariable is not defined '$(MyVariable)' ``` Now, let's create an Augments file, `def.json`, to define the variable and place it in `/tmp/feature-friday` (next to the policy entry). ```json { "variables": { "default:main.MyVariable": "defined in augments" } } ``` By executing the policy again, we can see the effect: ```command cf-agent -Kf /tmp/feature-friday/18-0.cf ``` ```output R: MyVariable is defined 'defined in augments' ``` Great, it's defined, but how can this be utilized to _override_ policy? Well, it can only override the policy if the policy accepts it. Let's see what happens if we define the variable in the policy. Here, we have modified the policy as `/tmp/feature-friday/18-1.cf`: ```cfengine3 {file="/tmp/feature-friday/18-1.cf"} bundle agent main { vars: "MyVariable" string => "defined in policy"; reports: "MyVariable $(with) defined '$(MyVariable)'" with => ifelse( isvariable( MyVariable ), "is", "is not" ); } ``` Running it, we see `MyVariable` holds `defined in policy`: ```command cf-agent -Kf /tmp/feature-friday/18-1.cf ``` ```output R: MyVariable is defined 'defined in policy' ``` Now we have a variable defined in policy, and policy always wins. So, we can readily define that variable in policy - conditionally - based on whether it's already defined. Here we have the modified policy as `/tmp/feature-friday/18-2.cf`: ```cfengine3 {file="/tmp/feature-friday/18-2.cf"} bundle agent main { vars: "MyVariable" string => "defined in policy", if => not( isvariable( MyVariable ) ); reports: "MyVariable $(with) defined '$(MyVariable)'" with => ifelse( isvariable( MyVariable ), "is", "is not" ); } ``` Now executing the policy, we see that the variable holds `defined in augments` as defined in `def.json`. ```command cf-agent -Kf /tmp/feature-friday/18-2.cf ``` ```output R: MyVariable is defined 'defined in augments' ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: -------------------------------------------------------------------------------- title: "Feature Friday #17: Tags for inventory and reporting" url: /blog/2024/feature-friday-17-tags-for-inventory-and-reporting/index.md date: "2024-07-05" -------------------------------------------------------------------------------- Let's talk about tags and how they can be useful for Inventory and Reporting. If you have been following along with the Feature Friday series you already heard about using tags to find currently defined classes, variables and bundles, but they are also very useful for reporting. In CFEngine Enterprise the `inventory` and `attribute_name` tags are _special_. A variable or class tagged with `inventory` becomes visible in the Inventory subsystem in Mission Portal with the name given in the `attribute_name` tag. For example, if we define a string variable and tag it with `inventory` and `attribute_name=Responsible Person`: ```cfengine3 vars: "owner" string => "Nick Anderson", meta => { "inventory", "attribute_name=Responsible person" }; ``` `Responsible Person` will show up in Mission Portal as Inventory for the host: {{< figure src="2024-02-23_13-04-25_Screenshot_2024-02-23_130042.png" >}} {{< figure src="2024-02-23_13-12-54_Screenshot_2024-02-23_131213.png" >}} Outside of Enterprise, you can leverage tags in creating your own reports, like the host info report in the Masterfiles Policy Framework which includes `inventory` tagged variables sourced by `variablesmatching_as_data()`. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Install and uninstall packages based on conditions" url: /blog/2024/conditional-installer/index.md date: "2024-07-01" -------------------------------------------------------------------------------- For security reasons, you generally want to uninstall `talk`, `samba`, and `apache2` in your infrastructure. However, on your webservers, which have the `webserver` CFEngine class defined, you might want Apache to be installed. With the [conditional-installer](https://build.cfengine.com/modules/conditional-installer/) module, you can put `talk`,`samba`,`apache2` in the list of packages to uninstall. And in the list of packages to install, you can put `apache2` with the condition `webserver`. Hence, the module will install `apache2` on your webservers and uninstall it everywhere else. `talk` and `samba`, on the other hand, will be uninstalled everywhere. As always with CFEngine, if the state is already correct - i.e., packages that should be installed are already installed and packages that should not be installed are not installed - no actions will be performed. When we use the Mission Portal Build application we can add this module and edit data to specify what we want to happen. First, search for "conditional-installer," click on the entry found from searching, and then the Edit button. {{< wideimg "images/conditional-installer-inputs.png" "conditional-installer inputs edited">}} Notice that we specify `data:webserver` instead of just `webserver`. This is because we want to use host-specific data, which defaults to the namespace `data`. After that, we can visit two hosts to add the `webserver` class: {{< wideimg "images/conditional-installer-x220-is-webserver.png" "adding webserver class to a Debian host" >}} {{< wideimg "images/conditional-installer-rhel8-is-webserver.png" "adding webserver class to a RHEL 8 host" >}} We can then test package removal by visiting `x220` (my Debian host): ```txt root@x220:~# apt install telnet root@x220:~# command -v telnet /usr/bin/telnet root@x220:~# echo $? 0 root@x220:~# cf-agent -KI info: Successfully removed package 'telnet' root@x220:~# command -v telnet root@x220:~# echo $? 1 ``` And then check on our webserver installations. First on Debian: ```sh root@x220:~# apache2ctl -v Server version: Apache/2.4.59 (Debian) Server built: 2024-04-05T12:08:04 ``` And on RHEL-8: ```sh root@rhel8:~# httpd -v Server version: Apache/2.4.37 (Red Hat Enterprise Linux) Server built: Feb 16 2024 04:23:20 ``` Looks good! Now, you can proceed by specifying more packages to install and uninstall based on how you classify your hosts with CFEngine classes. With our upcoming 3.24.0 release, our new Groups feature allows you to further refine how to specify vars and classes data on host records. {{< wideimg "images/conditional-installer-webservers-shared-group.png" "Groups app in Mission Portal showing Webservers group">}} Oops! Looks like I have 15 unkept promises, according to our Policy Analyzer. I better hop over there and get busy fixing those issues. For a more detailed preview of this functionality, check out our latest [The agent is in episode 38](https://cfengine.com/blog/2024/the-agent-is-in-episode-38/). ## Questions? If you have questions or need help, don't hesitate to reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #16: host info report" url: /blog/2024/feature-friday-16-host-info-report/index.md date: "2024-06-28" -------------------------------------------------------------------------------- Did you know the Masterfiles Policy Framework (MPF) ships with a host info report? That's right, you can simply run `cf-agent --bundlesequence host_info_report` and a report will be generated. ```command cf-agent --bundlesequence host_info_report ``` ```output R: Host info report generated and available at '/var/cfengine/reports/host_info_report.txt' ``` It's packed with information about the specific host. Let's peek: ```command head -n 9 /var/cfengine/reports/host_info_report.txt ``` ```output # Host Information Generated: Fri Feb 23 19:54:13 2024 ## Identity Fully Qualified Hostname: hub.example.com Host ID: SHA=41ebb680d136f82c57af6ee1a7b938c093fe8d773bf320213eae1c476dad4fb0 ## CFEngine Version: CFEngine Enterprise 3.21.4 ``` Here are the section headers: ```command grep ^# /var/cfengine/reports/host_info_report.txt ``` ```output # Host Information ## Identity ## CFEngine ## OS ## Hardware ## Network ### Interfaces ### IPv4 TCP Ports listening ## Policy ### Inventory #### Variables tagged for inventory ### Enterprise maintanance bundles available ``` Running with the `show_software` class defined will add information about installed packages ```command cf-agent --bundlesequence host_info_report --define show_software ``` ```output R: Host info report generated and available at '/var/cfengine/reports/host_info_report.txt' ``` ```command grep -A3 "^## Software" /var/cfengine/reports/host_info_report.txt ``` ```output ## Software [yum] hypervvssd: version 0-0.34.20180415git.el7, arch x86_64 [yum] grub2-common: version 2.02-0.81.el7.centos, arch noarch [yum] grub2-tools-extra: version 2.02-0.81.el7.centos, arch x86_64 ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 38 - Sneak peek of CFEngine 3.24" url: /blog/2024/the-agent-is-in-episode-38/index.md date: "2024-06-27" -------------------------------------------------------------------------------- Interested to see what's new in the next LTS version (3.24) of CFEngine? Nick joins Craig and Cody to see what's coming in 3.24. From the new groups feature which allows you to assign data to a group of hosts to improvements in filtering and new functionality in Build and other changes in behavior, checkout the video for all the details. ## Video {#video} The video recording is available on YouTube: {{< youtube nfZcVKF1zS0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - Changes in behavior since last LTS - [Self upgrade targets hub binary version by default](https://cfengine.com/blog/2023/change-in-behavior-self-upgrade-targets-hub-binary-version-by-default/) - [`ignore_interfaces.rx` has a new preferred location](https://cfengine.com/blog/2023/change-in-behavior-ignore_interfaces.rx-prefers-new-location/) - [The `arglist` attribute in _files_ promises now preserves spaces](https://cfengine.com/blog/2024/change-in-behavior-arglist/) - [`depth_search` can now be used (but warns) with an individual file as source](https://cfengine.com/blog/2024/change-in-behavior-depth-search-can-now-be-used-but-warns-with-an-individual-file-as-source/) - Post show chit-chat - Will there be another Basic Enterprise training? - Yes, but it's not scheduled yet. _Maybe_ early November. - Challenges with managing desired state with partial knowledge - `replace_patterns` in `edit_line` promises -------------------------------------------------------------------------------- title: "Feature Friday #15: bundlesmatching()" url: /blog/2024/feature-friday-15-bundlesmatching/index.md date: "2024-06-21" -------------------------------------------------------------------------------- Did you know _bundles_ can have tags too? That's right! You can tag a bundle by defining `tags` as a `meta` _promise_ on a bundle. For example: ```cfengine3 bundle agent example_bundle_tag { meta: "tags" slist => { "tag_1", "tag_2" }; } ``` You've likely encountered bundles tagged with `autorun`. These tags trigger automatic execution of bundles in lexical order whenever the `services_autorun` class is defined. However, you're not limited to `autorun`. You can create custom tags to suit your specific needs. Perhaps you want to tag bundles associated with a particular compliance framework or identify the primary developer/team responsible for maintenance. ```cfengine3 bundle agent example_bundle_tags { meta: "tags" slist => { "security", "maintainer=Nick Anderson" }; } ``` The tags can function as documentation, but you can also use them to build your own - specialized - autorun policy. For example, to run all the security bundles. ```cfengine3 {file="/tmp/feature-friday-15.cf"} bundle agent sec_2 { meta: "tags" slist => { "security", "stig", "maintainer=Nick Anderson" }; reports: "$(this.bundle) implements a STIG."; } bundle agent sec_1 { meta: "tags" slist => { "security", "cis", "maintainer=Nick Anderson" }; reports: "$(this.bundle) implements a CIS benchmark."; } bundle agent security_bundles { vars: "bundles" slist => sort( bundlesmatching( ".*", "security" ), lex); methods: "$(bundles)"; } bundle agent __main__ { methods: "security_bundles"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-15.cf ``` ```output R: sec_1 implements a CIS benchmark. R: sec_2 implements a STIG. ``` Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "CFEngine 3.18.8 and 3.21.5 released" url: /blog/2024/cfengine-3-18-8-and-3-21-5-released/index.md date: "2024-06-20" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.18.8 and 3.21.5\! These patch releases contain bug fixes and dependency updates. ## Changes We'd like to highlight one specific change in behavior, which some users will want to adjust to; ### Change in behavior - depth_search can now be used (but warns) with an individual file as source Users of the `depth_search` attribute of file promises should be aware of this change in behavior. See the blog post on the topic for more details: https://cfengine.com/blog/2024/change-in-behavior-depth-search-can-now-be-used-but-warns-with-an-individual-file-as-source/ ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.21.5 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.5 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.5 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.18.8 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.8 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.8 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades Compared to the previous releases, CFEngine [3.18.7 and 3.21.4](https://cfengine.com/blog/2023/cfengine-3-18-7-and-3-21-4-released/), the following dependencies were updated for the client packages: | CFEngine version | 3.18.7 | **3.18.8** | 3.21.4 | **3.21.5** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [libcurl](https://curl.haxx.se/download.html) | 8.5.0 | **8.7.1** | 8.5.0 | **8.7.1** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.12.3 | **2.12.6** | 2.12.3 | **2.12.6** | | [OpenSSL](https://openssl.org/) | 1.1.1w | 1.1.1w | 3.0.12 | **3.0.13** | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.3.1 | **2.3.2** | 2.3.1 | **2.3.2** | | [libattr](https://download.savannah.gnu.org/releases/attr/) | 2.5.1 | **2.5.2** | 2.5.1 | **2.5.2** | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.31 | **0.9.32** | 0.9.31 | **0.9.32** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.6 | **2.6.7** | 2.6.6 | **2.6.7** | | [zlib](https://www.zlib.net/) | 1.3 | **1.3.1** | 1.3 | **1.3.1** | And for the hub packages, these additional dependencies were also upgraded: | CFEngine version | 3.18.7 | **3.18.8** | 3.21.4 | **3.21.5** | | :-------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.58 | **2.4.59** | 2.4.58 | **2.4.59** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.43.0 | **2.45.1** | 2.43.0 | **2.45.1** | | [PHP](https://php.net/) | 8.0.30 | **8.0.30** | 8.2.13 | **8.2.19** | | [PostgreSQL](https://www.postgresql.org/) | 13.13 | **13.14** | 15.5 | **15.6** | | [rsync](https://download.samba.org/pub/rsync/) | 3.2.7 | **3.3.0** | 3.2.7 | **3.3.0** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.21/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## New supported platform - Ubuntu 24 Ubuntu 24 is now supported, with 4 enterprise packages available for the platform; 1. Ubuntu 24 hub package for 64-bit x86 (amd64) 2. Ubuntu 24 hub package for 64-bit ARM (arm64) 3. Ubuntu 24 client package for 64-bit x86 (amd64) 4. Ubuntu 24 client package for 64-bit ARM (arm64) See the **Downloads** section below for more details on how to download and install them. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.21.5 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.18.8 list ``` ```command cf-remote --version 3.21.5 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.18 LTS Please note that support for 3.18.8 ends in August (2024), and this 3.18.8 is the last planned release for the 3.18.x LTS series. If you still haven't upgraded to 3.21 LTS, we encourage you to [contact us](/contact) for help with upgrading or to discuss an extended support contract. **Edit:** Regular support for CFEngine 3.18 has now ended. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Change in behavior: depth_search can now be used (but warns) with an individual file as source" url: /blog/2024/change-in-behavior-depth-search-can-now-be-used-but-warns-with-an-individual-file-as-source/index.md date: "2024-06-19" -------------------------------------------------------------------------------- You may see a new warning in the upcoming releases of [3.21.5](/blog/2024/cfengine-3-18-8-and-3-21-5-released/) and 3.24.0. A new warning was introduced with a fix to the behavior of `depth_search` when combined with a `copy_from` source targeting a _file_: ``` depth_search (recursion) is promised for a base object '' that is not a directory ``` Prior to versions 3.21.5 and 3.24.0 CFEngine would copy the file initially but would subsequently avoid updating the file providing only a debug log message about the fact that the file was being skipped (because it's not possible to descend into a file). Beginning with 3.21.5 and 3.24.0 CFEngine will copy the file and properly update the file but will also emit a warning that recursion was promised for something that was not a directory. If you see these warnings consider adjusting your policy so that you do not promise `depth_search` with a non-directory source. For example, consider this policy which implements an agent bundle abstracting the copying of a single file or directory tree recursively. ```cfengine3 bundle agent copy_remote_file_or_dir(src, dest) # @brief Make `dest` a copy of `src`. # # This bundle copies recursively. If a single file is # specified only the single file is copied. If a directory # is specified that directory is copied recursively. # #@ **Examples:** #@ "Copy a single file" #@ usebundle => copy_remote_file_or_dir( #@ "/var/cfengine/masterfiles/promises.cf", #@ "/tmp/dest-test/dest_file.txt"); #@ #@ "Copy a directory tree" #@ usebundle => copy_remote_file_or_dir( #@ "/var/cfengine/masterfiles/.", #@ "/tmp/dest-dir/."); { files: "$(dest)" copy_from => remote_dcp_missing_ok($(src), $(sys.policy_hub)), depth_search => recurse_with_base(inf), classes => results("namespace", "$(src)__file_from_policy_hub"); } body copy_from remote_dcp_missing_ok(source_path,source_server) # @brief Copy resources from a remote server if the content hash differs from the local copy. # # @param `source_path` The location of the file on the remote server # @param `source_server` The hostname or IP of the server from which to download # # **See Also:** `local_dcp()` { servers => { "$(source_server)" }; source => "$(source_path)"; compare => "digest"; missing_ok => "false"; } ``` Using the above policy, you could trigger the old behavior if the source is actually a directory but not denoted as one with a trailing slash dot (`/.`). For example: ```cfengine3 methods: "Copy ambiguous dir to dir" usebundle => copy_remote_file_or_dir( "/src/source/file.txt", "/tmp/file.txt"); ``` With the old behavior, the contents of the source file (`/src/source/file.txt`) would be copied locally on the first agent execution, but changes to that file would not be realized and transmitted in the future. With the new behavior, the agent will copy the file locally on the first agent execution and any changes to the source file will be realized and updates will be transmitted on subsequent agent executions but a warning will be emitted: ```text warning: depth_search (recursion) is promised for a base object '/tmp/file.txt' that is not a directory ``` For more information, or if you find this change problematic in some way see [CFE-4403](https://northerntech.atlassian.net/browse/CFE-4403). -------------------------------------------------------------------------------- title: "Feature Friday #14: variablesmatching() & variablesmatching_as_data()" url: /blog/2024/feature-friday-14-variablesmatching-variablesmatching-as-data/index.md date: "2024-06-14" -------------------------------------------------------------------------------- Did you know you can find variables by name and tag? Like the ability to find currently defined classes (as described in [Feature Friday #13: `classesmatching()`](/blog/2024/feature-friday-13-classesmatching)) that match a name or tag, you can find variables by name and tag. It's a nifty capability. `variablesmatching()` returns a list of variable names that match the name and tag criteria.[^fn:1] `variablesmatching_as_data()` returns a data container of the matching variables along with their values[^fn:2]. Let's take a look at an example of installing packages. **NOTE:** Many different patterns can be used for managing packages. Sometimes, they are managed in a single place. Other times, across multiple bundles. The following example is not intended as a recommendation for any specific pattern. It's just to illustrate possibilities. The following bundle defines three lists of packages. Rather than making `packages` promises for each individually, we collect them - all together - into a single data structure from which we can derive a single list of package names to manage. ```cfengine3 {file="/tmp/feature-friday-14/example-0.cf"} bundle agent package_names_from_variablesmatching_as_data # @brief Example illustrating deriving a list of package names from variables named with a specific naming convention { vars: "packages_baseline" slist => { "bash-completion", "emacs" }; "packages_os" slist => { "httpd" }; "packages_barf" slist => { "apache" }; "_mypkgdata" data => variablesmatching_as_data( ".*\.packages_.*" ); "_package_names" slist => getvalues( _mypkgdata ); reports: '$(_package_names)'; } bundle agent __main__ { methods: "package_names_from_variablesmatching_as_data"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-14/example-0.cf ``` ```output R: bash-completion R: emacs R: httpd R: apache ``` Or, you might prefer to use tags instead of variable names. This allows us to collect information defined across multiple variables, which might even be defined in different bundles or namespaces. ```cfengine3 {file="/tmp/feature-friday-14/example-1.cf"} bundle agent package_names_from_variablesmatching_as_data # @brief Example illustrating deriving a list of package names from variables having a specific tag { vars: "packages_baseline" meta => { "desired_package_names=present" }, slist => { "bash-completion", "emacs" }; "packages_os" meta => { "desired_package_names=present" }, slist => { "httpd" }; "packages_barf" meta => { "desired_package_names=present" }, slist => { "apache" }; "_mypkgdata" data => variablesmatching_as_data( ".*", "desired_package_names=present" ); "_package_names" slist => getvalues( _mypkgdata ); reports: '$(_package_names)'; } bundle agent __main__ { methods: "package_names_from_variablesmatching_as_data"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-14/example-1.cf ``` ```output R: bash-completion R: emacs R: httpd R: apache ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: Documentation for `variablesmatching()` [^fn:2]: Documentation for `variablesmatching_as_data()` -------------------------------------------------------------------------------- title: "Feature Friday #13: classesmatching()" url: /blog/2024/feature-friday-13-classesmatching/index.md date: "2024-06-07" -------------------------------------------------------------------------------- Did you know you can find classes by name and tag? `classesmatching()` dynamically sources information from the current state. For example, let's say you have classes representing a system's role. Furthermore, let's say that we want a host to only have a single role class defined. Finally, if we have more than one role class defined, then we don't want to proceed. To achieve this without `classesmatching()`, we might have a policy file that looks like this (`/tmp/feature-friday-13/tags-on-classes-0.cf`) ```cfengine3 {file="/tmp/feature-friday-13/tags-on-classes-0.cf"} bundle agent __main__ { classes: "have_role" expression => "role_1|role_2|role_3"; "too_many_roles" expression => "(role_1.(role_2|role3))|(role_2.(role_1|role3))|role_3.(role_2|role_3)"; "no_roles" expression => "!(role_1|role_2|role_3)"; reports: have_role:: "I have a role"; role_1:: "I have role_1"; role_2:: "I have role_2"; role_3:: "I have role_3"; too_many_roles:: "Uh oh! I have more than 1 role class defined."; no_roles:: "Uh oh! I don't have any role classes defined."; } ``` We can run the policy defining different variations, checking that the various cases are covered. With no role classes: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-0.cf ``` ```output R: Uh oh! I don't have any role classes defined. ``` With all possible role classes: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-0.cf --define role_1,role_2,role_3 ``` ```output R: I have a role R: I have role_1 R: I have role_2 R: I have role_3 R: Uh oh! I have more than 1 role class defined. ``` With multiple role classes: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-0.cf --define role_1,role_3 ``` ```output R: I have a role R: I have role_1 R: I have role_3 R: Uh oh! I have more than 1 role class defined. ``` With a single role class defined: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-0.cf --define role_1 ``` ```output R: I have a role R: I have role_1 ``` This can be much more clear and maintainable if implemented using `classesmatching()`: ```cfengine3 {file="/tmp/feature-friday-13/tags-on-classes-1.cf"} bundle agent __main__ { vars: "role_classes" slist => classesmatching("role_.*"); classes: "have_role" expression => isgreaterthan(length(role_classes), 0); "too_many_roles" expression => isgreaterthan(length(role_classes), 1); "no_roles" expression => strcmp(length(role_classes), 0); reports: have_role:: "I have a role"; "I have $(role_classes)" if => "$(role_classes)"; too_many_roles:: "Uh oh! I have more than 1 role class defined."; no_roles:: "Uh oh! I don't have any role classes defined."; } ``` We can run it through the same checks we did previously and see that the result is identical. However, we no longer have to maintain expressions for each combination. All we need to do is to follow our role class naming convention. First with no role classes defined: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-1.cf ``` ```output R: Uh oh! I don't have any role classes defined. ``` With all possible role classes: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-1.cf --define role_1,role_2,role_3 ``` ```output R: I have a role R: I have role_1 R: I have role_3 R: I have role_2 R: Uh oh! I have more than 1 role class defined. ``` With multiple role classes: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-1.cf --define role_1,role_3 ``` ```output R: I have a role R: I have role_1 R: I have role_3 R: Uh oh! I have more than 1 role class defined. ``` With a single role class defined: ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-1.cf --define role_1 ``` ```output R: I have a role R: I have role_1 ``` Sometimes, naming conventions aren't enough. Sometimes we want a bit more. Well, `classesmatching()` can also be searched by _tag_. Let's refactor the above example to not care about the class name. Instead, let's care about having more than one class defined with the same tag. ```cfengine3 {file="/tmp/feature-friday-13/tags-on-classes-3.cf"} bundle agent __main__ { vars: "role_classes" slist => classesmatching(".*", "role_class"); classes: "have_role" expression => isgreaterthan(length(role_classes), 0); "too_many_roles" expression => isgreaterthan(length(role_classes), 1); "no_roles" expression => strcmp(length(role_classes), 0); reports: have_role:: "I have a role"; "I have $(role_classes)" if => "$(role_classes)"; too_many_roles:: "Uh oh! I have more than 1 role class defined."; no_roles:: "Uh oh! I don't have any role classes defined."; } ``` Unfortunately, we can't specify tags for a class when using the `--define` option as we did previously, so we will modify the policy to define the classes used. ```cfengine3 {file="/tmp/feature-friday-13/tags-on-classes-3.cf"} bundle common classification { classes: "Rick" expression => "any", meta => { "role_class" }; "Astley" expression => "any", meta => { "role_class" }; "Never_Gonna_Give_You_Up" expression => "any", meta => { "role_class" }; } bundle agent __main__ { vars: "role_classes" slist => classesmatching(".*", "role_class"); classes: "have_role" expression => isgreaterthan(length(role_classes), 0); "too_many_roles" expression => isgreaterthan(length(role_classes), 1); "no_roles" expression => strcmp(length(role_classes), 0); reports: have_role:: "I have a role"; "I have $(role_classes)" if => "$(role_classes)"; too_many_roles:: "Uh oh! I have more than 1 role class defined."; no_roles:: "Uh oh! I don't have any role classes defined."; } ``` ```command cf-agent -Kf /tmp/feature-friday-13/tags-on-classes-3.cf ``` ```output R: I have a role R: I have Astley R: I have Never_Gonna_Give_You_Up R: I have Rick R: Uh oh! I have more than 1 role class defined. ``` Take it for a spin, and see how this methodology can be applied in your own environment. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Use Ansible playbooks in CFEngine policy with promise-type-ansible module" url: /blog/2024/promise-type-ansible/index.md date: "2024-06-03" -------------------------------------------------------------------------------- Whether you are migrating from Ansible to CFEngine to gain some of the benefits of [scale](https://cfengine.com/blog/2021/ansible-and-cfengine-scalability-white-paper) or [autonomy](https://cfengine.com/blog/2021/comparing-ansible-and-cfengine/) or just need some functionality in an Ansible module, the [ansible promise type](https://build.cfengine.com/modules/promise-type-ansible/) can be a great tool to utilize. It also provides a compelling alternative to [ansible-pull](https://docs.ansible.com/ansible/latest/cli/ansible-pull.html) and works around some of the caveats included with that strategy. CFEngine has battle-tested features needed for the pull architecture: - [cf-execd](https://docs.cfengine.com/docs/master/reference-components-cf-execd.html) handles scheduling periodic runs as ansible-pull suggests using cron - [cf-agent](https://docs.cfengine.com/docs/master/reference-components-cf-agent.html) handles locking to avoid concurrent runs of the same playbooks ## A tiny Ansible project example Taking some first-step tips from [5 ways to harden a new system with Ansible](https://www.redhat.com/sysadmin/harden-new-system-ansible) let's make a sample playbook project which patches Linux systems. Ansible has some good documentation about [best practices](https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.html) and [roles](https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.html) that make it easy to develop a somewhat realistic mini ansible playbook project. For reference, I have shared a [cfbs](https://github.com/cfengine/cfbs) project in the [promise-type-ansible](https://github.com/cfengine/play-cfbs/tree/promise-type-ansible) branch of our repository of example projects at [play-cfbs](https://github.com/cfengine/play-cfbs). The crux of our tiny Ansible project is to patch our hosts. This will be implemented as a patch role in `roles/patch/tasks/main.yml`: ```yaml - name: Perform full patching package: name: '*' state: latest ``` This role is included in our main `site.yml` file as follows: ```yml --- - hosts: managed tasks: - ansible.builtin.debug: msg: first play roles: - patch ``` In this tiny project the `managed` group includes all hosts. ## CFEngine policy to ensure the playbooks are run The next step is to make an inventory file for Ansible which will enable running the playbook on each managed host but using a local connection so that no SSH credentials need be configured. For a typical set of playbooks you would likely have many groups in your inventory. We can map classes in CFEngine to groups in this localhost inventory with some CFEngine policy: ```cf3 classes: "managed" meta => { "ansible_group" }; "webserver" meta => { "ansible_group" }; vars: "localhost_inventory" string => "$(sys.statedir)/localhost_inventory.ini"; "ansible_groups" slist => classesmatching(".*", "ansible_group"); files: "$(localhost_inventory)" create => "true", content => string_mustache("[all:vars] ansible_connection=local {{#-top-}}[{{.}}] localhost {{/-top-}}", "ansible_groups"), classes => if_ok("localhost_inventory_created"); ``` This creates a file we will use later in our policy: ```ini [all:vars] ansible_connection=local [managed] localhost [webserver] localhost ``` We use the built-in group `all` to set `ansible_connection=local` so we can run the playbooks manually using the generated inventory. The `ansible` promise type defaults to this setting. Of course, we need Ansible to be installed on this host, so we use a package promise and set a class to wait until it is installed before trying to run the playbook. ```cf3 packages: "ansible" classes => if_ok("ansible_package_installed"); ``` Finally our policy handles copying the `ansible_project` directory to a nice place in `/var/cfengine/state` and runs `site.yml` the top-level site playbook. ```cf3 vars: "remote_ansible_project" string => string_replace("$(this.promise_dirname)", "inputs", "masterfiles"); "ansible_project" string => "$(sys.statedir)/ansible_project"; files: "$(ansible_project)" copy_from => remote_dcp("$(remote_ansible_project)", @(def.policy_servers)), depth_search => recurse(inf); ansible: ansible_package_installed.localhost_inventory_created:: "site.yml" playbook => "$(ansible_project)/site.yml", inventory => "$(localhost_inventory)"; ``` ## The complete policy ```cf3 {file="ansible_project/main.cf"} bundle agent ansible_main { vars: "ansible_project" string => "$(sys.statedir)/ansible_project"; "localhost_inventory" string => "$(sys.statedir)/localhost_inventory.ini"; "remote_ansible_project" string => string_replace("$(this.promise_dirname)", "inputs", "masterfiles"); classes: # initial thought about how to translate CFEngine classification into Ansible groups classification for dynamic inventory "managed" meta => { "ansible_group" }; "webserver" meta => { "ansible_group" }; vars: "ansible_groups" slist => classesmatching(".*", "ansible_group"); reports: "ansible_groups: ${ansible_groups}"; files: "$(localhost_inventory)" create => "true", content => string_mustache("[all:vars] ansible_connection=local {{#-top-}}[{{.}}] localhost {{/-top-}}", "ansible_groups" ), classes => if_ok("localhost_inventory_created"); "$(ansible_project)" copy_from => remote_dcp($(remote_ansible_project), @(def.policy_servers)), depth_search => recurse(inf); packages: "ansible" classes => if_ok("ansible_package_installed"); ansible: ansible_package_installed.localhost_inventory_created:: "site.yml" playbook => "$(ansible_project)/site.yml", inventory => "$(localhost_inventory)"; } ``` ## Debugging CFEngine and Ansible Debugging problems is a little tricky. You will have to use both your CFEngine and Ansible debugging skills. To get started you can run the policy with debug logging enabled and focus on the bundle in question with ```sh cf-agent -Kdb ansible_main | tee ~/log ``` And to debug the Ansible side run the playbook something like this: ```sh ansible-playbook -i /var/cfengine/state/localhost_inventory.ini /var/cfengine/state/ansible_project/site.yml ``` In the cf-agent debug output will look something like the following: ```sh debug: DeRefCopyPromise(): promiser:'site.yml' debug: DeRefCopyPromise(): copying constraint: 'playbook' debug: DeRefCopyPromise(): copying constraint: 'inventory' verbose: Starting custom promise module '/var/cfengine/modules/promises/ansible_promise.py' with command '/usr/bin/python3 /var/cfengine/modules/promises/ansible_promise.py' debug: Received header from promise module: 'ansible_promise_module 0.2.2 v1 json_based' debug: Received line from module: '{"operation": "validate_promise", "promiser": "site.yml", "attributes": {"inventory": "/var/cfengine/state/localhost_inventory.ini", "playbook": "/var/cfengine/state/ansible_project/site.yml"}, "result": "valid"}' debug: Received line from module: 'log_verbose=Task 'Gathering Facts' started on 'localhost'' verbose: Task 'Gathering Facts' started on 'localhost' debug: Received line from module: 'log_verbose=Task 'Gathering Facts' didn't change' verbose: Task 'Gathering Facts' didn't change debug: Received line from module: 'log_verbose=Task 'Perform full patching' started on 'localhost'' verbose: Task 'Perform full patching' started on 'localhost' debug: Received line from module: 'log_error=Task 'Perform full patching' failed' error: Task 'Perform full patching' failed debug: Received line from module: 'log_verbose=Summary of the tasks for 'localhost' is: ok=1 failures=1' verbose: Summary of the tasks for 'localhost' is: ok=1 failures=1 debug: Received line from module: '{"operation": "evaluate_promise", "promiser": "site.yml", "attributes": {"inventory": "/var/cfengine/state/localhost_inventory.ini", "playbook": "/var/cfengine/state/ansible_project/site.yml"}, "result": "not_kept", "result_classes": ["site.yml_failed"]}' debug: Setting class: default:site_yml_failed verbose: Additional promise info: version 'CFEngine Promises.cf 3.21.4' source path '/var/cfengine/inputs/services/cfbs/ansible_project/main.cf' at line 43 verbose: Promise with promiser 'site.yml' was not kept by promise module '/var/cfengine/modules/promises/ansible_promise.py' verbose: A: Promise NOT KEPT! verbose: P: END ansible promise (site.yml) ``` Let us know how it goes if you try it out! ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #12: special variables" url: /blog/2024/feature-friday-12-special-variables/index.md date: "2024-05-31" -------------------------------------------------------------------------------- Are you familiar with [CFEngine's special variables](https://docs.cfengine.com/latest/reference-special-variables.html)? Probably you are familiar with [sys variables](https://docs.cfengine.com/latest/reference-special-variables-sys.html) like `sys.fqhost` (the fully qualified host name) and `sys.policy_hub` (the IP address of the machine the host is bootstrapped to) but I want to highlight a few other special variables you may not be so familiar with. ## sys {#sys} Sys variables are derived from the system discovery done by the agent as it initializes. ### `sys.os_release` - A data structure derived from /etc/os-release {#sys-dot-os-release-a-data-structure-derived-from-etc-os-release} `/etc/os-release`, introduced by `systemd` provides a nice record of the current distribution's release information.[^fn:1] CFEngine prefers information from this file for determining system classification like the definition of the `redhat` and `debian` classes. The file can also be extended with custom keys, like I have done on my system to set `NORTHERN_TECH_OWNER=Nick Anderson`. Since files information is exposed as a data container in this sys variable it can be useful for influencing policy behavior, like selecting additional Augments to load.[^fn:2] ```cfengine3 bundle agent __main__ { reports: "My custom key 'NORTHERN_TECH_OWNER' contains $(sys.os_release[NORTHERN_TECH_OWNER])"; } ``` ```output R: My custom key 'NORTHERN_TECH_OWNER' contains Nick Anderson ``` ### `sys.policy_entry_filename` - The first policy file read by the agent (policy entry point) {#sys-dot-policy-entry-filename-the-first-policy-file-read-by-the-agent--policy-entry-point} By default when CFEngine runs it tries to load `$(sys.inputdir)/promises.cf` so that will normally be the policy entry but if you override the policy to read first using the `-f` or `--file` option that file will be the policy entry. This variable can be useful for knowing from within policy if we are executing from an expected location. For example here we have `/tmp/feature-friday-sys.policy_entry_filename.cf`: ```cfengine3 {file="/tmp/feature-friday-12/sys.policy_entry_filename.cf"} bundle agent __main__ { reports: "The policy entry is not running from a standard location, instead it's running from $(sys.policy_entry_filename)" unless => and( or( strcmp( "/var/cfengine/inputs/promises.cf", "$(sys.policy_entry_filename)" ), strcmp( "/var/cfengine/inputs/update.cf", "$(sys.policy_entry_filename)" ))); } ``` And running it we see the message that it's not running from a standard location. ```command cf-agent -Kf /tmp/feature-friday-12/sys.policy_entry_filename.cf ``` ```output R: The policy entry is not running from a standard location, instead it's running from /tmp/feature-friday-12/sys.policy_entry_filename.cf ``` ### `sys.policy_entry_dirname` - The directory in which the policy entry exists {#sys-dot-policy-entry-dirname-the-directory-in-which-the-policy-entry-exists} Similar to `sys.policy_entry_filename`, `sys.policy_entry_dirname` expands to the full path to the directory in which the policy entry exists. This can be very useful for referencing additional policy files to load relative to the policy root in a dynamic way. This allows for workflows where you check out a policy set to a temporary location and run it making sure that the policy files loaded are the policy files released to the policy set and not policy files that reside in some standard location. ### `sys.key_digest` - The hash of the executing agent's public key {#sys-dot-key-digest-the-hash-of-the-executing-agents-public-key} `sys.key_digest` can be useful for cases where you have assets that are related to a specific host instantiation. It can easily be paired with `connection.key` in an `access` promise to ensure that only that host identity has access to those files. For example, here we have policy that copies host specific files that are stored in a directory named for the key digest from the policy server which shares the directory based on the connecting key. ```cfengine3 bundle agent __main__ { files: "/tmp/my-files/." copy_from => remote_dcp( "/srv/host-specific-data/$(sys.key_digest)/.", "$(sys.policy_hub)" ), depth_search => recurse( "inf" ); } bundle server my_access_rules { access: policy_server:: "/srv/host-specific-data/$(connection.key)/." admit_keys => { "$(connection.key)" }; } ``` ## this {#this} This variables are context dependent but primarily they are useful for referencing things relative to the currently executing bundle. ### `this.bundle` - The current bundle name {#this-dot-bundle-the-current-bundle-name} `this.bundle` holds the current bundle name, this is often useful for `reports` promises to help you identify where the output is coming from. ```cfengine3 {file="/tmp/feature-friday-12/this.bundle.cf"} bundle agent __main__ { methods: "paisley"; } bundle agent paisley { reports: "In bundle $(this.bundle)"; } ``` ```command cf-agent -Kf /tmp/feature-friday-12/this.bundle.cf ``` ```output R: In bundle paisley ``` ### `this.namespace` - The current bundle namespace {#this-dot-namespace-the-current-bundle-namespace} `this.namespace` often pairs well with `this.bundle` to ensure that you know which bundle is being referred to in the case where namespaces are in use. ```cfengine3 {file="/tmp/feature-friday-12/this.namespace.cf"} bundle agent __main__ { methods: "go"; "CFEngine:go"; } bundle agent go { reports: "Running from $(this.namespace):$(this.bundle)"; } body file control { namespace => "CFEngine"; } bundle agent go { reports: "Running from $(this.namespace):$(this.bundle)"; } ``` ```command cf-agent -Kf /tmp/feature-friday-12/this.namespace.cf ``` ```output R: Running from CFEngine:go R: Running from default:go ``` ### `this.promise_dirname` - The current policy files directory {#this-dot-promise-dirname-the-current-policy-files-directory} `this.promise_dirname` is the directory in which the currently executing policy file exists. It's very useful for referencing assets like templates relative to the current policy file. For example, here we have `/tmp/feature-friday-12/this.promise_dirname.cf`: ```cfengine3 {file="/tmp/feature-friday-12/this.promise_dirname.cf"} bundle agent __main__ { reports: "This policy is running from $(this.promise_dirname)"; } ``` Running it we can see that it knows the directory in which it lives. ```command cf-agent -Kf /tmp/feature-friday-12/this.promise_dirname.cf ``` ```output R: This policy is running from /tmp/feature-friday-12 ``` ## connection {#connection} Connection variables are only available to `cf-serverd` they are useful in controlling which agents a directory is accessible by. ### `connection.key` - The hash of the connecting host's public key {#connection-dot-key-the-hash-of-the-connecting-hosts-public-key} As mentioned before, combined with `sys.key_digest` this can be very useful for configuring access rules that allow hosts to get only their own files. For example: ```cfengine3 bundle agent __main__ { files: "/tmp/my-files/." copy_from => remote_dcp( "/srv/host-specific-data/$(sys.key_digest)/.", "$(sys.policy_hub)"), depth_search => recurse( "inf" ); } bundle server my_access_rules { access: policy_server:: "/srv/host-specific-data/$(connection.key)/." admit_keys => { "$(connection.key)" }; } ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: Announcement of `/etc/os-release` [^fn:2]: Blog post about using `/etc/os-release` for use in loading additional Augments -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 37 - Windows package management" url: /blog/2024/the-agent-is-in-episode-37/index.md date: "2024-05-30" -------------------------------------------------------------------------------- Curious about package management with CFEngine on Windows? After sharing some history on Microsoft's global advertising campaign for "Where do you want to go today?" Craig shared some of his recent experiments with several windows based package managers as well as their related challenges. Craig discussed difficulties with the `msiexec` package module, such as distinguishing which packages need installation through msi while also identifying software for removal by name, a task that can be challenging. He demonstrated this using examples from winget, chocolatey, Scoop, and PowerShell's install-module commands. ## Policy {#policy} Here is the policy used during the episode. Most of the policy was added to `bundle agent mpf_main` in `services/main.cf` ```cfengine3 {file="services/main.cf"} bundle agent mpf_main # User Defined Service Catalogue { reports: "Which packages do you want to install today?"; # "Where do you want to go today?" was the title of Microsoft's second global image advertising campaign. # The broadcast, print and outdoor advertising campaign was launched in November 1994 through # the advertising agency Wieden+Kennedy. # The campaign had Microsoft spending $100 million through July 1995, # of which $25 million would be spent during the holiday shopping season ending in December 1994. # https://en.wikipedia.org/wiki/Where_do_you_want_to_go_today%3F packages: # built-in/standard msiexec package module # includes installed software reporting #"c:$(const.dirsep)Users$(const.dirsep)craig$(const.dirsep)Downloads$(const.dirsep)putty-64bit-0.81-installer.msi" # policy => "present", # package_module => msiexec; # wmic product get name,version | select-string putty #"PuTTY release 0.81 (64-bit)" # policy => "absent", # package_module => msiexec; # winget package method # winget lists software not installed by winget - could be included in reporting # winget search putty #"PuTTY.PuTTY" # package_policy => "add", # add or delete # package_method => winget; # choco package method # chocolatey installs normal style like msi/winget # licensed version synchronizes choco list with Programs and Features # https://docs.chocolatey.org/en-us/features/package-synchronization/ #"putty" # package_policy => "delete", # add or delete # package_method => choco; # scoop package method # has no view of Programs and Features # installs in user directory, can't install for for all users # gow package includes putty.exe (GNU on Windows) # https://github.com/bmatzelle/gow #"gow" # package_policy => "add", # add or delete # package_method => scoop; # install-module package method # focused on powershell code, not packages/programs/features # https://www.powershellgallery.com/packages/Putty/1.0 # The module contains functionality to download and extract the contents of Putty to a local path # https://www.powershellgallery.com/packages/Putty/1.0/Content/Putty.psm1 # $env:TEMP is C:\Users\craig\AppData\Local\Temp # putty.zip and PUTTY.EXE are there after "add" and running "putty" script "Putty" package_policy => "add", package_method => install_module; # nuget, dot net libraries? # TODO: Windows Features, Get-WindowsFeature, Install-, Uninstall- # https://learn.microsoft.com/en-us/powershell/module/servermanager/install-windowsfeature?view=windowsserver2022-ps # SMS # https://en.wikipedia.org/wiki/Microsoft_Configuration_Manager # InTune # PowerShell DSC (Design State Configuration) (dying?) 1.1 everything, 2.0/3.0 azure only, makes a lot of assumptions } body package_method winget { package_changes => "bulk"; package_name_convention => "$(name)"; package_delete_convention => "$(name)"; package_installed_regex => ".*"; # caveat, no commas between fields in Get-WinGetPackage | ConvertTo-Csv output allowed, so not in Id or InstalledVersion package_list_name_regex => '^"([^"]*)",.*'; package_list_version_regex => '.*,"([^"]*)".*'; # winget output doesn't work well, not "object" output # and not easily parseable because Name (column 1) includes spaces # package_add_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"winget install "; # package_delete_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"winget uninstall "; # package_list_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"winget list "; # so use PowerShell module instead package_add_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Install-WinGetPackage "; package_delete_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Uninstall-WinGetPackage "; package_list_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Get-WinGetPackage | Select Id,InstalledVersion | ConvertTo-Csv "; } body package_method choco { package_changes => "bulk"; package_name_convention => "$(name)"; package_delete_convention => "$(name)"; package_installed_regex => ".*"; package_list_name_regex => "(\S+).*"; package_list_version_regex => "\S+\s(\S+).*"; package_add_command => "choco install --yes "; package_delete_command => "choco uninstall --yes --force-dependencies "; package_list_command => "choco list "; } # scoop needs git to update itself so bootstrap with # scoop install git body package_method scoop { package_changes => "bulk"; package_name_convention => "$(name)"; package_delete_convention => "$(name)"; package_installed_regex => ".*"; package_list_name_regex => "(\S+).*"; package_list_version_regex => "\S+\s(\S+).*"; package_add_command => "scoop install "; package_delete_command => "scoop uninstall "; package_list_command => "scoop list "; } # https://learn.microsoft.com/en-us/powershell/module/powershellget/?view=powershellget-3.x body package_method install_module { package_changes => "bulk"; package_name_convention => "$(name)"; package_delete_convention => "$(name)"; package_installed_regex => ".*"; package_list_name_regex => '^"([^"]*)",.*'; package_list_version_regex => '.*,"([^"]*)".*'; package_add_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Install-Module -force -Name "; package_delete_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Uninstall-Module -Name "; package_list_command => "$(sys.winsysdir)\\WindowsPowerShell\\v1.0\\powershell.exe -Command \"Get-InstalledModule | Select Name,Version | ConvertTo-Csv"; } ``` The default package module for windows was commented out in `lib/packages.cf` as the examples leveraged the older _package_method_ based implementation of packages promises. ```cfengine3 {file="lib/packages.cf"} # In order for package methods to be used as list of installed software # must comment out the following package module platform_default. # windows:: # "platform_default" string => "msiexec"; ``` ## Video {#video} The video recording is available on YouTube: {{< youtube kJTjle4mVaE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - WinGet - Chocolatey - Scoop - Install-module - - - From post-show discussion - Reducing potential attack surface area with `bintointerface` and `allowconnects` attributes in `body server control`. - - - SHA2 signature hashing - Ensuring MicroStation image metadata, e.g. titles match file names - - - The future and challenges of preventing data exfiltration and impact on computing needs. - CFEngine Enterprise Basic Training - Default umask setting for directories - - Support options for Community users -------------------------------------------------------------------------------- title: "Feature Friday #11: namespaces" url: /blog/2024/feature-friday-11-namespaces/index.md date: "2024-05-24" -------------------------------------------------------------------------------- Did you know that CFEngine has namespaces? Let's see how they can facilitate policy sharing while avoiding "duplicate definitions of bundle" errors. Most of the Masterfiles Policy Framework (MPF) and policy examples for CFEngine use the `default` namespace. However, `body file control` allows you to specify a namespace that applies for the rest of the file or until it's set again by another `body file control`. Let's consider a contrived example. Say we have two policy files (`policy-1.cf`, `policy-2.cf`) for different services. In each policy file, we want to have a bundle where we store settings related to that policy. Traditionally this would be handled by using some bundle naming convention, so we might have `bundle agent policy_1_settings` and `bundle agent policy_2_settings`. Using namespaces you can keep your bundle names brief and use different namespaces to avoid "duplicate definitions of bundle" errors. Let's take a look at the implementation we might have for `policy-1.cf`. ```cfengine3 {file="/tmp/feature-friday-11/policy-1.cf"} body file control { namespace => "policy_1"; } bundle agent settings # @brief This policy is disabled by default { vars: "state" string => "disabled", if => not( isvariable( "state" ) ); reports: "In $(this.namespace):$(this.bundle): state = $(state)"; } ``` And similar for `policy-2.cf`: ```cfengine3 {file="/tmp/feature-friday-11/policy-2.cf"} body file control { namespace => "policy_2"; } bundle agent settings # @brief This policy is enabled by default. { vars: "state" string => "enabled", if => not( isvariable( "state" ) ); reports: "In $(this.namespace):$(this.bundle): state = $(state)"; } ``` Our policy entry then might look like this: ```cfengine3 {file="/tmp/feature-friday-11/promises.cf"} body common control { inputs => { "policy-1.cf", "policy-2.cf" }; bundlesequence => { "go" }; } bundle agent go { methods: "policy_1:settings"; "policy_2:settings"; reports: "State of policy 1 = $(policy_1:settings.state)"; "State of policy 2 = $(policy_2:settings.state)"; } ``` Executing the policy we can see our variable values in the different namespaces: ```command cf-agent -Kf /tmp/feature-friday-11/promises.cf ``` ```output R: In policy_1:settings: state = disabled R: In policy_2:settings: state = enabled R: State of policy 1 = disabled R: State of policy 2 = enabled ``` The most common gotcha when using namespaces is forgetting to specify the `default` namespace when using a body from the standard library. For example, if we wanted to use `body perms o` from the standard library to specify the owner of a file, we would write it as `perms => default:o( "owner" )`: ```cfengine3 {file="/tmp/feature-friday-11/policy-2.cf"} body file control { namespace => "policy_2"; } bundle agent settings # @brief This policy is enabled by default. { vars: "state" string => "enabled", if => not( isvariable( "state" ) ); files: "/etc/hosts" perms => default:o( "root" ); reports: "In $(this.namespace):$(this.bundle): state = $(state)"; } ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #10: cf-support" url: /blog/2024/feature-friday-10-cf-support/index.md date: "2024-05-17" -------------------------------------------------------------------------------- Found a bug, asking for help? Use [cf-support](https://docs.cfengine.com/latest/reference-components-cf-support.html) to collect info quickly. `cf-support` was born from interactions supporting Enterprise customers to streamline data collection and was introduced in late 2022 with the release of 3.18.3[^fn:1] and 3.21.0[^fn:2]. Furthermore, it was featured on The Agent is In[^fn:3] episode 21 [Troubleshooting with cf-support](https://cfengine.com/blog/2023/the-agent-is-in-episode-21-troubleshooting-with-cf-support/). It gathers various details about the system and creates an archive that you can attach to your ticket. While its inception was geared towards Enterprise customers, it's still both useful and available for community users as well. Let's check it out. Here we have a host running CFEngine 3.21.4 community: ```command cf-agent --version ``` ```output CFEngine Core 3.21.4 ``` ```command cf-support ``` ```output [root@host002 ~]# /var/cfengine/bin/cf-support If you have it, please enter your support case number: CFE-1234 Analyzing CFEngine core dumps Using coredumpctl to analyze CFEngine core dumps Captured output of command uname -a Captured output of command /var/cfengine/bin/cf-promises -V Captured output of command lscpu Captured output of command free -m Captured output of command hostname Captured output of command cat /etc/os-release Added file /etc/fstab Captured output of command mount Captured output of command df -h Captured output of command top -b -H -c -n1 Captured output of command netstat -ie Captured output of command ifconfig Captured output of command lsof Captured output of command /var/cfengine/bin/cf-key -p /var/cfengine/ppkeys/localhost.pub Captured output of command grep 'version =' /var/cfengine/inputs/promises.cf Captured output of command /var/cfengine/bin/cf-key -s -n Captured output of command /var/cfengine/bin/cf-check diagnose Captured output of command systemctl status cfengine3 Captured output of command systemctl status cf-* Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:24:34_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:24:35_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:36:37_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:36:38_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:44:31_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_18:44:32_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_19:24:37_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_19:24:38_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_19:32:01_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-15_19:32:02_UTC.log Added compressed copy of file /var/log/CFEngine-Install.log Captured output of journalctl filtered for cf-|CFEngine cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/ cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/core-dump.log cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/system-info.txt cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/fstab cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/lsof.txt cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/classes-and-vars.txt cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/update-evaluated-classes-and-vars.txt cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/promises-evaluated-classes-and-vars.txt cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:24:34_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:24:35_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:36:37_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:36:38_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:44:31_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_18:44:32_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_19:24:37_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_19:24:38_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_19:32:01_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install-2024-02-15_19:32:02_UTC.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/CFEngine-Install.log.gz cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950/syslog-filtered-for-cfengine.log.gz Please send cfengine_support_case_CFE-1234-host002.example.com-2024-02-15-1950.tar.gz to CFEngine support staff. [root@host002 ~]# ``` However, by running it on an Enterprise hub, even more will be collected. Note the prompt asking if `masterfiles` should be included (`Include masterfiles in support submission [Y/n]:`). Here is what it looks like: ```command cf-support ``` ```output If you have it, please enter your support case number: ENT-1234 Analyzing CFEngine core dumps Using coredumpctl to analyze CFEngine core dumps Captured output of command uname -a Captured output of command /var/cfengine/bin/cf-promises -V Captured output of command lscpu Captured output of command free -m Captured output of command hostname Captured output of command cat /etc/os-release Added file /etc/fstab Captured output of command mount Captured output of command df -h Captured output of command top -b -H -c -n1 Captured output of command netstat -ie Captured output of command ifconfig Captured output of command lsof Captured output of command /var/cfengine/bin/cf-key -p /var/cfengine/ppkeys/localhost.pub Captured output of command grep 'version =' /var/cfengine/inputs/promises.cf Captured output of command /var/cfengine/bin/cf-key -s -n Captured output of command /var/cfengine/bin/cf-check diagnose Captured output of command systemctl status cfengine3 Captured output of command systemctl status cf-* Added compressed copy of file /var/log/CFEngine-Install-2024-02-13_17:12:28_UTC.log Added compressed copy of file /var/log/CFEngine-Install-2024-02-13_17:12:58_UTC.log Added compressed copy of file /var/log/CFEngine-Install.log Captured output of journalctl filtered for cf-|CFEngine Added compressed copy of file /var/cfengine/outputs/previous Added file /var/cfengine/policy_server.dat Captured output of command cf-serverd -Fv | awk '/=== BEGIN summary of access promises ===/,/=== END summary of access promises ===/' Added compressed copy of file /var/log/postgresql.log Added file /var/cfengine/state/pg/data/postgresql.conf Added file /var/cfengine/state/pg/data/pg_hba.conf Captured output of command /var/cfengine/bin/psql cfdb -c "SELECT count(*) as host_count from __hosts;" Captured output of command /var/cfengine/bin/psql cfdb -c "select name, details, units, avg(value), count(value) from diagnostics group by name, details, units;" Captured output of command /var/cfengine/bin/psql cfdb -c "select host, status, type, count(*) from __status group by host, status, type order by host, status;" Captured output of command /var/cfengine/bin/psql cfdb -c "select count(*) from pg_stat_activity;" Captured output of command /var/cfengine/bin/psql cfdb -c "select max(now() - xact_start) from pg_stat_activity where state in ('idle in transaction', 'active');" Include masterfiles in support submission [Y/n]: Y Added directory /var/cfengine/masterfiles cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/core-dump.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/system-info.txt cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/fstab cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/lsof.txt cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/classes-and-vars.txt cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/update-evaluated-classes-and-vars.txt cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/promises-evaluated-classes-and-vars.txt cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/CFEngine-Install-2024-02-13_17:12:28_UTC.log.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/CFEngine-Install-2024-02-13_17:12:58_UTC.log.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/CFEngine-Install.log.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/syslog-filtered-for-cfengine.log.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/previous.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/policy_server.dat cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/postgresql.log.gz cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/application/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/application/log-2024-02-13.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/application/log-2024-02-14.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/application/log-2024-02-15.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/access_log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/error_log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/ssl_request_log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/httpd/logs/ssl_request_log.1 cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/deprecated/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/deprecated/cfengine_processes.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/watchdog/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/watchdog/templates/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/watchdog/templates/watchdog-windows.ps1.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/watchdog/templates/watchdog.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/watchdog/watchdog.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/host_info_report.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/limit_robot_agents.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/log_rotation.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/core/main.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/federation/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/federation/federation.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/ha/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/ha/ha.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/ha/ha_def.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/ha/ha_update.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/templates/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/templates/apachectl.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/templates/httpd.conf.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/templates/runalerts.php.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/templates/runalerts.sh.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/CFE_hub_specific.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/CFE_knowledge.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/file_change.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/main.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/enterprise/mission_portal.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/cfe_internal_dc_workflow.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/cfe_internal_update_from_repository.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/lib.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/systemd_units.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/update_bins.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/update_policy.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/update_processes.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/update/windows_unattended_upgrade.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/CFE_cfengine.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfe_internal/recommendations.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_agent.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_execd.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_hub.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_monitord.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_runagent.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/cf_serverd.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/def.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/def_inputs.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/reports.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/update_def.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/controls/update_def_inputs.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/aix.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/any.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/debian.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/freebsd.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/generic.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/linux.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/lsb.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/macos.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/os.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/redhat.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/suse.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/inventory/windows.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/templates/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/templates/junit.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/templates/tap.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/autorun.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/bundles.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/cfe_internal.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/cfe_internal_hub.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/cfengine_enterprise_hub_ha.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/commands.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/common.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/databases.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/edit_xml.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/event.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/examples.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/feature.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/files.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/guest_environments.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/monitor.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/packages-ENT-3719.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/packages.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/paths.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/processes.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/reports.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/services.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/stdlib.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/storage.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/testing.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/users.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/lib/vcs.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/WiRunSQL.vbs.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/apk.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/apt_get.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/freebsd_ports.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/msiexec-list.vbs.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/msiexec.bat.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/nimclient.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/pkg.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/pkgsrc.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/slackpkg.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/snap.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/yum.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/modules/packages/vendored/zypper.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/services/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/services/autorun/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/services/autorun/hello.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/services/init.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/services/main.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/ cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/10-base_filter.sed cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/50-merge_inserts.awk cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/cfsecret.py cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/config.sh.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/distributed_cleanup.py cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/dump.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/import.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/import_file.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/log.sh.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/nova_api.py cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/parallel.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/psql_wrapper.sh.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/pull_dumps_from.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/transfer_distributed_cleanup_items.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/federated_reporting/transport.sh cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-apache.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-execd.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-hub.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-monitord.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-postgres.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-reactor.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-runalerts.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cf-serverd.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cfengine3.service.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/cfengine_watchdog.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/host_info_report.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/json_multiline.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/json_serial.mustache cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/templates/vercmp.ps1 cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cf_promises_release_id cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cf_promises_validated cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/cfbs.json cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/promises.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/standalone_self_upgrade.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/var/cfengine/masterfiles/update.cf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/postgresql.conf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/pg_hba.conf cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/cfdb-biggest-tables.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-current-queries.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-age-of-nonidle-queries.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-waiting-not-idle.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-db-statistics.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-table-statisctics.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-database-cache-hit-ratio.log cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954/psql-current-rows-per-table.log Please send cfengine_support_case_ENT-1234-hub.example.com-2024-02-15-1954.tar.gz to CFEngine support staff. ``` Next time you file a bug report or you are trying to debug a problem, give `cf-support` a try. If you want to help improve `cf-support` we very much appreciate pull requests to the core repository. You can find it [here](https://github.com/cfengine/core/blob/master/misc/cf-support)! Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: CFEngine 3.18.3 LTS release blog post: [^fn:2]: CFEngine 3.21 LTS release blog post: [^fn:3]: The Agent is In is a monthly webinar covering a range of things, all related to CFEngine. Find past episode show notes on our blog . -------------------------------------------------------------------------------- title: "Efficient data/file copying on modern Linux" url: /blog/2024/efficient-data-copying-on-modern-linux/index.md date: "2024-05-15" -------------------------------------------------------------------------------- Editing and copying large files or large numbers of files is slow. For a configuration management tool, it is probably one of the slowest things we do, apart from waiting for other programs to finish or waiting for network communication. In this blog post, we look at how to copy files. More specifically, the most performant approaches available on modern Linux systems. We are working on implementing these techniques so CFEngine and all your policy will copy files more efficiently. ## Traditional approach Many programs need to copy data or whole files. Usually, this is something that doesn't bother developers too much because it is straightforward to do with the following steps: 1. Open the source file 2. Open or create the destination file 3. Read a chunk of data from the source file 4. Write the chunk into the destination file 5. Repeat steps 3 and 4 until all data is read from the source and written into the destination 6. Close both files And, if desired, fix up ownership and permissions on the destination file based on the source file. The above approach is very easy, especially in high-level languages or with libraries that don't require the developer to handle cases where the `read()` system call (syscall) is interrupted with a signal or even hiding the fact that it can easily read fewer bytes than requested. A very naive approach that, however, often works well enough is to just read the whole source file into one big buffer and then write the data into the destination file. This, of course, doesn't work for large files and is thus not a good approach for a generic `CopyFile()` function. Then there are a couple of better alternatives using `select()/poll()` syscalls or non-blocking/asynchronous I/O with two buffers or a ring buffer to better handle cases where the program could for example be writing into the destination file, but is instead stuck on trying to read from the source file which is not ready for reading (due to slow storage hardware, for example) and vice versa. ## Talking to the system All the above approaches, no matter how complex and smart, have one disadvantage -- data is shoveled from one file into another, however, the information that it is actually being copied is lost. In other words, when the source file is read and when the data is written into the destination file, the operating system doesn't know it's being copied. This doesn't allow the operating system and in particular the file system to do smarter things than just providing the data requested from somewhere to the program and then writing some other data somewhere else. The information that it is the same data is lost. When working on a [C implementation of the `mender-flash` tool](https://github.com/mendersoftware/mender-flash/pull/3) [^1], I was thinking that there must be a better way. Based on some past knowledge of storage and how file systems work, it was quite obvious that if some file systems can create copy-on-write (COW) copies of files, where the blocks of data are shared by multiple files as long as they all contain copies of the same blocks, then there must be some way to tell the operating system that data should be copied, not just read from one place and written to another. Also, since the `mender-flash` tool is expected to run on platforms with very little memory it made sense to keep the required buffers small or avoid them altogether. The size of the buffers used for the shoveling approach creates a trade-off between the number of system calls required to shovel the data and the amount of memory needed. The bigger the buffer, the fewer syscalls, but requiring more memory. If memory is not a problem, the very naive approach mentioned above with just one buffer as big as the source file potentially only requires 2 syscalls to copy all the data -- `read()` everything and then `write()` everything.[^4] However, the required size of the buffer for something like a system image (in case of `mender-flash`) is very likely too large for a small device to handle. On the other hand, using a small buffer means a large number of syscalls needed to shovel the data. And making system calls is not the cheapest operation a process can do. Usually this is not a problem because the related I/O operations eat the biggest part of the required resources (time, power, ...), but if the number of required syscalls can be lowered, it's always nice. ### sendfile() and splice() The Linux kernel has provided better alternatives addressing the problems described above since a long time ago. Even a long time before it started supporting any file systems with COW capabilities. Since version 2.2 it provides the `sendfile()` system call, originally designed for sending data from a file via a socket, but allowing copying data from one file into another one since version 2.6.33.[^2] Its man page literally says: > sendfile() copies data between one file descriptor and > another. Because this copying is done within the kernel, sendfile() is > more efficient than the combination of read(2) and write(2), which > would require transferring data to and from **user space**. where _user space_ means the buffer in the address space of the particular process (which thus is not in the kernel's address space). There's also a related system call `splice()` which > moves data between two file descriptors without copying between > kernel address space and user address space. It transfers up to > len bytes of data from the file descriptor fd_in to the file > descriptor fd_out, **where one of the file descriptors must refer > to a pipe.** And, since version 2.6.31, both file descriptors can refer to pipes.[^3] This similar system call allows efficient copying of data from a pipe, i.e., produced by some other process. ### copy_file_range() and FICLONE `sendfile()` has the benefit of replacing the repetition of `read()` and `write()` syscalls with a single one [^4], it doesn't require the buffer and thus avoids copying data from the kernel and back. With it, the program talks to the kernel more precisely, telling it that it needs to send data from one file into another file instead of telling it that it needs to read some data and write some (unrelated) data. However, it doesn't tell the kernel exactly that it needs to copy data from one file into another (keep in mind that its original purpose was to efficiently send data from a file via a socket). To talk to the kernel even more precisely, a program can use the `copy_file_range()` system call which is very similar to `sendfile()` at first glance, but has the following important note in its man page: > copy_file_range() gives filesystems an opportunity to implement > "copy acceleration" techniques, such as the use of reflinks > (i.e., two or more inodes that share pointers to the same > copy-on-write disk blocks) or server-side-copy (in the case of > NFS). where _reflinks_ means the copy-on-write mechanism mentioned above. This syscall has a short, but a bit wild, history (read the _copy_file_range(2)_ man page for details) mentioning version 5.19 as the one with the most recent and best implementation and it only reliably works if the source and destination files are on the same file system. However, in those cases, it provides (almost) the best possible approach to copy data between two files. Why only _almost_ best possible approach? Because it supports, as its name suggests, copying ranges of data between two files. Which is nice if that is required and of course copying the whole file is just as easy as telling it to copy the range between the beginning and the end of the file. On the other hand, if the goal is to create a copy of a whole file, it should be possible to tell the kernel exactly that, right? And as the above note says, it _\"gives filesystem an opportunity...\"_ which doesn't sound very reassuring. Then comes the last in the list of modern system calls for file/data copying. And it's not exactly a new syscall, but rather a new variant of the good old `ioctl()` system call that can be used for various I/O-related operations -- `FICLONE`. Its declaration looks like this: ```C int ioctl(int dest_fd, FICLONE, int src_fd); ``` which shows that it's exactly what we want to tell the kernel when copying a whole file -- _\"clone this file (`src_fd`) here (`dest_fd`)\"_. However, as its man page says: > If a filesystem supports files sharing physical storage between > multiple files ("reflink"), this ioctl(2) operation can be used to > make some of the data in the src_fd file appear in the dest_fd file by > sharing the underlying storage, which is faster than making a separate > physical copy of the data. it only works inside one file system and only if the file system supports sharing physical storage (a different description for the COW mechanism described above). When that is exactly the case, nothing can beat this system call in performance and also the (e)quality of the result. It exists since version 4.5 and the constant (macro) for the `ioctl()` call was actually originally called `BTRFS_IOC_CLONE`, revealing that it was introduced together with the _Btrfs_ file system. ## Sparse files When talking about a generic function to copy a file from a source to destination efficiently, special attention needs to be paid to _sparse files_ -- files that contain not only data, but also so-called _holes_, areas that contain no data, don't take up disk space, and when being read, they look like blocks of zeroes. There are many reasons for why such files are useful and quite commonly created and supported by modern file systems. It's not uncommon that the amount of _space_ occupied by a sparse file is an order of magnitude smaller than its reported _size_. The `stat` utility (and related system calls and functions) report both _size_ of a file and the number of _blocks_ (usually 512 bytes big) it occupies. In relation to copying, it should be quite obvious that the naive approach mentioned at the beginning of this post would result in a non-sparse copy of file. Because `read()` would simply return blocks of zeroes while getting across the holes in the source file and `write()` would simply write those zeroes into the destination file. And **there is a big difference between a sparse file and a file containing blocks of zeroes**, no matter how big those blocks are. Again, when a program wants to create a sparse file, it needs to tell the operating system instead of assuming that writing a block of zeroes will create a hole. The operating system doesn't care about the data being written, of course. An optimization, which we have in our CFEngine code[^5], is to detect such blocks of zeroes and create holes in the destination file instead of writing them there. However, it doesn't avoid reading all those zeroes from the source file. The easiest way to create a sparse file is to use the `truncate` utility and set a big size of a new empty file or an existing file containing some data using the `--size` option which creates a hole at the end of the file after the last data block. Another approach is to use the `dd` utility and its `seek` option to make it write some data at an offset that is larger than the end of the last data block in the file which creates a hole between such an offset and the end of the last data block with the newly written data by `dd` becoming the new last block of data. _Seek_ is the keyword when it comes to sparse files. As the example with `dd` shows, it provides one way to create holes in files. And since Linux 3.1, the `lseek()` syscall accepts two special values as its last argument `whence` -- `SEEK_DATA` and `SEEK_HOLE`. These two special values, as their names suggest, allow a program to _seek_ (reposition its file descriptor) in a file to the beginning of the next data block or hole (if any), respectively. This can be used to avoid reading all the zeroes from the holes in the source file in favor of skipping the holes and creating the exact same holes in the destination file. Out of the modern system calls for copying data between files described above, only the `FICLONE` variant of the `ioctl()` syscall takes care of mirroring holes between the source and the destination. Which, after all, makes sense as it literally creates a **clone** of the source file. Both `sendfile()` and `copy_file_range()` replicate holes as blocks of zeroes in the destination file and `lseek()` with `SEEK_DATA` and `SEEK_HOLE` needs to be used to avoid that. ## Conclusions Although copying files, or more generally, copying data from a source file into a destination file looks like an easy thing to do, this blog post shows that the naive approach has multiple issues that can easily result in an unnecessary consumption of various resources -- CPU, memory, disk space and also time. Modern Linux systems provide mechanisms that allow programs to be more explicit and tell the operating system that they are copying data and not just reading data here and writing data there. This blog post provides an overview of 3 specific system calls that serve such a purpose (plus `splice()` as related one). All with their limitations and with varying availability across different versions of Linux. An ideal implementation of a generic file copying function should check what the conditions are, then decide which of the approaches mentioned above to choose. And a defensive way of doing things is to fall back to using the very basic approach shoveling data from the source file into the destination file in case the chosen more efficient approach fails. At the time of writing this blog, we are currently [working on using the collected knowledge summarized above in CFEngine](https://github.com/cfengine/core/pull/5490) by [changing the implementation of our generic function `FileSparseCopy()`](https://github.com/NorthernTechHQ/libntech/pull/218) in our [_libntech_](https://github.com/cfengine/libntech) utility library ([now available under the Apache license](https://cfengine.com/blog/2023/libntech-1-0-released-with-new-license/)). It is not an easy task because CFEngine supports a wide variety of platforms including some exotic ones like HP-UX, Solaris and AIX as well as some old versions of GNU/Linux distributions like RHEL 6 and Ubuntu 16.04. All the syscalls mentioned above are Linux-specific (although similar syscalls may exist in other operating systems, even with similar or the exact same names) so we just use the good old shoveling approach with hole detection on all of the non-Linux platforms as well as on RHEL 6 which doesn't have `SEEK_DATA` and `SEEK_HOLE` (and `copy_file_range()`). Once fully implemented, tested and merged, these new mechanisms for data copying will allow CFEngine to perform this operation more efficiently with lower resource usage, at least on modern Linux-based platforms. Although CFEngine doesn't often copy large files and sparse files, it is nice to have an efficient generic implementation ready for such cases. And of course, the slower the HW is, especially regarding I/O operations, the bigger the impact can be. For example, copying files on a slow device using a micro-SD card for storage (like many of the ARM boards do) can yield very different results when the `FICLONE` `ioctl()` is used instead of slowly reading the data from an SD card and writing it even more slowly to the same storage when simply shoveling the data, let alone in case of large sparse files. [^1]: Which triggered a very long discussion about language choices and the necessary complexity of the tool, but that's another story. [^2]: According to the **HISTORY** section of the _sendfile(2)_ man page. [^3]: Our [blog post about processes](https://cfengine.com/blog/2022/processes-part-2/) describes, among many other things, what pipes are and how they work. [^4]: Up to a certain limit that the syscall has. If more data needs to be copied, it needs to be called multiple times, but the limit is `2 GiB` according to the man page. [^5]: Search for `FileSparseWrite()` in [the _file_lib.c_ file in the sources of our utility library](https://github.com/NorthernTechHQ/libntech/blob/master/libutils/file_lib.c). -------------------------------------------------------------------------------- title: "Feature Friday #9: body file control - inputs" url: /blog/2024/feature-friday-9-body-file-control-inputs/index.md date: "2024-05-10" -------------------------------------------------------------------------------- Did you know you can include one policy file from another? Traditionally you specify the files you want to make up a policy set using `inputs` in `body common control` found in your policy entry (`promises.cf` by default). ```cfengine3 body common control { # Paths are relative to $(sys.policy_entry_dirname) if not # fully qualified inputs => { "path/to/policy-1.cf", "path/to/policy-2.cf", }; } ``` `body file control` lets you specify additional inputs from any file that's included in the policy and those files can include other files. Let's check out a contrived example. First, we have our policy entry `/tmp/feature-friday-9/promises.cf`: ```cfengine3 {file="/tmp/feature-friday-9/promises.cf"} body common control { inputs => { "policy-1.cf", }; bundlesequence => { "go" }; } bundle agent go { methods: # Methods promises run before reports promises (Normal Order), we # want the message about the bundle we are in to be emitted before # we go running a different bundle, so we depend on that report. "policy_1" depends_on => { "$(this.namespace):$(this.bundle)-report-bundle" }; reports: "In $(this.namespace):$(this.bundle)" handle => "$(this.namespace):$(this.bundle)-report-bundle"; } ``` We see that `policy-1.cf` is included in the policy set, the `go` agent bundle gets run by default and it wants to run the `policy_1` bundle (which is found in `policy-1.cf`). Next we have `/tmp/feature-friday-9/policy-1.cf`: ```cfengine3 {file="/tmp/feature-friday-9/policy-1.cf"} body file control { # Remember that inputs is relative to sys.policy_entry_dirname # if it's not a fully qualified path. inputs => { "1/1.cf", }; } bundle agent policy_1 { methods: # Methods promises run before reports promises (Normal Order), we # want the message about the bundle we are in to be emitted before # we go running a different bundle, so we depend on that report. "policy_1_1" depends_on => { "$(this.namespace):$(this.bundle)-report-bundle" }; reports: "In $(this.namespace):$(this.bundle)" handle => "$(this.namespace):$(this.bundle)-report-bundle"; } ``` We see that `1/1.cf` is part of inputs in `body file control` and we see the definition for the `policy_1` agent bundle which emits a report and runs bundle `policy_1_1`. (which comes from `1/1.cf`) Remember, `inputs` if not a fully qualified path is relative to the policy entry, so `1/1.cf` will be looked for in `/tmp/feature-friday-9/1/1.cf`. ```cfengine3 {file="/tmp/feature-friday-9/1/1.cf"} body file control { # Remember that inputs is relative to sys.policy_entry_dirname if # it's not a fully qualified path. inputs => { "1/1-1.cf", }; } bundle agent policy_1_1 { methods: # Methods promises run before reports promises (Normal Order), we # want the message about the bundle we are in to be emitted before # we go running a different bundle, so we depend on that report. "policy_1_1_1" depends_on => { "$(this.namespace):$(this.bundle)-report-bundle" }; reports: "In $(this.namespace):$(this.bundle)" handle => "$(this.namespace):$(this.bundle)-report-bundle"; } ``` And finally, `/tmp/feature-friday-9/1/1-1.cf`: ```cfengine3 {file="/tmp/feature-friday-9/1/1-1.cf"} bundle agent policy_1_1_1 { reports: "In $(this.namespace):$(this.bundle)" handle => "$(this.namespace):$(this.bundle)-report-bundle"; } ``` Running our policy entry we can see from the output that all of the policy files were properly included: ```command cf-agent -Kf /tmp/feature-friday-9/promises.cf ``` ```output R: In default:go R: In default:policy_1 R: In default:policy_1_1 R: In default:policy_1_1_1 ``` We can also see that if we specify a different entry e.g. `/tmp/feature-friday-9/policy-1.cf` and bundlesequence (`policy_1`), that things still work as expected given the partial policy set. ```command cf-agent -Kf /tmp/feature-friday-9/policy-1.cf --bundlesequence policy_1 ``` ```output R: In default:policy_1 R: In default:policy_1_1 R: In default:policy_1_1_1 ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Configure which hosts can participate in CFEngine infrastructure management" url: /blog/2024/allow-hosts/index.md date: "2024-05-06" -------------------------------------------------------------------------------- Two modules are available for this task: [allow-all-hosts](https://build.cfengine.com/modules/allow-all-hosts/) and [allow-hosts](https://build.cfengine.com/modules/allow-hosts/). The first module, `allow-all-hosts`, configures the most open situation which is to accept hosts from anywhere. This is only recommended in network restricted environments such as a local machine's virtual machine network or other such closed down situations. The second module, `allow-hosts`, uses cfbs module input to let you decide which hosts (specified by IP addresses and subnets) are allowed to connect to your hub, authenticate, fetch policy, etc. This is by far the more common scenario. By default the hub will accept requests from hosts on its own subnet e.g. hub-ip/16. # Add the `allow-hosts` module When you install `allow-hosts` in the Build app in [Mission Portal](https://docs.cfengine.com/docs/master/getting-started-reporting-and-web-ui.html) you will see an input field to enter one or more allowed hosts. {{< wideimg "images/allow-hosts-edit.png" "allow-hosts module added and editing an entry">}} I add several IPs for hosts in my environment, including the hub itself, `192.168.1.196`: {{< figure width=197 src="../images/allow-hosts-several-ips.png" alt="list of IP address to allow to connect" >}} and then make sure I push and deploy the policy: {{< figure width=189 src="../images/allow-hosts-push-and-deploy.png" alt="Push and deploy button in Mission Portal Build application" >}} # Remove an IP and test Next I remove one host's IP: `192.168.1.103` and test if it can connect and pull policy from the hub. {{< figure width=197 src="../images/allow-hosts-fewer-ips.png" alt="list of IP addresses with 192.168.1.103 removed">}} The `ip a` command shows the IP address: ```command ip a | grep 192 ``` ```output inet 192.168.1.103/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp3s0 ``` If we try to fetch policy we see that the hub rejects the connection: ```command cf-agent -KIf update.cf ``` ```output error: Failed to establish TLS connection: underlying network error (Connection reset by peer) info: Unable to establish connection to '192.168.1.196' ``` The policy that has already been fetched will still be used but updates will not be received. When I add that IP address back in, it can fetch policy again as desired. # Test with `cf-net` An alternate way to test connectivity would be to use [`cf-net`](https://docs.cfengine.com/docs/3.21/reference-components-cf-net.html) command: ```command cf-net -H 192.168.1.196 opendir /var/cfengine/data ``` ```output ... host_specific.json ``` and if the IP is not included in the list of IPs you would get ```command cf-net -H 192.168.1.196 opendir /var/cfengine/data ``` ```output error: Failed to establish TLS connection: underlying network error (Connection reset by peer) Failed to connect to '192.168.1.196' ``` ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #8: bundlesequence" url: /blog/2024/feature-friday-8-bundlesequence/index.md date: "2024-05-03" -------------------------------------------------------------------------------- Ever want to run just a one or a few select bundles from your policy? While developing policy it's common to run `cf-agent -KI` so that you can quickly iterate on changes and then run the policy without locks. But if you are focused on select bundles you may not need the full policy to run, you can use the `--bundlesequence` option to specify one or more bundles overriding the bundlesequence defined in `body common control`. Let's check out an example using the Masterfiles Policy Framework (MPF). First, let's see how long it takes to run the full policy without locks: ```command for each in $(seq 3); do /bin/time -o /dev/tty cf-agent -K > /dev/null 2>&1 done ``` ```output 6.62user 3.52system 0:14.02elapsed 72%CPU (0avgtext+0avgdata 106144maxresident)k 0inputs+5312outputs (0major+92361minor)pagefaults 0swaps 6.27user 2.75system 0:11.49elapsed 78%CPU (0avgtext+0avgdata 103696maxresident)k 0inputs+2320outputs (0major+90882minor)pagefaults 0swaps 7.28user 3.74system 0:15.26elapsed 72%CPU (0avgtext+0avgdata 104092maxresident)k 0inputs+2576outputs (0major+92528minor)pagefaults 0swaps ``` From the output we can see that execution time is about 13 seconds. Now, I am just interested in one bundle, `host_info_report`. So, let's see how long running just that bundle takes: ```command for each in $(seq 3); do /bin/time -o /dev/tty cf-agent -K --bundlesequence host_info_report > /dev/null 2>&1 done ``` ```output 3.00user 0.59system 0:03.72elapsed 96%CPU (0avgtext+0avgdata 105616maxresident)k 0inputs+4920outputs (0major+52900minor)pagefaults 0swaps 2.51user 0.48system 0:03.12elapsed 95%CPU (0avgtext+0avgdata 104676maxresident)k 0inputs+1224outputs (0major+56042minor)pagefaults 0swaps 2.61user 0.44system 0:03.17elapsed 96%CPU (0avgtext+0avgdata 105568maxresident)k 0inputs+1240outputs (0major+57858minor)pagefaults 0swaps ``` As we can see from the output, execution time was dramatically lower, about 3 seconds for the execution. Sometimes you need more than one bundle, simply separate the bundles with a comma: ```command for each in $(seq 3); do /bin/time -o /dev/tty cf-agent -K --bundlesequence inventory_autorun,host_info_report > /dev/null 2>&1 done ``` ```output 2.84user 0.56system 0:03.52elapsed 96%CPU (0avgtext+0avgdata 104204maxresident)k 0inputs+4784outputs (0major+58614minor)pagefaults 0swaps 2.77user 0.42system 0:03.32elapsed 96%CPU (0avgtext+0avgdata 105372maxresident)k 0inputs+1432outputs (0major+57686minor)pagefaults 0swaps 2.68user 0.43system 0:03.22elapsed 96%CPU (0avgtext+0avgdata 104388maxresident)k 0inputs+1416outputs (0major+59159minor)pagefaults 0swaps ``` We see that adding `inventory_autorun` to the bundlesequence adds 10ths of a second to our execution time. Happy Friday! 🎉 Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #7: depends_on" url: /blog/2024/feature-friday-7-depends-on/index.md date: "2024-04-26" -------------------------------------------------------------------------------- Ever wanted to make sure a promise only runs if some other promise has succeeded? Consider this contrived example with two `reports` type promises, `It's Friday!` and `I love CFEngine Feature Friday.` Per normal ordering[^fn:1], these two promises will be emitted in the written order. ```cfengine3 {file="/tmp/feature-friday-7.cf"} bundle agent feature_friday { reports: "It's Friday, Friday"; "Gotta get down on Friday"; } ``` ```command cf-agent --no-lock --bundlesequence feature_friday --file /tmp/feature-friday-7.cf ``` ```output R: It's Friday! R: I love CFEngine Feature Friday. ``` If we want them in the opposite order, we could either change the order or define classes based on the results of the promises. But today's feature, `depends_on`, lets us influence the ordering using a more lightweight method via _handle_. ```cfengine3 {file="/tmp/feature-friday-7-1.cf"} bundle agent feature_friday { reports: "It's Friday!" depends_on => { "first" }; "I love CFEngine Feature Friday." handle => "first"; } ``` ```command cf-agent --no-lock --bundlesequence feature_friday --file /tmp/feature-friday-7-1.cf ``` ```output R: I love CFEngine Feature Friday. R: It's Friday! ``` As you can see from the output above, we effectively switched the order of the promises. The `depends_on` attribute requires that the promises with the named handles have been resolved as KEPT or REPAIRED before it's allowed to actuate. Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). [^fn:1]: Documentation on CFEngine Normal ordering -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 36 - Upgrading to Enterprise" url: /blog/2024/the-agent-is-in-episode-36/index.md date: "2024-04-25" -------------------------------------------------------------------------------- It's easy to switch from Community to Enterprise. Craig demonstrates the process of migrating from Community to Enterprise and touches on using many of the available components and tools from `cf-remote` to `cf-net` and so much in-between. ## Video {#video} The video recording is available on YouTube: {{< youtube mCYfONc-rpk >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - Craig's [policy and scripts from today](https://github.com/craigcomstock/play-cfbs/commit/c6860a74b83ff25791b1a4c8bde4110e340f2947) - From post show discussion - Run Enterprise at home with [Free 25](https://cfengine.com/download/) - Run CFEngine on your phone - [ish](https://apps.apple.com/us/app/ish-shell/id1436902243) - [termux](https://termux.dev/en/) - Learning resources - The [CFEngine Help Mailing List](https://groups.google.com/g/help-cfengine/) - [Github Discussions](https://github.com/cfengine/core/discussions) - [Learning CFEngine](https://leanpub.com/learning-cfengine/) by Diego Zamboni - [CFEngine Tutorial](https://www.softcover.io/read/72be7d4f/cf3/) by Aleksey Tsalolikhin - [CFEngine Newsletter](https://cfengine.com/#newsletter-signup-footer) - [Feature Friday](https://cfengine.com/tags/feature-friday/) blog series - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Feature Friday #6: cf-promises" url: /blog/2024/feature-friday-6-cf-promises/index.md date: "2024-04-19" -------------------------------------------------------------------------------- Will your policy work? `cf-promises` can check the CFEngine policy for syntax errors and give you an overview of the host's context. It's always a good idea to check your policy for syntax errors. Consider this policy file: ```cfengine3 {file="/tmp/feature-friday-6.cf"} bundle agent feature_friday { reports: "$(this.promise_filename)" printfile => cat( "$(this.promise_filename)" ) } ``` Can you spot the error? Let's see if `cf-promises` can help: ```command cf-promises -f /tmp/feature-friday-6.cf ``` ```output /tmp/feature-friday-6.cf:6:2: error: syntax error } ^ /tmp/feature-friday-6.cf:6:2: error: Check previous line, Expected ';', got '}' } ^ error: There are syntax errors in policy files ``` The output tells us that there is a syntax error near line 6, column 2. A semicolon (`;`) was expected but instead, a closing curly brace (`}`) was found. We are missing the semicolon that terminates the promise for bundle `feature_friday_6` which is called as a methods promise. If we correct it as follows, and run `cf-promises -f /tmp/feature-friday-6-1.cf`, it no longer complains. ```cfengine3 {file="/tmp/feature-friday-6-1.cf"} bundle agent feature_friday { reports: "$(this.promise_filename)" printfile => cat( "$(this.promise_filename)" ); } ``` `cf-promises` can also give you an overview of classes and variables that will be defined when your policy executes with the `--show-vars` and `--show-classes` options. It won't show you everything, but variables and namespace scoped classes that can be derived without a full three-pass evaluation will be shown. ```command cf-promises --show-vars ``` ```output Variable name Variable value Meta tags Comment default:cfe_autorun_inventory_dmidecode.dmi[bios-vendor] Dell Inc. source=promise,inventory,attribute_name=BIOS vendor default:cfe_autorun_inventory_dmidecode.dmi[bios-version] 1.20.0 source=promise,inventory,attribute_name=BIOS version default:sys.os_name_human Ubuntu source=agent,derived-from=ubuntu default:sys.os_version_major 22 source=agent,derived-from=flavor ``` ```command cf-promises --show-classes ``` ```output Class name Meta tags Comment 192_168_42_232 inventory,attribute_name=none,source=agent,hardclass 20_cpus source=agent,derived-from=sys.cpus,hardclass Afternoon time_based,cfengine_internal_time_based_autoremove,source=agent,hardclass mac_00_00_00_00_00_00 inventory,attribute_name=none,source=agent,hardclass ubuntu_22_04 inventory,attribute_name=none,source=agent,derived-from-file=/etc/os-release,hardclass x86_64 source=agent,derived-from=sys.machine,hardclass ``` But wait, that's not all! `cf-promises` also has the `--syntax-description` option which can output a JSON representation of the syntax. This can be useful for integration with other tooling like editors for syntax checking as well as a way to identify native promise types and attributes. For example, combined with `jq` we can get a list of native promise types: ```command cf-promises --syntax-description=json | jq -r '.promiseTypes|keys[]' ``` ```output access build_xpath classes commands databases defaults delete_attribute delete_lines delete_text delete_tree field_edits files guest_environments insert_lines insert_text insert_tree measurements meta methods packages processes replace_patterns reports roles services set_attribute set_text storage users vars ``` And we can count how many native functions exist: ```command cf-promises --syntax-description=json | jq '.functions|keys|length' ``` ```output 188 ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #5: cfbs" url: /blog/2024/feature-friday-5-cfbs/index.md date: "2024-04-12" -------------------------------------------------------------------------------- Do you maintain multiple policy sets? Do you leverage policy written by others? Ever wished for an easier way to upgrade your policy framework? `cfbs` can help to improve all of these cases. `cfbs` is a command line tool that aims to help simplify managing a policy set and working with [CFEngine Build](https://build.cfengine.com/), a website for finding and sharing modules. A policy set usually - but not always - builds on top of some base, like the Masterfiles Policy Framework (MPF). Custom policy is added on top, and you're off to the races. When a new version of CFEngine is released, the best practice is to upgrade the MPF (assuming you are using it) as the first step in the upgrade process. If you have not modified the MPF, re-integrating the custom policy on top of the new MPF is a relatively straightforward process. If you have modified the MPF it's 1) something you need to know that you did and 2) a process of managing your diffs against the newer version of the MPF. For the case of a non-modified MPF upgrade, `cfbs` makes this very straightforward using `cfbs upgrade`. For the case of a modified MPF, `cfbs` provides an ordered series of steps that can better help to keep track of and maintain modifications. Let's take a look at the process of initializing a `cfbs` project with the MPF and adding some custom policy files. After installing `cfbs` from the Python Package Index (pip), running `cfbs init` is all you need to do. You'll be prompted with a few questions (defaults can be accepted by pressing `ENTER` when prompted). ```command cfbs init ``` ```output Please enter the name of this CFEngine Build project [Example project]: Please enter the description of this CFEngine Build project [Example description]: Do you want cfbs to initialize a git repository and make commits to it? [YES/y/no/n] Please enter user name to use for git commits [Nick Anderson]: Please enter user email to use for git commits [nick.anderson@northern.tech]: Initialized empty Git repository in /home/nickanderson/northern.tech/cfengine/src/website/test/.git/ { "name": "Example project", "description": "Example description", "type": "policy-set", "git": true, "build": [] } The default commit message is 'Initialized a new CFEngine Build project' - edit it? [yes/y/NO/n] Committing using git: [main (root-commit) 64d9389] Initialized a new CFEngine Build project 1 file changed, 7 insertions(+) create mode 100644 cfbs.json Initialized an empty project called 'Example project' in 'cfbs.json' Do you wish to build on top of the default policy set, masterfiles? (Recommended) [YES/y/no/n]: Added module: masterfiles The default commit message is 'Added module 'masterfiles'' - edit it? [yes/y/NO/n] Committing using git: [main da7fbe0] Added module 'masterfiles' 1 file changed, 16 insertions(+), 1 deletion(-) ``` At this point, we still don't have a working policy set, because we need to build it. For that, we use the `cfbs build` command. ```command cfbs build ``` ```output Modules: 001 masterfiles @ 80374429aa8d9f1d5afe952727ae5659caf5e9ef (Downloaded) Steps: 001 masterfiles : run 'EXPLICIT_VERSION=3.21.4 EXPLICIT_RELEASE=1 ./prepare.sh -y' 001 masterfiles : copy './' 'masterfiles/' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` By now, our project directory contains the project file (`cfbs.json`), and the `out` directory where our completed build can be found in `out/masterfiles` contains a built policy set. If you're familiar with the MPF, it should look familiar. ```command ls ``` ```output cfbs.json out ``` ```command ls out/masterfiles | columns -c 50 ``` ```output cfbs.json cfe_internal controls inventory lib modules promises.cf services standalone_self_upgrade.cf templates update.cf ``` With a working base policy set, we can add some custom policy files to the project. Then, we can run `cfbs add` to integrate it into the project. ```command cat > hello-world.cf << EOF bundle agent hello_world #@ Emit hello world { reports: "Hello World!"; } EOF ``` ```command cfbs add ./hello-world.cf ``` ```output Which bundle should be evaluated (added to bundle sequence)? 1. ./hello-world.cf:hello_world (default) 2. (None) [1/2]: 1 Added module: ./hello-world.cf The default commit message is 'Added module './hello-world.cf'' - edit it? [yes/y/NO/n] Committing using git: [main bc51c13] Added module './hello-world.cf' 2 files changed, 17 insertions(+) create mode 100644 hello-world.cf ``` As you can see from the above output, `cfbs` prompted us to ask if the bundle should be added to the bundle sequence, since we want that bundle to run, not just use it from other policy we selected the appropriate option. We selected the default commit message and the module was added. We can now see that reflected in the `cfbs status` command output. ```command cfbs status ``` ```output Name: Example project Description: Example description File: cfbs.json Modules: 001 masterfiles @ 80374429aa8d9f1d5afe952727ae5659caf5e9ef (Downloaded) 002 ./hello-world.cf @ local (Copied) ``` Now, when we build the project using the `cfbs build` command, we will find the `hello-world.cf` policy file in the result. ```command find out/masterfiles -name "hello-world.cf" ``` ```output out/masterfiles/services/cfbs/hello-world.cf ``` Where that file lands in the resulting policy set is governed by the project file (`cfbs.json`), let's have a look. ```json { "name": "Example project", "description": "Example description", "type": "policy-set", "git": true, "build": [ { "name": "masterfiles", "description": "Official CFEngine Masterfiles Policy Framework (MPF).", "tags": ["supported", "base"], "repo": "https://github.com/cfengine/masterfiles", "by": "https://github.com/cfengine", "version": "3.21.4", "commit": "80374429aa8d9f1d5afe952727ae5659caf5e9ef", "added_by": "cfbs add", "steps": [ "run EXPLICIT_VERSION=3.21.4 EXPLICIT_RELEASE=1 ./prepare.sh -y", "copy ./ ./" ] }, { "name": "./hello-world.cf", "description": "Local policy file added using cfbs command line", "tags": ["local"], "added_by": "cfbs add", "steps": [ "copy ./hello-world.cf services/cfbs/hello-world.cf", "policy_files services/cfbs/hello-world.cf", "bundles hello_world" ] } ] } ``` Note the `policy_files` build step for `./hello-world.cf` targets `services/cfbs/hello-world.cf`. You can adjust the build steps to get the desired layout. If you manage just a single policy set, the benefits may not be immediately apparent, but that's not all `cfbs` does. It's also a great way to find and integrate policy written by others. For example, there is a `lynis` module providing policy to install and run CISOfy Lynis as well as a companion `compliance-report-lynis` module that provides a report in the CFEngine Enterprise Mission Portal showing a breakdown of all the detected findings. We can find these modules using the command `cfbs search lynis`: ```command cfbs search lynis ``` ```output compliance-report-lynis - Compliance report with Lynis checks. lynis - Automates the installation, running, and reporting of CISOfy's lynis system audits. ``` And install them with `cfbs add`: ```command cfbs add compliance-report-lynis ``` ```output Added module: compliance-report-imports (Dependency of compliance-report-lynis) Added module: lynis (Dependency of compliance-report-lynis) Added module: autorun (Dependency of compliance-report-imports) Added module: compliance-report-lynis The default commit message is: Added 4 modules - Added module 'lynis' - Added module 'compliance-report-imports' - Added module 'compliance-report-lynis' - Added module 'autorun' Edit it? [yes/y/NO/n] Committing using git: [main 066636b] Added 4 modules 1 file changed, 54 insertions(+) ``` CFEngine Enterprise users can leverage a subset of features in Mission Portal via the Build app. In Mission Portal, you can set up and switch between multiple projects, each connected to a specific branch in a git repository. {{< wideimg "mission-portal-build-project-setup.png" "Screenshot of setting up a Build project with git repo, branch, etc.">}} You can search for, install, and upgrade modules from the [CFEngine Build Index](https://build.cfengine.com) and remove (but not add) modules added from other sources, with a single click. {{< wideimg "mission-portal-build-add-module.png" "Screenshot of adding a module to a Build project">}} Check out our blog post on [migrating to cfbs](/blog/2023/migrating-to-cfbs/). Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #4: cf-remote" url: /blog/2024/feature-friday-4-cf-remote/index.md date: "2024-04-05" -------------------------------------------------------------------------------- What's the easiest way to _install_ cfengine? Have you heard of `cf-remote`? `cf-remote` was born out of a developer's itch for an easy way to get CFEngine installed on some host for testing. We have featured `cf-remote` in several posts[^fn:1] since it was first released in 2019, but today is Friday, so let's review its features. ## Overview {#overview} `cf-remote` (available via the Python Package Index) primarily targets _installing_ CFEngine on a _remote_ host, but it also provides some related conveniences including: - showing some basic remote host info - listing URLs for CFEngine packages - copying files - running commands - deploying policy - spawning and destroying hosts on supported cloud platforms ```command pip3 install cf-remote --user ``` ## Listing package URLs {#listing-package-urls} By default, `cf-remote list` shows URLs for CFEngine Enterprise assets from the most recent LTS, but if you want to know where the 3.21.4 packages for Debian 12 hubs are? It's a simple thing, but `cf-remote` really makes it easy: ```command cf-remote --version 3.21.4 list debian12 hub ``` ```output Available releases: master, 3.23.0, 3.21.x, 3.21.4, 3.21.3, 3.21.2, 3.21.1, 3.21.0, 3.18.x, 3.18.7, 3.18.6, 3.18.5, 3.18.4, 3.18.3, 3.18.2, 3.18.1, 3.18.0 Using 3.21.4 LTS: https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.21.4/hub/debian_12_x86_64/cfengine-nova-hub_3.21.4-1.debian12_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.21.4/hub/debian_12_arm_64/cfengine-nova-hub_3.21.4-1.debian12_arm64.deb ``` In addition to released versions, you can get nightly packages using `master` or the `.x` suffixed versions. For example, to get the URL for the latest nightly masterfiles for 3.18: ```command cf-remote --version 3.18.x list masterfiles ``` ```output Available releases: master, 3.23.0, 3.21.x, 3.21.4, 3.21.3, 3.21.2, 3.21.1, 3.21.0, 3.18.x, 3.18.7, 3.18.6, 3.18.5, 3.18.4, 3.18.3, 3.18.2, 3.18.1, 3.18.0 Using 3.18.x LTS: http://buildcache.cfengine.com/packages/testing-pr/jenkins-3.18.x-nightly-pipeline-685/PACKAGES_x86_64_linux_redhat_7/tarballs/cfengine-masterfiles-3.18.8a.efd0a8cf3-1.pkg.tar.gz ``` If you are looking for community assets, just specify `--edition community` for the `list` command: ```command cf-remote --version 3.18.x list --edition community el7 ``` ```output Available releases: master, 3.23.0, 3.21.x, 3.21.4, 3.21.3, 3.21.2, 3.21.1, 3.21.0, 3.18.x, 3.18.7, 3.18.6, 3.18.5, 3.18.4, 3.18.3, 3.18.2, 3.18.1, 3.18.0 Using 3.18.x LTS: http://buildcache.cfengine.com/packages/testing-pr/jenkins-community-nightly-3.18.x-519/PACKAGES_x86_64_linux_redhat_7/cfengine-community-3.18.8a.759009f84-31534.el7.x86_64.rpm ``` ## Spawning hosts {#spawning-hosts} Let's see what it's like to bring up a small environment of test hosts. Run `cf-remote spawn --list-platforms` to see the platforms `cf-remote` knows how to initialize, then spawn some hosts: ```command cf-remote spawn --platform ubuntu-22-04-x64 --count 1 --size t2.xlarge --name hub --role hub cf-remote spawn --platform ubuntu-18-04-x64 --count 1 --name ubuntu18 --role client cf-remote spawn --platform rhel-7-x64 --count 1 --name rhel7 --role client cf-remote spawn --platform rhel-9-x64 --count 2 --name rhel9 --role client ``` ```output Spawning VMs....DONE Waiting for VMs to get IP addresses...DONE Details about the spawned VMs can be found in /home/nickanderson/.cfengine/cf-remote/cloud_state.json Spawning VMs....DONE Waiting for VMs to get IP addresses...DONE Details about the spawned VMs can be found in /home/nickanderson/.cfengine/cf-remote/cloud_state.json Spawning VMs....DONE Waiting for VMs to get IP addresses...DONE Details about the spawned VMs can be found in /home/nickanderson/.cfengine/cf-remote/cloud_state.json Spawning VMs.....DONE Details about the spawned VMs can be found in /home/nickanderson/.cfengine/cf-remote/cloud_state.json ``` Now we can use `cf-remote show` to see information about spawned hosts: ```command cf-remote show ``` ```output @hub: (1 host in eu-west-1, aws) ubuntu@3.250.76.66 hub, nihub-ubuntu-22-04-x64hub0, ubuntu-22-04-x64, t2.xlarge, 172.31.40.73, 3.250.76.66, d61ec86a6640302a078da276c89cfb139eafd831, nickanderson-cfengine-dev-eu-west-1, cfengine-test @ubuntu18: (1 host in eu-west-1, aws) ubuntu@3.253.129.99 client, niubuntu18-ubuntu-18-04-x64client0, ubuntu-18-04-x64, m3.xlarge, 172.31.1.11, 3.253.129.99, 4c9b0dbaa0d2983f44d85ae071179ee6026e266a, nickanderson-cfengine-dev-eu-west-1, cfengine-test @rhel7: (1 host in eu-west-1, aws) ec2-user@34.243.111.152 client, nirhel7-rhel-7-x64client0, rhel-7-x64, t2.large, 172.31.42.214, 34.243.111.152, 295f5aa1b7f94d60519537bab6e93a916d170bcf, nickanderson-cfengine-dev-eu-west-1, cfengine-test @rhel9: (2 hosts in eu-west-1, aws) ec2-user@54.171.254.178 client, nirhel9-rhel-9-x64client0, rhel-9-x64, m3.xlarge, 172.31.0.193, 54.171.254.178, 12baa67b2527fbdfb8f77d09ad196e3ff855dd9d, nickanderson-cfengine-dev-eu-west-1, cfengine-test ec2-user@18.201.38.169 client, nirhel9-rhel-9-x64client1, rhel-9-x64, m3.xlarge, 172.31.5.170, 18.201.38.169, 31847baef8fcaf2a1764beeb51458e2e92c393c3, nickanderson-cfengine-dev-eu-west-1, cfengine-test Total: 5 hosts in 4 groups ``` ## Host info {#host-info} With some hosts running, we can use `cf-remote` to see some basic host info. We can look at a single host: ```command cf-remote info --hosts ec2-user@54.171.254.178 ``` ```output ec2-user@54.171.254.178 OS : RHEL 9 Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : rpm, yum ``` Or we can look at a group of hosts: ```command cf-remote info --hosts @rhel9 ``` ```output ec2-user@54.171.254.178 OS : RHEL 9 Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : rpm, yum ec2-user@18.201.38.169 OS : RHEL 9 Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : rpm, yum ``` ## Installing & bootstrapping CFEngine {#installing-and-bootstrapping-cfengine} Let's use `cf-remote` to install and bootstrap our provisioned hosts. Let's install an Enterprise hub, we can specify the IP of the host, or we can use the `name` specified during the spawn, `hub`. ```command cf-remote --version master install --remote-download --bootstrap hub --hub hub ``` ```output ubuntu@3.250.76.66 OS : Ubuntu 22 Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : dpkg, apt Downloading 'http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1481/PACKAGES_HUB_x86_64_linux_ubuntu_22/cfengine-nova-hub_3.24.0a.8f1bb17ac~31544.ubuntu22_amd64.deb' on 'ubuntu@3.250.76.66' using curl Installing: 'cfengine-nova-hub_3.24.0a.8f1bb17ac~31544.ubuntu22_amd64.deb' on 'ubuntu@3.250.76.66' CFEngine 3.24.0a.8f1bb17ac (Enterprise) was successfully installed on 'ubuntu@3.250.76.66' Bootstrapping: '3.250.76.66' -> '172.31.40.73' Bootstrap successful: '3.250.76.66' -> '172.31.40.73' ``` Note the use of `--remote-download`: By default, `cf-remote` downloads packages to the machine executing `cf-remote` and then copies the package to the host. Using `--remote-download` the package is downloaded directly by the remote host that is being installed. Now we can install some clients: ```command cf-remote --version master install --remote-download --bootstrap hub --clients ubuntu18 cf-remote --version master install --remote-download --bootstrap hub --clients rhel9 ``` ```output ubuntu@3.253.129.99 OS : Ubuntu 18 Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : dpkg, apt Downloading 'http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1481/PACKAGES_x86_64_linux_ubuntu_18/cfengine-nova_3.24.0a.8f1bb17ac~31544.ubuntu18_amd64.deb' on 'ubuntu@3.253.129.99' using curl Installing: 'cfengine-nova_3.24.0a.8f1bb17ac~31544.ubuntu18_amd64.deb' on 'ubuntu@3.253.129.99' CFEngine 3.24.0a.8f1bb17ac (Enterprise) was successfully installed on 'ubuntu@3.253.129.99' Bootstrapping: '3.253.129.99' -> '172.31.40.73' Bootstrap successful: '3.253.129.99' -> '172.31.40.73' Downloading 'http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1481/PACKAGES_x86_64_linux_redhat_9/cfengine-nova-3.24.0a.8f1bb17ac-31544.el9.x86_64.rpm' on 'ec2-user@18.201.38.169' using curl Downloading 'http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1481/PACKAGES_x86_64_linux_redhat_9/cfengine-nova-3.24.0a.8f1bb17ac-31544.el9.x86_64.rpm' on 'ec2-user@54.171.254.178' using curl Installing: 'cfengine-nova-3.24.0a.8f1bb17ac-31544.el9.x86_64.rpm' on 'ec2-user@18.201.38.169' Installing: 'cfengine-nova-3.24.0a.8f1bb17ac-31544.el9.x86_64.rpm' on 'ec2-user@54.171.254.178' CFEngine 3.24.0a.8f1bb17ac (Enterprise) was successfully installed on 'ec2-user@54.171.254.178' Bootstrapping: '54.171.254.178' -> '172.31.40.73' Bootstrap successful: '54.171.254.178' -> '172.31.40.73' CFEngine 3.24.0a.8f1bb17ac (Enterprise) was successfully installed on 'ec2-user@18.201.38.169' Bootstrapping: '18.201.38.169' -> '172.31.40.73' Bootstrap successful: '18.201.38.169' -> '172.31.40.73' ``` Note there are other interesting options like `--demo` and `--call-collect` that can be useful in development environments. To see all the options for _install_ run `cf-remote install --help`. ## Destroying spawned hosts {#destroying-spawned-hosts} Now that we are done with our little test infrastructure, we can clean it up with `cf-remote destroy`: ```command cf-remote destroy --all ``` ```output Destroying all hosts ``` Happy Friday! 🎉 [^fn:1]: Blog posts about `cf-remote` Checkout the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Inventory and remediate Red Hat Enterprise Linux with Security Technical Implementation Guides (STIGs)" url: /blog/2024/stig-rhel-7/index.md date: "2024-04-01" -------------------------------------------------------------------------------- [Security Technical Implementation Guides (STIGs)](https://public.cyber.mil/stigs/) are an excellent body of knowledge to leverage in securing your infrastructure. With the [stig-rhel-7](https://build.cfengine.com/modules/stig-rhel-7/) module you can easily add inventory and remediation policy for RHEL 7 with CFEngine. Do note that as of March 2024 this module does not provide comprehensive coverage but rather an initial 10 findings are implemented. ## Setup To start I installed CFEngine Enterprise on a [local virtual machine](https://docs.cfengine.com/docs/3.21/getting-started-installation-local-virtual-machine.html), logged in and started a new Build project with the `stig-rhel-7` module added and configured to enforce (as opposed to only warn). {{< wideimg "images/stig-rhel-7-module-added-and-configured.png" "stig-rhel-7 module added and configured">}} ## Try it out The STIG entries which we currently have remediation measures for involve uninstalling software which generally should not be installed for security reasons: rsh-server, ypserv, telnet-server, vsftpd and tftp-server. After some searching at pkgs.org I found a vsftpd rpm file to install to test compliance and remediation. ```shell curl -O http://mirror.centos.org/centos/7/os/x86_64/Packages/vsftpd-3.0.2-28.el7.x86_64.rpm sudo rpm -i vsftpd-3.0.2-28.el7.x86_64.rpm ``` After installing that package I run the policy and see that the software is removed as requested by the `enforce` setting. ```console R: rhel_7_stig:V_204497 is Enforcing without exception but automatic remediation is not available Warning: RPMDB altered outside of yum. info: Successfully removed package 'vsftpd' R: rhel_7_stig:V_204594 is Enforcing without exception but automatic remediation is not available R: rhel_7_stig:V_204502 is Enforcing without exception but automatic remediation is not available R: rhel_7_stig:V_204425 is Enforcing without exception but automatic remediation is not available R: rhel_7_stig:V_204424 is Enforcing without exception but automatic remediation is not available ``` ## Compliance reports Additionally I can view the state of these other findings in my entire infrastructure with our compliance reports. {{< wideimg "images/stig-rhel-7-module-compliance-report.png" "stig-rhel-7 module compliance report">}} A link is provided to drill down and see which hosts are affected by what findings. {{< wideimg "images/stig-rhel-7-module-host-report.png" "list of hosts failing stig-rhel-7 compliance">}} ## Contribute Help us make this module more comprehensive. Please consider [contributing](https://github.com/nickanderson/cfengine-security-hardening/tree/master/stig/red_hat_enterprise_linux_7)! ## Questions? If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). -------------------------------------------------------------------------------- title: "Feature Friday #3: execresult_as_data()" url: /blog/2024/feature-friday-3-execresult-as-data/index.md date: "2024-03-29" -------------------------------------------------------------------------------- When you want to inspect both the return code and output from a command [execresult_as_data()](https://docs.cfengine.com/latest/reference-functions-execresult_as_data.html) might be the function you are searching for. Most CFEngine policy writers have used [execresult()](https://docs.cfengine.com/latest/reference-functions-execresult.html) and [returnszero()](https://docs.cfengine.com/latest/reference-functions-returnszero.html). They are useful when you want to do something based on the output of a command or based on its successful execution (returning zero). For example: ```cfengine3 {file="/tmp/feature-friday-3.cf"} bundle agent __main__ { vars: "hostname" string => execresult( "$(paths.hostname)", "useshell" ); classes: "my_command_returned_zero" expression => returnszero( "$(paths.hostname)", "noshell" ); reports: "$(hostname)"; my_command_returned_zero:: "$(paths.hostname) returned 0"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-3.cf ``` ```output precision-5570 R: precision-5570 R: /bin/hostname returned 0 ``` But, sometimes, you care about specific return codes. However, you might also want to consider the output from a command. `execresult_as_data()` facilitates inspecting both output and return code from a command's execution, by returning a data structure. Let's take a look. ```cfengine3 {file="/tmp/feature-friday-3-1.cf"} bundle agent __main__ { vars: "i_got" data => execresult_as_data( "$(paths.hostname)", "useshell" ); reports: "$(paths.hostname) emitted '$(i_got[output])' and returned '$(i_got[exit_code])'"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-3-1.cf ``` ```output R: /bin/hostname emitted 'precision-5570' and returned '0' ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 35 - Automatic grouping of hosts with select_class" url: /blog/2024/the-agent-is-in-episode-35/index.md date: "2024-03-28" -------------------------------------------------------------------------------- Have you ever wanted some hosts to organize themselves into different groups, but without spending time to worry about assigning a specific group? Cody, Craig, and Nick talk about using the `select_class` feature in CFEngine. Nick implements automatic assignment of a maintainer for each host: - You want to assign a maintainer to each host that should be the primary point of contact. - Some hosts may have a specific maintainer, but generally, you don't care about the relationship between the person and the actual function of the machine. - It's okay if the groups are not _perfectly_ balanced. First we provide a list of maintainers to select from. ```cfengine3 bundle agent auto_maintainer { vars: "maintainers" slist => { "Alvin", "Simon", "Theodore" }; ``` Next we generate classes for each maintainer in the form `maintainer_Name`. ```cfengine3 "maintainer_options" slist => maplist( "maintainer_$(this)", @(maintainers) ); ``` Then if a maintainer has been automatically selected we define a variable to hold the maintainer's name, tagged for inventory so that it's easy to visualize in Mission Portal. ```cfengine3 maintainer_automatically_selected:: "maintainer" string => "$(maintainers)", if => "maintainer_$(maintainers)", meta => { "inventory", "attribute_name=Maintainer" }; ``` Finally, we define the `maintainer_automatically_selected` class along with the selected maintainer class if a maintainer has not already been defined in the `data:variables.maintainer` variable. ```cfengine3 classes: "maintainer_automatically_selected" select_class => { @(maintainer_options) }, if => not( isvariable( "data:variables.maintainer" ) ); } ``` We saw that we got a pretty even distribution of maintainers without perseverating about each assignment. {{< figure src="maintainer-inventory.png" >}} ## Video {#video} The video recording is available on YouTube: {{< youtube t9z7Ukzy860 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Post show chit-chat {#post-show-chit-chat} - Is there an API or functionality to queue up new hosts for later trust acceptance? - No, not currently, but please consider commenting on [CFE-4362](https://northerntech.atlassian.net/browse/CFE-4362) - PSA: Check out the [Hacker News post](https://news.ycombinator.com/item?id=39817497) for CFEngine's Star Trek and AI Origins (2023) - You can log promise outcomes with `log_kept`, `log_repaired` and `log_failed` attributes in `action` bodies.[^fn:1], [^fn:2], [^fn:3] - You can automatically attach a default body to all promises not using that body with default bodies.[^fn:4] - You can direct reports promises output to a file instead of standard out with the `report_to_file` attribute.[^fn:5] ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) [^fn:1]: Documentation for `log_kept` attribute in `action` bodies: [^fn:2]: Documentation for `log_repaired` attribute in `action` bodies: [^fn:3]: Documentation for `log_failed` attribute in `action` bodies: [^fn:4]: Documentation for default bodies: [^fn:5]: Documentation for `report_to_file` attribute for `reports` promises: -------------------------------------------------------------------------------- title: "Feature Friday #2: $(with)" url: /blog/2024/feature-friday-2-with/index.md date: "2024-03-22" -------------------------------------------------------------------------------- Ever wanted to manipulate a string - temporarily - for an individual promise? Check out the `with` attribute and its special, `$(with)` variable. Sometimes you need some variation on a string for a specific case. Traditionally, to achieve this you'd simply define another variable. Here is a contrived example: I have a string, `nginx` and I want to emit a report that contains both the string itself and the upper case version of the string. ```cfengine3 {file="/tmp/feature-friday-2.cf"} bundle agent __main__ { vars: "MyString" string => "nginx"; "MyString_upcase" string => string_upcase( "$(MyString)" ); reports: "'$(MyString)' upper-cased is '$(MyString_upcase)'"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-2.cf ``` ```output R: 'nginx' upper-cased is 'NGINX' ``` The `with` attribute lets us define the `$(with)` variable for a specific promise, letting us avoid the definition of a variable simply to hold the upper case version. ```cfengine3 {file="/tmp/feature-friday-2-1.cf"} bundle agent __main__ { vars: "MyString" string => "nginx"; reports: "'$(MyString)' upper-cased is '$(with)'" with => string_upcase( "$(MyString)"); } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-2-1.cf ``` ```output R: 'nginx' upper-cased is 'NGINX' ``` The benefits of this really start to shine when you need to do the same translation on a list being iterated: ```cfengine3 {file="/tmp/feature-friday-2-2.cf"} bundle agent __main__ { vars: "MyStrings" slist => { "nginx", "apache", "cherokee", "tomcat", "lighthttpd" }; reports: "'$(MyStrings)' upper-cased is '$(with)'" with => string_upcase( "$(MyStrings)"); } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-2-2.cf ``` ```output R: 'nginx' upper-cased is 'NGINX' R: 'apache' upper-cased is 'APACHE' R: 'cherokee' upper-cased is 'CHEROKEE' R: 'tomcat' upper-cased is 'TOMCAT' R: 'lighthttpd' upper-cased is 'LIGHTHTTPD' ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday #1: ifelse()" url: /blog/2024/feature-friday-1-ifelse/index.md date: "2024-03-15" -------------------------------------------------------------------------------- Looking for a way to concisely set a variable conditionally? Have you heard of `ifelse()`? In CFEngine, traditionally _class expressions_ are used to constrain promises to different contexts. Setting a variable to different values based on context might look like this: ```cfengine3 {file="/tmp/feature-friday-1.cf"} bundle agent __main__ { vars: "MyVariable" string => "My Default value"; redhat_8|centos_8|rocky_8:: "MyVariable" string => "My value for EL 8"; ubuntu_22:: "MyVariable" string => "My value for Ubuntu 22"; any:: "MyVariable" string => "My value on Friday", if => "Friday"; reports: "It's $(sys.date) and I am running on $(sys.os_release[PRETTY_NAME])"; "MyVariable is '$(MyVariable)'"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-1.cf ``` ```output R: It's Mon Mar 11 12:36:41 2024 and I am running on Ubuntu 22.04.4 LTS R: MyVariable is 'My value for Ubuntu 22' ``` That's great, lots of flexibility, but with an increasing number of options the policy can get quite long and it's easier for a human interpreter to lose track of the context. The same can be achieved in a single statement using `ifelse()`. From the [documentation for ifelse()](https://docs.cfengine.com/latest/reference-functions-ifelse.html) > The `ifelse()` function is like a multi-level if-else statement. It was inspired by Oracle's `DECODE` function. It must have an odd number of arguments (from 1 to N). The last argument is the default value, like the `else` clause in standard programming languages. Every pair of arguments before the last one are evaluated as a pair. If the first one evaluates true then the second one is returned, as if you had used the first one in a class expression. So the first item in the pair can be more than just a class name, it's a whole context like `Tuesday.linux.!verbose`). Using `ifelse()` the above example can be distilled into something like this: ```cfengine3 {file="/tmp/feature-friday-1-1.cf"} bundle agent __main__ { vars: "MyVariable" string => ifelse( "Friday", "My value on Friday", "ubuntu_22", "My value for Ubuntu 22", "redhat_8|centos_8|rocky_8", "My value for EL8", "My Default value" ); reports: "It's $(sys.date) and I am running on $(sys.os_release[PRETTY_NAME])"; "MyVariable is '$(MyVariable)'"; } ``` ```command cf-agent --no-lock --log-level info --file /tmp/feature-friday-1-1.cf ``` ```output R: It's Mon Mar 11 12:36:57 2024 and I am running on Ubuntu 22.04.4 LTS R: MyVariable is 'My value for Ubuntu 22' ``` Happy Friday! 🎉 Check out the rest of the [posts in the series](/blog/2024/feature-friday/). -------------------------------------------------------------------------------- title: "Feature Friday: A 42-part saga" url: /blog/2024/feature-friday/index.md date: "2024-03-11" -------------------------------------------------------------------------------- Last year in commemoration of 30 years of CFEngine I received this lovely coin. {{< figure src="agent-coin-42.jpg" >}} I thought it would be fun to celebrate by writing 42 blog posts, each with some little tip about CFEngine. So, be sure to come check in on Friday for the first post in the Feature Friday series. ## Series Posts {#series-posts} 1. [ifelse()](/blog/2024/feature-friday-1-ifelse) 2. [$(with)](/blog/2024/feature-friday-2-with) 3. [execresult_as_data()](/blog/2024/feature-friday-3-execresult-as-data) 4. [cf-remote](/blog/2024/feature-friday-4-cf-remote) 5. [cfbs](/blog/2024/feature-friday-5-cfbs) 6. [cf-promises](/blog/2024/feature-friday-6-cf-promises) 7. [depends_on](/blog/2024/feature-friday-depends-on) 8. [--bundlesequence](/blog/2024/feature-friday-8-bundlesequence) 9. [body file control - inputs](/blog/2024/feature-friday-9-body-file-control-inputs) 10. [cf-support](/blog/2024/feature-friday-10-cf-support) 11. [namespaces](/blog/2024/feature-friday-11-namespaces) 12. [special variables](/blog/2024/feature-friday-12-special-variables) 13. [classesmatching()](/blog/2024/feature-friday-13-classesmatching) 14. [variablesmatching() & variablesmatching_as_data()](/blog/2024/feature-friday-14-variablesmatching-variablesmatching-as-data) 15. [bundlesmatching()](/blog/2024/feature-friday-15-bundlesmatching) 16. [host info report](/blog/2024/feature-friday-16-host-info-report) 17. [Tags for Inventory and Reporting](/blog/2024/feature-friday-17-tags-for-inventory-and-reporting) 18. [Augments - def.json](/blog/2024/feature-friday-18-augments-def-json) 19. [What variables and classes are defined?](/blog/2024/feature-friday-19-what-variables-and-classes-are-defined) 20. [Macros](/blog/2024/feature-friday-feature-friday-20-macros) 21. [Promisees or Stakeholders](/blog/2024/feature-friday-21-promisees-or-stakeholders) 22. [Don't fix, just warn](/blog/2024/feature-friday-22-dont-fix-just-warn) 23. [Agent say!](/blog/2024/feature-friday-23-agent-say) 24. [Augments - host_specific.json](/blog/2024/feature-friday-24-augments-host-specific-json) 25. [Unprivileged execution](/blog/2024/feature-friday-25-unprivileged-execution) 26. [Groups custom promise type](/blog/2024/feature-friday-26-groups-custom-promise-type) 27. [Multiple outcomes](/blog/2024/feature-friday-27-multiple-outcomes) 28. [Restricting individual promises using if and unless](/blog/2024/feature-friday-28-restricting-individual-promises-using-if-and-unless) 29. [Variable class expressions](/blog/2024/feature-friday-29-variable-class-expressions) 30. [Agent svg](/blog/2024/feature-friday-30-agent-svg) 31. [Seeing a data structure with storejson()](/blog/2024/feature-friday-31-seeing-a-data-structure-with-storejson) 32. [Doing math in policy with eval()](/blog/2024/feature-friday-32-doing-math-in-policy-with-eval) 33. [Why associative arrays when data containers exist?](/blog/2024/feature-friday-33-why-associative-arrays-when-data-containers-exist) 34. [Self organizing groups](/blog/2024/feature-friday-34-self-organizing-groups) 35. [Groups in Mission Portal](/blog/2024/feature-friday-35-groups-in-mission-portal) 36. [Formatting policy with cffmt](/blog/2024/feature-friday-36-formatting-policy-with-cffmt/) 37. [Decisions based on arbitrary semantic versions](/blog/2024/feature-friday-37-decisions-based-on-arbitrary-semantic-versions) 38. [Developing modules that take input](/blog/2024/feature-friday-38_developing_modules_that_take_input/) 39. [The power of lists and implicit iteration](/blog/2024/feature-friday-39-the-power-of-lists-and-implicit-iteration/) 40. [What would CFEngine do?](/blog/2024/feature-friday-40-what-would-cfengine-do/) 41. [How can I quote thee, let me count the ways](/blog/2024/feature-friday-41-how-can-i-quote-thee/) 42. [ob-cfengine3](/blog/2024/feature-friday-42-ob-cfengine3/) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 34 - Host visibility with containerized CFEngine" url: /blog/2024/the-agent-is-in-episode-34/index.md date: "2024-02-29" -------------------------------------------------------------------------------- Curious about getting visibility of a host where you can't install CFEngine natively, but can run a container? Craig shows us how he's getting inventory from his [Torizon Verdin IMX8MP](https://developer.toradex.com/hardware/verdin-som-family/modules/verdin-imx8m-plus/) by deploying a container with a chroot containing volumes bind mounted from the host. A long post show discussion was had about [CfgMgmtCamp](https://cfgmgmtcamp.eu/) as well as a long troubleshooting session dealing with a prickly package management situation. ## Video {#video} The video recording is available on YouTube: {{< youtube JwXudOefnWE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 33 - Interview with Mark Burgess" url: /blog/2024/the-agent-is-in-episode-33/index.md date: "2024-01-25" -------------------------------------------------------------------------------- The CFEngine team celebrates 30 years of CFEngine and has a chat with Mark Burgess, the original creator and author of CFEngine. {{< figure src="mark-burgess-2016.jpg" >}} Mark delves into the early chapters of his journey, tracing back the roots of his inspiration to science fiction sagas like [Star Trek](https://en.wikipedia.org/wiki/Star_Trek) and [Dr. Who](https://en.wikipedia.org/wiki/Doctor_Who), and the literary works of [Isaac Asimov](https://en.wikipedia.org/wiki/Isaac_Asimov), [Aurthur C. Clarke](https://en.wikipedia.org/wiki/Arthur_C._Clarke), and [Paul Davies](https://en.wikipedia.org/wiki/Paul_Davies). Mark recalled his first computer, the fabulous, most programmable, decomposable, put back together-able, expandable thing, a BBC-b microcomputer. Mark moves on to recall his education and experience in academia, from taking courses from Paul Davies at New Castle and migration to Norway with a fellowship from the Royal Society of London to study with [Finn Ravndal](https://en.wikipedia.org/wiki/Finn_Ravndal). He ultimately became responsible for managing a clutch of Sun workstations brought into the department by [Jon Magne Leinaas](https://en.wikipedia.org/wiki/Jon_Magne_Leinaas), an experience that sparked ideas for simplifying automation, what eventually became CFEngine. Impressed with scripts written by [Bjorn Remseth](https://www.linkedin.com/in/remseth/) that ran across all hosts in the university doing backups and other simple tasks Mark began trying to find a way to separate the logic of environment detection from the configuration and wrote an initial version of CFEngine. After writing a [SLAC preprint introducing CFEngine](https://markburgess.org/papers/cfengine_history.pdf) and presenting CFEngine for CERN Mark decided to donate the code to the GNU foundation. Mark began lecturing about Unix and continued doing research in computing and working on CFEngine, eventually authoring [Adaptive Locks For Frequently Scheduled Tasks With Unpredictable Runtimes](https://www.usenix.org/legacy/publications/library/proceedings/lisa97/full_papers/15.burgess/15.pdf) with Demosthenes Skipitaris delivered at [LISA ’97](https://www.usenix.org/legacy/publications/library/proceedings/lisa97/). It was at this conference that Mark became aware of the growing interest in CFEngine when the Birds of a Feather session was overflowed. After publishing a paper on [Computer Immunology](https://www.usenix.org/legacy/event/lisa98/full_papers/burgess/burgess.pdf) while continuing to work on CFEngine Mark began to work with [Jan Bergstra](https://en.wikipedia.org/wiki/Jan_Bergstra) ultimately developing Promise Theory with its tenet of voluntary cooperation. {{< figure src="promise-theory.jpg" >}} By 2007, ready for a new challenge and disenchanted with academia Mark re-focused his efforts on CFEngine, overhauling the code-base and language bringing it in closer alignment with Promise Theory and released CFEngine 3. Together with [Thomas Ryd](https://www.linkedin.com/in/thomasryd/) he formed a company to bring Enterprise features and support. Mark discussed his thoughts on the wheel of time and how progress seems to be a bunch of strange loops inching forward. Many of the conversations and ideas from early CFEngine days continue to be discussed and implemented in new projects today and Promise Theory is being applied across many domains, from economics to business management. Mark also noted how Promise Theory was being used to help understand trust and mistrust, accurately predicting how mistrust fuels the coming together of people and information management on Wikipedia, interestingly also fitting [Robin Dunbar's](https://en.wikipedia.org/wiki/Robin_Dunbar) theories on group sizes. Mark closes with some thoughts about the future and AI and an open call for collaboration on [his passion of music](https://markburgess.org/music.html). {{< figure src="cfengine-30-coin.jpg" >}} ## Video {#video} The video recording is available on YouTube: {{< youtube KrF9R11AFis >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). - [00:00:18](https://youtu.be/KrF9R11AFis?t=18) - An incomplete introduction to Mark Burgess - [00:01:32](https://www.youtube.com/watch?v=KrF9R11AFis&t=92s) - In the beginning ... (before CFEngine) - [00:10:24](https://www.youtube.com/watch?v=KrF9R11AFis&t=624s) - The idea of CFEngine - [00:25:53](https://www.youtube.com/watch?v=KrF9R11AFis&t=1553s)-[00:26:45](https://youtu.be/KrF9R11AFis?t=1605) - Technical difficulties - [00:26:45](https://youtu.be/KrF9R11AFis?t=1605) - The seed for Promise Theory - [00:29:11](https://www.youtube.com/watch?v=KrF9R11AFis&t=1751s) - The formalization of Promise Theory - [00:40:53](https://www.youtube.com/watch?v=KrF9R11AFis&t=2453s) - Entropenaursihip (2007) - [00:49:04](https://youtu.be/KrF9R11AFis?t=2946) - Knowledge Management - [00:58:33](https://www.youtube.com/watch?v=KrF9R11AFis&t=3513s) - Progress - [01:03:32](https://www.youtube.com/watch?v=KrF9R11AFis&t=3812s) - Promise Theory in other domains - [01:08:12](https://www.youtube.com/watch?v=KrF9R11AFis&t=4092s) - Trust & mistrust - [01:20:08](https://www.youtube.com/watch?v=KrF9R11AFis&t=4808s) - AI & the future - [01:25:43](https://www.youtube.com/watch?v=KrF9R11AFis&t=5143s) - Open call for collaboration on music ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Change in behavior: The arglist attribute now preserves spaces" url: /blog/2024/change-in-behavior-arglist/index.md date: "2024-01-17" -------------------------------------------------------------------------------- When executing commands in a shell, the program and its arguments are typically separated by spaces. This command reads the content of two files and prints it out (concatenating it): ```command cat one.txt two.txt ``` In the example above, `cat` is the command / program, while `one.txt` is the first argument, and `two.txt` is the second argument. This is great because it makes commands really easy to read and type, however there is one obvious drawback: When a space has a special meaning (separating arguments), what do you do when you actually need a space? An example of this could be to want to print files with spaces in their filenames. The most common approach is to use quotes which also have a special meaning: ```command cat 'file one.txt' 'file two.txt' ``` Now, the first argument is what's between the first pair of quotes (`file one.txt`) and the second argument is predictably what's between the second pair of quotes (`file two.txt`) [^fn:1]. In CFEngine, `commands` promises are used to run commands like these, usually in one of three ways: ```cfengine3 {file="example.cf"} bundle agent main { commands: # First way "command with arguments"; # Second way "command" args => "with arguments"; # Third way "command" arglist => { "with", "arguments" }; } ``` In this blog post, we'll take a look at a recent bugfix for the `arglist` attribute. Because of the change in behavior required for this fix, some users may have to review and update their policy. The original behavior of the `arglist` attribute in the `commands` promise must clearly have been intended to allow for arguments with embedded spaces without having to quote or escape them, as you would normally have to do in a shell. The documentation seemed to support this claim as well: > As with `args`, it is convenient to separate command and arguments. With `arglist` you can use an `slist` directly instead of having to provide a single string as with `args`. That's particularly useful when there are embedded spaces and quotes in your arguments, but also when you want to get them directly from an `slist` without going through `join()` or other functions. - https://docs.cfengine.com/docs/3.15/reference-promise-types-commands.html#arglist However, this has never been the case. Causing numerous community members to (rightfully so) report confusion in our bug tracker with tickets dating back to the year 2017. See [CFE-2724](https://northerntech.atlassian.net/browse/CFE-2724), [CFE-2869](https://northerntech.atlassian.net/browse/CFE-2869), and [CFE-4253](https://northerntech.atlassian.net/browse/CFE-4253). Hence, it is about time we do something about it. As an example, the following policy; ```cf3 {file="example.cf"} bundle agent __main__ { files: "/tmp/test.py" # A program that echo's each argument content => "import sys; print(*(arg for arg in sys.argv), sep='\n')"; commands: "/usr/bin/python3 /tmp/test.py" arglist => { "one", "two three" }; } ``` would produce the following output; ```command cf-agent -Kf ~/test.cf ``` ```output notice: Q: "...in/python3 /tmp": /tmp/test.py Q: "...in/python3 /tmp": one Q: "...in/python3 /tmp": two Q: "...in/python3 /tmp": three ``` while one would expect; ```command cf-agent -Kf ~/test.cf ``` ```output notice: Q: "...in/python3 /tmp": /tmp/test.py Q: "...in/python3 /tmp": one Q: "...in/python3 /tmp": two three ``` For the next 3.24 LTS release we have a fix for this bug. Thus, whitespace is preserved in arguments given through the `arglist` attribute. This is **unless** the `useshell` attribute is set to `"useshell"` or `"powershell"`, as the shell itself would interpret spaces as argument separators. Also, due to limitations in the Win32 API, the fix is not available for Windows yet. ## What to do {#what-to-do} As a workaround, we have previously suggested the use of nested quoting as illustrated in the following example: ```cf3 {file="example.cf"} bundle agent __main__ { files: "/tmp/test.py" # A program that echo's each argument content => "import sys; print(*(arg for arg in sys.argv), sep='\n')"; commands: "/usr/bin/python3 /tmp/test.py" arglist => { "one", "'two three'" }; } ``` The example above would produce the following output: ```command cf-agent -Kf ~/test.cf ``` ```output notice: Q: "...in/python3 /tmp": /tmp/test.py Q: "...in/python3 /tmp": one Q: "...in/python3 /tmp": two three ``` However, the bug fix would pose a breaking change for any uses of this workaround. This is because the enclosed quotes would now be a part of the argument. The example above would produce the following output in CFEngine 3.24: ```command cf-agent -Kf ~/test.cf ``` ```output notice: Q: "...in/python3 /tmp": /tmp/test.py Q: "...in/python3 /tmp": one Q: "...in/python3 /tmp": 'two three' ``` **Note:** It is worth noting that this change only affects the `arglist` attribute and not `args`. Thus, when upgrading to CFEngine 3.24, you should walk through your policy and remove the enclosing quotes wherever this workaround is used in `arglist`, unless the `useshell` attribute is set to `"useshell"` or `"powershell"`. As an example, you can use the following command to locate all occurrences of `arglist` from within your [MPF](https://docs.cfengine.com/docs/3.21/reference-masterfiles-policy-framework.html#top) directory: ```command grep --colour -Irn . -e "arglist" --include="*.cf" ``` ```output ---snip--- ./services/autorun/bogus.cf:10: arglist => { "one", "'two three'" }; ./services/autorun/bogus.cf:12: arglist => { "five", "'$(six)'" }; ./services/autorun/bogus.cf:15: arglist => { "seven", "`eight nine`" }; ./services/autorun/bogus.cf:17: arglist => { "ten", "eleven" }; ---snip--- ``` For each located occurrence, determine if the promise uses a shell. If not, remove any nested quotes from strings in the `slist` or any referenced variables. See example below: ```cf3 {file="example.cf"} body contain example { useshell => "useshell"; } bundle agent main { commands: "/usr/bin/foo" # Requires fixing arglist => { "one", "'two three'" }; "/usr/bin/bar" # Requires further investigation arglist => { "five", "$(six)" }; "/usr/bin/baz" # This one is fine as is contain => example, arglist => { "seven", "`eight nine`" }; "/usr/bin/qux" # This one is fine as is arglist => { "ten", "eleven" }; } ``` If you have questions or need help, reach out on the [mailing list](https://groups.google.com/g/help-cfengine) or [GitHub discussions](https://github.com/cfengine/core/discussions). If you have a support contract, feel free to open a ticket in our [support system](https://support.northern.tech). [^fn:1]: An observant reader might notice that we need similar tricks now if we want quotes inside our arguments. For those quotes, we'd need to use escape characters, or more quotes, but let's skip this since it's not relevant for what we're looking at here. -------------------------------------------------------------------------------- title: "CFEngine 3.18.7 and 3.21.4 released" url: /blog/2024/cfengine-3-18-7-and-3-21-4-released/index.md date: "2024-01-11" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.18.7 and 3.21.4\! These patch releases contain bug fixes and dependency updates. ## Changes We'd like to highlight one specific change in behavior, which some users will want to adjust to; ### Change in behavior - New location for ignore_interfaces.rx Users who rely on the `ignore_interfaces.rx` file for ignoring certain network interfaces should be aware of its new _preferred_ location. See the blog post on the topic for more details: https://cfengine.com/blog/2023/change-in-behavior-ignore_interfaces.rx-prefers-new-location/ ### Complete changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.21.4 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.4 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.4 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) - [3.18.7 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.7 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.7 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ### Dependency upgrades Compared to the previous releases, CFEngine [3.18.6 and 3.21.3](https://cfengine.com/blog/2023/cfengine-3-18-6-and-3-21-3-released/), the following dependencies were updated for the client packages: | CFEngine version | 3.18.6 | **3.18.7** | 3.21.3 | **3.21.4** | | :------------------------------------------------ | :----- | :--------- | :----- | :--------- | | [libcurl](https://curl.haxx.se/download.html) | 8.2.1 | **8.5.0** | 8.2.1 | **8.5.0** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.11.5 | **2.12.3** | 2.11.5 | **2.12.3** | | [OpenSSL](https://openssl.org/) | 1.1.1v | **1.1.1w** | 3.0.10 | **3.0.12** | And for the hub packages, these additional dependencies were also upgraded: | CFEngine version | 3.18.6 | **3.18.7** | 3.21.3 | **3.21.4** | | :-------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.57 | **2.4.58** | 2.4.57 | **2.4.58** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.42.0 | **2.43.0** | 2.42.0 | **2.43.0** | | [PHP](https://php.net/) | 8.0.30 | 8.0.30 | 8.2.9 | **8.2.13** | | [PostgreSQL](https://www.postgresql.org/) | 13.12 | **13.13** | 15.4 | **15.5** | For a complete overview of all dependencies used in CFEngine and their licenses, please see [the **Legal and licenses** part of our documentation](https://docs.cfengine.com/docs/3.21/release-notes-legal-and-licenses.html). Thank you to the developers, maintainers, and contributors to open source software which makes CFEngine possible. ## New supported platform - Debian 12 Debian 12 is now supported, with 4 enterprise packages available for the platform; 1. Debian 12 hub package for 64-bit x86 (amd64) 2. Debian 12 hub package for 64-bit ARM (arm64) 3. Debian 12 client package for 64-bit x86 (amd64) 4. Debian 12 client package for 64-bit ARM (arm64) See the **Downloads** section below for more details on how to download and install them. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.21.4 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.18.7 list ``` ```command cf-remote --version 3.21.4 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## End of support for CFEngine 3.18 LTS Please note that support for all 3.18 versions of CFEngine will end later this year, in August 2024. One more final release is planned for the 3.18 LTS series before then. If you still haven't upgraded to 3.21 LTS, we encourage you to [contact us](/contact) for help with upgrading or to discuss an extended support contract. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "CFEngine 2023 retrospective" url: /blog/2023/2023-retrospective/index.md date: "2023-12-31" -------------------------------------------------------------------------------- As we bid farewell to 2023, it's once again time to reflect on the milestones and progress we've made throughout the year. This year is especially significant because it marks 30 years since CFEngine's birth. From its humble beginnings at the University of Oslo to the thousands of implementations across the world and counting, CFEngine has not only weathered the twists and turns of life, but has thrived in an industry where many have come and gone. While we here at CFEngine believe the technology itself is part of what makes it so special, it's also because of the loyal users and community who have contributed over the years. Personally speaking, it's an honor to have played a small part in its overall success. There aren't many times in life where you get an opportunity to work with such an instrumental technology. From the depths of the ocean floor, to the edges of space, to services that you depend on in your daily lives, CFEngine is still being used and deployed in the most critical infrastructure environments because of its reliability, battle-tested technology, and the fact that "it just works". Without further ado, here are the highlights from 2023 and a sneak peek into what to expect for 2024: ## Webinars: 'The agent is in' This year we celebrated the show's 2nd year anniversary. Whether you're an expert or new to CFEngine, join us every last Thursday of the month as we cover new tips and tricks and hangout as a community. Our January's webinar will feature the creator of CFEngine, Mark Burgess. Don't miss this chance to hear his story and how CFEngine came to be. [Reserve](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw) your spot now. You can catch all previous episodes [here](https://www.youtube.com/playlist?list=PLh71Vl9YjMaji_jnHFZ3k4mDbyOCE3A2x) on the official CFEngine YouTube channel. ## Feature releases: 3.22 and 3.23 (non-LTS) 2023 brought two new feature releases - [3.22](/blog/2023/cfengine-3-22-released-coordination/) and [3.23](/blog/2023/cfengine-3-23-released-anniversary/). These are not LTS releases, not supported, but allow us to introduce new features and receive feedback before the upcoming 3.24 LTS release next summer. CFEngine 3.23 was codenamed ['anniversary'](/blog/2023/cfengine-3-23-released-anniversary/) to commemorate 30 years of CFEngine. ### Feature: Groups in Mission Portal Using CFEngine there are many ways to group and classify your hosts. In order to group their hosts, our users use a combination of JSON files, CFEngine policy language (with variables, classes, and class expressions), host specific data and host filters in Mission Portal. There was, however, no straightforward way for a Mission Portal user to save a selection of hosts (a filter) and then start doing things (reports, changes) with those hosts. The new Groups feature enables making configuration changes to many hosts at one time, and also allows saving selections of hosts and reusing them in various reports, making Mission Portal more intuitive and user-friendly. {{< wideimg "images/groups.png" "Groups application showing a group of Ubuntu hosts across different versions.">}} ### Feature: Include / exclude hosts by hostname, IP address, subnet, MAC address We are continuing to improve the inventory reports and their host filters within Mission Portal by adding more ways to specify lists of hosts to include and exclude. By specifying lists of hosts by their IP, MAC, or hostname, they will stay in the list, even if their ID changes (for example after reinstalling CFEngine on a host, rotating a host's key, or replacing one host with another). When combining rules with lists of hosts to include and exclude, exclusion will always take precedence, so the hosts you put into the exclusion list will never show up in the results. {{< img "images/host-include-exclude.png" "Host include / exclude dialog showing lists of hosts can be included and excluded based on hostname, IP, MAC address and CFEngine ID">}} ## LTS releases: 3.18 and 3.21 patches Multiple patches have been released throughout 2023 for current LTS supported versions: * In April, [3.18.4 and 3.21.1](/blog/2023/cfengine-3-18-5-and-3-21-2-released/) were released. * In June, [3.18.5 and 3.21.2](/blog/2023/cfengine-3-18-5-and-3-21-2-released/) were released. * In October, [3.18.6 and 3.21.3](/blog/2023/cfengine-3-18-6-and-3-21-3-released/) were released. We encourage all of our users to stay up to date with the latest patch releases, to benefit from bug fixes, security patches, and dependency updates. Please note that 3.18 LTS 'end of support' is slated for next August 2023. If you need assistance upgrading to one of our supported versions, please [contact us](/contact/). For more information on previous and future 'end of support' dates, please visit [here](/extended-support/). ## Blog post: Migrating to CFEngine Build In 2021, we introduced CFEngine Build to help users consume reusable policy, making it easier to deploy and manage policy sets. As many users continue to explore how to leverage CFEngine Build within their environments, Nick Anderson wrote an extensive [blog post](/blog/2023/migrating-to-cfbs/) on how one can go about converting a traditionally managed policy set, that poses a number of challenges, into one that is managed and streamlined by the CFEngine Build. ## White paper: Efficient synchronization of tabular data Mission Portal is a system that collects invaluable information about managed hosts, by mirroring their states onto a centralized hub. These states are stored as tabular data structures and are synchronized using a log-based synchronization technique. The existing implementation of the reporting system is not optimized for minimal bandwidth consumption, which is a critical concern of CFEngine Enterprise customers. In response, [this paper](/white-papers/efficient-synchronization/) summarizes an enhanced log-based synchronization technique introduced by Lars Erik Wik, designed to address this challenge. {{< pdfpreview "/white-papers/Efficient_Synchronization_of_Tabular_Data_preview.png" "Preview of the first page of the white paper fading into the background." "/white-papers/efficient-synchronization/">}} ## What's next? In summer of 2024 we will launch CFEngine 3.24 LTS, bringing the functionality we've been working on for the last 1-1.5 years into an LTS supported release, including: * Groups in Mission Portal - save host filters with a name for easier reuse and overview. * Group data - assign CMDB data to groups of hosts, setting variables, classes and inventory attributes for many hosts at a time. * Improved host filters with class rules and host lists by IP / ID / MAC / hostname. * Various improvements to the UI of new and popular features: Build in Mission Portal, compliance reports, inventory reports. * New policy functions. * Several security improvements, especially around LDAP, passwords, and authentication. ## Fun facts * The largest CFEngine implementation known to date is over 3.5 million servers. * This year, [CFEngine turned 30](https://www.linkedin.com/search/results/content/?keywords=%23CFEngineTurns30), and we also held our 30th episode of ["The agent is in" webinars](/webinars/). Whether you've been a user since the 90's or 2000's or you just found CFEngine last week, we sincerely thank you from the bottom of our hearts for your contributions. We wish you and your families a wonderful new year and cheers to another 30 years of CFEngine. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 32 - Fireside chat w/ Mike Weilgart" url: /blog/2023/the-agent-is-in-episode-32/index.md date: "2023-12-28" -------------------------------------------------------------------------------- Curious about how users feel about using CFEngine? Cody and Nick are joined by Mike Weilgart, a long time CFEngine user about his history and how CFEngine compares to other tools he has been using recently. ## Video {#video} The video recording is available on YouTube: {{< youtube WGdLnCjb5pE >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Mike](https://www.linkedin.com/in/mike-weilgart-4a858719a/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.23 released - Anniversary" url: /blog/2023/cfengine-3-23-released-anniversary/index.md date: "2023-12-06" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.23.0\! This is a non-LTS (non-supported) release, where we introduce new features for users to test and give feedback on, allowing us to polish before the next LTS. (CFEngine 3.24 LTS is scheduled to release summer 2024). The codename for this release is _anniversary_, as this year is CFEngine's 30th anniversary. [CFEngine was initially released in 1993](https://mark-burgess-oslo-mb.medium.com/cfengines-star-trek-and-ai-origins-e99096fe845b), and to _mark_ this special occasion we've created a limited edition anniversary coin: {{< img "images/cfengine-coin.jpeg" "Photograph of the CFEngine 30 year anniversary coin.">}} ## What's new ### Include / exclude hosts by hostname, IP address, subnet, MAC address Within Mission Portal's inventory reports you can easily control what pieces of information to show (which columns), as well as which hosts (rows) to filter out. In the previous releases, we've made several improvements to the UI for inventory reports, making them more intuitive to use and edit, as well as giving you more flexibility with the dynamic rules. For 3.23 we are continuing to improve the inventory reports and their host filters, by adding more ways to specify lists of hosts to include and exclude. {{< img "images/host-include-exclude.png" "Host include / exclude dialog showing lists of hosts can be included and excluded based on hostname, IP, MAC address and CFEngine ID">}} By specifying lists of hosts by their IP, MAC, or hostname, they will stay in the list, even if their ID changes (for example after reinstalling CFEngine on a host, rotating a host's key, or replacing one host with another). When combining rules with lists of hosts to include and exclude, exclusion will always take precedence, so the hosts you put into the exclusion list will never show up in the results. ### Groups in Mission Portal The groups functionality introduced in [3.22](/blog/2023/cfengine-3-22-released-coordination/) also benefits from the host filter updates mentioned above. You now have a lot of flexibility when deciding what hosts should be in a group (editing its filter): {{< img "images/group-critical-systems.png" "Screenshot from Mission Portal which shows an example group called Critical systems, with a combination of rules and hosts included / excluded.">}} In the example above, we've included all RHEL 7, 8, and 9 hosts with a regular expression, as well as included 1 specific host by IP address and all hosts in a subnet. We've also excluded one specific host which we didn't want in the group (by its IP address). Groups can now be static (such as lists of CFEngine IDs and MAC addresses), or much more dynamic (conditions based on reported inventory data). Or a mix of these, or something in between (IP addresses usually are quite static, but can also be changed around). We will continue to improve and polish the groups and filter functionality before it lands in the next LTS release, CFEngine 3.24, scheduled for summer 2024. ### Policy language function: version_compare() In CFEngine policy language, we have many convenience functions and macros to make the policy behave differently for various versions of CFEngine. However, these always compare the _CFEngine version number_, and cannot be used to customize the behavior based on another version number (such as the version of an installed package). The new [`version_compare()`](https://docs.cfengine.com/docs/3.23/reference-functions-version_compare.html) function makes it easier to work with version numbers in general: ```cfengine3 bundle agent __main__ { vars: "version" string => "1.2.3"; reports: "That's newer!" if => version_compare("$(version)", ">", "1.0.0"); } ``` ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.23.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.23/release-notes-whatsnew-changelog-core.html) - [3.23.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.23/release-notes-whatsnew-changelog-enterprise.html) - [3.23.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.23/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to version [3.22](/blog/2023/cfengine-3-22-released-coordination/), these dependencies have been updated: | CFEngine version | 3.22.0 | **3.23.0** | | :-------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.57 | **2.4.58** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.41.0 | **2.42.1** | | [libcurl](https://curl.haxx.se/download.html) | 8.1.2 | **8.4.0** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.11.4 | **2.11.5** | | [LMDB](https://www.symas.com/lmdb) | 0.9.30 | **0.9.31** | | [OpenSSL](https://openssl.org/) | 3.1.1 | **3.1.4** | | [PHP](https://php.net/) | 8.2.7 | **8.2.12** | | [PostgreSQL](https://www.postgresql.org/) | 15.3 | **15.5** | | [zlib](https://www.zlib.net/) | 1.2.13 | **1.3** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, you have to specify this release explicitly: ```command cf-remote --version 3.23.0 download ``` (`cf-remote` defaults to the latest LTS release, which is currently 3.21.3). ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 31 - Sneak peek of CFEngine 3.23" url: /blog/2023/the-agent-is-in-episode-31/index.md date: "2023-11-30" -------------------------------------------------------------------------------- Join the team for a sneak peek of what's coming in 3.23.0. Herman joins Cody, Craig and Nick to discuss what's new in the upcoming release of CFEngine 3.23.0. We look at improvements to Groups in Mission Portal with easier ways to specify specific hosts that should or should not be part of the group based on reported attributes. This new functionality makes it much easier to affect change across a set of hosts without touching policy. ## Video {#video} The video recording is available on YouTube: {{< youtube QyGFbGpP4zM >}} ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) ## Post show discussion {#post-show-discussion} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ### How can I populate a variable from a script without executing the script many times? {#how-can-i-populate-a-variable-from-a-script-without-executing-the-script-many-times} Someone asked how they could minimize the number of times CFEngine runs a script to populate a variable: > I want to populate a variable with the result of a python script, currently I am using `execresult()` but the script is executed many times when the policy runs. How can I minimize the number of times the script is executed and populate my variable? The script is executed many times because of pre-evaluation[^fn:1]. CFEngine eagerly resolves `vars` and `classes` promises in `common` bundles and `vars` promises in `agent` bundles. There are various ways in which you could minimize the executions of the script. You could move the script to a `commands` promise and write its output to a file which can be read into a variable. With this approach, unless specifically guarded against, the value of the variable can change between the three passes of policy evaluation that occur on each agent run. This would cause any policy which uses this variable to execute once for each unique value of the variable. `readfile()` [^fn:2], `readdata()` [^fn:3], `readcsv()` [^fn:4], `readjson()` [^fn:5], `readyaml()` [^fn:6], `readenvfile()` [^fn:7] , and `read_module_protocol()` [^fn:8] are all good candidates depending on the output. You could adjust the script to run as a `commands` promise and use the module protocol to define variables and classes as a byproduct of its execution. With this methodology there won't be a file whose state must be managed but the variable will also not be available very early during policy execution. ### Setting up a new offline Enterprise Hub, how do I install cfbs? {#setting-up-a-new-offline-enterprise-hub-how-do-i-install-cfbs} Someone asked how they could install `cfbs` on a new Enterprise Hub that is not connected to the internet. > I am setting up a new hub and it is not connected to the internet. How can I install `cfbs`? Usually you install `cfbs` via Python's package index (`pip`). But for the case of an Enterprise Hub you don't need to. `cfbs` is vendored with the hub package. So, just install the hub package and you will have access to `cfbs`. ### How can I manage files with the immutable bit? {#how-can-i-manage-files-with-the-immutable-bit} Someone asked how they could manage files that have the immutable bit set. > In our environment sometimes administrators set the immutable bit on configuration files. CFEngine fails to modify these files. How can I manage these files? Unfortunately there isn't _currently_ native support in `files` promises for managing immutable files. Currently you must use a `commands` promise or function like `execresult()` or `execresult_as_data()` to unset the immutable bit. [^fn:1]: Documentation for pre-evaluation: [^fn:2]: Documentation for `readfile()`: [^fn:3]: Documentation for `readdata()`: [^fn:4]: Documentation for `readcsv()`: [^fn:5]: Documentation for `readjson()`: [^fn:6]: Documentation for `readyaml()`: [^fn:7]: Documentation for `readenvfile()`: [^fn:8]: Documentation for `read_module_protocol()`: -------------------------------------------------------------------------------- title: "Change in behavior: ignore_interfaces.rx prefers a new location" url: /blog/2023/change-in-behavior-ignore_interfaces.rx-prefers-new-location/index.md date: "2023-11-16" -------------------------------------------------------------------------------- `ignore_interfaces.rx` can be populated with regular expressions that match network interface names. When an interface matches CFEngine will ignore the interface. In the upcoming release of 3.23.0, and in the future release of 3.21.4 there is a change in behavior with respect to the preferred location of `ignore_interfaces.rx` from `$(sys.inputdir)` (typically `/var/cfengine/inputs`) to `$(sys.workdir)` (typically `/var/cfengine`). The change from `$(sys.inputdir)` to `$(sys.workdir)` makes it easier to ignore different interfaces on different hosts. With the change, if `ignore_interfaces.rx` is found in `$(sys.inputdir)`: - `cf-agent` will emit warnings ```text warning: Found interface exception file ignore_interfaces.rx in /var/cfengine/inputs but it should be in /var/cfengine. Please consider moving it to the appropriate location. ``` - Recommendation policy will emit reports about the state of `ignore_interfaces.rx` ```text R: NOTICE: 'ignore_interfaces.rx' is present in '$(sys.inputdir)' ('/var/cfengine/inputs/ignore_interfaces.rx'). We recommend that it be removed and migrated to '$(sys.workdir)' ('/var/cfengine/ignore_interfaces.rx') R: NOTICE: 'ignore_interfaces.rx' in '$(sys.workdir)' and '$(sys.inputdir)' but not identical. We recommend verifying the desired content of '$(sys.workdir)/ignore_interfaces.rx', correcting it if necessary and removing '$(sys.inputdir)/ignore_interfaces.rx' R: NOTICE: 'ignore_interfaces.rx' identical in '$(sys.workdir)' and '$(sys.inputdir)'. We recommend removing '$(sys.inputdir)/ignore_interfaces.rx' ``` - Recommendation policy will emit warnings about the presence of the file in inputs ```text warning: Should delete file '/var/cfengine/inputs/ignore_interfaces.rx', but only warning promised ``` ## What to do {#what-to-do} Adjust the policy so that `ignore_interfaces.rx` is not at the root of your policy set when running CFEngine versions 3.21.4 or newer, and 3.23.0 or newer. Setting the class `default:mpf_auto_migrate_ignore_interfaces_rx_to_workdir` will cause the recommendation policy to promise that `$(sys.workdir)/ignore_interfaces.rx` is a copy of `$(sys.inputdir)/ignore_interfaces.rx` which will suppress the warning from `cf-agent` about the file being found in inputs, but reports from the recommendation policy will continue to be active unless the recommendation policy is disabled by defining the class `default:cfengine_recommendations_disabled`. If you have questions or need help, reach out on the mailing list[^fn:1], or GitHub discussions[^fn:2]. If you have a support contract, feel free to open a ticket in our support system[^fn:3]. [^fn:1]: CFEngine Help mailing list [^fn:2]: CFEngine GitHub discussions [^fn:3]: Northern.tech Support -------------------------------------------------------------------------------- title: "Change in behavior: self upgrade now defaults to hub binary version" url: /blog/2023/change-in-behavior-self-upgrade-targets-hub-binary-version-by-default/index.md date: "2023-11-15" -------------------------------------------------------------------------------- In the upcoming release of 3.23.0, there is a change in behavior with respect to the self upgrade policy. Beginning with 3.23.0 the self upgrade policy will default to the binary version that is running on the hub instead of the version of the policy framework that is executing. When upgrading CFEngine[^fn:1] there are three major steps: - Upgrade the Masterfiles Policy Framework (MPF) - Upgrade the hub binaries - Upgrade the client binaries Generally it's desirable that the MPF version is equal to or greater than the hub binary version and the hub binary version is equal to or greater than the client binary version. This way the policy has necessary knowledge in place prior to a binary upgrade where behavior may change. Previously the MPF defaulted the self upgrade desired binary version to the version of the policy. If hosts are configured to self update when a new policy was released this could result in clients upgrading to a newer binary version before the hubs binary version is upgraded. While this is not typically problematic, we thought that defaulting to the hubs current binary version was better aligned with the recommended configuration. As before you can specify the desired binary version by defining `default:def.cfengine_software_pkg_version` [^fn:2] via Augments. For example: ```json { "classes": { "default:trigger_upgrade": { "class_expressions": [ "!(am_policy_hub|policy_server).!cfengine_3_21_3::" ], "comment": "We want clients to self upgrade their binary version if they aren't running the desired version." } }, "variables": { "default:def.cfengine_software_pkg_version": { "value": "3.21.3", "comment": "When self upgrading, this is the binary version we want to be installed." } } } ``` To facilitate this change as part of the update policy entry (`update.cf`), the hub now maintains `$(sys.statedir)/cf_version.txt` containing its current binary version in the form `$(sys.cf_version_major).$(sys.cf_version_minor).$(sys.cf_version_patch)-$(sys.cf_version_release)`. It's accessible to hosts described by `def.acl` and has a _shortcut_ `hub-cf_version`. Clients maintain a copy of the hubs binary version as `$(sys.statedir)/hub_cf_version.txt` and the self upgrade policy entry (`standalone_self_upgrade.cf`) uses the content of this file for determining the desired binary version. In summary, beginning with 3.23.0 if the self upgrade policy is active clients will default to the binary version that the hub is currently running. When the hubs binary version changes, clients will automatically start targeting the new binary version by default. [^fn:1]: Upgrade documentation [^fn:2]: Configuring self upgrade: -------------------------------------------------------------------------------- title: "Efficient synchronization of tabular data white paper" url: /blog/2023/efficient-synchronization-white-paper/index.md date: "2023-11-14" -------------------------------------------------------------------------------- To manage large infrastructures, efficient solutions for both making changes and observing the current state are necessary. As most information (inventory) about hosts is quite predictable and static, there are many opportunities for optimizations in terms of compression and avoiding re-transmission of the same data. In the CFEngine team, we are improving our reporting systems with a focus on correctness and low bandwidth consumption. This will benefit many users, both large data centers where bandwidth (networking equipment) is costly, as well as small IoT devices with limited connectivity. Inspired by git, we are implementing commits of reporting data, with table-based diffs, and compression of multiple changes, akin to squashing git commits. Read [our latest white paper](/white-papers/efficient-synchronization/), explaining the approach: {{< pdfpreview "/white-papers/Efficient_Synchronization_of_Tabular_Data_preview.png" "Preview of the first page of the white paper fading into the background." "/white-papers/efficient-synchronization/">}} -------------------------------------------------------------------------------- title: "CVE-2023-45684 - Mission Portal SQL injection vulnerability" url: /blog/2023/cve-2023-45684/index.md date: "2023-11-13" -------------------------------------------------------------------------------- We want to bring to your attention a critical security matter recently identified in CFEngine Enterprise version 3.6.0 and subsequent releases. This vulnerability pertains to a [A03:2021 - Injection](https://owasp.org/Top10/A03_2021-Injection/) flaw within the CFEngine Enterprise web UI, Mission Portal, which can lead to unauthorized access to the underlying database. The CVE identifier `CVE-2023-45684` has been assigned to this issue. At present, there is no evidence to suggest that this vulnerability has been exploited or that it was known beyond the CFEngine development team and the customer who brought it to our attention. ## Explanation Our vigilant CFEngine customers, Chad DeGuira and Jim Trater at the Oak Ridge National Laboratory, recently uncovered a vulnerability while scanning their attack surface. During their scan, they identified an unusual delay in the response time from the CFEngine Mission Portal login page. This delay raised concerns that an attacker might have the potential to manipulate the application's behavior and gain direct access to the underlying database. ## Impact In response to this discovery, we conducted an internal penetration test to assess the severity of the vulnerability. Our testing confirmed that the Mission Portal login page was indeed susceptible to SQL injection, specifically of the time-based variety. This means that you cannot send queries to dump the database tables directly (with the common `SELECT FROM ` statement). This is due to the fact that the queried results are not relayed in the server's response to the API request. However, since the input is not properly sanitized, you can still craft special queries, testing for logical conditions `AND`'ed with a call to the `PG_SLEEP ( interval )`, a function designed to delay the execution. By observing the time it takes for the server to respond, we can determine whether the first condition was true. This way we can slowly deduct the contents of the database, usually done one character at a time, making it a really slow process. By utilizing this vulnerability we were able to dump the contents of the entire underlying database, including tables storing confidential information such as access tokens and salted password hashes. Thus, we consider this vulnerability a significant concern that warrants immediate attention and remediation. An overall severity score of 7.5 (high severity) has been calculated using the following severity vector (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). ## Remediation To mitigate this risk, we strongly recommend installing one of the latest versions of the CFEngine hub packages as listed on our [download pages](https://cfengine.com/downloads/cfengine-enterprise/). The recently released CFEngine Enterprise versions 3.18.6 and 3.21.3 include the necessary fixes to address this vulnerability. It's important to note that this issue exclusively affects the CFEngine hub. Thus, by installing the updated hub package, you can effectively remediate the issue. For more information on how to upgrade an existing CFEngine installation, please see [our documentation](https://docs.cfengine.com/docs/3.21/getting-started-installation-upgrading.html). We appreciate your continued trust in CFEngine, and we are committed to ensuring the security and reliability of our software. If you have any questions or require further assistance, please don't hesitate to reach out to our [support team](https://support.northern.tech/). -------------------------------------------------------------------------------- title: "Migrating from Travis to Github Actions" url: /blog/2023/travis-to-github-actions/index.md date: "2023-10-30" -------------------------------------------------------------------------------- For CFEngine we manage several public and private repositories of code in GitHub for our Open Source and Enterprise products. In order to ensure quality we run many checks on the code both with nightly builds as well as on each pull request. We use a Jenkins server for nightlies which also includes more extensive deployment tests on all of the platforms we support. Previously we had used Travis for many of these checks but that system started to show its age and limitations. In this blog post I will discuss how we migrated part of our continuous integration environment from Travis to GitHub Actions. Let's get on with the major components of continuous integration in our case. For our open source CFEngine agent we have three repositories involved: [core](https://github.com/cfengine/core), [buildscripts](https://github.com/cfengine/buildscripts) and [masterfiles](https://github.com/cfengine/masterfiles). For Enterprise we have several additional repositories for the enterprise agent and the enterprise hub. Most of our build and test logic is in buildscripts. That is what we use to drive our Jenkins-based CI as well. # Secrets In order to enable GitHub Actions to build our Enterprise product we must give it read access to private repos. This is done through [secrets management](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) provided by GitHub. We also need to provide our builds with a few other secrets. All of the secrets need to be shared with both the organization as well as GitHub's [dependabot](https://docs.github.com/en/code-security/dependabot/working-with-dependabot) so that when dependabot creates pull requests for dependency upgrades, the github actions can access those secrets. # Public versus private One snag we ran into here though is that we want our public repos to have read access to build our Enterprise code. GitHub secrets are not available to pull requests, even if the pull request is submitted by a member of the organization. The workaround in this case is to push to the organization repo instead of our forks. This is easily done with the `gh` command which I have been fond of using lately so that I can avoid my browser a bit longer. Assuming I have two remotes, upstream and origin, and origin is my fork: ```sh $ git remote -v origin git@github.com:craigcomstock/buildscripts (fetch) origin git@github.com:craigcomstock/buildscripts (push) upstream git@github.com:cfengine/buildscripts (fetch) upstream git@github.com:cfengine/buildscripts (push) ``` gh will ask where I want to push my branch: ```sh $ gh pr create ? Where should we push the 'test' branch? [Use arrows to move, type to filter] > cfengine/buildscripts craigcomstock/buildscripts Skip pushing the branch Cancel ``` Choosing `cfengine` instead of `craigcomstock` will allow the github actions to receive the required secrets. # YAML explosion and contraction (DRY) Initially as we migrated we created many yaml files for various jobs in various repos. Often we would copy these yml files from one repo to another and make small changes to suit that different repo. This became a maintenance problem as we typically maintain a master and two LTS (long term support) branches for several repos which would essentially have identical yaml files for various jobs: build, package, test. To solve this issue we moved the yaml files into buildscripts and referenced them from the various other repos in a master ci.yml file which simply defined which jobs to run, not the content of the jobs. For example, in `buildscripts` we have `.github/workflows/deployment-tests.yml` which we want to use in several repositories. In each repository we have a main `.github/workflows/ci.yml` which is our main entry point that builds on `pull_request` activity. ```yaml on: pull_request ``` The `deployment-tests.yml` activates on `workflow_call` to enable it to be called from other workflows. ```yaml on: workflow_call: secrets: GH_SECRET_ONE: required: true jobs: deployment_tests: runs-on: ubuntu-latest steps: ... etc ... ``` And then from each repo we simply refer to the yml file: ```yaml jobs: deployment_tests: secrets: inherit uses: cfengine/buildscripts/.github/workflows/deployment-tests.yml@master ``` Generally, it is best practice to use a commit sha instead of `@master` but in this case we have tight controls over who can commit changes to `buildscripts` so felt it was worth staying with the latest as well as minimizing the work to update the commit shas whenever we changed the yaml in `buildscripts`. # Containers (docker) In the process we also found that using containers could be very helpful in making the jobs easy to run anywhere, both in GitHub as well as locally to debug without having to wait or pay for the infrastructure of GitHub. We already had a head start using Docker so decided to go with that. It is well supported on MacOS and Linux. We would like to gradually migrate to something a bit more lightweight such as [buildah](https://buildah.io/) in the future. As our enterprise product is a collection of several independent processes it was a bit non-standard how we had to implement this, starting systemd as the main process in the container and then handling other parts of the process through calls into the container instead of just a one-shot Dockerfile. [Dockerfile-cfengine-deployment-tests](https://github.com/cfengine/buildscripts/blob/master/ci/Dockerfile-cfengine-deployment-tests) ```Dockerfile FROM ubuntu:20.04 RUN apt-get update -y && apt-get install -y systemd sudo CMD [ "/lib/systemd/systemd" ] ``` and then later in a separate [shell script](https://github.com/cfengine/buildscripts/blob/master/ci/docker-deployment-tests.sh) we run various steps to run the test: ```sh docker exec -i $name bash -c 'cd /data; ./buildscripts/ci/deployment-tests.sh' ``` Not exactly best practices but it does work and it also enables us to run this `deployment-tests.sh` script in a plain host/virtual machine if we want to dig into things step by step for debugging. # Together Since we have a multi-repo setup for our software we needed a way to coordinate builds when a change involved PRs for multiple repos. Previously we had a way of looking in the pull request description for mentions (URLs) of PRs in other repos. The CI code would then checkout that PR instead of the branch we were on, thereby including all the changes needed. # Save time and money with caching As we worked through many builds during development of our GitHub Actions we burned through a LOT of minutes. In order to optimize this we needed to use any caching available to us. # GitHub cache We were surprised to see that github caching was specific to each pull request. We had hoped that we could leverage caches of builds in various repos from other repos and even from the same repo. e.g. If we made a change which involved 4 repos we wouldn't be building the same code 4 times but rather share the results. # Custom cache We include several third party dependencies as part of our package. Since these don't change all that often and can sometimes take quite a while to build we already had a cache of these built dependencies. Re-using this with GitHub proved to save a lot of build minutes. # Test-only changes reuse old cache When you are iterating on a pull request you might need to add or adjust testing code often based on the results in GitHub actions. Since we are saving minutes by caching our dependencies and packages we don't want to invalidate those caches with test expectation changes or additional tests. For this we implemented a package_sha which computes a sha from code-only files ```sh SHA=$(find "${REPO}" -type f -and \( -path "${REPO}/api/*" -or -path "${REPO}/db/*" -or -path "${REPO}/lib*" -or -path "${REPO}/report-collect-plugin/*" \) -print0 | xargs -0 sha1sum | awk '{print $1}' | sha1sum | cut -c -8) ``` Each repo has its own formula for what is code and what is not. We combine all of those shas into one which we use to decide whether to rebuild the package or not. All in all we have re-implemented most of the CI functionality we have in Travis and are generally happy with the result. # Continuous improvement As with many projects we weren't able to include every feature that we wanted. A few tasks stand out as things we will likely address in the near future: - shared caching between projects - improved logging to ease in debugging failures - reduced execution minutes with improved Dockerfiles and saving images for re-use Hope this summary helps you in your continuous integration work. If you have any thoughts or tips send them our way at contact@northern.tech. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 30 - Profiling CFEngine policy" url: /blog/2023/the-agent-is-in-episode-30/index.md date: "2023-10-19" -------------------------------------------------------------------------------- Imagine having the power to identify the exact lines of your CFEngine policy that are slowing down your executions. In this episode, we'll guide you through the art of profiling CFEngine policy for improved performance. In Episode 30 of "The agent is in," Nick and team dive into the topic of profiling CFEngine policy. We explore tools and techniques to identify performance bottlenecks and optimize CFEngine deployments. The episode covers the following main points: 1. Introduction to `cf-profile`: A useful tool for identifying long-running bundles in CFEngine policy. `cf-profile` analyzes the verbose log messages and provides a summary of the most time-consuming bundles. 2. Using `cf-profile`: Demonstration of how to install and use `cf-profile` interactively with `cf-agent`. By running `cf-profile` with the execution tree and summary options, users can get a high-level overview of bundle execution time and identify areas that require further investigation. 3. Fine-Grained Analysis: To dig deeper into individual promises within a bundle using the `timestamp` option of `cf-agent`. This allows users to track the execution time of each promise and pinpoint any performance issues. 4. Profiling in CFEngine Enterprise: For CFEngine Enterprise users, the benchmarks table provides execution time and standard deviation metrics. Nick explains how to access this table through the Mission portal and highlights the usefulness of attaching an `action` body with the `measurement_class` to measure specific promises. 5. Demonstrating Custom Measurement: Nick shows an example of attaching an `action` body using the `measurement_class` attribute on a command's promise and demonstrates how the measurement appears in the benchmarks table. The episode caters to both new and experienced CFEngine users, providing tips and tools for analyzing and optimizing CFEngine policies. ## Video {#video} The video recording is available on YouTube: {{< youtube kNlsLZO4n6k >}} ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/) or [Nick](https://www.linkedin.com/in/hithisisnick/) - `cf-profile` - [All Episodes](/webinars) ## Notes {#notes} ### `cf-profile` {#cf-profile} Provides high level summary of execution time in policy surfacing the bundles in which the most time was spent. Find it in `contrib/cf-profile` of the core repository. ```sh curl -s https://raw.githubusercontent.com/cfengine/core/master/contrib/cf-profile/cf-profile.pl --output /var/cfengine/bin/cf-profile chmod 755 /var/cfengine/bin/cf-profile ```
Code Snippet 1: Installing cf-profile
```sh cf-agent -Kv | /var/cfengine/bin/cf-profile -t -s ```
Code Snippet 2: Running cf-profile
Note: The script requires `perl`, and the `POSIX`, `Time::HiRes`, `Data::Dumper`, and `Getopt::Std` modules. ### `--timestamp` options adds a timestamp to CFEngine logs {#timestamp-options-adds-a-timestamp-to-cfengine-logs} Useful for digging down into the execution time of individual promises. ```sh cf-agent -Kv --timestamp ``` Note: it does not prefix lines emitted by `reports` promises with a timestamp. ### CFEngine Enterprise benchmarks table {#cfengine-enterprise-benchmarks-table} Provides last, average, and standard deviation for execution times of various things, including the Agent execution time per policy entry as well as the promise execution time for any promise using an `action` _body_ with the `measurement_class` attribute defined. ```cfengine3 body action measure_promise_time(identifier) # @brief Performance will be measured and recorded under identifier # # @param identifier Measurement name. { measurement_class => "$(identifier)"; } ```
Code Snippet 3: Snippet from standard library (lib/common.cf)
```cfengine3 commands: "sleep $(i)" inform => "false", contain => in_shell, handle => "bundle_agent_two_sleep", action => measure_promise_time("measure_my_command_that_is_slow"); ```
Code Snippet 4: Example Snippet
## Post show discussion {#post-show-discussion} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ### Config Management Camp CFP is open {#config-management-camp-cfp-is-open} - Don't forget to submit your talks! - - If you can help sponsor this free event, please do. - ### Where is the CFEngine Merch Shop? {#where-is-the-cfengine-merch-shop} Need some CFEngine swag? [Contact us](https://cfengine.com/contact/). ### History of CFEngine & differences in level of publicity with other tools {#history-of-cfengine-and-differences-in-level-of-publicity-with-other-tools} We discussed the history of CFEngine, covering the software, the company (now [Northern.tech](https://northern.tech)), user personalities (both individual and organization), as well as differences in company strategy in comparison to other products. -------------------------------------------------------------------------------- title: "libntech 1.0: now available to more projects" url: /blog/2023/libntech-1-0-released-with-new-license/index.md date: "2023-10-12" -------------------------------------------------------------------------------- The license of our in-house C utility and compatibility library *libntech* [was recently changed](https://github.com/NorthernTechHQ/libntech/pull/188) from GPLv3 to Apache License Version 2.0 which makes the library suitable for more projects thanks to the more permissive license. While GPLv3 practically required any project using *libntech* to be licensed under GPLv3 as well, the Apache License v2.0 allows any open source as well as proprietary software to utilize our utility library, keeping the copyright attributions. With this important change we also decided to [release the current version of *libntech* as version 1.0](https://github.com/NorthernTechHQ/libntech/releases). Maybe a big jump from 0.1 which could suggest an early development stage, but we believe that the code is very mature and want to reflect that in the version number. ## (Hi)Story CFEngine, being written in C and being a portable/multi-platform project started in 1993, has required implementations of many things that are usually provided by standard libraries in other languages and environments. Be it data structures like linked lists, sequences (automatically expanding arrays), hash maps and many others or utility functions for various string operations, locking and threading primitives, etc. Plus implementations of commonly, but not strictly generally available things like `getline()`, `getopt()`, `poll()`,... Around 2019 it was quite obvious that this code could be usable for [other projects](https://github.com/mendersoftware/mender-binary-delta) in our company and also for anybody else in need of a simple, fast and small C utility library. Building CFEngine and extracting the library from it was difficult, challenging and cumbersome and thus *libntech* as a separate project and repository [was created](https://github.com/NorthernTechHQ/libntech/commit/4d3bdc68931d404e582dc1acdcf9d04eb782c2c4) as a fork of the code from the [CFEngine core](https://github.com/cfengine/core) repository. And that\'s how it inherited CFEngine\'s license -- GPLv3. Or, to be more precise, the dual license GPLv3 + COSL (our in-house Commercial Open Source License). This posed no licensing problems for us because both use cases for the library were compatible. The situation changed this year, in 2023, when we [started rewriting](https://github.com/mendersoftware/mender/tree/feature-c%2B%2B-client) parts of our other product -- Mender -- in C++. With Mender using the Apache License, it was impossible to use *libntech* unless we would choose GPLv3 for at least some of its parts. Using different and incompatible FOSS licenses for parts of one project didn\'t sound like a good idea and we also quickly realized that other potential users of our utility library will likely have the same problem. Thus, since we had the option thanks to being copyright owners of the *libntech* code and [having the right to relicense it](https://github.com/cfengine/core/blob/master/AUTHORS#L93), we decided to change the license to Apache License, Version 2.0, the same license that the open-source parts of Mender use. ## What it has to offer *libntech* has two major parts -- [*libutils*](https://github.com/NorthernTechHQ/libntech/tree/1.0.0/libutils) and [*libcompat*](https://github.com/NorthernTechHQ/libntech/tree/1.0.0/libcompat) -- that contain the implementations of utility functions and data structures, and compatibility functions, respectively. *libcompat* is much smaller and provides implementations of around 50 functions. Much of its code was taken from various places and is under various authors\' copyrights and custom licenses. All of it [now being compatible](https://github.com/NorthernTechHQ/libntech/pull/186/commits/00140f1c1a5fea749c441ac410fafcb23da9480d) with Apache License v2.0. *libutils* is a much bigger part of the codebase and provides simple, fast and small implementations of the following data structures and some more: * [Sequences](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/sequence.h) (aka expandable arrays) * [Hash maps](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/map.h) and [sets](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/set.h) * [Expandable buffers](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/buffer.h) * [Double-linked lists](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/list.h) * [Queues (FIFO)](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/queue.h) and [stacks (LIFO)](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/stack.h) * [Red-black trees](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/rb-tree.h) * [Ring buffers](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/ring_buffer.h) * Thread-safe [queues](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/threaded_queue.h), [deques](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/threaded_deque.h) and [stacks](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/threaded_stack.h) It also provides a simple logging framework and memory allocation functions that check for OOM and kill the process accordingly. Then many helper and utility functions for working with strings, files, directories, file system paths, regular expressions, IP addresses and others. Last but not least, actually, some would say that this is the most valuable part, it also provides simple implementations for [*JSON*](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/json.h) and [*CSV*](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/csv_parser.h) file formats, for [rendering *Mustache templates*](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/mustache.h) and a custom very straightforward [format](https://github.com/NorthernTechHQ/libntech/blob/1.0.0/libutils/string_sequence.h#L28-L89) for storing and loading sequences of strings. There are still a couple places where the origin of the code is more obvious than what would be ideal (e.g. the `ManPageWrite()` function writes a hard-coded CFEngine header and footer and the `known_dirs.h` API is CFEngine specific), but we are [actively working](https://github.com/NorthernTechHQ/libntech/pull/194) on either (re)moving the code or, wherever possible, making it more generic. In the meantime, these few functions should either be avoided or patched with project-specific snippets. All of the code is truly open-source, not only from the licensing perspective, but also by being very easy to read and understand and thus to modify and tailor for the needs of the project it\'s being used by. Sometimes it even prefers simplicity and readability over optimization because we believe easier to read and thus easier to debug code is more useful than super-optimized code nobody understands and can work with. We'd like to thank everyone[^fn:1] in our community who has contributed to the CFEngine codebase (and libntech) over the years, and made it what it is today. [^fn:1]: An overview of all contributors can be found by viewing the [AUTHORS](https://github.com/cfengine/core/blob/76a7d0d01c63bcdf2a204c71a934f09d234b6588/AUTHORS) file and contributor graphs in individual repos - [libntech](https://github.com/NorthernTechHQ/libntech/graphs/contributors), [core](https://github.com/cfengine/core/graphs/contributors), [masterfiles](https://github.com/cfengine/masterfiles/graphs/contributors), [buildscripts](https://github.com/cfengine/buildscripts/graphs/contributors), [cf-remote](https://github.com/cfengine/cf-remote/graphs/contributors), [cfbs](https://github.com/cfengine/cfbs/graphs/contributors), [build-index](https://github.com/cfengine/build-index/graphs/contributors). -------------------------------------------------------------------------------- title: "CFEngine 3.18.6 and 3.21.3 released" url: /blog/2023/cfengine-3-18-6-and-3-21-3-released/index.md date: "2023-10-06" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.18.6 and 3.21.3\! These patch releases contain bug fixes and dependency updates. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.18.6 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.6 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.6 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.3 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.3 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.3 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.21.3 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.18.6 list ``` ```command cf-remote --version 3.21.3 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 29 - Basic Docker inventory with CFEngine" url: /blog/2023/the-agent-is-in-episode-29/index.md date: "2023-09-28" -------------------------------------------------------------------------------- Ever been curious about Docker details or the cruft that has built up and could be cleared out? Craig, Cody, and Nick chat about some of the work Craig has been doing recently, using Docker in development and CI. Craig shows how to develop policy to inventory various Docker details like image names, counts of dangling images, and reclaimable disk space. ## Video {#video} The video recording is available on YouTube: {{< youtube Iyc2Sg5FLpQ >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). - [00:20](https://youtu.be/Iyc2Sg5FLpQ?t=20) - What Craig has been doing with Docker lately - [01:48](https://youtu.be/Iyc2Sg5FLpQ?t=108) - Interesting information about Docker - [07:21](https://youtu.be/Iyc2Sg5FLpQ?t=441) - Implementing Docker Inventory - [23:58](https://youtu.be/Iyc2Sg5FLpQ?t=1438) - Inventorying associative arrays instead of lists to avoid truncation ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - Post show discussion - CFEngine layer in Spacemacs () - The agent is in [Episode 12: Spacemacs for CFEngine](https://cfengine.com/blog/2022/the-agent-is-in-episode-12-spacemacs-for-cfengine/) - Khoj, a personal AI assistant (use a local Large Language Model to chat with your notes) () -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 28 - Automating CFEngine policy testing" url: /blog/2023/the-agent-is-in-episode-28/index.md date: "2023-08-31" -------------------------------------------------------------------------------- Have you been interested in automating the testing of your CFEngine policy? Cody, Craig and Nick follow up on the Policy Examples episode and dive a bit deeper into testing. Nick walks through some policy and related tests that leverage `lib/testing.cf` from the Masterfiles Policy Framework and Craig walks through implementing a GitHub Workflow to run the tests in a Docker container for each Pull Request. ## Video {#video} The video recording is available on YouTube: {{< youtube jDaF-803HG0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [lib/testing.cf](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-testing.html) - [cfengine-apt](https://github.com/nickanderson/cfengine-apt) -------------------------------------------------------------------------------- title: "How can I get a list of specific key values from an array of objects in JSON?" url: /blog/2023/how-can-i-get-a-list-of-specific-key-values-from-an-array-of-objects-in-json/index.md date: "2023-07-27" -------------------------------------------------------------------------------- This question was covered in [The agent is in, Episode 27 - CFEngine Q&A: Policy questions](/blog/2023/the-agent-is-in-episode-27). Given the following JSON, how can I get a list containing just the values of name? ```json [ { "name": "Aurora", "description": "Illuminating" }, { "name": "Orion", "description": "Stellar" }, { "name": "Luna", "description": "Serene" }, { "name": "Phoenix", "description": "Resilient" }, { "name": "Atlas", "description": "Strong" } ] ``` ## Using `maparray()` {#using-maparray} The most concise and direct way to achieve something like this is to use the [maparray()](https://docs.cfengine.com/latest/reference-functions-maparray.html) function. It iterates over a list or data container applying a pattern based on `$(this.k)` and `$(this.v)` of the currently iterated element to produce a list. ```cfengine3 bundle agent __main__ { methods: "example"; } bundle agent example { vars: "d" data => '[ { "name": "Aurora", "description": "Illuminating" }, { "name": "Orion", "description": "Stellar" }, { "name": "Luna", "description": "Serene" }, { "name": "Phoenix", "description": "Resilient" }, { "name": "Atlas", "description": "Strong" } ]'; "names" slist => sort( maparray( "$(d[$(this.k)][name])", d ), lex ); } ``` Here we run with `--show-evaluated-vars=default:example` to show variable values at the end of the agent run filtered to show only variables in the `default` namespace from bundles matching `example` and `--file /tmp/example-1.cf` to specify the policy entry: ```command cf-agent --show-evaluated-vars=default:example --file /tmp/example-1.cf ``` ```output Variable name Variable value Meta tags Comment default:example.d [{"description":"Illuminating","name":"Aurora"},{"description":"Stellar","name":"Orion"},{"description":"Serene","name":"Luna"},{"description":"Resilient","name":"Phoenix"},{"description":"Strong","name":"Atlas"}] source=promise default:example.names {"Atlas","Aurora","Luna","Orion","Phoenix"} source=promise ``` ## Using an associative array {#using-an-associative-array} Associative arrays are a very useful and flexible construct. They allow you to build structured data up key value by key value, potentially applying specific rules at each step. We need to iterate over the data container and extract the names prior to getting the consolidated list. ```cfengine3 bundle agent __main__ { methods: "example"; } bundle agent example { vars: "d" data => '[ { "name": "Aurora", "description": "Illuminating" }, { "name": "Orion", "description": "Stellar" }, { "name": "Luna", "description": "Serene" }, { "name": "Phoenix", "description": "Resilient" }, { "name": "Atlas", "description": "Strong" } ]'; "d_keys" slist => getindices( d ); "name[$(d_keys)]" string => "$(d[$(d_keys)][name])"; "names" slist => sort( getvalues( name ), lex ); } ``` Again, we run the agent with `--show-evaluated-vars=default:example` to see these variables: ```command cf-agent --show-evaluated-vars=default:example --file /tmp/example-2.cf ``` ```output Variable name Variable value Meta tags Comment default:example.d [{"description":"Illuminating","name":"Aurora"},{"description":"Stellar","name":"Orion"},{"description":"Serene","name":"Luna"},{"description":"Resilient","name":"Phoenix"},{"description":"Strong","name":"Atlas"}] source=promise default:example.d_keys {"0","1","2","3","4"} source=promise default:example.name[0] Aurora source=promise default:example.name[1] Orion source=promise default:example.name[2] Luna source=promise default:example.name[3] Phoenix source=promise default:example.name[4] Atlas source=promise default:example.names {"Atlas","Aurora","Luna","Orion","Phoenix"} source=promise ``` In the above output you may wonder why `default:example.d_keys` is a list of numbers. That's because `getindices()` returns the positions of each element when targeting a list, data container array, or data container object of anonymous objects. Let's take it a bit further, this time getting the list of names where `description` begins with `S`. ```cfengine3 bundle agent __main__ { methods: "example"; } bundle agent example { vars: "d" data => '[ { "name": "Aurora", "description": "Illuminating" }, { "name": "Orion", "description": "Stellar" }, { "name": "Luna", "description": "Serene" }, { "name": "Phoenix", "description": "Resilient" }, { "name": "Atlas", "description": "Strong" } ]'; "d_keys" slist => getindices( d ); "name[$(d_keys)]" string => "$(d[$(d_keys)][name])", if => regcmp( "S.*", "$(d[$(d_keys)][description])" ); "names" slist => sort( getvalues( name ), lex ); } ``` And look at the result: ```command cf-agent --show-evaluated-vars=default:example --file /tmp/example-3.cf ``` ```output Variable name Variable value Meta tags Comment default:example.d [{"description":"Illuminating","name":"Aurora"},{"description":"Stellar","name":"Orion"},{"description":"Serene","name":"Luna"},{"description":"Resilient","name":"Phoenix"},{"description":"Strong","name":"Atlas"}] source=promise default:example.d_keys {"0","1","2","3","4"} source=promise default:example.name[1] Orion source=promise default:example.name[2] Luna source=promise default:example.name[4] Atlas source=promise default:example.names {"Atlas","Luna","Orion"} source=promise ``` We can even do something like uppercase names where descriptions starting with `S` have more than 6 characters and lowercase names where descriptions starting with `S` have less than 7 characters. ```cfengine3 bundle agent __main__ { methods: "example"; } bundle agent example { vars: "d" data => '[ { "name": "Aurora", "description": "Illuminating" }, { "name": "Orion", "description": "Stellar" }, { "name": "Luna", "description": "Serene" }, { "name": "Phoenix", "description": "Resilient" }, { "name": "Atlas", "description": "Strong" } ]'; "d_keys" slist => getindices( d ); "name[$(d_keys)]" string => string_upcase( "$(d[$(d_keys)][name])" ), if => and( regcmp( "S.*", "$(d[$(d_keys)][description])" ), isgreaterthan( string_length( "$(d[$(d_keys)][description])" ), 6)); "name[$(d_keys)]" string => string_downcase( "$(d[$(d_keys)][name])" ), if => and( regcmp( "S.*", "$(d[$(d_keys)][description])" ), islessthan( string_length( "$(d[$(d_keys)][description])" ), 7)); "names" slist => sort( getvalues( name ), lex ); } ``` Output: ```command cf-agent --no-lock --show-evaluated-vars=default:example --file /tmp/example-4.cf ``` ```output Variable name Variable value Meta tags Comment default:example.d [{"description":"Illuminating","name":"Aurora"},{"description":"Stellar","name":"Orion"},{"description":"Serene","name":"Luna"},{"description":"Resilient","name":"Phoenix"},{"description":"Strong","name":"Atlas"}] source=promise default:example.d_keys {"0","1","2","3","4"} source=promise default:example.name[1] ORION source=promise default:example.name[2] luna source=promise default:example.name[4] atlas source=promise default:example.names {"ORION","atlas","luna"} source=promise ``` -------------------------------------------------------------------------------- title: "How can I test CFEngine policy?" url: /blog/2023/how-can-i-test-cfengine-policy/index.md date: "2023-07-27" -------------------------------------------------------------------------------- This question was covered in [The agent is in, Episode 27 - CFEngine Q&A: Policy questions](/blog/2023/the-agent-is-in-episode-27). Testing is an important part of the software life-cycle. Writing tests for your CFEngine policy can help to bring improved assurance that your policy behaves as expected. Follow along and write your first test policy. ## Test stages {#test-stages} When writing tests there are three or four basic stages that typically need to be handled. 1. **Initialization** - Set up the necessary conditions for the test, e.g. create some files to be edited. 2. **Testing** - Running the policy whose behavior you wish to test. 3. **Checking** - Inspecting the results of the test policy to see if they conform with expectations. 4. **Cleanup** - You might need to cleanup artifacts produced by the test if your testing system does not handle it for you. These stages map well to a sequence of bundles. So, a simple test template could look like this: ```cfengine3 bundle agent __main__ { methods: "init"; "test"; "check"; "cleanup"; } bundle agent init {} bundle agent test {} bundle agent check {} bundle agent cleanup {} ``` Let's take this template and start filling it in. First, we need some test case. Let's do something simple, for example, making sure the `reports` promise type attribute [report_to_file](https://docs.cfengine.com/docs/3.18/reference-promise-types-reports.html#report_to_file) works as expected. `report_to_file` simply emits a report to a file instead of to standard output. To initialize the test we might want to make sure the file we report to does not exist, then we would test reporting to the file. After reporting to the file we would inspect the file to make sure the content within matches our expectations before finally cleaning up that test file. Let's get going with `bundle agent init`. ### bundle agent init {#bundle-agent-init} During test initialization let's make sure the test file does not exist when we exercise `report_to_file` in the `test` bundle. ```cfengine3 bundle agent init # @brief Initialize the environment to prepare for the test { vars: "testfile" string => "/tmp/test-reports-report_to_file.txt"; files: "$(testfile)" delete => my_tidy, if => fileexists( $(testfile) ); } body delete my_tidy # @brief Duplicated from the stdlib { dirlinks => "delete"; rmdirs => "true"; } ``` ### bundle agent test {#bundle-agent-test} In our test bundle we exercise `report_to_file` emitting our report to our test file. ```cfengine3 bundle agent test # @brief Test that report_to_file causes reports promises to be directed into a file. { reports: "Hello World!" report_to_file => "$(init.testfile)"; } ``` ### bundle agent check {#bundle-agent-check} Next we need to check that the file does indeed contain the expected content and emit something that lets us know if it passed or failed. ```cfengine3 bundle agent check # @brief Check that reports sent to a file are found as expected { classes: "testfile_exists" expression => fileexists( "$(init.testfile)" ); reports: testfile_exists:: "$(with)" with => ifelse( regcmp( "Hello World!", readfile( "$(init.testfile)" ) ), "Pass", "FAIL" ); !testfile_exists:: "FAIL"; } ``` ### bundle agent cleanup {#bundle-agent-cleanup} Let's cleanup that test artifact. It's perhaps duplicating efforts from init, but it's best to not leave unwanted artifacts behind if there is no test runner processing the artifacts from each test. ```cfengine3 bundle agent cleanup # @brief Clean up after checking test results { files: "$(init.testfile)" delete => my_tidy, if => fileexists( "$(init.testfile)" ); } ``` ### bundle agent \_\_main\_\_ {#bundle-agent} Finally, let's add `bundle agent __main__` to drive this bundle sequence if the test is the policy entry. ```cfengine3 bundle agent __main__ # @brief Clean up after checking test results { methods: "init"; "test"; "check"; "cleanup"; } ``` ### Full basic example test policy {#full-basic-example-test-policy} All together in `/tmp/example-test.cf` we now have this policy. ```cfengine3 bundle agent init # @brief Initialize the environment to prepare for the test { vars: "testfile" string => "/tmp/test-reports-report_to_file.txt"; files: "$(testfile)" delete => my_tidy, if => fileexists( $(testfile) ); } body delete my_tidy # @brief Duplicated from the stdlib { dirlinks => "delete"; rmdirs => "true"; } bundle agent test # @brief Test that report_to_file causes reports promises to be directed into a file. { reports: "Hello World!" report_to_file => "$(init.testfile)"; } bundle agent check # @brief Check that reports sent to a file are found as expected { classes: "testfile_exists" expression => fileexists( "$(init.testfile)" ); reports: testfile_exists:: "$(with)" with => ifelse( regcmp( "Hello World!", readfile( "$(init.testfile)" ) ), "Pass", "FAIL" ); !testfile_exists:: "FAIL"; } bundle agent cleanup # @brief Clean up after checking test results { files: "$(init.testfile)" delete => my_tidy, if => fileexists( "$(init.testfile)" ); } bundle agent __main__ # @brief Clean up after checking test results { methods: "init"; "test"; "check"; "cleanup"; } ``` #### Running the example test policy {#running-the-example-test-policy} Let's run it and inspect the output: ```command cf-agent --no-lock --file /tmp/example-test.cf ``` ```output R: Pass ``` Let's synthesize a failure by blocking writes to the test file. `chattr +i` sets a file to be immutable. A file with the `i` attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the `CAP_LINUX_IMMUTABLE` capability can set or clear this attribute. ```command touch /tmp/test-reports-report_to_file.txt chattr +i /tmp/test-reports-report_to_file.txt lsattr /tmp/test-reports-report_to_file.txt ``` ```output ----i---------e------- /tmp/test-reports-report_to_file.txt ``` ```command cf-agent --no-lock --file /tmp/example-test.cf ``` ```output error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt' error: Method 'init' failed in some repairs error: Could not open log file '/tmp/test-reports-report_to_file.txt', message 'Hello World!'. (fopen: Operation not permitted) R: FAIL error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt' error: Method 'cleanup' failed in some repairs ``` Great! Now we have a functioning test. Let's clear our synthesized failure before proceeding (again, you will need elevated privileges). ```command chattr -i /tmp/test-reports-report_to_file.txt lsattr /tmp/test-reports-report_to_file.txt rm /tmp/test-reports-report_to_file.txt ``` ```output --------------e------- /tmp/test-reports-report_to_file.txt ``` A next logical step would be to get this test result interpreted by something. The Masterfiles Policy Framework ships with a testing library (`lib/testing.cf`) to facilitate emitting output compatible for Test Anything Protocol (TAP) or JUnit. Let's extend the existing test to output in JUnit. ## Adding JUnit output support from lib/testing.cf {#adding-junit-output-support-from-lib-testing-dot-cf} First, we need to include the library as part of our inputs. ```cfengine3 body file control # @brief Include necessary files from the standard library { inputs => { "$(sys.libdir)/testing.cf", "$(sys.libdir)/reports.cf", # For printfile => default:cat() }; } ``` ### Refactoring bundle agent check {#refactoring-bundle-agent-check} Next refactor `bundle agent check` to emit in the JUnit format to a file and print it out. Normally your testing tools (Jenkins, GitHub, etc ...) would read that XML in to give you some nice visualization of the result. ```cfengine3 bundle agent check # @brief Check that reports sent to a file are found as expected { classes: "testfile_exists" expression => fileexists( "$(init.testfile)" ); testfile_exists:: "expected_string_found_in_testfile" scope => "namespace", if => regcmp( "Hello World!", readfile( "$(init.testfile)" ) ); methods: "Register test case" usebundle => testing_ok_if( "expected_string_found_in_testfile", "Test that the expected string is found in file created by printfile => report_to_file", "error: printfile => report_to_file failed to create a file with the expected string", "trace", "jUnit" ); "jUnit Report" usebundle => testing_junit_report( "/tmp/test-result.xml" ); reports: inform_mode:: "The full xml report:" printfile => default:cat( "/tmp/test-result.xml" ), if => fileexists( "/tmp/test-result.xml" ); !expected_string_found_in_testfile:: "$(init.testfile) did not have expected content. Contains:" printfile => default:cat( "$(init.testfile)" ), if => fileexists( "/tmp/test-result.xml" ); } ``` That gives us this full policy file: ```cfengine3 body file control # @brief Include necessary files from the standard library { inputs => { "$(sys.libdir)/testing.cf", "$(sys.libdir)/reports.cf", # For printfile => default:cat() }; } bundle agent init # @brief Initialize the environment to prepare for the test { vars: "testfile" string => "/tmp/test-reports-report_to_file.txt"; files: "$(testfile)" delete => my_tidy, if => fileexists( $(testfile) ); } body delete my_tidy # @brief Duplicated from the stdlib { dirlinks => "delete"; rmdirs => "true"; } bundle agent test # @brief Test that report_to_file causes reports promises to be directed into a file. { reports: "Hello World!" report_to_file => "$(init.testfile)"; } bundle agent check # @brief Check that reports sent to a file are found as expected { classes: "testfile_exists" expression => fileexists( "$(init.testfile)" ); testfile_exists:: "expected_string_found_in_testfile" scope => "namespace", if => regcmp( "Hello World!", readfile( "$(init.testfile)" ) ); methods: "Register test case" usebundle => testing_ok_if( "expected_string_found_in_testfile", "Test that the expected string is found in file created by printfile => report_to_file", "error: printfile => report_to_file failed to create a file with the expected string", "trace", "jUnit"); "jUnit Report" usebundle => testing_junit_report( "/tmp/test-result.xml" ); reports: inform_mode:: "The full xml report:" printfile => default:cat( "/tmp/test-result.xml" ), if => fileexists( "/tmp/test-result.xml" ); !expected_string_found_in_testfile:: "$(init.testfile) did not have expected content. Contains:" printfile => default:cat( "$(init.testfile)" ), if => fileexists( "/tmp/test-result.xml" ); } bundle agent cleanup # @brief Clean up after checking test results { files: "$(init.testfile)" delete => my_tidy, if => fileexists( "$(init.testfile)" ); } bundle agent __main__ # @brief Clean up after checking test results { methods: "init"; "test"; "check"; "cleanup"; } ``` ### Running the refactored example test {#running-the-refactored-example-test} Let's run it to see it pass. ```command cf-agent -KIf /tmp/example-test-with-junit.cf ``` ```output R: testing_ok_if: adding testing report for class expected_string_found_in_testfile at position 1 info: Updated rendering of '/tmp/test-result.xml' from mustache template '/home/nickanderson/.cfagent/inputs/lib/templates/junit.mustache' R: testing_generic_report: report summary: counts = 1/1/0/0/0 failed = { }, passed = { "testing_expected_string_found_in_testfile" }, skipped = { }, todo = { }, failed = { }; tests = [{"tap_message":"ok Test that the expected string is found in file created by printfile => report_to_file","test_message":"Test that the expected string is found in file created by printfile => report_to_file","test_offset":1,"testcase":"expected_string_found_in_testfile"}]+[]+[]+[] R: The full xml report: R: R: R: R: Test that the expected string is found in file created by printfile => report_to_file R: R: R: R: R: R: R: info: Deleted file '/tmp/test-reports-report_to_file.txt' ``` Now, let's synthesize failure again (you will need elevated privileges to use `chattr`). ```command touch /tmp/test-reports-report_to_file.txt chattr +i /tmp/test-reports-report_to_file.txt lsattr /tmp/test-reports-report_to_file.txt ``` ```output ----i---------e------- /tmp/test-reports-report_to_file.txt ``` Now, let's run the policy to see what failure with JUnit output looks like. ```command cf-agent -KIf /tmp/example-test-with-junit.cf ``` ```output error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt' error: Method 'init' failed in some repairs error: Could not open log file '/tmp/test-reports-report_to_file.txt', message 'Hello World!'. (fopen: Operation not permitted) R: testing_ok_if: adding testing report for class expected_string_found_in_testfile at position 1 info: Updated rendering of '/tmp/test-result.xml' from mustache template '/home/nickanderson/.cfagent/inputs/lib/templates/junit.mustache' R: testing_generic_report: report summary: counts = 1/0/0/0/1 failed = { "testing_expected_string_found_in_testfile_failed" }, passed = { }, skipped = { }, todo = { }, failed = { "testing_expected_string_found_in_testfile_failed" }; tests = []+[]+[]+[{"fail_message":"error: printfile => report_to_file failed to create a file with the expected string","tap_message":"not ok Test that the expected string is found in file created by printfile => report_to_file","test_message":"Test that the expected string is found in file created by printfile => report_to_file","test_offset":1,"testcase":"expected_string_found_in_testfile_failed","trace_message":"trace"}] R: The full xml report: R: R: R: R: R: R: Test that the expected string is found in file created by printfile => report_to_filetrace R: R: R: R: R: R: R: R: /tmp/test-reports-report_to_file.txt did not have expected content. Contains: error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Couldn't unlink '/tmp/test-reports-report_to_file.txt' tidying. (unlink: Operation not permitted) error: Errors encountered when actuating files promise '/tmp/test-reports-report_to_file.txt' error: Method 'cleanup' failed in some repairs ``` Let's clean up that immutable file. ```command chattr -i /tmp/test-reports-report_to_file.txt rm /tmp/test-reports-report_to_file.txt ``` -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 27 - CFEngine Q&A: Policy questions" url: /blog/2023/the-agent-is-in-episode-27/index.md date: "2023-07-27" -------------------------------------------------------------------------------- Unlock the power of CFEngine with expert insights and get your burning policy questions. Cody, Craig and Nick discuss and answer CFEngine policy questions submitted by users. ## Video {#video} The video recording is available on YouTube: {{< youtube q5NAZSL0-qg >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Questions {#questions} These are the questions and policy used during the episode but each question has a separate blog post that goes into more detail. ### How can I get a list of specific key values from an array of objects in JSON {#how-can-i-get-a-list-of-specific-key-values-from-an-array-of-objects-in-json} ```cfengine3 bundle agent __main__ { vars: "d" data => '[ {"name": "Aurora", "value": "1"}, {"name": "Orion", "value": "2"}, {"name": "Luna", "value": "3"}, {"name": "Phoenix", "value": "4"}, {"name": "Atlas", "value": "5"} ]'; "d_names" slist => sort( maparray( "$(d[$(this.k)][name])", d ), lex ); } ```
Code Snippet 1: Question 1 iteration 0
Output: ```text Variable name Variable value Meta tags Comment default:main.d [{"name":"Aurora","value":"1"},{"name":"Orion","value":"2"},{"name":"Luna","value":"3"},{"name":"Phoenix","value":"4"},{"name":"Atlas","value":"5"}] source=promise default:main.d_names {"Atlas","Aurora","Luna","Orion","Phoenix"} source=promise ``` Refactored to use an associative array. ```cfengine3 bundle agent __main__ { vars: "d" data => '[ {"name": "Aurora", "value": "1"}, {"name": "Orion", "value": "2"}, {"name": "Luna", "value": "3"}, {"name": "Phoenix", "value": "4"}, {"name": "Atlas", "value": "5"} ]'; "di" slist => getindices( d ); "d_names[$(di)]" string => "$(d[$(di)][name])"; "names" slist => sort( getvalues( d_names ), lex); } ```
Code Snippet 2: Question 1 iteration 1
Output: ```text Variable name Variable value Meta tags Comment default:main.d [{"name":"Aurora","value":"1"},{"name":"Orion","value":"2"},{"name":"Luna","value":"3"},{"name":"Phoenix","value":"4"},{"name":"Atlas","value":"5"}] source=promise default:main.d_names[0] Aurora source=promise default:main.d_names[1] Orion source=promise default:main.d_names[2] Luna source=promise default:main.d_names[3] Phoenix source=promise default:main.d_names[4] Atlas source=promise default:main.di {"0","1","2","3","4"} source=promise default:main.names {"Atlas","Aurora","Luna","Orion","Phoenix"} source=promise ``` Refactoring to show additional flexibility building a data structure key by key. ```cfengine3 bundle agent __main__ { vars: "d" data => '[ {"name": "Aurora", "value": "1"}, {"name": "Orion", "value": "2"}, {"name": "Luna", "value": "3"}, {"name": "Phoenix", "value": "4"}, {"name": "Atlas", "value": "5"} ]'; "di" slist => getindices( d ); "d_names[$(di)]" string => "$(d[$(di)][name])", if => isgreaterthan( "$(d[$(di)][value])", 3 ); "names" slist => sort( getvalues( d_names ), lex); } ```
Code Snippet 3: Question 1 iteration 2
```text Variable name Variable value Meta tags Comment default:main.d [{"name":"Aurora","value":"1"},{"name":"Orion","value":"2"},{"name":"Luna","value":"3"},{"name":"Phoenix","value":"4"},{"name":"Atlas","value":"5"}] source=promise default:main.d_names[3] Phoenix source=promise default:main.d_names[4] Atlas source=promise default:main.di {"0","1","2","3","4"} source=promise default:main.names {"Atlas","Phoenix"} source=promise ``` Read more in the [related blog post](/blog/2023/how-can-i-get-a-list-of-specific-key-values-from-an-array-of-objects-in-json). ### How can I Test CFEngine Policy {#how-can-i-test-cfengine-policy} ```cfengine3 bundle agent __main__ { methods: "init"; "test"; "check"; } bundle agent init { reports: "$(this.bundle)"; } bundle agent test { reports: "$(this.bundle)"; } bundle agent check { reports: "$(this.bundle)"; } ```
Code Snippet 4: Question 2 iteration 0
Output: ```text R: init R: test R: check ``` Refactored to use lib/testing.cf for emitting JUnit. ```cfengine3 body file control { # We need to include the policy which we are leveraging inputs => { "$(sys.libdir)/lib/testing.cf", # The testing library "$(sys.libdir)/lib/files.cf", # contains copy_from => default:local_dcp, edit_line => default:lines_present "$(sys.libdir)/lib/common.cf", # Note: files.cf loads common.cf which contains printfile => default:cat, but we include it explicitly since we are using it here }; } bundle agent __main__ { methods: "init"; "test"; "check"; "cleanup"; } bundle agent init # @brief Setup the stage for the test { files: # Let's initialize our test file from our pre-recorded starting state. This # is the file that we will make our promise against. "/tmp/test-file.txt" copy_from => default:local_dcp( "/tmp/starting-state.txt" ); } bundle agent test # @brief Excercise the policy we wish to test { files: "/tmp/test-file.txt" edit_line => default:lines_present( "Big ones, small ones, some as big as your head!" ); } bundle agent check # @brief Verify that the state is as we expect { vars: "test_check_required_files" slist => { "/tmp/test-file.txt", "/tmp/expected-end-state.txt" }; classes: "edit_line_default_lines_present_ok" if => filesexist( @(test_check_required_files) ), scope => "namespace", expression => returnszero( concat( "/usr/bin/diff", " /tmp/test-file.txt", " /tmp/expected-end-state.txt" ), "noshell" ); methods: # Register each passing test "Register test for appending a missing line" usebundle => testing_ok_if("edit_line_default_lines_present_ok", "Test that bundle edit_line default:lines_present appends a line that is not present in the file", "error: bundle edit_line default:lines_present did not produce the expected result", "trace", "jUnit"); # Write out the jUnit report "jUnit Report" usebundle => testing_junit_report( "/tmp/test-result.xml" ); reports: inform_mode:: "The full xml report:" printfile => default:cat( "/tmp/test-result.xml" ); } bundle agent cleanup # @brief Cleanup the files we used in our testing. { files: "/tmp/test-file.txt.*" delete => default:tidy; } ```
Code Snippet 5: Question 2 iteration 1
```text info: Copied file '/tmp/starting-state.txt' to '/tmp/test-file.txt.cfnew' (mode '600') info: Moved '/tmp/test-file.txt.cfnew' to '/tmp/test-file.txt' info: Updated file '/tmp/test-file.txt' from 'localhost:/tmp/starting-state.txt' info: Inserted the promised line 'Big ones, small ones, some as big as your head!' into '/tmp/test-file.txt' after locator info: insert_lines promise 'Big ones, small ones, some as big as your head!' repaired info: Edited file '/tmp/test-file.txt' R: testing_ok_if: adding testing report for class edit_line_default_lines_present_ok at position 1 info: Updated rendering of '/tmp/test-result.xml' from mustache template '/home/nickanderson/.cfagent/inputs/lib/templates/junit.mustache' R: testing_generic_report: report summary: counts = 1/1/0/0/0 failed = { }, passed = { "testing_edit_line_default_lines_present_ok" }, skipped = { }, todo = { }, failed = { }; tests = [{"tap_message":"ok Test that bundle edit_line default:lines_present appends a line that is not present in the file","test_message":"Test that bundle edit_line default:lines_present appends a line that is not present in the file","test_offset":1,"testcase":"edit_line_default_lines_present_ok"}]+[]+[]+[] R: The full xml report: R: R: R: R: Test that bundle edit_line default:lines_present appends a line that is not present in the file R: R: R: R: R: R: R: info: Deleted file '/tmp/test-file.txt' info: Deleted file '/tmp/test-file.txt.cf-before-edit' ``` Read more in the [related blog post](/blog/2023/how-can-i-test-cfengine-policy). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [Core policy acceptance tests and framework](https://github.com/cfengine/core/tree/master/tests/acceptance) - [lib/testing.cf](https://docs.cfengine.com/latest/reference-masterfiles-policy-framework-lib-testing.html) in the Masterfiles Policy Framework -------------------------------------------------------------------------------- title: "Iterating on CFEngine policy for pinning packages in APT" url: /blog/2023/iterating-on-cfengine-policy-for-pinning-pacakges-in-apt/index.md date: "2023-07-20" -------------------------------------------------------------------------------- I was chatting with someone recently about some security maintenance tasks and they were bemoaning that some software updates had turned into a yak shaving[^fn:1]. Updating this required updating that, required updating that on N hosts of varying platforms and flavors. So, they asked me how could they avoid updating a specific package and naturally I said, let's just prototype some policy. The incipiency of said yak shaving was updating packages via `apt`, Debian flavored systems default package manager. `apt upgrade` is being used to update all packages, but we need to exclude a specific package from the updates until the dependency chain has been resolved. What we are talking about is simply package pinning. This can be done by creating a file `/etc/apt/preferences.d/yourpackage`. ```cfg Package: yourpackage Pin: version 1.2.3 Pin-Priority: 999 ```
Code Snippet 1: /etc/apt/preferences.d/yourpackage
## Iteration 0 - A very simple prototype {#iteration-0-a-very-simple-prototype} We can write a very simple policy to achieve this. The most simple solution that I could think of looked like this: ```cfengine3 bundle agent __main__ { files: "/etc/apt/preferences.d/zpackage" content => "Package: zpackage Pin: version 1.2.3 Pin-Priority: 999 "; } ```
Code Snippet 2: Prototype policy to pin the version of a package in apt
It's not an ideal policy, it's just a simple prototype. Let's iterate on it a bit. ## Iteration 1 - Specifying the proper file permissions {#iteration-1-specifying-the-proper-file-permissions} First, what about permissions? By default CFEngine will create files with mode of `600` (Owner read-write, group nothing, other nothing). I failed to find any canonical reference as to what the permissions for the file _should_ be, but intuitively `644` (owner read-write, group read, other read) with user and group owners being `root` seems appropriate and comports with the default permissions of the directory on some spot checked Debian systems.[^fn:2] ```cfengine3 bundle agent __main__ { files: "/etc/apt/preferences.d/zpackage" perms => default:mog( "644", "root", "root" ), content => "Package: zpackage Pin: version 1.2.3 Pin-Priority: 999 "; } ```
Code Snippet 3: Prototype policy to pin the version of a package in apt after adding permissions
I don't like that way that looks, I find the string wrapping ugly. ## Iteration 2 - Use concat() to make the policy more readable {#iteration-2-use-concat-to-make-the-policy-more-readable} We can use `concat()` to make things align. ```cfengine3 bundle agent __main__ { files: "/etc/apt/preferences.d/zpackage" perms => default:mog( "644", "root", "root" ), content => concat( "Package: zpackage", "$(const.n)", "Pin: version 1.2.3", "$(const.n)", "Pin-Priority: 999", "$(const.n)" ); } ```
Code Snippet 4: Prototype policy to pin the version of a package in apt after using concat to keep strings aligned
We are repeating the package name, that's not very DRY.[^fn:3] ## Iteration 3 - Use `with` to avoid repetition {#iteration-3-use-with-to-avoid-repetition} We can use the `with` attribute to avoid a case of repetition. ```cfengine3 bundle agent __main__ { files: "/etc/apt/preferences.d/$(with)" perms => default:mog( "644", "root", "root" ), with => "zpackage", content => concat( "Package: $(with)", "$(const.n)", "Pin: version 1.2.3", "$(const.n)", "Pin-Priority: 999", "$(const.n)" ); } ```
Code Snippet 5: Prototype policy to pin the version of a package in apt after using with to avoid package name repetition
That's about as far as I can get at the current level of abstraction. It's still not reusable, let's move it to a bundle so that it's more reusable. ## Iteration 4 - Use a parameterized bundle to make the policy reusable {#iteration-4-use-a-parameterized-bundle-to-make-the-policy-reusable} Here we move the promise to its own agent bundle `apt_package_version_pinned` taking parameters for the name of the package and the version we wish to pin. ```cfengine3 bundle agent __main__ { methods: "Ensure apt package is pinned to version" usebundle => apt_package_version_pinned( "zpackage", "1.2.3" ); } bundle agent apt_package_version_pinned( name, version ) # @brief Ensure that `name` is pinned to `version` for the apt package manager # @param `name` The name of the package to pin # @param `version` The name of the package to pin { files: "/etc/apt/preferences.d/$(name)" perms => default:mog( "644", "root", "root" ), content => concat( "Package: $(name)", "$(const.n)", "Pin: version $(version)", "$(const.n)", "Pin-Priority: 999", "$(const.n)" ); } ```
Code Snippet 6: Prototype policy to pin the version of a package in apt after using a bundle and parameters for abstraction
What about when I am ready to clear this pinned configuration? Let's make its pinned state configurable. ## Iteration 5 - Adding a parameter to specify the state of the pin {#iteration-5-adding-a-parameter-to-specify-the-state-of-the-pin} Here we have added the `present_absent` parameter to the bundle which lets us specify if we want the package pin configuration to be _present_ or _absent_. This iteration also added a promise to ensure the configuration directory itself was present with appropriate permissions when we are requesting a present state and the configuration file also gets a header comment indicating CFEngine manages the file. ```cfengine3 bundle agent __main__ { methods: "Ensure apt package is pinned to version" usebundle => apt_package_version_pinned( "zpackage", "1.2.3", "present" ); } bundle agent apt_package_version_pinned( name, version, present_absent ) # @brief Ensure that `name` is pinned to `version` for the apt package manager or not based on the desired state # @param `name` The name of the package to pin, case matters. # @param `version` The version of the package to pin # @param `present_absent` The desired state of the package pin (present|absent), case does not matter. Note: the version does not matter when absent is specified. { files: "/etc/apt/preferences.d/." perms => default:mog( "755", "root", "root" ), if => regcmp( "(?i)present", "$(present_absent)" ), comment => concat( "If we are promising that a package", " should be pinned, the configuration", " directory should exist." ); "/etc/apt/preferences.d/$(name)" perms => default:mog( "644", "root", "root" ), content => concat( "# Managed by CFEngine", "$(const.n)", "Package: $(name)", "$(const.n)", "Pin: version $(version)", "$(const.n)", "Pin-Priority: 999", "$(const.n)" ), if => regcmp( "(?i)present", "$(present_absent)" ); "/etc/apt/preferences.d/$(name)" delete => default:tidy, if => regcmp( "(?i)absent", "$(present_absent)" ); } ```
Code Snippet 7: Prototype policy to pin the version of a package in apt after instrumenting the bundle for a pinned state to allow for removal
Now we have a re-usable bundle which can be used to specify packages that should be pinned to a specific version or not. The next logical step would be to elevate this policy to a CFEngine Build[^fn:4] module so that it's easier for others to access and use. Will someone beat me to the punch? [^fn:1]: Yak shaving, a task that leads you to perform another related task and so on, and so on -- distracting from the original goal. [^fn:2]: I would appreciate it greatly if you would point me to a canonical answer for what the proper permissions for these configuration files should be. I was unable to find any specific mention across Debian bug trackers and man pages. [^fn:3]: DRY: Don't Repeat Yourself [^fn:4]: The CFEngine Build website and main index of modules -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 26 - Demo of CFEngine 3.22" url: /blog/2023/the-agent-is-in-episode-26/index.md date: "2023-06-29" -------------------------------------------------------------------------------- Have you seen what's new in CFEngine 3.22.0? Ole Herman Elgesem, CFEngine Product Manager joins Cody, Craig and Nick to give a tour of the changes in recently released CFEngine 3.22.0 Mission Portal. See how filters have been improved and how the new Groups feature makes it easier to affect change across your infrastructure and enforce package compliance with a new module, `packages-allowlist-snapshot` from CFEngine Build. ## Video {#video} The video recording is available on YouTube: {{< youtube OrkDfxMShuU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). - [00:58](https://youtu.be/OrkDfxMShuU?t=58) - New filter UI - [03:33](https://youtu.be/OrkDfxMShuU?t=213) - Groups in Mission Portal - [19:28](https://youtu.be/OrkDfxMShuU?t=1168) - `packages-allowlist-snapshot` module from CFEngine Build - [27:21](https://youtu.be/OrkDfxMShuU?t=1661) - Controlling `packages-allowlist-snapshot` using Group data ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [CFEngine 3.22.0 release blog post](https://cfengine.com/blog/2023/cfengine-3-22-released-coordination/) - [CFEngine Build](https://build.cfengine.com/) - [packages-allowlist-snapshot](https://build.cfengine.com/modules/packages-allowlist-snapshot/) - [delete-files](https://build.cfengine.com/modules/delete-files/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.22 released - Coordination" url: /blog/2023/cfengine-3-22-released-coordination/index.md date: "2023-06-16" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.22.0\! The focus of this new version has been _coordination_. This is a non-LTS (non-supported) release, where we introduce new features for users to test and give feedback on, allowing us to polish before the next LTS. (CFEngine 3.24 LTS is scheduled to release summer 2024). ## What's new ### New host filters The host filter from inventory reports has been upgraded. You can now add rules based on classes, such as `linux`, `windows`, `redhat`, `ubuntu`, `xen`, `policy_server`, `cfengine_3_21`, `ipv4_172_31`, etc: {{< img "images/filter.png" "Host filter selecting hosts with an OS regex for Ubuntu 18, 20 or 22, and ipv4_172_31 class as well as 2 hosts explicitly.">}} Below the rules, which are dynamic conditions, there are lists of hosts to include and exclude in a more static manner. The new filters allow you to view a dynamically changing list of hosts, a static selection, or a mix of both. When used together, the list of hosts to include is added in addition to the condition based rules. You can easily add some hosts to your results which were not captured by the rules. ### Groups in Mission Portal You can now create groups of your hosts in Mission Portal. They are based on the same host filter system shown above. > A group is a saved host filter with a name. Each group has a name and a filter, and allows you to see details about the hosts in it: {{< wideimg "images/groups.png" "Groups application showing a group of Ubuntu hosts across different versions.">}} With groups, you can split up your infrastructure into logically distinct groups, based on operating system, function, environment, CFEngine version, etc. {{< img "images/list-of-groups.png" "Listing of groups, with names such as All hosts, CFEngine 3.22, Ubuntu, and Production.">}} #### Personal and shared groups 2 types of groups are available: personal and shared. _Personal groups_ allow individual users to organize hosts on their own, without affecting what others see or the behavior of the hosts. _Shared groups_ are available to the entire organization, and can be used to set data for the hosts in that group, controlling what the hosts in that group are doing. The RBAC (Role Based Access Control) system specifies who can do what with groups. By default users are able to create personal groups, but only see shared groups (not create, delete, or edit). #### Group data (CMDB) Through the **Data** tab of a shared group, you can assign variables and classes to the hosts, in a similar manner to **Host specific data** in the host's info page: {{< wideimg "images/group-data.png" "UI for editing group data, showing the Production group with the class to enable autorun and a list of files to delete as a variable.">}} In case of conflicts, users should be aware of the merging behavior: - As before, data from the CMDB will be preferred over data from the augments file (`def.json`). - This allows you to completely overwrite the values set in this JSON file which is commonly kept together with the policy set, with data from your (per-hub) database. - CMDB data is merged: - Shared groups with data are sorted by their priority, with host specific data at the end. (Host specific data is treated as more specific and thus preferred over group data). - Values (strings) are overwritten, lists and data containers are extended. We plan to expand the possibilities of the CMDB, allowing different merging behavior and enabling users to add CFEngine Build module input per group. ### Adding columns in inventory reports As shown in the release blog post for 3.21.2, there is a new UI to modify columns in an inventory report: {{< wideimg "images/add-columns.png" "Modal window with options to add or remove columns of inventory attributes.">}} The new window allows you to easily find columns you want to add (among a large collection of inventory attributes), and also enables adding multiple columns and deleting columns at the same time. ### Policy language function: isreadable() In some situations, reading a file can be slow, especially if it's an NFS drive over a slow network. Blindly attempting to read a large file over a slow network can cause the agent to block for an extended period of time. To try to mitigate this, we've added a function which allows you to try reading a file, with some timeout, and then letting you react based on the result; ```cfengine3 bundle agent __main__ { vars: "filename" string => "/tmp/foo.txt"; "timeout" int => "3"; reports: "File '$(filename)' is readable" if => isreadable("$(filename)", "$(timeout)"); } ``` This function attempts to read 1 byte from the specified file, and times out in the specified number of seconds. Thus, if the file is not _readable within x seconds_ you can choose to not read it and print an error message, for example. It should be noted that this is very much a best-effort approach, there is no guarantee about that file. After having checked that the file is readable, it can still become unreadable, or very slow to read before you actually try to read the contents of the file. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.22.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.22/release-notes-whatsnew-changelog-core.html) - [3.22.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.22/release-notes-whatsnew-changelog-enterprise.html) - [3.22.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.22/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the recently released [3.21.2](/blog/2023/cfengine-3-18-5-and-3-21-2-released/), these dependencies have been updated: | CFEngine version | 3.21.2 | **3.22.0** | | :-------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.55 | **2.4.57** | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | 3.9 | **3.10** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.40.1 | **2.41.0** | | [libcurl](https://curl.haxx.se/download.html) | 8.0.1 | **8.1.2** | | [libexpat](https://libexpat.github.io/) | - | **2.5.0** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.11.2 | **2.11.4** | | [OpenSSL](https://openssl.org/) | 3.0.8 | **3.1.1** | | [PHP](https://php.net/) | 8.1.12 | **8.2.7** | | [PostgreSQL](https://www.postgresql.org/) | 15.2 | **15.3** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, you have to specify this release explicitly: ```command cf-remote --version 3.22.0 download ``` (`cf-remote` defaults to the latest LTS release, which is currently 3.21.2). ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "CFEngine 3.18.5 and 3.21.2 released" url: /blog/2023/cfengine-3-18-5-and-3-21-2-released/index.md date: "2023-06-09" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.18.5 and 3.21.2\! These releases mainly contain bug fixes, but there is one UI improvement to highlight here; ## Adding columns in inventory reports {{< wideimg "images/add-columns.png" "Modal window with options to add or remove columns of inventory attributes.">}} This new window allows you to easily find the columns you want to add (among a large collection of inventory attributes), and also enables adding multiple columns and deleting columns at the same time. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.18.5 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.5 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.5 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.2 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.2 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.2 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.21.2 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.18.5 list ``` ```command cf-remote --version 3.21.2 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Migrating to cfbs" url: /blog/2023/migrating-to-cfbs/index.md date: "2023-06-01" -------------------------------------------------------------------------------- Traditionally, CFEngine policy sets are managed as a whole. When upgrading the Masterfiles Policy Framework (MPF)[^fn:1] users must download the new version of the policy framework and integrate it into the existing policy set, carefully diffing the vendored policy files against their currently integrated policy. Updates to policy authored by others must be sought out and similarly integrated. The burden is on the user to maintain the knowledge of where policy is sourced, if updates are available, and how it is integrated into the policy set as a whole. `cfbs` [^fn:2]the command line tool for the CFEngine Build System[^fn:3] streamlines management of a CFEngine policy set. It facilitates upgrades of the Masterfiles Policy Framework (MPF), consumption of modules (inventory and management policy, custom promise types, compliance reports, etc ...) from build.cfengine.com and other sources. In this blog post we will cover the process of initializing a traditionally managed policy set integrating custom policy in a couple of typical ways and then migrating to a policy set managed by `cfbs`. ## Initializing a traditionally managed policy set {#initalizing-a-traditionally-managed-policy-set} First, we need to initialize a traditional policy set. Let's start with masterfiles from the `3.18.3` release. This way we can go through the process of migrating to the same version with `cfbs`. We need to download it and unpack it into the root of our repository. ```sh MPF_TARBALL_URL=$(cf-remote --version 3.18.3 list masterfiles | tail -n 1) MPF_TARBALL_FILENAME=$(basename $MPF_TARBALL_URL) curl -s -O $MPF_TARBALL_URL tar -zxf $MPF_TARBALL_FILENAME --strip-components=2 rm -f $MPF_TARBALL_FILENAME ls ``` We can see it's a standard stock policy set. ```text cfe_internal controls inventory lib modules promises.cf services standalone_self_upgrade.cf templates update.cf ``` ### Keep generated files from getting committed to the policy set {#keep-generated-files-from-getting-committed-to-the-policy-set} Next we need to lay down the typical `.gitignore` for a policy set. ```sh printf "# These files are generated and should not become part of the policy set\n" > .gitignore printf "cf_promises_validated\n" >> .gitignore printf "cf_promises_release_id\n" >> .gitignore ``` Let's go ahead and initialize the git repository and commit the current changes. ```sh git init git add . git commit -m "In the beginning there was darkness, then Nick said let there be a stock 3.18.3 policy set" | grep -v "create mode" ``` ```text hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: git config --global init.defaultBranch hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: git branch -m Initialized empty Git repository in /tmp/migrating-to-cfbs/.git/ [master (root-commit) fba5378] In the beginning there was darkness, then Nick said let there be a stock 3.18.3 policy set 131 files changed, 27107 insertions(+) ``` Now we have a stock `3.18.3` policy set. ### Integrating custom policy {#integrating-custom-policy} Next we need to add some custom policy. Let's integrate policy in a few different ways to be representative of what you might find in a policy set that has been around for some time. Long, long ago it was standard practice to _modify_ the default policy to add custom policy to `inputs` and the `bundlesequence`. Let's create a custom policy file and add it to _inputs_ and _bundlesequence_. ```sh cat << EOF > custom-policy-1.cf bundle agent custom_policy_1 { reports: 'Policy from \$(this.bundle)'; } EOF sed -i '/^\s*"services\/main.cf",/a "custom-policy-1.cf",' promises.cf sed -i '/^\s*@(def.bundlesequence_end),/a custom_policy_1,' promises.cf ``` These days we have Augments (`def.json`) that allow us to define variables and classes very early during the bundlesequence and the MPF is instrumented with many variables which can be leveraged to influence the behavior of the policy. Let's add a couple more policies integrated explicitly via augments as well as via autorun. ```sh cat << EOF > services/custom-policy-2.cf bundle agent custom_policy_2 { reports: 'Hello from \$(this.bundle) \$(with)' with => join( ",", callstack_promisers() ); } EOF cat << EOF > services/autorun/custom-policy-3.cf bundle agent custom_policy_3 { meta: "tags" slist => { "autorun" }; reports: 'Hello from \$(this.bundle)'; } EOF cat << EOF > def.json { "inputs": [ "services/custom-policy-2.cf" ], "classes": { "services_autorun": [ "any::" ] }, "vars": { "control_common_bundlesequence_end": [ "custom_policy_2" ] } } EOF #git commit -m "Added custom policy integrated via Augments" ``` `services/main.cf` also commonly contains custom policy, typically `methods` type promises to call other custom policy. ```sh sed -i '/^\s*# Activate your custom policies here/a "Calling custom_policy_2 from main" usebundle => custom_policy_2;' services/main.cf ``` With custom policy in place, let's commit. ```sh git add . git commit -m "Added custom policy integrated in various ways" ``` ```text [master e390bb0] Added custom policy integrated in various ways 6 files changed, 28 insertions(+) create mode 100644 custom-policy-1.cf create mode 100644 def.json create mode 100644 services/autorun/custom-policy-3.cf create mode 100644 services/custom-policy-2.cf ``` ## Initialize the cfbs project {#initialize-the-cfbs-project} Now, let's initialize this repository as a cfbs project. The easiest way to get a cfbs project started is to use `cfbs init`. ```sh cfbs init --non-interactive ``` ```text [master f58dc2f] Initialized a new CFEngine Build project 1 file changed, 7 insertions(+) create mode 100644 cfbs.json [master 72ce205] Added module 'masterfiles' 1 file changed, 13 insertions(+), 1 deletion(-) Initialized an empty project called 'Example project' in 'cfbs.json' Added module: masterfiles ``` By default it will use the most recent version of masterfiles, but we want to first get cfbs replicating our existing policy set so we will remove masterfiles. ```sh cfbs remove masterfiles --non-interactive ``` ```text [master c9aeb2a] Removed module 'masterfiles' 1 file changed, 7 insertions(+), 19 deletions(-) rewrite cfbs.json (80%) Removing module 'masterfiles' ``` And add it back at a specific version. ```sh cfbs add masterfiles@3.18.3 --non-interactive ``` ```text [master efc200f] Added module 'masterfiles' 1 file changed, 20 insertions(+), 1 deletion(-) Added module: masterfiles ``` At this point we can `cfbs build`. ```sh cfbs build ``` ```text 001 masterfiles @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded) 001 masterfiles : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh' 001 masterfiles : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y' 001 masterfiles : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g'' 001 masterfiles : run 'find . -name '*.bak' | xargs rm' 001 masterfiles : copy './' 'masterfiles/' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` ### Find the differences between cfbs built policy and traditionally integrated policy set {#find-the-differences-between-cfbs-built-policy-and-traditionally-integrated-policy-set} Now we can see how our `cfbs` built `3.18.3` policy differs from our current policy set. We recursively diff the root of our policy set against the `cfbs build` result in `out/masterfiles`. ```sh diff --exclude=out --recursive --unified . out/masterfiles ``` ```diff Only in .: cfbs.json Only in out/masterfiles/cfe_internal/core/watchdog: README.md Only in out/masterfiles/cfe_internal/enterprise/ha: ha_info.json Only in .: custom-policy-1.cf Only in .: def.json Only in .: .git Only in out/masterfiles: .github Only in .: .gitignore Only in out/masterfiles/inventory: README.md Only in out/masterfiles/lib: README.md Only in out/masterfiles: LICENSE Only in out/masterfiles/modules: promises Only in out/masterfiles: .no-distrib diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf --- ./promises.cf 2023-06-06 08:09:17.303386890 -0500 +++ out/masterfiles/promises.cf 2023-06-06 08:11:05.094842107 -0500 @@ -59,7 +59,6 @@ main, @(cfengine_enterprise_hub_ha.management_bundles), @(def.bundlesequence_end), -custom_policy_1, }; @@ -86,7 +85,6 @@ @(services_autorun.inputs), "services/main.cf", -"custom-policy-1.cf", }; version => "CFEngine Promises.cf 3.18.3"; Only in ./services/autorun: custom-policy-3.cf Only in out/masterfiles/services/autorun: README.md Only in ./services: custom-policy-2.cf diff '--exclude=out' --recursive --unified ./services/main.cf out/masterfiles/services/main.cf --- ./services/main.cf 2023-06-06 08:09:25.299346646 -0500 +++ out/masterfiles/services/main.cf 2023-06-06 08:11:05.098842087 -0500 @@ -10,6 +10,5 @@ { # Activate your custom policies here -"Calling custom_policy_2 from main" usebundle => custom_policy_2; } Only in out/masterfiles/templates: README.md ``` #### Integrate custom policy into cfbs project {#integrate-custom-policy-into-cfbs-project} We can see there are a few differences, but first let's focus on the files that are missing from the cfbs built policy. ```sh diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." ``` ```diff Only in .: cfbs.json Only in .: custom-policy-1.cf Only in .: def.json Only in .: .git Only in .: .gitignore Only in ./services/autorun: custom-policy-3.cf Only in ./services: custom-policy-2.cf ``` We can ignore the `.git`, `.gitignore`, and `cfbs.json`. ```sh diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json" ``` ```diff Only in .: custom-policy-1.cf Only in .: def.json Only in ./services/autorun: custom-policy-3.cf Only in ./services: custom-policy-2.cf ``` ##### custom-policy-1.cf {#custom-policy-1-dot-cf} Now we can easily see that we need to get `custom-policy-1.cf`, `def.json`, `services/autorun/custom-policy-3.cf`, and `services/custom-policy-2.cf` into the `cfbs` built policy set. Let's `cfbs add` them. ```sh cfbs add custom-policy-1.cf --non-interactive ``` ```text [master 05ee342] Added module './custom-policy-1.cf' 1 file changed, 11 insertions(+) Added module: ./custom-policy-1.cf ``` Let's take a look at what `cfbs add` did. ```sh git log -p -n 1 ``` ```text commit 05ee342d1c58ea62b449614f622cd234f60a413a Author: Nick Anderson Date: Tue Jun 6 08:11:40 2023 -0500 Added module './custom-policy-1.cf' diff --git a/cfbs.json b/cfbs.json index 885c056..5a30ddc 100644 --- a/cfbs.json +++ b/cfbs.json @@ -20,6 +20,17 @@ "run find . -name '*.bak' | xargs rm", "copy ./ ./" ] + }, + { + "name": "./custom-policy-1.cf", + "description": "Local policy file added using cfbs command line", + "tags": ["local"], + "added_by": "cfbs add", + "steps": [ + "copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf", + "policy_files services/cfbs/custom-policy-1.cf", + "bundles custom_policy_1" + ] } ], "git": true ``` `copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf` : This copies `custom-policy-1.cf` to `services/cfbs/custom-policy-1.cf` relative to the root of the built policy. `policy_files services/cfbs/custom-policy-1.cf` : This adds `services/cfbs/custom-policy-1.cf` to `inputs` in `def.json` so that the policy file is parsed. `bundles custom_policy_1` : This adds `custom_policy_1` to `default:def.control_common_bundlesequence_end` in `def.json` so that the bundle will be run as part of the bundlesequence. At least for now, we are looking to replicate the existing policy set while trying to avoid modifications to the MPF, so let's edit `cfbs.json` so that `custom-policy-1.cf` is placed in the root of the built policy. ```sh sed -i 's|services/cfbs/custom-policy-1.cf|custom-policy-1.cf|g' cfbs.json ``` Let's check that the build steps for `custom-policy-1.cf` are as desired. ```sh git diff ``` ```text diff --git a/cfbs.json b/cfbs.json index 5a30ddc..9429cf3 100644 --- a/cfbs.json +++ b/cfbs.json @@ -27,8 +27,8 @@ "tags": ["local"], "added_by": "cfbs add", "steps": [ - "copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf", - "policy_files services/cfbs/custom-policy-1.cf", + "copy ./custom-policy-1.cf custom-policy-1.cf", + "policy_files custom-policy-1.cf", "bundles custom_policy_1" ] } ``` That looks correct, so we can commit those changes to the cfbs project. ```sh git add cfbs.json git commit -m "Fixed custom-policy-1.cf build integration target location, inputs and bundlesequence" ``` ```text [master c36c917] Fixed custom-policy-1.cf build integration target location, inputs and bundlesequence 1 file changed, 2 insertions(+), 2 deletions(-) ``` ##### custom-policy-2.cf {#custom-policy-2-dot-cf} Next we do similar with the other custom policy files, first `services/custom-policy-2.cf`. First we add the local policy to the project. ```sh cfbs add services/custom-policy-2.cf --non-interactive ``` ```text Which bundle should be evaluated (added to bundle sequence)? 1. ./services/custom-policy-2.cf:custom_policy_2 (default) 2. (None) [1/2]: Added module: ./services/custom-policy-2.cf The default commit message is 'Added module './services/custom-policy-2.cf'' - edit it? [yes/y/NO/n] [master 497fdbb] Added module './services/custom-policy-2.cf' 1 file changed, 11 insertions(+) ``` We see that its target location is not as desired. ```sh git log -p -n 1 ``` ```text commit 497fdbb63f2ce48f2c11616049168835221aed37 Author: Nick Anderson Date: Tue Jun 6 08:12:07 2023 -0500 Added module './services/custom-policy-2.cf' diff --git a/cfbs.json b/cfbs.json index 9429cf3..f075886 100644 --- a/cfbs.json +++ b/cfbs.json @@ -31,6 +31,17 @@ "policy_files custom-policy-1.cf", "bundles custom_policy_1" ] + }, + { + "name": "./services/custom-policy-2.cf", + "description": "Local policy file added using cfbs command line", + "tags": ["local"], + "added_by": "cfbs add", + "steps": [ + "copy ./services/custom-policy-2.cf services/cfbs/services/custom-policy-2.cf", + "policy_files services/cfbs/services/custom-policy-2.cf", + "bundles custom_policy_2" + ] } ], "git": true ``` We correct the target location. ```sh sed -i 's|services/cfbs/services/custom-policy-2.cf|services/custom-policy-2.cf|g' cfbs.json ``` We inspect our change. ```sh git diff ``` ```text diff --git a/cfbs.json b/cfbs.json index f075886..e5811ea 100644 --- a/cfbs.json +++ b/cfbs.json @@ -38,8 +38,8 @@ "tags": ["local"], "added_by": "cfbs add", "steps": [ - "copy ./services/custom-policy-2.cf services/cfbs/services/custom-policy-2.cf", - "policy_files services/cfbs/services/custom-policy-2.cf", + "copy ./services/custom-policy-2.cf services/custom-policy-2.cf", + "policy_files services/custom-policy-2.cf", "bundles custom_policy_2" ] } ``` We commit our change fixing the target location for `custom-policy-2.cf` ```sh git add cfbs.json git commit -m "Fixed custom-policy-2.cf build integration target location, inputs and bundlesequence" ``` ```text [master 4868aeb] Fixed custom-policy-2.cf build integration target location, inputs and bundlesequence 1 file changed, 2 insertions(+), 2 deletions(-) ``` ##### custom-policy-3.cf {#custom-policy-3-dot-cf} Finally, we start integrating the missing `custom-policy-3.cf`. ```sh cfbs add services/autorun/custom-policy-3.cf --non-interactive ``` ```text WARNING: Found bundle tagged with autorun in local policy file './services/autorun/custom-policy-3.cf': Note that the autorun tag is ignored when adding local policy files or subdirectories. [master 4894b2f] Added module './services/autorun/custom-policy-3.cf' 1 file changed, 11 insertions(+) Added module: ./services/autorun/custom-policy-3.cf ``` We again see the target is not as desired, but also we see that we don't need to explicitly include this policy file in inputs or the bundlesequence as it leverages autorun for inclusion. ```sh git log -p -n 1 ``` ```text commit 4894b2fd4c47746e4bc734da13d79a6c881b8858 Author: Nick Anderson Date: Tue Jun 6 08:13:04 2023 -0500 Added module './services/autorun/custom-policy-3.cf' diff --git a/cfbs.json b/cfbs.json index e5811ea..f57ad96 100644 --- a/cfbs.json +++ b/cfbs.json @@ -42,6 +42,17 @@ "policy_files services/custom-policy-2.cf", "bundles custom_policy_2" ] + }, + { + "name": "./services/autorun/custom-policy-3.cf", + "description": "Local policy file added using cfbs command line", + "tags": ["local"], + "added_by": "cfbs add", + "steps": [ + "copy ./services/autorun/custom-policy-3.cf services/cfbs/services/autorun/custom-policy-3.cf", + "policy_files services/cfbs/services/autorun/custom-policy-3.cf", + "bundles custom_policy_3" + ] } ], "git": true ``` We adjust the target path and commit the changes to the cfbs project, and we redact the addition of the file to inputs (since it's in the `services/autorun` directory as well as its inclusion to the top level bundlesequence. ```sh # Fix the target path for custom-policy-3 sed -i 's|services/cfbs/services/autorun/custom-policy-3.cf|services/autorun/custom-policy-3.cf|g' cfbs.json # Remove custom-policy-3.cf from inputs and bundlesequnce sed -i '/.*policy_files services\/autorun\/custom-policy-3.cf.*/d' cfbs.json sed -i '/.*bundles custom_policy_3.*/d' cfbs.json # Now we need to correct the JSON since the copy is the only build step and the # last entry of a JSON array must not contain a comma sed -ri 's/("copy.*)custom-policy-3.cf",/\1custom-policy-3.cf"/g' cfbs.json ``` ```sh git diff ``` ```text diff --git a/cfbs.json b/cfbs.json index f57ad96..bae7213 100644 --- a/cfbs.json +++ b/cfbs.json @@ -49,9 +49,7 @@ "tags": ["local"], "added_by": "cfbs add", "steps": [ - "copy ./services/autorun/custom-policy-3.cf services/cfbs/services/autorun/custom-policy-3.cf", - "policy_files services/cfbs/services/autorun/custom-policy-3.cf", - "bundles custom_policy_3" + "copy ./services/autorun/custom-policy-3.cf services/autorun/custom-policy-3.cf" ] } ], ``` ```sh git add cfbs.json git commit -m "Fixed custom-policy-3.cf build integration target location, redacted inputs and bundlesequence" ``` ```text [master ae38672] Fixed custom-policy-3.cf build integration target location, redacted inputs and bundlesequence 1 file changed, 1 insertion(+), 3 deletions(-) ``` Finally we can see that the policy set resulting from `cfbs build` is not missing anything. ```sh cfbs build diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json" ``` ```diff 001 masterfiles @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded) 002 ./custom-policy-1.cf @ local (Copied) 003 ./services/custom-policy-2.cf @ local (Copied) 004 ./services/autorun/custom-policy-3.cf @ local (Copied) 001 masterfiles : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh' 001 masterfiles : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y' 001 masterfiles : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g'' 001 masterfiles : run 'find . -name '*.bak' | xargs rm' 001 masterfiles : copy './' 'masterfiles/' 002 ./custom-policy-1.cf : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf' 002 ./custom-policy-1.cf : policy_files 'custom-policy-1.cf' 002 ./custom-policy-1.cf : bundles 'custom_policy_1' 003 ./services/custom-policy-2.cf : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : policy_files 'services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : bundles 'custom_policy_2' 004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` ### Check for files in cfbs build result that are not present in tarball initialized policy set {#check-for-files-in-cfbs-build-result-that-are-not-present-in-tarball-initalized-policy-set} Let's look at the files in the cfbs built result that are not in the current policy. ```sh diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out" ``` ```text Only in out/masterfiles/cfe_internal/core/watchdog: README.md Only in out/masterfiles/cfe_internal/enterprise/ha: ha_info.json Only in out/masterfiles: .github Only in out/masterfiles/inventory: README.md Only in out/masterfiles/lib: README.md Only in out/masterfiles: LICENSE Only in out/masterfiles/modules: promises Only in out/masterfiles: .no-distrib Only in out/masterfiles/services/autorun: README.md Only in out/masterfiles/templates: README.md ``` We have some _extra_ files, let's take care of those by editing `cfbs.json` and deleting them in the build steps of the masterfiles module. ```sh # Modify masterfiles build steps to remove files not delivered by release tarball sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/core\/watchdog\/README.md",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/enterprise\/ha\/ha_info.json",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .github",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete inventory\/README.md",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete lib\/README.md",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete LICENSE",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete modules\/promises",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .no-distrib",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete services\/autorun\/README.md",\n\1\2/' cfbs.json sed -ri 's/(.*)("copy .\/ .\/")/\1"delete templates\/README.md",\n\1\2/' cfbs.json ``` After making the edits to the masterfiles module build steps we can review our changes. ```sh git diff ``` ```text diff --git a/cfbs.json b/cfbs.json index bae7213..947f990 100644 --- a/cfbs.json +++ b/cfbs.json @@ -18,6 +18,16 @@ "run EXPLICIT_RELEASE=2 ./prepare.sh -y", "run grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\\.18\\.3a.*\\b/3.18.3/g'", "run find . -name '*.bak' | xargs rm", + "delete cfe_internal/core/watchdog/README.md", + "delete cfe_internal/enterprise/ha/ha_info.json", + "delete .github", + "delete inventory/README.md", + "delete lib/README.md", + "delete LICENSE", + "delete modules/promises", + "delete .no-distrib", + "delete services/autorun/README.md", + "delete templates/README.md", "copy ./ ./" ] }, ``` And we can make sure that the result of `cfbs build` is not missing any files. ```sh cfbs build diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out" ``` ```text 001 masterfiles @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded) 002 ./custom-policy-1.cf @ local (Copied) 003 ./services/custom-policy-2.cf @ local (Copied) 004 ./services/autorun/custom-policy-3.cf @ local (Copied) 001 masterfiles : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh' 001 masterfiles : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y' 001 masterfiles : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g'' 001 masterfiles : run 'find . -name '*.bak' | xargs rm' 001 masterfiles : delete 'cfe_internal/core/watchdog/README.md' 001 masterfiles : delete 'cfe_internal/enterprise/ha/ha_info.json' 001 masterfiles : delete '.github' 001 masterfiles : delete 'inventory/README.md' 001 masterfiles : delete 'lib/README.md' 001 masterfiles : delete 'LICENSE' 001 masterfiles : delete 'modules/promises' 001 masterfiles : delete '.no-distrib' 001 masterfiles : delete 'services/autorun/README.md' 001 masterfiles : delete 'templates/README.md' 001 masterfiles : copy './' 'masterfiles/' 002 ./custom-policy-1.cf : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf' 002 ./custom-policy-1.cf : policy_files 'custom-policy-1.cf' 002 ./custom-policy-1.cf : bundles 'custom_policy_1' 003 ./services/custom-policy-2.cf : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : policy_files 'services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : bundles 'custom_policy_2' 004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` ### Review and resolve remaining differences {#review-and-resolve-remaining-differences} Now all that remains is reviewing any other differences between our traditionally managed policy with the result of `cfbs build`. ```sh diff --exclude=out --recursive --unified . out/masterfiles ``` ```text Only in .: cfbs.json diff '--exclude=out' --recursive --unified ./def.json out/masterfiles/def.json --- ./def.json 2023-06-06 08:09:22.719359634 -0500 +++ out/masterfiles/def.json 2023-06-06 08:13:48.942006067 -0500 @@ -1,11 +1,6 @@ { - "inputs": [ "services/custom-policy-2.cf" ], - - "classes": { - "services_autorun": [ "any::" ] - }, - + "inputs": ["custom-policy-1.cf", "services/custom-policy-2.cf"], "vars": { - "control_common_bundlesequence_end": [ "custom_policy_2" ] + "control_common_bundlesequence_end": ["custom_policy_1", "custom_policy_2"] } } Only in .: .git Only in .: .gitignore diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf --- ./promises.cf 2023-06-06 08:09:17.303386890 -0500 +++ out/masterfiles/promises.cf 2023-06-06 08:13:48.738007113 -0500 @@ -59,7 +59,6 @@ main, @(cfengine_enterprise_hub_ha.management_bundles), @(def.bundlesequence_end), -custom_policy_1, }; @@ -86,7 +85,6 @@ @(services_autorun.inputs), "services/main.cf", -"custom-policy-1.cf", }; version => "CFEngine Promises.cf 3.18.3"; diff '--exclude=out' --recursive --unified ./services/main.cf out/masterfiles/services/main.cf --- ./services/main.cf 2023-06-06 08:09:25.299346646 -0500 +++ out/masterfiles/services/main.cf 2023-06-06 08:13:48.742007092 -0500 @@ -10,6 +10,5 @@ { # Activate your custom policies here -"Calling custom_policy_2 from main" usebundle => custom_policy_2; } ``` The diff above highlights differences in `def.json`, `.git`, `.gitignore`, `promises.cf`, and `services/main.cf`. - The difference in the `inputs` key and the `control_common_bundlesequence_end` array in the `vars` key is expected as we migrated a modification including `custom-policy-1.cf` in inputs and bundlesquence from `promises.cf` to `def.json` via the build steps for that module, removing an unnecessary modification to the vendored files which is also reflected in the differences of `promises.cf` - We are lacking the class definition for `services_autorun` to enable the autorun service. - We are lacking the modification to `services/main.cf` explicitly running the `custom_policy_2` bundle. We can address `services/main.cf` as we have addressed other files deployed from our traditionally managed policy set, first adding the file. ```sh cfbs add services/main.cf --non-interactive ``` ```text [master 482f67c] Added module './services/main.cf' 1 file changed, 21 insertions(+) Added module: ./services/main.cf ``` Then adjusting the build steps as necessary. ```sh # Overwrite the vendored services/main sed -i 's|services/cfbs/services/main.cf|services/main.cf|g' cfbs.json # Remove build steps for services/main.cf for inputs and bundlesequnce as it's # already inclided by default in the framework as a vendored file sed -i '/.*policy_files services\/main.*/d' cfbs.json sed -i '/.*bundles main.*/d' cfbs.json # Now we need to correct the JSON since the copy is the only build step and the # last entry of a JSON array must not contain a comma sed -ri 's/("copy.*)main.cf",/\1main.cf"/g' cfbs.json ``` We can address enabling autorun by simply adding in the autorun module. ```sh cfbs add autorun --non-interactive ``` ```text Added module: autorun The default commit message is 'Added module 'autorun'' - edit it? [yes/y/NO/n] [master e5f9c9a] Added module 'autorun' 1 file changed, 13 insertions(+), 5 deletions(-) ``` This leaves only the expected differences between the traditionally managed policy set and the cfbs built policy set. ```sh cfbs build diff --exclude=out --recursive --unified . out/masterfiles ``` ```text 001 masterfiles @ c92106b72ac9a9f12e412df7ecba1ea22bcb373a (Downloaded) 002 ./custom-policy-1.cf @ local (Copied) 003 ./services/custom-policy-2.cf @ local (Copied) 004 ./services/autorun/custom-policy-3.cf @ local (Copied) 005 ./services/main.cf @ local (Copied) 006 autorun @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 001 masterfiles : run 'sed -i.bak 's|prefix="/var/cfengine/"|prefix="/var/cfengine"|g' prepare.sh' 001 masterfiles : run 'EXPLICIT_RELEASE=2 ./prepare.sh -y' 001 masterfiles : run 'grep --files-with-matches -R 3.18.3 . | xargs -n1 sed -i.bak 's/3\.18\.3a.*\b/3.18.3/g'' 001 masterfiles : run 'find . -name '*.bak' | xargs rm' 001 masterfiles : delete 'cfe_internal/core/watchdog/README.md' 001 masterfiles : delete 'cfe_internal/enterprise/ha/ha_info.json' 001 masterfiles : delete '.github' 001 masterfiles : delete 'inventory/README.md' 001 masterfiles : delete 'lib/README.md' 001 masterfiles : delete 'LICENSE' 001 masterfiles : delete 'modules/promises' 001 masterfiles : delete '.no-distrib' 001 masterfiles : delete 'services/autorun/README.md' 001 masterfiles : delete 'templates/README.md' 001 masterfiles : copy './' 'masterfiles/' 002 ./custom-policy-1.cf : copy './custom-policy-1.cf' 'masterfiles/custom-policy-1.cf' 002 ./custom-policy-1.cf : policy_files 'custom-policy-1.cf' 002 ./custom-policy-1.cf : bundles 'custom_policy_1' 003 ./services/custom-policy-2.cf : copy './services/custom-policy-2.cf' 'masterfiles/services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : policy_files 'services/custom-policy-2.cf' 003 ./services/custom-policy-2.cf : bundles 'custom_policy_2' 004 ./services/autorun/custom-policy-3.cf : copy './services/autorun/custom-policy-3.cf' 'masterfiles/services/autorun/custom-policy-3.cf' 005 ./services/main.cf : copy './services/main.cf' 'masterfiles/services/main.cf' 006 autorun : json 'def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy Only in .: cfbs.json diff '--exclude=out' --recursive --unified ./def.json out/masterfiles/def.json --- ./def.json 2023-06-06 08:09:22.719359634 -0500 +++ out/masterfiles/def.json 2023-06-06 08:14:27.481808290 -0500 @@ -1,11 +1,7 @@ { - "inputs": [ "services/custom-policy-2.cf" ], - - "classes": { - "services_autorun": [ "any::" ] - }, - + "inputs": ["custom-policy-1.cf", "services/custom-policy-2.cf"], "vars": { - "control_common_bundlesequence_end": [ "custom_policy_2" ] - } + "control_common_bundlesequence_end": ["custom_policy_1", "custom_policy_2"] + }, + "classes": { "services_autorun": ["any"] } } Only in .: .git Only in .: .gitignore diff '--exclude=out' --recursive --unified ./promises.cf out/masterfiles/promises.cf --- ./promises.cf 2023-06-06 08:09:17.303386890 -0500 +++ out/masterfiles/promises.cf 2023-06-06 08:14:27.221809626 -0500 @@ -59,7 +59,6 @@ main, @(cfengine_enterprise_hub_ha.management_bundles), @(def.bundlesequence_end), -custom_policy_1, }; @@ -86,7 +85,6 @@ @(services_autorun.inputs), "services/main.cf", -"custom-policy-1.cf", }; version => "CFEngine Promises.cf 3.18.3"; ``` ## Moving ahead {#moving-ahead} At this point we have completed the process of bringing our policy set under `cfbs` management. Look around build.cfengine.com for modules that interest you, try adding and removing modules, you can even try `cfbs update masterfiles` to see how the policy framework upgrade has been simplified. If you have ideas for improving the feature set or behavior of `cfbs` please open a ticket on the [issue tracker](https://northerntech.atlassian.net/jira/software/c/projects/CFE). [^fn:1]: The Masterfiles Policy Framework (MPF) is the default policy set which contains the standard library and provides a framework for custom policy sets to be built on top. [^fn:2]: `cfbs` is the command line tool used to manage a policy set and interact with the CFEngine Build System (build.cfengine.com). It's available via the Python Package Index (PyPi). [^fn:3]: The CFEngine Build System provides tooling and a catalog of modules to simplify the process of building and maintaining an infrastructure with CFEngine. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 25 - Migrating to cfbs" url: /blog/2023/the-agent-is-in-episode-25/index.md date: "2023-05-25" -------------------------------------------------------------------------------- Been a CFEngine user for a while? Have you migrated to a `cfbs` managed policy set yet? Live from the Northern.tech Summit in Castell de Sant Mori[^fn:1]! Cody, Craig and Nick walk through the process of migrating a policy set to `cfbs` management. Go through the process yourself following the detailed [Migrating to cfbs blog post](/blog/2023/migrating-to-cfbs/). ## Video {#video} The video recording is available on YouTube: {{< youtube Q6Yk374ua70 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/). - Find [all Episodes](/webinars) listed on our webinars page. [^fn:1]: Northern.tech employees gather for an annual Summit. This year it was held in [Castell de Sant Mori](https://castellsantmori.com/wordpress/index.php/en/home-eng/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 24 - Pretty printer (cffmt) demo with Miek Gieben" url: /blog/2023/the-agent-is-in-episode-24/index.md date: "2023-04-27" -------------------------------------------------------------------------------- Tired of hand crafting policy and arguing with people about spacing and alignment? Longing for regularity and easier scanning of your policy no matter who wrote it? Cody, Craig and Nick wrap up the second year of The agent is in with Miek Gieben, CFEngine Community user and author of `cffmt`, a formatter written in go for CFEngine policy files. Check out the discussion about opinionated formatting, possible future developments and other tooling to improve quality of life as a CFEngineer. ## Video {#video} The video recording is available on YouTube: {{< youtube C-8LDCPJLmU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ Miek, [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - Miek Gieben on [LinkedIn](https://www.linkedin.com/in/miek-gieben-932aaa20a/), [GitHub](https://github.com/miekg), [Mastodon](https://mastodon.cloud/@miek), [Twitter](https://twitter.com/miekg) - [cffmt](https://github.com/miekg/cf) - A CFEngine pretty printer on github - [gitopper](https://github.com/miekg/gitopper) - Git Ops for non-Kubernetes folks on github - [testing.cf](https://docs.cfengine.com/docs/3.21/reference-masterfiles-policy-framework-lib-testing.html#top) - TAP and JUnit output for policy testing in Masterfiles Policy Framework Reference documentation - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.18.4 and 3.21.1 released" url: /blog/2023/cfengine-3-18-4-and-3-21-1-released/index.md date: "2023-04-24" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.18.4 and 3.21.1\! These releases only contain security fixes for our recently discovered vulnerability; [CVE-2023-26560](./cve-2023-26560). ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.18.4 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.4 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.4 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.21.1 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.1 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.1 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.21.1 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.18.4 list ``` ```command cf-remote --version 3.21.1 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "CVE-2023-26560 - Unauthorized access to system files through scheduled reports" url: /blog/2023/cve-2023-26560/index.md date: "2023-04-24" -------------------------------------------------------------------------------- We are writing to inform you about a security issue that was discovered in CFEngine 3.6.0 and later versions. Our development team found the vulnerability relating to inadequate access control / unauthorized access to system files. MITRE assigned the CVE identifier CVE-2023-26560. We have no indications that this vulnerability has been used or known outside of the CFEngine development team. ## Explanation The issue is that Mission Portal users can access certain files through scheduled reports, as these reports are run with elevated privileges, without additional checks to limit what can be queried. Within SQL queries (in PostgreSQL) you can use functions like `pg_read_binary_file` to access files on the file system. This issue is limited to scheduled reports, due to the different context where those queries are run. ## Impact Using scheduled reports and PostgreSQL functions like `pg_read_binary_file`, Mission Portal users can read files they should not have access to, such as database contents from `/var/cfengine/state/pg/data/`, logs and other files which the `cfpostgres` system user has access to. It should be noted that the user needs access to scheduled reports and this is not enabled by default when creating new users. The concern is around non-admin users which have been granted permission to use scheduled reports through the **RBAC settings**. The database files could allow someone to restore the database, and among other things, get access to: - Information about hosts they should not see due to RBAC filtering. - The LDAP username and password used in settings to set up LDAP authentication (if this feature is used). - Usernames and password hashes for all (non LDAP) Mission Portal users. ## Remediation We recommend installing one of the new versions of the CFEngine hub packages listed [on our download pages](https://cfengine.com/downloads/cfengine-enterprise/), as well as rotating users' passwords and CFEngine secrets. The newly released CFEngine 3.18.4 and 3.21.1 versions contain only fixes for the mentioned issue, and thus should be non-problematic to install on top of CFEngine 3.18.3 or 3.21.1. If you have LDAP configured, please also rotate the LDAP user password you have in the **Authentication settings**. For your information, we do not store Mission Portal users' raw passwords, they are hashed. Note that since this issue only affects the CFEngine Enterprise hub, installing the hub package is enough to remediate the issue. Pay attention to the upgrade instructions, especially if you are upgrading from a version older than 3.18.3, as you should first upgrade your policy set. ### Workaround / hotfix In the case where updating hub packages is not possible, you can use this hotfix. The hotfix restricts access to the affected files and rotates secrets. https://cfengine.com/blog/2023/hotfix-CVE-2023-26560-v1.0.0.cf However, we would like to make you aware that after applying the hotfix, the root database user will no longer be able to make schema changes to the database. This is because the hotfix removes the superuser role from it. But you can still manage the database under the `cfpostgres` user. -------------------------------------------------------------------------------- title: "Improved software compliance with packages-allowlist" url: /blog/2023/improved-software-compliance-with-packages-allowlist/index.md date: "2023-04-06" -------------------------------------------------------------------------------- Having a list of software that is allowed to be installed on a host is a strategy to prevent and fix security gaps and maintain compliance with operational guidelines. This zero-trust methodology ensures that only explicitly permitted applications are allowed to be present on a host unlike package block-listing which enumerates an explicit list of software that is not allowed to be present. In fact, with a software allow-list, you are essentially block-listing everything except the software you allow. The new [packages-allowlist module on CFEngine Build](https://build.cfengine.com/modules/packages-allowlist/) can help you facilitate this. The `packages-allowlist` module targets the platform's default package manager and provides an inventory of packages installed that are _not_ in the allow-list (as defined by the `packages_allowlist:state.allowed` list variable) by default. {{< figure src="packages-allowlist-module-input.png" alt="packages-allowlist module with input data configuring allowed list of packages" link="packages-allowlist-module-input.png" >}} Any packages found installed that are not in the allow-list are inventoried as `Packages installed not in allow list`. {{< figure src="inventory.png" alt="Inventory showing a package not in the allowed list of software packages" link="inventory.png" >}} This inventory can be used to build alerts and compliance checks to quickly notify stakeholders of non-compliance. {{< figure src="alert.png" alt="My inventory screenshot" link="alert.png" >}} By default `packages_allowlist:state.enforcement` is set to `disabled`. While `packages_allowlist:state.enforcement` is _not_ `enabled` warnings are emitted for each package that is not found in `packages_allowlist:state.allowed`. {{< figure src="console-info-warning.png" alt="Screenshot showing behavior when package not in allowlist is found and enforcement is not enabled" link="console-info-warning.png" >}} When the variable `packages_allowlist:state.enforcement` has a value of `enabled` these packages will be automatically removed. {{< figure src="console-info-removed.png" alt="Screenshot showing behavior when package not in allowlist is found and enforcement is enabled" link="console-info-removed.png" >}} {{< figure src="changes.png" alt="Screenshot showing Changes report highlighting package removal in Mission Portal" link="changes.png" >}} Try out the module and let us know what you think on [GitHub Discussions](https://github.com/cfengine/core/discussions) or on the [help mailing list](https://groups.google.com/g/help-cfengine). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 23 - Detecting Previously Hidden Malware With Invary & CFEngine" url: /blog/2023/the-agent-is-in-episode-23/index.md date: "2023-03-30" -------------------------------------------------------------------------------- Can you trust the integrity of your base operating system runtime? {{< figure src="CFEngine-x-Invary.png" >}} Jason Rogers and Dr. Wesley Peck of [Invary](https://invary.com) join Cody, Craig and Nick to chat about their Runtime Integrity technology. They discuss the challenges of Trust, Information Technology Knowledge Management, and how Invary fits in the SecOps, Systems Automation, Security and Compliance landscape. Nick shares an example of an early integration between CFEngine and the Invary RISe agent[^fn:1] with reporting in Mission Portal and talks about the different ways to approach integration. - [00:37](https://youtu.be/soiDcqiqX68?t=37) - Introductions - [03:39](https://youtu.be/soiDcqiqX68?t=219) - What is Invary?[^fn:2] - [04:49](https://youtu.be/soiDcqiqX68?t=289) - Technology Origins - [06:42](https://youtu.be/soiDcqiqX68?t=402) - How Invary works[^fn:3] - [16:01](https://youtu.be/soiDcqiqX68?t=961) - Other technology applications - [33:32](https://youtu.be/soiDcqiqX68?t=2012) - Integrating with Invary ## Video {#video} The video recording is available on YouTube: {{< youtube soiDcqiqX68 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Jason](https://www.linkedin.com/in/jasonlrogers/), [Wesley](https://www.linkedin.com/in/wesley-peck-4589a75a/), [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) [^fn:1]: Try Invary's Runtime Integrity Score (RISe) for free to detect hidden malware in your OS. [^fn:2]: Runtime Integrity Measurement, Kernel Mode Rootkits & how the Drovorub rootkit works. [^fn:3]: Ongoing, monthslong malware campaign that's targeting and stealing data from pharmaceutical, IT services and consulting firms through their internet routers. -------------------------------------------------------------------------------- title: "How to properly check if files are readable" url: /blog/2023/check-if-file-is-readable/index.md date: "2023-03-28" -------------------------------------------------------------------------------- Opening and reading files may cause your program to block indefinitely, which is a major problem for configuration management tools like CFEngine. What if we make a program that can check if these file operations would block indefinitely, before using them? Unfortunately our good friend Alan Turing proved that such programs are not theoretically possible. But if we let theory be theory; we can in practice make some compromises by changing the definition of an unreadable file, to be a file that could not be read in N number of seconds. The first possible solution that comes to mind, is creating a program that opens a file using the `O_NONBLOCK` flag. Followed by using [`poll(2)`](https://man7.org/linux/man-pages/man2/poll.2.html) to wait a limited amount of time for the file descriptor to become ready for reading. But the man page states that the file will be opened in a "non-blocking" manner whenever possible, and that applications should not depend upon the blocking behavior when specifying this flag for regular files and block devices. Thus this option is not ideal. Instead, we can try performing the file operations in a separate thread, or in a forked process, waiting a limited amount of time for the thread/process to terminate. Both suggestions would work; but since fork creation is more expensive than thread creation, and since the sharing of information between processes is both more difficult and expensive, we chose to solve this problem using threads. ## Solution We'll start off by declaring a function that takes two parameters. A filename for specifying the file that we want to check. And a timeout interval for specifying how long we're willing to wait. The function will return an integer, indicating whether the read was successful, failed, or an unexpected error occurred. ```c enum { READABLE_SUCCESS = 0, // Successfully read file. READABLE_FAILURE, // Failed to read file. READABLE_ERROR, // Unexpected error occurred. }; /** * @brief Determine if a file is readable. * @param[in] filename path to file. * @param[in] timeout number of milliseconds to wait. * @return READABLE_SUCCESS on successful read, READABLE_FAILUIRE on failed * read, and READABLE_ERROR in case of unexpected error. */ int Readable(const char *filename, long timeout); ``` Our plan is to use the [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html) function in order to provide the timeout functionality. To do this, we'll need to create a mutex for synchronizing the threads, and a condition to signal a state change. We'll also need to calculate an expiration time for when to give up waiting. ```c static pthread_mutex_t MUTEX = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t CONDITION = PTHREAD_COND_INITIALIZER; int Readable(const char *const filename, const long timeout) { // Calculate absolute expiration time from timeout interval in milliseconds. struct timespec ts = {0}; if (clock_gettime(CLOCK_REALTIME, &ts) != 0) { perror("Failed to get system's real time"); return READABLE_ERROR; } ts.tv_sec += ((timeout * 1000000) + ts.tv_nsec) / 1000000000; ts.tv_nsec = ((timeout * 1000000) + ts.tv_nsec) % 1000000000; } ``` Both the mutex and condition variables are statically initialized using the `PTHREAD_MUTEX_INITIALIZER`- and `PTHREAD_COND_INITIALIZER` macros respectively. To initialize them dynamically, you'll need to use the [`pthread_mutex_init(3)`](https://man7.org/linux/man-pages/man3/pthread_mutex_init.3p.html) and [`pthread_cond_init(3)`](https://www.man7.org/linux/man-pages/man3/pthread_cond_init.3p.html) functions from the pthread library, passing them to the thread routine in a struct along with the filename. Our program takes a timeout interval _(in milliseconds)_ as an argument. But since [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html) expects an absolute expiration time, we'll need to do some calculations. We'll get the system's real time using [`clock_gettime(3)`](https://man7.org/linux/man-pages/man3/clock_gettime.3.html). Then we'll convert our timeout interval argument into seconds and nanoseconds, before adding it to the timespec struct. Next we'll lock the mutex with [`pthread_mutex_lock(3)`](https://man7.org/linux/man-pages/man3/pthread_mutex_lock.3p.html) before spawning the thread with [`pthread_create(3)`](https://man7.org/linux/man-pages/man3/pthread_create.3.html), giving the main thread time to call [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html) before the spawned thread goes about its business. ```c int Readable(const char *const filename, const long timeout) { ... int ret = pthread_mutex_lock(&MUTEX); if (ret != 0) { fprintf(stderr, "Failed to lock mutex: %s\n", strerror(ret)); return READABLE_ERROR; } pthread_t thread; ret = pthread_create(&thread, NULL, &TryRead, (void *)filename); if (ret != 0) { fprintf(stderr, "Failed to create thread: %s\n", strerror(ret)); return READABLE_ERROR; } } ``` The spawned thread will execute the - soon to be defined - `TryRead()` routine, where it will try to acquire the mutex we just locked. The call to [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html) causes the main thread to release the mutex, allowing the spawned thread to acquire it and carry on. ```c int Readable(const char *const filename, const long timeout) { ... ret = pthread_cond_timedwait(&CONDITION, &MUTEX, &ts); const bool time_expired = (ret == ETIMEDOUT); if (time_expired) { ret = pthread_cancel(thread); if (ret != 0) { fprintf(stderr, "Failed to cancel thread: %s\n", strerror(ret)); return READABLE_ERROR; } } else if (ret != 0) { fprintf(stderr, "Failed to wait for condition: %s\n", strerror(ret)); return READABLE_ERROR; } void *status; ret = pthread_join(thread, &status); if (ret != 0) { fprintf(stderr, "Failed to join thread: %s\n", strerror(ret)); return READABLE_ERROR; } } ``` Once we return from [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html) we determine whether or not the spawned thread timed out. If the thread did not finish in time, we'll need to cancel its execution before joining with it; preventing the main thread from getting blocked. There is a gotcha when it comes to using [`pthread_cancel(3)`](https://man7.org/linux/man-pages/man3/pthread_cancel.3.html); i.e. precisely what happens to a thread when it receives a cancellation request depends on the thread's cancellation state and type. If the cancellation state is `PTHREAD_CANCEL_DISABLE`; any cancellation request will remain pending until its state is changed to `PTHREAD_CANCEL_ENABLE`. If cancellation type is `PTHREAD_CANCEL_DEFFERED`, any cancellation request will remain pending until a cancellation point is reached. This is opposed to the cancellation type `PTHREAD_CANCEL_ASYNCHRONOUS`, in which the thread can be canceled at any time. Cancellation state and type can be set using [`pthread_setcancelstate(3)`](https://man7.org/linux/man-pages/man3/pthread_setcancelstate.3.html) and [`pthread_setcanceltype(3)`](https://man7.org/linux/man-pages/man3/pthread_setcanceltype.3p.html) respectively. We'll use cancellation state `PTHREAD_CANCEL_ENABLE` to make sure the thread can be canceled. And we'll use the cancellation type `PTHREAD_CANCEL_DEFFERED` since [`open(2)`](https://man7.org/linux/man-pages/man2/openat.2.html), [`read(2)`](https://man7.org/linux/man-pages/man2/read.2.html) and [`close(2)`](https://man7.org/linux/man-pages/man2/close.2.html) are all cancellation points. Luckily these are the default cancellation state and type, so there is no need to change them. A list of functions that are cancellation points can be found in [`pthreads(7)`](https://man7.org/linux/man-pages/man7/pthreads.7.html) man page. Please note that setting the cancellation type to `PTHREAD_CANCEL_ASYNCHRONOUS` is rarely useful, and can lead to some serious challenges when it comes to managing resources. Since a thread can be canceled at any time; the cancellation can happen before, after, or even during the locking of a mutex, or a memory allocation. Thus there is no way to know the exact state when the cancellation takes place; making it impossible to safely release these resources in a cleanup handler. So if you're planning to use this cancellation type, you better think twice. With the spawned thread safely joined, we need to clean up after ourselves by releasing the mutex that was reacquired when returning from [`pthread_cond_timedwait(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_timedwait.3p.html), before we can return the code indicating whether or not we were able to successfully read the file. ```c int Readable(const char *const filename, const long timeout) { ... ret = pthread_mutex_unlock(&MUTEX); if (ret != 0) { fprintf(stderr, "Failed to unlock mutex: %s\n", strerror(ret)); return READABLE_ERROR; } return (time_expired) ? READABLE_FAILURE : (int)status; } ``` We'll program the `TryRead()` function - to be executed by the spawned thread - to return whether or not the file was readable through its exit status, which we'll obtain in the `status` variable when joining the thread. Please note that we don't return the value of the `status` variable in case of timeout. As it may contain the value `PTHREAD_CANCELED`, if the thread was indeed canceled. Instead we return `READABLE_FAILURE` as we consider the file unreadable anyway, in this case. The `TryRead()` function will be defined to take a void pointer, which will point to the name of the respective file. The function will return a void pointer indicating success, failure or error. ```c static void *TryRead(void *data) { int ret = pthread_mutex_lock(&MUTEX); if (ret != 0) { fprintf(stderr, "Failed to lock mutex: %s\n", strerror(ret)); return (void *)READABLE_ERROR; } int fd = -1; bool success; pthread_cleanup_push(&CleanupHandler, &fd); ret = pthread_mutex_unlock(&MUTEX); if (ret != 0) { fprintf(stderr, "Failed to unlock mutex: %s\n", strerror(ret)); return (void *)READABLE_ERROR; } } ``` We'll synchronize with the main thread by acquiring the mutex using [`pthread_mutex_lock(3)`](https://man7.org/linux/man-pages/man3/pthread_mutex_lock.3p.html). Once synchronized, we'll push a cleanup handler to the thread's cleanup handler stack using [`pthread_cleanup_push(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_push.3.html). This way we can assume that the file we're about to open will be closed in the case of receiving a cancellation request. With the cleanup handler pushed to the stack, we release the mutex, so that the main thread is allowed to timeout. Some UNIX implementations - including Linux - implement [`pthread_cleanup_push(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_push.3.html) and [`pthread_cleanup_pop(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_pop.3p.html) as macros that expand into statement sequences including the opening (`{`) and closing (`}`) of a scope. Thus every use of [`pthread_cleanup_push(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_push.3.html) must be paired with a corresponding [`pthread_cleanup_pop(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_pop.3p.html), within the same lexical scope. This also means that any variables declared within this scope, cannot be accessed outside it. Thus we'll have to declare our `success` variable just before [`pthread_cleanup_push(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_push.3.html) so that we can still use it after the call to [`pthread_cleanup_pop(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_pop.3p.html). The cleanup handler passed to [`pthread_cleanup_push(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_push.3.html) is simply defined to close the file descriptor using [`close(2)`](https://man7.org/linux/man-pages/man2/close.2.html) if it has a valid value (i.e. greater or equal to zero). ```c static void CleanupHandler(void *data) { const int fd = *((int *)data); if (fd >= 0) { close(fd); } } ``` Continuing with our thread routine, we'll attempt to open the respective file with [`open(2)`](https://man7.org/linux/man-pages/man2/openat.2.html), and we'll try to read 1 Byte with [`read(2)`](https://man7.org/linux/man-pages/man2/read.2.html). If the opening of the file succeeds, and if we're able to read 1 Byte or reach end-of-file, we'll consider the file readable. Last, but not least, we'll pop off the cleanup handler passing `1` as the argument to [`pthread_cleanup_pop(3)`](https://man7.org/linux/man-pages/man3/pthread_cleanup_pop.3p.html) in order to also execute the cleanup handler. ```c static void *TryRead(void *data) { ... char buffer[1]; const char *const filename = data; fd = open(filename, O_RDONLY); success = ((fd > 0) && (read(fd, buffer, sizeof(buffer)) >= 0)); pthread_cleanup_pop(1); } ``` The last remaining piece of the thread routine is to signal the main thread using [`pthread_cond_signal(3)`](https://man7.org/linux/man-pages/man3/pthread_cond_signal.3p.html), before returning code indicating success, failure or error. ```c static void *TryRead(void *data) { ... ret = pthread_cond_signal(&CONDITION); if (ret != 0) { fprintf(stderr, "Failed to signal waiting thread: %s\n", strerror(ret)); return (void *)READABLE_ERROR; } return (void *)((success) ? READABLE_SUCCESS : READABLE_FAILURE); } ``` In order for our program to execute, we'll also create a main function that parses the command line arguments, calls the `Readable()` function, and outputs the results. ```c int main(int argc, char *argv[]) { time_t timeout = 10; int opt; char *endptr; while ((opt = getopt(argc, argv, "+ht:")) != -1) { switch (opt) { case 'h': printf("%s [-h] [-t TIMEOUT] [FILENAME ...]\n", argv[0]); return EXIT_SUCCESS; case 't': errno = 0; timeout = strtol(optarg, &endptr, 10); if (errno != 0 || endptr == optarg) { fprintf(stderr, "Failed to parse integer '%s'\n", optarg); return EXIT_FAILURE; } break; default: return EXIT_FAILURE; } } for (int i = optind; i < argc; i++) { const char *const filename = argv[i]; switch (Readable(filename, timeout)) { case READABLE_SUCCESS: printf("File '%s' is readable.\n", filename); break; case READABLE_FAILURE: printf("File '%s' is not readable.\n", filename); break; default: fprintf(stderr, "An unexpected error occured while checking file '%s'\n", filename); return EXIT_FAILURE; } } return EXIT_SUCCESS; } ``` ## Testing The complete source code is available in the GitHub repository: https://github.com/larsewi/isreadable Feel free to clone it and test it out. ``` $ git clone git@github.com:larsewi/isreadable.git Cloning into 'isreadable'... remote: Enumerating objects: 34, done. remote: Counting objects: 100% (34/34), done. remote: Compressing objects: 100% (26/26), done. remote: Total 34 (delta 15), reused 18 (delta 8), pack-reused 0 Receiving objects: 100% (34/34), 12.23 KiB | 6.11 MiB/s, done. Resolving deltas: 100% (15/15), done. $ make gcc -g -Wall -Wextra -c main.c -o main.o gcc -g -Wall -Wextra -Wno-int-to-void-pointer-cast -Wno-void-pointer-to-int-cast -c readable.c -o readable.o gcc main.o readable.o -o readable -pthread $ ./readable -t100 *.c *.h not-a-file File 'main.c' is readable. File 'readable.c' is readable. File 'readable.h' is readable. File 'not-a-file' is not readable. ``` We can also check that the timeout functionality works, by adding a `sleep()` just before or after the file operations, as [`sleep(3)`](https://man7.org/linux/man-pages/man3/sleep.3.html) is also a cancellation point according to [`pthreads(7)`](https://man7.org/linux/man-pages/man7/pthreads.7.html) man page. ```c static void *TryRead(void *data) { ... fd = open(filename, O_RDONLY); success = ((fd > 0) && (read(fd, buffer, sizeof(buffer)) >= 0)); sleep(1); ... } ``` As you can see from the output below, the program times out after approximately 900 milliseconds, deeming the file unreadable. Upping the timeout interval deems the file readable. ``` $ time ./readable -t900 main.c File 'main.c' is not readable. ./readable -t900 main.c 0.00s user 0.00s system 0% cpu 0.907 total $ time ./readable -t1100 main.c File 'main.c' is readable. ./readable -t1100 main.c 0.00s user 0.00s system 0% cpu 1.007 total ``` ## Conclusion We've discussed how to properly check if a file is readable, without potentially blocking indefinitely. Although this program may not be perfect, in that other system calls used in the implementation may block, it definitely reduces the probability. Another point that can be made, is that even though this program correctly concludes that the file is readable, it may not be readable immediately after this decision is made. Thus checking the file before reading it, gives no definitive guarantees. If you have any questions in regards to this blogpost, or any feedback on how this program could be improved. Please feel free to hit us up on [GitHub Discussions](https://github.com/cfengine/core/discussions), we are eager to hear from you. ## Sources - Kerrisk, M. (1961). The Linux Programming Interface. - Kerrisk, M. et al. (2023). Linux man pages online. -------------------------------------------------------------------------------- title: "Conference recap: CfgMgmtCamp 2023" url: /blog/2023/cfgmgmtcamp/index.md date: "2023-02-28" -------------------------------------------------------------------------------- In February, our team attended both FOSDEM and CfgMgmtCamp (Configuration Management Camp) in Belgium. At CfgMgmtCamp we held several talks, and we'll upload some of the recordings. The first one is already available: {{< youtube 6q00Eyf3E4Y >}} In the list below, you will find the abstracts, slides, and videos. As more videos become available we will update the list. ## Conference talks - State of the CFEngine ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/GZKQV8/), [slides](/slides/cfgmgmtcamp-2023-state-of-the-cfengine.pdf), video coming) - CFEngine on odd platforms ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/AQVDZV/), [slides](/slides/cfgmgmtcamp-2023-cfengine-on-odd-platforms.pdf), video coming) - Using Ansible modules in CFEngine ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/R9RGZ3/), [slides](/slides/cfgmgmtcamp-2023-using-ansible-modules-in-cfengine.pdf), video coming) - An introduction to CFEngine Build ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/3CSJWW/), [slides](/slides/cfgmgmtcamp-2023-introduction-to-cfengine-build.pdf), [video](/videos/?videoID=6q00Eyf3E4Y)) - Writing Modules for CFEngine Build ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/3QKHK3/), [slides](/slides/cfgmgmtcamp-2023-writing-modules-for-cfengine-build.pdf), video coming) - Personal Configuration Management with CFEngine ([abstract](https://cfp.cfgmgmtcamp.org/2023/talk/FAYZEG/), [slides](/slides/cfgmgmtcamp-2023-personal-configuration-management-with-cfengine.pdf), video coming) ## Connecting and meeting up Thank you to everyone we met up with during these 2 events! If you'd like the opportunity to have a chat or share a meal with us when we attend events, follow us on [LinkedIn](https://www.linkedin.com/company/northern.tech) and [Twitter](https://twitter.com/cfengine), or reach out to us directly via [cfengine.com/contact](/contact). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 22 - Hackathon: Termux Services" url: /blog/2023/the-agent-is-in-episode-22-hackathon-termux-services/index.md date: "2023-02-23" -------------------------------------------------------------------------------- Have a burning desire to run `sshd` or another service on your VR headset? Cody, Craig and Nick do time-boxed live hackathon working on developing CFEngine services promise type support for [Termux](https://termux.dev/). Watch Nick and Craig race to implement basic services support before the timer buzzes. ## Video {#video} The video recording is available on YouTube: {{< youtube l-TtILO2y48 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Termux services policy {#termux-services-policy} Here is the snippet of policy developed during the episode. ```cfengine3 bundle agent __main__ { services: "sshd" service_policy => "stopped", service_method => termux_services; } bundle agent termux_services(service, policy) # @brief Manage service state in Termux # @param service The name of the service to manage, e.g. `sshd` # @param policy The desired state of the service (`running`|`stopped`) { classes: "service_running" expression => regcmp( "^run:.*", execresult("sv status ${service}", "useshell") ); "service_stopped" expression => regcmp( "^down:.*", execresult("sv status ${service}", "useshell") ); "policy_running" expression => strcmp("${policy}", "running"); "policy_stopped" expression => strcmp("${policy}", "stopped"); commands: service_stopped.policy_running:: "sv up ${service}" contain => useshell; service_running.policy_stopped:: "sv down ${service}" contain => useshell; } body service_method termux_services { service_bundle => default:termux_services( ${this.promiser}, ${this.service_policy}); } body contain useshell { useshell => "useshell"; } ``` **Note:** Further work remains, notably "enabling" the service if it's not yet been enabled and integrating it into the Masterfiles Policy Framework and the `standard_services` bundle. ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [Termux](https://termux.dev) - [Termux Services](https://wiki.termux.com/wiki/Termux-services) - [CFEngine services promise type](https://docs.cfengine.com/latest/reference-promise-types-services.html) - [Masterfiles Policy Framework](https://github.com/cfengine/masterfiles) - [standard_services bundle](https://github.com/cfengine/masterfiles/blob/df815a0f1278e171f88ff13e768ff2f8786aa3ac/lib/services.cf#L94) - From the post show chit-chat - RSS Feed for the cfengine.com/blog? We have one, find the [rss feed here](https://cfengine.com/blog/index.xml)! - Struggling with package management, unauthorized software installed in your infrastructure? Check out the [packages-allowlist](https://build.cfengine.com/modules/packages-allowlist/) and [conditional-installer](https://build.cfengine.com/modules/conditional-installer/) modules developed during hack day at [CfgMgmtCamp 2023](https://cfgmgmtcamp.eu/ghent2023/) -------------------------------------------------------------------------------- title: "Sneak peek: Groups in Mission Portal" url: /blog/2023/groups-app-sneak-peek/index.md date: "2023-02-15" -------------------------------------------------------------------------------- {{< wideimg "images/groups-blog-post.png" "Decorative image for the blog post">}} Using CFEngine there are many ways to group and classify your hosts. In order to group their hosts, our users use a combination of JSON files, CFEngine policy language (with variables, classes, and class expressions), host specific data and host filters in Mission Portal. With these features you can choose which hosts to show in reports, and you can make decisions on what changes to make on which hosts. There is, however, no straightforward way for a Mission Portal user to save a selection of hosts (a filter) and then start doing things (reports, changes) with those hosts. Our next big feature is _groups_ in Mission Portal: > A group is a saved filter with a name. (Optionally with some data assigned). There are 2 big reasons why we're adding groups to Mission Portal: * Allow saving selections of hosts (filters) and reusing them in various reports, making Mission Portal more intuitive and user-friendly. * Enable making configuration changes to many hosts at a time. ## All groups at a glance In the **Groups** app, users can organize their personal groups, and view all the shared groups from others on their team. Similar to our recent improvements to the reports page, users can click the star icon next to shared groups to make them show up within **My groups**. {{< wideimg "images/video1.gif" "The gif shows the overview of the host groups. The groups are categorized into two lists, which are My groups and Shared groups.">}} ### Creating groups Upon clicking the **Add** button, a new group is created, and the title and filter can be customized. {{< wideimg "images/video5.gif" "The gif shows how it looks like when creating a new group.">}} Within the filter you specify a list of rules that have to be satisfied for a host to be in the group. These can be things like: * Hosts running Microsoft Windows (class `windows` is defined) * Hosts without a maintainer (inventory attribute `Maintainer` is not reported) * Hosts in AWS us-east-1 region (class `aws_us_east_1` is defined) In addition to these rules, you can also add and remove (include and exclude) individual hosts from the filter. ## Host data Now with the **Groups** app, users can edit the data for a group of hosts simultaneously, similar to the **Host specific data** inside the host info page. The data is stored in PostgreSQL on the hub, and transferred to hosts as JSON files, where the values can be used in your policy and by modules. ### Variables and classes The **Data** tab has a similar UI to **Host specific data** and allows you to enter variables and classes: {{< wideimg "images/video2.gif" "The gif shows the data tab page.">}} ### Module input The **Modules** tab has a similar UI to CFEngine Build in Mission Portal, especially for modules which accept input: {{< wideimg "images/editing-module-data.png" "">}} With this functionality, you can use modules like the [`delete-files`](https://build.cfengine.com/modules/delete-files/) module and make it delete a file only on groups of hosts in your infrastructure. ## Access control By default, users can create their own personal groups and view groups shared across the organization. This enables viewing useful info in Portal and using the groups functionality to make reports, without the risk of actually making changes to hosts. The more powerful features, such as creating/deleting/editing shared groups and group data, are only available to administrators by default. This prevents other users from accidentally or maliciously causing problems to hosts in the infrastructure. As with other features, an administrator can go to the RBAC settings to customize which functionality a user / role has access to. ## Getting started with CFEngine The **Groups** app will arrive later this year. If you would like to get to grips with Mission Portal first, you can follow [this tutorial](https://docs.cfengine.com/docs/3.21/getting-started.html) to get started with CFEngine for free. Or [get in touch](/contact/) with our team to see how we can help you with your project. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 21 - Troubleshooting with cf-support" url: /blog/2023/the-agent-is-in-episode-21-troubleshooting-with-cf-support/index.md date: "2023-01-26" -------------------------------------------------------------------------------- What's the best way to collect information when troubleshooting something with CFEngine? Cody and Nick chat with Craig about `cf-support` a new tool shipping in the latest (and future) versions of CFEngine. ## Video {#video} The video recording is available on YouTube: {{< youtube F9dwkI-aFW8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - `cf-support` is packaged as part of [3.15.7, 3.18.3](https://cfengine.com/blog/2022/cfengine-3-15-7-and-3-18-3-released/), and [3.21.0](https://cfengine.com/blog/2022/cfengine-3-21-lts-released-unification/) and later releases - Post recording chit chat: - [Security Advent Calendar - Part 2](https://cfengine.com/blog/2022/security-holiday-calendar-part-2/) - [Security Advent Calendar - Part 1](https://cfengine.com/blog/2022/security-holiday-calendar-part-1/) -------------------------------------------------------------------------------- title: "Guest blog post: Don't use your distro's package manager" url: /blog/2023/dont-use-a-package-manager/index.md date: "2023-01-23" -------------------------------------------------------------------------------- I have stopped using my Linux distro's package manager, and you should, too. Maybe I should clarify that. I don't install software with my distro's package manager any more. I still upgrade my system. I became influenced by a few different factors. Top among these is something required in certain industries called a change advisory board or committee. This requirement says that changes to production computers have to be reviewed and approved by all stakeholders in that computer's operations. Second among these is the concept of reproducible builds. Every so often, I get a new computer or a new hard drive. This always means installing a set of software packages and configuring to my liking. And third was my commitment to configuring my machines with CFEngine. This includes installing software. Indeed, CFEngine has a powerful packages promise type that works across multiple distros (by identifying the underlying package system and calling that package manager behind the scenes). ## Public companies require a change management process {#public-companies-require-a-change-management-process} Let's look a little deeper into the change management process. My first introduction to this was working for a company that had recently become publicly traded and we instituted a change advisory board meeting every day. Being a member of the technical operations team, I had to bring my case for changes I needed to make before the board during the regularly scheduled meeting. This was generally a simple process, I said this or that machine needs to update packages, or we need to install this service. The impact will be the following. The risk is such and such. The rollback procedure is like this. It will be scheduled for this time on this day. And everybody at the meeting says it's ok and I go do the work. Now, there was one situation that never sat quite right with me. I don't want to out anybody for not following procedures correctly because the impact here was minimal, but here we go. The rule was, don't install software on any production machine without an approved change ticket. But sometimes you log into a machine and something simple, which you need to diagnose a problem, isn't installed. Something like `traceroute`. So I would ask, is it ok if I install `traceroute` on this production server? My boss said yes. Like I said, it's a little thing, but it wasn't documented. I decided there's a better way to handle this scenario, which I can easily implement with CFEngine. Create a list of software you want installed on every computer. That's easy enough. Using CFEngine means we can guarantee it actually gets installed. Now when we want to add another package to that list, we have a short vote in the change management meeting, and add it. CFEngine will do the rest. That package will be installed on every production machine. Now we create what I call a baseline of packages that are approved to be installed on every machine, but can be lumped into one bundle because they don't require any configuration. ## Reproducibility {#reproducibility} I use Fedora, btw. I also use Debian, btw. Sometimes I use Ubuntu, btw. You get the picture. Last year I got a new hard drive for my laptop. I set about to install a new operating system on it, but I made the decision I was going to manage this computer exclusively with CFEngine. This is something I read Mark Burgess did in his book on CFEngine version 2. That means I will install all my software with CFEngine, which is the point of this article. I want to point out that some distros will call the same package differently than another distro. For example, RedHat family distros package the `ag` command as `the_silver_searcher`, but Debian family distros package it as `silversearcher-ag`. This requires some manipulation of the package list which I will demonstrate further on. ## Introducing the baseline bundle {#introducing-the-baseline-bundle} Let's start with a new CFEngine policy file. Of course, I'm using Emacs. I named my file `baseline.cf`. If you followed my previous article, [Quick-start guide to using Emacs for CFEngine](https://cfengine.com/blog/2022/quick-start-guide-to-using-emacs/), you know how I create a new policy bundle using the YASnippet Emacs extension to insert various templates into my file. Let's start with the outline. ```cfengine3 bundle agent baseline { meta: vars: defaults: classes: users: files: packages: guest_environments: methods: processes: services: commands: storage: databases: reports: } ``` This is going to be a very simple bundle. We're going to add a few `vars` promises, and a `packages` promise. Add the following under the `vars:` heading. ```cfengine3 vars: "lists" -> { "def.json" } slist => "@{def.baseline}"; "pkg[${lists}]" if => fileexists("${lists}"), slist => readstringlist( "${lists}", "\s*#[^\n]*$", "\n", "inf", "inf"); "packages" slist => getvalues("pkg"); ``` For those who may be new to CFEngine, I will walk through all of these. The first variable is a string list called `lists`. This list will be retrieved from the augments file, `def.json`. This is indicated in the promisee definition. I always indicate that the values of certain variables will be found in `def.json` in the promisee. That way, when I am re-using my policy files, I remind myself where I need to define something. I will go over the contents needed in `def.json` below. The second variable is an associative array variable called `pkg` holding a list of strings (it's actually a collection of individual variables which CFEngine can treat as a cohesive unit like `data` type vars for some operations). The keys used in this structure will be the file names coming from `lists`. The contents of each of the string lists will be read from the corresponding files. And there's an `if` property to prevent any error occurring if one of those named files is not found. The final variable is a string list of the actual packages to install called `packages`. I named it that to correspond to the `packages` promise. And speaking of the `packages` promise, let's add that next. ```cfengine3 packages: "${packages}" comment => "Ensure ${this.promiser} is installed", handle => "${this.bundle}_packages_${this.promiser}", policy => "present"; ``` There is only one `packages` promise statement, thanks to CFEngine's automatic looping over list values. The first property in this promise is the comment. This will be printed if `cf-agent` is run in verbose mode. The second property is the handle, which is also printed in verbose mode. I always include the bundle name and the current promiser, which in this case will be the package name currently being processed, in the handle. This allows me to identify the exact location of any promise iterations that cause problems. The final property is the policy, which is set to "present." This relies on the _package module_ [^fn:1] implementation of `packages` promises available in the masterfiles promise library, which doesn't require declaring which package manager to use. `apt` will be used on Ubuntu and `zypper` will be used on SUSE, for example. This is all we have to do for the promise bundle. All the actual configuration will be handled in `def.json` and text lists referenced therein. ```json { "inputs": [ "services/baseline.cf" ], "vars": { "control_common_bundlesequence_end": [ "baseline" ], "baseline": [ "${sys.inputdir}/baseline.default.dat", "${sys.inputdir}/baseline.${sys.flavor}.dat", "${sys.inputdir}/baseline.${sys.os}.dat", "${sys.workdir}/baseline.local.dat" ] } } ``` If you are already familiar with `def.json` and its contents, yours may include far more than what is presented above. This is just a simple example to support the bundle we have created. **Note:** The above snippet uses the original style syntax. Since 3.18.0 CFEngine supports a more expressive structure which allows specification of comments, meta data and targeting variables in any bundle of any namespace. This mirrors the above: ```json { "inputs": [ "services/baseline.cf" ], "variables": { "default:def.control_common_bundlesequence_end": { "value": [ "baseline" ], "comment": "Custom policies to run at the end of the standard MPF bundlesequence" }, "default:def.baseline": { "value": [ "${sys.inputdir}/baseline.default.dat", "${sys.inputdir}/baseline.${sys.flavor}.dat", "${sys.inputdir}/baseline.${sys.os}.dat", "${sys.workdir}/baseline.local.dat" ], "comment": "Data files used to specify baseline configurations" } } } ``` Adding the policy file name to `inputs` means I can add the file to the `inputs` attribute of `body common control` without editing `promises.cf`. Likewise, adding the name of the bundle to the `control_common_bundlesequence_end` variable means I can add this bundle to the `bundlesequence` without editing `promises.cf`. I consider the ability to modify these properties without editing a file shipped with the Masterfiles Policy Framework (MPF) important because if I don't have to edit any of these files, I won't introduce my own bugs. Finally is a list of filenames which will be the value of the variable `def.baseline`. Notice I have included a few variables in the file names, coming from `sys`. These variables are determined in CFEngine's C code and available before `def.json` is read. This is only true of `sys` (and `const`) variables so they are the only ones which can be included in `def.json`. All of these files will be found relative to the `sys.inputdir` directory, which is usually `/var/cfengine/inputs`. I have named all of these files ending with "`.dat`" because when `cf-agent` processes `update.cf`, it will only copy certain file names to the target `inputs` directory. These files must match the following. ```shell cf-promises -f update.cf --show-vars=update_def.input_name_patterns ``` This results in the following file patterns. * `*.awk` * `*.cf` * `*.conf` * `*.csv` * `*.dat` * `*.json` * `*.mustache` * `*.pl` * `*.ps1` * `*.py` * `*.rb` * `*.sed` * `*.sh` * `*.txt` * `*.yaml` * `cf_promises_release_id` I thought `*.dat` seemed logical. I think `*.txt` would have been equally acceptable. My `baseline.default.dat` file contains that list of packages that I always want installed. If change management approves a new package, it should get added here. As a systems administrator, it is quite frustrating when the following are not installed. ```text bash-completion curl gzip iperf3 iproute iputils jq less lynx man-db mg mlocate more moreutils nmap pcre-tools procps-ng screen strace sysstat tar tcpdump traceroute tree wget which whois ``` The variable `sys.flavor` represents the Linux distro, if it is Linux. Otherwise `sys.os` is the operating system, which would be Linux, Solaris, or perhaps AIX. Recall the `if` property in the `vars` promise. This allows either of these to be absent. Here is an example of a `baseline.debian_11.dat` file. ```text apt-transport-https bind9-dnsutils bind9-host mailutils needrestart silversearcher-ag ``` And the following is an example of a `baseline.centos_7.dat` file. ```text bind-utils mailx the_silver_searcher yum-plugin-copr ``` I use my `baseline.local.dat` to contain packages which I want on my desktop, but the name provides a convenient location to store any package lists for a specific host. Don't put this file in `masterfiles`, save it in `/var/cfengine/`. ```text alacritty emacs firefox rofi thunderbird ``` ## Conclusion {#conclusion} Using CFEngine on my desktops or laptops, I can keep my machines installed with all my favorite software following an Infrastructure-as-Code model. Once CFEngine is set up, any program can be installed just by adding the package name to one of the data files. Additionally, this method works well in a corporate environment where changes to machines are managed carefully. This same configuration works equally and allows the admin team to know the programs they require are installed on every server. [^fn:1]: The _package module_ based implementation of packages promises was introduced in CFEngine 3.7.0. A package module can be implemented in any language. Package modules communicate with `cf-agent` via the [package module api](https://docs.cfengine.com/docs/master/reference-language-concepts-modules-package-module-api.html#the-api). Its implementation is the preferred syntax for policy writing and usage documentation is found under the [packages promise type](https://docs.cfengine.com/docs/3.21/reference-promise-types-packages.html). -------------------------------------------------------------------------------- title: "CFEngine 2022 retrospective" url: /blog/2022/2022-retrospective/index.md date: "2022-12-31" -------------------------------------------------------------------------------- It's that time of year again where we reflect & recap all things new with CFEngine from this year. You may recall from [the 2021 retrospective](/blog/2021/2021-retrospective/) that our focus for 2022 would be on collaboration, ease of use, and community engagement. I'm proud to summarize our progress below in these key areas for 2022's Retrospective and give you a sneak peek at what's to come in 2023. ## Revamped documentation CFEngine is a powerful, flexible, and complex piece of software, but we are committed to make it as easy to use as possible, and are looking at all ways we can improve the new user experience. The documentation is an important tool for both new and experienced users to find the information they need. We identified multiple areas for improvement in terms of structure, navigation, search, and content, we decided to completely overhaul it in 2022. The new documentation was launched this fall, and includes several new improvements: * **New look** - A modernized design that eliminates clutter and enables you to more quickly find information. * **Updated navigation** - Everything is organized into one hierarchy, which can always be found on the left side. * **Improved search** - The new search bar is much better at finding what you're looking for. * **Responsive design** - Our documentation now looks great no matter the device, screen size, or window size. For more details and screenshots of the redesign, see [our blog post from June](/blog/2022/5-improvements-coming-to-the-cfengine-docs/). ## Getting started guide and videos Along with the new documentation site, we've also started improving a lot of the content. One core part of this is the new [getting started guide](https://docs.cfengine.com/docs/3.21/getting-started.html). These tutorials take you through testing CFEngine for the first time, including installing, using the Web UI, modules, reports, etc. There are also video versions available: {{< youtube GSaRfDlfh7I >}} All of the videos are available in [a playlist on YouTube](https://www.youtube.com/playlist?list=PLh71Vl9YjMahQyC93A3-DGgs6TJvfvp4F). ## CFEngine Build in Mission Portal {{< wideimg "images/build-in-mp-3-21.png" "Screenshot of the Build in MP application">}} One of the most significant additions last year was the introduction of CFEngine Build, which enables users to more quickly and easily automate infrastructure. This year, we made using CFEngine Build even easier by integrating the tooling directly into Mission Portal. In the latest release, we added several powerful features, including: * **Module input** - allowing end users to configure what a module should do, without writing code. * **Multiple projects** - to manage multiple policy sets, either for different hubs / environments, or for testing out new changes. * **Local deploy** - to allow easily testing modules and CFEngine Build functionality without setting up a remote git repo. ## Reports UI The Reports UI in Mission Portal has also been redesigned allowing you to quickly access your most relevant reports: {{< wideimg "images/new-report-page.png" "Screenshot of new report page UI">}} There are now multiple tabs for **My reports**, **CFEngine Build**, **Default reports**, and **Reports made by others**, which makes finding individual reports easier and eliminates the need to sift through all reports in one view. You can read more about the CFEngine Build in MP and the new reports UI in the latest [3.21 LTS release blog post](/blog/2022/cfengine-3-21-lts-released-unification/). ## Compliance reports Compliance reports were introduced in 2021, and give you high level overviews of your infrastructure's compliance with various security checks, as well as the ability to drill down and see why individual checks are failing on individual hosts. This year, we improved the report design, especially for the PDFs, to make them more organized and easy to read: [![](./images/example-compliance-report.png)](./images/example-compliance-report.png) ## Releases We released 6 new versions of CFEngine this year: * [CFEngine 3.15.6 and 3.18.2 LTS](/blog/2022/cfengine-3-15-6-and-3-18-2-released/) * [CFEngine 3.20.0 - Modularity](/blog/2022/cfengine-3-20-released-modularity) * [CFEngine 3.15.7 and 3.18.3 LTS](/blog/2022/cfengine-3-15-7-and-3-18-3-released/) * [CFEngine 3.21.0 LTS - Unification](/blog/2022/cfengine-3-21-lts-released-unification/) To test the new features mentioned above, get one of the newest releases today, from our [download](/download) page. Additionally, [we released 28 new versions](https://github.com/cfengine/cfbs/releases) of the CFEngine Build command line tool, [`cfbs`](https://pypi.org/project/cfbs/). The most important changes are summarized in these 2 blog posts: * [CFEngine Build System version 2](/blog/2022/cfbs-2-0-0/) * [CFEngine Build System version 3](/blog/2022/cfbs-3-0-0/) Install or upgrade `cfbs` via the Python Package Index: `pip3 install --upgrade cfbs`. ## Holiday security calendar Due to popular demand, the CFEngine team brought back the Holiday security calendar that provided daily security recommendations and best practices that can be easily implemented into your CFEngine deployment. You can catch up on these daily tips [here](/blog/2022/security-holiday-calendar-part-2/). {{< img "images/day-0-square.png" "Holiday / Christmas themed calendar with 25 boxes in the shape of a Christmas tree.">}} ## Trends Most of the content we publish ([blog posts](/blog), [webinars](/webinars), [videos](/videos), [white papers](/white-papers)), focuses on the CFEngine product, new functionality and the design and engineering behind it. Over the past few years, we've been picking up development pace, adding big improvements and new features. An indicator of this, is the number of blog posts per year, and we're happy to share this nice graph/trivia - The number of blog posts on cfengine.com has been increasing every year, for the past 6 years: {{< img "images/blog-posts-2017-2022.png" "Graph showing an increasing number of blog posts per year in 2017-2022.">}} ## Holiday giveaway As a thank you to our loyal users, we'd like to send you some CFEngine SWAG. Fill out the ["Contact us" form](/contact/) on our website, include your mailing address in the 'How can we help?' section, and our team will hook you up with the latest CFEngine gear. ## Next year For next year we will continue a lot of the great things we did this year, including: * Improving the documentation content with more easy tutorials and examples. * Focusing on ease of use in every area (MP, Docs, websites, policy language, etc.). * Adding more modules to help you with compliance and security hardening, especially in more modern contexts such as IoT, hybrid cloud, Kubernetes clusters, etc. Finally, I'd like to leave you with one little teaser of the next big feature coming to CFEngine: {{< wideimg "images/groups-teaser.png" "Screenshot of MP UI showing a group of hosts titled Ubuntu hosts in AWS.">}} As always, we wouldn't have had the success we did this year without our Enterprise customers and Community users. Thank you for your continued contributions, feedback, and support. We look forward to working together in 2023. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 20 - Reviewing the 2022 CFEngine holiday security calendar" url: /blog/2022/the-agent-is-in-episode-20-security-holiday-calendar/index.md date: "2022-12-29" -------------------------------------------------------------------------------- For the holiday season gift yourself an improved infrastructure security posture. Join Craig, Cody, and Nick as they wrap up 2022 and the 20th episode of "The agent is in" reviewing CFEngines' 2022 Holiday Security Calendar which has advice picked straight from industry standard security hardening guides like the [OpenSCAP Security Policies](https://www.open-scap.org/security-policies/) and [Security Technical Implementation Guides](https://public.cyber.mil/stigs/) (STIGs). Craig demos new modules like [maintainers-in-motd](https://build.cfengine.com/modules/maintainers-in-motd/), [file-permissions](https://build.cfengine.com/modules/file-permissions/), [enable-aslr](https://build.cfengine.com/modules/enable-aslr/), highlights guidance on [writing your own security policies](https://cfengine.com/blog/2022/five-security-policy-examples/) and more. ## Video {#video} The video recording is available on YouTube: {{< youtube nvRhjVsU5W0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - Blog post: [Security holiday calendar - Part 1](https://cfengine.com/blog/2022/security-holiday-calendar-part-1/) - Blog post: [Security holiday calendar - Part 2](https://cfengine.com/blog/2022/security-holiday-calendar-part-2/) -------------------------------------------------------------------------------- title: "Security holiday calendar - Part 2" url: /blog/2022/security-holiday-calendar-part-2/index.md date: "2022-12-25" -------------------------------------------------------------------------------- Thank you for following along with our security themed holiday calendar. Today, we summarize the last half of the calendar, in case you missed some days. ## Part 1 recap (12/25) A couple of weeks ago, on the 12th of December, we posted a recap of the first 12 days: [cfengine.com/blog/2022/security-holiday-calendar-part-1](https://cfengine.com/blog/2022/security-holiday-calendar-part-1/) [![Holiday / Christmas themed calendar with 25 boxes in the shape of a christmas tree.](images/day-12-square.png)](/blog/2022/security-holiday-calendar-part-1/) ## File integrity monitoring with CFEngine (13/25) On the 13th, we took a look at how you can use File Integrity monitoring in CFEngine for similar functionality to [AIDE](https://aide.github.io/): [cfengine.com/blog/2022/file-integrity-monitoring-with-cfengine](https://cfengine.com/blog/2022/file-integrity-monitoring-with-cfengine/) **Recommendation:** Set up monitoring for changes in important files, such as `/etc/passwd`, `/etc/crontab`, installed software, etc. This could help you notice an accidental misconfiguration, or a malicious attacker making changes. ## Track maintainers and purpose for hosts in your infrastructure (14/25) When a host is failing compliance checks, acting suspiciously, or stops reporting, what do you do? Knowing what the machine does and who to contact is crucial: [cfengine.com/blog/2022/track-maintainer-for-hosts](https://cfengine.com/blog/2022/track-maintainer-for-hosts/) **Recommendation:** Keep track of who is responsible and what is the purpose for each machine in your infrastructure. As shown in the blog post above, an inventory report showing maintainer's email address and other relevant information for each host is useful: {{< wideimg "images/maintainers-inventory.png" "Inventory with maintainer and purpose information added">}} ## Delete .rhosts files in home directories (15/25) Similar to the `.shosts` files, the `.rhosts` files in home directories were historically used for per-user defined host based authentication, in rsh (and ssh). A global, system-wide, file is preferred, and these files should be deleted if present: [build.cfengine.com/modules/delete-home-dotrhosts](https://build.cfengine.com/modules/delete-home-dotrhosts/) **Recommendation:** Do not allow user defined host based authentication. Use this `delete-home-dotrhosts` module to inventory and delete `~/.rhosts` files. This ensures you are protected in the future, if somebody adds a `~/.rhosts` file. ## It's always a DNS problem (16/25) The `bind` package provides the `named` service, for running a DNS server. Most machines are not DNS servers, and don't need this package. To reduce attack surface, this package should be uninstalled when not needed: [build.cfengine.com/modules/uninstall-bind](https://build.cfengine.com/modules/uninstall-bind/) **Recommendation:** Ensure only DNS servers use the `bind` package by uninstalling it on all other hosts (by default). Explicitly define hosts which are DNS servers and thus need the `bind` package. ## Secure permissions by default (17/25) Limiting access to system-wide configuration files and folders managed by the root user is very important. In CFEngine 3.21.0 we are changing the default permissions when creating directories to be more strict (secure default): [cfengine.com/blog/2022/change-in-behavior-default-directory-permissions-now-700](https://cfengine.com/blog/2022/change-in-behavior-default-directory-permissions-now-700/) **Recommendation:** When creating directories and permissions using scripts / policy or other software, consider what the permissions should be. If unclear / unspecified, choose a restrictive default, such as `600` for files and `700` for folders. ## Samba! (18/25) The Samba software enables file and printer sharing, and is typically used in mixed Linux and Windows environments. It can provide an attack vector and has been affected by vulnerabilities in the past. If not used, it should be removed: [build.cfengine.com/modules/uninstall-samba](https://build.cfengine.com/modules/uninstall-samba/) **Recommendation:** Uninstall Samba when possible (when it is not needed). For infrastructures or environments which don't use it at all, uninstall it everywhere. If some of your hosts need to use it, explicitly define where it is needed, and uninstall it everywhere else. ## Enforce anything with policy language (19/25) What do you do when you have a security requirement custom to your organization? When there's no ready-to-use module? You can write the policy yourself, just like we've been writing the modules: [cfengine.com/blog/2022/five-security-policy-examples](https://cfengine.com/blog/2022/five-security-policy-examples/) **Recommendation:** Learn CFEngine policy language and enforce any security requirements by writing your own policy / modules. ## Uninstall the Apache httpd web server (20/25) Apache `httpd` is one of the most widely used web servers on the internet. If your hosts are not serving websites, and don't need `httpd`, it's best to uninstall it, preventing accidental or malicious activation: [build.cfengine.com/modules/uninstall-apache](https://build.cfengine.com/modules/uninstall-apache/) **Recommendation:** Ensure the Apache web server is only running where necessary, by uninstalling it on all other machines (by default). Explicitly specify which machines are allowed to run the web server. ## Delete files indicating security issues (21/25) Many configuration files for cron, at, SSH etc. enable insecure configuration, but are still supported for backwards compatibility. With the delete-files module and the upcoming module input feature, in CFEngine 3.21.0, you can easily delete files across your infrastructure from within Mission Portal: [build.cfengine.com/modules/delete-files](https://build.cfengine.com/modules/delete-files/) **Recommendation:** Enforce the removal of files known to be malicious or indicating insecure configuration. Examples of such files are `~/.shosts` files, `/etc/cron.deny`, etc. ## Uninstall unnecessary packages (22/25) A lot of software is still available to install in package managers, but not recommended to use for security reasons. Old and unmaintained software often has bugs, security vulnerabilities or lack important security features such as strong authentication and encryption. Different compliance frameworks and security hardening guidelines recommend uninstalling various packages, such as: * `bind` * `dhcp` / `isc-dhcp-server` * `dovecot` / `dovecot-core` * `httpd` / `apache2` * `samba` * `squid` * `talk` / `talk-server` / `talkd` * `telnet-server` / `telnetd` * `xinetd` The `uninstall-packages` module allows you to specify a list of all the packages you want uninstalled: [build.cfengine.com/modules/uninstall-packages](https://build.cfengine.com/modules/uninstall-packages/) **Recommendation:** Uninstall packages you don't need, especially focusing on servers / daemons, and software known to be insecure or have vulnerabilities. Use this module and specify the packages you want uninstalled with module input. Compliance frameworks and security hardening guidelines can help you with ideas beyond the list above. ## Upgrade CFEngine (23/25) Newer versions of CFEngine include features around security and compliance, bug fixes, and other improvements. Take a look at what's changed and what's new in the latest release, 3.21.0: [cfengine.com/blog/2022/cfengine-3-21-lts-released-unification](https://cfengine.com/blog/2022/cfengine-3-21-lts-released-unification/) **Recommendation:** Upgrade to 3.18.3 as soon as possible, and start looking at 3.21, the new LTS series. Stay up to date by always running one of the latest releases on an LTS series. ## Enforce strict file permissions (24/25) It's important to enforce strict access to system-wide configuration files for software such as SSH, cron, at, etc. Our `file-permissions` module allows you to manage permission bits of files, using module input: [build.cfengine.com/modules/file-permissions](https://build.cfengine.com/modules/file-permissions/) **Recommendation:** Enforce strict permissions on important configuration files such as `/etc/ssh/sshd_config`, `/etc/motd`, `/etc/crontab`, etc. ## Happy holidays (25/25) Today, we've wrapped up the holiday calendar for 2022 with this summary. Happy holidays from the CFEngine team: {{< wideimg "images/day-25-square.png" "Happy holidays! Thank you for helping us secure the world's connected devices">}} -------------------------------------------------------------------------------- title: "CFEngine 3.21 LTS released - Unification" url: /blog/2022/cfengine-3-21-lts-released-unification/index.md date: "2022-12-21" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.21.0\! The focus of this new version has been _unification_. Across our websites and UI, you should see that it's a much more modern and unified experience, whether you're reading this blog post on [cfengine.com](https://cfengine.com), browsing [the new documentation site](https://docs.cfengine.com), looking for modules on [the CFEngine Build website](https://build.cfengine.com), or [adding input to modules within Build in Mission Portal](https://cfengine.com/webinars/?videoID=SCbZ-M7XFN4). This release also marks an important event, the beginning of the 3.21 LTS series, which will be supported for 3 years. Several new features have been added since the release of CFEngine 3.18 LTS, in the form of non-LTS releases. In this blog post we'll primarily focus on what is new in 3.21, but we'll also highlight some things released in 3.19 and 3.20. ## What's new ### CFEngine Build in Mission Portal {{< wideimg "images/build-in-mp-3-21.png" "Screenshot of the Build in MP application">}} With CFEngine Build in Mission Portal, you can use modules to make changes to your infrastructure as well as add reports and reporting data all from within Mission Portal. Since it was introduced in [3.20](/blog/2022/cfengine-3-20-released-modularity/) several features have been added, including adding input for modules, managing multiple projects, and deploying your policy locally. #### Module input CFEngine Build modules can now accept input. This allows module authors to write modules which prompt the user about what to do. Examples of this could be allowing the users to specify files to delete, packages to install, processes to kill, etc. The [`delete-files` module](https://build.cfengine.com/modules/delete-files/) lets you specify a list of files to delete: {{< wideimg "images/delete-files-example-input-mp.png" "Module input UI showing 3 files the user wants to delete">}} When you have this module in your project and add the input as above, it means the module will check if those files exist on your hosts, and delete them if they do. Another example of a module accepting input is [`file-permissions`](https://build.cfengine.com/modules/file-permissions/), which allows you to enforce the permission bits of files. In the coming months, we will publish more modules like this, allowing you to easily make changes to your hosts and enforce security requirements from within Mission Portal. #### Local deploy and new deploy button The new deploy button has 3 options: {{< img "images/new-deploy-button.png" "Button with dropdown of 3 options, Push & Deploy, Push, Deploy locally">}} **Push and deploy** lets you push your changes to the remote git repository (for example GitHub or GitLab), and ensures your version control settings are correct, so the policy set is deployed on this hub. (The policy set will also be deployed to other hubs set up to use the same repository). **Push** allows you to push your changes to the remote git repository without affecting what policy is deployed on this hub. (If you have other hubs configured to pull policy from that git repository, the policy changes will be deployed there). The third option, **Deploy locally** enables deploying projects without syncing with a remote git repo. This is great for testing out CFEngine 3.21.0 and modules together. You can just install CFEngine, go to the **Build** application, skip entering git settings and start adding modules and deploying to your hub. (Previously you had to set up a remote repository on github.com or similar). #### Multiple projects Within the Build application, you can now create multiple projects: {{< img "images/multiple-projects.png" "Dropdown for selecting between projects or adding new ones">}} When combined with the new deploy button shown above, this means you can easily switch between projects and test out modules. If you already have one project being deployed to your test hub, you can create another project, add the modules you want to test, and use **Deploy locally** to see how they work, then switch back to your main project and deploy that again. #### Video If you haven't seen it yet, we showed a lot of the CFEngine Build related functionality in the last episode of our [webinar series](/webinars): {{< youtube SCbZ-M7XFN4 >}} ### New report page The new report pages allows you to more easily organize and browse your reports: {{< wideimg "images/new-report-page.png" "Screenshot of new report page UI">}} The ones you care most about show up in **My reports**, meaning the ones you created, or added with the star buttons. Reports which come with CFEngine by default show up in **Default reports**, while the reports made by other Mission Portal users show up in **Reports made by others**. Finally, reports imported from CFEngine Build modules show up in **CFEngine Build**. All reports from **CFEngine Build** and **Default reports** now have special icons instead of authors, indicating that they were not made by Mission Portal users. ### New compliance report PDFs One feature which was already released in [3.18.3](/blog/2022/cfengine-3-15-7-and-3-18-3-released/), a few weeks ago, is the new compliance report design: [![](./images/example-compliance-report.png)](./images/example-compliance-report.png) ### New documentation site Although not strictly a part of the release packages, we'd like to highlight that our documentation was revamped this fall: [docs.cfengine.com](https://docs.cfengine.com/docs/3.21/) The documentation site has a new design, with improved navigation and search. To learn more about the changes done to the documentation site, see [this blog post](/blog/2022/5-improvements-coming-to-the-cfengine-docs/). ### Policy language changes A couple of new attributes have been introduced: - [`system_log_level`](https://docs.cfengine.com/docs/3.21/reference-components.html#system_log_level) - for controlling what log messages go to syslog / Windows Event Log. - [`copyfrom_restrict_keys`](https://docs.cfengine.com/docs/3.21/reference-components-cf-agent.html#copyfrom_restrict_keys) - for restricting which hosts `cf-agent` will copy files from. Additionally, several attributes were implemented for [custom promise types](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html): - [`action_policy`](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html#action-policy) - for telling modules to produce warnings instead of fixing (repairing) promises. - [`comment`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#comment) - for documenting the reasons and context around a promise. - [`handle` and `depends_on`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#depends_on) - for controlling evaluation order of promises. - [`ifelapsed`](https://docs.cfengine.com/docs/3.21/reference-components-cf-agent.html#ifelapsed) - to delay repeated evaluations of a promise until some amount of time has elapsed. - [`meta`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#meta) - for attaching metadata / tags about a promise. - [`with`](https://docs.cfengine.com/docs/3.21/reference-promise-types.html#with) - gives you an easy way to expand results of function calls into a temporary variable. For a complete list of changes in policy language, see the changelogs linked in the section below. ### New tooling A new tool `cf-support` has been introduced. This tool is intended to streamline data collection when reporting issues. ### Other changes There are other big changes introduced in the past 1.5 years, but not shown here. We encourage you to read our previous release blog posts to see more of these changes: - [CFEngine 3.19 - Collaboration](/blog/2021/cfengine-3-19-released-collaboration/) - [CFEngine 3.20 - Modularity](/blog/2022/cfengine-3-20-released-modularity/) - [CFEngine Build System version 2](/blog/2022/cfbs-2-0-0/) - [CFEngine Build System version 3](/blog/2022/cfbs-3-0-0/) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.21.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-core.html) - [3.21.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-enterprise.html) - [3.21.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.21/release-notes-whatsnew-changelog-masterfiles-policy-framework.html) If you are upgrading from the 3.18 LTS series, scroll down in the changelog to find changes made in 3.19 and 3.20 to see the older changes. Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the recently released [3.18.3](/blog/2022/cfengine-3-15-7-and-3-18-3-released/), these dependencies have been updated: | CFEngine | 3.18.3 | **3.21.0** | | :------------------------------------------------ | :----- | :--------- | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.9.14 | **2.10.3** | | [OpenSSL](https://www.openssl.org/) | 1.1.1q | **3.0.6** | | [PHP](https://php.net/) | 8.0.24 | **8.1.12** | | [PostgreSQL](https://www.postgresql.org/) | 13.8 | **15.1** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ### Python Our hub packages now require Python (3.5 or newer), to enable the CFEngine Build in MP functionality. If you use `cf-remote`, or `apt`, or `yum`, to install the package, this should not be a problem, in most cases Python is already installed, and if not, it will be installed as it's declared as a dependency. ## Platform support We now support **Ubuntu 22**, **Debian 11**, and **RHEL 9**. And Ubuntu and Debian are now packaged for aarch64, bringing CFEngine Enterprise to Raspberry Pis near you! Our general policy is that we support platforms as long as they are under regular support by the OS vendor. Due to this, we are dropping support for **AIX 6**, **Solaris 10**, **SLES 11**, and **Windows Server 2008**. ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. If you are using `cf-remote`, it will now default to 3.21.0, since this is the latest LTS release available. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following avenues: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - [Contribute modules to CFEngine Build](https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) -------------------------------------------------------------------------------- title: "5 security hardening CFEngine policy examples" url: /blog/2022/five-security-policy-examples/index.md date: "2022-12-19" -------------------------------------------------------------------------------- Throughout the [security holiday calendar](/calendar), we've looked at modules for enforcing security requirements. Writing the policy to achieve these security hardening goals is easy. By learning how, you can write policy (or modules) for any requirements, including those specific to your organization. In this blog post, we'll take a look at five beginner-level examples to get you started, focusing on the most common resources to manage with CFEngine; files and packages. All file names, package names, etc. are just examples and should be easy to modify to your desire. The examples are available in this repository: https://github.com/olehermanse/five-security-hardening-examples If you are unsure about how to write and deploy policy, we recommend [the getting started guide in our docs](https://docs.cfengine.com/docs/master/). ## Installing a package This policy ensures the `aide` package is installed, using the default package manager (`yum`, `apt`, etc.) to install it if necessary: ```cfengine3 bundle agent install_aide { packages: "aide" -> { "OpenSCAP", "CCE-27096-7" } policy => "present"; } ``` **Hint:** The stakeholders (` -> { "OpenSCAP", "CCE-27096-7" }`) document where this requirement comes from. It's purely for documentation purposes, the policy would behave the same without this part. ## Uninstalling a package It is quite common to want to get rid of packages and configuration files to eliminate potential security issues and reduce attack surface. `squid` is an example of a package you don't want installed, as running a proxy server can open you up for attack and circumvent network security measures. From the example above, just switch `present` with `absent`, to ensure the package is **not** installed: ```cfengine3 bundle agent uninstall_squid { packages: "squid" -> { "OpenSCAP", "CCE-80286-8" } policy => "absent"; } ``` **Hint:** CFEngine checks the current state to determine if it needs to make changes. For this promise, if `squid` is not installed, nothing will be done. ## Deleting a file Ensuring a file does not exist is quite similar to the example with a package: ```cfengine3 bundle agent delete_shosts_equiv { files: "/etc/ssh/shosts.equiv" -> { "OpenSCAP", "CCE-80513-5" } delete => tidy; } ``` ## Enforcing access to a file Many system-wide configuration files can leak information or be used for privilege escalation. Enforcing strict access to them, through file ownership and permission bits is important: ```cfengine3 bundle agent etc_cron_allow_access { files: "/etc/cron.allow" -> { "OpenSCAP", "NIST", "CCI-000366" } perms => mog("640", "root", "root"), if => fileexists("/etc/cron.allow"); } ``` There are different ways of specifying the `perms` for a file. In the example above, we've used `mog` to specify the _Mode, Owner, and Group_. **Hint:** Checking if the file exists is often not strictly necessary. Adding it can make your policy more explicit and easy to understand. Even someone completely unfamiliar with CFEngine can read the snippet above, and understand that this is only relevant when the file actually exists. ## Managing the content of a file The `/etc/motd` file is used to show useful information to users after logging in. CFEngine can ensure both the content and the permissions of this file are correct: ```cfengine3 bundle agent etc_motd { files: "/etc/motd" -> { "OpenSCAP", "CCE-83737-7", "CCE-83727-8", "CCE-83337-6" } perms => mog("644", "root", "root"), content => "$(sys.host) - Contact: Alice@example.com"; } ``` **Hint:** In the example above, we've used variable expansion; `$(sys.host)` gets replaced by the hostname on each machine in the infrastructure. ## Testing it In order to test our policy, we can introduce some issues and watch CFEngine fix them: ```bash touch /etc/ssh/shosts.equiv touch /etc/cron.allow chmod 0777 /etc/cron.allow echo "blah" > /etc/motd apt install -y squid >/dev/null 2>/dev/null apt remove -y aide >/dev/null 2>/dev/null cf-agent -KI ``` Running the script above we see that CFEngine fixes 5 issues: ``` info: Deleted file '/etc/ssh/shosts.equiv' info: Object '/etc/cron.allow' had permissions 0777, changed it to 0640 info: Updated content of '/etc/motd' with content 'ip-172-31-6-47 - Contact: Alice@example.com' info: Successfully installed package 'aide' info: Successfully removed package 'squid' ``` ## Conclusion Hopefully, these examples give you some ideas for how to improve the security of your infrastructure. This is just the beginning, take a look at [processes](https://docs.cfengine.com/docs/master/reference-promise-types-processes.html), [services](https://docs.cfengine.com/docs/master/reference-promise-types-services.html), [users](https://docs.cfengine.com/docs/master/reference-promise-types-users.html), and even [custom promise types](https://docs.cfengine.com/docs/master/reference-promise-types-custom.html) for what other things you can work with. **Security recommendation:** Implement reports and automation to enforce strict security requirements for your organization. When [ready-to-use modules](https://build.cfengine.com) are not available, write policy to do exactly what you need. In addition to your certification requirements, look at [STIGs](https://www.stigviewer.com/), [NIST / NCP](https://ncp.nist.gov/), [OpenSCAP](https://www.open-scap.org/), [CIS benchmarks](https://www.cisecurity.org/benchmark), [Lynis](https://cisofy.com/lynis/), and other similar resources for ideas on what security checks to implement. -------------------------------------------------------------------------------- title: "Change in behavior: Directories are now created with 700 instead of 755" url: /blog/2022/change-in-behavior-default-directory-permissions-now-700/index.md date: "2022-12-16" -------------------------------------------------------------------------------- In the upcoming release of CFEngine 3.21.0 there is a change in behavior with respect to default permissions of created directories. From 3.21.0 and later directories will be created with read, write, execute permissions only for the file owner. No permissions are granted for group or other. This change improves the default security posture to make sure that only the user executing CFEngine (typically root) will have access to content in newly created directories. This also aligns default directory permissions with default file permissions. ## Illustrating the change in behavior {#illustrating-the-change-in-behavior} This policy snippet illustrates the change in behavior: ```cfengine3 bundle agent __main__ { files: "/tmp/a/promised/directory/." create => "true"; reports: "CFEngine $(sys.cf_version)"; "Directory has $(with)" with => concat( "permoct='", filestat( "/tmp/a/promised/directory/.", "permoct" ), "', ", "permstr='", filestat( "/tmp/a/promised/directory/.", "permstr" ), "'" ); } ``` On versions prior to 3.21.0 we see that newly created directories grant access to both _group_ and _other_. ```text [root@hub ~]# cf-agent -KIf ./example-default-directory-permissions.cf info: Created directory '/tmp/a/promised/directory/.' R: CFEngine 3.18.0 R: Directory has permoct='755', permstr='drwxr-xr-x' ``` On 3.21.0 and later we see that only the directory _owner_ has permission: ```text [root@hub ~]# cf-agent -KIf ./example-default-directory-permissions.cf info: Created directory '/tmp/a/promised/directory/.' R: CFEngine 3.21.0 R: Directory has permoct='700', permstr='drwx------' ``` ## Explicit permissions {#explicit-permissions} The change in default does not affect the behavior when promising explicit permissions. The directory is initially created with the default permissions and then immediately changed to comport with the explicit permissions promised. ```cfengine3 bundle agent __main__ { files: "/tmp/a/promised/directory/." create => "true", perms => permoct( 655 ); reports: "CFEngine $(sys.cf_version)"; "Directory has $(with)" with => concat( "permoct='", filestat( "/tmp/a/promised/directory/.", "permoct" ), "', ", "permstr='", filestat( "/tmp/a/promised/directory/.", "permstr" ), "'" ); } body perms permoct( oct ) { mode => "$(oct)"; rxdirs => "false"; } ``` The output below illustrates the behavior on 3.21.0. ```text [root@hub ~]# cf-agent -KIf ./example-explicit-directory-permissions.cf info: Created directory '/tmp/a/promised/directory/.' info: Object '/tmp/a/promised/directory' had permissions 0700, changed it to 0655 R: CFEngine 3.21.0 R: Directory has permoct='655', permstr='drw-r-xr-x' ``` And running on 3.18.0 we can see that the behavior when promising explicit permissions is identical except for the initial default: ```text [root@hub ~]# cf-agent -KIf ./example-explicit-directory-permissions.cf info: Created directory '/tmp/a/promised/directory/.' info: Object '/tmp/a/promised/directory' had permissions 0755, changed it to 0655 R: CFEngine 3.18.0 R: Directory has permoct='655', permstr='drw-r-xr-x' ``` -------------------------------------------------------------------------------- title: "Track maintainers and purpose for hosts in your infrastructure" url: /blog/2022/track-maintainer-for-hosts/index.md date: "2022-12-14" -------------------------------------------------------------------------------- When something goes wrong or looks fishy for a particular host in your infrastructure how do you know who to ask about it? In an infrastructure managed by many and used by many it is also helpful to know what each host's purpose is. In this article we show how to add maintainer and purpose information to individual hosts in your infrastructure via the CMDB feature of Mission Portal. We will also add a Build Module to add this information to the `/etc/motd` file for each associated host. ## Leverage a Build Module For this task we will first add the [maintainers-in-motd](https://build.cfengine.com/modules/maintainers-in-motd) Build Module to our current [Build project](https://cfengine.com/blog/2022/cfengine-3-20-released-modularity/). This module uses reasonable variable names and automates the process of adding our information to `/etc/motd`. Looking at that module's [README.md](https://github.com/cfengine/modules/blob/master/security/maintainers-in-motd/README.md) file we see that we can add three CMDB variables per host to provide the needed information: `maintainer`, `maintainer_email`, and `purpose`. We additionally add tags of `inventory` and `attribute_name=...` so that we can write reports, compliance checks and an alert widget on our dashboard to show us which hosts don't have information. {{< img "images/maintainers-cmdb.png" "Maintainers variables in Mission Portal host info CMDB">}} As a lazy sysadmin I like to sit back, take a break, and wait 5-10 minutes to let the policy and its effects take hold. ## Inventory and a Compliance report Checking back in, I run an Inventory Report and add the items we added: `Purpose`, `Maintainer` and `Maintainer Email`. I see that a few hosts have information but many do not. {{< wideimg "images/maintainers-inventory.png" "Inventory with maintainer and purpose information added">}} Next we can create a Compliance Report which fails a host if all three items are not reported. See the previous blog post [Building a Compliance report based on inventory modules](https://cfengine.com/blog/2022/compliance-report-based-on-inventory-modules/) for how to accomplish this in detail. {{< wideimg "images/maintainers-compliance.png" "Compliance report for Maintainer info">}} Clicking on the details (bottom-right blue box with arrow) we see which hosts we need to work on. {{< wideimg "images/maintainers-detail.png" "Maintainers info, compliance report detail">}} ## Message of the day ``` [craig@mbp]~% ssh x220 Linux x220 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 ::: use this machine for music and tv, contact My Dog(mydog@somewhere) with any questions/issues. ::: Last login: Tue Dec 13 14:17:10 2022 from fd8a:f257:36bd::99f craig@x220:~$ ``` ``` [craig@mbp]~% ssh 192.168.1.200 Linux raspberrypi 5.15.74-v8+ #1595 SMP PREEMPT Wed Oct 26 11:07:24 BST 2022 aarch64 ::: use this machine for CFEngine personal hub, contact Craig Comstock(craig.comstock@somewhere) with any questions/issues. ::: Last login: Tue Dec 13 14:17:25 2022 from 192.168.1.235 craig@raspberrypi:~$ ``` ## ICE (In Case of Emergency) When a security issue arises on a machine that you aren't completely familiar with it can be essential to know quickly who to contact and what the machine should be used for. **Recommendation:** Add maintainer and purpose information about each host in your infrastructure. We showed how to add maintainer and purpose details with Mission Portal, create a Compliance report to get a high-level view of which hosts lack this information and need to be updated as well as added a message where users will see it first thing on logging in: `/etc/motd`. -------------------------------------------------------------------------------- title: "File integrity monitoring with CFEngine" url: /blog/2022/file-integrity-monitoring-with-cfengine/index.md date: "2022-12-13" -------------------------------------------------------------------------------- File integrity monitoring is an important aspect in managing your infrastructure. Tripwire and AIDE are often cited as necessary tools by compliance frameworks[^fn:1],[^fn:2],[^fn:3]. Of course CFEngine can _manage_ a file to make sure it contains desired content, but did you know that CFEngine also has the capability to simply monitor a file for change? In this blog post we take a look at CFEngines' `changes` attribute for files promises. ## File promises, changes body To monitor a file for change in CFEngine you must have a files promise with a changes body attached. For example: ```cfengine3 files: "/etc/motd" changes => default:detect_all_change; ``` Here we promise to watch `/etc/motd` and detect "all changes". We can look at the documentation for [`changes` bodies](https://docs.cfengine.com/docs/master/reference-promise-types-files.html#changes) and [the definition of `body changes detect_all_change`](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-lib-files.html#detect_all_change) to see what it really means. Using `cf-locate` from [core/contrib](https://github.com/cfengine/core/tree/master/contrib/cf-locate) makes it easy to find the definition of a body or bundle: ```text $ cf-locate "detect_all_change$" -f | ansi2txt -> body or bundle matching 'detect_all_change$' found in /home/nickanderson/.cfagent/inputs/lib/files.cf:2054 body changes detect_all_change # @brief Detect all file changes using the best hash method # # This is fierce, and will cost disk cycles # { hash => "best"; report_changes => "all"; update_hashes => "yes"; } ``` We can see from the definition that it will use the best available hashing algorithm (`md5` on community, `sha512` on enterprise) to hash the files content, that it will report _all_ changes (options include `all`, `stats`, `content`, `none`) and that it will update hashes. This update of hashes is something that is common in CFEngine use patterns but atypical for other File Integrity Monitoring tools. Setting the value of `update_hashes` to false will result in CFEngine warning loudly about the differences each time it evaluates the file for change until the database is updated. The attribute `report_diffs` which is not included in `files.cf` above is specific to CFEngine Enterprise and defaults to `false`. When enabled, diffs are reported, letting us visualize the change(s) in Mission Portal. ## A practical example Now that we know how to express the desire to monitor a file for change, let's check out how it functions in practice. I added the following example policy which monitors `/etc/motd` for changes if it exists and emits information in `inform_mode` (when `cf-agent` is run with `-I` , `--inform`, or `--log-level info`, which is typically only when run manually): ```cfengine3 bundle agent example_fim { files: "/etc/motd" changes => default:my_fim, classes => default:results( "bundle", "FIM_etc_motd"), if => fileexists( "/etc/motd" ); reports: inform_mode.FIM_etc_motd_reached:: "Promise to monitor /etc/motd for change was reached"; inform_mode.FIM_etc_motd_repaired:: "Promise to monitor /etc/motd for change was repaired"; } body changes my_fim # @brief Detect file content changes using sha256 # and report the diff to CFEngine Enterprise { hash => "sha256"; report_changes => "all"; report_diffs => "true"; update_hashes => "yes"; } ``` Running the policy manually we can see what the output looks like when a file is first noticed: ```text $ sudo cf-agent -KIf update.cf; sudo cf-agent -KI info: Stored sha512 hash for '/etc/motd' (SHA=18edda964ea45642cb610982df47e34e1d6fd1e23f7a1c811305902c90ca53e7dda634ed51945c080aca93fd8122768835328a94c08f72809620c80826bd4a8a) info: Wrote stat information for '/etc/motd' to database R: Promise to monitor /etc/motd for change was reached R: Promise to monitor /etc/motd for change was repaired ``` Since the file changes information was updated it's seen as a promise repaired. Subsequent runs of the policy simply indicate the promise was reached but no change was detected: ```text $ sudo cf-agent -KIf update.cf; sudo cf-agent -KI R: Promise to monitor /etc/motd for change was reached ``` ## Modifications are reported Now, I'll rewrite the content of `/etc/motd` and run the policy again to see what the output looks like when a change is observed: ```text sudo bash -c 'cat << EOF > /etc/motd You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only. By using this IS (which includes any device attached to this IS), you consent to the following conditions: -The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations. -At any time, the USG may inspect and seize data stored on this IS. -Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose. -This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not for your personal benefit or privacy. -Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details. EOF' $ sudo cf-agent -KIf update.cf; sudo cf-agent -KI info: Hash 'sha256' for '/etc/motd' changed! info: Updated sha256 hash for '/etc/motd' (SHA=998e7013530b712ebfc901d5e3279815b08534637dd52d080f7419ac042b2aa8) info: Recorded integrity changes in '/etc/motd' info: Reported on file changes to '/etc/motd' info: Copied file '/etc/motd' to '/etc/motd_cfchanges.cfnew' (mode '600') info: Backed up '/etc/motd_cfchanges' as '/etc/motd_cfchanges.cfsaved' info: Moved '/etc/motd_cfchanges.cfnew' to '/etc/motd_cfchanges' notice: Last modified time for '/etc/motd' changed 'Mon Dec 12 20:36:41 2022' -> 'Mon Dec 12 20:43:25 2022' info: Recorded mtime changes in '/etc/motd' info: Wrote stat information changes for '/etc/motd' to database R: Promise to monitor /etc/motd for change was reached R: Promise to monitor /etc/motd for change was repaired ``` ## Examine reports and details in Mission Portal After a few minutes we can visit the **File integrity monitoring** report in Mission Portal: {{< wideimg "images/fim-reports.png" "Accessing the File integrity monitoring report">}} {{< wideimg "images/fim-report.png" "File integrity monitoring report">}} Next we can adjust the filters to show only the file we are interested in: {{< img "images/fim-filter-etc-motd.png" "Filter to limit files to /etc/motd">}} And we can adjust the types of change we wish to see, for example let's show only file integrity monitoring reports that contain `DIFF`: {{< img "images/fim-filter-change-type.png" "Filter to limit change type to DIFF">}} And finally, hovering over the **Change details** we are offered the opportunity to show the diff: {{< img "images/fim-change-details-show-diff.png" "Hover over change details column to expose show diff button" >}} Clicking the button we see a diff of the changes made to the file: {{< img "images/fim-diff.png" "Diff of change to /etc/motd">}} [^fn:1]: CISOfy Lynis FINT-4350 suggests AIDE, Samhain or Tripwire: [^fn:2]: STIGs require a file integrity tool: [^fn:3]: The SCAP Security Guide calls for AIDE to be installed: -------------------------------------------------------------------------------- title: "Security holiday calendar - Part 1" url: /blog/2022/security-holiday-calendar-part-1/index.md date: "2022-12-12" -------------------------------------------------------------------------------- As it was well received [last year](/blog/2021/security-hardening-holiday-calendar-week-4), we decided to do another security-focused holiday calendar this year. The concept was roughly the same, but instead of only adding security hardening modules, we've also added in some other security advice and blog posts to improve the variety. Now that we're halfway through to 24 (or 25), let's recap the first half of the calendar. {{< wideimg "images/day-0-square.png" "Holiday / Christmas themed calendar with 25 boxes in the shape of a christmas tree.">}} ## The problematic remote shell (rsh) (1/25) [Remote shell (`rsh`)](https://en.wikipedia.org/wiki/Remote_Shell) allows you to log in and send commands to another computer over the network. It is notoriously insecure, sending traffic in an unencrypted manner. In some implementations of `rsh`, passwords are also sent over the network in plaintext. `rsh` should no longer be used, as much more secure alternatives exist, such as [`ssh`](https://en.wikipedia.org/wiki/Secure_Shell). This module helps you uninstall `rsh`: [build.cfengine.com/modules/uninstall-rsh-server](https://build.cfengine.com/modules/uninstall-rsh-server) **Recommendation:** Do not have the `rsh-server` module installed on any of the machines in your infrastructure, as it presents a substantial security risk. Use CFEngine Enterprise's software inventory to verify that it isn't installed anywhere, and this module to enforce it (uninstalling the module if it appears somewhere). This ensures you are protected in the future, if somebody installs the package on a machine, or if somebody adds a machine with the package installed to the infrastructure. ## Updates, upgrades, and uptime (2/25) On December 2nd, we discussed updates to software packages as well as operating systems, how you can find hosts which need attention and why `uptime` can help you look for the most neglected hosts in your infrastructure: [cfengine.com/blog/2022/updates-upgrades-and-uptime](https://cfengine.com/blog/2022/updates-upgrades-and-uptime) **Recommendations:** * Ensure you are running recent operating systems and at the very least operating systems supported by the vendor. * Use automation to help you effectively upgrade software packages. * Set up reports / alerts to help highlight which hosts need attention and prioritize which hosts to work on first. ## Limit access to /etc/motd (3/25) In unix-like systems the `/etc/motd` file is used to display relevant information to users after logging in. When a user logs in, it is quite common that they get information about the machine, where to find documentation, who to contact if you need help, etc. Since users may trust and rely on the information there, it is important to prevent misuse by limiting access. Possible attack vectors include telling users to run a specific command or give them a wrong (malicious) web address or email address for help. This module enforces access to `/etc/motd`: [build.cfengine.com/modules/etc-motd-access](https://build.cfengine.com/modules/etc-motd-access) **Recommendation:** Limit access to `/etc/motd` by setting its owner and group to `root`, and permission bits to `0644`. ## No longer talking (4/25) The `talk` program allowed users on early unix systems in the 80's and 90's to chat with each other. As it does not encrypt communication, and has been proven vulnerable in several ways, its use today is discouraged. The best approach is to uninstall it: [build.cfengine.com/modules/uninstall-talk](https://build.cfengine.com/modules/uninstall-talk) **Recommendation:** Do not use `talk` as it is insecure (provides no encryption for communication). Uninstall both `talk` and `talk-server`, and ensure it isn't used / installed in your infrastructure. ## Address space layout randomization (ASLR) (5/25) [Address space layout randomization (ASLR)](https://en.wikipedia.org/wiki/Address_space_layout_randomization) helps prevent exploits by randomly changing the address at which programs are located. Without ASLR an exploit can rely on the location of various aspects of a program and more easily perform an attack. This feature was first introduced around 2005 in Linux and subsequently in most other operating systems in the following years. 64-bit systems benefit more from this feature because the larger address space allows for greater entropy in randomizing the locations of programs. At this time there are known exploitations even of this protection mechanism but it is still best practice to ensure this feature is fully enabled on your systems. [build.cfengine.com/modules/enable-aslr](https://build.cfengine.com/modules/enable-aslr) **Recommendation:** Ensure that ASLR is fully enabled on all systems to make exploits relying on address space layout more difficult. ## Delete .shosts files in home directories (6/25) [Host-based authentication with ~/.shosts](https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication) is the equivalent to `/etc/ssh/.shosts.equiv`. It allows individual users to define a list of trusted remote machines, or user-machine pairs, which are allowed to try host-based authentication. However a global `.shosts.equiv` is preferable to having `.shosts` in each and every home directory. This module helps you delete the undesirable files: [build.cfengine.com/modules/delete-home-dotshosts](https://build.cfengine.com/modules/delete-home-dotshosts) **Recommendation:** Do not use user defined host based authentication. Use this `delete-home-dotshosts` module to inventory and delete `~/.shosts` files. This ensures you are protected in the future, if somebody adds a `~/.shosts` file. ## Limit access to /etc/issue (7/25) In unix-like systems the `/etc/issue` file is used to display relevant information to users before logging in. Since users may trust and rely on the information there, it is important to prevent misuse by limiting access. Possible attack vectors include telling users to run a specific command or give them a wrong (malicious) web address or email address for help. This module helps you enforce strict access to `/etc/issue` across your infrastructure: [build.cfengine.com/modules/etc-issue-access](https://build.cfengine.com/modules/etc-issue-access) **Recommendation:** Limit access to `/etc/issue` by setting its owner and group to `root`, and permission bits to `0644`. ## Unshadowed passwords in /etc/passwd (8/25) The `/etc/passwd` file contains information about local users on the system. This file is readable by different users on the system to allow them to look up usernames and translate between UIDs and usernames. The `/etc/shadow` file was introduced to prevent users from seeing each other's password hashes (thus prevent them from performing an offline brute force attack). Use this inventory module to add reporting data for users which don't use shadowed passwords, as they present a security risk: [build.cfengine.com/modules/inventory-unshadowed-users](https://build.cfengine.com/modules/inventory-unshadowed-users) **Recommendation:** In the `/etc/passwd` file, only `x` should be allowed in the second column (password field). This indicates that the `/etc/shadow` file is used. Any other users ("unshadowed" users) should be deleted or forced to change password. ## Building compliance reports (9/25) On December 9th, Craig Comstock wrote about using modules which provide reporting data (called inventory) to create custom compliance reports: [cfengine.com/blog/2022/compliance-report-based-on-inventory-modules](https://cfengine.com/blog/2022/compliance-report-based-on-inventory-modules) **Recommendation:** Build custom Compliance Reports to check the reporting data from your hosts, ensuring they are compliant with your security requirements. Update and expand these reports over time as your infrastructure grows and security requirements change. ## Access to cron-related files (10/25) The `cron` utility enables running commands or programs periodically. System-wide `cron`-related configuration files exist in the `/etc` directory. For these files, enforcing strict access is essential, as editing them allows you to run commands as any user, including `root`. This module continuously enforces owners, groups and permission bits on cron-related files and directories: [build.cfengine.com/modules/cron-access](https://build.cfengine.com/modules/cron-access) **Recommendation:** Limit access to the various `cron`-related files and directories in `/etc`. All of them should be owned by `root` and have group set to `root` as well. For the directories, permissions should be `0700`. The `/etc/crontab` file should have permissions `0600`, and `/etc/cron.allow` can have slightly less restrictive permissions, `0640`. ## Use AIDE for integrity checking (11/25) The [AIDE (Advanced Intrusion Detection Environment)](https://aide.github.io/) software enables integrity checking of installed software. It works by taking a snapshot of files, storing metadata about them (including hashes), and then comparing with the known values later, when you perform a "check". When software / files change unexpectedly, this can indicate an attacker is making changes to the system (Intrusion Detection). [build.cfengine.com/modules/install-aide](https://build.cfengine.com/modules/install-aide) **Recommendation:** Install and use AIDE to detect changes to installed software, potentially from attackers. Run the check at least once per week, and set it up to alert you (via email) when there are unexpected changes. ## Subscribe to the calendar We are just halfway through the calendar, there's about 12 more days left, so it's not too late to sign up. To receive daily security recommendations through Dec. 25th, [sign up here](/calendar). -------------------------------------------------------------------------------- title: "Building a Compliance Report based on inventory modules" url: /blog/2022/compliance-report-based-on-inventory-modules/index.md date: "2022-12-09" -------------------------------------------------------------------------------- In CFEngine Enterprise we collect information from each system in the infrastructure as [inventory](https://docs.cfengine.com/docs/3.18/enterprise-cfengine-guide-reporting-reporting_ui.html#inventory-management). Some inventory is available by default, and more can be added using modules or writing policy. You can use inventory information to create a Compliance Report with checks that determine if the information complies with your security requirements. In this blog post, we will use some modules from [CFEngine Build](https://build.cfengine.com/) which provide inventory data, and build a Compliance Report on top of those. ## Add some modules Assuming you have a hub with a [nightly build](https://cfengine.com/blog/2020/cfengine-nightlies/) and have created a [Build Project](https://cfengine.com/blog/2022/cfengine-3-20-released-modularity/), let's add some modules which collect inventory: - [enable-aslr](https://build.cfengine.com/modules/enable-aslr/) - Address space layout randomization (ASLR) - collects inventory about the setting on each host. - [delete-home-dotshosts](https://build.cfengine.com/modules/delete-home-dotshosts/) - collects inventory about hosts with `~/.shosts` files and those hosts which have been given an exception to automated deletion of such files. - [inventory-unshadowed-users](https://build.cfengine.com/modules/inventory-unshadowed-users/) - collects inventory about users in `/etc/passwd` which aren't using `/etc/shadow`. - [lynis](https://build.cfengine.com/modules/lynis/) - uses CISofy's lynis system audits - we will add a check for their Hardening Index - an overall number to gauge the health of a system - [inventory-local-users-non-root-has-uid-0](https://build.cfengine.com/modules/inventory-local-users-non-root-has-uid-0/) - which users have a uid of 0? - [inventory-yum-update-info](https://build.cfengine.com/modules/inventory-yum-update-info/) - information about security updates from `yum updateinfo` We deploy this policy to our infrastructure and wait 5-10 minutes for the information to be available so we can create our new Compliance Report. ## Exceptions to the Rule Many of these modules include providing for exceptions to the policy. To allow the administrator to skip enforcing policy on specific hosts. For example, the `delete-home-dotshosts` module asks for a variable named `delete_home_dotshosts:main.exception` to be defined for a host to make an exception: {{< img "images/crboim-shosts-exception.png" "Adding exception for ~/.shosts enforcement">}} We add an exception for one host for the `enable-aslr` module as well: {{< img "images/crboim-aslr-exception.png" "Adding exception for ASLR enforcement">}} ## Break things Now in order for us to see some problems we need to break some things. I used [cf-remote](https://github.com/cfengine/cf-remote) to provision some instances in the cloud and now will use it again to break things: #### **`break-things.sh`** ```bash #!/usr/bin/bash set -e clients=ubuntu-22-04-arm64,debian-11-arm64,rhel-8-x64,centos-7-x64 cf-remote sudo -H $clients "sysctl -w kernel.randomize_va_space=0" cf-remote run -H $clients "touch .shosts" cat << EOF > bad emptyfield::9991:9991:testa:/home/emptyfield:/sbin/nologin starfield:*:9992:9992:testb:/home/starfield:/sbin/nologin EOF cf-remote scp -H $clients bad cf-remote sudo -H $clients "cat bad >> /etc/passwd" cat << EOF > bad2 toor:x:0:0:toor:/home/toor:/sbin/nologin EOF cf-remote scp -H $clients bad2 cf-remote sudo -H $clients "cat bad2 >> /etc/passwd" ``` ## Inventory report At this point I run an inventory report and add some appropriate columns to get an overview of the results: {{< wideimg "images/crboim-inventory-report.png" "Inventory report showing data collected by modules">}} We see that the Ubuntu 22 host has `Disabled (0)` for an ASLR value. This is because we added the exception to not enforce our policy on that host. ## Create a Compliance Report From the **Reports** app in Mission Portal, select the **Compliance** section and **Create a report**. {{< wideimg "images/crboim-compliance-report-section.png" "Compliance Report section">}} The report is created and you can start editing it (adding checks): {{< img "images/crboim-new-compliance-report.png" "Editing a new compliance report">}} ## Add a check for the `enable-aslr` module Checks can be based on several different types of information: Policy, Inventory, Software Updates, File Changes or Custom SQL. You can add multiple rules to your check and for each rule can compare the value in various ways such as less-than, greater-than, matches, is or is not reported. Your checks can be grouped together in the report by Category, such as Security, Hardware or whatever custom category you need. Finally, you can filter the hosts involved in this Check by Host tree group, hostname, IP address or CFEngine classes. The first check I made, checks that Address space layout randomization (ASLR) is enabled, and looks like this: {{< wideimg "images/crboim-check-for-alsr.png" "Editing check for ASLR inventory">}} ## Add more checks Similar to the check above, I created a few more for the other modules: {{< wideimg "images/crboim-edit-report.png" "Editing Security Compliance Report">}} ## A shiny new Compliance Report! After saving the checks above, we have a functioning Compliance Report: {{< wideimg "images/crboim-result.png" "Resulting Security Compliance Report">}} This highlights the state of our infrastructure, showing overall compliance as well as passing / failing host counts and percentages per check. When individual checks are failing, we can click the icon on the right to drill down and find the individual hosts. **Security recommendation:** Build custom Compliance Reports to check the reporting data from your hosts, ensuring they are compliant with your security requirements. Inventory data, checks and compliance reports are fully customizable. You can write policy or use [modules from CFEngine Build to add additional inventory data](https://build.cfengine.com/modules/?query=inventory). Update and expand these reports over time as your infrastructure grows and security requirements change. -------------------------------------------------------------------------------- title: "Updates, upgrades, and uptime" url: /blog/2022/updates-upgrades-and-uptime/index.md date: "2022-12-02" -------------------------------------------------------------------------------- All software of any significant size has bugs, vulnerabilities, and other weaknesses. This includes the operating system (OS), libraries, command line tools, services and graphical applications. Across your infrastructure, you should have an overview of what operating systems and software you have installed. Additionally, automated ways of upgrading the OS, as well as packages are desirable. Finally, ways of highlighting problematic hosts (with old operating systems and software) and prioritizing them helps your efforts to upgrade and secure your machines. Let's take a look at some ways which CFEngine can help in these areas. ## The dashboard From the first time you [install CFEngine Enterprise](https://docs.cfengine.com/docs/master/getting-started-installation.html) and open up Mission Portal, its Web UI, you get some useful information: {{< img "images/dashboard-os-and-software.png" "Pie chart of OSes and alert widget indicating software updates available">}} These widgets, available on the default dashboard, show us our distribution of OSes, and an alert that software updates are available. ### Operating systems Looking at the OS pie chart widget, we can already see some older operating systems. Through some quick googling, I find that Debian 9 and Ubuntu 16 are no longer supported, so we should upgrade them. If you'd like CFEngine to quickly highlight all hosts running unsupported operating systems, there is a module with a compliance report for that: https://build.cfengine.com/modules/compliance-report-os-is-vendor-supported/ ### Software updates The other widget, titled **Software updates**, indicates that all of our machines have software updates available. The data here is based on the systems package manager - `yum` on Red Hat based systems, and `apt` on Debian based systems. By clicking on the widget and then the alert inside of it, we get an overview of how the alert works, along with the hosts affected and how many updates are available for each one: {{< wideimg "images/updates-available.png" "Screenshot with Alert details, lists of hosts and number of updates available">}} Drilling further down, by clicking on individual hostnames, we can see a report with the exact software and versions available for each machine. Alerts, widgets, and reports are fully customizable, you can take this one step further and highlight only specific packages which you deem important. At this point, we could start upgrading packages with `yum`: ``` $ yum update ``` Or `apt`: ``` $ apt update && apt upgrade ``` However, this can be automated with CFEngine. You can do this in policy, or use this module: https://build.cfengine.com/modules/upgrade-all-packages/ ## Inventory reports An inventory report highlights your inventory of hosts (assets), with different reported information about each of them. Use the left navigation bar, click **Reports**, then **Inventory** to open one: {{< img "images/reports-inventory-nav.png" "Left navigation bar with Reports selected and sub menu for Inventory">}} Once inside the inventory report, you can customize what information is shown by clicking the 3 vertical dots, then adding and removing columns: {{< img "images/add-remove-columns.png" "UI showing a dropdown with options to add and remove columns">}} ### Uptime A system's "uptime" (as indicated by the `uptime` command) tells you how long the system has been "up", i.e. how long it's been since the last reboot. Since upgrades, especially OS upgrades, typically require a reboot, looking for machines with high uptime can help you identify machines which need attention (upgrades). In the example below, I removed the column of available memory, and added columns for uptime and Kernel release (versions of the linux kernel in this case), and then clicked on Uptime minutes to sort based on that column: {{< wideimg "images/inventory-os-uptime-kernel.png" "Table showing OS, Uptime minutes and Kernel release version for all hosts">}} The sorting allows us to see the oldest rebooted machines first. In this case, 108 minutes is pretty good, so no action needed. If you have machines with months or even years of uptime, it might be time for an upgrade and reboot(!). ## Security recommendations Hopefully this guide has given you some ideas on ways to improve the security of your servers and other devices. We recommend that you: * Ensure you are running recent operating systems and at the very least operating systems supported by the vendor. * Use automation to help you effectively upgrade software packages. * Set up reports / alerts to help highlight which hosts need attention and prioritize which hosts to work on first. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 19 - Sneak peek of CFEngine 3.21" url: /blog/2022/the-agent-is-in-episode-19-sneak-peek-at-3-21-lts/index.md date: "2022-11-22" -------------------------------------------------------------------------------- The next LTS is coming ... Join Cody Valle, Craig Comstock, Nick Anderson, and Ole Herman Elgesem for a preview of the coming in CFEngine 3.21. ## Video {#video} The video recording is available on YouTube: {{< youtube SCbZ-M7XFN4 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) - [CFEngine 3.15.7 & 3.18.3 released (November 14th)](https://cfengine.com/blog/2022/cfengine-3-15-7-and-3-18-3-released/) -------------------------------------------------------------------------------- title: "CFEngine Build System version 3" url: /blog/2022/cfbs-3-0-0/index.md date: "2022-11-15" -------------------------------------------------------------------------------- Our beloved `cfbs` CLI tool for working with CFEngine Build is rapidly evolving. At the time of writing, we are currently at version 3.2.1. Thus I would like to take this opportunity to talk a bit about the latest and greatest features; including support for users to manipulate input parameters in modules, as well as a couple of new build steps. If you haven't yet got a hold of the latest version of `cfbs`, you can update it with pip using the following command: ``` $ pip3 install --upgrade cfbs Collecting cfbs Using cached cfbs-3.2.1-py3-none-any.whl (46 kB) Installing collected packages: cfbs Successfully installed cfbs-3.2.1 ``` We'll start off with a simple CFEngine Build project containing only the default master policy files. Note that I have disabled git, in order to make the examples a bit less noisy. ``` $ cfbs status Name: My awesome CFEngine Build project Description: A project to demo new features in cfbs 3 File: cfbs.json Modules: 001 masterfiles @ a87b7fea6f7a88808b327730a4ba784a3dc664eb (Not downloaded) ``` ## New build steps Build steps are usually added by module developers or advanced `cfbs` users in order to manipulate temporary files while building the policy set. An overview of all available build steps - and what they do - can be found in the `cfbs` repository [here](https://github.com/cfengine/cfbs/blob/master/JSON.md#all-available-build-steps). The build steps `policy_files`, `bundles` and `input` were added in `cfbs` version 3. The two former provide a more powerful way in which module developers can choose which policy files and bundles shall be evaluated by the agent during an agent run. This is in contrast to the autorun functionality that was previously used. While the latter build step is used to generate and merge augments from user input. ## Adding local policy files The main motivation for adding the `policy_files` and `bundles` build steps, was to improve the ease of use when it comes to adding local policy files for new CFEngine users. Previously bundles in local policy files had to be tagged with `"autorun"` using the `meta` promise type in order for the bundle to be evaluated. See example below: ```cf3 # policy.cf bundle agent my_bundle { meta: "tags" slist => { "autorun" }; reports: "Hello from $(this.bundle)"; } ``` ``` $ cfbs add policy.cf Added module: autorun (Dependency of ./policy.cf) Added module: ./policy.cf ``` Adding a local policy file should be a primitive operation that can be used as a beginner example for new CFEngine users. Thus having to explain the autorun feature is undesirable. With the powers of our new build steps, `cfbs` version 3 is now able to ask you which bundle to run instead of users having to implicitly state this into their bundles. ```cf3 # policy.cf bundle agent my_bundle { reports: "Hello from $(this.bundle)"; } ``` ``` $ cfbs add policy.cf Which bundle should be evaluated (added to bundle sequence)? 1. ./policy.cf:my_bundle (default) 2. (None) [1/2]: 1 Added module: ./policy.cf ``` Let's see what this looks like if we have multiple bundles spread over multiple policy files. Given the two policy files below, we can add `bogus/` as a local module directory. The CLI will browse the module directory looking for candidate bundles for you to choose from. I'll choose the `foobar` bundle. ```cf3 # bogus/bogus.cf bundle agent foo { reports: "Hello from $(this.bundle)"; } bundle agent bar { reports: "Hello from $(this.bundle)"; } bundle agent baz { reports: "Hello from $(this.bundle)"; } ``` ```cf3 # bogus/doofus/doofus.cf bundle agent foobar { reports: "Hello from $(this.bundle)"; } bundle agent foobaz { reports: "Hello from $(this.bundle)"; } ``` ``` $ cfbs add bogus/ Which bundle should be evaluated (added to bundle sequence)? 1. ./bogus/bogus.cf:foo (default) 2. ./bogus/bogus.cf:bar 3. ./bogus/bogus.cf:baz 4. ./bogus/doofus/doofus.cf:foobar 5. ./bogus/doofus/doofus.cf:foobaz 6. (None) [1/2/3/4/5/6]: 4 Added module: ./bogus/ ``` If we build the project we can see the build steps in action: ``` $ cfbs build Modules: 001 masterfiles @ a87b7fea6f7a88808b327730a4ba784a3dc664eb (Downloaded) 002 ./policy.cf @ local (Copied) 003 ./bogus/ @ local (Copied) Steps: 001 masterfiles : run './prepare.sh -y' 001 masterfiles : copy './' 'masterfiles/' 002 ./policy.cf : copy './policy.cf' 'masterfiles/services/cfbs/policy.cf' 002 ./policy.cf : policy_files 'services/cfbs/policy.cf' 002 ./policy.cf : bundles 'my_bundle' 003 ./bogus/ : directory './' 'masterfiles/services/cfbs/bogus/' 003 ./bogus/ : policy_files 'services/cfbs/bogus/bogus.cf' 'services/cfbs/bogus/doofus/doofus.cf' 003 ./bogus/ : bundles 'foobar' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: sudo cfbs install To deploy on remote hub(s): cf-remote deploy ``` By installing the policy set, followed by running the agent, we can see that the correct bundles are in fact run. ``` $ sudo cfbs install Installed to /var/cfengine/masterfiles $ sudo cf-agent --no-lock --file update.cf $ sudo cf-agent --no-lock R: Hello from my_bundle R: Hello from foobar ``` Pretty neat, right? ## Adding user input The greatest new feature by far is nothing less than the added support for user input in modules. This new feature allows for module developers to specify prompts that allow users to customize module parameters. User inputs can be added using the `cfbs input` command, or during `cfbs add`. Input is stored as JSON in the project repository, and can be converted into augments and merged into the policy framework using the `input` build step. An example of a module that currently uses this feature, is the `delete-files` promise module, written by none other than the notorious [CFEngine ninja Nick Anderson](https://github.com/nickanderson). Let's check it out! ``` $ cfbs add delete-files Added module: delete-files The added module 'delete-files' accepts user input. Do you want to add it now? [yes/y/NO/n] yes Collecting input for module 'delete-files' Path to file: /tmp/virus Why should this file be deleted? [Unknown] We don't want it to do malicious stuff on our host. Specify another file you want deleted on your hosts? [yes/y/NO/n] no ``` We build and install using `cfbs`, followed by running the agent with the `update.cf` policy, just like we did in the previous example. Then we create a "good" file and a "bad" one in the `/tmp/` directory. Running the agent now makes sure the bad file is deleted. ``` $ echo "Malicious code" > /tmp/virus $ echo "Important stuff" > /tmp/important $ sudo /var/cfengine/bin/cf-agent --no-lock --inform info: Deleted file '/tmp/virus' ``` ## A sneak peek We are currently working hard on integrating the new input feature into the CFEngine Enterprise Web UI, Mission Portal. Without revealing too much of what is coming next, here is a little sneak peek on what we can expect. {{< wideimg "images/cfengine-build-sneak-peek.png" "CFEngine Build sneak peek">}} ## Share your thoughts If you have any ideas of new features or tips on how we can improve CFEngine Build? Please share your thoughts over at [GitHub Discussions](https://github.com/cfengine/core/discussions), we are eager to hear your feedback. -------------------------------------------------------------------------------- title: "CFEngine 3.15.7 and 3.18.3 released" url: /blog/2022/cfengine-3-15-7-and-3-18-3-released/index.md date: "2022-11-14" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.15.7 and 3.18.3\! These releases mainly contain bug fixes and dependency updates. ## 3.15: Last release and end of life 3.15.7 is the last planned release for the 3.15 LTS series, which is supported until December 2022. Please reach out to support if you need help with upgrading or need to purchase extended support; on January 1st 2023, 3.15 is no longer supported. ## What's new As these are patch releases, there isn't a lot of new functionality added or big changes introduced. Compliance reports and their exported PDFs have been significantly improved in 3.18.3: [![](./images/example-compliance-report.png)](./images/example-compliance-report.png) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.15.7 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [3.15.7 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.15.7 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.18.3 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.3 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.3 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. | CFEngine version | 3.15.6 | **3.15.7** | 3.18.2 | **3.18.3** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.53 | **2.4.54** | 2.4.53 | **2.4.54** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.36.1 | **2.38.1** | 2.36.1 | **2.38.1** | | [libcurl](https://curl.haxx.se/download.html) | 7.83.1 | **7.86.0** | 7.83.1 | **7.86.0** | | [libiconv](https://www.gnu.org/software/libiconv/) | 1.16 | **1.17** | 1.16 | **1.17** | | [OpenSSL](https://www.openssl.org/) | 1.1.1l | **1.1.1q** | 1.1.1l | **1.1.1q** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.4.58 | 2.4.58 | 2.6.1 | **2.6.3** | | [PHP](https://php.net/) | 7.4.29 | **7.4.32** | 8.0.19 | **8.0.24** | | [PostgreSQL](https://www.postgresql.org/) | 12.11 | **12.12** | 13.7 | **13.8** | | [rsync](https://rsync.samba.org/) | 3.2.4 | **3.2.7** | 3.2.4 | **3.2.7** | | [SASL2](https://github.com/cyrusimap/cyrus-sasl/) | 2.1.27 | **2.1.28** | 2.1.27 | **2.1.28** | | [zlib](https://zlib.net/) | 1.2.12 | **1.2.13** | 1.2.12 | **1.2.13** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. At the time of writing, version 3.18.3 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). (This will change when 3.21.0 LTS is released, before the end of the year). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.15.7 list ``` ```command cf-remote --version 3.18.3 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "November 2022: Severe vulnerabilities in OpenSSL 3" url: /blog/2022/november-2022-severe-vulnerabilities-in-openssl-3/index.md date: "2022-11-01" -------------------------------------------------------------------------------- On October 25th 2022 the OpenSSL project team announced [^fn:1] the forthcoming release of OpenSSL version 3.0.7. From the announcement we know that a fix will be made available on Tuesday November 1st, 2022 for a CRITICAL security issue. **Note:** CVE-2022-3786 and CVE-2022-3602 (X.509 Email Address Buffer Overflows) have been published [^fn:2]. CVE-2022-3602 originally assessed as CRITICAL was downgraded to HIGH after further review prior to being published. ## Affected versions The vulnerability is reported to affect version 3.0.x and does not impact OpenSSL 1.1.1 or LibreSSL[^fn:3] [^fn:4] [^fn:5]. The first stable version of OpenSSL 3.0 was released in September 2021. Older operating systems are likely using OpenSSL 1.1.1, which is not affected. ## CFEngine's use of OpenSSL CFEngine includes OpenSSL as a part of its packages. No supported releases of CFEngine are using the affected OpenSSL versions. CFEngine 3.20.0 (a non-LTS release) is the only version of CFEngine released built with OpenSSL 3.0.x. It was released July 1st, 2022 with OpenSSL 3.0.4[^fn:6]. ## Mitigation We encourage you to read through the official OpenSSL team blog post for detailed information about the vulnerabilities: https://www.openssl.org/blog/blog/2022/11/01/email-address-overflows/ You will want all hosts and software projects using OpenSSL 3 upgraded to version 3.0.7 (or an appropriately patched version for your distribution, for example 3.0.5-2 in Fedora 36). We recommend you start by identifying all machines with a vulnerable version of OpenSSL installed. Below, we have outlined 2 ways of doing this using CFEngine. Once you have an overview of all affected hosts, start rolling out updates, prioritizing the most critical / exposed parts of your infrastructure first. ### Using Custom reports to find OpenSSL version in software inventory If you are using CFEngine Enterprise you can create a custom report to find hosts with OpenSSL 3 installed. ```sql SELECT Hosts.HostName AS "Host name", Software.SoftwareName AS "Software name", Software.SoftwareVersion AS "Software version" FROM Software LEFT JOIN Hosts ON Hosts.HostKey = Software.HostKey WHERE Software.SoftwareName = 'openssl' AND Software.SoftwareVersion ILIKE '3%' ``` If you don't know where to enter SQL queries for custom reports: 1. Click on "Reports" in the left navigation bar. 2. Click on "Custom report". 3. Scroll down and click on "Show query". 4. Paste the query into the text field and click "Run". If you get 0 rows, it means OpenSSL 3 was not found anywhere. You can remove the last line of the query to see all versions of OpenSSL. You don't need to know SQL to make reports like these, this SQL query was made using the report builder GUI. Instead of pasting in the SQL query, you can fill out the appropriate checkboxes and fields: [![](images/report-builder-openssl-3.png)](images/report-builder-openssl-3.png) ### A CFEngine Build module to look for installed versions of OpenSSL We have released a new module to help identify OpenSSL versions installed: https://build.cfengine.com/modules/inventory-openssl-versions/ This module tries to look for OpenSSL versions in multiple places (it runs `openssl`, `yum`, `apt`, `brew`) and can detect when multiple versions are installed with different package managers. It puts the resulting list of versions in an inventory attribute, which can be added to an inventory report: {{< wideimg "images/inventory-openssl-versions-mp-screenshot.png" "Screenshot of inventory report (table) showing different OpenSSL versions installed on different machines">}} This is somewhat different from what is available in CFEngine out of the box ("software inventory" used above): * Software inventory relies on one chosen package manager for information about installed versions (`apt` on Ubuntu, `yum` on CentOS, and so on). * Installed software names and versions are not normally put in Inventory (Software inventory is separate). We hope you find one of these methods of identifying potentially vulnerable machines based on OpenSSL versions helpful, and that these vulnerabilities do not have a significant negative impact on your organization and software. [^fn:1]: The OpenSSL project team announcement of forthcoming release of OpenSSL 3.0.7 [^fn:2]: OpenSSL CVE announcement [^fn:3]: Sentinel One reports the vulnerability affects 3.0.x and does not impact OpenSSL 1.1.1 or LibreSSL [^fn:4]: SANS reports the update will only affect OpenSSL 3.0.x, not 1.1.1 [^fn:5]: Synk.io reports the vulnerability will not impact versions of OpenSSL prior to 3.0 [^fn:6]: CFEngine 3.20.0 release announcement -------------------------------------------------------------------------------- title: "Scary stories you won't believe until they happen to you!" url: /blog/2022/halloween/index.md date: "2022-10-27" -------------------------------------------------------------------------------- For halloween this year, we wanted to share some scary scenarios along with security recommendations to help avoid them. All the names, companies and characters are made up, but the events and experiences are based on things which could happen, or have happened in the real world. {{< wideimg "images/halloween_wide.png" "Halloween terrors coming true in IT security">}} ## 1. Horrors of the logging library {#number-one} Mary the sysadmin looks over at her monitoring system, noticing an increase in requests with special characters. She recognizes the strings as log4shell vulnerability exploit attempts. Months earlier, when the vulnerability first appeared, she concluded they were safe, since the vulnerability was in a Java library. She was wrong. One machine goes offline, then another. She tries to look online for scanners, but it's already too late. Slowly, one by one, the attackers succeed, they are remotely executing code and bringing down her entire datacenter. **Recommendation:** Having up-to-date and complete software inventory and automated ways of running scans and patching across your hosts is essential to reacting to vulnerabilities. CFEngine provides extendable software inventory out of the box, and you can use modules to scan for vulnerable hosts, such as the log4shell module: [cve-2021-44228-log4j](https://build.cfengine.com/modules/cve-2021-44228-log4j/). ## 2. SSH-ut out by mistake {#number-two} James gets a low disk space alert for one of his production web servers and logs in with SSH. He notices that the configuration for SSH is wrong: it allows root login and empty passwords. He quickly fixes it and moves on to the disk cleanup. Suddenly, his connection drops, and he tries to log back in, but SSH is not working! He must have made a mistake in the config and has locked himself out. All their management is based on SSH and he has no way back in to fix his error! The server runs out of space and the website starts showing error messages to users. He calls the datacenter to get someone to walk over to the server with a laptop, well aware that for every minute of downtime, they are losing thousands of dollars, and he is the one who will be blamed. **Recommendation:** Don't rely on SSH as a single point of failure - it is very risky. With CFEngine running on all your machines you can manage the SSH configuration, or just fix it if you've accidentally locked yourself out. For help with installing and using CFEngine for the first time, [see our getting started tutorials](https://docs.cfengine.com/docs/master/getting-started.html). ## 3. New boss in town {#number-three} Susan, the new CEO of DE-Corp, is tasked with turning the company around. She starts looking for ways to modernize and upgrade their IT systems, as well as reduce unnecessary cloud spending. For this, she needs David, head of IT, to provide reports of host counts across data centers, departments and cloud providers, as well as information on outdated operating systems. David talks to different managers and logs into multiple web UIs to collect the necessary data, but Susan starts noticing discrepancies in his reports, as it becomes obvious he does not have a good and complete overview. Ultimately, she finds out about hundreds of machines in AWS which are not being used and costing them thousands of dollars per month. David gets fired on the spot and the company starts looking for a new head of IT. **Recommendation:** With CFEngine Enterprise on all your machines, you get a complete overview of hosts, including operating systems, versions and other customizeable inventory data. To learn more about the benefits of our reporting, see this part of the getting started tutorials: [Reporting and Web UI](https://docs.cfengine.com/docs/master/getting-started-reporting-and-web-ui.html). ## 4. The rogue won {#number-four} One of the senior admins, John, is unhappy with the direction of the company under new management. He hands in his resignation and goes home, but continues arguments with his now ex-boss and colleagues over email. As the arguments escalate he loses his temper and decides to take revenge. During the night he first goes through the company's backup servers, then production servers, deleting as much data as possible, and then issuing a reboot for good measure. The next day, all of the company's systems are offline and it takes them months to recover. Their reputation is damaged beyond repair. **Recommendation:** Use CFEngine's powerful automation to manage access keys on all machines, allowing you to quickly revoke someone's access when necessary. See this example policy to get started: [cfengine-example-ssh-management](https://github.com/olehermanse/cfengine-example-ssh-management). ## 5. The kiddo {#number-five} Railio, the hottest self-driving train startup in Silicon Valley is in the news again, but this time to CISO Kevin's dismay. A 13 year old boy shows how he hacked the train company's systems, using very basic exploits found online. This is just 1 month ahead of the planned first driverless voyage. The minor didn't harm the servers as he got in, but Kevin is concerned and embarrassed, as the brand damage is substantial. Their stock price plummets and the city of San Francisco cancels their contract. The worst part is, it could have been avoided very easily. **Recommendation:** Run basic security hardening on all your systems, enforcing secure configuration and detecting security issues. CFEngine can be used both for enforcement and detection, for example using the lynis module and compliance report: [compliance-report-lynis](https://build.cfengine.com/modules/compliance-report-lynis/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 18 - Policy examples" url: /blog/2022/the-agent-is-in-episode-18-policy-examples/index.md date: "2022-10-27" -------------------------------------------------------------------------------- Do you know how to use **every** function available in CFEngine? Join Cody, Craig, Herman to see how Nick uses org-mode, org-roam, and ob-cfengine3 to manage his personal collection of CFEngine Function Examples. ## Video {#video} The video recording is available on YouTube: {{< youtube RHUy9Hyis8Y >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Notes {#notes} Live coded examples for the following functions: * `regline( regex, filename )` * Returns whether the _anchored_ regular expression `regex` matches a line in `filename`. * `reglist( list, regex )` * Returns whether the anchored regular expression `regex` matches any item in `list`. * Very similar to `some( regex, list )` but it differs in reversed parameters and the regular expression is _unanchored_. * `parseyaml( 'inline-yaml' )` * Parses YAML data directly from a string and returns the result as a data variable. * Not to be confused with `readyaml()` which is for parsing a YAML _file_, a much more common use case. ### `regline()` {#regline} Look to see if quiet is a boot option. ```cfengine3 bundle agent __main__ { classes: "have_quiet_boot" expression => regline("$(anchored_regex)", $(file)); vars: # e.g. BOOT_IMAGE=/vmlinuz-5.15.0-52-generic root=/dev/mapper/vgkubuntu-root ro quiet splash vt.handoff=7 "anchored_regex" string => ".*\s+quiet\s*.*"; "file" string => "/proc/cmdline"; reports: have_quiet_boot:: "Have quiet in body cmdline args"; "$(file)" printfile => cat($(file)); } ``` ```text R: Have quiet in body cmdline args R: /proc/cmdline R: BOOT_IMAGE=/vmlinuz-5.15.0-52-generic root=/dev/mapper/vgkubuntu-root ro quiet splash vt.handoff=7 ``` ### `reglist()` {#reglist} Determine if at least one element of a list starts with `c` (case insensitive). ```cfengine3 bundle agent __main__ { vars: "names" slist => {"Cody", "Craig", "Herman", "Nick"}; # This regex is unanchored, because the function itself anchors it. "anchored_regex" string => "(?i)c.*"; reports: # Emit report if at least one element starts with c or C "$(this.namespace):$(this.bundle).names has at least one element matching '$(anchored_regex)'" if => reglist(names, $(anchored_regex)); "names: $(with)" with => join(", ", names); } ``` ```text R: default:main.names has at least one element matching '(?i)c.*' R: names: Cody, Craig, Herman, Nick ``` ### `parseyaml()` {#parseyaml} Demonstrate how to parse inline YAML and access different parts of the data structure. ```cfengine3 bundle agent __main__ { vars: "some_yaml" data => parseyaml(' mykey: myvalue domain: name: example.com subdomain: asdf nesting: levelzero: levelone: levelone'); "top_level_keys" slist => getindices(some_yaml); "nesting_keys" slist => getindices("some_yaml[nesting]"); "nesting_levelzero_keys" slist => getindices("some_yaml[nesting][levelzero]"); reports: "$(with)" with => storejson(some_yaml); } ``` ```text R: { "domain": { "name": "example.com", "subdomain": "asdf" }, "mykey": "myvalue", "nesting": { "levelzero": { "levelone": "levelone" } } } Variable name Variable value Meta tags Comment default:main.nesting_keys {"levelzero"} source=promise default:main.nesting_levelzero_keys {"levelone"} source=promise default:main.some_yaml {"domain":{"name":"example.com","subdomain":"asdf"},"mykey":"myvalue","nesting":{"levelzero":{"levelone":"levelone"}}} source=promise default:main.top_level_keys {"mykey","domain","nesting"} source=promise ``` **Note:** In the above example output, the `cf-agent` was executed with `--show-evaluated-vars=main\.` which outputs variables, their values, tags, and comments. ## Links {#links} - Org-roam A plain-text personal knowledge management system - Org-mode A GNU Emacs major mode for keeping notes, authoring documents, computational notebooks, literate programming, maintaining to-do lists, planning projects, and more - in a fast and effective plain text system - ob-cfengine3 Execute CFEngine policy within org-mode - Spacemacs a community driven Emacs distribution - CFEngine Spacemacs Layer - Documentation for `regline()` - Documentation for `reglist()` - Documentation for `some()` - Documentation for `parseyaml()` - Documentation for `readyaml()` - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Accessing CFEngine nightly packages" url: /blog/2022/nightly-packages/index.md date: "2022-10-12" -------------------------------------------------------------------------------- Did you know that nightly builds of CFEngine are available? `cf-remote` is the most convenient way to get nightly packages. If you're not familiar with it, or if you need a refresher, check out our [other blog posts about cf-remote](/tags/cf-remote/). ## Listing packages By default `cf-remote list` will emit a list of available releases and the URLs for the newest CFEngine Enterprise LTS release. ```command cf-remote list ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using 3.18.2 LTS (default): https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/debian_9_x86_64/cfengine-nova-hub_3.18.2-1.debian9_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/debian_10_x86_64/cfengine-nova-hub_3.18.2-1.debian10_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/debian_11_x86_64/cfengine-nova-hub_3.18.2-1.debian11_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/redhat_6_x86_64/cfengine-nova-hub-3.18.2-1.el6.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/redhat_7_x86_64/cfengine-nova-hub-3.18.2-1.el7.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/redhat_8_x86_64/cfengine-nova-hub-3.18.2-1.el8.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/ubuntu_16_x86_64/cfengine-nova-hub_3.18.2-1.ubuntu16_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/ubuntu_18_x86_64/cfengine-nova-hub_3.18.2-1.ubuntu18_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/hub/ubuntu_20_x86_64/cfengine-nova-hub_3.18.2-1.ubuntu20_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_debian9_x86_64/cfengine-nova_3.18.2-1.debian9_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_debian10_x86_64/cfengine-nova_3.18.2-1.debian10_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_debian11_x86_64/cfengine-nova_3.18.2-1.debian11_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_ubuntu16_x86_64/cfengine-nova_3.18.2-1.ubuntu16_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_ubuntu18_x86_64/cfengine-nova_3.18.2-1.ubuntu18_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_ubuntu20_x86_64/cfengine-nova_3.18.2-1.ubuntu20_amd64.deb https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_rhel6_x86_64/cfengine-nova-3.18.2-1.el6.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_rhel7_x86_64/cfengine-nova-3.18.2-1.el7.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_rhel8_x86_64/cfengine-nova-3.18.2-1.el8.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_suse11_x86_64/cfengine-nova-3.18.2-1.suse11.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_suse12_x86_64/cfengine-nova-3.18.2-1.suse12.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/agent_suse15_x86_64/cfengine-nova-3.18.2-1.suse15.x86_64.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/windows_i686/cfengine-nova-3.18.2-1-i686.msi https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/windows_x86_64/cfengine-nova-3.18.2-1-x86_64.msi https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/solaris_10_sparc/CFEcfengine-nova-3.18.2.1-solaris10-sparc.pkg https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/solaris_11_sparc/CFEcfengine-nova-3.18.2.1-solaris11-sparc.pkg https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/solaris_10_x86/CFEcfengine-nova-3.18.2.1-solaris10-i386.pkg https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/hpux/cfengine-nova-3.18.2.1-B.11.23-ia64.depot https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/aix_5_ppc/cfengine-nova-3.18.2-1.aix5.ppc.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/aix_5_ppc/cfengine.cfengine-nova-3.18.2.1.aix5.bff https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/aix_7_ppc/cfengine-nova-3.18.2-1.aix7.ppc.rpm https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/aix_7_ppc/cfengine.cfengine-nova-3.18.2.1.aix7.bff https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/agent/xfs_filesystem_image/cfengine-nova-3.18.2-1.x86_64.fs-img.pkg.tar.gz https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/misc/cfengine-masterfiles-3.18.2-1.pkg.tar.gz https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/misc/cfengine-nova-3.18.2-1.x86_64.pkg.tar.gz https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.2/misc/CFEngine_Enterprise_vagrant_quickstart-3.18.2-1.tar.gz ``` If you want to get a list of URLs for nightly packages from an LTS branch, specify the branch name as the version: ```command cf-remote --version 3.18.x list ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using 3.18.x LTS: http://buildcache.cfengine.com/packages/testing-pr/jenkins-3.18.x-nightly-pipeline-325/PACKAGES_HUB_x86_64_linux_debian_9/cfengine-nova-hub_3.18.3a.33cc458e4~22969.debian9_amd64.deb [...] ``` **Tip:** Replacing `list` with `download` will download all the packages in the list, for cached use in other commands or for you to use manually. (Filenames are printed, files are stored in `~/.cfengine/cf-remote/packages/`). ## Tags and filtering You can provide a space separated list of tags to filter the list. For example, if you want the URL for the very latest Enterprise Linux Hub package specify `master` as the version and add `el7 hub`. ```command cf-remote --version master list el7 hub amd64 ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using master: http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1113/PACKAGES_HUB_x86_64_linux_redhat_7/cfengine-nova-hub-3.21.0a.3c9610266-22964.el7.x86_64.rpm ``` ## Community Of course, you can get CFEngine Community packages this way as well, just specify `community` as the edition: ```command cf-remote --version master list --edition community el7 ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using master LTS: http://buildcache.cfengine.com/packages/testing-pr/jenkins-community-nightly-master-158/PACKAGES_x86_64_linux_redhat_7/cfengine-community-3.21.0a.432050af6-22949.el7.x86_64.rpm ``` ## Other assets You can even get other assets like the default policy set (Masterfiles Policy Framework): ```command cf-remote --version master list masterfiles ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using master: http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1113/PACKAGES_x86_64_linux_redhat_7/tarballs/cfengine-masterfiles-3.21.0a.5d60668d3-1.pkg.tar.gz ``` Or the Vagrant quickstart environment: ```command cf-remote --version master list vagrant ``` ```output Available releases: master, 3.20.0, 3.18.x, 3.18.2, 3.18.1, 3.18.0, 3.15.x, 3.15.6, 3.15.5, 3.15.4, 3.15.3, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1 Using master: http://buildcache.cfengine.com/packages/testing-pr/jenkins-master-nightly-pipeline-1113/PACKAGES_x86_64_linux_redhat_7/../vagrant_quickstart/builds/CFEngine_Enterprise_vagrant_quickstart-3.21.0a-1.tar.gz ``` [`cf-remote`](https://pypi.org/project/cf-remote/) can do more than list URLs, install it and give it a try! Nightly packages for CFEngine Community can also be found on our website, under [Resources -> Downloads -> Nightly packages](/downloads/cfengine-community-nightly/). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 17 - Compliance" url: /blog/2022/the-agent-is-in-episode-17-compliance/index.md date: "2022-09-29" -------------------------------------------------------------------------------- The good we secure for ourselves is precarious and uncertain until it is secured for all of us and incorporated into our common life. How often do you verify your compliance? Once or twice a year? Have you considered reporting on compliance continually? The usual suspects, Cody Valle (Head of community), Craig Comstock (Digger), and Nick Anderson (Doer of Things) see how CFEngine Enterprise can be used to implement and report on compliance, specifically the Ubuntu 20.04 Security Technical Implementation Guide (STIG). Nick shows how tagging variables for inventory and Mission Portals compliance reports can be used to implement compliance reporting that is continually verified. ## Video The video recording is available on YouTube: {{< youtube T7KTNWzR4CU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links * [CFEngine Build and vagrant projects](https://gitlab.com/nickanderson/cfengine-enterprise-demo/-/commits/TAII-17) * [Ubuntu 20.04 Security Technical Implementation Guide (STIG)](https://www.stigviewer.com/stig/canonical_ubuntu_20.04_lts/) * Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) * [All Episodes](/webinars/) ## Further work Help Nick implement Ubuntu 20.04 STIG inventory by opening pull requests [here](https://github.com/nickanderson/cfengine-security-hardening/tree/master/stig/canonical_ubuntu_20.04_lts). Share your own compliance inventory policy and reports with other users by publishing a module on [build.cfengine.com](https://build.cfengine.com). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 16 - CFEngine Enterprise for ARM64" url: /blog/2022/the-agent-is-in-episode-16-cfengine-enterprise-for-arm64/index.md date: "2022-08-28" -------------------------------------------------------------------------------- Ever wish that you could run Mission Portal at Home? Some of the CFEngine team gathers in Oslo Norway to do the show live, together. Craig Comstock (Digger) demonstrates how to use cf-remote to access new ARM64 packages for CFEngine Enterprise (Hub and Clients) and experiments with CFEngine Build in Mission Portal. ## Video The video recording is available on YouTube: {{< youtube vbnoxvkmocw >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links * [cf-remote](https://github.com/cfengine/cf-remote) * Connect w/ [Chia-Cheng](https://www.linkedin.com/in/chiachenglu/) , [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), [Lars](https://www.linkedin.com/in/larsewi/), [Nick](https://www.linkedin.com/in/hithisisnick/), or [Vratislav](https://www.linkedin.com/in/vratislav-podzimek-03489b43/) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Guest blog post: Quick-start guide to using Emacs for CFEngine" url: /blog/2022/quick-start-guide-to-using-emacs/index.md date: "2022-08-26" -------------------------------------------------------------------------------- This guide is designed for the novice user of CFEngine who wishes to explore the power of Emacs while developing CFEngine policy files -- and will introduce the use of some Emacs features and plugins along the way. There are two types of editors available in the Unix and Linux world: line and visual. Examples of line editors are ed and sed. These allow you to edit a file one line at a time. Examples of visual editors are vi, nano and Emacs. These are also known as full-screen editors. They display as much of the entire file being edited as will fit on the screen at a time. Emacs is unique among visual editors because it is capable of operating in both GUI (using Gtk widgets) and console or terminal modes. If you are working in a graphical environment, simply running `emacs` at the command line will start in GUI mode. If you are running in a non-graphical environment, such as from within [PuTTY](https://docs.cfengine.com/docs/master/getting-started-installation-pre-installation-checklist-putty-quick-start-guide.html), it will start in terminal mode. If you wish to start terminal mode from within a GUI environment, run `emacs -nw`. Emacs is more than just an editor; it is a complete programming language and interpreter for that language. Emacs consists of a Lisp interpreter written in C, but most of it is written in Emacs-Lisp, called elisp. Emacs implements modes written in elisp. There are two types of modes, major and minor. Major modes are essentially programs that Emacs runs. The most common of these is Fundamental mode, which implements the basic text editor. Other programming modes are extensions of Fundamental mode. There is a major mode for CFEngine called cfengine3-mode. You can think of minor modes as plugins or features that run along side of major modes. You can only run one major mode at a time but multiple minor modes can be active at the same time. There are minor modes which alter the Emacs key bindings to work like other editors, such as vi. This tutorial will use only standard Emacs key bindings. If you are interested in vi-style key bindings, please investigate viper-mode, evil-mode, Spacemacs[^fn:1], or Doom Emacs on your own. ## Emacs Key Bindings {#emacs-key-bindings} To start Emacs, either start it from your operating system's application menu or type `emacs` at a command prompt. This will start a GUI window of Emacs if you are operating in a GUI environment as discussed above. Emacs modes are not like regular and insert modes in vi. Emacs is always in insert mode; there is no regular mode like in vi. Therefore, it is necessary to hit special keys in Emacs to perform other functions. These are Control, Alt, Shift, and Super (most commonly the Windows key). Here are the conventions used to display these keys: | Symbol | Meaning | |--------|-------------| | C-x | Control + x | | M-x | Meta + x | | S-x | Shift + x | | s-x | Super + x | Wait! What is meta? Once upon a time, not every keyboard had Alt keys, but there might have been a meta key. And if the keyboard had neither, it likely had an Escape key. So, if you see a key combination that is `M-something`, either hold down Alt while hitting that other key, or hit Escape and then the other key. Sometimes it is necessary to use Escape instead of Alt, in particular when using SSH to access machines with different key mappings. `S-x` is Shift + x. This will only result in an upper case `X` being printed. You may find other key combinations that require shift, like `C-%`, which is `C-S-5` (Control + Shift + 5). This starts the query-replace function, otherwise known as search and replace string. Once started, you will see some information about Emacs. No file will be open. If you started Emacs from the command line with `emacs `, it will begin with that file open. You can specify multiple files. Emacs is capable of opening multiple files at a time. Each file is opened in a buffer. Emacs can also hold buffers which are not associated with a file. To visit a file, use `C-x C-f`. That is, hold down Control and hit `x` and while still holding down Control hit `f`. This will prompt you for a file name. The file does not have to exist. You can use tab-completion if the file does already exist. By default, Emacs will start looking for files in the directory where it was started, most likely your `$HOME` directory. When you are done editing a file, save it with `C-x C-s`. Why do the commands to visit and save files start with `C-x`? Think of `C-x` as the "extra functions" prefix. `C-f` on its own moves the cursor forward one character. `C-s` on its own starts the search function within the buffer. To exit Emacs, use `C-x C-c`. But you don't exit Emacs every time you finish editing a file. Most of us Emacs users start it when we log in and just close buffers when we finish editing a file. To do this, just use `C-x k`. That's hold down Control and hit `x`, release Control and then hit `k`. You will be prompted for the buffer name to close, and it defaults to the current buffer. Emacs will remain open and you can open another file with `C-x C-f`. To switch between buffers, use `C-x b`. By default, it will attempt to switch back to the most previously used buffer. You can type in any other buffer name, with Tab key completion. You can also type the name of a non-existent buffer to create a new one not associated with a file. There is a buffer named `*scratch*` which you can use to store notes or even evaluate Lisp commands (which is beyond the scope of this document). Buffers not associated with files, like the `*scratch*` buffer, are not saved when Emacs exits. Now for some generic movement. `C-f` moves the cursor forward one character, like the right arrow. `C-b` moves it back like the left arrow. `C-p` moves to the previous line like the up arrow. `C-n` moves to the next line like the down arrow. `C-a` moves to the beginning of the line like the home key. `C-e` moves to the end of the line like the end key. `C-v` moves a page down like the page-down key. And `M-v` moves a page up like, you guessed it, the page-up key. Why use all these funny Control key combos when arrow keys exist? Because it is more ergonomic to keep your hands in the home row of the keyboard than to constantly move them around. `C-d` will delete the character under the cursor. `M-d` will delete the rest of the word at the cursor. `C-k` will delete from the cursor to the end of the line. If the cursor is at the beginning of the line, `C-k` will delete the whole line. `Backspace` will delete the character behind the cursor. `C-backspace` will delete from the cursor to the beginning of the current word. Normally, the position of the cursor is called the point. Hit `C-space` and move the cursor. The position where the cursor started is called the mark, and wherever you have moved the cursor is still called the point. If you have transient-mark-mode enabled, the region between the point and the mark should be highlighted. In more recent versions of Emacs, this is the default so you should see what you have marked. To cut this text, hit `C-w`, but to just copy it, hit `M-w`. When you do this, the copied or cut text is stored on the kill-ring. To get it back, or paste it, you use the yank command. To start yanking text off the kill-ring, first hit `C-y`. This yanks the most recent item on the kill-ring. To yank something else off the kill-ring, use `M-y`. This yanks the second item off the kill-ring, replacing the first item yanked. The kill-ring defaults to holding 60 items. Items deleted with `C-k`, `M-d` or `C-backspace` are also stored on the kill-ring, but not single characters erased with `C-d` or `backspace`. Finally, some important keys you will frequently need. To cancel from the middle of anything, use `C-g`. And to undo things, use `C-/`. If you mess up, thinking that undo is `C-z`, that just puts Emacs in the background, possibly minimizing it. From the terminal, use the `fg` command, from a GUI it's probably docked on one of your panels and Alt-Tab should bring it back. ## Editing CFEngine Policy with Emacs {#editing-cfengine-policy-with-emacs} At the bottom of the screen in Emacs you will see the mode-line. It tells you what file you have open and what mode you are in. The blank under that is called the mini-buffer, which is where Emacs prompts you for file names and other information. Emacs may not start in `cfengine3-mode` when you open a file ending in `.cf`. To put Emacs in `cfengine3-mode`, hit `M-x` then enter `cfengine3-mode` in the mini-buffer. You should be able to Tab-complete this. Editing CFEngine policy files in `cfengine3-mode` is helpful because Emacs will handle indentation for you and highlight parts of the syntax. I am going to show you how to write CFEngine policy in Emacs using the YASnippet extension and a library I wrote for it. YASnippet means "yet another snippet" and is an extension which inserts templates where requested. My library provides these templates specific to CFEngine 3. `M-x` allows you to enter Emacs commands into the mini-buffer. Such commands include changing modes and starting certain functions. First, make sure Emacs's package cache is up to date with `M-x package-refresh-contents`. Next, install YASnippet with `M-x package-install yasnippet`. Open a terminal on your computer. Clone my snippet library from my GitHub repository[^fn:2]. ```shell git clone https://github.com/jeffcarlson72/emacs-yasnippet-cfengine3-mode.git mkdir -p .emacs.d/snippets mv emacs-yasippet-cfengine3-mode .emacs.d/snippets/cfengine3-mode ``` Now turn on YASnippet in Emacs with `M-x yasnippet-mode`. Load the CFEngine 3 snippets with `M-x yas/reload-all`. Make sure you are in a new `cfengine3-mode` buffer with `C-x C-f example.cf` followed by `M-x cfengine3-mode`. Let's insert the first template. We want to make an agent type bundle so start by typing the word "agent" followed by the Tab key. Name your bundle "main" by typing that where the string is highlighted. You should see the following. ```cfengine3 bundle agent main { meta: vars: defaults: classes: users: files: packages: guest_environments: methods: processes: services: commands: storage: databases: reports: } ``` What you have is a template for an agent bundle with all the promise types valid for agent bundles, in standard execution order. You can leave the promise types blank, it will not cause an error. Now, move the cursor below the promise type for `files:`, hit the Enter key, type the word "files" one more time and hit Tab. You will expand the template for a "files" type promise. Name the promiser (which is highlighted) to "/tmp/test." ```cfengine3 "/tmp/test" action classes comment depends_on handle if ifvarclass meta unless with acl changes copy_from create delete depth_search edit_defaults edit_line edit_template edit_template_string edit_xml file_select file_type link_from move_obstructions pathtype perms rename repository template_data template_method touch transformer ``` What you see is every optional property for a files-promise, first the common properties in alphabetical order, then all the files-specific properties in alphabetical order. Move the cursor to the end of a property and hit Tab, it will expand with a hash-rocket and possibly any default values which might be body types, booleans, or other common answers. Be sure to add commas or semicolons at the ends of lines where necessary. Delete any template lines which are not needed for this promise. Let's just create a blank file for this example. ```cfengine3 "/tmp/test" create => "true", perms => mog("0644", "root", "root"); ``` When you hit Tab after "create," YASnippet prompted you with the values of `true` or `false`. And when you hit Tab after `perms`, YASnippet prompted you with all the perms body names from the masterfiles policy framework library. When you hit Tab after `mog`, YASnippet prompted you for the mode, owner, and group of this file. ## Conclusion {#conclusion} In this tutorial you have seen some basic usage of the Emacs editor. You have learned to open, save, and close files as well as how to change modes and buffers within Emacs. You have learned how to delete, copy, and paste text within an Emacs buffer. Also, you have seen how to install an external package from Emacs's default package repository, which is called ELPA. Then you set up the YASnippet package to edit your CFEngine policy files with ease. To continue using Emacs, run the tutorial with `C-h t` which will show you many more Emacs keybindings and explain concepts in more depth. You should look into configuring your own Emacs init file so that files ending in .cf automatically start in `cfengine3-mode` and so that YASnippet loads all snippets at startup. I personally believe that Emacs is the most powerful editor on Earth and I use it to write all of my own CFEngine policy as well as just about everything else I need to write on a computer. [^fn:1]: Spacemacs conveniently provides a [CFEngine layer](https://develop.spacemacs.org/layers/LAYERS.html#cfengine) [^fn:2]: Jeff Carlson's YASnippet library for CFEngine -------------------------------------------------------------------------------- title: "Debian 11 and Ubuntu 22 aarch64 (arm64) packages available!" url: /blog/2022/aarch64-debian11-ubuntu22-nightly-packages-available/index.md date: "2022-08-18" -------------------------------------------------------------------------------- As a person who tries to work with as few resources as possible, whether it's editing everything with [ed(1)](https://bsd.network/@ed1conf) or using old laptops without screens for servers or [turning off computers](./turn-off-computers) as much as possible I am happy to announce _nightly_ packages are available for the aarch64 (ARM 64-bit) architecture. This enables low-power, low-cost devices such as the Raspberry Pi and many others to run CFEngine Enterprise. Why run CFEngine? It is lean on resources and rich in features! It helps keep your systems secure and compliant with whatever policy you may require. So far we are supporting what seem to be the most common operating systems: Ubuntu 22 and Debian 11. Our work on providing [community packages in Yocto](https://northerntech.atlassian.net/browse/CFE-3748) is ongoing and should come before too long. ## Support for Linux Devicetree Some notable changes were needed, such as instead of relying on the `x86_64` / `amd64` specific [Desktop Management Interface(DMI)](https://en.wikipedia.org/wiki/Desktop_Management_Interface) we now also support getting some inventory information from the [Linux Devicetree](https://www.kernel.org/doc/html/latest/devicetree/usage-model.html) instead which is quite prevalent on ARM devices such as phones, SBCs, etc. This paves the way for future in-roads with phones running Linux-based systems such as [Android](https://source.android.com/devices/architecture/dto), [PostmarketOS(Alpine)](), [Mobian(Debian)](https://wiki.mobian.org/doku.php?id=devices), Yocto and more. ## Easy development with cross-architecture containers During development I made use of the very helpful technique of running linux containers with a different architecture via `qemu-user-static` package available in most distributions. Here's what I used in a Debian 11 VM: ```command sudo apt install qemu-user-static buildah echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf deb11=$(buildah from arm64v8/debian:11) buildah run $deb11 bash ``` and from there I simply followed the basic [community package generating steps](https://github.com/cfengine/buildscripts) in our buildscripts project to build community packages. ## Move to the cloud, aarch64 is there After I made various adjustments to core, masterfiles and buildscripts I added a couple of cloud images to our CI system. In this case we are using AWS and Jenkins. Handily AWS has aarch64 support and images for both Debian and Ubuntu! ![part of AWS web UI to start instances of 64-bit arm ubuntu and debian machines](./images/aarch64-aws.png) ## Testing on real hardware For testing I have used a Raspberry Pi 4 as well as a [Toradex Verdin](https://www.toradex.com/computer-on-modules/verdin-arm-family) board with an [NXP iMX 8M Plus](https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS) processor running [Torizon](https://www.toradex.com/torizon) ([Yocto](https://www.yoctoproject.org/) + [Docker](https://www.docker.com/)) to run various distributions on aarch64 more natively. [![Fairphone fp2, PinePhone, Motorola Surnia, LG Hammerhead, Toradex Verdin board with IMX 8M Plus SoM , Raspberry Pi 4, all aarch64 devices that could be tested with](./images/aarch64-devices.png)](./images/aarch64-devices.png) ## Installation A few pre-requisites: - ARM 64 board, for example Raspberry Pi 4 or similar - A 64 bit Debian based distribution installed - Debian 11 or Raspberry Pi OS (based on Debian 11) - Ubuntu 22 - Ubuntu 20 (use Debian 11 package) - SSH access with password-less sudo to run the installer ```command ssh-copy-id $HOME/.ssh/id_rsa rpi4 ssh rpi4 sudo whoami ``` ```output root ``` Given that, install `cf-remote` (see our [blog posts about `cf-remote`](/tags/cf-remote/)) and install a nightly package! We build nightlies 5 days a week so there is plenty of opportunity for you to find bugs, submit PRs and enjoy the results in upgrading packages from nightlies. ```command pip3 install cf-remote cf-remote --version master install --clients rpi4 ``` (This only installs CFEngine, to "start" it, add `--bootstrap `, making it run every 5 minutes, pulling new policy from the policy server and enforcing it). ## Run an Enterprise Hub! Free 25! And don't forget that you can certainly run an enterprise hub as well. Our Enterprise Hub is available free for download and installing for up to 25 hosts. It's a great way to do some proof of concept work or setup something at home for tinkering. ```command cf-remote --version master install --hub rpi4 --bootstrap rpi4 ``` I have been running hub on a Raspberry Pi 4 for several weeks now and am quite pleased with the results. Here you can see I have my `raspberrypi` hub (Raspberry Pi 4, Debian 11) with two containers bootstrapped on the Toradex board. [![Mission Portal served by a Raspberry Pi 4](./images/aarch64-hub-2.png)](./images/aarch64-hub-2.png) ## Community packages Nightly builds are also available as [community packages](https://cfengine.com/downloads/cfengine-community-nightly/) To install with `cf-remote` add the `--edition community` argument: ```command cf-remote --version master install --edition community --clients rpi4 ``` ## Dust off, Fire up! So go find those ARM devices and let CFEngine keep them running well! Let us know how it goes or what other devices you'd like to see supported on [GitHub discussions](https://github.com/cfengine/core/discussions), [the mailing list](https://groups.google.com/g/help-cfengine) and/or chat via [#cfengine on irc.libera.chat](https://web.libera.chat/gamja/#cfengine) or [gitter.im/cfengine.core](https://gitter.im/cfengine/core) (also available [via Matrix](https://matrix.to/#/%23cfengine_core:gitter.im)). -------------------------------------------------------------------------------- title: "Extending autorun" url: /blog/2022/extending-autorun/index.md date: "2022-08-11" -------------------------------------------------------------------------------- What's _autorun_? [Autorun](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html) is a feature of the _Masterfiles Policy Framework_ (MPF)[^fn:1] that simplifies the process of adding and executing new policy. We have talked about [Modular policies with autorun and the Augments](https://cfengine.com/blog/2015/modular-policies-with-autorun-and-the-augments_file/) before. This time, we dig into autorun a bit deeper to explore some of its current features and look at how to implement your own as we did during [The agent is in, Episode 15 - Extending autorun](https://cfengine.com/webinars/?videoID=dxmjlxssZ_4) **Note:** All paths unless otherwise noted are relative to the root of your policy set (typically `/var/cfengine/masterfiles` is the distribution point). `cf-agent` and other commands are run as the `root` user. When enabled (by defining the `default:services_autorun` _class_) `.cf` suffixed files located in `services/autorun` are automatically included in `inputs`[^fn:2] and bundles tagged with `autorun` are actuated in lexical order (a bundle gets tagged by having a _meta_ promise, `"tags"`, defined as a list with `"autorun"` as a value). Let's try it out. ## Taking autorun for a spin {#taking-autorun-for-a-spin} First, add the following policy in `services/autorun/example.cf`. ```cfengine3 bundle agent __main__ # @brief Drive default bundlesequence if called as policy entry { methods: "illustrating autorun" usebundle => example(); } bundle agent example() # @brief example illustrating autorun { meta: "tags" slist => { "autorun" }; reports: "Hello World! from '$(this.namespace):$(this.bundle)' in '$(this.promise_filename)'"; } ``` Now, let's run the policy with `cf-agent --no-lock --file update.cf; cf-agent --no-lock`. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock ``` Nothing happened, as expected since we neglected to define the `services_autorun` class. Let's try it again, this time with the appropriate class defined. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock --define services_autorun R: Hello World! from 'default:example' in '/var/cfengine/inputs/services/autorun/example.cf' ``` Great! It works. But, what if we want to load and run policy files that live in some other directory? ### Additional autorun directories {#additional-autorun-directories} When autorun is enabled the MPF will also look for `.cf` suffixed files in directories defined in `default:def.mpf_extra_autorun_inputs` in addition to `services/autorun`. Let's move our example policy file to a different directory. Create a `MyAutorun` directory and move the example policy there. ```text $ mkdir -p /var/cfengine/masterfiles/MyAutorun $ mv /var/cfengine/masterfiles/services/autorun/example.cf /var/cfengine/masterfiles/MyAutorun/ ``` Next, let's use Augments[^fn:3] to enable autorun and define the path to our additional directory. Populate `def.json` with the following content. ```json { "classes": { "default:services_autorun": { "class_expressions": ["any::"], "comment": "This simplifies the process of integrating new policy. Simply drop it in place." } }, "variables": { "default:def.mpf_extra_autorun_inputs": { "value": ["MyAutorun"], "comment": "Unqualified directories relative to policy entry dirname and fully qualified directories that we should load policy files from" } } } ``` Note: In the above JSON we use an Augments format that is valid for CFEngine agents version 3.18.0 and greater which allow us to be explicit in targeting the namespace a class should be defined in, and the namespace and bundle variables should be defined in, as well as applying other meta-data like tags and comments. This is an equivalent version that works for older versions as well. ```json { "classes": { "services_autorun": ["any::"] }, "vars": { "mpf_extra_autorun_inputs": ["MyAutorun"] } } ``` Now that autorun is configured, run the policy again, this time without having to explicitly define `services_autorun` since we defined it in Augments. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock R: Hello World! from 'default:example' in '/var/cfengine/inputs/MyAutorun/./example.cf' ``` Note, you are not limited to including policy that is within your main policy set. Policy can be included from anywhere. Create `/etc/cfengine/inputs-annex/another-example.cf` with the following content. ```cfengine3 bundle agent another_example() { meta: "tags" slist => { "autorun" }; reports: "Hello World! from '$(this.namespace):$(this.bundle)' in '$(this.promise_filename)'"; } ``` Now, this policy file is outside of our policy set so be sure to set appropriate permissions on the file. ```text $ mkdir -p /etc/cfengine/inputs-annex $ vi /etc/cfengine/inputs-annex/another-example.cf $ chmod 600 /etc/cfengine/inputs-annex/another-example.cf ``` Edit `def.json` in the root of your policy set and add the new directory. ```json { "classes": { "default:services_autorun": { "class_expressions": ["any::"], "comment": "This simplifies the process of integrating new policy. Simply drop it in place." } }, "variables": { "default:def.mpf_extra_autorun_inputs": { "value": ["MyAutorun", "/etc/cfengine/inputs-annex"], "comment": "Unqualified directories relative to policy entry dirname and fully qualified directories that we should load policy files from" } } } ``` Run the policy again to prove that the new policy file outside the policy set was picked up as expected. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock R: Hello World! from 'default:another_example' in '/etc/cfengine/inputs-annex/./another-example.cf' R: Hello World! from 'default:example' in '/var/cfengine/inputs/MyAutorun/./example.cf' ``` Great, it works! But, what if you only want part of that behavior, for example to only find and load policy files but not automatically actuate any bundles? ### Auto inputs (without autorun) {#auto-inputs--without-autorun} **Note:** This feature was first introduced to the MPF in CFEngine 3.18.1. Unfortunately, a bug prevented it from working until 3.18.3 (which has yet to be released, so please use nightlies if you would like to enjoy this feature before then). Let's amend `def.json` and switch from `default:services_autorun` to `default:services_autorun_inputs`. After editing it should look similar to the following. ```json { "classes": { "default:services_autorun_inputs": { "class_expressions": ["any::"], "comment": "This simplifies the process of integrating new policy inputs. Simply drop them in services/autorun or any of the directories listed in default:def.mpf_extra_autorun_inputs." } }, "variables": { "default:def.mpf_extra_autorun_inputs": { "value": ["MyAutorun", "/etc/cfengine/inputs-annex"], "comment": "Unqualified directories relative to policy entry dirname and fully qualified directories that we should load policy files from" } } } ``` With that in place, we can see that running `cf-agent --no-lock --file update.cf; cf-agent --no-lock` does not result in the same reports being emitted as we saw previously. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock ``` Now, let's explicitly run `default:another_example` to run by editing `services/main.cf` as shown below. ```cfengine3 ############################################################################### # # bundle agent main # - User/Site policy entry # ############################################################################### bundle agent main # User Defined Service Catalogue { methods: # Activate your custom policies here "Actuating a bundle in a file that was autoloaded" usebundle => default:another_example; } ``` Now, when we run the policy, we see that the bundle from an automatically included file was found and emitted the expected report. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock R: Hello World! from 'default:another_example' in '/etc/cfengine/inputs-annex/./another-example.cf' ``` Some people (including yours truly) are uncomfortable with dynamically loading policy files and prefer to explicitly enumerate all policy inputs, let's take a look at automatically running bundles without automatically loading inputs. ### Auto run bundle (without inputs) {#auto-run-bundle--without-inputs} The MPF supports this methodology via the `default:services_autorun_bundles` class. When defined (assuming that neither `services_autorun` or `services_autorun_inputs` is defined) no policy files will be automatically added to inputs. However, any bundle tagged with `autorun` will be actuated in lexical order. Edit `def.json` to explicitly include `MyAutorun/example.cf` and switch from defining the class `default:services_autorun_inputs` to `default:services_autorun_bundles` as shown below. ```json { "inputs": ["MyAutorun/example.cf"], "classes": { "default:services_autorun_bundles": { "class_expressions": ["any::"], "comment": "This simplifies the process of running bundles." } }, "variables": { "default:def.mpf_extra_autorun_inputs": { "value": ["/etc/cfengine/inputs-annex"], "comment": "Unqualified directories relative to policy entry dirname and fully qualified directories that we should load policy files from" } } } ``` Comment out or remove the promise to run `default:another_example` from `services/main.cf` or you will get errors due to `another-example.cf` no longer being loaded. For example, if you neglect to comment out or remove the promise, expect to get an error similar to this: ```text error: Apparent bundle 'default:another_example' was undeclared, but used in a promise near line 14 of /var/cfengine/inputs/services/main.cf (possible unquoted literal value) error: A method attempted to use a bundle 'default:another_example' that was apparently not defined error: Fatal CFEngine error: Aborting due to missing bundle 'default:another_example' ``` Even though we left `default:def.mpf_extra_autorun_inputs` defined, since neither of the classes `default:services_autorun` or `default:services_autorun_inputs` are defined, the policy file was not parsed as part of inputs and thus the bundles from those files were not found. When executing the policy we find that only the bundle from the explicitly included file is actuated and emits a report. ```text $ cf-agent --no-lock --file update.cf; cf-agent --no-lock R: Hello World! from 'default:example' in '/var/cfengine/inputs/MyAutorun/example.cf' ``` But, what if you want more? What if you want to descend into sub-directories to find policy files or use other logic to select which policy files should be loaded and which bundles should be actuated? For that, consider implementing your own autorun. ## Implementing your own autorun {#implementing-your-own-autorun} To implement your own autorun mechanism, you need a few things: - Some policy files to find and load as part of inputs - A bundle to find the policy files - A file control body to load the found files - A bundle to find and run bundles - Configuration to integrate your custom autorun policy ### Policy files to load {#policy-files-to-load} So that we have something to work with, let's generate some contrived policy and structure. Please do not consider any of this arrangement as an endorsement of any specific layout, it's simply something to work with. All of these policy files will share this common pattern: ```cfengine3 # Generated contrived policy file ¯\_(ツ)_/¯ bundle agent __main__ { methods: "canonified_policy_filename"; } bundle agent canonified_policy_filename { meta: "tags" slist => { "my_autorun", "", ""}; reports: "From bundle '$(this.namespace):$(this.bundle)' tagged with '$(with)' in '$(this.promise_filename)'" with => join( ", ", "$(this.namespace):$(this.bundle)_meta.tags" ); } ``` This will allow us to easily see which bundles get automatically run and where they are from. ```sh exec 2>&1 rm -rf /tmp/services for each in /tmp/services/MyPolicies/onefish/twofish.cf \ /tmp/services/MyPolicies/SomeWhere/OutThere.cf \ /tmp/services/MyPolicies/beneath_the_pale/moonlight.cf \ /tmp/services/MyPolicies/DOW/Monday/monday.cf \ /tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf \ /tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf \ /tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf \ /tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf \ /tmp/services/MyPolicies/DOW/Thursday/thursday.cf \ /tmp/services/MyPolicies/DOW/Friday/then.cf \ /tmp/services/MyPolicies/DOW/Saturday/sat.cf \ /tmp/services/MyPolicies/DOW/Sunday/what.cf \ /tmp/services/MyPolicies/Platform/linux/three.cf \ /tmp/services/MyPolicies/more/enabled/four.cf \ /tmp/services/MyPolicies/more/disabled/five.cf; do chars=$(echo -n ${each} | wc -c ) dirname=$(dirname ${each}) dir_basename=$(basename ${dirname}) bundlename=$(basename --suffix=".cf" ${each} | tr [[:punct:][:space:]] _ | sed 's/_$//') if [ $((chars%2)) -eq 0 ]; then odd_even="even" else odd_even="odd" fi mkdir -p $(dirname $each) printf "# Generated contrived policy file ¯\_(ツ)_/¯ bundle agent __main__ { methods: \"%s\"; } bundle agent %s { meta: \"tags\" slist => { \"my_autorun\", \"%s\", \"%s\" }; reports: \"From bundle '\$(this.namespace):\$(this.bundle)' tagged with '\$(with)' in '\$(this.promise_filename)'\" with => join( \", \", \"\$(this.namespace):\$(this.bundle)_meta.tags\" ); } " ${bundlename} ${bundlename} ${odd_even} ${dir_basename} > ${each} chmod 600 ${each} lastfile=${each} printf "Created '${each}'\n" done printf "Preview '${each}':\n$(cat ${each})\n" printf "Done generating contrived policy" : ``` Now that we have some policy files to find let's get to it. ### A bundle to find policy files {#a-bundle-to-find-policy-files} All of the policy files generated previously are within `services/MyPolicies`. Let's start by recursively searching for `.cf` suffixed files. ```cfengine3 bundle agent find_my_policies_recursively { vars: "found" slist => findfiles( "/tmp/services/MyPolicies/**/*.cf" ); reports: "Found '$(found)'"; } ``` Running the policy, we see that all the various policy files are found as desired. ```text $ cf-agent --no-lock --log-level info --bundlesequence find_my_policies_recursively --file find_my_policies_recursively.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: Found '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: Found '/tmp/services/MyPolicies/more/disabled/five.cf' R: Found '/tmp/services/MyPolicies/more/enabled/four.cf' R: Found '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: Found '/tmp/services/MyPolicies/DOW/Sunday/what.cf' R: Found '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: Found '/tmp/services/MyPolicies/onefish/twofish.cf' ``` Now, let's take a look at some variations on the idea. #### Find policy files that are relevant for this day of the week {#find-policy-files-that-are-relevant-for-this-day-of-the-week} Here we look for files in directories named for the current day of the week. This can be useful if you have policies that you only want to run during certain periods of time, the rest of the time don't bother loading the policy files you won't use. ```cfengine3 bundle agent find_my_policies_for_this_dow { vars: "today" string => strftime( "localtime", "%A", now() ); "found" slist => findfiles( "/tmp/services/MyPolicies/DOW/$(today)/**.cf" ); reports: "$(today) Found '$(found)'"; } ``` Running the policy, we see only the policy files under a directory named for the current day of the week are found. ```text $ cf-agent --bundlesequence find_my_policies_for_this_dow --no-lock --log-level info --file find_my_policies_for_this_dow.cf info: Using command line specified bundlesequence R: Monday Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' ``` #### Find policy files that are relevant for the current platform {#find-policy-files-that-are-relevant-for-the-current-platform} How about some policies that are platform specific, for example that only run on _linux_. ```cfengine3 bundle agent find_my_policies_for_my_platform { vars: "found" slist => findfiles( "/tmp/services/MyPolicies/**/$(sys.class)/*.cf" ); reports: "Found '$(found)'"; } ``` We can see from running the policy that only policies under a directory named `linux` are found. ```text $ cf-agent --bundlesequence find_my_policies_for_my_platform --no-lock --log-level info --file find_my_policies_for_my_platform.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' ``` **Tip:** `sys.os_release` contains more specific info on most modern linux systems where `/etc/os-release` is present. #### Find policy files whose parent directory maps to a class suffixed with enabled {#find-policy-files-who-s-parent-directory-maps-to-a-class-suffixed-with-enabled} In this variation, we only want to use a file if we have a class that indicates we need it. This is a handy pattern for only parsing files which are needed by an individual host and reducing execution overhead. ```cfengine3 bundle agent find_my_policies_in_enabled_dirs { classes: # Classes defined here for examples sake. Consider using Augments # so that your classification information is ready from the very # start of cf-agent initialization "SomeWhere_enabled"; "beneath_the_pale_enabled"; vars: "candidates" slist => findfiles( "/tmp/services/MyPolicies/**/*.cf" ); "picked[$(candidates)]" string => "$(candidates)", with => lastnode( dirname( $(candidates) ), "/" ), if => "$(with)_enabled"; "found" slist => getindices( picked ); reports: "Found '$(found)'"; } ``` Executing the policy we see that only policy files below the directories named for the classes with `_enabled` suffixes were found. ```text $ cf-agent --bundlesequence find_my_policies_in_enabled_dirs --no-lock --log-level info --file find_my_policies_in_enabled_dirs.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' ``` #### Find policy files that are not in a directory named disabled {#find-policy-files-that-are-not-in-a-directory-named-disabled} How about omitting policies that are in directories named `disabled`? ```cfengine3 bundle agent find_my_policies_for_my_platform { vars: "candidates" slist => findfiles( "/tmp/services/MyPolicies/**/*.cf" ); "found" slist => filter(".*/disabled/.*", candidates, true, true, inf ); reports: "Found '$(found)'"; } ``` Again, we see that none of the found policy files live inside a directory named _disabled_. ```text $ cf-agent --bundlesequence find_my_policies_for_my_platform --no-lock --log-level info --file find_my_policies_for_my_platform.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: Found '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: Found '/tmp/services/MyPolicies/more/enabled/four.cf' R: Found '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: Found '/tmp/services/MyPolicies/DOW/Sunday/what.cf' R: Found '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: Found '/tmp/services/MyPolicies/onefish/twofish.cf' ``` Hopefully this gives you some ideas about the possibilities. ### body file control to load the found files {#body-file-control-to-load-the-found-files} Once the files we want to include in `inputs` have been found we need to get them parsed along with the rest of `inputs`. For that, we use `body file control` and simply reference the variable that holds the list of file paths we want to include. Here we use the first example to find policy files recursively. ```cfengine3 bundle agent find_my_policies_recursively { vars: "found" slist => findfiles( "/tmp/services/MyPolicies/**/*.cf" ); reports: "Found '$(found)'"; } body file control { agent:: inputs => { @(default:find_my_policies_recursively.found) }; } ``` **Note:** We guard the inclusion of the files with the class `agent`, this protects other components like `cf-serverd`, `cf-monitord`, and `cf-execd` from trying to parse the files. ### A bundle to find and run bundles {#a-bundle-to-find-and-run-bundles} Now that we have included the files in `inputs` we can proceed to make a bundle to run our bundles. Note: this methodology does not restrict itself to bundles that live within a specific directory. Bundles for running need to be found based on their name and any tags they have and the bundles must not take any parameters. ```cfengine3 bundle agent find_and_run_bundles_tagged_my_autorun { vars: "b" slist => sort( bundlesmatching( ".*", "my_autorun" ), lex ); methods: "$(b)"; } ``` When we combine that with our bundle to find policy files and body file control to load the found policy files to `inputs` we get the following. ```cfengine3 bundle agent find_my_policies_recursively { vars: "found" slist => findfiles( "/tmp/services/MyPolicies/**/*.cf" ); reports: "Found '$(found)'"; } bundle agent find_and_run_bundles_tagged_my_autorun { vars: "b" slist => sort( bundlesmatching( ".*", "my_autorun" ), lex ); methods: "$(b)"; } body file control { agent:: inputs => { @(default:find_my_policies_recursively.found) }; } ``` If we run it we can see that all the policy files are found as before and this time all the bundles tagged with `my_autorun` were actuated. ```text $ cf-agent --no-lock --log-level info --bundlesequence find_my_policies_recursively,find_and_run_bundles_tagged_my_autorun --file find_my_policies_recursively_with_find_and_run_bundles_tagged_my_autorun.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: Found '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: Found '/tmp/services/MyPolicies/more/disabled/five.cf' R: Found '/tmp/services/MyPolicies/more/enabled/four.cf' R: Found '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: Found '/tmp/services/MyPolicies/DOW/Sunday/what.cf' R: Found '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: Found '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:OutThere' tagged with 'my_autorun, even, SomeWhere' in '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: From bundle 'default:five' tagged with 'my_autorun, even, disabled' in '/tmp/services/MyPolicies/more/disabled/five.cf' R: From bundle 'default:four' tagged with 'my_autorun, odd, enabled' in '/tmp/services/MyPolicies/more/enabled/four.cf' R: From bundle 'default:monday' tagged with 'my_autorun, odd, Monday' in '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: From bundle 'default:moonlight' tagged with 'my_autorun, even, beneath_the_pale' in '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: From bundle 'default:patch_tuesday' tagged with 'my_autorun, odd, Tuesday' in '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: From bundle 'default:sat' tagged with 'my_autorun, even, Saturday' in '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: From bundle 'default:then' tagged with 'my_autorun, odd, Friday' in '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: From bundle 'default:three' tagged with 'my_autorun, even, linux' in '/tmp/services/MyPolicies/Platform/linux/three.cf' R: From bundle 'default:thursday' tagged with 'my_autorun, odd, Thursday' in '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: From bundle 'default:twofish' tagged with 'my_autorun, odd, onefish' in '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:wed_afternoon' tagged with 'my_autorun, even, pm' in '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: From bundle 'default:wed_morn' tagged with 'my_autorun, odd, am' in '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: From bundle 'default:wednesda' tagged with 'my_autorun, even, Wednesday' in '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: From bundle 'default:what' tagged with 'my_autorun, odd, Sunday' in '/tmp/services/MyPolicies/DOW/Sunday/what.cf' ``` As with finding files, we can take it further. Let's try a variation. #### Find and run bundles tagged with my_autorun & odd {#find-and-run-bundles-tagged-with-my-autorun-and-odd} Let's find and run the bundles that are tagged with both `my_autorun` and `odd`. Since `bundlesmatching()` function takes `n` tags that are OR'd we need to do two searches and find the intersecting bundles. ```cfengine3 bundle agent find_and_run_bundles_tagged_my_autorun_and_odd { vars: "b_my_autorun" slist => sort( bundlesmatching( ".*", "my_autorun" ), lex ); "b_odd" slist => sort( bundlesmatching( ".*", "odd" ), lex ); "b" slist => sort( intersection( b_my_autorun, b_odd ) ); methods: "$(b)"; } ``` Running the policy we can see that it finds all the same policy files but only the subset tagged with both `my_autorun` and `odd` emit the reports indicating they were actuated. ```text $ cf-agent --no-lock --log-level info --bundlesequence find_my_policies_recursively,find_and_run_bundles_tagged_my_autorun_and_odd --file find_my_policies_recursively_with_find_and_run_bundles_tagged_my_autorun_and_odd.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: Found '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: Found '/tmp/services/MyPolicies/more/disabled/five.cf' R: Found '/tmp/services/MyPolicies/more/enabled/four.cf' R: Found '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: Found '/tmp/services/MyPolicies/DOW/Sunday/what.cf' R: Found '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: Found '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:four' tagged with 'my_autorun, odd, enabled' in '/tmp/services/MyPolicies/more/enabled/four.cf' R: From bundle 'default:monday' tagged with 'my_autorun, odd, Monday' in '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: From bundle 'default:patch_tuesday' tagged with 'my_autorun, odd, Tuesday' in '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: From bundle 'default:then' tagged with 'my_autorun, odd, Friday' in '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: From bundle 'default:thursday' tagged with 'my_autorun, odd, Thursday' in '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: From bundle 'default:twofish' tagged with 'my_autorun, odd, onefish' in '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:wed_morn' tagged with 'my_autorun, odd, am' in '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: From bundle 'default:what' tagged with 'my_autorun, odd, Sunday' in '/tmp/services/MyPolicies/DOW/Sunday/what.cf' ``` We can even compress this down into a single statement. ```cfengine3 bundle agent find_and_run_bundles_tagged_my_autorun_and_odd { vars: "b" slist => sort( intersection( sort( bundlesmatching( ".*", "my_autorun" ), lex ), sort( bundlesmatching( ".*", "odd" ), lex ) ) ); methods: "$(b)"; } ``` Running the policy again, we get the same result. ```text $ cf-agent --no-lock --log-level info --bundlesequence find_my_policies_recursively,find_and_run_bundles_tagged_my_autorun_and_odd --file find_my_policies_recursively_with_find_and_run_bundles_tagged_my_autorun_and_odd.cf info: Using command line specified bundlesequence R: Found '/tmp/services/MyPolicies/DOW/Saturday/sat.cf' R: Found '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: Found '/tmp/services/MyPolicies/beneath_the_pale/moonlight.cf' R: Found '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: Found '/tmp/services/MyPolicies/Platform/linux/three.cf' R: Found '/tmp/services/MyPolicies/SomeWhere/OutThere.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/wednesda.cf' R: Found '/tmp/services/MyPolicies/more/disabled/five.cf' R: Found '/tmp/services/MyPolicies/more/enabled/four.cf' R: Found '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/pm/wed-afternoon.cf' R: Found '/tmp/services/MyPolicies/DOW/Sunday/what.cf' R: Found '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: Found '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: Found '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:four' tagged with 'my_autorun, odd, enabled' in '/tmp/services/MyPolicies/more/enabled/four.cf' R: From bundle 'default:monday' tagged with 'my_autorun, odd, Monday' in '/tmp/services/MyPolicies/DOW/Monday/monday.cf' R: From bundle 'default:patch_tuesday' tagged with 'my_autorun, odd, Tuesday' in '/tmp/services/MyPolicies/DOW/Tuesday/patch-tuesday.cf' R: From bundle 'default:then' tagged with 'my_autorun, odd, Friday' in '/tmp/services/MyPolicies/DOW/Friday/then.cf' R: From bundle 'default:thursday' tagged with 'my_autorun, odd, Thursday' in '/tmp/services/MyPolicies/DOW/Thursday/thursday.cf' R: From bundle 'default:twofish' tagged with 'my_autorun, odd, onefish' in '/tmp/services/MyPolicies/onefish/twofish.cf' R: From bundle 'default:wed_morn' tagged with 'my_autorun, odd, am' in '/tmp/services/MyPolicies/DOW/Wednesday/am/wed-morn.cf' R: From bundle 'default:what' tagged with 'my_autorun, odd, Sunday' in '/tmp/services/MyPolicies/DOW/Sunday/what.cf' ``` More variations are left to the reader, if you come up with some clever patterns, please share them on [GitHub discussions](https://github.com/cfengine/core/discussions), [the mailing list](https://groups.google.com/g/help-cfengine) and or chat via [#cfengine on irc.libera.chat](https://web.libera.chat/gamja/#cfengine) or [gitter.im/cfengine.core](https://gitter.im/cfengine/core) (also available [via Matrix](https://matrix.to/#/%23cfengine_core:gitter.im)). ### Configuration to activate your custom autorun policy {#configuration-to-activate-your-custom-autorun-policy} Now the last step is to configure the policies so they are run as part of the normal scheduled executions. For that, we use Augments, `def.json`. We place the finding of policy files early in the `bundlesequence` using `default:def.control_common_bundlesequence_classification` and then find and run bundles at the end of the main `bundlesequence` using `default:def.control_common_bundlesequence_end`. If you have autorun policies for classification, consider running those early as well so that the variables and classes are resolved by the time they get to any of your other policies that use them. ```json { "inputs": [ "find_my_policies_recursively_with_find_and_run_bundles_tagged_my_autorun_and_odd_short.cf" ], "variables": { "default:def.control_common_bundlesequence_classification": { "value": ["find_my_policies_recursively"], "comment": "Unqualified directories relative to policy entry dirname and fully qualified directories that we should load policy files from" }, "default:def.control_common_bundlesequence_end": { "value": ["find_and_run_bundles_tagged_my_autorun_and_odd_short"], "comment": "Find and run bundles tagged with both 'my_autorun' and 'odd'" } } } ``` [^fn:1]: The Masterfiles Policy framework is the default policy set, with documentation here: . [^fn:2]: `inputs` is the list of policy files which will be loaded by CFEngine. After a policy is loaded, you still need some way to specify which bundles inside it should be run. This can be achieved through `methods` promises, `autorun` (tags), or modifying the `bundlesequence`. [^fn:3]: Augments, sometimes referred to as `def.json`, allows you to easily define classes, variables, input files, and more, through a JSON file without modifying `.cf` policy files; . -------------------------------------------------------------------------------- title: "Processes, forks and executions - part 2" url: /blog/2022/processes-part-2/index.md date: "2022-07-28" -------------------------------------------------------------------------------- This is the second blog post in a short series about processes on UNIX-like systems. It is a followup to the [previous post](processes-part-1) which focused on basic definitions, creation of processes and relations between them. This time we analyze the semantics of two closely related system calls that play major roles in process creation and program execution. ## fork() and exec() The UNIX-based operating systems provide the `fork()` system call[^1] to create a clone of an existing process and the `execve()` system call to start executing a program in a process. Windows, on the other hand, provide the `CreateProcess()` function which starts a given program in a newly created process. Why are UNIX-based systems doing things in a more complicated way? There are many reasons for that, some simply historical, as described in [The Evolution of the Unix Time-sharing System](https://www.bell-labs.com/usr/dmr/www/hist.html): > Process control in its modern form was designed and implemented within > a couple of days. It is astonishing how easily it fitted into the > existing system; at the same time it is easy to see how some of the > slightly unusual features of the design are present precisely because > they represented small, easily-coded changes to what existed. A good > example is the separation of the fork and exec functions. > In fact, the PDP-7\'s fork call required precisely 27 lines of > assembly code. Of course, other changes in the operating system and > user programs were required, and some of them were rather interesting > and unexpected. But a combined fork-exec would have been considerably > more complicated,... Based on the above words one could say it was merely a coincidence or even a result of some laziness. Laziness of programmers[^2] has over the decades definitely led to many great things (including great disasters). So this would not be something new or special. However, was it a coincidence that it was easy to add `fork()` and then `exec()` keeping them as two separate actions? Finding the answer would definitely be a nice research and analysis, but maybe a topic for a separate blog post. One thing is definitely true, though -- the semantics of `fork()` and `exec()` fit very well in the philosophy and design of the UNIX-based operating systems. One of the key UNIX principles is the *Do One Thing And Do It Well* principle. And even just this approach alone requires the operating system to be able to create new processes quickly and efficiently. After all, there are many things to be done! Creating a clone of an existing process with `fork()` only to throw most of it away again with `exec()` may sound inefficient. But a closer look, for example at the *fork(2)* man page on Linux, shows something else: > Under Linux, fork() is implemented using copy-on-write pages, so > the only penalty that it incurs is the time and memory required to > duplicate the parent\'s page tables, and to create a unique task > structure for the child. In other words, `fork()` creates something like a shallow copy of an existing process where only the memory that is accessed for writing is actually copied before writing into it. This is very similar to creation of a new thread which, unlike a new child process, remains in the same address space even for writes into the memory. In fact, on Linux the `fork()` C function is actually implemented using the `clone()` system call which is also used when creating new threads, only with different arguments. If a `fork()` is followed by an `exec()`, which loads the new program image and initializes the process to be able to run the program, very little of the work done for `fork()` comes in vain. And thus such combination is actually almost as efficient as doing everything at once. Of course, making two system calls generally requires a bit more resources than making a single system call, but the overhead is very small compared to all the things that need to happen when a program is started. If, on the other hand, a process only needs to do something in a separate process (for practical or security reasons) which means there\'s no `exec()` a lot of resources are saved in comparison to starting a program, doing all the initialization, replicating the state of the original process and doing that something that required to be run in a new process. Again, in UNIX, this is a common thing happening because of the UNIX philosophy. Even a simple usage of a subshell in a shell script to avoid worrying about reverting the current working directory like this: ```sh echo "Doing something in $PWD" (cd /tmp; echo "Now in $PWD") echo "Back to the original shell in $PWD!" ``` is using a `fork()` without an `exec()`. We can see that if we run it with `strace`: ``` example $ strace -e clone,execve,write,chdir -ff bash /tmp/test.sh > /dev/null execve("/usr/bin/bash", ["bash", "/tmp/test.sh"], 0x7ffe730ee4a0 /* 71 vars */) = 0 write(1, "Doing something in /home/vpodzim"..., 34) = 34 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 709650 attached , child_tidptr=0x7f1b9cce0a10) = 709650 [pid 709650] chdir("/tmp") = 0 [pid 709650] write(1, "Now in /tmp\n", 12) = 12 [pid 709650] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=709650, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- write(1, "Back to the original shell in /h"..., 46) = 46 +++ exited with 0 +++ ``` There is one `execve()` system call when the `bash` program is started, then it executes the first `echo` command resulting in a `write()` system call, then there is a `clone()` as a result of the `fork()` function call which creates a new (child) process. The new process changes its current working directory, writes a message to standard output and exits (terminates) about which the parent process is notified (signaled with `SIGCHLD`). And before the parent itself terminates, it again writes a message to the standard output. There is no `execve()` system call from the child subshell process and the change of the current working directory only affects the forked child process. Another example of when `fork()` without `exec()` makes a lot of sense is concurrent (and potentially parallel) processing. A process can do some initialization and then replicate itself N times to create N child processes as workers that then concurrently process some data or handle requests. Of course this could be done by starting the workers separately, each running the same program, but then the initialization would have to happen in all of them which would be wasting resources at least or even impossible (because of them running concurrently). Of course threads can be used for such workers, but processes provide better separation and make the workers really independent -- they cannot write into each other\'s address space and when, for example, one of the workers makes a bad memory access, only that one worker is stopped by the operating system due to a violation of memory constraints. An `execve()` system call without a previous `fork()` (or `clone()`) makes sense too. When a program\'s code is done running in a process and all it wants to do is to start a different program, it can simply `exec()` the other program. We can again use `strace` with a simple, although very artificial, shell command to demonstrate this: ``` example $ strace -e execve,clone,write -ff -- bash -c 'echo "Running"; exec echo Done' >/dev/null execve("/usr/bin/bash", ["bash", "-c", "echo \"Running\"; exec echo Done"], 0x7ffcb2148ed0 /* 71 vars */) = 0 write(1, "Running\n", 8) = 8 execve("/usr/bin/echo", ["echo", "Done"], 0x5562fb9cb250 /* 69 vars */) = 0 write(1, "Done\n", 5) = 5 +++ exited with 0 +++ ``` This time we see there is no `clone()` system call, the process runs `bash` and then the command tells `bash` to execute `echo` with a message. ## Between fork() and exec() The last, but not least, benefit of the separate `fork()` and `exec()` is that there might be instructions and operations happening between the two system calls. Some people actually even consider this the biggest benefit and the most useful feature of the two-step approach. And some consider it the biggest drawback of the approach. Let\'s see why. The above sections and paragraphs describe process creation on UNIX-like systems as creation of clones of existing processes. However, that\'s a bit of a simplification because the newly created process is not a 1:1 copy of the process it was cloned from. As the `fork(3)` man page on Linux says: > The child process is an exact duplicate of the parent process except > for the following points: after which a list of 9 points follows, only to be followed by > The parent and child also differ with respect to the following > Linux-specific process attributes: with another list of 7 points. Some of those points are quite obvious -- for example, the new (child) process has a different PID than the parent process and its parent PID is the PID of the parent process. The rest are more technical and potentially more subtle, but definitely not more important. ### File descriptors When scrolled even a bit further, the `fork(3)` man page says: > Note the following further points: and lists 5 points that are very important. First, let\'s look at the last 3 ones: > * The child inherits copies of the parent\'s set of open file > descriptors. Each file descriptor in the child refers to the same open > file description (see open(2)) as the corresponding file descriptor in > the parent. This means that the two file descriptors share open file > status flags, file offset, and signal-driven I/O attributes (see the > description of F_SETOWN and F_SETSIG in fcntl(2)). > > * The child inherits copies of the parent\'s set of open message > queue descriptors (see mq_overview(7)). Each file descriptor in the > child refers to the same open message queue description as the > corresponding file descriptor in the parent. This means that the two > file descriptors share the same flags (mq_flags). > > * The child inherits copies of the parent\'s set of open directory > streams (see opendir(3)). POSIX.1 says that the corresponding > directory streams in the parent and child may share the directory > stream positioning; on Linux/glibc they do not. The message queues are not so common, at least not the POSIX-defined ones mentioned in the second point. But file descriptors (FDs) and directory streams (*\"directory descriptors\"*) are. And as we can see, these **are** inherited and so in this regard the new process **is** an exact copy of the parent process. This basically means that whatever[^3] the parent process had open for reading or writing before it called `fork()` is available for reading and writing in the child (new) process. And once again, this fits very well into the UNIX philosophy and way of doing things. The *do one thing and do it well* principle means that multiple programs (and processes) often need to operate on some data in a pipeline where the output from one step is the input for the next step. One way to achieve that is to use temporary files for the outputs (and thus inputs) of the individual steps. However, requiring each step to finish before the next one starts is, in most cases, a waste of resources because the individual steps hardly fully utilize them all. A common technique is to let the steps process the data in parallel with a step processing a new chunk of the data while the previous chunk is being processed by the next step. To achieve this, UNIX-like systems provide so-called *pipes* -- unidirectional data channels with one end for writing and one end for reading. What do pipes have to do with forks and file descriptor inheritance? Let\'s take a look at another simple example (with a few lines omitted for better clarity): ``` example $ strace -ff -e clone,execve,pipe,dup2,read,write -- bash -c '/bin/echo Hello|grep -i hell' 2>&1 >/dev/null execve("/usr/bin/bash", ["bash", "-c", "/bin/echo Hello|grep -i hell"], 0x7ffc78c1cc50 /* 72 vars */) = 0 pipe([3, 4]) = 0 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4c42b6ca10) = 19493 strace: Process 19493 attached [pid 19492] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD , child_tidptr=0x7f4c42b6ca10) = 19494 strace: Process 19494 attached [pid 19493] dup2(4, 1) = 1 [pid 19494] dup2(3, 0) = 0 [pid 19493] execve("/bin/echo", ["/bin/echo", "Hello"], 0x559f63e212b0 /* 71 vars */) = 0 [pid 19494] execve("/usr/bin/grep", ["grep", "-i", "hell"], 0x559f63e212b0 /* 71 vars */) = 0 [pid 19493] write(1, "Hello\n", 6) = 6 [pid 19493] +++ exited with 0 +++ [pid 19494] read(0, "Hello\n", 98304) = 6 [pid 19494] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19493, si_uid=1000, si_status=0, si_utime=0, si_stime=0} --- +++ exited with 0 +++ ``` Just a few lines, but showing multiple ingenious things! We can see that the `bash` process (PID `19492`) first creates *a pipe* with its ends being file descriptors `3` and `4`.[^4] Note how there is no path associated with the pipe, it virtually only exists inside of the `bash` process and cannot be accessed by any other process. Then, magic starts happening. The `bash` process forks (`clone()`) of course, twice, to create processes for both `echo` and `grep`. However, keep in mind that the new cloned child processes still run the `bash` program! And so it\'s the `bash` code that then does `dup2(4, 1)` and `dup2(3, 0)` in the child processes, respectively, duplicating the file descriptor `4` to `1` in the process `19493` and `3` to `0` in the process `19494`. *Duplicating a file descriptor* practically means copying/mirroring -- the descriptors then point to the same place, be it somewhere in a file or device, to a socket,..., or pipe, of course. `4` to `1` and `3` to `0`, why these numbers? As mentioned above, the descriptors `3` and `4` are the ends of the pipe, the read end and the write end, respectively. It\'s thanks to the FD inheritance, why these descriptors are actually available in the new forked/cloned processes even though the `pipe()` syscall was made in their parent process. The `0` and `1` are the standard input and standard output descriptors. Put together, what happens in the `dup2()` system calls is that the standard **input** of the process `19494`, is \"hooked up\" to the **read** end of the pipe and the standard **output** of the process `19493` is \"hooked up\" to the **write** end of the pipe. Which means that if the process `19494` writes something to its standard output, the process `19493` will be able to read it from its standard input. So the two processes can now communicate, and they can do that without using any special system calls or anything, they just read/write from/to their standard input/output as usual! The pipe and file descriptor inheritance allows two processes to be used in a pipeline without the programs they run being modified or specially written for such a use case! And of course, there can be more steps in the pipeline, two is just the minimal length. Following the `dup2()` system calls we can see that **only then** the processes actually start executing the `echo` (`19493`) and `grep` (`19494`) programs which just normally do `write()` and `read()`, respectively, as usual. And there\'s one last interesting thing to notice! The `echo` process (`19493`) actually terminates before the `grep` reads the data from it! This is possible because the pipe uses a buffer which holds the data. So the `write()` syscall doesn\'t have to wait for the respective `read()` syscall, after writing the data into the pipe, as long as the data fits into the buffer, the process can continue executing (incl. termination). Another one of many use cases for file descriptor inheritance and the ability to run code before `fork()` and `exec()` is related to security, in particular to permissions and privileges. A process running with high privileges and many permissions can do some initialization and even open some file descriptors that require those privileges and permissions only to fork itself, drop some of the privileges and permissions (usually by changing the user under which the process runs) in the forked child process and continuing work in the child process. This way the child process has limited access only to carefully chosen assets. Again, we can look at an example, though this time a bit more complicated and run as the *root* user (again with lines omitted for better clarity): ``` example # strace -ff -e openat,clone,execve,setuid,write -- bash -c 'runuser -u vpodzime -- /bin/echo Hello > /file.txt' execve("/bin/bash", ["bash", "-c", "runuser -u vpodzime -- /bin/echo"...], 0x7ffff56872f0 /* 55 vars */) = 0 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f95123bda10) = 837086 strace: Process 837086 attached [pid 837086] openat(AT_FDCWD, "/file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 [pid 837086] dup2(3, 1) = 1 [pid 837086] execve("/sbin/runuser", ["runuser", "-u", "vpodzime", "--", "/bin/echo", "Hello"], 0x55bc8b5f7250 /* 54 vars */) = 0 [pid 837086] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fbf36c73a50) = 837087 strace: Process 837087 attached [pid 837087] setuid(1000) = 0 [pid 837087] execve("/bin/echo", ["/bin/echo", "Hello"], 0x7ffe92f922a0 /* 54 vars */) = 0 [pid 837087] write(1, "Hello\n", 6) = 6 [pid 837087] +++ exited with 0 +++ [pid 837086] +++ exited with 0 +++ --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=837086, si_uid=0, si_status=0, si_utime=0, si_stime=1} --- +++ exited with 0 +++ ``` This time we can see `bash` started which forks (`clone()`), in the forked child process (`837086`) it opens the `/file.txt` file for writing (and creates it), copies the associated file descriptor (`3`) to standard output descriptor (`1`) and executes `runuser`. `runuser` then forks and in the new child process (`837087`) it sets user ID to `1000` (UID of the user *vpodzime*) and executes `echo` which then writes data to its standard output, i.e. its parent\'s (`runuser`) standard output, i.e. the `/file.txt` file. If we check the permissions of the file: ``` example $ ls -l /file.txt -rw-r--r--. 1 root root 6 Jul 21 16:56 /file.txt ``` we can see it is owned by the *root* user and of course an attempt to append some data to the file as the *vpodzime* user fails: ``` example $ echo World >> /file.txt bash: /file.txt: Permission denied ``` So a process running as the user *vpodzime* (with UID `1000`) was able to write data into the file without the user actually having (write) access to the file. File descriptor inheritance and actions happening between `fork()` and `exec()` made this possible. ### Threads, locks, signals,... Going back to the list of 5 points from the `fork(3)` man page, to complete it, let\'s see the first two points: > * The child process is created with a single thread---the one that > called fork(). The entire virtual address space of the parent is > replicated in the child, including the states of mutexes, condition > variables, and other pthreads objects; the use of pthread_atfork(3) > may be helpful for dealing with problems that this can cause. > > * After a fork() in a multithreaded program, the child can safely > call only async-signal-safe functions (see signal-safety(7)) until > such time as it calls execve(2). This tells us that the new process created by `fork()` only has one thread and it is the thread that called `fork()`, i.e. the new process continues to execute the instructions of that thread. But it also mentions that states of mutexes, condition variables and other objects are cloned (replicated) and that when using `fork()` in multi-threaded programs, special care needs to be taken. These are the first hints that the cloning behavior of `fork()` and the ability to execute instructions before calling `exec()` might cause issues if things are not handled carefully. Especially in multi-threaded programs, but even the file descriptor inheritance itself can easily cause security issues, for example. However, more about that in an upcoming blog post focused on how CFEngine deals with these potential issues. [^1]: Since version 2.3.3, rather than invoking the kernel's `fork()` system call, the glibc fork() wrapper function invokes `clone()` with flags. So, fork is both a system call and a library function, and in many cases the system call is not really used (depending on library, versions and platforms), but in practice, this ambiguity / distinction does not really matter because the API and behavior are the same. [^2]: Some people basically consider this one of the required qualities of a good programmer [^3]: Many things are represented as file descriptors on UNIX-like systems, on Linux including the POSIX message queues, actually. [^4]: Don\'t get confused by the FDs showing up as arguments to the system call, the real argument is an `int fd[2]` array, i.e. a place where to store two integers (FDs). -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 15 - Extending autorun" url: /blog/2022/the-agent-is-in-episode-15-extending-autorun/index.md date: "2022-07-28" -------------------------------------------------------------------------------- How can I run my own bundles automatically, like autorun from the MPF (Masterfiles Policy Framework), but with different logic? Cody Valle (Head of community), Craig Comstock (Digger), Ole Herman Elgesem (Product Manager) and Nick Anderson (Doer of Things) review the existing capabilities and limitations of autorun in the MPF. After reaching the limits offered by the stock framework they explore implementing a custom autorun, for example recursively finding policy files or only including policy files with associated enablement classes. ## Video The video recording is available on YouTube: {{< youtube dxmjlxssZ_4 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links * [Autorun in the MPF](https://docs.cfengine.com/docs/master/reference-masterfiles-policy-framework-services-autorun.html) * Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse) or [Nick](https://www.linkedin.com/in/hithisisnick/) * [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Processes, forks and executions - part 1" url: /blog/2022/processes-part-1/index.md date: "2022-07-26" -------------------------------------------------------------------------------- While working on the [integration of CFEngine Build into Mission Portal](https://cfengine.com/blog/2022/cfengine-3-20-released-modularity/#cfengine-build-in-mission-portal) we came to the point where we needed to start executing separate tools from our recently added daemon - [`cf-reactor`](https://docs.cfengine.com/docs/3.18/reference-components-cf-reactor.html). Although it may seem like nothing special, knowing a bit about the process creation and program execution specifics (and having to fight some really hard to solve bugs in the past) we spent a lot of time and effort on this step. Now we want to share the story and the results of the effort, but since understanding of the reasons behind the work together with how the implementation works requires quite deep knowledge of how processes are being created and programs are being started on UNIX-like systems, we first start with a series of blog posts focused on this seemingly simple area. They cover the basics as well as some advanced topics in two parts: 1. Process creation and relations between processes 2. The two-step mechanism of an external program execution Let's start with the first part right away with the second part being covered in a follow-up blog post. ## Process creation and process relations Operating systems work with entities called *processes* where a *process* is an instance of a computer *program*. A *program* is a set of data and instructions that can be executed. So a program is a static/passive entity, it is stored in some memory and it only has its static data (including initial values of dynamic data). A process, on the other hand, is an active entity, it has a runtime state stored in its address space and CPU registers and it executes instructions of the program (and dynamically added/generated instructions). The operating system running a process then maintains information about the process -- its identifier, resource counters, address space boundaries, security aspects, open files, etc. While a process is a unique entity and it has a unique identifier, there can, of course, be multiple processes running the same program. And there can be multiple copies of a program stored in multiple places, but they are all the same program (the same set of data and instructions). ### Process creation Processes are created by operating systems, either because of decisions of the operating systems themselves or as reactions to requests (system calls) made by existing processes. A natural way of thinking is that **a process** is created when **a program** is started being executed. However, this is not generally true. On some operating systems, the UNIX-based ones, a new process is created when an existing process *forks* (clones/replicates) itself. That is, when the operating system creates a clone of an existing process that requested it. The new **process** is really a clone of its parent process and right after the fork both processes continue executing the instructions that followed the `fork()` system call. For **a program** to be started, some process needs to use the `execve()` system call (usually by means of one of the `exec*()` functions) which loads and executes the given program, in the process that made the syscall. ### Process tree With the way processes are created, they actually form a process tree where every process has one parent process and zero or more child processes. When a child process terminates, it becomes a so-called *zombie* process and its parent process is responsible for *reaping* it. To do that, the parent process calls one of the `wait*()` functions waiting for the child process to terminate and obtaining information about their run (in particular their exit status). The information about a process is maintained by the operating system until it is reaped. And so until then it also shows up in process listings/tables/etc. If processes form a tree, what is the root node in the tree and how is this process created? And what if a parent process terminates and is reaped before one of its child processes breaking the path from the root process to the still-running child process(es)? ### Special processes The answer to both of the above questions is: Not all processes are born equal. As mentioned in the first paragraph of this post, every process has an identifier -- *PID*. PIDs are assigned to processes when they are created and typically, they are something like sequential numbers of processes. So when there\'s a process with PID `12345`, one can assume there have been at least 12345 processes started in the system since it itself was started[^1]. There are systems with randomized PIDs and various mechanisms for assigning PIDs, but we can ignore those specific cases for now. It then becomes obvious that the root node of the process tree must be a process with PID `0`. And that\'s exactly the case. However, for example on modern Linux systems, the process with PID `0` does not show up in any process table or listing. As described above, processes on UNIX-based systems (incl. Linux) are created by cloning existing processes. But this is not 100 % true! Because for the very first process there is, of course, no process to clone it from! The first process, with PID `0`, is actually a part of the operating system itself, one of the so-called *kernel threads*, *kernel tasks* or *kernel processes* (further referred to as *kernel threads*) and it is usually called *swapper* or *sched* because it is responsible for paging (part of memory management) which is one of the first things necessary for running multiple processes. The first, let\'s say, \"normal\" process is the process with PID `1` -- the *init* process. It is, well..., the *initial* process and it is responsible for, well..., *initializing* the system by setting things up and, in particular, creating child processes and starting programs that themselves will create new processes and start other programs to, eventually, bring the operating system and components running on it into a ready-to-use state. One could expect that the *init* would be the root of the process tree. But looking at the first few lines of output from the `ps` command on a modern GNU/Linux system, we can see it is not the case: ``` example $ ps -eo pid,ppid,cmd|head -n10 PID PPID CMD 1 0 /usr/lib/systemd/systemd --switched-root --system --deserialize 31 2 0 [kthreadd] 3 2 [rcu_gp] 4 2 [rcu_par_gp] 5 2 [netns] 7 2 [kworker/0:0H-events_highpri] 9 2 [kworker/0:1H-events_highpri] 10 2 [mm_percpu_wq] 11 2 [rcu_tasks_kthread] ``` As we can see, there are processes with parent PID (`ppid`) equal to `0`. So their parent process is the hidden very first process with PID `0`. The *init* itself (*systemd* in this case) has `ppid` equal to 0 and then there is another such process -- *\[kthreadd\]*. Its name is in square brackets and it suggests it has something to do with **k**ernel, **thread**s, and acting as a **d**aemon. In fact, it is again a kernel thread, a part of the operating system itself, not a normal process. And we can see other processes with names in square brackets, a convention used for marking kernel threads. Their `ppid`, however, is `2` so their parent process is the *kthreadd* which is thus the kernel thread responsible for creation of other kernel threads. Whether the kernel code actually uses the same mechanisms for creating new kernel threads from *kthreadd* as when forking/cloning new processes is an exercise left to the reader[^2]. An interesting observation from the above example output is that the kernel threads have PIDs greater than the *init* process even though they are actually needed for the *init* process to run. But again, PIDs are assigned when processes are **created** not when they start running a program. And the *init* process is guaranteed to always have PID equal to `1`. The first few lines of the `ps` output only show the *init* process and kernel threads. Conveniently, `ps` sorts the processes based on their PID by default. So what can we see if we look a bit further down in the output? Of course there are more kernel threads. And then, the first \"normal\" processes show up -- *systemd-journald* and *systemd-udevd* in our case. At this point it should be no surprise that they have parent PID equal to `1`. Even further down there is the first process with `ppid` different from `1` and `2` -- *sedispatch* -- a child process of the *auditd* process. ``` example ... 764 2 [xfs-log/dm-4] 765 2 [xfs-cil/dm-4] 766 2 [xfsaild/dm-4] 870 1 /usr/lib/systemd/systemd-journald 892 1 /usr/lib/systemd/systemd-udevd 944 2 [ktpacpid] 950 2 [cfg80211] 951 2 [nv_queue] 952 2 [nv_queue] 953 1 /usr/sbin/dmeventd -f 959 2 [irq/141-iwlwifi] ... 1116 1 /usr/lib/systemd/systemd-oomd 1117 1 /usr/lib/systemd/systemd-resolved 1118 1 /sbin/auditd 1120 1118 /usr/sbin/sedispatch ``` #### The reaper(s) As mentioned above in the [*Process tree section*](#process-tree), when a process terminates, it becomes a *zombie* process and its parent process is responsible for *reaping* it. Until that happens, the operating system maintains the information about the process. And it is actually **only the parent process** that can reap the child process by using one of the `wait*()` functions. But what if the parent process terminates before one or more of its child processes which then become orphan processes? Well, then reaping of such orphan process is a job for *the reaper* (process) which becomes a new parent of the orphan process. Traditionally, the *init* process is the reaper and all orphan processes are assigned to it. Their parent PID is set to `1`, the *init* process is notified (with a `SIGCHLD` signal) every time one of such processes terminates and it can call one of the `wait*()` functions to reap the terminated process. Linux, however, since version 3.4 supports so-called *subreapers* that are described in the `prctl(2)` man page: > A subreaper fulfills the role of init(1) for its descendant processes. When a process becomes > orphaned (i.e., its immediate parent terminates), then that process will be reparented to the > nearest still living ancestor subreaper. Subsequently, calls to getppid(2) in the orphaned process > will now return the PID of the subreaper process, and when the orphan terminates, it is the > subreaper process that will receive a SIGCHLD signal and will be able to wait(2) on the process to > discover its termination status. If we use a trivial example of an orphaned process, we can see these mechanisms in action: ``` example $ echo $$ 333288 $ bash -c 'sleep 10 &'; ps -o pid,ppid,cmd PID PPID CMD 333288 5816 bash 496832 2536 sleep 10 496833 333288 ps -o pid,ppid,cmd ``` A `bash` process started (from an interactive `bash` session with PID `333288` as the value of the special `$$` variable shows) with a command that tells it to create a `sleep 10` process in the background. This `bash` process then terminates immediately and so the `sleep` process it started becomes an orphan process. In the `ps` output we can see the process of the interactive `bash` session, the `sleep 10` process and the `ps` process itself. And as it shows, the parent PID of the `sleep 10` process is `2536`. By using `ps` again: ``` example $ ps -p 2536 -o pid,ppid,cmd PID PPID CMD 2536 1 /usr/lib/systemd/systemd --user ``` we can see that the process with PID `2536` is actually *systemd*! But this time, it is not running as the *init* process (with PID `1`), but as a *User Manager* -- a process that manages user services and many other things and that is a subreaper for orphaned processes started by the given user. And of course, if we run `ps -p 496832` more than 10 seconds after the `sleep 10` was started, it shows no results -- the `496832` process no longer exists which means it was reaped by its (new) parent process. Hopefully, this post helped shed some light on how processes work in UNIX-like systems, the meaning of PIDs and PPIDs, and familiarized you with terminology like process trees, parent and child processes, reaping, zombies, etc. In [the part 2 blog post](processes-part-2), we go more into depth about what happens inside the process, with `fork()`, `exec()`, file descriptors, threads, locks, and more. [^1]: The PIDs of the daemon processes that start during the boot process can actually be used to determine if the system uses a shell script based boot process or something else (like *systemd* or *upstart*) because shell scripts create many processes and so even the PIDs of the daemon processes that start early in the boot process are quite high. [^2]: Hint: `git grep 'pid_t kernel_thread('` in the [Linux sources](https://github.com/torvalds/linux). -------------------------------------------------------------------------------- title: "The state of configuration management" url: /blog/2022/state-of-configuration-management/index.md date: "2022-07-06" -------------------------------------------------------------------------------- Since joining the CFEngine team in 2019 I've heard and [read](https://chorally.com/configuration-management-is-dead/) numerous times that the configuration management market is dying and becoming obsolete. While I and [many others](https://opensource.com/article/19/2/configuration-management-camp) don't personally adopt this line of thinking, I can understand why one would come to this conclusion being that we're in an ever-changing industry and talking about solutions that have been around for decades. Configuration management solutions like CFEngine are certainly not a new concept, however there are many changes that are happening across the industry that will continue to drive usage and will ultimately pave the way for a new era in this market. First, let's look at the market holistically. If you take a look at the technology adoption curve below most would probably agree that the configuration management market is currently in either the "Late Majority" or "Laggard" phase. If true, then based on the graph, adoption is near its end, right?... It's not that simple. ![](https://assets.speakcdn.com/assets/2304/17qmyfkqueidu8iwb9mxezq.png) Most everyone has heard the saying, "the more things change, the more they stay the same." That's especially true when it comes to configuration management. Over the last several years, newer technologies like containers have created a lot of hype, some of it very real and useful. On the other hand, in the early days of containers we were [reading](https://www.networkworld.com/article/2910559/will-containers-kill-the-virtual-machine.html) about how they would displace "the cloud". However, we know that in IT one size doesn't fit all. In fact, according to a 2020 [survey](https://www.oreilly.com/radar/cloud-adoption-in-2020/) by O'Reilly, almost 50% of organizations surveyed continue to run traditional, on-premise infrastructure and more than 90% of organizations expect their cloud based infrastructure to increase. IT environments are only becoming more heterogenous and showing no signs of changing in the near future, creating more of a need for tools like CFEngine that can help minimize complexity vs. multiple point solutions that only increase complexity. > Config management is definitely not dead. Infrastructure-as-code is also not dead, but it all is evolving. *-Kris Buytaert, Founder of CfgMgmtCamp* We've also seen an expansion in use cases for configuration management solutions. Improving efficiency through automation has always been and continues to be a key use case, but as needs & requirements of organizations have evolved and the increasing threat of security breaches, the scope of configuration management solutions has only expanded to become a more critical component of infrastructure environments. Trend Micro [claims](https://www.trendmicro.com/en_us/research/21/a/the-top-worry-in-cloud-security-for-2021.html) that 65-70% of all security challenges arise from simple misconfigurations. In this same report, they also break down the 12 most significant breaches in recent years, including Capital One, US DoD, Accenture, etc. What do each of these 12 breaches have in common? Every one of them was caused by at least one misconfiguration. These types of security concerns have been the driver for various compliance frameworks such as CIS, NIST, PCI, HIPAA, and others. Mix all this with the [consolidations](https://www.geekwire.com/2022/portland-cloud-infrastructure-company-puppet-to-be-acquired-by-perforce-software/) and [acquisitions](https://www.globenewswire.com/news-release/2020/10/06/2104339/0/en/Progress-Completes-Acquisition-of-Chef.html) in the space and it's clear that we are entering a new era in configuration management that encompasses Development, Security & Operations, more commonly known as DevSecOps. While the days of seeking "hockey stick" growth that many mainstream configuration management solutions aim to achieve may be behind them, this was never the intention for CFEngine. Born out of [academic research](https://people.scs.carleton.ca/\~soma/biosec/readings/burgess-immunology.pdf) and the concept of [promise theory](https://en.wikipedia.org/wiki/Promise_theory), CFEngine has always embraced being a little different. Of course generating revenue to sustain a profitable company is necessary to continue innovating, however we have prioritized customer satisfaction over the traditional milestones of endless VC funding and/or "going public". This strategy has served us well as we continue to scale organically. Because of our unique position in the market, we are now in an equally unique position for continued success as the needs & requirements of organizations mature and the new era of configuration management is ushered in. -------------------------------------------------------------------------------- title: "CFEngine 3.20 released - Modularity" url: /blog/2022/cfengine-3-20-released-modularity/index.md date: "2022-07-01" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.20.0\! Over the past few years we've focused on ease of use, new user experience, and out of the box value, giving you the ability to do much more through only the Mission Portal Web UI. This has resulted in several important steps forward; policy analyzer, compliance reports, host specific data (CMDB), and CFEngine Build with custom promise types and other modules. In this release we take this one step further, focusing on modules and modularity, and the ability to find and use modules, making changes to your infrastructure, directly from Mission Portal (API or GUI). The 3.20 release is not supported, but it's an excellent opportunity for our users to see and test what will be in the next release - CFEngine 3.21 LTS, about 6 months from now. ## What's new ### CFEngine Build in Mission Portal With CFEngine Build, we want to enable collaboration between community members and the CFEngine team, and ease the management of policy and modules from multiple sources. Additionally we want to deliver valuable functionality in the form of modules directly to our users outside of the regular release cycle. Until now, this has been done with the use of [build.cfengine.com](https://build.cfengine.com) and the `cfbs` command line tool. The website is great for finding new modules and seeing what others have made, and the tooling allows you to download, manage, update, and build modules into a deployable policy set. Today, we bring this functionality into Mission Portal; you can browse for modules, add, and deploy on your hub, all within the web UI: [![](./images/welcome-screenshot.png)](./images/welcome-screenshot.png) #### Getting started The first time you open the Build application, you will be prompted to create a project, and set it up to synchronize with your git repository. If you are unsure here, we recommend creating a repository on GitHub, choosing to generate an SSH key in Mission Portal, and adding that to your GitHub repository settings, under "Deploy keys". If you just want to check out the UI, you can skip the git repo settings, but we recommend setting it up, so your changes are saved and pushed somewhere and you can deploy your project to your hub(s). After creating your project, you see the Build UI and a guide for quickly getting started: [![](./images/welcome-screenshot-tight.png)](./images/welcome-screenshot-tight.png) You can search for modules just like on [build.cfengine.com](https://build.cfengine.com): [![](./images/search-inventory.png)](./images/search-inventory.png) Clicking on modules gives you some of the same information as on the website, and a button to add (or remove) the module: [![](./images/module-info.png)](./images/module-info.png) After adding modules you can see them on the left: [![](./images/added-modules.png)](./images/added-modules.png) #### Deploying In the top right corner you can deploy your changes: [![](./images/deploy-button.png)](./images/deploy-button.png) The first time you click the deploy button, you will be asked if you want Mission Portal to change the settings so it starts using this repository to pull policy from: [![](./images/deploy-modal.png)](./images/deploy-modal.png) After doing this, any `git push` to that repository, including the ones you make by clicking _Deploy_ in Mission Portal, will result in the policy being updated on your hub, and thus the rest of your hosts on this hub. In the background, Mission portal uses the `cfbs` and `git` command line tools to manage your project. This also means you can work with the same project in your terminal and editor, or whatever tools and websites you use to work with git repositories. Here we can see the commits we made in Mission Portal from the GitHub UI: [![](./images/commits.png)](./images/commits.png) **Tip:** CFEngine Build in Mission Portal will never _force push_ to your git repository, so commit history is never lost. This means, in case of bugs or human errors, you always have the option to go back and revert undesired changes. #### Updating If there is a new version available for one of your modules, this will be highlighted by the UI: [![](./images/update.png)](./images/update.png) And you can easily upgrade: [![](./images/update-modal.png)](./images/update-modal.png) **Tip:** The same goes for `masterfiles`, the first module and the default policy set we provide. The traditional way of managing your policy is to make a _fork_ of our default policy set and have all your policy mixed into it. The first step of upgrading to a new version of CFEngine is to upgrade your policy, which can be painful with this approach. Once you are using CFEngine Build to manage your policy, upgrading to a new version of the framework and standard library is done in seconds, not days. #### Separation of data and policy It is quite common to want to separate policy from data (configuration values). Policy might be shared across multiple environments and teams, while each hub has its own CMDB of configuration values. Additionally you may want some teams to be able to edit policy, while others only change values in the database. Today, with CFEngine Build in Mission Portal, and the CMDB functionality we've introduced previously, this is possible. Some modules already look for data from the CMDB and act based on it, for example: [![](./images/telnet-exceptions.png)](./images/telnet-exceptions.png) The module for uninstalling telnet allows you to make an exception for specific hosts, using _Host Specific Data_. We will go further along this path, enabling more powerful input-based modules, and giving you a nice UI to specify inputs on many levels: project, hub, subset of hosts on a hub, or individual host. #### Video In this developer demo video, you can see the new functionality in action: {{< youtube kkmBdJW8HcM >}} ### CFEngine Build System command line tool Now, with CFEngine Build in Mission Portal, the `cfbs` command line tool is included in all Hub installations. We've also made several improvements to this tool, such as interactive prompts, automatic git commits, version selection, and more. Read more about this in the [`cfbs` version 2 blog post](/blog/2022/cfbs-2-0-0/). ### Bodies for custom promise types Custom promise types can now use custom bodies. These new bodies look exactly like the built in body types, to a policy writer there is no difference. For example, take a look at the experimental groups promise type: ```cf3 body members foo { include => { "alice", "bob" }; exclude => { "malcom" }; } bundle agent main { groups: "foo" policy => "present", members => foo; } ``` To learn more about custom bodies, [read the blog post](https://cfengine.com/blog/2022/custom-bodies/). ### Changes in behavior In the past few months we've announced some changes coming in this release: - [Renaming bundle agent main in MPF](/blog/2022/renaming-bundle-agent-main-in-mpf/) - [Files promises creating files by default](/blog/2022/create-true/) - [Multiple cf-execd processes](/blog/2022/multiple-cfexecd-processes/) Although we believe most users won't be affected by this in a negative way, it is a good idea to look through them. ### Inventory refresh and compliance report widgets This release includes the new inventory refresh handling, compliance report design and widget which we also included in 3.18.2. See the [3.18.2 release announcement](/blog/2022/cfengine-3-15-6-and-3-18-2-released/) for more details on these features. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.20.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.20/guide-latest-release-whatsnew-changelog-core.html) - [3.20.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.20/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.20.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.20/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the [3.19.0](/blog/2021/cfengine-3-19-released-collaboration/) release from half a year ago, these dependencies have been updated: | CFEngine version | 3.19.0 | **3.20.0** | | :-------------------------------------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.51 | **2.4.53** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.34.1 | **2.36.1** | | [libcurl](https://curl.haxx.se/download.html) | 7.80.0 | **7.83.1** | | [libiconv](https://www.gnu.org/software/libiconv/) | 1.16 | **1.17** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.6.0 | **2.6.2** | | [OpenSSL](https://openssl.org/) | 1.1.1l | **3.0.4** | | [PHP](https://php.net/) | 8.1.0 | **8.1.7** | | [PostgreSQL](https://www.postgresql.org/) | 13.5 | **14.3** | | [rsync](https://rsync.samba.org/) | 3.2.3 | **3.2.4** | | [zlib](https://zlib.net/) | 1.2.11 | **1.2.12** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 14 - Sneak peek of CFEngine 3.20" url: /blog/2022/the-agent-is-in-episode-14-sneak-peak-of-3-20-release/index.md date: "2022-06-30" -------------------------------------------------------------------------------- Interested in seeing what's around the corner in CFEngine Enterprise? Herman (Product Manager) walks us through the features that we will find in the upcoming release of CFEngine 3.20. Being a non-LTS release, it's a great opportunity to try out new features and get your feedback in before our next LTS series. ## Video {#video} The video recording is available on YouTube: {{< youtube LlCVcGY_kM4 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [Download CFEngine Enterprise](/downloads/cfengine-enterprise/) - [CFEngine Build](https://build.cfengine.com) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.15.6 and 3.18.2 released" url: /blog/2022/cfengine-3-15-6-and-3-18-2-released/index.md date: "2022-06-29" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.15.6 and 3.18.2\! These releases mainly contain bug fixes and dependency updates. ## What's new Some smaller features and improvements were added to 3.18.2. Most of these are centered around newer functionality, such as compliance reports. ### Compliance report widgets and improved UI Compliance reports are one of our most powerful report types, allowing you to compile all your security and compliance requirements into _one_ checklist, and easily see exactly how many hosts are failing and passing each check. These reports can now be turned into widgets on your Mission Portal dashboard: [![](./images/compliance-report-widget-only.png)](./images/compliance-report-widget-only.png) By clicking on "Open full report" you'll see that the interface has been updated and simplified a bit: [![](./images/new-compliance-report.png)](./images/new-compliance-report.png) And by clicking the results of an individual check, you can quickly see which hosts are passing and failing, the relevant inventory attribute, and filter for only one or the other: [![](./images/compliance-check-results.png)](./images/compliance-check-results.png) ### New add widget menu As we've been working on the compliance report widget, we wanted to improve the user experience of adding new widgets as well. The new menu makes a clear distinction between creating new and reusing existing widgets, and also adds helpful visuals and descriptions to ensure finding the right widget is as easy as possible: [![](./images/add-widget.png)](./images/add-widget.png) (This screenshot shows the dark mode UI, which we are continuously improving, but all of the smaller changes would be too many to list here). ### Faster inventory refresh and host specific data update Thanks to a new component, `cf-reactor`, more things are now handled in an event-based manner. This means that when you change host specific data (CMDB), this will be written to file immediately and ready for the host to fetch. Similarly, when collecting reports from a host, the refresh of inventory information on that host in the database is also performed immediately. The end result of this is that you don't have to wait for one or multiple agent runs for these things to happen. You can now change a variable on the host info page, click the play button to trigger an agent run, and you will see the updated variable reported back from the host. For more information on how `cf-reactor` reacts to events and synchronizes data between PostgreSQL and files, see our developer blog post: https://cfengine.com/blog/2022/postgresql-to-json-files-sync/ ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.15.6 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [3.15.6 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.15.6 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.18.2 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.2 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.2 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. | CFEngine version | 3.15.5 | **3.15.6** | 3.18.1 | **3.18.2** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.51 | **2.4.53** | 2.4.51 | **2.4.53** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.33.1 | **2.36.1** | 2.33.1 | **2.36.1** | | [libcurl](https://curl.haxx.se/download.html) | 7.79.1 | **7.83.1** | 7.79.1 | **7.83.1** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.4.58 | 2.4.58 | 2.6.0 | **2.6.1** | | [PHP](https://php.net/) | 7.4.25 | **7.4.29** | 8.0.12 | **8.0.19** | | [PostgreSQL](https://www.postgresql.org/) | 12.8 | **12.11** | 13.4 | **13.7** | | [rsync](https://rsync.samba.org/) | 3.2.3 | **3.2.4** | 3.2.3 | **3.2.4** | | [zlib](https://zlib.net/) | 1.2.11 | **1.2.12** | 1.2.11 | **1.2.12** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. Since version 3.18.2 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.15.6 list ``` ```command cf-remote --version 3.18.2 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - [Explore CFEngine Build](https://build.cfengine.com/) for modules made by the CFEngine team and community, and share your own modules - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Change in behavior: multiple cf-execd processes" url: /blog/2022/multiple-cfexecd-processes/index.md date: "2022-06-15" -------------------------------------------------------------------------------- Recently we introduced new feature where you can [trigger agent runs and report collection](https://cfengine.com/blog/2021/trigger-agent-runs-and-report-collection/) from the Mission Portal UI. [![UI Screenshot showing play and update buttons, with a tooltip saying Trigger Agent Run.](https://cfengine.com/blog/2021/images/trigger.png)](https://cfengine.com/blog/2021/images/trigger.png) This required our daemon `cf-execd` to behave a bit differently when periodic agent runs occur. Previously the daemon would create a new thread in which to run `cf-agent`, capture output, wait for completion and move on. We changed the behavior so that the daemon forks itself and then fork/execs `cf-agent` as before, with the forked `cf-execd` processing agent run output. When the agent is finished the forked `cf-execd` process is left a zombie/defunct. The daemon wakes up every minute to see if it should do an agent run. The next time the original daemon `cf-execd` "wakes up" it will clean up that defunct forked `cf-execd`. So now two cf-execd processes are expected around the time of each cf-agent run, either the periodic run or the runs triggered by Mission Portal UI. ## Watchdogs keep things tidy Many system administrators wisely use watchdog tools to ensure that the correct number of processes are present for the services they implement. Many commercial and open source projects exist to monitor systems and take action. If you use such a system to ensure that there is only one cf-execd process, you will need to adjust your setup to allow two processes if you are running version 3.18.0 or greater. There is a [built-in watchdog](https://github.com/cfengine/masterfiles/tree/master/cfe_internal/core/watchdog) in the Masterfiles Policy Framework that is not enabled by default, which we had to adjust when this change was implemented. This watchdog currently works on three platforms: Linux, AIX and Windows, each with different functionality. The [linux version](https://github.com/cfengine/masterfiles/blob/master/templates/cfengine_watchdog.mustache) is a crontab entry which ensures that at least one cf-execd process is present. The [Windows version](https://github.com/cfengine/masterfiles/blob/master/cfe_internal/core/watchdog/templates/watchdog-windows.ps1.mustache) ensures a threshold of time and number of processes of cf-agent. And most similar to what many administrators may have configured is the [AIX watchdog](https://github.com/cfengine/masterfiles/blob/master/cfe_internal/core/watchdog/templates/watchdog.mustache) which tracks several different problem cases or pathologies and takes action if needed. ## Adjust your thresholds So remember to adjust any monitoring software which expects only one `cf-execd` daemon to run and allow it to run 2. And if you don't have monitoring software already doing this work, consider using our [built-in watchdog](https://github.com/cfengine/masterfiles/tree/master/cfe_internal/core/watchdog). -------------------------------------------------------------------------------- title: "CFEngine Build System version 2" url: /blog/2022/cfbs-2-0-0/index.md date: "2022-06-14" -------------------------------------------------------------------------------- A while back we released version 2 of [cfbs](https://pypi.org/project/cfbs/), and even though we release versions of this tool quite [frequently](https://github.com/cfengine/cfbs/releases), without announcing it on the blog, we thought this was a good opportunity to talk a bit about the tool, what's new and our direction with it in the future. The reason why we called this the "2.0" release is that we are trying to follow semantic versioning, and there were some big new features in the release which could be considered breaking changes. ## Philosophy The `cfbs` command line tool is responsible for 2 main tasks: 1. Editing CFEngine Build projects (adding modules, policy files, updating etc.) 2. Building a project into a ready to deploy policy set An important distinction here is that we imagine the first part, working with projects, is mainly something users will be doing, manually. The second part, building policy for deployment, is often done in automation, you can have scripts running `git clone ; cfbs build` on your hub or as part of your CI/CD pipeline. ### The build command Since users are building scripts and automation around `cfbs build` we want this command to be non-interactive and backwards compatible. We will try our very best to not break your policy builds by changing the behavior here, and if we have to at some point, we will communicate it widely. ### Other commands The other commands (`init`, `add`, `update`, `remove`) we expect users to be running from a terminal, and we want them to be as easy to use as possible, especially for new users and as we add more functionality. This means we are changing them a bit more than the `build` command, and in many cases making them have interactive prompts to guide users to achieving what they want. Backwards compatibility is not our primary focus here. So, if you intend to build scripts or tests or automation that use these commands, just be aware that they might change in the future. That being said, we are using these commands inside tests and upcoming functionality internally, so we won't be changing them without good reasons. ## What's new in version 2+ As mentioned before, we release quite frequently, and are already at version 2.2.2. So, instead of focusing on exactly what was added between 1.3.3 and 2.0.0, we'd just like to highlight some of the biggest improvements we made in the last months. ### Release numbers and matching versions from outside of CFEngine Build It is now possible for modules like [masterfiles](https://build.cfengine.com/modules/masterfiles/) and [SURF CFEngine Library](https://build.cfengine.com/modules/surf-cfengine-library/) to follow their existing versioning, and if they need to make small changes and module releases between versions, they can add a release number at the end of their version number (`-1`, `-2`, `-3`): [![](./images/mpf.png)](./images/mpf.png) ### Interactive prompts As hinted at above, we want many of `cfbs`'s commands to be interactive and help the user with what they need: ``` $ cfbs init Please enter name of this CFBS repository [Example]: My new project Please enter description of this CFBS repository [Example description]: Just an example for the blog post ``` In all of the prompts you can keep pressing enter to use the defaults, and we try to pick defaults that make sense for most users, especially new ones. Currently, the prompts are quite basic, but in the future we'd like to add more interactivity like choosing between different options using the arrow keys, as an example. ### Helping users add masterfiles Most users will want to add masterfiles as their first module, especially new users. This module contains the default policy that is probably already running on their hub, and which does many important things for CFEngine and CFEngine Enterprise to work as expected. Previously we had a log message which told the user to run `cfbs add masterfiles`. Now that has been changed to a prompt, and by default, if you press enter, it will be added: ``` Do you wish to build on top of the default policy set, masterfiles? (Recommended) [YES/y/no/n]: y Added module: masterfiles ``` ### Git commits The commands you use to edit your project (`cfbs.json`) now make git commits by default. This allows you to easily track the changes made to your project and push them to a remote git repository. ``` $ cfbs add autorun Added module: autorun The default commit message is 'Added module 'autorun'' - edit it? [yes/y/NO/n] Committing using git: [main 9403c3c] Added module 'autorun' 1 file changed, 12 insertions(+) $ ``` To disable this functionality, you can answer `no` to the question about using git in `cfbs init`, or set `"git": false` in your `cfbs.json`. ### Specify versions If there is a problem with the latest version of a module, you can now easily look through the other versions on [the website](https://build.cfengine.com/) and add the one you want: ``` $ cfbs add promise-type-git@0.1.0 ``` ## A sneak peek Without revealing too much, here is a sneak peek of what comes next: [![](./images/build-in-mp.png)](./images/build-in-mp.png) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 13 - Custom promise types" url: /blog/2022/the-agent-is-in-episode-13-custom-promise-types/index.md date: "2022-05-26" -------------------------------------------------------------------------------- Interested in extending the CFEngine DSL to support your own custom promise types? Herman (Product Manager) walks us through implementing custom promise type in python. ## Video {#video} The video recording is available on YouTube: {{< youtube CBttMFhTZuU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Example code](https://github.com/cfengine/example-promise-type-alerting-system) - [Library for custom promise types in python](https://build.cfengine.com/modules/library-for-promise-types-in-python/) - [Learning CFEngine](https://leanpub.com/learning-cfengine) - [Thinking in Promises](https://www.oreilly.com/library/view/thinking-in-promises/9781491917862/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "5 improvements coming to the CFEngine Docs" url: /blog/2022/5-improvements-coming-to-the-cfengine-docs/index.md date: "2022-05-20" -------------------------------------------------------------------------------- This year, we are improving CFEngine's ease of use, so it should come as no surprise that we need to rethink our documentation site. For software like CFEngine, great documentation is not just "nice to have", but a must for our users, both beginners and experienced. CFEngine has helped DevOps teams to automate their day-to-day tasks and make better decisions by providing a holistic overview of their systems. Part of our responsibility is to help users quickly understand how to conduct standard jobs while giving them insight into the possibilities of the product and features they might find useful next. In starting the revamping of our docs site, we assessed the content available on [docs.cfengine.com](https://docs.cfengine.com/docs/master/) and listed the most fundamental improvements that can maximize the user experience. In this blog post, we will share the top 5 things we do to improve CFEngine docs and why we do it. Enjoy! ## Work in progress We've already started working on the following 5 topics, and so we'd like to share some teasers of what to expect. ### 1. New look For the new docs site, we are updating the visual design to be more modern, intuitive, and in line with our other new websites, [cfengine.com](https://cfengine.com) and [build.cfengine.com](https://build.cfengine.com). This brings consistency throughout different platforms of CFEngine. [![](./images/new-look.png)](./images/new-look.png) ### 2. New information architecture We started the project by reviewing the information architecture of the content because it is the essence of the site and a key criteria for user satisfaction. After listing all the existing content and comparing them to other sites, we came up with a more logical and easy-to-use structure. We increased the main categories from four items to eight. Those will cover a holistic overview of the content and offer users a good and fast entry to find the information they need in the most logical way. [![](./images/new-information-architecture.png)](./images/new-information-architecture.png) ### 3. New homepage A good homepage allows users to orient themselves and look for information effectively. Therefore, we designed a brand new homepage for the CFEngine docs site. Here we divided the page into two sections which aim to serve different user competence levels. The getting started section enables users to find the installation instructions easily, and the popular section is where some of the key resources are highlighted and easily accessible to users. [![](./images/new-homepage.png)](./images/new-homepage.png) ### 4. New Navigation In the past, we have received feedback from the users that navigation was hard to understand and use. For this reason, we simplified the site navigation by integrating the top navigation bar into side one and alleviating its functionality to expand the content without jumping to another page. Users can, therefore, get a good overview of where they currently are and where they can go next. Since the top navigation bar is removed and all navigation happens in the left navigation tree, the experience is a lot more consistent, and you avoid the situations where the left navigation menu was changing unexpectedly. [![](./images/new-before-navigation.png)](./images/new-before-navigation.png) [![](./images/new-after-navigation.png)](./images/new-after-navigation.png) ### 5. Mobile view Do you need to find something on the go? No problem! We've got you covered. The new design is responsive, with custom navigation for mobile so that you can find the information you need while having lunch! ![](./images/mobile-view.gif) ## What's next? We're excited about the improvements that are in progress and we plan to continue improving the site. Here are a few more things to look forward to: * Better in-site search * More examples throughout the documentation * Simplified and updated getting started tutorial series with videos Like always, we value your feedback and would love to hear from you. You can share your ideas, and thoughts by sending us an email. If you would like to contribute to our documentation, you can find the information here [https://github.com/cfengine/documentation](https://github.com/cfengine/documentation "https://github.com/cfengine/documentation"). I hope you enjoy the new CFEngine Docs. -------------------------------------------------------------------------------- title: "The agent is in - Episode 12 - Spacemacs for CFEngine" url: /blog/2022/the-agent-is-in-episode-12-spacemacs-for-cfengine/index.md date: "2022-04-28" -------------------------------------------------------------------------------- Looking to be more efficient writing CFEngine policy? Nick Anderson (Doer of Things) walks us through setting up Spacemacs for CFEngine. Get syntax highlighting, on the fly error checking, function prototypes, integration with the venerable org-mode and more! ## Video {#video} The video recording is available on YouTube: {{< youtube rlJiIdhHYUY >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Spacemacs](https://www.spacemacs.org/) - [CFEngine Spacemacs Layer](https://develop.spacemacs.org/layers/+tools/cfengine/README.html) - [Other editors with CFEngine support](https://docs.cfengine.com/docs/master/examples-tutorials-writing-and-serving-policy-editors.html#:~:text=editors) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Change in behavior: Creating files by default" url: /blog/2022/create-true/index.md date: "2022-04-22" -------------------------------------------------------------------------------- In the upcoming CFEngine 3.20 release we are making a change in the behaviour of the `create` attribute for the files promises that manage the entire content of a file. This includes promises with the template methods `mustache`, `inline_mustache` and `cfengine`; as well as promises with the `content` attribute. The motivation behind these new changes is two-fold; make it easier to learn CFEngine policy language and understand what policy is doing, and to prevent CFEngine from creating empty configuration files. ## Problem 1 - Readability and ease of use **Note:** The examples below are for the `content` attribute for simplicity, but the same exact arguments apply to templates as well. Consider the following policy: ```cf3 bundle agent main { files: "/tmp/hello.txt" content => "Hello, world!"; } ``` This might be the first piece of policy someone reads. It seems quite clear what the intent is here, the user wants a file called `/tmp/hello.txt`, with the content `Hello, world!`. However, historically, and before the `content` attribute was introduced, the behavior for files promises in CFEngine was to not create files by default. This can make sense in some scenarios, if you are only interested in changing very specific parts of a file (remove a line, change a permission, etc.). The old-school way of specifying you want the file to be created if it doesn't exist is using the create attribute: ```cf3 bundle agent main { files: "/tmp/hello.txt" content => "Hello, world!", create => "true"; } ``` Nobody knows instinctively that files promises work this way, that you have to specify `create => "true"`, this is something you just have to learn (maybe try and fail first). On the other hand, our first example without `create` made a lot of sense, it seemed clear, and should just work. If we only want to do something if the file exists, we can express that perfectly in policy language: ```cf3 bundle agent main { files: "/tmp/hello.txt" content => "Hello, world!", if => fileexists("/tmp/hello.txt"); # Or $(this.promiser) } ``` Now, this policy reflects how it was described in an English sentence, and it can even be understood by someone who doesn't know CFEngine policy language. So, to make things easier to learn, read, and understand we are changing the default here so that files are created by default in the cases where it definitely makes sense; templates and promises which manage the full contents of the file. Making this change across all files promises, even those that just change a permission or a single line, would be a big change, and not something we want to rush. ## Problem 2 - Empty files Another related problem we'd like to fix is promises which should create a file and fill it with data from a template. Say you have some policy for hosting an Apache 2 web server. You may have a bundle including promises to make sure that the `apache2` package is installed: ```cf3 bundle agent webserver { meta: webserver:: "tags" slist => { "autorun" }, comment => "Enable autorun"; packages: "apache2" policy => "present", package_module => apt_get, comment => "Install apache2 package"; ``` Furthermore you may have a promise to render the contents of the Apache 2 configuration file. In this example I've used the template method `cfengine`. ```cf3 vars: "name" string => "www.website.com", comment => "Website hostname"; "admin" string => "admin@localhost", comment => "Website administrator"; "root" string => "/var/www/website", comment => "Website root directory"; files: "/etc/apache2/apache2.conf" create => "true", template_method => "cfengine", edit_template => "$(sys.workdir)/templates/apache2.conf.tmpl", classes => results("bundle", "config"), comment => "Edit apache2 config"; ``` Finally, whenever changes are made to the Apache 2 config file, we'll need to restart the Apache 2 service in order to bring the changes into effect. ```cf3 services: config_repaired:: "apache2" service_policy => "restart", comment => "Restart apache2 service"; } ``` If you now bootstrap a fresh host to a hub with the following policy; the files promises would be executed first - due to CFEngine [normal ordering](https://docs.cfengine.com/docs/3.18/reference-language-concepts-normal-ordering.html) - creating the configuration file with the rendered content. Next the packages promises would be executed, installing the "apache2" package. The installation script would see that the configuration file was already there, thus it would avoid overwriting it. Perfect! Everything works as expected. But what if the rendering of the configuration file would fail to render? In this case, an empty configuration file would be created, causing the web server to break. This is not really what we want, is it? Of course you could work around this by controlling the policy execution; using the [`depends_on`](https://docs.cfengine.com/docs/3.18/reference-promise-types.html#depends_on) attribute: ```cf3 files: "/etc/apache2/apache2.conf" create => "true", template_method => "cfengine", edit_template => "$(sys.workdir)/templates/apache2.conf.tmpl", classes => results("bundle", "config"), comment => "Edit apache2 config", depends_on => { "apache2_package_present" }; packages: "apache2" policy => "present", handle => "apache2_package_present", package_module => apt_get, comment => "Install apache2 package"; ``` But still, it does not make sense that the files promise leaves behind an empty file. We'd like CFEngine to render the template and only create the file if there is content to put in it. ## Conclusion In its essence, CFEngine is a configuration management tool. The most important promise type is the files promise, which is used to configure a system through managing the content of configuration files. CFEngine should therefore be designed around this use case. To avoid issues as a result of empty configuration files, the new behavior of files promises is as follows: * Files are created by default after rendering succeeds (for content / templates). * Specify `create => "true"` if you want the file to always be created. * Use `if => fileexists("$(this.promiser)")` to only edit files which exist (This has the same effect as `create => "false"`, but the `if` is preferred). * The behavior of other files promises is unchanged (such as permissions, line editing, etc.) ## Share your thoughts If you want to play around with these new changes today; feel free to check out our packages from the master branch. For instructions on how, please see [Installing CFEngine Nightlies using cf-remote](https://cfengine.com/blog/2020/cfengine-nightlies/). Please let us know what you think over at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear your feedback. -------------------------------------------------------------------------------- title: "Change in behavior: Renaming bundle agent main" url: /blog/2022/renaming-bundle-agent-main-in-mpf/index.md date: "2022-04-11" -------------------------------------------------------------------------------- A [recent change in the Masterfiles Policy Framework](https://github.com/cfengine/masterfiles/pull/2272) (MPF) is renaming `bundle agent main` to `bundle agent mpf_main`. [![What's in a name: Source https://stijndewitt.files.wordpress.com/2019/11/whats-in-a-name-shakespeare.jpg](images/whats-in-a-name-shakespeare.jpg)](images/whats-in-a-name-shakespeare.jpg) This change is intended to make it easier to run individual parts of your policy leveraging the [library main bundle](https://docs.cfengine.com/docs/3.18/reference-language-concepts-bundles.html#library-main-bundles) functionality (`bundle agent __main__`). Library main bundles were first introduced in [CFEngine 3.12.0](https://cfengine.com/blog/2018/cfengine-3-12-0-lts-released/). The functionality allows for the definition of `bundle agent __main__`. When this bundle definition is present in the policy entry (the first policy file that CFEngine reads) the bundle is understood to be used as the default bundlesequence. This example illustrates the behavior: ```cfengine3 bundle agent __main__ { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` When we run the policy without providing a bundlesequence (the `--bundlesequence` or `-b` option to `cf-agent`), and since `body common control` is absent, not defining `bundlesequence`, the `__main__` bundle is used in the place of `main`. ```sh cf-agent --no-lock --file /tmp/example-0.cf ``` ```text R: Hello World, from /tmp/example-0.cf ``` We can see that if we instead define the bundle as `main`: ```cfengine3 bundle agent main { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` The behavior is identical when the policy is executed: ```sh cf-agent --no-lock --file /tmp/example-1.cf ``` ```text R: Hello World, from /tmp/example-1.cf ``` So, why are we renaming `bundle agent main` in `services/main.cf` to `bundle agent mpf_main`? We are renaming it because you will get errors if you execute a policy entry that has `bundle agent __main__` and there also exists `bundle agent main` within the inputs of that policy execution. For example, if we place both `bundle agent main` and `bundle agent __main__` in the same file: ```cfengine3 bundle agent main { reports: "Hello World, from $(sys.policy_entry_filename)"; } bundle agent __main__ { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` When executing, we get an error about duplicate definition of bundle. ```sh cf-agent --no-lock --file /tmp/example-2.cf | head -n 2 ``` ```text /tmp/example-2.cf:1:0: error: Duplicate definition of bundle main with type agent /tmp/example-2.cf:6:0: error: Duplicate definition of bundle main with type agent error: There are syntax errors in policy files error: Policy failed validation with command '"/home/nickanderson/.cfagent/bin/cf-promises" -c "/tmp/example-2.cf"' ``` This error is emitted because `bundle agent __main__` is internally translated to `bundle agent main` and since there already exists a `bundle agent main` it's seen as a duplicate. The same thing happens if there is `bundle agent __main__` in the policy entry and `bundle agent main` in another file added to _inputs_. In this example, `/tmp/example-3.cf` (which has `bundle agent __main__`) loads `/tmp/example-4.cf` (which has `bundle agent main`) as part of its inputs. ```cfengine3 body file control { inputs => { "example-4.cf" }; } bundle agent __main__ { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` ```cfengine3 bundle agent main { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` Again, we see that running `/tmp/example-3.cf` as the policy entry results in duplicate definition of bundle errors: ```sh cf-agent --no-lock --file /tmp/example-3.cf | head -n 2 ``` ```text /tmp/example-3.cf:5:0: error: Duplicate definition of bundle main with type agent /tmp/example-4.cf:1:0: error: Duplicate definition of bundle main with type agent error: Policy failed validation with command '"/home/nickanderson/.cfagent/bin/cf-promises" -c "/tmp/example-3.cf"' error: CFEngine was not able to get confirmation of promises from cf-promises, so going to failsafe ``` Note, it's not a problem in the other direction. If you have `bundle agent main` in your policy entry that loads another file to inputs which has `bundle agent __main__`, there is no error since `bundle agent __main__` in the non policy entry file is not internally translated to `bundle agent main`. In this example we reverse the previous example so that the policy entry file contains `bundle agent main` and `bundle agent __main__` is present in a file loaded to inputs. ```cfengine3 body file control { inputs => { "example-5.cf" }; } bundle agent main { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` ```cfengine3 bundle agent __main__ { reports: "Hello World, from $(sys.policy_entry_filename)"; } ``` When executing we can see that no errors are emitted. ```sh cf-agent --no-lock --file /tmp/example-5.cf ``` ```text R: Hello World, from /tmp/example-5.cf ``` So, we are renaming `bundle agent main` to `bundle agent mpf_main` so that you can run some arbitrary policy file which itself somehow includes `services/main.cf`. What do I need to do? Upgrade your policy framework as usual just be sure that you don't ignore the upstream change to `services/main.cf`. If you have variables (e.g. `main.myvar`, `default:main.my_var`) or methods promises ( `usebundle => main`, `usebundle => default:main`) you should update them to `mpf_main`. This change comes with policy from the upcoming 3.20.0 release, it's backwards compatible, so it's perfectly safe to run new policy with this change on binaries prior to 3.20.0. -------------------------------------------------------------------------------- title: "Student competition - Build a module and win cash" url: /blog/2022/student-competition/index.md date: "2022-04-07" -------------------------------------------------------------------------------- The CFEngine team is pleased to announce a competition for students in Norway. We want you to write a module in Python, and submit it to [CFEngine Build](https://build.cfengine.com). Your module will be Open Source (MIT License), available for our community of users worldwide. CFEngine is a programming language, and modules can be added to do whatever the user needs, so the possibilities are endless. You can look at some examples for inspiration at the end of this blog post. ## Prizes The prizes are as follows: 1. 🥇 **10 000 NOK** for the winner(s) 2. 🥈 **2 000 NOK** for 2nd place 3. 🥉 **1 000 NOK** for 3rd place We will judge the quality and value of your module (how well it's made and how useful it is to our users). Anyone can submit modules to CFEngine Build, but only students at UiO, UiB, OsloMet, and NTNU are eligible for prizes. (If you are studying at another Norwegian university / college, please send us an email and we may be able to accept you as well.) We will ask for some transcripts / documentation from the winners to verify this. Teams of up to 3 are allowed, but you will have to split the prize money. ## What you have to do 1. Complete [getting started tutorial](https://docs.cfengine.com/docs/3.21/getting-started.html) 2. _Optional:_ Talk to us about what you want to make: [student-2022@cfengine.com](mailto:student-2022@cfengine.com) 3. Use [our template](https://github.com/cfengine/promise-type-template) and start coding! **Estimated time spend:** 1-2 work days **Deadline:** The deadline for submitting your module is April 25th 2022. ## The python module you will make To give you an idea of what your module will look like, here is our template: ```python from cfengine import PromiseModule, ValidationError, Result class ExamplePromiseTypeModule(PromiseModule): def validate_promise(self, promiser, attributes, metadata): # Your code here, do error checking and raise ValidationError pass def evaluate_promise(self, promiser, attributes, metadata): # Your code here, make changes to the system and return Result pass if __name__ == "__main__": ExamplePromiseTypeModule().start() ``` Filling out the 2 functions, `validate_promise()` and `evaluate_promise()` is all you need to do. A basic implementation of a git module looks like this: https://github.com/cfengine/promise-type-template/blob/main/git_example.py ## Help Participants are encouraged to contact us via email: [student-2022@cfengine.com](mailto:student-2022@cfengine.com) We can help you with: - Help getting started / setting up a Virtual Machine - Any questions you have or problems you encounter - Ideas / suggestions for what to build - Feedback on your idea / what you want to build ## Inspiration Some examples of modules to look at for inspiration: - [Module for managing git repositories](https://build.cfengine.com/modules/promise-type-git/) - [Module for working with systemd services](https://build.cfengine.com/modules/promise-type-systemd/) - [Module for making HTTP requests](https://build.cfengine.com/modules/promise-type-http/) ![](./images/student-competition.png) -------------------------------------------------------------------------------- title: "Synchronize data between PostgreSQL and files" url: /blog/2022/postgresql-to-json-files-sync/index.md date: "2022-04-06" -------------------------------------------------------------------------------- Databases are great for data processing and storage. However, in many cases it is better or easier to work with data in files on a file system, some tools even cannot access the data in any other way. When a database (DB) is created in a database management system (DBMS) using a file system as its data storage, it of course uses files on the given file system to store the data. But working with those files outside of the DBMS, even for read-only access to the data stored in the DB, is practically impossible. So what can be done if some setup requires data in files while at the same time, the data processing and storage requires a use of a DB(MS)? The answer is synchronization between two storage places -- a DB and files. It can either be from the DB to the files where the files are then treated as read-only for the parties working with the data, or with modifications of the files being synchronized to the DB. In the former setup, the DB is the single source of truth -- the data in the files may be out of sync, but the DB has the up to date version. In the latter setup, **the DB** provides a backup or alternative **read-only** access to the data that is primarily stored in **the files** or **the files** provide an alternative **write-only** access to the DB. A two-way synchronization and thus a combination of read and write access in both places, the DB and the files, should be avoided because it\'s very hard (one could even say impossible) to properly implement mechanisms ensuring data consistency. Both between the two storages, but even in each of them alone. ## Synchronizing data to JSON files In CFEngine Enterprise, we needed to provide file-based access to host-specific data stored in our PostgreSQL database. Users define the host-specific data in our web user interface (Mission Portal), but the individual hosts expect their specific `host-specific.json` files on the CFEngine hub\'s file system. This could have been achieved by extending the web UI code to write the data to both places, but we enforce privilege separation and don\'t want the web UI code to have access to CFEngine policy, the part of which the host-specific data is. And because we already needed a daemon reacting to changes and events in the PostgreSQL database, we used techniques described in a [previous blog post](https://cfengine.com/blog/2022/postgresql-triggers-notifications/) to implement a synchronization of the host-specific data between the DB and the respective JSON files. The daemon is called *cf-reactor* and it is written in C. Let\'s take a look at how it works and what it does. ### Listening to notifications from PostgreSQL The *cf-reactor* daemon needs to know when some JSON file with host-specific data needs to be updated with the new data from the PostgreSQL database. The data in question is in the following `__cmdb` table: ```sql CREATE TABLE IF NOT EXISTS __cmdb ( hostkey text PRIMARY KEY, value jsonb NOT NULL, updated_at timestamptz NOT NULL DEFAULT now(), epoch bigint NOT NULL DEFAULT 0 ); ``` *cf-reactor* could check and compare the last modification timestamp of each of the JSON files with the last modification timestamp of the respective row (`updated_at`) in the table storing the host-specific data. Another thing that it would need to do is to check if there is some new data, for hosts that had no host-specific data before. And this is actually how things used to be implemented before *cf-reactor* was introduced -- CFEngine policy on the hub periodically compared the timestamps, updated the files that were out of date, and created new files for hosts which the user defined host-specific data for since the last check. It was a working solution, but since an agent run evaluating the policy only happens every 5 minutes (by default), it took up to 5 minutes for the changes made in the `__cmdb` table to propagate to the respective host-specific JSON files. Instead of using the same approach as described above, potentially with a shorter interval for the periodic checks, *cf-reactor* uses the technique described in a [previous blog post](https://cfengine.com/blog/2022/postgresql-triggers-notifications/) -- asynchronous notifications sent by a trigger. Whenever a new row is added to the `__cmdb` table (`INSERT`) or an existing row is updated (`UPDATE`), it receives a notification through the `cmdb_refresh` channel and knows it needs to synchronize the `host-specific.json` file for the particular host (identified by the `hostkey` column) with the data in the database. This is done by the following SQL commands in our schema: ```sql CREATE FUNCTION update_cmdb_data_file() RETURNS TRIGGER AS $$ BEGIN EXECUTE 'NOTIFY cmdb_refresh, ''' || NEW.hostkey || ''''; RETURN NULL; -- result is ignored since this is an AFTER trigger END; $$ LANGUAGE plpgsql; CREATE TRIGGER update_cmdb_data_file AFTER INSERT OR UPDATE ON __cmdb FOR EACH ROW EXECUTE FUNCTION update_cmdb_data_file(); ``` To be able to listen to the notifications sent by the trigger defined above and react on them, *cf-reactor* does this: (in pseudo-code) ```python conn = PQ_Connection(db="cfdb") while not terminate: wait_for_new_notifications() while PQ_have_notifications(): notif = PQ_next_notification() hostkey = notif.payload UpdateHostSpecificJSONFile(hostkey) ``` ### C code to work with notifications The `wait_for_new_notifications()` function call above may look like *cf-reactor* is constantly polling PostgreSQL for new notifications creating a heavy load on the system. However, the real implementation using C actually uses the `poll()` function (system call) which, contradictory to its name, provides an efficient way of waiting for resources being available. Simply put, the `poll()` function takes a set of file descriptors plus a timeout specification and only returns when one or more of the given file descriptors are ready for the specified operation (read, write), if an error occurs, or if the given timeout expires. While waiting, the process making the `poll()` system call is put out of the queue of processes waiting to be executed on the CPUs and so such processes consume little to no resources. The real code then looks similar to the following snippet (with error and result checking omitted): ```c #define TIMEOUT (60 * 1000) /* milliseconds */ bool terminate = false; PGconn *conn = PQconnectdb("dbname=cfdb"); int sock = PQsocket(conn); PGresult *res = PQexec(conn, "LISTEN cmdb_refresh"); PQclear(res); struct pollfd sock_poll = { .fd = sock, .events = POLLIN, .revents = 0 }; int ret; while (!terminate) { ret = poll(&sock_poll, 1, TIMEOUT); if (ret > 0) { if ((sock_poll.revents & (POLLERR|POLLHUP|POLLNVAL)) != 0) { /* some error on the socket, try to reconnect */ PQfinish(conn); conn = PQconnectdb("dbname=cfdb"); sock = PQsocket(conn); sock_poll.fd = sock; continue; } PQconsumeInput(conn); PGNotify *notify; while ((notify = PQnotifies(conn)) != NULL) { assert(StringEqual(notify->relname, "cmdb_refresh"); printf("CMDB refresh request received from PID %d: '%s'\n", notify->be_pid, notify->extra ? notify->extra : ""); PQfreemem(notify); PQconsumeInput(conn); } } else if (ret == 0) { /* handle timeout */ } else { /* handle error (can be EINTR because of signal received) */ } } PQfinish(conn); ``` Of course, instead of just using `printf()` to inform about the received CMDB refresh request, the real code extracts the hostkey from the notification\'s payload (`notify->extra`), executes an SQL query to get the JSON data (`value`) for the given host and stores the data in the respective file and directory. The data is already stored as JSON in the DB so it's ready to be written, but this code could easily be expanded to transform the data or even combine data from multiple tables. ## Timing of the events Our [previous blog post](https://cfengine.com/blog/2022/postgresql-triggers-notifications/) describes in which cases and when notifications are sent by triggers like the one defined above and emphasizes that the notifications are *asynchronous*. So synchronization of the data from the DB to files implemented using such notifications **is not** synchronous. What does this beautiful sentence mean in practice? First of all, there\'s no guarantee that when something reads the respective JSON file after a change of the data in the DB, it will get the same data. In other words, there are periods of time when the data in the DB differs from the data in the respective files. And when the data for a host is added to the table (or deleted from it), even the existence of the respective file is not guaranteed. On the other hand, as we can see in the following screencast, the timing is in reality very fast, just not guaranteed to be fast enough for real-time restrictions. For our use case in CFEngine Enterprise it is good enough because we only need the future agent runs on the respective hosts to have their host-specific data files up to date and available on the CFEngine hub. {{< wideimg "images/cmdb_sync_timing.gif" "GIF showing CMDB data synchronization timing">}} The above screencast shows a couple things. First, when *cf-reactor* starts, it checks whether the data in the JSON files is up to date because some modifications of the data in the DB could be done (and were done, in this case) while it was not running. It sees that the JSON file\'s last modification timestamp is older than the `updated_at` timestamp in the DB so it updates the file contents. Then, when another modification of the data in the DB is done by changing the value of a CMDB variable in Mission Portal (web UI), it gets a notification and updates the contents of the file again. As the timestamps demonstrate, the reaction is very fast. The *Updating CMDB data file...* log message has the same timestamp in seconds as the `updated_at` timestamp in the DB and as the last modification timestamp of the respective JSON file. So the update of the file happens in the same second in which the user submits the new value. And if we look closer at the `updated_at` and the file\'s last modification timestamps we can see that the update of the file\'s contents happens in less than 20 ms after the change in the DB. The update is asynchronous and window of time when the file contents are out of date, is less than 20 ms, which is much more than good enough for our use case. ## Timeouts The C code example above has the following block: ```c } else if (ret == 0) { /* handle timeout */ } ``` A timeout from the `poll()` function/system call should be handled. It actually doesn\'t have to be handled and the timeout interval given to the `poll()` function can be a negative number meaning infinite timeout. However, this timeout is actually a very useful mechanism for throttling -- controlling when and how often things happen. For example, if we knew that there would be many changes of the data in the DB in a short sequence, it would make more sense to only synchronize the data into the respective file after a quiet period -- a period of time with no changes of the data. Or if we knew that nothing will read the data file until a specific time, we could avoid updating the file on every change of the data in the DB and only do it right before the data file is read. If the reactions to the events (notifications) are resource-intensive, they could even be held off until the system load drops below a certain level. And so on, there are many options. And since the implementation of the code handling the notifications can use an arbitrary language and run in an arbitrary environment, there are basically no limits. However, all these approaches require that the code knows that there were no notifications for some time and that\'s what the timeout argument given to the `poll()` function ensures. The value of it doesn\'t have to be a static hard-coded one, it can be a dynamic value set by the code processing the notifications because that code knows when it will need to continue doing things due to past notifications. ## Conclusions This blog post describes how PostgreSQL\'s asynchronous notifications sent by triggers can be used to synchronize data from a database into a set of JSON files. In CFEngine Enterprise, this is a job of the recently introduced *cf-reactor* daemon which is also responsible for other tasks, reacting to other types of events and requests. Generally, asynchronous notifications can be used for replacing mechanisms using periodic checks and updates with more event-driven implementations saving resources while making changes propagate faster, more interactively and more efficiently. We definitely plan to extend the area of tasks and mechanisms in CFEngine Enterprise using this approach to make our product better, both regarding the improved more interactive user experience, and regarding lower system resource usage. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 11 - Infrastructure hardening with CFEngine & Lynis" url: /blog/2022/the-agent-is-in-episode-11-infrastructure-hardening-cfengine-lynis/index.md date: "2022-03-31" -------------------------------------------------------------------------------- Looking to be more efficient writing CFEngine policy? Michael Bolen (Founder, CISOfy and author of Lynis) gives us some history on Lynis (including how to pronounce it, spoiler it's "lee nus"). Nick Anderson (Doer of Things, Northern.tech) shows off reporting Lynis scan findings with CFEngine Enterprise and the lynis CFEngine build module. ## Video {#video} The video recording is available on YouTube: {{< youtube KTV9HfqdEAQ >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Lynis Enterprise](https://cisofy.com/solutions/#lynis-enterprise) - [Lynis CFEngine Build Module](https://build.cfengine.com/modules/lynis/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), [Michael Bolen](https://www.linkedin.com/in/mboelen/) or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Trigger arbitrary code from PostgreSQL" url: /blog/2022/postgresql-triggers-notifications/index.md date: "2022-03-31" -------------------------------------------------------------------------------- In this blog post we show how it is possible to run an arbitrary program, script, or execute arbitrary code in reaction to changes and generally events in a PostgreSQL database. ## Triggers Database management systems (DBMS) provide mechanisms for defining reactions to certain actions or, in other words, for defining that specific actions should *trigger* specific reactions. PostgreSQL, the DBMS used by CFEngine Enterprise, is no exception. These *triggers* can be used for ensuring consistency between tables when changes in one table should be reflected in another table, for recording information about actions, and many other things. PostgreSQL\'s [*Overview of Trigger Behavior*](https://www.postgresql.org/docs/current/trigger-definition.html) describes the basics of triggers with the following sentences: > A trigger is a specification that the database should automatically > execute a particular function whenever a certain type of operation is > performed. Triggers can be attached to tables (partitioned or not), > views, and foreign tables. > > On tables and foreign tables, triggers can be defined to execute > either before or after any INSERT, UPDATE, or DELETE operation, either > once per modified row, or once per SQL statement. UPDATE triggers can > moreover be set to fire only if certain columns are mentioned in the > SET clause of the UPDATE statement. Triggers can also fire for > TRUNCATE statements. If a trigger event occurs, the trigger\'s > function is called at the appropriate time to handle the event. > > On views, triggers can be defined to execute instead of INSERT, > UPDATE, or DELETE operations. Such INSTEAD OF triggers are fired once > for each row that needs to be modified in the view. ... Triggers are created by the `CREATE TRIGGER` SQL command and the [PostgreSQL documentation](https://www.postgresql.org/docs/current/sql-createtrigger.html) specifies the syntax as follows: ``` sql CREATE [ OR REPLACE ] [ CONSTRAINT ] TRIGGER name { BEFORE | AFTER | INSTEAD OF } { event [ OR ... ] } ON table_name [ FROM referenced_table_name ] [ NOT DEFERRABLE | [ DEFERRABLE ] [ INITIALLY IMMEDIATE | INITIALLY DEFERRED ] ] [ REFERENCING { { OLD | NEW } TABLE [ AS ] transition_relation_name } [ ... ] ] [ FOR [ EACH ] { ROW | STATEMENT } ] [ WHEN ( condition ) ] EXECUTE { FUNCTION | PROCEDURE } function_name ( arguments ) where event can be one of: INSERT UPDATE [ OF column_name [, ... ] ] DELETE TRUNCATE ``` As the two above snippets show, triggers can define reactions to a range of actions very specifically. However, the only type of reaction they can define is described on this line: ``` sql EXECUTE { FUNCTION | PROCEDURE } function_name ( arguments ) ``` with a bit more information following further on [the page](https://www.postgresql.org/docs/current/sql-createtrigger.html) providing details for the specific parts of the `CREATE TRIGGER` SQL command, the `function_name` part in this case: > A user-supplied function that is declared as taking no arguments and returning type trigger, which > is executed when the trigger fires. > > In the syntax of CREATE TRIGGER, the keywords FUNCTION and PROCEDURE are equivalent, > but the referenced function must in any case be a function, not a procedure. The use of the keyword > PROCEDURE here is historical and deprecated. Such function is defined by the [`CREATE FUNCTION`](https://www.postgresql.org/docs/current/sql-createfunction.html) SQL command and it can do many things and be written in various supported languages, but it is still quite limited. In CFEngine, we needed to react on some simple actions, but we needed a lot of logic in the reactions and we needed the reactions to use our code and very specific mechanisms. ## Asynchronous Notifications Fortunately, PostgreSQL provides [*Asynchronous Notifications*](https://www.postgresql.org/docs/current/libpq-notify.html). This mechanism can be used to notify listening clients about events, or, when looked at from a different angle, to broadcast a message to the listening clients. The usage is very simple. A *channel name* needs to be defined and then clients can subscribe to notifications from the particular channel by executing the `LISTEN channel_name` SQL command. The sender(s)/broadcaster(s) can then use the `NOTIFY channel_name [, payload]` command to send notifications to the channel, with an optional `payload` string providing more details about the event. There\'s no need (or even way, actually) to create or register the channel first. It\'s easy to test the mechanism using the *psql* tool both sending and listening to the notifications in a channel: {{< wideimg "images/pg_listen_notify.gif" "psql LISTEN NOTIFY">}} We can see that whenever the tool sends a notification, it also immediately receives it, including the payload (if any) and the identifier of the sending process. Two *psql* processes can of course be used to simulate a more realistic situation where the senders and listeners of the notifications are separate processes and to check that the messages really go through the PostgreSQL DB and not just through some loopback in the *psql* process itself. The only tricky part of that is that the listening *psql* process only checks for new notifications when it executes an SQL command (communicates with the DB), it doesn\'t have any background watch thread. ## Triggers and Asynchronous Notifications The above two sections of this blog post describe two mechanisms for reacting to events in PostgreSQL -- triggers and asynchronous notifications. Triggers can be used for automatic (implicit) reactions to various specific events happening on DB tables where the reactions are functions executed by the PostgreSQL DBMS in its limited environment. Asynchronous notifications can, on the other hand, be used for explicitly announcing some events or simply broadcasting messages and the clients listening to them are independent of the PostgreSQL DBMS, they only need a connection to it. So how can arbitrary programs or scripts be run or arbitrary code be executed when something happens in a PostgreSQL database? One option is, of course, to add explicit `NOTIFY` SQL commands to the code that causes the changes in the database. And often this is the best approach because such code can decide whether to send the notification and when exactly to send it. But what if it is not desired or even possible to modify the code causing the changes in the database? The answer is, as it often is with modern technologies, integration and combination. Integration and combination of the two mechanisms described in this post -- triggers and asynchronous notifications. Triggers cannot run arbitrary things, they can only execute functions in the DBMS. Clients listening to notifications can do whatever they want, but they need something or someone to send the notifications. Well, what if the functions that the triggers execute send the notifications? That way, every change in the database the trigger is hooked to will be announced with the respective notification. As an example, let\'s take a look at what we are doing in CFEngine Enterprise (more details about this in an [upcoming blog post](./postgresql-to-json-files-sync/)): ``` sql CREATE TABLE IF NOT EXISTS __cmdb ( hostkey text PRIMARY KEY, value jsonb NOT NULL, updated_at timestamptz NOT NULL DEFAULT now(), epoch bigint NOT NULL DEFAULT 0 ); CREATE FUNCTION update_cmdb_data_file() RETURNS TRIGGER AS $$ BEGIN EXECUTE 'NOTIFY cmdb_refresh, ''' || NEW.hostkey || ''''; RETURN NULL; -- result is ignored since this is an AFTER trigger END; $$ LANGUAGE plpgsql; CREATE TRIGGER update_cmdb_data_file AFTER INSERT OR UPDATE ON __cmdb FOR EACH ROW EXECUTE FUNCTION update_cmdb_data_file(); ``` The above SQL code from our schema basically says: whenever a row in the `__cmdb` table is modified or a new row is added to the table, send a notification to the `cmdb_refresh` channel with the `hostkey` (first column) as the notification payload. This is done by a definition of a trigger `AFTER INSERT OR UPDATE ON __cmdb` that runs a trigger function executing a `NOTIFY` SQL command. The clients listening on the channel then know that data for the particular host (identified by the `hostkey`) was modified or added and can do whatever they want in reaction. How to implement such clients, what they can do and what our real clients in CFEngine actually do will be the main topic of an [upcoming blog post](./postgresql-to-json-files-sync/). ## Notification-triggered code is not a trigger function Writing trigger functions is not particularly easy. They use special languages and they have a quite limited execution environment and possibilities. Of course there are many PostgreSQL extensions adding many extra features and possibilities, but writing a bit of code in one\'s language of choice in a separate program that can do anything and run under an arbitrary user is definitely easier. And it can even run on a separate machine because all it needs is a connection to the PostgreSQL DB and an ability to run the `LISTEN` SQL command. However, there is one crucial word in the description of what was introduced above -- using triggers for asynchronous notifications in PostgreSQL -- the word *asynchronous*. While trigger functions are run by the DBMS as part of the same transaction and (from [Overview of Trigger Behavior](https://www.postgresql.org/docs/current/trigger-definition.html) again): > In all cases, a trigger is executed as part of the same transaction as > the statement that triggered it, so if either the statement or the > trigger causes an error, the effects of both will be rolled back. for asynchronous notification this applies (from [`NOTIFY`](https://www.postgresql.org/docs/current/sql-notify.html) documentation): > NOTIFY interacts with SQL transactions in some important ways. > Firstly, if a NOTIFY is executed inside a transaction, the notify > events are not delivered until and unless the transaction is > committed. Although this may seem similar, there\'s a fundamental difference -- while the code in a trigger function is executed as part of the transaction that triggered it, a transaction is completely independent (and unaware) of the code handling notifications that were sent as part of it. In practice, if a trigger function is responsible for updating some related relation (table), there\'s a guarantee that such update will happen if and only if the change triggering the function happens and when the transaction containing that change ends, the trigger function code, and thus the update of the related relation, is done. If a trigger function sends a notification then the **notification** is sent if and only if the triggering changes happen and as part of the same transaction, but the clients handling the notification execute their reactions asynchronously and independently. There might be no such clients, one client or even multiple clients, remember the notifications are broadcasted to any number of clients that did `LISTEN` on the particular channel. In the example case shown above, the code handling notifications in the `cmdb_refresh` channel cannot even assume that data for the host given by the payload (`hostkey`) is still in the table. There could have been another change in the DB after the notification was sent and before it was delivered that could have deleted the data. On the other hand, if there was a trigger `ON DELETE` on the same table sending a notification to the same channel, the two notifications, one triggered by the `UPDATE` or `INSERT` and the other one triggered by the `DELETE` would be guaranteed to be delivered in the same order as in which the changes happened. ## Conclusions Asynchronous notifications are great for many things, whether sent automatically (implicitly) by trigger functions or explicitly by the code making changes in a database. They provide a powerful mechanism for reacting to changes happening in a DB while giving an option to implement those reactions in an arbitrary language and execute them in an arbitrary environment. However, the code handling them cannot make any assumptions about the state of the DB and thus this mechanism cannot be used for things like ensuring consistency between multiple relations or something similar. In an [upcoming blog post](./postgresql-to-json-files-sync/) we will see how they are being used in CFEngine Enterprise to make it more event-driven and less resource hungry. Both in combination with triggers and with code explicitly sending notifications in certain situations. -------------------------------------------------------------------------------- title: "Change in behavior: Directory permissions and the execute bit" url: /blog/2022/rxdirs-default-changing-from-true-to-false/index.md date: "2022-03-29" -------------------------------------------------------------------------------- `rxdirs` has provided a convenient default when setting permissions recursively. When enabled (the default prior to version 3.20.0) a promise to grant read access on a directory is extended to also include execution since quite commonly if you want to read a directory you also want to be able to list the files in the directory. However, the convenience comes with the cost of complicating security reviews since the state requested on the surface is more strict than what is actually granted. This can both undermine the understanding of the desired state of the permissions as well as confidence that the policy accurately describes the resulting state and we have decided the convenience is not worth the cost. [![](images/rxdirs.png)](images/rxdirs.png) In the upcoming release of CFEngine 3.20.0 the default setting for [rxdirs](https://docs.cfengine.com/docs/3.18/reference-promise-types-files.html#rxdirs) is changing from _true_ to _false_. In this post we discuss changes you may want to make to your policy as a result of the change. For example, here we promise that both a file and a directory should exist and be readable by the owner (_mode_ `0400`) while `rxdirs` is enabled: ```cfengine3 bundle agent __main__ { files: # A directory promises to be readable, with rxdirs enabled it's also executable "/tmp/a-directory/." create => "true", perms => m_rxdirs( "0400", "true" ); # A file promises to be readable, with rxdirs enabled it's just readable, not executable "/tmp/a-file.txt" create => "true", perms => m_rxdirs( "0400", "true" ); reports: # Show human readable and octal permissions of each file "stat -c '%A %a' /tmp/a-directory: $(with)" with => concat( filestat( "/tmp/a-directory", "permstr" ), " ", filestat( "/tmp/a-directory", "permoct" )); "stat -c '%A %a' /tmp/a-file.txt: $(with)" with => concat( filestat( "/tmp/a-file.txt", "permstr" ), " ", filestat( "/tmp/a-file.txt", "permoct" )); } body perms m_rxdirs( mode, rxdirs ) { mode => "$(mode)"; rxdirs => "$(rxdirs)"; } ``` Running the policy we can see that the directory but not the file gets the additional permission of execute: ```text # cf-agent --no-lock --log-level info --file /tmp/rxdirs-illustrated.cf info: Created directory '/tmp/a-directory/.' info: Object '/tmp/a-directory' had permissions 0755, changed it to 0500 info: Created file '/tmp/a-file.txt', mode 0400 R: stat -c '%A %a' /tmp/a-directory: dr-x------ 500 R: stat -c '%A %a' /tmp/a-file.txt: -r-------- 400 ``` With the change in default value directories will no longer get the additional permissions unless rxdirs is _explicitly_ enabled. Furthermore, beginning with binary version 3.18.2 or greater _Warnings_ will be emitted when a `perms` body is used setting `mode` _without_ explicitly having set `rxdirs`. For example, `warning: Using the default value 'true' for attribute rxdirs (promiser: /tmp/rxdirs/.), please set it explicitly` is already emitted by 3.18 nightly packages for the following policy snippet. ```cfengine3 bundle agent main # User Defined Service Catalogue { files: "/tmp/rxdirs/." create => "true", perms => mode( "0400" ); } body perms mode( perm_or_oct_string ) # @brief Set the file mode # @param perm_of_oct_string The new mode, e.g. 0400, o+rx-w, go-rwx { mode => "$(perm_or_oct_string)"; } ``` The MPF (Masterfiles Policy Framework) has been instrumented to maintain the old `rxdirs` default value expectation until you are running binaries version 3.20.0 or later meaning you can safely run the newer policy with older binaries and expect to have the same behavior until your binaries have been upgraded to 3.20.0 or newer. If you upgrade binaries to 3.18.2 or later _before_ upgrading the policy framework you can expect to see _many_ warnings as illustrated by the following output (which has been uniquified for brevity) from a host running update.cf with an out-dated policy framework and newer binaries ```text warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/masterfiles/., file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:283), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:777), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/apk, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/apt_get, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/freebsd_ports, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/msiexec.bat, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/msiexec-list.vbs, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/nimclient, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/pkg, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/pkgsrc, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/slackpkg, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/snap, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/WiRunSQL.vbs, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/yum, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly warning: Using the default value 'false' for attribute rxdirs (promiser: /var/cfengine/modules/packages/zypper, file: /var/cfengine/inputs/cfe_internal/update/update_policy.cf:763), please set it explicitly ``` What should you do? Upgrade your policy with builds from master or 3.18.x and address any Warnings. **Tip:** Getting nightly packages for 3.18.x is quick and easy with [cf-remote](/tags/cf-remote/), `cf-remote --version 3.18.x list` Review your policy for cases where perms bodies affect _mode_ to see if they are applied to directories and adjust as necessary. Typical treatment will be to use a perms body with `rxdirs` explicitly enabled or to make one promise that affects non directories and another promise that affects directories. To make a promise that applies recursively to files but not directories use a [file_select](https://docs.cfengine.com/docs/3.18/reference-promise-types-files.html#file_select) body. For example, here [file_types](https://docs.cfengine.com/docs/3.18/reference-promise-types-files.html#file_types) specifies `dir` and `file_result` specifies `!file_types`, thus selecting all files that are not of type dir. Note: [body file_select not_dir](https://docs.cfengine.com/docs/3.18/reference-masterfiles-policy-framework-lib-files.html#not_dir) was recently added to the standard library and will be present in 3.18.2 and later releases. ```cfengine3 body file_select not_dir # @brief Select all files that are not directories { file_types => { "dir" }; file_result => "!file_types"; } ``` For example, this policy illustrates how to make promises including the execute bit for just directories and promises without the executable bit for files. The commented out promise being the equivalent prior to the change in `rxdirs` default. ```cfengine3 bundle agent init { vars: "init_files" slist => { "/tmp/rxdirs/A/file.txt", "/tmp/rxdirs/A/directory/file.txt", "/tmp/rxdirs/A/directory/promised/file.txt", "/tmp/rxdirs/B/file.txt", "/tmp/rxdirs/B/directory/file.txt", "/tmp/rxdirs/B/directory/promised/file.txt" }; files: "$(init_files)" create => "true"; } bundle agent example_rxdirs_treatment { files: # A promise that would have resulted in owner getting execute on # directories prior to the rxdirs behavior change. # # "/tmp/rxdirs/." # depth_search => recurse_with_base( "inf" ), # perms => m( "0400" ); # Adjusted implementation to promise permissions and files separately "/tmp/rxdirs/A/." file_select => dirs, # Select directories only depth_search => recurse_with_base( "inf" ), perms => m( "0500" ); "/tmp/rxdirs/A/." file_select => not_dir, # Select anything that isn't a directory depth_search => recurse_with_base( "inf" ), perms => m( "0400" ); # Alternative implementation, using rxdirs "/tmp/rxdirs/B/." depth_search => recurse_with_base( "inf" ), perms => m_rxdirs( "0400" ); # Define a custom perms body with rxdirs enabled } bundle agent __main__ { methods: "Initialize file tree for illustrating rxdirs behavior" usebundle => init; "Illustrate implemntation options for dealing with change in rxdirs behavior" usebundle => example_rxdirs_treatment; } body perms m_rxdirs( mode ) # @brief Custom body with behavior of body perms m from standard library with rxdirs enabled { inherit_from => default:m( "$(mode)" ); rxdirs => "true"; } # Plucked from lib/files.cf # # # From here down are bodies defined in the standard library, duplicated here for # convenience of running a standalone example policy body perms m(mode) # @brief Set the file mode # @param mode The new mode { mode => "$(mode)"; #+begin_ENT-951 # Remove after 3.20 is not supported rxdirs => "true"; @if minimum_version(3.20) rxdirs => "false"; @endif #+end } body depth_search recurse_with_base(d) # @brief Search files and directories recursively up to the specified # depth, starting from the base directory excluding directories on # other devices. # # @param d The maximum search depth { depth => "$(d)"; xdev => "true"; include_basedir => "true"; } body file_select dirs # @brief Select directories { file_types => { "dir" }; file_result => "file_types"; } body file_select not_dir # @brief Select all files that are not directories { file_types => { "dir" }; file_result => "!file_types"; } ``` In the output we can see that the resulting permissions are the same between `/tmp/rxdirs/A` and `/tmp/rxdirs/B` with the permissions in `/tmp/rxdirs/A` set by independent promises for files and directories and the permission under `/tmp/rxdirs/B` set with `rxdirs` enabled. ```text # cf-agent --no-lock --log-level info --file /tmp/rxdirs-inherit-from-same-name.cf info: Created directory for '/tmp/rxdirs/A/file.txt' info: Created file '/tmp/rxdirs/A/file.txt', mode 0600 info: Created directory for '/tmp/rxdirs/A/directory/file.txt' info: Created file '/tmp/rxdirs/A/directory/file.txt', mode 0600 info: Created directory for '/tmp/rxdirs/A/directory/promised/file.txt' info: Created file '/tmp/rxdirs/A/directory/promised/file.txt', mode 0600 info: Created directory for '/tmp/rxdirs/B/file.txt' info: Created file '/tmp/rxdirs/B/file.txt', mode 0600 info: Created directory for '/tmp/rxdirs/B/directory/file.txt' info: Created file '/tmp/rxdirs/B/directory/file.txt', mode 0600 info: Created directory for '/tmp/rxdirs/B/directory/promised/file.txt' info: Created file '/tmp/rxdirs/B/directory/promised/file.txt', mode 0600 info: Object '/tmp/rxdirs/A/./directory/promised' had permissions 0755, changed it to 0500 info: Object '/tmp/rxdirs/A/./directory' had permissions 0755, changed it to 0500 info: Object '/tmp/rxdirs/A/.' had permissions 0755, changed it to 0500 info: Object '/tmp/rxdirs/A/./file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/A/./directory/file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/A/./directory/promised/file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/B/./file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/B/./directory/file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/B/./directory/promised/file.txt' had permissions 0600, changed it to 0400 info: Object '/tmp/rxdirs/B/./directory/promised' had permissions 0755, changed it to 0500 info: Object '/tmp/rxdirs/B/./directory' had permissions 0755, changed it to 0500 info: Object '/tmp/rxdirs/B/.' had permissions 0755, changed it to 0500 ``` -------------------------------------------------------------------------------- title: "Turn off your devices" url: /blog/2022/turn-off-computers/index.md date: "2022-03-17" -------------------------------------------------------------------------------- Saint Patrick's Day makes us think of the color green. Spring is coming. Plants are starting to sprout amongst the dead grass and leaves from Fall/Winter: ![](./images/clover-starts.jpg) [Earth Day](https://www.earthday.org/) is just around the corner on April 22nd. This reminds us of our commitment to the environment and ecosystems that surround us. As we at Northern.tech state in our [corporate social responsibilities](https://northern.tech/our-company/our-corporate-social-responsibilities): > We have set an ambitious company-objective to "Become a net-zero carbon business by the end of 2022". CFEngineers concern themselves with the proper configuration of computers, but we can also consider how computers affect us as humans and the world as an ecosystem. ## No pixels time One of our core [beliefs](https://northern.tech/blog/elements-of-a-meaningful-business-the-beliefs) is "You don't have to stare into pixels to work". > Pay attention to being effective (doing the correct things), not only efficient (doing things correctly). > There is probably no correlation between frequency of the number of key-presses or hours looking at a monitor and the amount of value produced. > By working proactively, stepping "out" and contemplating improvements, or solving a concrete challenge, we can avoid double-doing, or worse wrong-doing. > Humans should work smart, and should leave the speed thing to computers. > We encourage coffee breaks and frequent walks or activities outside of office to think things over. > Invest in that extra conversation with other stakeholders to ensure future alignment. And so this is the human impact of computers: they can distract us from what is really important sometimes. Computers are of course a wonderful tool: for computation and especially communication which is vital to us humans. ## Ecosystem impact The impact of computers on the ecosystem is significant, from their use of electricity to the mining of precious metals to create the devices in the first place. Where does your electricity come from? ![](./images/power-lines.jpg) Mine comes from a complicated and [ever changing mix](https://www.evergy.com/smart-energy/renewable-resources/our-energy-mix), both day-to-day based on trading and generation as well as over time. ![](./images/Energy-Mix-Graphic.gif) Companies like Fairphone state ["we care for people and planet"](https://www.fairphone.com/en/story/) > From the earth to your pocket, a smartphone's journey is filled with unfair practices. > We believe a fairer electronics industry is possible. > By making change from the inside, we're giving a voice to people who care. ## Power off Our mission is to "secure the worlds connected devices" and to that end I have been considering lately that one of the best ways of securing a device is to turn it off! I know this might sound like a joke, but of course it is really true. If a connected device is turned off it certainly can't be hacked via the network. Also, when a device is in transit it is at risk of being physically taken and hacked. There are [known risks](https://blog.f-secure.com/sleep-mode-vulnerable-mode/) in leaving a computer in sleep mode versus in hibernate or simply turned off. So turning off a computer both saves energy as well as improves its security in general and specifically in transit when it could be taken physically. Reducing wear and tear on devices is a way of extending the life of products which helps the environment. ## Steps to take There are some manual steps you can take: - Turn computers off when you won't use them for a while, such as at night or when on vacation. - Use sleep-to-disk/hibernate along with inactivity timeouts to automate putting computers in a low-power and secure state - Try some no-pixels time: turn your computer off, go for a walk, figure out that complex problem you are working on away from a computer As well as automated ones: - Use a tool like rtcwake on linux to automate when the computer will wake, make it on only when needed - Configure wake-on-lan to only have computers on when they are needed, such as for a personal media or git server Our prolific policy writer, [Nick Anderson](https://cfengine.com/author/nick-anderson/), has written some experimental support for `rtcwake` on linux: https://github.com/nickanderson/cfengine-rtcwake Maybe someone can take that as a start and make a pull request to convert it into a [CFEngine Build](https://build.cfengine.com) module. For another time. And in the meantime, turn those devices off when you can. Be well, and thanks for reading. -------------------------------------------------------------------------------- title: "Secure your hosts with CFEngine Build modules" url: /blog/2022/secure-your-hosts-with-cfengine-build-modules/index.md date: "2022-03-16" -------------------------------------------------------------------------------- [Last year](/blog/2021/2021-retrospective/), we launched functionality for users to add policy for reporting data, compliance reports, promise types, and other code as modules. With [CFEngine Build](https://build.cfengine.com/), users can manage and update their own policy, the default policy and any additional modules separately. This makes it very easy to utilize policy or other modules written by the CFEngine team, or other community members. In this post we will take a look at using some modules to improve the security of our infrastructure. ## Getting started If you are not familiar with CFEngine or how to use CFEngine Build, we highly recommend you go through the getting started guide: https://docs.cfengine.com/docs/3.21/getting-started.html We won't go over the basics of how CFEngine works and how to set it up here, so some prior knowledge is assumed. ## New project For this I will create a new project, but you can also keep using your existing `cfbs` project if you have one: ```command mkdir -p ~/security_policy ; cd ~/security_policy ; cfbs init ; cfbs add masterfiles ``` We will be using a similar workflow as in the getting started guide; 1. Explore CFEngine Build and add modules. 2. Build your policy set. 3. Deploy to your CFEngine hub. 4. Observe the results in the CFEngine Web UI, Mission Portal. [![](images/workflow.png)](images/workflow.png) ## Exploring security related modules On the CFEngine Build website, we can use the search functionality to find interesting modules: https://build.cfengine.com/modules/?query=&sort=most-download&page=1&perPage=100&tag=supported&tag=security In this case I searched for modules with `security` and `supported` tags and sorted by popularity (number of downloads). From that list of search results, I've chosen a few modules I'd like to use: - Modules which give more reporting data (inventory): - [`inventory-sudoers`](https://build.cfengine.com/modules/inventory-sudoers/) - Report who can use sudo on each host. - [`inventory-yum-update-info`](https://build.cfengine.com/modules/inventory-yum-update-info/) - Report security update information on RHEL / yum systems. - Modules related to detecting and fixing CVEs: - [`cve-2021-3156-sudo`](https://build.cfengine.com/modules/cve-2021-3156-sudo/) - Detect a vulnerability in sudo and fix it by updating. - [`cve-2021-44228-log4j`](https://build.cfengine.com/modules/cve-2021-44228-log4j/) - Detect potentially vulnerable instances of the log4j library. - Security hardening modules which files or packages on the machine: - [`prelinking-disabled`](https://build.cfengine.com/modules/prelinking-disabled/) - Disable the prelinking option which can change binaries. - [`ftp-server-not-installed`](https://build.cfengine.com/modules/ftp-server-not-installed/) - Uninstall the FTP server packages if they are installed. - [`tmp-nosuid`](https://build.cfengine.com/modules/tmp-nosuid/) - Ensure `/tmp` is mounted with the `nosuid` option. The command to add all of these is simply: ```command cfbs add inventory-sudoers inventory-yum-update-info cve-2021-3156-sudo cve-2021-44228-log4j prelinking-disabled ftp-server-not-installed tmp-nosuid ``` ## Build and deploy: Building is straight forward and fast: ```command cfbs build ``` If you're already using `cf-remote` and have spawned / saved hubs with it, you can deploy like this: ```command cf-remote deploy ``` If you have a CFEngine hub and need to save it in `cf-remote`, this is easy: ```command cf-remote save -H root@192.168.56.2 --role hub --name hub ``` (Replace the `root@192.168.56.2` with the appropriate SSH username and IP for your hub). ## Observing the results After deploying my policy set, I opened up the CFEngine Enterprise GUI, Mission Portal. It has multiple useful features for security and reporting, such as: Inventory reports, Policy analyzer, Compliance reports and individual host info pages with host specific data and actions. ### Inventory reports I created an Inventory Report with some of the relevant data: [![](images/inventory.png)](images/inventory.png) We now have a list of users with sudo, and can double check that it's correct, we want as few users as possible to have this privilege. We also see that many of our hosts are affected by the sudo CVE. That's good to know, let's look at updating sudo later. Additionally, we see that our RHEL 7 machine has security updates available. That is easy to fix, since it's only 1 machine, we can just log in with SSH and update; ``` $ ssh ec2-user@34.245.224.59 Last login: Wed Mar 16 19:57:57 2022 from 84.213.201.188 [ec2-user@ip-172-31-1-116 ~]$ sudo yum update -y ``` Alternately, I have the machine saved with the name `rhel7`, so it's easier to just use `cf-remote`: ```command cf-remote sudo -H rhel7 "yum update -y" ``` **Tip:** If you want all your hosts to be upgraded automatically, [there is a module for that](https://build.cfengine.com/modules/all-packages-upgraded/). ### Action buttons in host info page Now that we've updated with yum, let's refresh the data in Mission Portal. By clicking on the hostname in the inventory report, we get to the host info page, where we can trigger an agent run and report collection: [![](images/trigger-agent-run.png)](images/trigger-agent-run.png) Now, when we go back to the inventory report, we see that there are no security updates available: [![](images/security-updates-0.png)](images/security-updates-0.png) ### Policy analyzer Interestingly, we can see a red counter on the Policy analyzer app, indicating problems in our infrastructure: [![](images/pa-counter.png)](images/pa-counter.png) By opening up the app, we can find out which policy file is encountering issues: [![](images/pa-file.png)](images/pa-file.png) And even the individual line number: [![](images/pa-line.png)](images/pa-line.png) Below that, we can see the affected hosts and even the log messages indicating what's wrong: [![](images/pa-hosts.png)](images/pa-hosts.png) So, it seems our module detected a vulnerable version of `sudo` and wants to update, but is only configured to log warnings. We can see this documented in [the module's page on CFEngine Build](https://build.cfengine.com/modules/cve-2021-3156-sudo/): > This policy tests for the presence of the vulnerability and inventories it's presence. If the vulnerability is present sudo promises to run the most recent version available in warning mode unless `default:cve_2021_3156_remediate` or `northerntech_security_hardening:cve_2021_3156_remediate` is defined in which case sudo is automatically upgraded to the latest version available. So, we just need to define a class to tell the module to upgrade sudo, let's use _Host specific data_ for that. ### CMDB - Host specific data By clicking on one of the hosts affected by the sudo vulnerability, either in the policy analyzer results, or in the inventory report, we go to its host info page: [![](images/host-info.png)](images/host-info.png) Once here, we can use the _Host specific data_ section to enable or configure specific policy on that host. Let's enable the upgrade of `sudo`: [![](images/remediate-class.png)](images/remediate-class.png) After saving that class, triggering an agent run and waiting for the reporting to happen, we can see in the Inventory report that it's been patched on Ubuntu 20: [![](images/patched-ubuntu-20.png)](images/patched-ubuntu-20.png) At this point I could remove the class again, if I don't want the automatic upgrade to happen without me knowing of it. **Note:** On some older platforms, a new version of `sudo` might not be available, and the remediation will not succeed. In that case the module will still tell you that you are affected by the CVE, and it's up to you to find a way to upgrade `sudo` (or preferably the OS). ### Compliance reports While we are on the topic of upgrading Operating Systems, let's take a look at a newer experimental module: [compliance-report-os-is-vendor-supported](https://build.cfengine.com/modules/compliance-report-os-is-vendor-supported/) When added, this module will import a compliance report, to help you keep track of which Operating Systems need to be upgraded. Let's add and deploy it: ```command cfbs add compliance-report-os-is-vendor-supported cfbs build && cf-remote deploy ``` (When using `cf-remote` and `cfbs`, these are the 3 commands you need to add a module from our website, build it, and deploy it to your hub, and subsequently the rest of your infrastructure). We can open up Mission Portal again and go to _Reports -> Compliance_, to find our new report: [![](images/os-is-vendor-supported.png)](images/os-is-vendor-supported.png) In general, that looks good, but let's see which Ubuntu version is not supported by clicking on the link for _failing_: [![](images/os-is-vendor-supported-failing.png)](images/os-is-vendor-supported-failing.png) In this view, we can clearly see that Ubuntu 16 is not supported. Which is correct, regular support for Ubuntu 16 ended in April 2021, so we should upgrade! **Note:** This is an experimental module, as we just introduced the ability to import compliance reports. Using experimental modules is great for testing new things and giving feedback, just be aware that they might not be working in the best way or they might change a lot in the future. ## Exploring for more modules Feel free to look around CFEngine Build to find more modules you might be interested in: https://build.cfengine.com/ If you didn't see it already, the security calendar we did in December has more examples of security modules for CFEngine: https://cfengine.com/blog/2021/security-hardening-holiday-calendar-week-4/ -------------------------------------------------------------------------------- title: "CVE-2021-44215 & CVE-2021-44216 - Log file permissions" url: /blog/2022/cve-2021-44215-and-cve-2021-44216/index.md date: "2022-03-03" -------------------------------------------------------------------------------- The CFEngine engineering team has recently discovered two security issues in the CFEngine Enterprise product, specifically in the hub package: * CVE-2021-44215 - PostgreSQL log file world readable. * CVE-2021-44216 - Apache and Mission Portal Application log files world readable. `CVE-2021-44215` is a regression affecting currently supported versions 3.18.0 and 3.15.4 as well as some unsupported versions. `CVE-2021-44216` affects all supported versions prior to 3.18.1 and 3.15.5 as well as some unsupported versions. ## CVE-2021-44215 - PostgreSQL log file world readable ### Description The CFEngine Enterprise Hub package uses PostgreSQL for storing data about the history and current state of reporting agents. The log file `/var/log/postgresql.log` was found to be world readable. ### Detection Currently supported versions of the CFEngine Enterprise Hub package affected include 3.18.0 and 3.15.4 (3.15.x prior to 3.15.4 are not affected). Some older currently unsupported versions of the CFEngine Enterprise Hub package are affected as well. Use [this script](https://cfengine.com/2022/check-CVE-2021-44215-and-CVE-2021-44216.sh) to check if you are affected. Note, by default this script checks for both `CVE-2021-44215` and `CVE-2021-44216`. This example checks only for `CVE-2021-44215`. ```text [root@hub ~]# curl -sO https://cfengine.com/2022/check-CVE-2021-44215-and-CVE-2021-44216.sh [root@hub ~]# bash check-CVE-2021-44215-and-CVE-2021-44216.sh --no-check-cve-2021-44216 Running CFEngine Enterprise 3.15.4 Checking for the presence of CVE-2021-44215 WARNING: CVE-2021-44215 found ``` ### Impact An attacker with local access to the CFEngine Enterprise Hub can read database logs which may contain sensitive data depending on the configuration. ### Mitigation To mitigate the issue, upgrade to the newest supported releases of CFEngine Enterprise ([3.18.1, 3.15.5](https://cfengine.com/blog/2021/cfengine-3-15-5-and-3-18-1-released/)) or use [this policy](https://cfengine.com/2022/hotfix-cve-2021-44215-and-cve-2021-44216.cf) to manage the permissions actively until an upgrade can be completed. Please contact [support](https://support.northern.tech) if you need assistance. This issue has been registered as _CVE-2021-44215_ in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44215). ## CVE-2021-44216 - Apache and Mission Portal Application log files world readable ### Description The CFEngine Enterprise Hub package uses Apache to serve the Mission Portal web application. Logs from both Apache and Mission Portal were found to be world readable. ### Detection Currently supported versions of the CFEngine Enterprise Hub package affected include 3.18.0, 3.15.4, 3.15.3, 3.15.2, 3.15.1 and 3.15.0 (versions prior to 3.18.1 and 3.15.5). Older, currently unsupported versions of the CFEngine Enterprise Hub package are affected as well. Use [this script](https://cfengine.com/2022/check-CVE-2021-44215-and-CVE-2021-44216.sh) to check if you are affected. Note, by default this script checks for both `CVE-2021-44215` and `CVE-2021-44216`. This example checks only for `CVE-2021-44216`. ```text [root@hub ~]# curl -sO https://cfengine.com/2022/check-CVE-2021-44215-and-CVE-2021-44216.sh [root@hub ~]# bash check-CVE-2021-44215-and-CVE-2021-44216.sh --no-check-cve-2021-44215 Running CFEngine Enterprise 3.15.4 Checking for the presence of CVE-2021-44216 WARNING: CVE-2021-44216 found /var/cfengine/httpd/logs/application/log-2022-01-26.log affected WARNING: CVE-2021-44216 found /var/cfengine/httpd/logs/error_log affected WARNING: CVE-2021-44216 found /var/cfengine/httpd/logs/access_log affected WARNING: CVE-2021-44216 found /var/cfengine/httpd/logs/ssl_request_log affected ``` ### Impact An attacker with local access to the CFEngine Enterprise Hub can read web-server and web application log files which may contain sensitive data depending on the configuration. ### Mitigation To mitigate the issue, upgrade to the newest supported releases of CFEngine Enterprise ([3.18.1, 3.15.5](https://cfengine.com/blog/2021/cfengine-3-15-5-and-3-18-1-released/)) or use [this policy](https://cfengine.com/2022/hotfix-cve-2021-44215-and-cve-2021-44216.cf) to manage the permissions actively until an upgrade can be completed. This issue has been registered as _CVE-2021-44216_ in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44216). Please contact [support](https://support.northern.tech) if you need assistance. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 10 - Event-driven CFEngine" url: /blog/2022/the-agent-is-in-episode-10-event-driven-cfengine/index.md date: "2022-02-24" -------------------------------------------------------------------------------- Interested in the efforts underway to make CFEngine manage the environment even faster? Vratislav (Software Engineer) joins the show to talk about cf-reactor ## Video {#video} The video recording is available on YouTube: {{< youtube -KS6GRFKY_0 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [cf-reactor](https://docs.cfengine.com/docs/master/reference-components-cf-reactor.html) - [cf-runagent sockets](https://docs.cfengine.com/docs/master/reference-components-cf-runagent.html#sockets) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), [Nick](https://www.linkedin.com/in/hithisisnick/), or [Vratislav](https://www.linkedin.com/in/vratislav-podzimek-03489b43) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Writing a cfbs module for your custom policy update" url: /blog/2022/example-custom-policy-update-cfbs-module/index.md date: "2022-02-14" -------------------------------------------------------------------------------- I re-stumbled across [this mailing list post](https://groups.google.com/g/help-cfengine/c/ZtsTN5rcry0/m/c5WQmMVFCQAJ) from Bryan Burke about some policy framework upgrade issues where he also asked about hooking in and customizing the update policy. I thought this sounded like a good opportunity for an example using a `cfbs` module. So, let's take a look at making a `cfbs` module for a custom update policy. As mentioned in the thread there are just a couple of things you need to do in order to hook in and customize the behavior of the update policy. To define your own update mechanism: 1. Add your custom update policy to update inputs 2. Set the name of the bundle to use for policy update In creating the `cfbs` module I first created a [repository to host it](https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update) and [added a README and LICENSE](https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update/commit/d3ae965fb928c0d7443d8cb7b602ad310e103518). Then following my own `cfbs` module layout style I made [policy/custom-policy-update.cf](https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update/blob/d6470d23835c0dd2d39c3c1c600c89f2b73fa4eb/policy/custom-policy-update.cf), [cfbs/def.json](https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update/blob/d6470d23835c0dd2d39c3c1c600c89f2b73fa4eb/cfbs/def.json) and [cfbs.json](https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update/blob/d6470d23835c0dd2d39c3c1c600c89f2b73fa4eb/cfbs.json). For this simple example we won't expose any tunables for the module, maybe we will do that in a future post. `policy/custom-policy-update.cf` introduces the `my_organization` namespace so as to not conflict with other pre-existing bundle names. ```cf3 body file control { namespace => "my_organization"; } ``` `bundle agent custom_policy_update`, a bundle to run some things before and after the default policy update. Note: the use of `action => defaut:u_immediate`. This body is part of the update policy, it's a copy of body action immediate from the standard library which makes sure that a promise is not skipped during this execution of cf-agent because of a held lock (remember, CFEngine locks each promise for 1 minute by default). The update policy tries to be completely stand alone not loading any of the same policy files as another policy entry to try and decrease chances of breaking policy update from some other change. ```cf3 bundle agent custom_policy_update { vars: # The bundle that the MPF runs by default: # https://github.com/cfengine/masterfiles/search?q=%22bundle+agent+cfe_internal_update_policy_cpv%22 "mpf_default_policy_update_bundle" string => "default:cfe_internal_update_policy_cpv"; methods: # We want to do some things before the stock policy update runs "pre_default_policy_update" action => default:u_immediate; # We want to use the stock policy update (or, maybe we don't ...) "$(mpf_default_policy_update_bundle)" action => default:u_immediate; # We want to do some things after the stock policy update runs "post_default_policy_update" action => default:u_immediate; } ``` `bundle agent pre_default_policy_update`, a bundle to run before the default policy update bundle which simply emits a report that it is running for illustrative purposes. ```cf3 bundle agent pre_default_policy_update { reports: default:verbose_mode|my_orginization:DEBUG_custom_policy_update:: "Running $(this.namespace):$(this.bundle) ..." action => default:u_immediate; } ``` And finally, `bundle agent post_default_policy_update` a bundle to run after the default policy update, perhaps fetching additional information to prepare for the full policy run. Of course we don't have to use the default bundle at all, we could implement our own bundle to update inputs instead. `cfbs/def.json` contains the data that we want to merge into `def.json` in the root of the policy set during `cfbs build` to include and run the policy. ```json { "vars": { "update_inputs": ["services/custom-update-policy/custom-update-policy.cf"], "mpf_update_policy_bundle": "my_organization:custom_policy_update" } } ``` `cfbs.json` contains the metadata necessary to allow the module to be added directly from the repository (since this isn't in the index at [build.cfengine.com](https://build.cfengine.com/modules/promise-type-git/)). ```json { "name": "custom-policy-update", "description": "Customize the CFEngine Masterfiles Policy Framework Update Policy.", "type": "module-repo", "provides": { "custom-policy-update": { "description": "Customize the CFEngine Masterfiles Policy Framework Update Policy.", "tags": ["example"], "repo": "https://github.com/nickanderson/https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update", "by": "https://github.com/nickanderson/", "steps": [ "copy policy/custom-policy-update.cf services/custom-update-policy/custom-update-policy.cf", "json cfbs/def.json def.json" ] } } } ``` Now, with everything in place we bring up a test host and try it out. First, we initialize a `cfbs` project. ```text [root@hub ~]# mkdir -p /tmp/cfbs; cd /tmp/cfbs [root@hub cfbs]# cfbs init Initialized - edit name and description cfbs.json To add your first module, type: cfbs add masterfiles ``` Next we add masterfiles from the default index and `custom-policy-update` from its repository directly. ```text [root@hub cfbs]# cfbs add masterfiles Added module: masterfiles [root@hub cfbs]# cfbs add https://github.com/nickanderson/example-cfengine-mpf-custom-policy-update custom-policy-update Added module: custom-policy-update ``` Finally we build and install ... ```text [root@hub cfbs]# cfbs build Modules: 001 masterfiles @ f3a8f65e77428a6ab9d62c34057a7ace6ae54ce9 (Downloaded) 002 custom-policy-update @ d6470d23835c0dd2d39c3c1c600c89f2b73fa4eb (Downloaded) Steps: 001 masterfiles : run './prepare.sh -y' 001 masterfiles : copy './' 'masterfiles/' 002 custom-policy-update : copy 'policy/custom-policy-update.cf' 'masterfiles/services/custom-update-policy/custom-update-policy.cf' 002 custom-policy-update : json 'cfbs/def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz [root@hub cfbs]# cfbs install Installed to /var/cfengine/masterfiles ``` Now, we can see that it works running the update policy. With `--define my_orginization:DEBUG_custom_policy_update` the pre and post bundles emit reports about the fact that they are running. ```text [root@hub cfbs]# cf-agent -KIf /var/cfengine/masterfiles/update.cf --define my_orginization:DEBUG_custom_policy_update R: Running my_organization:pre_default_policy_update ... info: Copied file '/var/cfengine//masterfiles/services/custom-update-policy/custom-update-policy.cf' to '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf.cfnew' (mode '600') info: Backed up '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf' as '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf.cfsaved' info: Moved '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf.cfnew' to '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf' info: Updated '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf' from source '/var/cfengine//masterfiles/services/custom-update-policy/custom-update-policy.cf' on 'localhost' info: Purged '/var/cfengine/inputs/services/custom-update-policy/custom-update-policy.cf.cfsaved' copy dest directory info: Object '/var/cfengine//masterfiles/./services/custom-update-policy/custom-update-policy.cf' had permissions 0644, changed it to 0600 info: Object '/var/cfengine//masterfiles/./services/custom-update-policy' had permissions 0755, changed it to 0700 info: Object '/var/cfengine//masterfiles/./def.json' had permissions 0644, changed it to 0600 info: Object '/var/cfengine//masterfiles/.' had permissions 0755, changed it to 0700 R: Running my_organization:post_default_policy_update ... R: Found user specified update bundle. R: User specified update bundle: my_organization:custom_policy_update [root@hub cfbs]# ``` That's all folks! -------------------------------------------------------------------------------- title: "Introducing bodies with custom promise types" url: /blog/2022/custom-bodies/index.md date: "2022-02-08" -------------------------------------------------------------------------------- Last year we had a look at [managing local groups](https://cfengine.com/blog/2021/managing-local-groups/) with the custom [`groups` promise type](https://build.cfengine.com/modules/promise-type-groups/). As you may or may not recall, we used JSON-strings to imitate CFEngine bodies. This was due to the fact that the promise module protocol did not support bodies at that time. Today, on the other hand, we're happy to announce that as of CFEngine 3.20, this will no longer be the case. In this blog post we'll introduce the long awaited feature; custom bodies. We'll have a look at it from both the policy writers- and the promise module developers point of view. ## Dear policy writers Let's rewind back to the olden days, when we rode horse 'n carriage, and imitated custom bodies using JSON-strings. Back then we wrote our groups promises like this: ```cf3 bundle agent __main__ { groups: "foo" policy => "present", members => '{ "include": ["alice", "bob"], "exclude": ["malcom"] }'; } ``` It may have been somewhat ugly; but back then we managed to get by with what we had. More importantly; it worked! We've come a long way since then. Nowadays, the new kids on the block write policy utilizing custom bodies: ```cf3 body members foo { include => { "alice", "bob" }; exclude => { "malcom" }; } bundle agent __main__ { groups: "foo" policy => "present", members => foo; } ``` It's beautiful indeed! Not only do custom bodies provide a cleaner way of writing custom policy, they also unlock access to powerful features like body inheritance, just like you would find in built-in policy. ```cf3 body members parent { include => { "alice", "bob" }; } body members child(user) { exclude => { "$(user)" }; inherit_from => parent; } bundle agent __main__ { groups: "foo" policy => "present", members => child("malcom"); } ``` Okay. So, maybe you're one of those old folks who hates change, or maybe you run multiple versions of CFEngine. No matter who you are, there is no need to fear. The groups promise type is currently compatible with both the new and the old way of doing things. If you were to run multiple versions of CFEngine; you could write your policy as follows: ```cf3 @if minimum_version(3.20) body members foo { include => { "alice", "bob" }; exclude => { "malcom" }; } @endif bundle agent __main__ { groups: "foo" policy => "present", @if minimum_version(3.20) members => foo; @else members => '{ "include": ["alice", "bob"], "exclude": ["malcom"] }'; @endif } ``` Or if you're one of them old folks; you could just stick to the old way of doing things. ## Promise module developers For promise module developers, the only difference in the new and the old way of doing things, is that before, the "body" was sent as a JSON string to the promise module. And you had to manually parse it. ```json "attributes": { "policy": "present", "members": "{ \"include\": [\"alice\", \"bob\"], \"exclude\": [\"malcom\"] }" } ``` The difference in how custom bodies are sent to the promise module, is that they are sent as an actual JSON object. ```json "attributes": { "policy": "present", "members": { "include": ["alice", "bob"], "exclude": ["malcom"] } } ``` Therefore there is essentially no difference in terms of what is sent by the protocol when using body or data container. In fact the attributes part of the JSON would be equivalent for both the `foo` and the `bar` promise. ```cf3 body members foo_members { include => { "alice", "bob" }; exclude => { "malcom" }; } bundle agent __main__ { vars: "bar_members" data => '{ "include": ["alice", "bob"], "exclude": ["malcom"] }'; groups: "foo" members => bar_members; "bar" members => "@bar_members"; } ``` For more information on how the JSON protocol works, please checkout the master documentation [custom promise types](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html). ## Share your thoughts By now you know all there is to know about custom bodies. If you have any questions or thoughts on this matter; please let us know over at [GitHub Discussions](https://github.com/cfengine/core/discussions). We are eager to hear your feedback. -------------------------------------------------------------------------------- title: "CFEngine bootstrap with Ansible" url: /blog/2022/cfengine-ansible-module/index.md date: "2022-02-03" -------------------------------------------------------------------------------- CFEngine and Ansible are two [complementary infrastructure management tools](https://cfengine.com/white-papers/ansible-and-cfengine/). Findings from our analysis show that they can be combined and used side by side with joint forces to handle all areas in the best possible way. Part of infrastructure management is hosts deployment, either when building a brand new infrastructure or when growing one by adding new hosts. This is something Ansible truly excels in as it makes it very easy to run a sequence of steps on all hosts to initialize (deploy) them and it only requires SSH access to the hosts and Python installed on them. [^1] Many other aspects of infrastructure management that follow the initial deployment are much better handled by CFEngine which, however, requires all hosts to have a CFEngine package installed and to be bootstrapped to a CFEngine hub (policy server). So a logical thing to do is to install and bootstrap CFEngine as part of the initialization (deployment) of the hosts. ## Installing and bootstrapping CFEngine Installing and bootstrapping CFEngine as part of the host deployment Ansible playbook is of course possible with standard Ansible tasks. Nevertheless, it\'s not as easy as it may seem at the first glance. CFEngine supports many platforms (AIX, HP-UX, Solaris, Windows and a number of GNU/Linux distributions), but it is not platform-independent. Each platform needs a different package (with different binaries), even though in case of GNU/Linux distributions it is a matter of optimal performance (and various CPU architectures). Also, there are separate packages for CFEngine hubs (policy servers) and CFEngine *clients* (servers managed by CFEngine, sometimes also called *agents*). So installing and bootstrapping CFEngine requires quite a bit of logic which is, of course, possible to express in an Ansible playbook using facts, conditions, variables, etc. However, it\'s generally not an easy thing to do and requiring every user to solve it for themselves is wrong. ## CFEngine Ansible module Ansible playbooks (and role definitions) are sequences of tasks. Each task is of some type handled either by the Ansible core or by one of the Ansible modules. To make things as easy as possible for users, we wanted to implement a *cfengine* task type and so we created the [CFEngine Ansible module](https://gitlab.com/Northern.tech/CFEngine/cfengine-ansible-collection/-/blob/master/plugins/modules/cfengine.py), which is part of the [CFEngine Ansible collection](https://gitlab.com/Northern.tech/CFEngine/cfengine-ansible-collection). With the collection installed on the system on which *ansible-playbook* (or *ansible*) is run, installing and bootstrapping hosts on the deployed machines is as easy as adding the following task to the deploy playbook: ```yaml - name: Install and bootstrap CFEngine cfengine.cfengine.cfengine: policy_server: hub.example.com version: 3.18.1 ``` Wondering why the triple `cfengine`? Because, from right to left, it\'s the `cfengine` module (task type), in the `cfengine` collection in the `cfengine` namespace. The rest is straightforward -- `policy_server` determines the CFEngine policy server (hub) the hosts should be bootstrapped to and `version` (optional) determines the version of CFEngine to be installed on the hosts. If the `policy_server` is the host name or IP address of the host the task is run on, it becomes a CFEngine hub (policy server), otherwise it becomes a CFEngine client. ## Demo Let\'s see the CFEngine Ansible module in action! First, let\'s create some demo infrastructure using the [*cf-remote*](https://pypi.org/project/cf-remote/) tool: ``` example [vpodzime@vpodzime-laptop ~]$ cf-remote spawn --platform centos-7-x64 --count 1 --role hub --name c7-hub Waiting for VMs to get IP addresses..............DONE Details about the spawned VMs can be found in /home/vpodzime/.cfengine/cf-remote/cloud_state.json [vpodzime@vpodzime-laptop ~]$ cf-remote spawn --platform rhel-8-x64 --count 2 --role clients --name r8-clients Spawning VMs.....DONE Waiting for VMs to get IP addresses.............DONE Details about the spawned VMs can be found in /home/vpodzime/.cfengine/cf-remote/cloud_state.json [vpodzime@vpodzime-laptop ~]$ cf-remote spawn --platform ubuntu-18-04-x64 --count 2 --role clients --name u18-clients Spawning VMs.....DONE Waiting for VMs to get IP addresses.........................DONE Details about the spawned VMs can be found in /home/vpodzime/.cfengine/cf-remote/cloud_state.json [vpodzime@vpodzime-laptop ~]$ cf-remote spawn --platform debian-10-x64 --count 2 --role clients --name d10-clients Spawning VMs.....DONE Waiting for VMs to get IP addresses...........DONE Details about the spawned VMs can be found in /home/vpodzime/.cfengine/cf-remote/cloud_state.json ``` Now we can use the new feature of *cf-remote* -- the `--ansible-inventory` option that can be used to generate Ansible inventory data for the spawned hosts: ``` example [vpodzime@vpodzime-laptop ~]$ cf-remote show --ansible-inventory [c7-hub] vpc7-hub-centos-7-x64hub0 ansible_host=54.229.177.67 ansible_user=centos ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [r8-clients] vpr8-clients-rhel-8-x64client0 ansible_host=34.244.251.174 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpr8-clients-rhel-8-x64client1 ansible_host=54.246.161.133 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [u18-clients] vpu18-clients-ubuntu-18-04-x64client0 ansible_host=54.229.140.112 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpu18-clients-ubuntu-18-04-x64client1 ansible_host=34.247.55.222 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [d10-clients] vpd10-clients-debian-10-x64client0 ansible_host=34.243.247.62 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpd10-clients-debian-10-x64client1 ansible_host=52.50.38.227 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [all] vpc7-hub-centos-7-x64hub0 ansible_host=54.229.177.67 ansible_user=centos ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpr8-clients-rhel-8-x64client0 ansible_host=34.244.251.174 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpr8-clients-rhel-8-x64client1 ansible_host=54.246.161.133 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpu18-clients-ubuntu-18-04-x64client0 ansible_host=54.229.140.112 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpu18-clients-ubuntu-18-04-x64client1 ansible_host=34.247.55.222 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpd10-clients-debian-10-x64client0 ansible_host=34.243.247.62 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpd10-clients-debian-10-x64client1 ansible_host=52.50.38.227 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [hub] vpc7-hub-centos-7-x64hub0 ansible_host=54.229.177.67 ansible_user=centos ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" [clients] vpr8-clients-rhel-8-x64client0 ansible_host=34.244.251.174 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpr8-clients-rhel-8-x64client1 ansible_host=54.246.161.133 ansible_user=ec2-user ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpu18-clients-ubuntu-18-04-x64client0 ansible_host=54.229.140.112 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpu18-clients-ubuntu-18-04-x64client1 ansible_host=34.247.55.222 ansible_user=ubuntu ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpd10-clients-debian-10-x64client0 ansible_host=34.243.247.62 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" vpd10-clients-debian-10-x64client1 ansible_host=52.50.38.227 ansible_user=admin ansible_ssh_extra_args="-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" ``` As we can see, there are groups of hosts based on the names given to *cf-remote spawn* followed by the special groups `all`, `hub` and `clients` with the respective hosts in them. We can store the output in a file so that we can feed *ansible-playbook* with it: ``` example [vpodzime@vpodzime-laptop ~]$ cf-remote show --ansible-inventory > /tmp/inventory ``` With the Ansible inventory ([not to be confused with CFEngine inventory](https://cfengine.com/blog/2021/cfengine-inventory-to-ansible/)) ready, we can now run an Ansible playbook with a `cfengine.cfengine.cfengine` task to see what it does. Here is an example of such playbook: ```yaml --- - name: Deploy new hosts in the infrastructure hosts: all become: true tasks: - name: Install and bootstrap CFEngine cfengine.cfengine.cfengine: policy_server: 172.31.23.2 version: 3.18.1 ``` where the IP address of the policy server is the private IP address of the hub host (because it is spawned in AWS). First, we want to run the playbook on the policy server itself because that\'s required for other hosts to be able to bootstrap to it: ``` example [vpodzime@vpodzime-laptop ~]$ ansible-playbook --inventory /tmp/inventory --limit hub ~/tmp/deploy.yml [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details ERROR! couldn't resolve module/action 'cfengine.cfengine.cfengine'. This often indicates a misspelling, missing collection, or incorrect module path. The error appears to be in '/home/vpodzime/tmp/deploy.yml': line 6, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: tasks: - name: Install and bootstrap CFEngine ^ here ``` Wait, what happened?! Well, as the error suggests, it\'s likely one of the mentioned issues. And since we didn\'t install the CFEngine Ansible collection, the second common cause, *missing collection* is the one. So let\'s install the collection: ``` example [vpodzime@vpodzime-laptop ~]$ ansible-galaxy collection install https://gitlab.com/Northern.tech/CFEngine/cfengine-ansible-collection/uploads/71362febba022da55db9db748d7d399f/cfengine-cfengine-1.0.0.tar.gz Process install dependency map Starting collection install process Installing 'cfengine.cfengine:1.0.0' to '/home/vpodzime/.ansible/collections/ansible_collections/cfengine/cfengine' ``` As you can see, we are installing the collection directly from the GitLab project where it lives. That is because the collection is not yet available in [Ansible Galaxy](https://galaxy.ansible.com/).[^2] OK, let\'s give it another try with the collection installed! And this time, let\'s also use the `--verbose` option so that we get some extra bits of information. ``` example [vpodzime@vpodzime-laptop ~]$ ansible-playbook --verbose --inventory /tmp/inventory --limit hub ~/tmp/deploy.yml Using /etc/ansible/ansible.cfg as config file [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [Deploy new hosts in the infrastructure] ***************************************************************************************************************************************************** TASK [Gathering Facts] **************************************************************************************************************************************************************************** ok: [vpc7-hub-centos-7-x64hub0] TASK [Install and bootstrap CFEngine] ************************************************************************************************************************************************************* changed: [vpc7-hub-centos-7-x64hub0] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/hub/redhat_7_x86_64/cfengine-nova-hub-3.18.1-1.el7.x86_64.rpm", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "hub", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} PLAY RECAP **************************************************************************************************************************************************************************************** vpc7-hub-centos-7-x64hub0 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` As we can see, the `Install and bootstrap CFEngine` task has the status `changed` and thanks to the `--verbose` option we can also see extra details -- the package it installed was the `cfengine-nova-hub` package for RHEL/CentOS 7 in version 3.18.1, the host is a CFEngine hub and its policy server is `172.31.23.2`, thus the host itself. So exactly what we wanted and expected. Ansible tasks are idempotent -- if the state they describe is already the state of the host, they are supposed to do nothing. So it should be safe to run the playbook on the hub host again, right? Well, let\'s try it: ``` example [vpodzime@vpodzime-laptop ~]$ ansible-playbook --verbose --inventory /tmp/inventory --limit hub ~/tmp/deploy.yml Using /etc/ansible/ansible.cfg as config file [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [Deploy new hosts in the infrastructure] ***************************************************************************************************************************************************** TASK [Gathering Facts] **************************************************************************************************************************************************************************** ok: [vpc7-hub-centos-7-x64hub0] TASK [Install and bootstrap CFEngine] ************************************************************************************************************************************************************* ok: [vpc7-hub-centos-7-x64hub0] => {"cfengine_policy_server": "172.31.23.2", "cfengine_role": "hub", "changed": false, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} PLAY RECAP **************************************************************************************************************************************************************************************** vpc7-hub-centos-7-x64hub0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` This time the state of the task is `ok` so nothing was changed. And again, thanks to the `--verbose` option, we can also see the extra details and we can check they are the same as when the task actually installed CFEngine and bootstrapped the host. Now we can deploy the remaining hosts, which are in the `clients` group: ``` example [vpodzime@vpodzime-laptop ~]$ ansible-playbook --verbose --inventory /tmp/inventory --limit clients ~/tmp/deploy.yml Using /etc/ansible/ansible.cfg as config file [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [Deploy new hosts in the infrastructure] ***************************************************************************************************************************************************** TASK [Gathering Facts] **************************************************************************************************************************************************************************** [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client0 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client0] ok: [vpr8-clients-rhel-8-x64client1] ok: [vpr8-clients-rhel-8-x64client0] ok: [vpu18-clients-ubuntu-18-04-x64client1] ok: [vpu18-clients-ubuntu-18-04-x64client0] [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client1 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client1] TASK [Install and bootstrap CFEngine] ************************************************************************************************************************************************************* changed: [vpd10-clients-debian-10-x64client0] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_debian10_x86_64/cfengine-nova_3.18.1-1.debian10_amd64.deb", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} changed: [vpu18-clients-ubuntu-18-04-x64client1] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_ubuntu18_x86_64/cfengine-nova_3.18.1-1.ubuntu18_amd64.deb", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} changed: [vpd10-clients-debian-10-x64client1] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_debian10_x86_64/cfengine-nova_3.18.1-1.debian10_amd64.deb", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} changed: [vpr8-clients-rhel-8-x64client1] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_rhel8_x86_64/cfengine-nova-3.18.1-1.el8.x86_64.rpm", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} changed: [vpr8-clients-rhel-8-x64client0] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_rhel8_x86_64/cfengine-nova-3.18.1-1.el8.x86_64.rpm", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} changed: [vpu18-clients-ubuntu-18-04-x64client0] => {"cfengine_package": "https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.1/agent/agent_ubuntu18_x86_64/cfengine-nova_3.18.1-1.ubuntu18_amd64.deb", "cfengine_policy_server": "172.31.23.2", "cfengine_role": "agent", "changed": true, "state": "CFEngine installed and bootstrapped to 172.31.23.2"} PLAY RECAP **************************************************************************************************************************************************************************************** vpd10-clients-debian-10-x64client0 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpd10-clients-debian-10-x64client1 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client0 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client1 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client0 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client1 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` We can see that the task made changes on all the hosts, that it installed the respective packages for Debian 10, RHEL 8 and Ubuntu 18.04 and that all the hosts were bootstrapped to the given policy server so they became CFEngine agents (clients). One last remaining thing is to just run the playbook again on all hosts because that is what should be happening regularly to make sure the hosts are in the desired state. This time we drop the `--verbose` option to make the output less noisy: ``` example [vpodzime@vpodzime-laptop ~]$ ansible-playbook --inventory /tmp/inventory ~/tmp/deploy.yml [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [Deploy new hosts in the infrastructure] ***************************************************************************************************************************************************** TASK [Gathering Facts] **************************************************************************************************************************************************************************** ok: [vpu18-clients-ubuntu-18-04-x64client0] ok: [vpu18-clients-ubuntu-18-04-x64client1] ok: [vpr8-clients-rhel-8-x64client1] ok: [vpr8-clients-rhel-8-x64client0] ok: [vpc7-hub-centos-7-x64hub0] [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client0 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client0] [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client1 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client1] TASK [Install and bootstrap CFEngine] ************************************************************************************************************************************************************* ok: [vpc7-hub-centos-7-x64hub0] ok: [vpu18-clients-ubuntu-18-04-x64client0] ok: [vpu18-clients-ubuntu-18-04-x64client1] ok: [vpr8-clients-rhel-8-x64client1] ok: [vpr8-clients-rhel-8-x64client0] ok: [vpd10-clients-debian-10-x64client1] ok: [vpd10-clients-debian-10-x64client0] PLAY RECAP **************************************************************************************************************************************************************************************** vpc7-hub-centos-7-x64hub0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpd10-clients-debian-10-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpd10-clients-debian-10-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` As we can see, the `Install and bootstrap CFEngine` task just reports that all hosts are `ok`, i.e. in the desired state. And what happens if we add new hosts to the infrastructure? ``` example [vpodzime@vpodzime-laptop ~]$ cf-remote spawn --platform centos-7-x64 --count 2 --role clients --name c7-clients Spawning VMs.....DONE Waiting for VMs to get IP addresses..........DONE Details about the spawned VMs can be found in /home/vpodzime/.cfengine/cf-remote/cloud_state.json [vpodzime@vpodzime-laptop ~]$ cf-remote show --ansible-inventory > /tmp/inventory [vpodzime@vpodzime-laptop ~]$ ansible-playbook --inventory /tmp/inventory ~/tmp/deploy.yml [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [Deploy new hosts in the infrastructure] ***************************************************************************************************************************************************** TASK [Gathering Facts] **************************************************************************************************************************************************************************** ok: [vpr8-clients-rhel-8-x64client1] ok: [vpu18-clients-ubuntu-18-04-x64client1] ok: [vpu18-clients-ubuntu-18-04-x64client0] ok: [vpr8-clients-rhel-8-x64client0] ok: [vpc7-hub-centos-7-x64hub0] [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client0 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client0] [WARNING]: Platform linux on host vpd10-clients-debian-10-x64client1 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. ok: [vpd10-clients-debian-10-x64client1] ok: [vpc7-clients-centos-7-x64client0] ok: [vpc7-clients-centos-7-x64client1] TASK [Install and bootstrap CFEngine] ************************************************************************************************************************************************************* ok: [vpc7-hub-centos-7-x64hub0] ok: [vpu18-clients-ubuntu-18-04-x64client1] ok: [vpu18-clients-ubuntu-18-04-x64client0] ok: [vpr8-clients-rhel-8-x64client1] ok: [vpr8-clients-rhel-8-x64client0] ok: [vpd10-clients-debian-10-x64client0] ok: [vpd10-clients-debian-10-x64client1] changed: [vpc7-clients-centos-7-x64client0] changed: [vpc7-clients-centos-7-x64client1] PLAY RECAP **************************************************************************************************************************************************************************************** vpc7-clients-centos-7-x64client0 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpc7-clients-centos-7-x64client1 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpc7-hub-centos-7-x64hub0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpd10-clients-debian-10-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpd10-clients-debian-10-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpr8-clients-rhel-8-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 vpu18-clients-ubuntu-18-04-x64client1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` As expected, the task makes no change on the already deployed hosts and it does make changes on the new hosts. With `--verbose` we would see that it indeed installed the CentOS 7 CFEngine package only this time the `cfengine-nova` package, so the client package not the hub package, and that it bootstrapped the hosts to the CFEngine hub (policy server). ## Conclusions In this blog post we presented the CFEngine Ansible module that allows CFEngine to be installed and bootstrapped on the hosts using Ansible in a very easy way. We believe that using Ansible when deploying hosts in an infrastructure is a great and smart choice. And we also believe that CFEngine can handle many following aspects of infrastructure management once the hosts are deployed much better. So providing an easy mechanism to ensure CFEngine can start doing its job when Ansible is done deploying hosts is a logical step. The module provides the basic required functionality and works well on all GNU/Linux distributions supported by CFEngine. But it can surely be extended and improved, especially for environments that cannot rely on private network mechanisms handling the trust aspects. The [module](https://gitlab.com/Northern.tech/CFEngine/cfengine-ansible-collection/-/blob/master/plugins/modules/cfengine.py) is of course open-source and we welcome all contributions and [issue reports](https://gitlab.com/Northern.tech/CFEngine/cfengine-ansible-collection/-/issues). [^1]: At least theoretically. In practice, many of the Ansible modules require certain tools all libraries to be installed on the hosts. [^2]: Last time we tried to add it there, the *Sign up* didn\'t work properly and when it somehow worked, we were not really happy about the *Login with GitHub* integration which required access to quite a bit of information about us at *GitHub*. -------------------------------------------------------------------------------- title: "Using cfbs with a traditionally managed policy set" url: /blog/2022/using-cfbs-with-traditionally-managed-policy-set/index.md date: "2022-01-31" -------------------------------------------------------------------------------- With the recent release of [build.cfengine.com](https://build.cfengine.com) and `cfbs` I have been thinking about the process of converting a traditionally managed policy set. I consider a traditionally managed policy set one where you have a repo with the root of masterfiles being the root of the repository, or even having no repository at all and managing masterfiles by editing directly in the distribution point (e.g. `/var/cfengine/masterfiles`). Before jumping in with both feet and converting to a `cfbs` managed policy set you might want a hybrid situation where you can leverage some of the benefits of `cfbs` but without making drastic changes to the way policy is currently managed. That's what this post is about, using `cfbs` with your traditionally managed policy set. Note: This post assumes that you already have `cfbs` installed and understand the basics of how it works. Check out our previous blog posts if you want to review how to [get started with `cfbs`](https://cfengine.com/blog/2021/cfengine-build-launched/). When running `cfbs init` it's suggested that you `cfbs add masterfiles` as your first module. While modules can declare dependencies, none of the modules in the [default index](https://raw.githubusercontent.com/cfengine/build-index/master/cfbs.json) currently depend on masterfiles. So, if we don't `cfbs add masterfiles`, running `cfbs add` on other modules won't result in the MPF ([Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/reference-masterfiles-policy-framework.html)) being included in the `cfbs` build. Let's see: Here we make a directory for our `cfbs` project, switch into it and run `cfbs init`. ```sh cfbs_dir=/tmp/cfbs-playground mkdir -p ${cfbs_dir}; cd ${cfbs_dir} cfbs init ``` ```text Initialized - edit name and description cfbs.json To add your first module, type: cfbs add masterfiles ``` So, the project was successfully initialized and we are instructed to `cfbs add masterfiles` but we will skip that bit since we are looking to use `cfbs` with a traditionally managed policy set, or even a policy set that's not based on the MPF. Next we can add `autorun`, a module that simply defines the class `services_autorun` to enable a feature to automatically load policy files in services/autorun and run any bundles tagged with `autorun` (a feature of the MPF). ```sh cfbs add autorun && cfbs build ``` ```text Added module: autorun Modules: 001 autorun @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) Steps: 001 autorun : json 'def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz ``` The build completed successfully and we can see that the directory `out/masterfiles` contains the resulting assets (as well an accompanying archive at `out/masterfiles.tgz`). Let's take a look at the artifacts produced by the build. ```sh find out/masterfiles/* ``` ```text out/masterfiles/def.json ``` We see just one file, `def.json`. Looking at the content we can see that `services_autorun` gets defined and nothing else. ```sh jq < out/masterfiles/def.json ``` ```text { "classes": { "services_autorun": [ "any" ] } } ``` So, by simply adding and building the `autorun` module we get an [*augments file*](https://docs.cfengine.com/docs/3.18/reference-language-concepts-augments.html) defining the class to enable the feature. This could be merged with the *augments* as they sit in your traditionally managed policy set. Let's look at a `def.json` from a contrived policy set based on the MPF. ```sh jq < /tmp/contrived-masterfiles/def.json ``` ```text { "inputs": [ "services/my_custom.cf" ], "vars": { "msg": "Hello" }, "classes": { "contrived_policy_set": [ "any::" ] } } ``` From the output we can see that there is a custom policy file (`services/my_custom`) added to inputs a variable, `msg` in the `def` bundle scope is defined with the value `Hello` and the class `contrived_policy_set` is defined for any hosts. We can use `jq` to merge the `cfbs` built `def.json` into the one that already exists in our traditionally managed contrived policy set. ```sh jq -n 'reduce inputs as $i ({}; .* $i)' \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json \ | tee /tmp/cfbs-playground/out/masterfiles/def.json.merged ``` ```text { "inputs": [ "services/my_custom.cf" ], "vars": { "msg": "Hello" }, "classes": { "contrived_policy_set": [ "any::" ], "services_autorun": [ "any" ] } } ``` We can `diff` the original with the merged result and see that the classes object was extended to include the `services_autorun` class definition. ```sh diff -u \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json.merged | cat ``` ```text --- /tmp/contrived-masterfiles/def.json 2022-01-28 13:27:04.845285598 -0600 +++ /tmp/cfbs-playground/out/masterfiles/def.json.merged 2022-01-28 13:27:08.681439446 -0600 @@ -8,6 +8,9 @@ "classes": { "contrived_policy_set": [ "any::" + ], + "services_autorun": [ + "any" ] } } ``` We can continue down this path, let's add another module that vendors more than *augments*. In the `cfbs` project directory run let's add and build with `tmp-file-age`, a module to inventory and manage old files in `/tmp`. ```sh cfbs add tmp-file-age && cfbs build ``` ```text Added module: tmp-file-age Modules: 001 autorun @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 002 tmp-file-age @ 56a7c149f33808db6796de77eff6eb0502c01e61 (Downloaded) Steps: 001 autorun : json 'def.json' 'masterfiles/def.json' 002 tmp-file-age : copy './tmp-file-age.cf' 'masterfiles/services/security-hardening/tmp-file-age/' 002 tmp-file-age : json 'cfbs/def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz ``` Looking at the artifacts resulting from the build we can see that a policy file `services/security-hardening/tmp-file-age/tmp-file-age.cf` is added. ```sh tree /tmp/cfbs-playground/out/masterfiles/ ``` ```text /tmp/cfbs-playground/out/masterfiles/ ├── def.json └── services └── security-hardening └── tmp-file-age └── tmp-file-age.cf 3 directories, 2 files ``` Looking at the `def.json` resulting from the `cfbs build` we can see that the policy file was added to inputs and a bundle was added to the bundlesequence. ```sh jq < /tmp/cfbs-playground/out/masterfiles/def.json ``` ```text { "classes": { "services_autorun": [ "any" ] }, "vars": { "control_common_bundlesequence_end": [ "northerntech_security_hardening:tmp_file_age" ] }, "inputs": [ "services/security-hardening/tmp-file-age/tmp-file-age.cf" ] } ``` As done previously, we can merge this with our preexisting `def.json`. ```sh jq -n 'reduce inputs as $i ({}; .* $i)' \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json \ | tee /tmp/cfbs-playground/out/masterfiles/def.json.merged ``` ```text { "inputs": [ "services/security-hardening/tmp-file-age/tmp-file-age.cf" ], "vars": { "msg": "Hello", "control_common_bundlesequence_end": [ "northerntech_security_hardening:tmp_file_age" ] }, "classes": { "contrived_policy_set": [ "any::" ], "services_autorun": [ "any" ] } } ``` Again we can `diff` to see the detailed changes. ```sh diff -u \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json.merged | cat ``` ```text --- /tmp/contrived-masterfiles/def.json 2022-01-28 13:27:04.845285598 -0600 +++ /tmp/cfbs-playground/out/masterfiles/def.json.merged 2022-01-28 14:12:24.653577092 -0600 @@ -1,13 +1,19 @@ { "inputs": [ - "services/my_custom.cf" + "services/security-hardening/tmp-file-age/tmp-file-age.cf" ], "vars": { - "msg": "Hello" + "msg": "Hello", + "control_common_bundlesequence_end": [ + "northerntech_security_hardening:tmp_file_age" + ] }, "classes": { "contrived_policy_set": [ "any::" + ], + "services_autorun": [ + "any" ] } } ``` Oh no! Good thing we checked, that's not quite right. The inputs array was replaced instead of extended. We need a different incantation for `jq`. Let's try again ... ```sh jq -s 'def deepmerge(a;b): reduce b[] as $item (a; reduce ($item | keys_unsorted[]) as $key (.; $item[$key] as $val | ($val | type) as $type | .[$key] = if ($type == "object") then deepmerge({}; [if .[$key] == null then {} else .[$key] end, $val]) elif ($type == "array") then (.[$key] + $val | unique) else $val end) ); deepmerge({}; .)' \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json \ | tee /tmp/cfbs-playground/out/masterfiles/def.json.merged ``` ```text { "inputs": [ "services/my_custom.cf", "services/security-hardening/tmp-file-age/tmp-file-age.cf" ], "vars": { "msg": "Hello", "control_common_bundlesequence_end": [ "northerntech_security_hardening:tmp_file_age" ] }, "classes": { "contrived_policy_set": [ "any::" ], "services_autorun": [ "any" ] } } ``` Let's check that our merge behaved as desired. ```sh diff -u \ /tmp/contrived-masterfiles/def.json \ /tmp/cfbs-playground/out/masterfiles/def.json.merged | cat ``` ```text --- /tmp/contrived-masterfiles/def.json 2022-01-28 14:33:45.518200635 -0600 +++ /tmp/cfbs-playground/out/masterfiles/def.json.merged 2022-01-28 14:34:41.658773202 -0600 @@ -1,13 +1,20 @@ { "inputs": [ - "services/my_custom.cf" + "services/my_custom.cf", + "services/security-hardening/tmp-file-age/tmp-file-age.cf" ], "vars": { - "msg": "Hello" + "msg": "Hello", + "control_common_bundlesequence_end": [ + "northerntech_security_hardening:tmp_file_age" + ] }, "classes": { "contrived_policy_set": [ - "any::" + "any::" + ], + "services_autorun": [ + "any" ] } } ``` Now that we are confident in the changes to `def.json` we can put the merged result into place and copy the policy built by cfbs into our traditionally managed contrived policy set. ```sh mv -v \ /tmp/cfbs-playground/out/masterfiles/def.json.merged \ /tmp/contrived-masterfiles/def.json rsync -av\ /tmp/cfbs-playground/out/masterfiles/ \ /tmp/contrived-masterfiles/ \ --exclude def.json ``` ```text renamed '/tmp/cfbs-playground/out/masterfiles/def.json.merged' -> '/tmp/contrived-masterfiles/def.json' sending incremental file list ./ services/ services/security-hardening/ services/security-hardening/tmp-file-age/ services/security-hardening/tmp-file-age/tmp-file-age.cf sent 3,097 bytes received 54 bytes 6,302.00 bytes/sec total size is 2,832 speedup is 0.90 ``` That's it, we integrated both the `autorun` module and the `tmp-file-age` module into our traditionally managed policy set. This is a good way to easily consume content with cfbs before moving to a fully managed cfbs policy set. Moving fully to cfbs brings other great features like removing modules you no longer want and cleaning up unused dependencies. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 9 - Introduction to cf-secret" url: /blog/2022/the-agent-is-in-episode-9-cf-secret/index.md date: "2022-01-27" -------------------------------------------------------------------------------- How can I work with secrets using CFEngine? Craig (Digger) demoed `cf-secret` and how he uses it for protecting secrets used to mount LUKS encrypted drives. ## Video {#video} The video recording is available on YouTube: {{< youtube ElnfCTtr_lc >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [cf-secret](https://docs.cfengine.com/docs/master/reference-components-cf-secret.html) - [LUKS](https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Happy Chinese New Year | 新年快乐 (CFEngine 总年回顾)" url: /blog/2022/happy-chinese-new-year/index.md date: "2022-01-26" -------------------------------------------------------------------------------- [![](images/chinese_new_year.jpg)](images/chinese_new_year.jpg) _(This is a blog post to celebrate Chinese New Year for our Chinese-speaking users.)_ 作为年前的最后一篇文章,并延续我们的传统,我们想回顾一下CFEngine在这一年中的所有成果,并对新的一年我们的开发计画做一个简要的介绍。 ## 现代化的 Mission Portal UI 在6月发布的CFEngine Enterprise 3.18.0 LTS中,我们对网络用户界面进行了全面的修改。您可以在我们的[文章中](/blog/2021/the-cfengine-ui-gets-a-fresh-look/)了解到相关的变化。明年我们将继续在Mission Portal内进行实质性的设计修改,目的是创造一个友善的使用环境并让使用者能够更直观得使用用户界面。 [![](/blog/2021/images/frame-5516.png)](/blog/2021/images/frame-5516.png) ## 主机特定数据(CMDB) 我们还在Mission Portal中增加了主机特定数据。你现在可以将变量和类别分配给各个主机。这对于设置机器的所有者,为其分配角色,或控制其策略(policy)的作用是非常好的。举个例子,您可以制定策略,自动更新一台机器上的所有软件包。您可能不希望在任何时候都启用这个功能,但对特定的主机暂时启用它是非常有用的。 [![](/blog/2021/images/cmdb.png)](/blog/2021/images/cmdb.png) ## 触发代理(agent)运行和报告收集 除了从主机信息页面分配数据外,还有新的按钮来执行有用的行动,触发代理运行和报告收集。这个开发者演示视频显示了新的代理运行和报告收集按钮的作用。 {{< youtube -ZAMFb2msgA >}} ## 全新网站 今年初,我们公布了我们了全新的[CFEngine网站](https://cfengine.com/)。其主要的目标是为了让新用户更好地突出CFEngine的功能,然而我们也希望改善现有用户的导航,使其更容易尽快准确地找到您正在寻找的东西。我们实现这一目标的方法之一是创建一个CFEngine[开发者区域](/developers),它专门为现有用户提供服务,能更快得利用客服系统、文档和其他技术资源。 我们还创建了一个全新的资源区域,在这里您可以找到我们所有以营销为重点的内容,包括网络[研讨会的录影集](/webinars/)、[教学视频](/videos/)、[白皮书](/white-papers/)、[案例研究](/case-studies/)等等。 [![](/blog/2021/images/new-website.png)](https://cfengine.com) ## 代理人网络研讨会系列 由于COVID的旅行限制和无法参加会议,我们希望找到继续吸引社区的方法。因此,我们推出了名为 "代理人会议 "的每月网络研讨会系列。我们把这个网络研讨会系列作为一种方式来强调新的功能,如CFEngine Build,物联网的的使用范例,甚至从其他社区成员那里学习新的技巧。所有的网络研讨会都可以在我们的[网站](https://cfengine.com/webinars/)上找到。 自从研讨会推出以来我们得到了许多肯定与支持,我们将把这个网络研讨会系列延续到2022年,并期待展示社区和CFEngine团队的更多创造力。 ## GitHub 论坛 我们已经有多种方式与CFEngine团队和其他社区成员互动,包括邮件讨论群、IRC、定期的时事通讯,然而随着用户群的增长和GitHub变得更加突出,我们认为使用他们新的讨论功能可以更好地让用户以他们最舒适的方式参与。 如果论坛式的交流是您喜欢的方式,那么欢迎来[介绍您自己](https://github.com/cfengine/core/discussions/4546),并告诉我们您的CFEngine故事,让彼此有更多机会交流。 ## CFEngine Build 共同协作是我们想在2021年特别关注的一个领域。虽然每一个新的软件都有一个学习曲线,但我们希望为新的CFEngine用户提供一种开箱即用的方法,同时也为现有的用户提供一种方法来轻松简化他们的系统执行,而不必总是创建策略。这是一个雄心勃勃的目标,但也成就了[CFEngine Build](https://build.cfengine.com/)的诞生。在这个新的网站上,用户可以很容易地实现并相互分享自定义策略和模块。 [![](.//blog/2021/images/cfengine-build.png)](https://build.cfengine.com/) ## 为安全加固的圣诞假期日历 为了说明CFEngine Build的力量,我们创建了安全加固圣诞日历,在那里我们每天发布一个以安全为重点的模块,直到圣诞节。直到圣诞节。这将帮助您改善您的组织的安全相关状态,并为您的系统安全等级加分。虽然圣诞已经过去了,但现在阅读我们的总结并在您自己的系统环境中实施它们也不算太晚。 * [第一周](/blog/2021/security-hardening-holiday-calendar-week-1) * [第二周](/blog/2021/security-hardening-holiday-calendar-week-2) * [第三周](/blog/2021/security-hardening-holiday-calendar-week-3) * [第四周](/blog/2021/security-hardening-holiday-calendar-week-4) [![](/blog/2021/images/1-25.png)](https://twitter.com/cfengine/status/1465772009024012305) ## 发布概述 我们总共发布了6个新版本的CFEngine,请在各自的博文中阅读它们的内容。 * [CFEngine 3.12.7 and 3.15.4 LTS](/blog/2021/cfengine-3-12-7-and-3-15-4-released) * [CFEngine 3.18.0 LTS - 扩展性](/blog/2021/cfengine-3-18-lts-released-extensibility) * [CFEngine 3.15.5 and 3.18.1 LTS](/blog/2021/cfengine-3-15-5-and-3-18-1-released) * [CFEngine 3.19.0 - 合作性](/blog/2021/cfengine-3-19-released-collaboration) ## 新的一年 2021年将是难以超越的,但我们对2022年的发展感到兴奋。共同协作、易用性和社区参与是今年的一大重点,明年我们将继续在这些领域上加倍努力。您将看到我们的系統文件未来的改进,使其比以往更容易优化您的CFEngine实施。我们还将引入新的功能,目的是使Mission Portal更具有可操作性,减少为基本功能创建策略的需要。 最后,我们想要感谢 CFEngine 社群用户的贡献和持续支持。您的反馈和想法对 CFEngine 的成功至关重要,它帮助我们开发更好的产品。我们祝福您在新的一年身体健康,虎虎生风,並期盼着新的一年与您的合作。 CFEngine 團隊敬上 -------------------------------------------------------------------------------- title: "CFEngine 2021 retrospective" url: /blog/2021/2021-retrospective/index.md date: "2021-12-31" -------------------------------------------------------------------------------- For our final blog post of 2021 and continuing our tradition, we'd like to reflect on all the CFEngine accomplishments throughout the year and provide a sneak peek of what to expect in 2022. ## Modernized Mission Portal UI In CFEngine Enterprise 3.18.0 LTS, released in June, we overhauled the web user interface. You can read about the changes in our [blog post on the subject](/blog/2021/the-cfengine-ui-gets-a-fresh-look/). We will continue to make meaningful design changes within Mission Portal next year with the goal of making it more intuitive and user friendly. [![](images/frame-5516.png)](images/frame-5516.png) ## Host specific data (CMDB) We also added host specific data to Mission Portal. You can now assign variables and classes to individual hosts. This is great for setting the owner of a machine, assigning it a role, or controlling what its policy is doing. As an example, you can have policy to automatically update all packages on a machine. You might not want this enabled at all times, but temporarily enabling it for specific hosts is very useful. [![](images/cmdb.png)](images/cmdb.png) ## Trigger agent runs and report collection In addition to assigning data from the host info page, there are new buttons to perform useful actions, triggering agent runs and report collection. This developer demo video shows the new agent run and report collection buttons in action: {{< youtube -ZAMFb2msgA >}} ## New website Earlier this year we revealed our [new CFEngine website](https://cfengine.com/). One main goal was to better highlight CFEngine's capabilities for potential new users, however we also wanted to improve navigation for existing users, making it easier to find exactly what you're looking for as quickly as possible. One way we achieved this is by creating a CFEngine [developer section](/developers) that is specifically meant for existing users who are more likely looking for customer support, documentation, and other technical resources. We've also created a whole new Resources section where you can find all of our marketing focused content including recorded [webinars](/webinars/), [videos](/videos/), [white papers](/white-papers/), [case studies](/case-studies/) and more. [![](images/new-website.png)](https://cfengine.com) ## "The agent is in" webinar series Due to COVID travel restrictions and being unable to attend conferences, the CFEngine team wanted to find ways to continue engaging the community. As a result, we launched our monthly webinar series titled "The agent is in". We've used this webinar series as a way to highlight new features like CFEngine Build, new use cases for IoT, and even learn new tricks from other community members. All of the webinars are available to [stream on our website](https://cfengine.com/webinars/). Because of its success, we will continue this webinar series into 2022 and look to show off more creativity from the community and the CFEngine team. ## GitHub Discussions There are already several ways to interact with the CFEngine team and other community members, including the Mailing List, IRC and newsletter, however as the user base grows and GitHub becomes more prominent, we felt that using their new Discussions feature would better allow users to engage in a way that they're most comfortable. If forum style communication is your preferred method, then come introduce yourself and [tell us your CFEngine story](https://github.com/cfengine/core/discussions/4546). As a thank you we will send you some cool CFEngine SWAG. ## CFEngine Build Collaboration was an area that we wanted to specifically focus on in 2021. While there's a learning curve with every new piece of software, we wanted to provide a way for new CFEngine users to get instant value out-of-the-box, but also provide existing users a way to easily optimize their implementation without always having to create policy. This was an ambitious goal, but led to the creation of [CFEngine Build](https://build.cfengine.com/). On this new website, users can easily implement and share custom policy & modules with one another. [![](./images/cfengine-build.png)](https://build.cfengine.com/) ## Security hardening holiday calendar To illustrate the power of CFEngine Build, we created our Security Hardening Holiday Calendar where we released a new security-focused module every day until Christmas that will help you improve your organization's security posture and win you some brownie points with your boss. Christmas may be over, but it's not too late to read our recap and implement them in your own environment: * [Week 1](/blog/2021/security-hardening-holiday-calendar-week-1) * [Week 2](/blog/2021/security-hardening-holiday-calendar-week-2) * [Week 3](/blog/2021/security-hardening-holiday-calendar-week-3) * [Week 4](/blog/2021/security-hardening-holiday-calendar-week-4) [![](images/1-25.png)](https://twitter.com/cfengine/status/1465772009024012305) ## Releases In total, we released 6 new versions of CFEngine, read about them in their respective blog posts: * [CFEngine 3.12.7 and 3.15.4 LTS](/blog/2021/cfengine-3-12-7-and-3-15-4-released) * [CFEngine 3.18.0 LTS - Extensibility](/blog/2021/cfengine-3-18-lts-released-extensibility) * [CFEngine 3.15.5 and 3.18.1 LTS](/blog/2021/cfengine-3-15-5-and-3-18-1-released) * [CFEngine 3.19.0 - Collaboration](/blog/2021/cfengine-3-19-released-collaboration) ## Next year 2021 will be hard to top, but we're excited for what's in store for 2022. Collaboration, ease of use and community engagement were big focuses this year and we will continue to double down in these areas next year. You will see future improvements in our documentation, making it easier than ever to optimize your CFEngine implementation. We will also be introducing new features that aim to make Mission Portal more actionable, reducing the need to create policy for basic functions. Finally, we'd also like to thank the CFEngine community for all their dedication and continued support. Your feedback and ideas are critical to the success of CFEngine and help us develop meaningful improvements. We wish you and your family good health, a Happy Holiday season, and look forward to working together in 2022. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 8 - Security hardening holiday calendar" url: /blog/2021/the-agent-is-in-episode-8-security-hardening-holiday-calendar/index.md date: "2021-12-30" -------------------------------------------------------------------------------- Looking for ways to improve the security of your infrastructure? Craig (Digger) and Nick (Doer of Things) walk us through some of the policies shared during the [2021 CFEngine security holiday hardening calendar](/blog/2021/security-hardening-holiday-calendar-week-4/). ## Video {#video} The video recording is available on YouTube: {{< youtube yo77elKo5nY >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [CFEngine security hardening holiday calendar week 1](/blog/2021/security-hardening-holiday-calendar-week-1) - [CFEngine security hardening holiday calendar week 2](/blog/2021/security-hardening-holiday-calendar-week-2) - [CFEngine security hardening holiday calendar week 3](/blog/2021/security-hardening-holiday-calendar-week-3) - [CFEngine security hardening holiday calendar week 4](/blog/2021/security-hardening-holiday-calendar-week-4/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Security hardening holiday calendar - Week 4" url: /blog/2021/security-hardening-holiday-calendar-week-4/index.md date: "2021-12-25" -------------------------------------------------------------------------------- This is the final summary of our 2021 security hardening holiday calendar. We wanted to provide educational, useful, and actionable security advice, and we're really pleased with the reception! Thank you for reading and following along. [![CFEngine themed advent calendar design with 25 boxes, 25 of them are revealed with the same titles as the following sections](images/25.12.png)](images/25.12.png) ## Week 1-3 summary (1-21/25) We posted summaries for the 3 first weeks of the calendar: * [Week 1](/blog/2021/security-hardening-holiday-calendar-week-1) * [Week 2](/blog/2021/security-hardening-holiday-calendar-week-2) * [Week 3](/blog/2021/security-hardening-holiday-calendar-week-3) ## Enforce specific list of allowed sudoers (22/25) As discussed previously, the root user and sudo tool provide a lot of access to the system, both in terms of making changes, and reading sensitive data. You should tightly control who can use sudo (the sudoers). This module allows you to specify a list of sudoers, and have it enforced; no other users will be allowed to use sudo: [build.cfengine.com/modules/sudo-enforce-allowed-users](https://build.cfengine.com/modules/sudo-enforce-allowed-users) **Recommendation:** Specify the list of users allowed to use sudo, and limit it as much as possible. Ensure it is enforced everywhere with CFEngine and this module. ## Who's logging into your hosts? (23/25) Reports of who logged on to hosts are important for security monitoring and audits / compliance. After some change or unexpected event, it is especially valuable to see which users have logged in to the relevant systems. You should set up all your hosts to automatically report this information, continuously: [build.cfengine.com/modules/inventory-lastlog](https://build.cfengine.com/modules/inventory-lastlog) [![](images/inventory-lastlog.png)](images/inventory-lastlog.png) **Recommendation:** Use this module to get a complete overview of who is logging into your hosts in the CFEngine Enterprise Web UI (Mission Portal). Look for unexpected user logins. ## Uninstall ftp package (24/25) Using FTP servers is an easy way to share files over the network. FTP clients can connect and download the files they want. If you use FTP, it's probably on a few specific / dedicated machines. We provide a module for uninstalling ftp everywhere you don't need it: [build.cfengine.com/modules/ftp-server-not-installed](https://build.cfengine.com/modules/ftp-server-not-installed) As a test, you can install `vsftpd` and then run CFEngine with this module, to see that it's uninstalled: ``` $ sudo apt install vsftpd $ cf-agent -KI info: Successfully removed package 'vsftpd' ``` **Recommendation:** Disable and uninstall FTP everywhere it's not needed, to avoid unintended data exposure and reduce attack surface. ## Supported version of CFEngine (25/25) We are constantly improving CFEngine and fixing issues. The 3.15 and 3.18 LTS series are currently supported. To get the latest bug fixes and security updates, you should upgrade to the latest release for your LTS series (3.15.5 and 3.18.1). This module adds reporting information around whether your hosts are running supported versions or not: [build.cfengine.com/modules/cfengine-supported](https://build.cfengine.com/modules/cfengine-supported) **Recommendation:** Always use a supported version of CFEngine and try to upgrade to new releases as quickly as possible. Use the reporting system in CFEngine Enterprise to stay on top of which versions you are using and what hosts need to be upgraded. ## Thank you Thank you for following our holiday themed security hardening calendar. We look forward to sharing more useful content, security advice, and modules with you in 2022. Stay up to date by following us on [LinkedIn](https://www.linkedin.com/company/northern.tech/) and [Twitter](https://twitter.com/cfengine). Happy holidays! -------------------------------------------------------------------------------- title: "Hunting and tracking remediation of Log4Shell (CVE-2021-44228)" url: /blog/2021/cve-2021-44228-log4j-log4shell-cfbs-module/index.md date: "2021-12-22" -------------------------------------------------------------------------------- The internet has been ablaze since the announcement of [Log4Shell](https://en.wikipedia.org/wiki/Log4Shell), the nickname for [CVE-2021-44228](https://nvd.nist.gov/vuln/detail/CVE-2021-44228), an arbitrary remote code execution vulnerability in the Java logging utility Log4j. So far two additional vulnerabilities ([CVE 2021-45046](https://nvd.nist.gov/vuln/detail/CVE-2021-45046), [CVE-2021-45105](https://nvd.nist.gov/vuln/detail/CVE-2021-45105)) have been identified. If you are interested in how the vulnerability works, this graphic from [SecurityZines](https://securityzines.com/flyers/log4j.html) explains it well: [![](images/log4j.png)](images/log4j.png) The code has been vulnerable since 2013 and millions of hosts and services are affected. The US Cybersecurity and Infrastructure Security Agency (CISA) issued an [emergency directive](https://www.cisa.gov/emergency-directive-22-02) on December 17th, 2021 ordering all civilian federal agencies to take a series of measures to identify, patch, or mitigate vulnerable systems. Agencies have until 5pm EST on December 23rd, 2021 to comply with the requirements of the directive. [Many projects](https://github.com/search?q=log4shell) have been kicked off in pursuit of hunting down and remediating affected instances. As the deadline fast approaches, we wanted to show how CFEngine can be used in combination with these other projects to facilitate identifying potentially vulnerable instances of log4j within your infrastructure and decided to publish a module, [cve-2021-44228-log4j](https://build.cfengine.com/modules/cve-2021-44228-log4j/) to help you identify potentially affected instances within CFEngine managed infrastructure and track remediation efforts. [![](images/cve-2021-44228-log4j-screenshot.png)](images/cve-2021-44228-log4j-screenshot.png) -------------------------------------------------------------------------------- title: "Security hardening holiday calendar - Week 3" url: /blog/2021/security-hardening-holiday-calendar-week-3/index.md date: "2021-12-21" -------------------------------------------------------------------------------- This December, we are posting security advice and modules, every day until December 25th. Now, it's December 21st, and we've gotten through most of the security hardening holiday calendar: [![CFEngine themed advent calendar design with 25 boxes, 21 of them are revealed with the same titles as the following sections](images/21.12.png)](images/21.12.png) ## Week 1 & 2 summary (1-14/25) We posted summaries for the 2 first weeks of the calendar: * [Week 1](/blog/2021/security-hardening-holiday-calendar-week-1) * [Week 2](/blog/2021/security-hardening-holiday-calendar-week-2) ## Disable prelinking (15/25) A technique called prelinking can be used to optimize programs, making them start up faster. As this feature will change the binary file, it interferes with security functionality such as checksumming and signatures. For these reasons it is generally a good idea to disable prelinking, unless you really need it. [build.cfengine.com/modules/prelinking-disabled](https://build.cfengine.com/modules/prelinking-disabled) Running CFEngine with this module in your policy set on a system which has prelinking enabled should produce similar output as below: ``` # cf-agent -KI info: Replaced pattern '^\s*(PRELINKING\s*=\s*(?!no$).*|PRELINKING)$' in '/etc/default/prelink' info: replace_patterns promise '^\s*(PRELINKING\s*=\s*(?!no$).*|PRELINKING)$' repaired info: Edited file '/etc/default/prelink' info: Executing 'no timeout' ... '/usr/sbin/prelink -ua' info: Completed execution of '/usr/sbin/prelink -ua' ``` **Recommendation:** If you don't need it, disable prelinking by editing `/etc/default/prelink`, either manually, or using our module. ## Disable Empty Passwords in SSH (16/25) Some systems create users with empty passwords. These users should not be allowed to log in remotely, as it is an extremely easy "password" for attackers to guess. You should configure SSH to not allow logging in with an empty password: [build.cfengine.com/modules/ssh-permit-empty-passwords-no](https://build.cfengine.com/modules/ssh-permit-empty-passwords-no) **Recommendation:** Ensure logging in with an empty password is not possible, by changing `sshd` configuration. ## Permissions for directories in root PATH (17/25) On [day 11](https://mailchi.mp/cfengine/hardening-11), we showed reports of writable directories in root's PATH variable. These represent a security risk, as other users can modify the programs run by the root user. To fix this, you should enforce the permissions of all directories in root's PATH variable: [build.cfengine.com/modules/root-path-enforce-permissions](https://build.cfengine.com/modules/root-path-enforce-permissions) **Recommendation:** Control the permissions of all directories in root's PATH. Use automation to enforce this everywhere, continuously. ## Add nosuid option to /tmp (18/25) The SUID and SGID flags enable specific executables, such as sudo, to be run with additional access (setting the UID or GID). This should be tightly controlled. Users should not be able to execute SUID or SGID binaries from temporary storage partitions. You should make /tmp a separate mount point with a `nosuid` mount option to prevent execution of SUID programs there: [build.cfengine.com/modules/tmp-nosuid](https://build.cfengine.com/modules/tmp-nosuid) **Recommendation:** Ensure the `/tmp` directory is mounted with the `nosuid` option. ## Remove old files in /tmp (19/25) The /tmp directory holds temporary files used by various programs. This can be logs, cache files, partial downloads, etc. Since many programs use the /tmp folder, an attacker could get some software to write there, or look for valuable data to exfiltrate. It is a good idea to clean up regularly: [build.cfengine.com/modules/tmp-file-age](https://build.cfengine.com/modules/tmp-file-age) **Recommendation:** Clean up the /tmp directory regularly to free up disk space and remove unwanted old files. ## Use only strong ciphers (20/25) SSH has been around for a long time and supports many different cryptographic algorithms. Older ciphers are generally less secure than the newer additions. You should ensure it's configured to only use modern, strong cryptographic ciphers: [build.cfengine.com/modules/ssh-ciphers-strong](https://build.cfengine.com/modules/ssh-ciphers-strong) **Recommendation:** Use our module to ensure only strong ciphers are allowed in SSH. Deploy it as part of your policy set so it's enforced everywhere in your infrastructure. ## Reports of the results of yum's update info (21/25) Not all updates are the same. Most systems have some pending updates all the time. But security updates should be treated more carefully, especially the important or even critical ones. First step is to know that there are such updates available. On Red Hat Enterprise Linux, the `yum updateinfo sumary` command gives you information about security updates. Our module on CFEngine Build uses this functionality in `yum`: [build.cfengine.com/modules/inventory-yum-update-info](https://build.cfengine.com/modules/inventory-yum-update-info) This gives you the desired information as part of the CFEngine Enterprise reporting data: [![](images/security-updates.png)](images/security-updates.png) (To see this, go to Reports -> Inventory -> Click the 3 dots at the top of a column -> Insert Column -> Security updates and similar for the other attributes.) **Recommendation:** Pay attention to security upgrades and perform them as quickly as possible. Run `yum updateinfo summary` from the command line or use this module to get the information from all of your RHEL servers. Look for similar solutions if you are using other platforms. ## Subscribe to the calendar To sign up for the remaining days of the calendar [use this link](/calendar). -------------------------------------------------------------------------------- title: "What happened to sudo and who is Baron Samedit?" url: /blog/2021/cve-2021-3156-sudo/index.md date: "2021-12-16" -------------------------------------------------------------------------------- In January of 2021 [Qualys security researchers discovered a heap overflow vulnerability in `sudo`](https://blog.qualys.com/vulnerabilities-threat-research/2021/01/26/cve-2021-3156-heap-based-buffer-overflow-in-sudo-baron-samedit), an extremely common tool installed in most Unix and Linux operating systems. Sudo allows users to execute programs with the privileges of another user but the vulnerability allows any unprivileged user to gain root on a vulnerable host. This specific vulnerability was nicknamed "Baron Samedit". The [Buffer overflow in command line escaping blog post](https://www.sudo.ws/security/advisories/unescape%5Foverflow/) on sudo.ws notes that the vulnerability can be tested by executing `sudoedit -s /`. When run as `root` a vulnerable version of `sudo` will display an error `sudoedit: /: not a regular file`. We can define a class based on this: ```cfengine3 classes: "cve_2021_3156_vulnerable" scope => "namespace", expression => strcmp( "sudoedit: /: not a regular file", execresult( "$(_sudoedit) -s /", "noshell" )); ```
Code Snippet 1: Define the class cve_2021_3156_vulnerable on hosts that are vulnerable
Using this class we can make other decisions in policy, for example to try and remediate automatically by upgrading the `sudo` package what the policy does when it's found. ```cfengine3 packages: cve_2021_3156_vulnerable:: "sudo" policy => "present", version => "latest"; ```
Code Snippet 2: Upgrade sudo to the latest version available when it is found to be vulnerable
We published a [module for this on build.cfengine.com](https://build.cfengine.com/modules/cve-2021-3156-sudo/). Integrate it into your policy with `cfbs` with `cfbs add cve-2021-3156-sudo`. Learn more about [build.cfengine.com](https://build.cfengine.com/modules/cve-2021-3156-sudo/) and `cfbs` from [our blog post introducing the new tooling](https://cfengine.com/blog/2021/cfengine-build-launched/). -------------------------------------------------------------------------------- title: "Security hardening holiday calendar - Week 2" url: /blog/2021/security-hardening-holiday-calendar-week-2/index.md date: "2021-12-14" -------------------------------------------------------------------------------- This December, we are posting security advice and modules, every day until December 25th. Now, it's December 14th, and we've gotten to the fourteenth day of the security hardening holiday calendar: [![CFEngine themed advent calendar design with 25 boxes, 14 of them are revealed with the same titles as the following sections](images/14.12.png)](images/14.12.png) ## Week 1 summary (1-7/25) If you didn't see it yet, we posted a summary last week. [Click here](/blog/2021/security-hardening-holiday-calendar-week-1) to read the security tips for day 1-7. ## Non-root users with uid 0 (8/25) On most UNIX-like systems, there is a user called root, with an ID number 0 (uid). Software may assume or expect that to be the case. If the root user is renamed, or another user has 0 as its uid, this can cause problems or indicate a misconfiguration. Whenever there are other user accounts with uid 0, you should know about it and investigate it: [build.cfengine.com/modules/inventory-local-users-non-root-has-uid-0](https://build.cfengine.com/modules/inventory-local-users-non-root-has-uid-0) This host adds inventory data to each host: [![](images/non-root-with-uid-0.png)](images/non-root-with-uid-0.png) **Recommendation:** Look for instances of non-root users with uid 0. Investigate why and fix the misconfiguration (unless there is a good reason why). ## Vendor supported OS (9/25) When an operating system vendor stops supporting a version of their OS, it no longer receives bugfixes or security patches. Running outdated and vulnerable operating systems can be a big risk. You should have compliance reports which clearly highlight when you are running unsupported operating systems: [build.cfengine.com/modules/compliance-report-os-is-vendor-supported](https://build.cfengine.com/modules/compliance-report-os-is-vendor-supported) CFEngine Enterprise 3.18 LTS includes compliance reports, and the report provided by this module easily shows you which hosts are compliant and which ones are not: [![](images/os-is-vendor-supported.png)](images/os-is-vendor-supported.png) **Recommendation:** Use the reports from this module to see what operating systems are no longer supported. Upgrade the outdated operating systems as soon as possible. ## Uninstall openldap-servers package (10/25) LDAP is a commonly used solution for user authentication and user information storage, but only LDAP servers need the server packages, other hosts only need client packages. You should disable and uninstall the software where it isn't needed. [build.cfengine.com/modules/openldap-server-not-installed](https://build.cfengine.com/modules/openldap-server-not-installed) If you add this module and run cf-agent on a machine with openldap-server installed, it is uninstalled: ``` [root@hub]# cf-agent -KI info: Successfully removed package 'openldap-servers' ``` **Recommendation:** Ensure only the necessary machines have OpenLDAP server installed, uninstall everywhere else. ## Reports of writable directories in root PATH (11/25) The PATH variable defines where a shell, or other software will look for programs to run. It's a list of directories (folders) and differs per user. By writing to the folders mentioned in a user's PATH variable, you can change the programs ran by that user. If that user has more access than you, this enables a privilege escalation vulnerability. You should monitor all folders in the root's PATH variable and ensure other users cannot write to them. [build.cfengine.com/modules/inventory-writable-directories-in-root-path](https://build.cfengine.com/modules/inventory-writable-directories-in-root-path/) The reporting data provided by the module looks like this: [![](images/writable-dirs-in-root-path.png)](images/writable-dirs-in-root-path.png) **Recommendation:** Monitor the permissions on all folders in root's PATH variable. Ensure that other users cannot add, modify or delete programs in those folders. ## Continuously poll ntp servers (12/25) Network Time Protocol (NTP) can be used to automatically set the system time. Having accurate time is important for many different pieces of software, and makes the logs and other timestamped data from the host a lot more usable. You should ensure the system time is always correct by regularly polling the NTP server(s). [build.cfengine.com/modules/ntp-maxpoll](https://build.cfengine.com/modules/ntp-maxpoll) In CFEngine Enterprise, you can use the built in CMDB (Host specific data) to configure this per host: [![](images/ntp-config-maxpoll.png)](images/ntp-config-maxpoll.png) **Recommendation:** Ensure all your systems have the correct time, by frequently using NTP to poll time servers and set the time. ## Allow Only SSH Protocol 2 (13/25) SSH protocol version 2 (SSH-2) was released in 2006 and includes significant improvements for security, including Diffie-Hellman based key exchange, and integrity checking via message authentication codes. At this point, in 2021, all your SSH servers and clients should support SSH-2, and it is a good idea to enforce only using this newer version of the protocol. [build.cfengine.com/modules/ssh-protocol-2](https://build.cfengine.com/modules/ssh-protocol-2) **Recommendation:** Use this module to ensure SSH protocol 1 is not allowed on any of your servers. ## Sudo should require passwords (14/25) The sudo program allows other users to temporarily become the root user, to run commands which require additional privileges. Users should be aware and explicitly approve when programs need to use sudo. You should configure sudo to require passwords, to prevent both accidental commands and malicious code from damaging the system. [build.cfengine.com/modules/sudo-requires-passwords](https://build.cfengine.com/modules/sudo-requires-passwords) **Recommendation:** Require users to type their passwords when using sudo, if possible. ## Subscribe to the calendar There are 11 days left of the calendar! To sign up and receive daily security hardening content through Dec. 25th, [use this link](/calendar). -------------------------------------------------------------------------------- title: "CFEngine 3.19 released - Collaboration" url: /blog/2021/cfengine-3-19-released-collaboration/index.md date: "2021-12-10" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.19.0\! In 2021, for this release, and the [launch of CFEngine Build](/blog/2021/cfengine-build-launched/), our focus has been on _collaboration_. We want to deliver a lot of value to our users through modules, and enable you to share and cooperate on policy, promise types, compliance reports, etc. CFEngine 3.19 is not an LTS release, so the intention for us is to give you a chance to start testing and giving feedback on the new features we are developing, before they land in an LTS version next year. ## What's new ### Faster report collection The CFEngine reporting hub now intelligently prioritizes the order of hosts to collect reports from. Newly bootstrapped hosts which don't have any reporting data yet will be contacted first, then hosts are sorted based on the time of their last report, so the oldest data is refreshed first. In the end, this should result in more fresh data in Mission Portal, both for newly bootstrapped hosts, and in general. ### Importable Compliance Reports Compliance reports can now be stored and managed as JSON files. This functionality is available in both the UI and API, and enables us to deliver ready-to-use compliance reports as modules. The first module like this is [available on CFEngine Build](https://build.cfengine.com/modules/compliance-report-os-is-vendor-supported/). Importing a report looks like this in the Mission Portal UI: [![](images/importable-compliance-reports.png)](images/importable-compliance-reports.png) This feature is also available in CFEngine 3.18.1 LTS. All new features from 3.18.1 are available in 3.19.0. We won't go over them again here, see the [3.18.1 release blog post](/blog/2021/cfengine-3-15-5-and-3-18-1-released) for details on those changes. ### Custom promise types As users can share promise types on CFEngine Build, we continue to expand functionality which enhances what you can do in modules. `cf-agent` now supports more attributes for [custom promise types](https://docs.cfengine.com/docs/master/reference-promise-types-custom.html): - [The `handle` attribute is useful for ordering promises to happen after a custom promise](https://northerntech.atlassian.net/browse/CFE-3439) - [The `depends_on` attribute can be used to make a custom promise happen after another promise](https://northerntech.atlassian.net/browse/CFE-3438) - [The `with` attribute allows you to expand function calls inside strings](https://northerntech.atlassian.net/browse/CFE-3441) ### HTTP promise type There is a new promise type, for doing HTTP requests in CFEngine, [available on CFEngine Build](https://build.cfengine.com/modules/promise-type-http/). In contrast to our `url_get()` function, this promise type supports more HTTP methods, like `POST` and `DELETE`, and can also stream responses of arbitrary size directly to a file. Once added, it looks like this in use: ```cf3 bundle agent main { http: "https://cfengine.com/images/cfengine-logo.svg" file => "/var/cfengine/cfengine-logo.svg", if => not(fileexists("/var/cfengine/cfengine-logo.svg")); } ``` Other promise types we've announced earlier this year are also available via CFEngine Build: - [`systemd`](https://build.cfengine.com/modules/promise-type-systemd) - Manage systemd services - [`ansible`](https://build.cfengine.com/modules/promise-type-ansible) - Run Ansible playbooks - [`groups`](https://build.cfengine.com/modules/promise-type-groups) - Manage local groups ### Security modules in a holiday-themed calendar In December, we decided to provide daily security hardening tips and modules as a holiday calendar. You can read a summary of the first week, with the first 7 modules, in [this blog post](/blog/2021/security-hardening-holiday-calendar-week-1/). To sign up for the rest of this calendar (day 10 to 25) use this link: [cfengine.com/calendar](/calendar) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.19.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.19/guide-latest-release-whatsnew-changelog-core.html) - [3.19.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.19/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.19.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.19/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the recently released [3.18.1](/blog/2021/cfengine-3-15-5-and-3-18-1-released), these dependencies have been updated: | CFEngine version | 3.18.1 | **3.19.0** | | :-------------------------------------------------- | :----- | :--------- | | [libcurl](https://curl.haxx.se/download.html) | 7.79.1 | **7.80.0** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.33.1 | **2.34.1** | | [PostgreSQL](https://www.postgresql.org/) | 13.4 | **13.5** | | [PHP](https://php.net/) | 8.0.12 | **8.1.0** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/download/) to go to the download pages with new packages. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Static checking of CFEngine code" url: /blog/2021/static-checking-of-cfengine-code/index.md date: "2021-12-09" -------------------------------------------------------------------------------- Software quality has been a topic and an area of interest since the dawn of software itself. And as software evolved so did the techniques and approaches to assuring its high quality. Better computers providing more computing power, bigger storage and faster communication have allowed software developers to detect issues in their code sooner and faster. And so we got from getting a syntax error after two days of waiting for the box of punch cards to go through the queue of boxes and get loaded into a computer running a compiler to getting such errors from a compiler in seconds or even in real-time from the code editor. And we got from bugs being detected by actually seeing real bugs on punch cards with machine instructions to operating systems providing bug reports with coredumps, tracebacks and lots of information helping the developers to identify the problem, tests detecting problems before the code gets into production, or compilers and tooling detecting them before the code is even executed the first time. We can afford doing things like *fuzz testing*, we have enough computational power for compilers and special tools to analyze the code and check all possible paths through it and much more. At the same time, software has become a part of almost everything we use or interact with every day and so with the incomparably greater amount of software potentially affecting our lives there is an incomparably greater amount of bugs that need to be detected and fixed or at least handled gracefully. With some software being more critical than other and with bugs ranging from minor annoyances to losses of human lives. Many things have changed in this evolution, but one rule has always been key: **The sooner a bug is detected, the cheaper it is to fix it.** ## Static checking *Static checking* or, more formally, *Static program analysis* is a set of software quality assurance techniques that analyze software without executing (running) it. Instead, they work on the source code level, with some intermediate representation or with object (machine) code. The complexity of the static checking techniques and tools ranges from basic warnings and errors provided by the compilers over special tools analyzing paths through the code and combinations of possible values of variable and parameters to very advanced tools doing formal verification. The more advanced and powerful the analysis is, the longer it takes, but it\'s generally much faster and easier to do than testing the code in a production-like environment. And unlike dynamic testing of particular pieces of code (*unit testing*) and testing of specific cases in *integration testing*, static analysis is much more generic. Often the results from static analysis are warnings and/or errors that cannot practically happen in real world due to some implicit facts or assumptions that are not expressed in the code. But even those are worth attention because those implicit assumptions should either be expressed in the code, if the programming language (or environment) allows it, or at least in comments, potentially suppressing the specific errors/warnings from the static checking tools. It is then up to the person doing *code review* (another software quality assurance technique) to check those assumptions and later if/when the execution environment for the program changes a revision of those assumptions needs to be performed. However, the assumptions need to be specified there first and the closer to the code it is the better. Thus techniques requiring or at least encouraging that can prevent bugs. On the other hand, while static program analysis is evolving and getting better and better, it often yields false positives. For this reason, all the tools provide mechanisms for suppressing some types of warnings/errors, for suppressing specific warnings/errors in specific places in the code and/or for completely disabling some types of checks. However, sometimes it is best to change the code to avoid the particular false positive because then no suppression is needed and the code often becomes easier to read and understand for developers. As described above, the very basic, easiest to do and very cheap static checking technique is to enable as many warnings from a compiler as possible and, ideally, to treat them as errors. Or in other words, for people familiar with the *GCC* compilers, use of the `-Wall`, `-Wextra` and `-Werror` flags. This sounds like a trivial thing to do and one could even ask why those are not enabled by default. Well, even the mere existence of the `-Wextra` option adding extra warnings on top of the `-Wall` option gives the answer -- those warnings break many builds when treated as errors or produce a lot of noise when treated as warnings. This is a problem especially with code that has been developed without all the warnings enabled and very often that is the case of old code because many of the warnings are new or at least providing much better results (i.e. fewer false positives) than they used to. Last but not least, some of those warnings actually trigger on code using techniques that used to be considered good/best practice at some point or are still considered good/best practice in special cases. Static program analysis has been evolving rapidly and greatly in the last couple years, new tools are appearing and the existing tools (e.g. compilers) are being enhanced in this regard. New programming languages are more and more adapting techniques from programming languages focused on critical software that make static checking easier, better or possible to extend to cover more aspects. However, much of the software (still) \"running the world\" was written years or even decades ago in languages and using techniques that didn\'t care about static analysis too much. ## Static checking of CFEngine CFEngine's origins go back to year 1993 and while most of the code was rewritten over the years, there is still some code written more than 20 years ago that is part of the packages released these days. C code, of course. The naïve approach like *\"Let\'s just add `-Wall -Wextra -Werror` everywhere and fix all the issues\"* was thus impossible. And not only because some parts of the code are quite old and weren\'t written with static checking in mind, but also because CFEngine is being built on many platforms that only provide old compilers. Good news is that static checking is almost platform-independent. Of course some parts of the code depend on some platform-specific macros and may even not be compiled on all platforms, but vast majority of the code is shared and platform-independent, much more than machine code, of course. And even though compilers are good elementary static checking tools, static program analysis working on the source code level doesn\'t require the code to be compiled or even compiling/compilable. Thus issues found by static checking on one platform are very often a problem on other platforms and very little is left uncovered. ### Container-based checking Of course every developer working on CFEngine can run the compiler with special flags on their own system. However, as mentioned above, newer versions of compilers provide better results, i.e. more warnings/errors. Thus we decided to use containers because while it\'s unfortunately not easy to **run** CFEngine in a container, it\'s very easy to **build** it in a container. And with that flexibility, we were able to choose a container image that has new versions of the tools we want to use -- *gcc* (GNU project C and C++ compiler), *Clang* (C front-end for LLVM) and *cppcheck* (a static analysis tool for C and C++ code). We decided to use *Fedora* images because Fedora is a distribution that very early adopts new versions of compilers, standard libraries, etc. To make it simple for the developers, we added a new *make* target -- [*static-check*](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/Makefile.am#L167) -- which simply runs [a script](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/tests/static-check/run.sh) that creates a new container from a base Fedora image with the required tools installed, copies all sources into it and then runs [another script](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/tests/static-check/run_checks.sh) **inside** the container that runs builds of CFEngine using *gcc* and *Clang* and then runs *cppcheck*. The builds use the `-Werror -Wall -Wextra -Wno-sign-compare` options to enable all warnings except for the *sign-compare* warning which is triggered by lines of code that are very hard to change (but we still hope to do it in the future). The *cppcheck* run uses [a list of suppressions](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/tests/static-check/cppcheck_suppressions.txt) that prevent some (currently 5) false-positives from being reported. We also experimented with the `-fanalyzer` option of *gcc*, but it was producing too many false-positives for us to be practically usable. However, many improvements have been made in the recent versions of *gcc* [^1] so we will surely give it another try soon. This simple solution has detected many issues in our code. Real bugs as well as just missing or badly expressed assumptions or just pieces of code that were correct, but too cryptic for the above mentioned tools and of course it turned out they were cryptic for human readers as well. Plus, with every new release of Fedora (and thus the tools we use), we discover and fix new issues. Having an easy way to run the checks locally is one thing. It\'s nice and handy and it takes less than 5 minutes on an average developer laptop, but we don\'t want to require all developers to do it and swear they did it before they push or merge some changes. That\'s why we also have a job in our CI system that runs the *static-check* target on all pull requests and also as part of our nightly builds. [![](./images/static-checks-GH-badge.png)](./images/static-checks-GH-badge.png) [^1]: See and ### LGTM Apart from using *gcc*, *Clang* and *cppcheck* all of which we have available locally, [we also use LGTM](https://lgtm.com/projects/g/cfengine/core/history/) -- an online code analysis platform which is available for free for open source projects. [![](./images/cfengine-core-lgtm.png)](./images/cfengine-core-lgtm.png) Even with the default configuration it has many rules that detect issues in code written in various programming languages plus it can be easily extended with custom rules. We have two custom rules in CFEngine: 1. boolean type mismatch between function type and return value and 2. missing `NULL` check for arguments. [The first one](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/.lgtm/cpp-queries/bool-type-mismatch-return.qhelp) detected, and prevents us from re-introducing, problems where a function returning `bool` values actually returns `int` values. `true` and `false` are `1` and `0`, respectively, in C so they are integer values. But the problem is that all other integer values can be treated as boolean values -- `0` meaning `false` and everything else meaning `true`. A typical problem is when the code in a function with return type `bool` returns `-1` or some other negative values to indicate an error. This is a common (**good**) practice, but then the function needs to have `int` (or some other signed integer type) as its return type and the calling code should check the return value for being less than, equal or greater than `0` instead of just doing `if (retval)` which evaluates to `true` if `retval` is `-1`. [The second rule](https://github.com/cfengine/core/blob/9c9736e55c63d1be00c67f6ddf9179ee36f05add/.lgtm/cpp-queries/missing-argument-null-check.qhelp) requires all function pointer arguments that are dereferenced in function bodies to be checked for not being `NULL`. Either the code dereferencing the arguments needs to check that the argument is not `NULL` before dereferencing it or there must be an `assert(arg != NULL)` for every argument that is dereferenced in the function body. This rule actually has two benefits -- a failure of such `assert()` stops the program in a way that makes it easy to detect where the problem is (instead of potentially reading from or writing to random bits of memory) and the `assert()` gives information to the developer using the function that the particular argument(s) cannot be `NULL`. The asserts are only effective in code compiled with debugging options and so they don\'t introduce any performance penalty in production code. The above two rules helped us to identify bugs as well as to improve our code quality and readability. The amazing thing about *LGTM* is that it can be easily used with a codebase that triggers alerts and warnings (ideally eliminating them over time) with only newly-introduced alerts and warnings being reported for open pull requests or merged commits. This greatly facilitates integration of *LGTM* into existing projects like CFEngine. [![](./images/lgtm-GH-badge.png)](./images/lgtm-GH-badge.png) ## Summary Static checking is a great software quality assurance technique which has recently gone through a great evolution and development that keeps continuing. It provides fast analysis of existing codebase and quick feedback for newly introduced code and can detect serious issues. It has also become easily available and easy to use. With CFEngine, a project the codebase of which goes back to 90s, we are showing that it can be of great help not just for new projects using modern programming languages and modern coding practices. We definitely plan to keep using static checking and even extend it in the future and we are definitely open to share our experience with it. If you have any questions related to how we do static checking, don\'t hesitate to ask, for example at [GitHub discussions](https://github.com/cfengine/core/discussions). -------------------------------------------------------------------------------- title: "Security hardening holiday calendar - Week 1" url: /blog/2021/security-hardening-holiday-calendar-week-1/index.md date: "2021-12-07" -------------------------------------------------------------------------------- This year we decided to provide security focused modules and content for the holiday season. These are parts of the security configuration we implement on our own infrastructure, based on OpenSCAP and other sources. By putting these into easy to use modules and writing about it, we hope to give our community of users something valuable: Educational and easy to understand security tips, along with configuration which can quickly be automated across your entire infrastructure, using CFEngine. Today, at the seventh day of the calendar, we will share a summary of the first week. [![CFEngine themed advent calendar design with 25 boxes, 7 of them are revealed with the same titles as the following sections](images/07.12.png)](images/07.12.png) ## Reports of all local users with sudo access (1/25) sudo enables you to run a command with elevated privileges. You should monitor and restrict who is allowed to use it: [build.cfengine.com/modules/inventory-sudoers](https://build.cfengine.com/modules/inventory-sudoers) This module provides the attribute `Users with sudo` which can be added to inventory reports: [![Tabular report containing columns for Host name, Users with sudo, OS, and Ports listening](images/inventory-sudoers.png)](images/inventory-sudoers.png) **Recommendation:** Review this list on a regular basis, and ensure all of the users really need `sudo` access. Remove `sudo` access where possible by [editing the `/etc/sudoers` file](https://www.digitalocean.com/community/tutorials/how-to-edit-the-sudoers-file). ## Use SHA-512 to hash passwords by default (2/25) MD5, DES, and other older algorithms are no longer secure, and shouldn't be used for any cryptographic purposes. Use a better default, like SHA-512: [build.cfengine.com/modules/default-encrypt-method-sha512](https://build.cfengine.com/modules/default-encrypt-method-sha512) Running CFEngine with this module in your policy set on a system which uses MD5 looks like this: ``` $ cat /etc/login.defs | grep '^ENCRYPT_METHOD' ENCRYPT_METHOD MD5 $ cf-agent -KI info: Replaced pattern '^\s*(ENCRYPT_METHOD\s+(?!SHA512$).*|ENCRYPT_METHOD)$' in '/etc/login.defs' info: replace_patterns promise '^\s*(ENCRYPT_METHOD\s+(?!SHA512$).*|ENCRYPT_METHOD)$' repaired info: Edited file '/etc/login.defs' $ cat /etc/login.defs | grep '^ENCRYPT_METHOD' ENCRYPT_METHOD SHA512 ``` **Recommendation:** Don't use MD5 or other insecure hashing algorithms. Use this module to enforce a strong default across your systems. Look into what other places you might be relying on outdated cryptographic algorithms and how you can eliminate them. (See for example [FIPS mode](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/chap-federal_standards_and_regulations)). ## Disable root login over SSH (3/25) SSH allows you to securely log into a remote machine and run commands there (in a shell). Attackers will commonly try to log in as root, since the root user exists almost everywhere, and generally has a lot of access. It is a good idea to disable this completely. [build.cfengine.com/modules/ssh-permit-root-login-no](https://build.cfengine.com/modules/ssh-permit-root-login-no) This module ensures that the SSH daemon (sshd) does not permit logging in with the `root` user account. There are several reasons why allowing this could be problematic, from a security perspective: - It is a good target for attackers, to try logging in as `root`: - The `root` user exists on almost all UNIX-like systems. - The `root` user normally has a lot of privileges (access), to read data or change the system. - Sometimes the `root` user has a default / insecure password. - The `root` user is not personal. - When people log in with root you cannot know who did what, or effectively restrict access based on person / role. **Recommendation:** Ensure logging in with SSH as `root` is not possible. This can be achieved by adding this module to your policy set and deploying it to all your infrastructure (via the policy server). Due to the reasons above, this is beneficial, even if you allow users to *become* root after they log in. ## Uninstall telnet-server package (4/25) Telnet is a protocol for providing remote terminal access to hosts over network. It is generally considered obsolete and dangerous. The first step to make sure it's not used, is to make sure it's not installed. [build.cfengine.com/modules/telnet-server-not-installed](https://build.cfengine.com/modules/telnet-server-not-installed) Example of a `cf-agent` run on a host that has the `telnet-server` package installed: ``` [root@hub]# cf-agent -KI info: Successfully removed package 'telnet-server' ``` **Recommendation:** Ensure it is not possible to use telnet to log into your machines. This can be achieved by using this module, which uninstalls the telnet server if it is installed. ## Reports of kernel boot parameters (5/25) What parameters are used to boot the Linux kernel can affect performance, functionality and security. Seeing the configuration of all your servers allows you to detect misconfigurations and fix them. [build.cfengine.com/modules/inventory-kernel-boot-params](https://build.cfengine.com/modules/inventory-kernel-boot-params) This module makes CFEngine report a new inventory attribute, `Kernel Boot Params`: [![Screenshot showing the kernel boot params of a host in CFEngine Enterprise UI](images/kernel-boot-params.png)](images/kernel-boot-params.png) **Recommendation:** Use this module to get up to date reports of the Linux kernel boot parameters used in your infrastructure. Review them and look for misconfigurations or outliers. ## Security checks with Lynis (6/25) Lynis is a security tool for hosts running Linux, macOS, or other Unix-based operating systems. It performs an extensive health scan of your host to support system hardening and compliance testing. [build.cfengine.com/modules/lynis](https://build.cfengine.com/modules/lynis) This module will automate installation, running and reporting of Lynis results, on all UNIX-like hosts you have in CFEngine. If used from the command line, you can add the `--info` option to see all the results from Lynis: ``` $ cf-agent --no-lock --info R: ----- Warnings ----- R: NETW-2704 -- Nameserver 192.168.1.1 does not respond R: PKGS-7392 -- Found one or more vulnerable packages. R: NETW-2705 -- Couldn't find 2 responsive nameservers R: ----- Suggestions ----- R: NETW-2704 -- Check connection to this nameserver and make sure no outbound DNS queries are blocked (port 53 UDP and TCP). R: KRNL-6000 -- One or more sysctl values differ from the scan profile and could be tweaked R: HRDN-7222 -- Harden compilers like restricting access to root user only R: PKGS-7392 -- Update your system with apt-get update, apt-get upgrade, apt-get dist-upgrade and/or unattended-upgrades R: ACCT-9626 -- Enable sysstat to collect accounting (no results) R: MACF-6208 -- Check output of aa-status ... ``` **Recommendation:** Run Lynis on all UNIX-like hosts in your infrastructure. Investigate the results and fix as much as possible. Learn more about Lynis on [their official website](https://cisofy.com/lynis/). ## SSH authentication attempts (7/25) Attackers can try to log in to SSH many times, testing different passwords. To stop this, you can configure the SSH daemon with a maximum number of attempts: [build.cfengine.com/modules/ssh-max-auth-tries](https://build.cfengine.com/modules/ssh-max-auth-tries) This module looks at the SSH daemon configuration in `/etc/ssh/sshd_config`. If the `MaxAuthTries` parameter is not already set to the correct value (3, by default), the necessary changes will be made: * Backup the config * Edit the config * Replace old config with new config * Restart SSH daemon **Recommendation:** Stop brute-force attacks which try to authenticate many times with SSH by limiting the number of possible attempts. You can achieve this by adding the module to your policy set, and it will be enforced on all hosts you have in CFEngine. ## Subscribe to the calendar This is just the beginning, we have 18 days left! To sign up and receive daily security hardening content through Dec. 25th, [use this link](/calendar). -------------------------------------------------------------------------------- title: "CFEngine 3.15.5 and 3.18.1 released" url: /blog/2021/cfengine-3-15-5-and-3-18-1-released/index.md date: "2021-11-29" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.15.5 and 3.18.1\! These releases mainly contain bug fixes and dependency updates. ## What's new In addition to bug fixes, some smaller features and improvements were added to 3.18.1. These additions are focused on the Mission Portal UI and API, and were added due to user feedback. ### Enterprise APIs The new API endpoint - [`/api/hosts/deleted`](https://docs.cfengine.com/docs/3.18/reference-enterprise-api-ref-host.html#get-deleted-hosts-list) can be used to list previously deleted hosts. As mentioned in [a previous blog post](/blog/2021/cfengine-inventory-to-ansible/), the [Ansible compatible Host API](https://docs.cfengine.com/docs/3.18/reference-enterprise-api-ref-host.html#hosts-list-grouped-by-hard-classes) got another option: `inventoryFile`. This option allows you to control whether to use the format expected by Ansible when reading the output of a script (default) or a file (`inventoryFile=true`). ### Visual / UX improvements Various buttons, icons, logos and images in Mission Portal were improved. As an example, here are the new "toast" notifications: [![](images/notifications.png)](images/notifications.png) URLs inside reports are now clickable links: [![](images/urls.png)](images/urls.png) There are many more for you to discover once you upgrade. If you are interested in these UI/UX changes, check out [our blog posts on design](/tags/design/). ### Importable Compliance Reports Compliance reports can now be stored and managed as JSON files. You can click Export within a compliance report, and choose the new "Report definition - json" type. This will give you a JSON file containing all the information needed to create the report, including the checks. (The report definition does not include data about your hosts, only the report). The JSON definition can be edited / managed outside of Mission Portal, and will overwrite as necessary when importing. This functionality enables sharing reports between hubs / organizations, and will allow us to publish compliance reports as [CFEngine Build](https://build.cfengine.com/) modules in the future. Importing a report looks like this: [![](images/importable-compliance-reports.png)](images/importable-compliance-reports.png) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.15.5 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [3.15.5 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.15.5 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.18.1 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.1 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.1 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. | CFEngine version | 3.15.4 | **3.15.5** | 3.18.0 | **3.18.1** | | :-------------------------------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | - | - | 3.7 | **3.8** | | [libcurl](https://curl.haxx.se/download.html) | 7.75.0 | **7.79.1** | 7.77.0 | **7.79.1** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.4.58 | 2.4.58 | 2.5.5 | **2.6.0** | | [OpenSSL](https://openssl.org/) | 1.1.1k | **1.1.1l** | 1.1.1l | **1.1.1l** | | [PCRE](https://www.pcre.org/) | 8.44 | **8.45** | 8.44 | **8.45** | | [Apache](https://httpd.apache.org/) | 2.4.47 | **2.4.51** | 2.4.48 | **2.4.51** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.31.1 | **2.33.1** | 2.32.0 | **2.33.1** | | [PHP](https://php.net/) | 7.4.19 | **7.4.25** | 8.0.7 | **8.0.12** | | [PostgreSQL](https://www.postgresql.org/) | 12.7 | **12.8** | 13.3 | **13.4** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](/download/) to go to the download pages with new packages. Since version 3.18.1 is currently the latest available LTS release, it will be the default version picked by [cf-remote](https://pypi.org/project/cf-remote/) and the [Quick install script](/download/). If you are using `cf-remote`, you can target versions like this: ```command cf-remote --version 3.15.5 list ``` ```command cf-remote --version 3.18.1 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 7 - Creating a module with CFEngine Build" url: /blog/2021/the-agent-is-in-episode-7-creating-a-module-with-cfengine-build/index.md date: "2021-11-26" -------------------------------------------------------------------------------- Join us as we embark on an adventure to create and publish a new CFEngine Build module. Nick (Doer of Things) demonstrates he knows the proper offerings the gods require by writing and publishing a new CFEngine Build Module from scratch, **live, with no safety net**! ## Video {#video} The video recording is available on YouTube: {{< youtube zqnCgJxvGb8 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [CFEngine Build](https://build.cfengine.com/modules/library-parsed-local-users/) - [cfbs](https://github.com/cfengine/cfbs) - [library-parsed-local-users](https://build.cfengine.com/modules/library-parsed-local-users/) - [inventory-non-root-users-non-root-has-uid-0](https://build.cfengine.com/modules/inventory-local-users-non-root-has-uid-0/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Solving specific use cases with CFEngine policy and providing reusable modules" url: /blog/2021/solving-specific-use-cases-with-cfengine-policy-and-providing-reusable-modules/index.md date: "2021-11-25" -------------------------------------------------------------------------------- With the release of [build.cfengine.com](https://build.cfengine.com/), I have been working to migrate some of our own security related policy into modules of their own. CFEngine Build and the `cfbs` tooling allows us to organize policy into modules, which are easy to update independently and share with other users. Let's take the scenic route and look at what life is like with `cfbs`. One of our security policies requires that the password hashing algorithm in `/etc/login.defs` is set to `SHA512`. [![Compliance check showing /etc/login.defs ENCRYPT_METHOD=SHA512 not passing.](images/compliance-check-non-compliant-etc-login-defs-encrypt-method-sha512.png)](images/compliance-check-non-compliant-etc-login-defs-encrypt-method-sha512.png) Traditionally, with CFEngine we would simply promise the desired state: ```cf3 bundle agent _etc_login_defs_ENCRYPT_METHOD { meta: "tags" slist => { "autorun" }; vars: "_config[ENCRYPT_METHOD]" string => "SHA512"; "_etc_login_defs" string => "/etc/login.defs"; files: "$(_etc_login_defs)" edit_line => set_line_based( "$(this.namespace):$(this.bundle)._config", " ", "\s+", ".*", "^\s*(#.*|$)"); } ``` When the policy runs, it simply fixes the configuration issue. For example: ```example [root@hub]# grep ENCRYPT_METHOD /etc/login.defs ENCRYPT_METHOD MD5 [root@hub]# cf-agent -KI info: Replaced pattern '^\s*(ENCRYPT_METHOD\s+(?!SHA512$).*|ENCRYPT_METHOD)$' in '/etc/login.defs' info: replace_patterns promise '^\s*(ENCRYPT_METHOD\s+(?!SHA512$).*|ENCRYPT_METHOD)$' repaired info: Edited file '/etc/login.defs' ``` But, I don't want to blindly apply this policy to the infrastructure. I could warn instead of fixing the state, but that will only tell me where it's not correct and it won't give me any information about what it's currently set to. In order to get an overview of the current configuration before making promises about the state I want, I need to *inventory* the configuration. In [CFEngine Enterprise reporting](https://docs.cfengine.com/docs/3.18/enterprise-cfengine-guide-reporting.html), inventory is achieved by tagging variables and/or classes with metadata. Specifically, tagging a variable or class with `inventory` and `attribute_name=My Name` will result in a new inventory attribute for `My Name`. So, we need to parse the file into one or more variables that we can tag for *inventory* so that we can report about the currently configured state. [![Inventory report showing /etc/login.defs current state.](images/inventory-report-etc-login-defs.png)](/blog/2021/images/inventory-report-etc-login-defs.png) The following policy generates the necessary data, we won't dive into the details here, but the result is a new inventory attribute, `/etc/login/.defs` holding the *KEY=VALUE* pairs parsed from the configuration file as seen in the preceding image. ```cf3 bundle agent inventory_etc_login_defs { vars: "_etc_login_defs_path" string => "/etc/login.defs"; # A KEY VALUE formatted file "_etc_login_defs" data => data_readstringarrayidx( "$(_etc_login_defs_path)", # File "#[^\n]*", # Comment "\s+", # Split inf, # Max entries inf), # Max bytes if => fileexists( "$(_etc_login_defs_path)" ); # We need to iterate over the data structure created by # data_readstringarrayidx() so that we can build tag each item. "_etc_login_defs_idx" slist => getindices( _etc_login_defs ); # Here we use the index to iterate and build up an associative array of # KEY=Values We use an associative array so that we can stay under the # data limit reporting a single variable "etc_login_defs[$(_etc_login_defs_idx)]" string => "$(_etc_login_defs[$(_etc_login_defs_idx)][0])=$(_etc_login_defs[$(_etc_login_defs_idx)][1])", meta => { "inventory", "attribute_name=$(_etc_login_defs_path)" }; } bundle agent __main__ { methods: "inventory_etc_login_defs"; } ``` Using `cfbs` you no longer have to copy and paste this policy to integrate it into your policy set. Let's use `cfbs` to build a policy set that includes this inventory leveraging the [inventory-etc-login-defs module](https://build.cfengine.com/modules/inventory-etc-login-defs/). First, let's bring up an environment to play with. Here I use the [CFEngine Enterprise Vagrant Environment](https://docs.cfengine.com/docs/3.18/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Note: Alternatively you can also use `cf-remote` to provision hosts on AWS or GCP. Once your hub is up, log in as `root` and get `cfbs` installed. First, install pip and get it upgraded, `yum -y install python3-pip; pip3 install pip --upgrade`. Next, install `cfbs`, and put it in PATH for ease of use `pip3 install cfbs; export PATH=$PATH:/usr/local/bin`. Now that we have `cfbs` let's make a project directory, switch into it, and initialize it, `mkdir -p /root/cfbs-project; cd /root/cfbs-project; cfbs init`. ```example [root@hub ~]# mkdir -p /root/cfbs-project; cd /root/cfbs-project; cfbs init Initialized - edit name and description cfbs.json To add your first module, type: cfbs add masterfiles ``` Let's do as instructed and run `cfbs add masterfiles`. ```example [root@hub cfbs-project]# cfbs add masterfiles Added module: masterfiles ``` [Currently](https://northerntech.atlassian.net/browse/CFE-3872), you need `autotools` to build masterfiles from source, so let's install that as well: ``` [root@hub ~]# yum -y install automake ``` Now, let's add the `inventory-etc-login-defs` module. ```example [root@hub cfbs-project]# cfbs add inventory-etc-login-defs Added module: inventory-etc-login-defs ``` We can see that `cfbs` populated `cfbs.json` with the information needed to build the project. ```example [root@hub cfbs-project]# cat cfbs.json { "name": "Example", "type": "policy-set", "description": "Example description", "build": [ { "name": "masterfiles", "description": "Official CFEngine Masterfiles Policy Framework (MPF)", "tags": ["supported", "base"], "repo": "https://github.com/cfengine/masterfiles", "by": "https://github.com/cfengine", "version": "0.1.1", "commit": "5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f", "steps": [ "run ./autogen.sh", "delete ./autogen.sh", "run ./cfbs/cleanup.sh", "delete ./cfbs/cleanup.sh", "copy ./ ./" ], "added_by": "cfbs add" }, { "name": "inventory-etc-login-defs", "description": "Inventory useful bits from /etc/login.defs", "tags": ["supported", "inventory"], "repo": "https://github.com/nickanderson/cfengine-inventory-etc-login-defs", "by": "https://github.com/nickanderson", "version": "0.0.1", "commit": "52f659510ddd3bc65e804306a1bff17d6c6f4299", "steps": [ "copy ./inventory-etc-login-defs.cf services/inventory-etc-login-defs/inventory-etc-login-defs.cf", "json cfbs/def.json def.json" ], "added_by": "cfbs add" } ] } ``` Now, let's build and install! ```example [root@hub cfbs-project]# cfbs build Modules: 001 masterfiles @ 5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f (Downloaded) 002 inventory-etc-login-defs @ 52f659510ddd3bc65e804306a1bff17d6c6f4299 (Downloaded) Steps: 001 masterfiles : run './autogen.sh' 001 masterfiles : delete './autogen.sh' 001 masterfiles : run './cfbs/cleanup.sh' 001 masterfiles : delete './cfbs/cleanup.sh' 001 masterfiles : copy './' 'masterfiles/' 002 inventory-etc-login-defs : copy './inventory-etc-login-defs.cf' 'masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf' 002 inventory-etc-login-defs : json 'cfbs/def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz [root@hub cfbs-project]# cfbs install Installed to /var/cfengine/masterfiles ``` Within a few minutes we should be able to find `/etc/login.defs` reported in inventory. By querying the API: ```example [root@hub cfbs-project]# curl --silent --insecure --user $AUTHUSER:$PASSWORD \ --request POST \ https://$HUB/api/inventory \ -H 'content-type: application/json' \ -d '{ "sort":"Host name", "select":[ "Host name", "/etc/login.defs" ]}' ``` ```json { "data": [ { "header": [ { "columnName": "Host name", "columnType": "STRING" }, { "columnName": "\/etc\/login.defs", "columnType": "STRING" } ], "queryTimeMs": 17, "rowCount": 1, "rows": [ [ "hub.example.com", "UID_MIN=1000, UID_MAX=60000, UMASK=077, USERGROUPS_ENAB=yes, SYS_GID_MAX=999, CREATE_HOME=yes, ENCRYPT_METHOD=MD5, MD5_CRYPT_ENAB=yes, MAIL_DIR=\/var\/spool\/mail, GID_MAX=60000, SYS_GID_MIN=201, SYS_UID_MAX=999, GID_MIN=1000, PASS_MIN_LEN=5, PASS_MAX_DAYS=99999, PASS_MIN_DAYS=0, PASS_WARN_AGE=7, SYS_UID_MIN=201" ] ] } ], "meta": { "count": 1, "page": 1, "timestamp": 1636491107, "total": 1 } } ``` Or by looking in Mission Portal: [![Mission Portal Inventory Report shows 2 hosts bootstrapped and reporting.](images/show-all-inventory-attributes-etc-login-defs.png)](/blog/2021/images/show-all-inventory-attributes-etc-login-defs.png) From here we could continue to build a compliance report checking that `/etc/login.defs` matches `ENCRYPT_METHOD=SHA512`, but this post is about `cfbs`. [![Compliance check showing /etc/login.defs ENCRYPT_METHOD=SHA512 not passing.](images/compliance-check-non-compliant-etc-login-defs-encrypt-method-sha512.png)](/blog/2021/images/compliance-check-non-compliant-etc-login-defs-encrypt-method-sha512.png) Here is where things get *really* great! While authoring this post, I noticed that the comment regex in the [policy could be improved](https://github.com/nickanderson/cfengine-inventory-etc-login-defs/commit/8f4747e4856737aa44fd96b9b8d38d941afec9f3). So, I updated the module, and now I can quickly get that change integrated into my `cfbs` project. First, let's initialize our `cfbs` project as a git repository. This will make it very easy to see the changes `cfbs` is making to the project. ```example [root@hub cfbs-project]# git init Initialized empty Git repository in /root/cfbs-project/.git/ [root@hub cfbs-project]# git add cfbs.json [root@hub cfbs-project]# git commit -m "Initialized cfbs project with masterfiles and inventory-etc-login-defs" ``` Okay, now, let's run `cfbs update` to update all of our modules to the latest versions available in the index. We can use `git diff` to see the changes made by `cfbs`. ```example [root@hub cfbs-project]# cfbs update [root@hub cfbs-project]# git diff diff --git a/cfbs.json b/cfbs.json index 1540e08..bc0a03f 100644 --- a/cfbs.json +++ b/cfbs.json @@ -27,8 +27,8 @@ "tags": ["supported", "inventory"], "repo": "https://github.com/nickanderson/cfengine-inventory-etc-login-defs", "by": "https://github.com/nickanderson", - "version": "0.0.1", - "commit": "52f659510ddd3bc65e804306a1bff17d6c6f4299", + "version": "0.0.3", + "commit": "8f4747e4856737aa44fd96b9b8d38d941afec9f3", "steps": [ "copy ./inventory-etc-login-defs.cf services/inventory-etc-login-defs/inventory-etc-login-defs.cf", "json cfbs/def.json def.json" ``` Using `cfbs status` we can see that the latest `inventory-etc-login-defs` has yet to be downloaded. ```example [root@hub cfbs-project]# cfbs status Name: Example Description: Example description File: cfbs.json Modules: 001 masterfiles @ 5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f (Downloaded) 002 inventory-etc-login-defs @ 8f4747e4856737aa44fd96b9b8d38d941afec9f3 (Not downloaded) ``` Let's build the new policy set. ```example [root@hub cfbs-project]# cfbs build Modules: 001 masterfiles @ 5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f (Downloaded) 002 inventory-etc-login-defs @ 8f4747e4856737aa44fd96b9b8d38d941afec9f3 (Downloaded) Steps: 001 masterfiles : run './autogen.sh' 001 masterfiles : delete './autogen.sh' 001 masterfiles : run './cfbs/cleanup.sh' 001 masterfiles : delete './cfbs/cleanup.sh' 001 masterfiles : copy './' 'masterfiles/' 002 inventory-etc-login-defs : copy './inventory-etc-login-defs.cf' 'masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf' 002 inventory-etc-login-defs : json 'cfbs/def.json' 'masterfiles/def.json' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz ``` We can see the diff of the policy file does indeed show the change in comment regex. ```example [root@hub cfbs-project]# diff -u /var/cfengine/masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf out/masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf --- /var/cfengine/masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf 2021-11-09 20:38:52.342254892 +0000 +++ out/masterfiles/services/inventory-etc-login-defs/inventory-etc-login-defs.cf 2021-11-09 21:58:45.229784693 +0000 @@ -13,7 +13,7 @@ "_etc_login_defs" data => data_readstringarrayidx( "$(_etc_login_defs_path)", # File - "#[^\n]*", # Comment + "\s*#[^\n]*", # Comment "\s+", # Split inf, # Max entries inf), # Max bytes ``` Let's go ahead and `cfbs install` to deploy the updated policy set. ```example [root@hub cfbs-project]# cfbs install Installed to /var/cfengine/masterfiles ``` **Tip:** `cfbs install` requires elevated privileges to write in `/var/cfengine/masterfiles`, execute as `root` user or use `sudo` to elevate privileges. All the other `cfbs` commands only edit files inside current working directory and your home directory, and don't need elevated privileges. Throughout this post I've been using the `root` user for convenience, but this is inside a "throw-away" virtual machine. If you are running this on your development machine or production infrastructure, you should only use the `root` user when necessary. I hope this kindles the same excitement in you as it has for me. This is definitely going to make it easier to manage a policy set, especially when considering working with various policy authors, and that's without even considering the other kinds of content already in `cfbs`, like [custom promise types](https://build.cfengine.com/modules/?tag=promise-type) or content that is on the horizon like compliance reports. -------------------------------------------------------------------------------- title: "How we designed CFEngine Build, a complete journey" url: /blog/2021/how-we-design-cfengine-build-a-complete-journey/index.md date: "2021-11-19" -------------------------------------------------------------------------------- Have you ever wondered how a site was designed and how the ideas were conceptualized into the final result? If your answer is yes, you are in the right place! In this post, I will show you our journey to create our latest website, [CFEngine Build](https://build.cfengine.com/). From start to finish, how did we do the design and make the design decisions? So without further delay, let's jump straight in! ![CFEngine build logo](images/cfengine-build-logo.gif "CFEngine build logo") ## Branding and logo To have a logo that matches our objective for the website, we first need to reflect on our goal for the website. Our goal is clear: create a place for users to find and share reusable modules for CFEngine. With that in mind, we decided on the name **CFEngine Build**. The name represents the intention of our goal, which is to allow users to build and assemble the modules that meet their needs. What's more, we wanted to add a block shape icon next to the name to enhance the image of building blocks and notably differentiate it from the original CFEngine icon. This animation shows some of the ideas and experimentation we went through when determining how the block should look: ![Ideas for CFEngine build logos](images/cfengine-build-logos.gif "Ideas for CFEngine build logos") ## User research Through customer meetings and user surveys, we knew that this kind of website was highly requested among our users. To better understand their needs and challenges, we conducted a series of user interviews. We asked about their use cases and how they use online resources for solving problems. Questions like: * What are the most common online resources you use for your work? * What is the hardest part of using those resources? * What are the goals you try to achieve when you use them? After the interviews, I summarized the data and concluded the common user patterns based on participants' answers. For example, users heavily rely on keywords to find answers, they also prefer reliable/trustworthy sources and will not run any random code on their machines, and generally want to improve their current way of doing things. These common patterns help me to develop a design question as follows: > _How might we provide an informative, useful and robust site for our visitors?_ The question then further guided me to create the initial design strategies. To be more specific, I developed the following three user stories that include the features and functionalities that hope to answer the question. 1. As a user, I want to use the search bar or apply multiple tags to find the most relevant modules (based on keyword and tag matching) which meet my needs. 2. As a user, I want to bookmark some of the modules so I can easily find them (on the home page or using the drop-down menu) and check updates when I return to the website. 3. As a user, I want to be able to comment and to like other people's comments so I can share my ideas and brainstorm with others. ## From concept to wireframe With the user stories in mind, I created a series of user flows in a wireframe format. It conceptualizes the ideas into the design and provides a general understanding of when and where to put certain features. The goal is to enable users to effectively find what they are looking for, by focusing on their needs and ease-of-use in the design process. ![CFEngine build wireframe](images/cfengine-build-wireframes.png "CFEngine build wireframe") Following are two examples of how we design the features based on users' needs. Take featured modules as an example. It provides users quick access to the featured modules and allows them to see what types of modules are available at a glance. ![](images/cfengine-build-concept-example.png) ## Final design We added CFEngine brand colors and the font into the design. The page has ample white space because we want the page to be simple and uncluttered and deliver the information and features that our users would enjoy and appreciate. ![The final design version for CFEngine build](images/cfengine-build-final-design.png "The final design version for CFEngine build") ## What's next? We have launched the first iteration of CFEngine Build and are planning to continuously add more features which would benefit our users. If you have any feedback or thoughts, please kindly let us know in the community discussion in the below link: [https://github.com/cfengine/core/discussions/4846](https://github.com/cfengine/core/discussions/4846 "https://github.com/cfengine/core/discussions/4846") -------------------------------------------------------------------------------- title: "Announcing CFEngine Build" url: /blog/2021/cfengine-build-launched/index.md date: "2021-11-01" -------------------------------------------------------------------------------- Earlier this year, [we hinted at what we were working on](/blog/2021/collaboration-and-sharing/) - a place for users to find and share reusable modules for CFEngine. Today, the CFEngine team is pleased to announce the launch of [CFEngine Build](https://build.cfengine.com/): [![](./images/cfengine-build.png)](https://build.cfengine.com/) The new website, [build.cfengine.com](https://build.cfengine.com), allows you to browse for modules, and gives you information about how to use each one of them. When you've found the module you were looking for, it can be downloaded and built using the command line tooling. ## Downloading and using modules The new command line tool, [`cfbs`](https://pypi.org/project/cfbs/) (CFEngine Build System) allows you to download multiple modules, update them, and build them (combine them into a policy set). Once built, you can deploy your policy set to your CFEngine hub (policy server). First, install the tooling with pip: ``` $ pip install cfbs ``` (Depending on which platform and python version you are on, you may have to use `pip3` instead of `pip`, and possibly add `sudo` to the beginning). To get started, use the `init` command in a fresh folder: ``` $ mkdir my_policy $ cd my_policy $ cfbs init Initialized - edit name and description cfbs.json To add your first module, type: cfbs add masterfiles ``` This created a small `cfbs.json` project file. A lot of functionality in CFEngine relies on our default policy set, the Masterfiles Policy Framework (MPF). For this reason, you should add it as the first module in your build, as a base: ``` $ cfbs add masterfiles ``` (At this point you could build and deploy the policy set, but it doesn't do much yet, it's just the default policy you are used to from our packages). In addition to the website, you can also find modules from the command line: ``` $ cfbs search promise-type-ansible (ansible) - Promise type to manage systemd services autorun - Enable autorun functionality library-for-promise-types-in-bash (bash-lib) - Library enabling promise types implemented in bash [...] ``` We can add some of them: ``` $ cfbs add autorun inventory-sudoers git systemd groups git is an alias for promise-type-git systemd is an alias for promise-type-systemd groups is an alias for promise-type-groups Added module: library-for-promise-types-in-python (Dependency of promise-type-git) Added module: autorun Added module: inventory-sudoers Added module: promise-type-git Added module: promise-type-systemd Added module: promise-type-groups ``` What modules you add is entirely up to you. In our example above, we have achieved the following: 1. Enabled `autorun` functionality 2. Started tracking which users have sudo access (visible in Mission Portal) 3. Added 3 new promise types: `git`, `systemd`, and `groups` * Each of them is ready to be used in policy To complete the example, let's create a simple policy file, `clone_lynis.cf`: ```cfengine3 bundle agent clone_lynis { meta: "tags" slist => { "autorun" }; git: "/opt/lynis" version => "master", repository => "https://github.com/CISOfy/lynis.git"; } ``` Here, we use the new `git` promise type, to clone `lynis`. Let's add it using `cfbs`: ``` $ cfbs add clone_lynis.cf Added module: ./clone_lynis.cf ``` Finally, let's build our new policy set: ``` $ cfbs build Modules: 001 masterfiles @ 5c7dc5b43088e259a94de4e5a9f17c0ce9781a0f (Downloaded) 002 library-for-promise-types-in-python @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 003 autorun @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 004 inventory-sudoers @ 01fe2d60ecea4e2e0aefbbd9b4bc9ad570425f9f (Downloaded) 005 promise-type-git @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 006 promise-type-systemd @ c3b7329b240cf7ad062a0a64ee8b607af2cb912a (Downloaded) 007 promise-type-groups @ 087a2fd81e1bbaf241dfa7bf39013efd9d8d348f (Downloaded) 008 ./clone_lynis.cf @ local (Copied) Steps: 001 masterfiles : run './autogen.sh' 001 masterfiles : delete './autogen.sh' 001 masterfiles : run './cfbs/cleanup.sh' 001 masterfiles : delete './cfbs/cleanup.sh' 001 masterfiles : copy './' 'masterfiles/' 002 library-for-promise-types-in-python : copy 'cfengine.py' 'masterfiles/modules/promises/' 003 autorun : json 'def.json' 'masterfiles/def.json' 004 inventory-sudoers : copy './policy/main.cf' 'masterfiles/services/inventory-sudoers/main.cf' 004 inventory-sudoers : json './cfbs/def.json' 'masterfiles/def.json' 005 promise-type-git : copy 'git.py' 'masterfiles/modules/promises/' 005 promise-type-git : append 'enable.cf' 'masterfiles/services/init.cf' 006 promise-type-systemd : copy 'systemd.py' 'masterfiles/modules/promises/' 006 promise-type-systemd : append 'enable.cf' 'masterfiles/services/init.cf' 007 promise-type-groups : copy 'groups.py' 'masterfiles/modules/promises/' 007 promise-type-groups : append 'enable.cf' 'masterfiles/services/init.cf' 008 ./clone_lynis.cf : copy './clone_lynis.cf' 'masterfiles/services/autorun/clone_lynis.cf' Generating tarball... Build complete, ready to deploy 🐿 -> Directory: out/masterfiles -> Tarball: out/masterfiles.tgz To install on this machine: cfbs install To deploy on remote hub(s): cf-remote deploy --hub hub out/masterfiles.tgz ``` As the last 2 lines of output indicate, we now have a policy set we can deploy. I already have a CFEngine hub in cf-remote, named `hub`, so I will deploy it there: ``` $ cf-remote deploy --hub hub out/masterfiles.tgz Deploying to: ubuntu@54.246.14.140 OS : ubuntu (debian) Architecture : x86_64 CFEngine : 3.18.0 (Enterprise) Policy server : None Binaries : dpkg, apt Copying: '/Users/olehermanse/mypolicy/out/masterfiles.tgz' to 'ubuntu@54.246.14.140' Renaming 'masterfiles.tgz' -> 'masterfiles.tgz' on 'ubuntu@54.246.14.140' Running: 'systemctl stop cfengine3 && rm -rf /var/cfengine/masterfiles && mv masterfiles /var/cfengine/masterfiles && systemctl start cfengine3 && cf-agent -Kf update.cf && cf-agent -K' ``` I can log into the machine to see if my policy worked: ``` $ ssh ubuntu@54.246.14.140 $ sudo ls /opt/lynis/ CHANGELOG.md FAQ README db include CODE_OF_CONDUCT.md HAPPY_USERS.md README.md default.prf lynis CONTRIBUTING.md INSTALL SECURITY.md developer.prf lynis.8 CONTRIBUTORS.md LICENSE TODO.md extras plugins ``` Indeed it worked. I can also see the new inventory attribute in Mission Portal: [![](./images/sudoers.png)](./images/sudoers.png) ## The future CFEngine follows a 6-month release schedule, with 18 months between each new LTS series. This is great for stability and predictability, but it does unfortunately mean that the time until a new feature arrives can be long. With CFEngine Build, we can continuously publish new modules and functionality in the tooling, and users can start using it the same day. It does not change what is running on your servers - they will still have a policy set, just like before (located in `/var/cfengine/masterfiles` on the hub). The policy set is built and managed using CFEngine Build, but you choose when to deploy it, and you can test it first. CFEngine Build enables you to use new policy and modules, at ease, when you want to. Without spoiling everything we have planned, here are some things you can look forward to in the near future: * Functionality and tutorials for migrating existing policy sets to CFEngine Build * The ability to share Mission Portal reports as modules * New promise types for HTTP requests and firewall management ## Open source, for all users of CFEngine Every part of this is open source; [the command line tool](https://github.com/cfengine/cfbs), [the website](https://github.com/cfengine/build-website), [the modules](https://github.com/cfengine/modules), etc. We want everyone to be able to use CFEngine Build and contribute, whether you're a CFEngine Enterprise customer, a long time Community user, or someone who just got started with the [Free 25 Enterprise Licenses](https://cfengine.com/download/). The website is built with [hugo](https://gohugo.io/) and [Less](https://lesscss.org/), and served using [nginx](https://www.nginx.com/). This is the same technology stack we use for [cfengine.com](https://cfengine.com/). The command line tool is written in [Python](https://www.python.org/), and released on [pypi](https://pypi.org/project/cfbs/). (That's very similar to [how we develop cf-remote](/videos/?videoID=U-aIPTS580s).) It uses [git](https://git-scm.com/), [requests](https://python-requests.org), [rsync](https://linux.die.net/man/1/rsync) internally, to clone repos, interact with APIs and copy files around for the build process. We'd like to take this opportunity to thank the authors and maintainers of all the mentioned open source software. ## Contributing Before starting to write your own modules, we recommend getting familiar with the tooling and how things work. Once you feel like you understand the basics, there is a guide for how to author your own modules: https://github.com/cfengine/build-index/blob/master/CONTRIBUTING.md ## Getting help If you have questions, suggestions or need help, please use our GitHub Discussions: https://github.com/cfengine/core/discussions -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 6 - Running CFEngine on IoT (Part 2)" url: /blog/2021/the-agent-is-in-episode-6-running-cfengine-on-iot-part-2/index.md date: "2021-10-28" -------------------------------------------------------------------------------- Still interested in running CFEngine on IoT? Craig (Digger) shows building CFEngine Enterprise for Yocto and deploys a Raspberry Pi Zero with a sensor to measure the height of Nick's (Doer of Things) desk. ## Video {#video} The video recording is available on YouTube: {{< youtube QFhSi5qFLlI >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Yocto](https://www.yoctoproject.org/) - [Raspberry Pi Zero](https://www.raspberrypi.com/products/raspberry-pi-zero/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CVE-2021-38379 & CVE-2021-36756 - Exported report permissions and certificate checking in Federated Reporting" url: /blog/2021/cve-2021-38379-and-cve-2021-36756/index.md date: "2021-10-27" -------------------------------------------------------------------------------- The CFEngine engineering team has recently discovered two security issues in the *CFEngine Enterprise* product: * CVE-2021-38379 - Publicly available exported reports * CVE-2021-36756 - Certificate not checked in Federated Reporting While the latter one (`CVE-2021-36756`) only affects *CFEngine Enterprise* deployments using the *Federated Reporting* functionality, the former one (`CVE-2021-38379`) affects all deployments running all supported versions of *CFEngine Enterprise* (and many unsupported versions, 3.5 or newer, to be more precise). Both issues were discovered internally during development and testing and we have no indications of these vulnerabilities being exploited or known of outside of the development team. ## CVE-2021-38379 - Publicly available exported reports ### Description *CFEngine Enterprise* uses the `tmp` directory in the HTML document root (for example https://hub.example.com/tmp/) for temporary files that need to be accessed by users of *Mission Portal*, the *CFEngine Enterprise* web user interface. This directory needs to be publicly accessible because it includes files that are needed for the basic functionality. Unfortunately, this directory has also been used for files with exported reports, for example https://hub.example.com/tmp/CFEngine_Enterprise_advancedreports-09-15-2021-abcdef.pdf. An attacker can try making requests to the hub machine with the URLs in the above easily predictable format and get access to reports generated by users using *Mission Portal*. ### Detection All deployments running *CFEngine Enterprise* version 3.5 and newer are affected. ### Impact An attacker with network access to the hub machine (port `443`) can obtain reports generated by users in *Mission Portal* with potentially sensitive data. ### Mitigation Because the mechanisms for making the exported reports available for users are part of the *CFEngine Enterprise* binaries, the only possible mitigation for this CVE is to upgrade the hub installations to one of the newest supported releases of *CFEngine Enterprise* - version *3.15.4-2* or *3.18.0-2*. **Client machines are not affected by this CVE as they don\'t run the *Mission Portal* web user interface.** The newest supported releases of *CFEngine Enterprise* use a separate directory for exported reports which is not publicly accessible and the names of the files with the exported reports are practically impossible to predict so even logged in users cannot get access to other users\' exported reports. This issue has been registered as *CVE-2021-38379* in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-38379). ## CVE-2021-36756 - Certificate not checked in Federated Reporting ### Description *CFEngine Enterprise* deployments using the *Federated Reporting* feature to collect reports from multiple hubs (called *feeders*) to one aggregating hub (called *superhub*) don\'t properly check the TLS certificates of *feeder* hubs when connecting them to the *superhub*. An attacker can use IP spoofing, DNS spoofing or other common techniques to direct the traffic from the *superhub* to their own machine instead of the real *feeder* hub and get it connected to the *superhub*. The *superhub*-*feeder* connection process exchanges secrets that are then used to ensure integrity and authenticity of the data sent by the *feeder*. So the attacker can feed the *superhub* fake data or, even worse, harmful data that can damage the *superhub* database. **Feeder hubs and their data are not affected by this vulnerability.** ### Detection All *CFEngine Enterprise* deployments using the *Federated Reporting* feature are affected by this vulnerability. ### Mitigation The only possible mitigation of this vulnerability is re-installation of all *superhubs* and re-connection of all the *feeder* hubs. The newest supported releases of *CFEngine Enterprise* - version *3.15.4-2* and *3.18.0-2* - ensure that the TLS certificates of *feeder* hubs are verified, either automatically if they are using standard certificates with certificate chains signed by trusted certification authorities, or manually by the admin user if they use self-signed certificates. Fortunately, the *superhubs* usually don\'t serve policy (except to themselves) and the only exclusive data they have is their own reporting data and its history. The configuration, saved queries, dashboards etc. can be preserved using the [Import & Export API](https://docs.cfengine.com/docs/3.18/reference-enterprise-api-ref-export-import-api.html). To re-install all *superhubs* and re-connect all *feeder* hubs, please follow the [instructions for a superhub upgrade](https://docs.cfengine.com/docs/3.18/enterprise-cfengine-guide-federated-reporting.html#superhub-upgrade) provided as part of the [online *CFEngine Enterprise* documentation](https://docs.cfengine.com/docs/3.18/enterprise-cfengine-guide.html). Please notice that all the *feeder* hubs need to be disabled (reverted to normal hubs) before they can be re-connected to *superhubs*. This issue has been registered as *CVE-2021-36756* in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-36756). ## Packages for the newest releases of CFEngine Enterprise The previously released 3.18.0 and 3.15.4 hub packages are affected by these vulnerabilities and have been replaced by new packages. Only the vulnerabilities were fixed in these re-releases. The new packages have a `-2` in the filename and new checksums. They are now available in the [Downloads section of the website](/downloads/cfengine-enterprise/), [cf-remote](https://pypi.org/project/cf-remote/), etc. For completeness, the links to the new packages and their checksums are provided below: ### 3.18.0-2 packages | Operating System | Checksum | | :----------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------ | | [Debian 9](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/debian_9_x86_64/cfengine-nova-hub_3.18.0-2.debian9_amd64.deb) | `sha256=05bceb8b50085f857d4d19a2b6a566efa9cae1ecae18fbfb237d56eef9ad1599` | | [Debian 10](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/debian_10_x86_64/cfengine-nova-hub_3.18.0-2.debian10_amd64.deb) | `sha256=a770b2cac3f09083e77f39611a8e8b6b52dd2b95d9b0419aa5aa5a933aae368b` | | [RHEL/CentOS 6](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/redhat_6_x86_64/cfengine-nova-hub-3.18.0-2.el6.x86_64.rpm) | `sha256=adfb77b1bf0804abd1ff0c0e3d642b35f32f740399888d0d3b10ab1cbaf4fdce` | | [RHEL/CentOS 7](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/redhat_7_x86_64/cfengine-nova-hub-3.18.0-2.el7.x86_64.rpm) | `sha256=c6cd356b2d8da807520c2ceeee5ce8ffc5f30fc035d47129bc2caa7525fbbe5f` | | [RHEL/CentOS 8](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/redhat_8_x86_64/cfengine-nova-hub-3.18.0-2.el8.x86_64.rpm) | `sha256=d68a5351c7784de370a21fc4d91dfeb403e81143681b07d9fc468505192f2b3d` | | [Ubuntu 16](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/ubuntu_16_x86_64/cfengine-nova-hub_3.18.0-2.ubuntu16_amd64.deb) | `sha256=dc319d1416e09f27f470d57deca0413936f00b52f81eb6f07edf22e2426db1c5` | | [Ubuntu 18](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/ubuntu_18_x86_64/cfengine-nova-hub_3.18.0-2.ubuntu18_amd64.deb) | `sha256=cf803129761ef27efab452c82d4078b81c7f8827474a4138f948858df25c7f97` | | [Ubuntu 20](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.18.0/hub/ubuntu_20_x86_64/cfengine-nova-hub_3.18.0-2.ubuntu20_amd64.deb) | `sha256=101eefe4ffa69dfb3d3139dfdf805db909f41d20f13d362aa69e9ed0917516b2` | ### 3.15.4-2 packages | Operating System | Checksum | | :----------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------ | | [Debian 8](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/debian_8_x86_64/cfengine-nova-hub_3.15.4-2.debian8_amd64.deb) | `sha256=2739b0cf23b08408305ccc93e63d4a10bf70b5c00651500f1c03d46a2fe57109` | | [Debian 9](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/debian_9_x86_64/cfengine-nova-hub_3.15.4-2.debian9_amd64.deb) | `sha256=f85eccab440fbc818e0dbfb09462a8eb700a3e20b16aeb3b77ceb19af0b20106` | | [Debian 10](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/debian_10_x86_64/cfengine-nova-hub_3.15.4-2.debian10_amd64.deb) | `sha256=a54a0293af1449d244742e80b0812a9f84a48b7b9d90838decdbeb1666d3d0c9` | | [RHEL/CentOS 6](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/redhat_6_x86_64/cfengine-nova-hub-3.15.4-2.el6.x86_64.rpm) | `sha256=f64330a48d23f152ac45da6426664f92f920474585662d1506f4b66769cbc59d` | | [RHEL/CentOS 7](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/redhat_7_x86_64/cfengine-nova-hub-3.15.4-2.el7.x86_64.rpm) | `sha256=993f36a660f63acce1da1723da0db66d62c5ad6c4a55f1c293ad50656d94f2b7` | | [RHEL/CentOS 8](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/redhat_8_x86_64/cfengine-nova-hub-3.15.4-2.el8.x86_64.rpm) | `sha256=6ea8f76335d6a5950ef636f9e3d44737cfa9bc092a6dba702c5d2e4c9e779329` | | [Ubuntu 14](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/ubuntu_14_x86_64/cfengine-nova-hub_3.15.4-2.ubuntu14_amd64.deb) | `sha256=e4fccfc3b7fa6f5f31ec1d91c388cf344901bbfcb110e446d326c277f6965dde` | | [Ubuntu 16](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/ubuntu_16_x86_64/cfengine-nova-hub_3.15.4-2.ubuntu16_amd64.deb) | `sha256=7f9540ec066cd971e7a223fdd93d85b31be58c76325c22388389e59c06745b3c` | | [Ubuntu 18](https://cfengine-package-repos.s3.amazonaws.com/enterprise/Enterprise-3.15.4/hub/ubuntu_18_x86_64/cfengine-nova-hub_3.15.4-2.ubuntu18_amd64.deb) | `sha256=5817cfb9bf5048866cb8554585cc3a24183e1f0290316371429801b023184807` | In case of any questions, feel free to [contact us](https://cfengine.com/support/). -------------------------------------------------------------------------------- title: "Working with external data, a look at classfiltercsv()" url: /blog/2021/classfiltercsv/index.md date: "2021-10-21" -------------------------------------------------------------------------------- When working with CFEngine, it's common to hear advice about separating data from policy. Separating data from policy allows for separation of concerns, delegation of responsibilities and integration with other tooling. Each organization is different, and a strategy that works well in one environment may not work as well in a similar environment of another organization, so CFEngine looks to provide various generic ways to leverage external data. For example, [Augments](https://docs.cfengine.com/docs/3.18/reference-language-concepts-augments.html) (`def.json`) is useful for setting classes and defining variables very early during the agent execution which can be applied to the entire policy having [differences based on system characteristics](https://docs.cfengine.com/docs/3.18/reference-language-concepts-augments.html#augments) as well as being used for [host specific data](https://docs.cfengine.com/docs/3.18/reference-language-concepts-augments.html#host_specific-json). **Functions for parsing strings:** * [`parseintarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-parseintarray.html) * [`parsejson()`](https://docs.cfengine.com/docs/3.18/reference-functions-parsejson.html) * [`parserealarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-parserealarray.html) * [`parsestringarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-parsestringarray.html) * [`parseyaml()`](https://docs.cfengine.com/docs/3.18/reference-functions-parseyaml.html) **Functions for parsing files:** * [`readcsv()`](https://docs.cfengine.com/docs/3.18/reference-functions-readcsv.html) * [`readdata()`](https://docs.cfengine.com/docs/3.18/reference-functions-readdata.html) * [`readenvfile()`](https://docs.cfengine.com/docs/3.18/reference-functions-readenvfile.html) * [`readfile()`](https://docs.cfengine.com/docs/3.18/reference-functions-readenvfile.html) * [`readintarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-readintarray.html) * [`readintlist()`](https://docs.cfengine.com/docs/3.18/reference-functions-readintlist.html) * [`readjson()`](https://docs.cfengine.com/docs/3.18/reference-functions-readjson.html) * [`readrealarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-readrealarray.html) * [`readreallist()`](https://docs.cfengine.com/docs/3.18/reference-functions-readreallist.html) * [`readstringarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-data_readstringarray.html) * [`readstringarrayidx()`](https://docs.cfengine.com/docs/3.18/reference-functions-data_readstringarrayidx.html) * [`readstringlist()`](https://docs.cfengine.com/docs/3.18/reference-functions-readstringlist.html) * [`readyaml()`](https://docs.cfengine.com/docs/3.18/reference-functions-readyaml.html) * [`data_readstringarray()`](https://docs.cfengine.com/docs/3.18/reference-functions-data_readstringarray.html) * [`data_readstringarrayidx()`](https://docs.cfengine.com/docs/3.18/reference-functions-data_readstringarrayidx.html) * [`classfiltercsv()`](https://docs.cfengine.com/docs/3.18/reference-functions-classfiltercsv.html) Let's take a look at one of the newer additions, `classfiltercsv()` which was introduced as part of 3.14.0 and how it simplifies using a CSV file and class expressions for deriving appropriate values. CSVs have been a common pattern in CFEngine for a long time. A column containing a class expression is a common way to express the appropriate context for a data value. Historically they would be parsed by something like `readstringarray()` or `data_readstringarray()`, then the index of the data would be pulled (see `getindices()`) and used for iterating over the data to find the rows that are relevant based on testing the class expression. For example, let's use this *fact based* data for crafting the content of a report. ``` #ClassExpression,Feeling,Reason linux.Afternoon,Good,I am running my preferred platform and it's near luch. I hope it's Ramen! linux.Morning,Good,I just had breakfast and I don't have broken windows nor rotten fruit. windows,Bad,nothing is ever good if I am running Windows. darwin,Bad,who want's to compute with rotten fruit? ``` This policy parses the file, figures out which rows are relevant and then uses the data. ```cf3 bundle agent __main__ { vars: # Parse the file into a data container "_d" data => data_readstringarray("/tmp/my.csv", # filename, "#[^\n]*", # comment, ",", # split, inf, #maxentries, inf); #maxbytes) # Get an index to iterate with "_di" slist => getindices( _d ); reports: # Here we iterate over the class expressions $(_di) and print out the data from the matching rows "I feel $(_d[$(_di)][0]) because $(_d[$(_di)][1])" if => "$(_di)"; # Let's also take a look at the data structure resulting from data_readstringarray() "data_readstringarray() returned: $(with)" with => storejson( _d ), unless => "no_show_data"; } ``` *Execution output:* ``` R: I feel Good because I am running my preferred platform and it's near luch. I hope it's Ramen! R: data_readstringarray() returned: { "darwin": [ "Bad", "who want's to compute with rotten fruit?" ], "linux.Afternoon": [ "Good", "I am running my preferred platform and it's near luch. I hope it's Ramen!" ], "linux.Morning": [ "Good", "I just had breakfast and I don't have broken windows nor rotten fruit." ], "windows": [ "Bad", "nothing is ever good if I am running Windows." ] } ``` Now let's use the same data with `classfiltercsv()`. ```cf3 bundle agent __main__ { vars: # Parse the file into a data container "_d" data => classfiltercsv("/tmp/my.csv", # filename, "no", # has_header "0"); # class_column # optional sort column # Get an index to iterate with "_di" slist => getindices( _d ); reports: # Here we iterate over the class expressions $(_di) and print out the data from the matching rows "I feel $(_d[$(_di)][0]) because $(_d[$(_di)][1])"; # Let's also take a look at the data structure resulting from classfiltercsv() "classfiltercsv() returned: $(with)" with => storejson( _d ), unless => "no_show_data"; } ``` *Execution output:* ``` R: I feel Good because I am running my preferred platform and it's near luch. I hope it's Ramen! R: classfiltercsv() returned: [ { "0": "Good", "1": "I am running my preferred platform and it's near luch. I hope it's Ramen!" } ] ``` While the implementation of these policies is very similar, `classfiltercsv()` can bring *significant* performance and efficiency improvements as the volume of data increases since we no longer have to iterate over the data. Let's add some more rows to the data for consideration. ``` for i in $(seq 9000); do echo "Class$RANDOM,Feel$RANDOM,Reason$RANDOM" >> /tmp/my.csv done unix2dos /tmp/my.csv unix2dos: converting file /tmp/my.csv to DOS format... ``` You might wonder why `unix2dos` is run on the `.csv` file. This is because `classfiltercsv()` requires `CRLF` line endings, per [RFC 4180](https://www.ietf.org/rfc/rfc4180.txt). Now, let's compare the results of running each, first with `data_readstringarray()`: ``` /bin/time cf-agent -KIf /tmp/data_readstringarray.cf | head -n 10 R: I feel Good because I am running my preferred platform and it's near luch. I hope it's Ramen! R: data_readstringarray() returned: { "Class10000": [ "Feel19773", "Reason27476" ], "Class1001": [ "Feel15419", "Reason11564" ], 14.91user 0.00system 0:14.95elapsed 99%CPU (0avgtext+0avgdata 24424maxresident)k 0inputs+320outputs (8major+7905minor)pagefaults 0swaps ``` And again with `classfiltercsv()`: ``` /bin/time cf-agent -KIf /tmp/classfiltercsv.cf | head -n 10 R: I feel Good because I am running my preferred platform and it's near luch. I hope it's Ramen! R: classfiltercsv() returned: [ { "0": "Good", "1": "I am running my preferred platform and it's near luch. I hope it's Ramen!" } ] 0.14user 0.00system 0:00.16elapsed 89%CPU (0avgtext+0avgdata 11504maxresident)k 0inputs+320outputs (8major+1879minor)pagefaults 0swaps ``` As we can see from the output above, for the given data set, `classfiltercsv()` is about 100 times faster. I hope this motivates you to become familiar with the [wide assortment of functions](https://docs.cfengine.com/docs/3.18/reference-functions-readcsv.html) available and find other areas of your policy that can be improved. -------------------------------------------------------------------------------- title: "CFEngine for IoT" url: /blog/2021/cfengine-for-iot/index.md date: "2021-10-19" -------------------------------------------------------------------------------- CFEngine is well suited for use in IoT environments due to its portability, size, and performance. There already exists a meta layer for including the CFEngine community client and Masterfiles Policy Framework in Yocto Project builds. This enables developing policy to: * ensure a service stays running * track changes to important files * monitor a value over time for normalcy Let's walk through bringing up a qemu environment with CFEngine and ensure that a few basic things work: ensure the `udev` service stays running, tracking changes to important files like `/etc/group` and a look at monitoring capabilities. For the next blog in this series we will show using the CFEngine Enterprise client which will enable us to collect reporting data including monitoring data. This will enable us to track the health of my co-worker Nick Anderson by monitoring the height of his desk. He should really be adjusting it about once an hour and is definitely willing to add some automation via CFEngine to optimize his ergonomics. The Enterprise client is not yet available though we do hope to provide a binary yocto layer in the future. ## Yocto Image with CFEngine Community The Yocto project has a well documented [Quick Build](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html) process to get up and running. I used Ubuntu 20 for a base OS since it seemed the most supported. Here are the condensed setup steps to perform: ```sh $ sudo apt-get install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm python3-subunit mesa-common-dev $ git clone git://git.yoctoproject.org/poky -b hardknott $ cd poky $ source oe-init-build-env Note that this will change your current working directory to `$HOME/poky/build` $ bitbake core-image-minimal ``` I like to use `core-image-minimal` because it takes the least amount of time to iterate on changes and I don't need any graphics environment to play around with CFEngine. This last step of calling `bitbake` will take QUITE a long time the first time you execute it on a machine. On an old laptop with a 2 core Intel Core Duo at 2.80GHz it took about 4 and a half hours. On a newer laptop with a 4 core Intel Core i7-6500U @ 2.50GHz it took about 3 hours. Once the image has been built, run the image with qemu with options `serial nographic` so you can stay in your terminal. ```sh $ runqemu qemux86-64 serial nographic ``` Login as `root` with no password (the default) ```sh ... lots of kernel messages ... INIT: Entering runlevel: 5 Configuring network interfaces... ip: RTNETLINK answers: File exists Starting syslogd/klogd: done Poky (Yocto Project Reference Distro) 3.3.3 qemux86-64 /dev/ttyS0 qemux86-64 login: root root@qemux86-64:~# uname -a Linux qemux86-64 5.10.63-yocto-standard #1 SMP PREEMPT Thu Sep 9 02:58:13 UTC 2021 x86_64 GNU/Linux ``` Use `Ctrl-a x` to exit qemu, which will shut down the instance running the image you built. ## Add CFEngine Layer We will use the meta-openembedded, meta-oe layer to add CFEngine. Clone their repository under the `poky` directory you cloned earlier and checkout the latest stable release: `hardknott`. ```sh $ pwd /home/user/poky $ git clone https://github.com/openembedded/meta-openembedded.git $ cd meta-openembedded $ git checkout hardknott ``` Add the meta-oe layer to poky/build/conf/bblayers.conf so that the `BBLAYERS` section looks like the following: ```python3 BBLAYERS ?= " \ /home/user/poky/meta \ /home/user/poky/meta-poky \ /home/user/poky/meta-yocto-bsp \ /home/user/poky/meta-openembedded/meta-oe \ " ``` Now append three lines to the end of `poky/build/conf/local.conf`. * `procps` is needed to enable CFEngine to properly operate `processes` * `cfengine` provides the core agent and other daemons ```python3 IMAGE_INSTALL_append = " procps" IMAGE_INSTALL_append = " cfengine" IMAGE_ROOTFS_EXTRA_SPACE_append = " + 10240" ``` Running the build again results in a much quicker build. Only about 5 minutes on my slower laptop. ```sh $ time bitbake core-image-minimal real 5m5.388s user 0m1.331s sys 0m0.200s ``` 5 minutes isn't bad compared to our initial build time of 4.5 hours! ## Create Initial Policy In a typical CFEngine environment we would bootstrap to a policy server that would provide a masterfiles policy set customized to our needs. If we were creating a production yocto image we would likely pre-bootstrap the device to our policy server so that when it first booted it would already be able to setup a trusted connection and pull down any updates to policy. But in this case we want a more self-contained example so will simply copy masterfiles to inputs and work from inputs. This is a common practice when iterating heavily on policy in a development environment. Start qemu again with ```sh $ runqemu qemux86-64 serial nographic ``` and confirm that CFEngine is available: ```sh # cf-agent --version CFEngine Core 3.15.0 ``` And now we can create our first policy which will ensure that the `udev` service is restarted if it stops. Create the following `/var/libntech/inputs/promises.cf` file. Due to [CFE-3745](https://northerntech.atlassian.net/browse/CFE-3745) we must use `/var/libntech` instead of the normal `/var/cfengine` for the top-level directory. Both `vi` and `cat` are available by default in `core-image-minimal`. ```cf3 bundle agent main { processes: "udevd" restart_class => "udevd_not_running"; commands: udevd_not_running:: "/etc/init.d/udev start" classes => results( "bundle", "udevd_restarted" ); reports: udevd_not_running:: "udevd not running at ${sys.date}"; udevd_restarted_not_kept:: "failed to restart udevd at ${sys.date}"; udevd_restarted_repaired:: "restart of udevd succeeded at ${sys.date}"; } body classes results(scope, class_prefix) { scope => "${scope}"; promise_repaired => { "${class_prefix}_repaired" }; repair_failed => { "${class_prefix}_not_kept" }; repair_denied => { "${class_prefix}_not_kept" }; repair_timeout => { "${class_prefix}_not_kept" }; } ``` Next follow along with this process of running the agent, killing `udevd` and ensuring that the agent causes the service to be restarted properly. ```sh root@qemux86-64:~# ps -efl | grep udev[d] 5 S root 115 1 0 80 0 - 869 do_epo 19:24 ? 00:00:00 /sbin/udevd -d root@qemux86-64:~# cf-agent -KI root@qemux86-64:~# ``` Here we see no output from `cf-agent` because by default CFEngine is quiet when things are as expected. Now kill `udevd` manually and run the agent again. ```sh root@qemux86-64:~# pkill -9 udevd root@qemux86-64:~# ps -efl | grep udev[d] root@qemux86-64:~# root@qemux86-64:~# cf-agent -KI info: Executing 'no timeout' ... '/etc/init.d/udev start' [ 516.810936] udevd[307]: starting version 3.2.10 [ 516.913521] udevd[308]: starting eudev-3.2.10 notice: Q: "...nit.d/udev star": Starting udev info: Last 1 quoted lines were generated by promiser '/etc/init.d/udev start' info: Completed execution of '/etc/init.d/udev start' R: udevd not running at Tue Oct 12 19:33:05 2021 R: restart of udevd succeeded at Tue Oct 12 19:33:05 2021 ``` The service was restarted properly by running `/etc/init.d/udev start`. The [Masterfiles Policy Framework](https://github.com/cfengine/masterfiles/) and CFEngine support managing many types of `service` promises. Unfortunately for our example the name of the service in `/etc/init.d`, `udev`, doesn't match the daemon name `udevd` so we had to use a workaround in the policy above using `processes` and `commands` promises instead. ## Track File Changes Replace `/var/libntech/inputs/promises.cf` with the following policy: ```cf3 bundle agent main { vars: "track_files" slist => { "/etc/passwd", "/etc/group" }; files: "${track_files}" changes => track_changes; } body changes track_changes { hash => "sha256"; report_changes => "all"; update_hashes => "yes"; } ``` The first run of the agent after changing `promises.cf` should show that a new hash is stored for each file: ```sh root@qemux86-64:~# cf-agent -KI notice: Storing sha256 hash for '/etc/passwd' (SHA=4db85105a3a3b6ba18f4d119076e0d9682be470b3b66f03d46c4d324cdc05668) notice: Storing sha256 hash for '/etc/group' (SHA=f7c9f78c9fb2a002256cb993649c0f691abff1b9594ad1e986acd611cfcc0d53) ``` Now if we touch or modify these files, agent runs will report the changes: ```sh root@qemux86-64:~# touch /etc/group root@qemux86-64:~# cf-agent -KI notice: Last modified time for '/etc/group' changed 'Fri Mar 9 12:34:56 2018' -> 'Tue Oct 12 18:20:31 2021' root@qemux86-64:~# echo "junk" >> /etc/group root@qemux86-64:~# cf-agent -KI notice: Hash 'sha256' for '/etc/group' changed! notice: Updating sha256 hash for '/etc/group' (SHA=7bf20f1cafa94cd6227e809e4f7e5a497b41027c3a3882a66c4bb9c348a900dd) notice: Last modified time for '/etc/group' changed 'Tue Oct 12 18:20:31 2021' -> 'Tue Oct 12 18:20:41 2021' ``` ## Monitoring in Community Edition Here we will simply start the daemon `cf-monitord` so that some built in measurements are collected and then report with policy what has been collected. By default, `cf-monitord` stores information about the average value, standard deviation, and most recent value for each monitor. It is possible to create [custom measurements](https://docs.cfengine.com/docs/master/reference-promise-types-measurements.html) but we won't touch on that here. Run `cf-monitord`, wait for the log line `info: Updated averages at ...` and press Ctrl-c. ```sh # cf-monitord --no-fork --inform cf-monitord info: Updated averages at 'Tue:Hr18:Min25_30' ``` We can run `cf-promises --show-vars` to see the _non-zero_ values collected so far: ```sh cf-promises --show-vars=default:mon\. | grep -v 00 | grep -v \"\" Variable name Variable value Meta tags default:mon.av_cpu 5.04 monitoring,source=environment default:mon.av_cpu0 5.04 monitoring,source=environment default:mon.av_diskfree 13.80 monitoring,source=environment default:mon.av_loadavg 0.08 monitoring,source=environment default:mon.av_mem_cached 7.68 monitoring,source=environment default:mon.av_mem_total 136.13 monitoring,source=environment default:mon.av_messages 303.60 monitoring,source=environment default:mon.av_rootprocs 34.20 monitoring,source=environment default:mon.av_users 0.60 monitoring,source=environment default:mon.dev_cpu 8.22 monitoring,source=environment default:mon.dev_cpu0 8.22 monitoring,source=environment default:mon.dev_diskfree 22.54 monitoring,source=environment default:mon.dev_loadavg 0.13 monitoring,source=environment default:mon.dev_mem_cached 12.55 monitoring,source=environment default:mon.dev_mem_free 143.70 monitoring,source=environment default:mon.dev_mem_total 222.30 monitoring,source=environment default:mon.dev_messages 495.78 monitoring,source=environment default:mon.dev_rootprocs 55.85 monitoring,source=environment default:mon.dev_users 0.98 monitoring,source=environment default:mon.env_time Tue Oct 12 18:27:40 2021 time_based,source=agent default:mon.value_cpu 8.39 monitoring,source=environment default:mon.value_cpu0 8.39 monitoring,source=environment default:mon.value_loadavg 0.13 monitoring,source=environment default:mon.value_mem_cached 12.80 monitoring,source=environment default:mon.value_mem_free 146.67 monitoring,source=environment default:mon.value_mem_total 226.88 monitoring,source=environment ``` This monitoring data is presented to policy as variables which can then be used to take actions such as pushing data up to a server via a REST API or making needed configuration changes to adapt such as cleaning up disk space. ## Extend and Revise our Remarks We showed that it is easy to include CFEngine in a yocto image built for IoT devices and that there are many use cases which can provide value in this ecosystem. Next time when we use the Enterprise client we will see how these measurements are collected automatically and can be viewed as charts and data in Mission Portal, our Enterprise Web Interface. We have an epic in our Jira, [CFE-3748](https://northerntech.atlassian.net/browse/CFE-3748), in which we are tracking issues. Feel free to check that out and help out! -------------------------------------------------------------------------------- title: "Using CFEngine inventory as Ansible inventory" url: /blog/2021/cfengine-inventory-to-ansible/index.md date: "2021-10-07" -------------------------------------------------------------------------------- CFEngine and Ansible are two [complementary infrastructure management tools](https://cfengine.com/white-papers/ansible-and-cfengine/) that both work with so-called *inventories*. However, the common term can be quite confusing because the way they are defined and created is very different for an *Ansible Inventory* and for a *CFEngine Inventory*. In the most basic case, an *Ansible Inventory* is just a file with a list of hosts and groups of hosts that Ansible then manages when fed the inventory file. On the other hand, *CFEngine Inventory* is a database of information about all the hosts in the infrastructure managed by CFEngine which the hosts themselves report. In a more complex scenario, an *Ansible Inventory* can also contain a lot of information about the hosts in the infrastructure, but those need to be pulled from somewhere else and given to Ansible. With CFEngine, hosts talk to a *CFEngine Hub*, pull policy from it and report information back to it. On the other hand, with Ansible, policy is pushed to the hosts from one place which thus must have a list of all hosts available in advance, potentially with some extra information (parameters) of the hosts. So the information for an *Ansible Inventory* has to be pulled from somewhere and then given to Ansible. And of course that *somewhere* can be *CFEngine Inventory* which has all the information. Using such data, Ansible can be used in a more efficient way only targeting the hosts where some actions need to be performed. Like shown in the image below and further explained later in this post. [![](./images/cfengine_ansible_targeting.png)](./images/cfengine_ansible_targeting.png) ## CFEngine Hosts and Classes One of the key concepts in CFEngine are *classes*. Policy and CFEngine components can define classes which are used for making decisions and some of which are also reported to the hub. The word *class* has, in this case, its mathematical meaning -- a generalization of a set. Looking at individual hosts, classes *classify* the hosts on which they are defined and for the policy they define *a context* (that is why they are sometimes called *contexts*). However, looking from the global view, classes actually define sets of hosts.[^1] So for example the class `linux` used in a policy as a so-called *class guard* `linux::` defines a required context for the following promises to be evaluated. If the policy is evaluated in a `linux` context, i.e. in an environment where the class `linux` is defined, the promises that follow the class guard are evaluated, otherwise they are skipped. And the hosts where the class `linux` is defined create a set of, well, hosts running Linux. ## CFEngine Hosts by hard class API The recent versions of CFEngine provide a new API to get hosts grouped by *hard classes*. [*Hard classes*](https://docs.cfengine.com/docs/3.18/reference-language-concepts-classes.html#hard-classes) are special classes defined implicitly by CFEngine components based on the environment discovery. So they reflect the OS the host is running, network parameters, CFEngine version, virtualization technology (if any), etc. With a simple API request, the user can get a JSON- or YAML-formatted file with all the hosts known to CFEngine grouped by *hard classes* defined on them. The API request is supposed to be of the following form: ``` URI: https://hub.example.com/api/hosts/by-class Method: GET Parameters: context-include comma delimited string of regular expressions format Output format. Default value is "json". Allowed values: "json", "yaml". withInventory Include inventory data to the API response. Default value is "false". Allowed values: "true", "false" ``` and with no parameters specified (and thus defaults being used), the response looks like this (with parts omitted for brevity): ```json { "cfengine": { "hosts": [ "ip-172-31-21-241.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal", "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-25-102.eu-west-1.compute.internal", "ip-172-31-22-152.eu-west-1.compute.internal" ] }, "127_0_0_1": { "hosts": [ "ip-172-31-22-152.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal", "ip-172-31-25-102.eu-west-1.compute.internal", "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-21-241.eu-west-1.compute.internal" ] }, "172_31_21_241": { "hosts": [ "ip-172-31-21-241.eu-west-1.compute.internal" ] }, "172_31_22_152": { "hosts": [ "ip-172-31-22-152.eu-west-1.compute.internal" ] }, "172_31_23_151": { "hosts": [ "ip-172-31-23-151.eu-west-1.compute.internal" ] }, "linux": { "hosts": [ "ip-172-31-23-151.eu-west-1.compute.internal", "ip-172-31-21-241.eu-west-1.compute.internal", "ip-172-31-25-102.eu-west-1.compute.internal", "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-22-152.eu-west-1.compute.internal" ] }, "centos": { "hosts": [ "ip-172-31-21-241.eu-west-1.compute.internal", "ip-172-31-22-152.eu-west-1.compute.internal", "ip-172-31-25-102.eu-west-1.compute.internal" ] }, "centos_7": { "hosts": [ "ip-172-31-25-102.eu-west-1.compute.internal", "ip-172-31-22-152.eu-west-1.compute.internal", "ip-172-31-21-241.eu-west-1.compute.internal" ] }, "ubuntu": { "hosts": [ "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal" ] }, "ubuntu_16": { "hosts": [ "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal" ] }, "ubuntu_16_04": { "hosts": [ "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal" ] }, "xen": { "hosts": [ "ip-172-31-28-22.eu-west-1.compute.internal", "ip-172-31-25-102.eu-west-1.compute.internal", "ip-172-31-22-152.eu-west-1.compute.internal", "ip-172-31-23-151.eu-west-1.compute.internal", "ip-172-31-21-241.eu-west-1.compute.internal" ] } } ``` The first part of the example above shows the group of hosts with the `cfengine` class defined. This special class is defined on all hosts managed by CFEngine so this group can conveniently be used as an equivalent of what is commonly known as the `all` group. Then we can see some groups (classes) based on the IP addresses assigned to the hosts, the group of Linux hosts and multiple groups based on the operating system (GNU/Linux distribution) the hosts are running. Those groups nicely demonstrate a common pattern of how CFEngine defines the hard classes -- a specific class (e.g. `ubuntu_16_04`), is defined together with more generic classes (`ubuntu_16` and `ubuntu`) that are constructed from it by removing the trailing parts separated by underscores. Last but not least, there\'s a group of hosts running in the Xen environment which is the same as the `cfengine` group because all the hosts that were used to generate the above example were running in AWS. ## CFEngine to Ansible The above example response from the CFEngine hosts by class API shows the format of the response (in the JSON format, but the YAML format is just a direct equivalent). Each class/group has its top-level object with a `hosts` array of host names. This format is directly recognized by Ansible, not by accident, of course, in fact, the API response format was designed that way as part of the efforts to make integration of CFEngine and Ansible easier. ### Ansible inventory data formats Ansible supports various [inventory sources](https://docs.ansible.com/ansible/latest/dev_guide/developing_inventory.html#id14) providing inventory data in various formats. The most common source is a simple file in a JSON, YAML or ini format, but if given an executable, Ansible uses the [*script-plugin*](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/script_inventory.html#script-inventory) that runs the executable and parses its output. Such executable is supposed to output a JSON in the format shown above in the example. Unfortunately, **this format differs from the JSON/YAML format of an inventory file**, even though the difference is very subtle.[^2] The CFEngine hosts by class API was designed to be used in a script (usually just a simple *curl* invocation) and thus the format of the response follows the specification of the *script plugin*. Which means **it doesn\'t work as an inventory file**. Future versions of CFEngine will recognize an extra parameter `inventoryFile=true/false` to support scenarios where the response is saved into a file instead of directly fed into one of the Ansible utilities. ### Example inventory script As described above, the CFEngine hosts by class API returns inventory data suited for Ansible\'s inventory *script-plugin* which expects an executable to run. The most trivial script for such use is just: ```bash #!/bin/bash curl -u ${CFE_USER}:${CFE_PWD} https://hub.example.com/api/hosts/by-class ``` with the `CFE_USER` and `CFE_PWD` environment variables to be set and exported with the `export` command.[^3] With a script like the above saved as, let\'s say *get~cfeinventory~.sh*, with the executable bit set (`chmod u+x get_cfe_inventory.sh`) the *CFEngine Inventory* data can be used with the Ansible utilities like this: ```bash history -a export CFE_USER="admin" CFE_PWD="testingCFEngine" ansible -m setup -i ./get_cfe_inventory.sh debian unset CFE_PWD history -c ``` where the last argument to the *ansible* command (`debian`) tells Ansible to only target hosts from the `debian` group as defined in the inventory data, i.e. as returned by the CFEngine hosts by class API. Of course it\'s possible to execute some tasks only on some specific hosts with Ansible itself, (although the `debian` class in CFEngine is defined on all hosts running Debian-like/Debian-derived GNU/Linux distributions), but with adding a condition like `where ansible.distro_family == debian` (if such special variable exists/existed) would mean Ansible would connect to all the hosts **and only then** skip the particular task(s) on those hosts that fulfill the condition. Using the information from CFEngine speeds things up significantly if there are many hosts in the infrastructure and it can use the full potential of CFEngine\'s hard classes. ## Host variables As was explained in [the first part of this post](#Introduction), *CFEngine Inventory* contains information about hosts in the infrastructure managed by CFEngine which the hosts report. *Ansible Inventory*, on the other hand, may contain host-specific variables with host-specific values used in the tasks and playbooks. Obviously, these two mechanisms can be chained up just like getting the groups of hosts from CFEngine and feeding them to Ansible. By adding the `withInventory=true` argument to the API query (i.e. at the end of the URL used in the `curl` command), the response is extended with a piece of data like this: ```json { "_meta": { "hostvars": { "ip-172-31-22-152.eu-west-1.compute.internal": { "CFEngine Inventory": { "OS": "CentOS 7", "Kernel": "linux", "Timezone": "UTC", "CPU model": "Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz", "Host name": "ip-172-31-22-152.eu-west-1.compute.internal", "Interfaces": "eth0", "BIOS vendor": "Xen", "CFEngine ID": "SHA=0854031414e708d8c647496b75580fb09972b6a108737af08fcaa03e43af7de6", "CPU sockets": "1", "System UUID": "35A626EC-2164-5893-1476-75B7BC9325D4", "Architecture": "x86_64", "BIOS version": "4.2.amazon", "Virtual host": "xen", "Disk free (%)": "91.00", "MAC addresses": "06:3e:2f:ff:99:cf", "IPv4 addresses": "172.31.22.152", "Kernel Release": "3.10.0-957.27.2.el7.x86_64", "Policy Servers": "172.31.21.241", "System version": "4.2.amazon", "Uptime minutes": "96", "Ports listening": "22, 25, 111, 5308", "CFEngine version": "3.18.0", "Memory size (MB)": "14875.92", "CPU logical cores": "4", "Policy Release Id": "79d0f8cfca33efdc0aea676048ec4b7b9889cc0a", "CPU physical cores": "2", "System manufacturer": "Xen", "System product name": "HVM domU", "Timezone GMT Offset": "+0000", "Physical memory (MB)": "15360", "System serial number": "ec26a635-6421-9358-1476-75b7bc9325d4", "Primary Policy Server": "172.31.21.241", "Allowed hosts for cf-runagent": "172.31.21.241", "Allowed users for cf-runagent": "root" } } } } } ``` Then, host-specific variables are defined for all the hosts in all the groups that are part of the response. These so-called *attributes* are user-defined (with some pre-defined) in the CFEngine policy and thus can be easily extended. Once again, the format is ready-to-use with Ansible and so the ansible tasks and playbooks can refer to these values, using the special `hostvars[]` dictionary. This can be a nice extension or a complement for the [*Ansible Facts*](https://docs.ansible.com/ansible/latest/user_guide/playbooks_vars_facts.html) functionality. ## Conclusions This post is another piece in the [series of posts](https://cfengine.com/tags/ansible/) focused on integration of Ansible and CFEngine and using the best of what the two tools can provide. It shows how the *CFEngine Inventory* and *Ansible Inventory*, two seemingly conceptually different things, are actually very similar and how they can be used together to provide enhanced performance and better ease of use. Once again, the combination of Ansible for the deployment part of the infrastructure management and for the well-targeted real-time changes with CFEngine for the general long-term maintenance and infrastructure knowledge and overview proves to be very useful. [^1]: And *class expressions* which are logical expressions combining classes with the logical operators *AND*, *OR* and *NOT*, define sets that are results of the respective set operations *intersection*, *union* and *complement*. [^2]: The `hosts` key must be an object and the individual hosts need to be objects too, not just strings. [^3]: Unfortunately, the Ansible *script-plugin* doesn\'t support passing through standard error output and standard input so just having something like `curl -u admin https://hub.example.com/api/hosts/by-class` and letting *curl* to prompt for the password is not possible. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 5 - Running CFEngine on IoT (Part 1)" url: /blog/2021/the-agent-is-in-episode-5-running-cfengine-on-iot/index.md date: "2021-10-06" -------------------------------------------------------------------------------- Interested in running CFEngine for IoT? Craig (Digger) shows building CFEngine for Yocto. ## Video {#video} The video recording is available on YouTube: {{< youtube 8FxJNImDCx4 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Yocto](https://www.yoctoproject.org/) - [Raspberry Pi Zero](https://www.raspberrypi.com/products/raspberry-pi-zero/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Managing local groups" url: /blog/2021/managing-local-groups/index.md date: "2021-10-01" -------------------------------------------------------------------------------- Manually managing groups on a large infrastructure can be a tedious task, and is therefore best suited through automation software like CFEngine. Unfortunately - at time of writing - CFEngine does not have any built-in promise types for managing groups. But fear not; in CFEngine 3.17, custom promise types were introduced. This new exhilarating feature does not only allow for members of our community to make their own custom promise types, but also lets the CFEngine Core developers prototype new future promise types. In this blog post, I'd like to introduce one of our prototypes; the experimental `groups` promise type. ## Getting started Using CFEngine 3.18 LTS with the default policy set, it is quite easy to get started using the `groups` promise type. The first step consists of adding the respective promise module and its support library to `/var/cfengine/masterfiles/modules/promises/`. The files you will need to copy are the following: 1. [`groups.py`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/groups/groups.py) _(Promise module)_ 2. [`cfengine.py`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) _(Library)_ Next we'll need to tell `cf-agent` about our custom promise type, where to find it, as well as what interpreter to use in order to execute it. This is typically done through a promise type definition in `/var/cfengine/masterfiles/services/init.cf`. Open the respective file in your favorite editor, and add the following lines: ```cf3 promise agent groups { path => "$(sys.workdir)/modules/promises/groups.py"; interpreter => "/usr/bin/python3"; } ``` Awesome! We're all set to start writing some groups promises. ## The `groups` promise type Groups promises are promises about local groups on a host. They express whether a group shall be present or absent; which users shall be included or excluded as members of the group; as well as what GID the group shall have. These properties are expressed through the attributes `policy`, `members` and `gid` respectively. In the example bundle below we specify promises about three groups named `foo`, `bar` and `baz`. ```cf3 bundle agent example { groups: "foo" policy => "present", members => "$(foo.members)"; "bar" members => "$(bar.members)", gid => "1234"; "baz" policy => "absent"; } bundle agent foo { vars: "members" string => '{ "include": ["alice", "bob"], "exclude": ["malcom"] }'; } bundle agent bar { vars: "members" string => '{ "only": ["alice"] }'; } ``` In the first promise statement, we promise that a group named `foo` shall be present on the host and that its members shall include the users `alice` and `bob`, but exclude the user `malcom`. Note that in the second promise statement, we do not specify the `policy` attribute. Still we promise that a group named `bar` shall be present. This is due to the fact that the `policy` attribute defaults to `"present"` if not specified. Additionally, in the second promise statement, we promise that the members of `bar` shall _only_ contain the user `alice`, excluding all others. Please note that the `only` member-attribute cannot be used in combination with `include` or `exclude`. In the last promise statement, we promise that a group named `baz` shall be absent from the host, meaning that this group shall not exist. By now, you may have noticed that the `member` attribute takes a JSON string. And you're probably wondering why it does not take a member body as illustrated below. ```cf3 bundle agent example_2 { groups: "c_experts" policy => "present", members => c_expert_members; } body member c_expert_members { include => { "vpodzime", "olehermanse" }; exclude => { "larsewi" }; } ``` The reasoning behind this is that the use of bodies - unfortunately - is still not supported with custom promise types. But this is definitely something we will implement in the near future. A complete example can be found [here](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/groups/groups.cf). By running this example, you can expect the following output: ``` $ /var/cfengine/bin/cf-agent -KIf /var/cfengine/masterfiles/services/groups.cf info: User promise repaired info: User promise repaired info: User promise repaired info: Created group 'foo' info: Added user 'alice' to group 'foo' info: Added user 'bob' to group 'foo' info: Created group 'bar' info: Members of group 'bar' set to only users ['alice'] ``` ## Managing local groups with CMDB Combining the `groups` promise type with CMDB can lead to some powerful use cases. Let's say you have an issue with one of your hosts. And in order for the developers to debug the host, you would need to grant them temporary access using a local group. With CMDB you can use Mission Portal to create a `temp_members_debug` list variable, containing your most blessed developers: [![Managing local groups with CMDB](images/managing-local-groups-cmdb.png)](images/managing-local-groups-cmdb.png) In policy, a promise can add the users from this list as members to the group `debug`, if and only if the variable is present: ```cf3 bundle agent add_temp_members_to_debug { groups: "debug" policy => "present", members => '{ "include": ["$(data:variables.temp_members_debug)"] }', if => isvariable("data:variables.temp_members_debug"); } bundle agent __main__ { methods: "add_temp_members_to_debug"; } ``` Awesome, right? To expand on this, you could write policy to remove the users when the variable is removed from CMDB. Here is the output from running the example above: ``` $ cf-agent -KIf /var/cfengine/masterfiles/services/add_temp_members_to_debug.cf info: Created group 'debug' info: Added user 'vpodzime' to group 'debug' info: Added user 'olehermanse' to group 'debug' info: Added user 'larsewi' to group 'debug' ``` ## Share your thoughts By now you know all there is to know about the `groups` promise type. If you choose to use this promise type, or if you have any tips on how we can improve it; please let us know over at [GitHub Discussions](https://github.com/cfengine/core/discussions), we are eager to hear your feedback. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 4 - CFEngine Build system (cfbs)" url: /blog/2021/the-agent-is-in-episode-4-cfengine-build-system-cfbs/index.md date: "2021-08-27" -------------------------------------------------------------------------------- Come see what's new in CFEngine policy management! Herman (Product Manager) introduces and demonstrates new tooling, the CFEngine Build System (cfbs). `cfbs` is a command line tool to facilitate policy management and consuming modules written by others. ## Video {#video} The video recording is available on YouTube: {{< youtube j4eKHTtFQd4 >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [cfbs](https://github.com/cfengine/cfbs) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Cloning git repos and creating systemd services with CFEngine" url: /blog/2021/git-and-systemd-promises/index.md date: "2021-08-16" -------------------------------------------------------------------------------- Using modules, you can add custom promise types to CFEngine, to manage new resources. In this blog post, I'd like to introduce some of the first official modules, namely `git` and `systemd` promise types. They were both written by [Fabio Tranchitella](https://github.com/tranchitella), who normally works on our other product, [Mender.io](https://mender.io). He decided to learn some CFEngine and within a couple of weeks he's contributed 3 modules, showing just [how easy it is to implement new promise types](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/). Thanks, Fabio! ## Getting started If you are using CFEngine 3.18 LTS, and a policy set based on the default one we distribute, it's very easy to get started with custom promise types. First, add the modules and any supporting libraries to `/var/cfengine/masterfiles/modules/promises/`. For example, you can add these 3 python files: 1. [`cfengine.py`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) (Library) 2. [`git.py`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/git/git.py) (module for `git` promise type) 3. [`systemd.py`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/systemd/systemd.py) (module for `systemd` promise type) (If you have your policy in git, or another version control system, make sure you commit these modules to version control). Additionally, you need to enable the promise types using a promise type definition, typically placed `/var/cfengine/masterfiles/services/init.cf`: ```cf3 promise agent git { path => "$(sys.workdir)/modules/promises/git.py"; interpreter => "/usr/bin/python3"; } promise agent systemd { path => "$(sys.workdir)/modules/promises/systemd.py"; interpreter => "/usr/bin/python3"; } ``` There are 3 pieces of information to be aware of here. `promise agent git` defines a new promise type, only for the `cf-agent` component. This means you can use it in policy files by writing `git:` inside agent bundles. `path` is the path to the module itself (a script / executable). `interpreter` is the path to a program which runs the module. In this case, two python processes will be started by the agent, like this: ``` /usr/bin/python3 /var/cfengine/modules/promises/git.py /usr/bin/python3 /var/cfengine/modules/promises/systemd.py ``` ## Cloning git repos Once you have added and enabled the modules, you can use them from anywhere within your policy set. As an example, we can clone two git repos: ```cf3 bundle agent website_git_repos { git: "/opt/hugoBasicExample" repository => "https://github.com/gohugoio/hugoBasicExample", version => "master"; "/opt/hugoBasicExample/themes/hugo-PaperMod" repository => "https://github.com/adityatelange/hugo-PaperMod", version => "master"; } ``` More documentation and examples for the `git` promise type is available [here](https://github.com/cfengine/modules/tree/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/git). ## Creating a webserver systemd service After cloning, we have a hugo site and theme, and would like to start a server. `systemd` is great for starting a process, and ensuring it keeps running, so let's create a systemd service: ```cf3 bundle agent website_systemd_service { systemd: "website" name => "website", state => "started", unit_description => "My example hugo website service", service_exec_start => {"/bin/sh -c 'cd /opt/hugoBasicExample && hugo server -t hugo-PaperMod'"}; } ``` When writing `systemd:` as our promise type, we are using the `systemd` module, which handles creating a service, as well as enabling it. More documentation and examples for the `systemd` promise type is available [here](https://github.com/cfengine/modules/tree/54a41ff399e0f918fa5cf50f4e1e4b858d678915/promise-types/systemd). ## The complete example The 2 bundles above are really all it takes to ensure a webserver is running, with the help of systemd, and content from git repos. Here is a complete example which can be added as `/var/cfengine/masterfiles/services/autorun/website.cf`: ```cf3 bundle agent website_hugo_install { packages: "hugo" policy => "present"; } bundle agent website_git_repos { git: "/opt/hugoBasicExample" repository => "https://github.com/gohugoio/hugoBasicExample", version => "master"; "/opt/hugoBasicExample/themes/hugo-PaperMod" repository => "https://github.com/adityatelange/hugo-PaperMod", version => "master"; } bundle agent website_systemd_service { systemd: "website" name => "website", state => "started", unit_description => "My example hugo website service", service_exec_start => {"/bin/sh -c 'cd /opt/hugoBasicExample && hugo server -t hugo-PaperMod'"}; } bundle agent website { meta: "tags" slist => { "autorun" }; methods: "website_hugo_install"; "website_git_repos"; "website_systemd_service"; } ``` The first bundle, `website_hugo_install` uses the default package module to install hugo, if not installed already. Then, bundles for cloning git repos and creating systemd services are implemented with the modules, as explained above. Finally, the last bundle, `website` ties it all together, it is the entry point for the policy, and defines the order of the other bundles. This bundle is tagged so it is automatically run, if you've enabled [`autorun`](https://docs.cfengine.com/docs/3.18/reference-masterfiles-policy-framework-services-autorun.html) functionality. Otherwise, you will have to manually add the policy file to _inputs_ and the bundle to the _bundle sequence_. You can enable autorun using `/var/cfengine/masterfiles/def.json`: ``` { "classes": { "services_autorun": [ "any::" ] } } ``` If you run the policy, you should see something like this: ``` $ /var/cfengine/bin/cf-agent -KI info: Successfully installed package 'hugo' info: Cloning 'https://github.com/gohugoio/hugoBasicExample:master' to '/opt/hugoBasicExample' info: Cloning 'https://github.com/adityatelange/hugo-PaperMod:master' to '/opt/hugoBasicExample/themes/hugo-PaperMod' info: Enabled the service website [...] ``` You can see it working using `ps`: ``` $ ps aux | grep hugo root 64075 0.0 0.0 2608 608 ? Ss 18:32 0:00 /bin/sh -c cd /opt/hugoBasicExample && hugo server -t hugo-PaperMod root 64076 1.6 0.3 966980 64212 ? Sl 18:32 0:00 hugo server -t hugo-PaperMod root 64502 0.0 0.0 8160 740 pts/0 S+ 18:32 0:00 grep --color=auto hugo ``` Or `curl`: ``` $ curl localhost:1313 Hugo Themes [...] ``` Complete documentation for custom promise types is available [here](https://docs.cfengine.com/docs/3.18/reference-promise-types-custom.html). ## Tooling to manage policy and modules We are working on tooling which makes it much easier to manage your policy, modules, and their dependencies. See the video below for a sneak peek: {{< youtube fzjN2UM-Jvo >}} -------------------------------------------------------------------------------- title: "Optional arguments with getopt_long(3)" url: /blog/2021/optional-arguments-with-getopt-long/index.md date: "2021-08-13" -------------------------------------------------------------------------------- I recently had a minor task involving changing an option - on one of our command line tools - from taking a required argument, to taking an optional argument. This should be easy they said; just change the respective `option` struct to take an optional argument, add a colon to the `optstring`, and get on with your life. Well, it proved to be easier said than done. My initial expectation was that a solution similar to the one below should just work. And it does work, just not in the way I expected. ```c #include #include #include int main(int argc, char *argv[]) { int opt; const struct option options[] = { {"no-arg", no_argument, 0, 'n'}, {"opt-arg", optional_argument, 0, 'o'}, {"req-arg", required_argument, 0, 'r'}, {NULL, 0, 0, '\0'} }; while ((opt = getopt_long(argc, argv, "no::r:", options, NULL)) != -1) { switch(opt) { case 'n': // option with no argument printf("No argument\n"); break; case 'o': // option with optional argument printf("Optional argument: %s\n", (optarg == NULL) ? "default" : optarg); break; case 'r': // option with required argument printf("Required argument: %s\n", optarg); break; default: fprintf(stderr, "Usage: %s [-n] [-o [arg]] [-r arg]\n", argv[0]); return EXIT_FAILURE; } } return EXIT_SUCCESS; } ``` After making these minor changes - I manually tested the code - and to my despair, the code kept giving me the default argument on the options with optional arguments, using short options. ```sh $ ./main -r testarg # -r with required argument 'testarg' Required argument: testarg $ ./main -o testarg # -o with optional argument 'testarg' Optional argument: default ``` Taking the red pill; I went head first into the Linux man pages, in order to figure out what was wrong with my code. It turned out there was nothing wrong with the code at all. > *_Two colons mean an option takes an optional arg; if there is text in the > current argv-element (i.e., in the same word as the option name itself, for > example, "-oarg"), then it is returned in optarg, otherwise optarg is set to > zero._* > - [getopt_long(3) - Linux man page](https://www.man7.org/linux/man-pages/man3/getopt_long.3.html) Confusing, right? In other words, `getopt_long` expects no whitespace between an option and its optional argument. For options with required arguments on the other hand, it simply does not care. Who would have guessed? ```sh $ ./main -otestarg # -o with optional argument 'testarg' Optional argument: testarg ``` Yes it works! But we cannot really expect our users to magically know that optional arguments have to be passed in this special way, right? Thus, we'll need to do something about it! ## Solution The solution I arrived at to solve this issue is as follows: - If `optarg` is `NULL`, it means there are either no arguments, or the option and the argument are separated by a whitespace character. In this case, the potential argument - if present - would be the next string in `argv`. - The variable `optind` has the index of the next argument to be processed in `argv`, thus we can use it to get a hold of our potential argument. But before accessing `argv`, we need to make sure we don't read out of bounds of the array. We can do this by simply checking that `optind` is less than the argument count `argc`, before accessing `argv` at this index. - We determine whether or not the next string in `argv` is actually an argument, based on the fact that another option would have to start with the `'-'` character. Thus we compare the first character of the next string in `argv` with the `'-'` character, assuming it's an argument in case of inequality. - When all these conditions are met, our conclusion is that the next string in `argv` is in fact an argument. We can extract the argument by setting `optarg` to point at the next string in `argv`, this followed by advancing `optind` accordingly. The following is an example of how you could implement and use this. ```c case 'o': // option with optional argument if (optarg == NULL && optind < argc && argv[optind][0] != '-') { optarg = argv[optind++]; } if (optarg == NULL) { // Handle is present } else { // Handle is not present } break; ``` Hooray! The options now work as previously expected. ```sh $ ./main -r testarg # -r with required argument 'testarg' Required argument: testarg $ ./main -o testarg # -o with optional argument 'testarg' Optional argument: testarg ``` I have to admit, the solution above is kind of hacky. For a neater solution, that improves readability and avoids great code duplication, you can wrap it all up in a macro like I did below. ```c #define OPTIONAL_ARGUMENT_IS_PRESENT \ ((optarg == NULL && optind < argc && argv[optind][0] != '-') \ ? (bool) (optarg = argv[optind++]) \ : (optarg != NULL)) ``` (We use all caps to signify that this is a macro, not a normal function or variable). And use the macro like this: ```c case 'o': // option with optional argument if (OPTIONAL_ARGUMENT_IS_PRESENT) { // Handle is present } else { // Handle is not present } break; ``` This macro, and many other macros, parsing functions, data structures, etc. are available in our standalone library, [libntech](https://github.com/cfengine/libntech). Feel free to use it in your own C/C++ projects. If this blog post was helpful in any way, or if you know of any better ways to get options with optional arguments working like one would expect. Please let us know over at [GitHub Discussions](https://github.com/cfengine/core/discussions)! Remember, sharing is caring. -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 3 - What's new in 3.18?" url: /blog/2021/the-agent-is-in-episode-3-whats-new-in-3-18/index.md date: "2021-07-29" -------------------------------------------------------------------------------- Come see the new hotness in the latest LTS series, 3.18! Craig (Digger) and Nick (Doer of Things) take a tour of 3.18.0, the first release in the latest LTS series. Join them in exploring dark mode, compliance reports, host specific data via Mission Portals CMDB, manually triggered agent runs, report collections, and CFEngine Build. ## Video {#video} The video recording is available on YouTube: {{< youtube s1NzJqJVw0o >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [CFEngine 3.18 release announcement](/blog/2021/cfengine-3-18-lts-released-extensibility/) - [Groups custom promise type](https://build.cfengine.com/modules/promise-type-groups/) - [Python custom promise type library](https://build.cfengine.com/modules/library-for-promise-types-in-python/) - [Bash custom promise type library](https://build.cfengine.com/modules/library-for-promise-types-in-bash/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), [Herman](https://www.linkedin.com/in/olehermanse), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 2 - Community user demo with Jeff Carlson" url: /blog/2021/the-agent-is-in-episode-2-community-user-demo-with-jeff-carlson/index.md date: "2021-06-25" -------------------------------------------------------------------------------- Interested writing CFEngine policy faster? Jeff (CFEngine Community user) demonstrates his YASnippet library for CFEngine to make writing CFEngine policy significantly faster. ## Video {#video} The video recording is available on YouTube: {{< youtube pCJzzJe9bAw >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Jeff Carlson's YASnippet library for CFEngine](https://github.com/jeffcarlson72/emacs-yasnippet-cfengine3-mode) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Jeff Carlson](https://www.linkedin.com/in/jeff-carlson-1912091/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "CFEngine 3.18 LTS released - Extensibility" url: /blog/2021/cfengine-3-18-lts-released-extensibility/index.md date: "2021-06-24" -------------------------------------------------------------------------------- Today, we are pleased to announce the release of CFEngine 3.18.0\! The focus of this new version has been _extensibility_. It also marks an important event, the beginning of the 3.18 LTS series, which will be supported for 3 years. Several new features have been added since the release of CFEngine 3.15 LTS, in the form of non-LTS releases. In this blog post we'll primarily focus on what is new in 3.18, but we will also highlight some things released in 3.16 and 3.17. ## What's new ### A new look [![](images/new_look.png)](images/new_look.png) We are updating the visual look and usability of many parts of Mission Portal. The new style is more clean and modern, and focused on improving ease of use and accessibility. Some changes are in 3.18.0, but we are still working on this, so expect many improvements in this area going forward. ### Host Specific Data (CMDB) [![](images/cmdb.png)](images/cmdb.png) You can now assign data (CFEngine variables and classes) to individual hosts from Mission Portal. This is great for setting the owner of a machine, assigning it a role, or controlling what its policy is doing. As an example, you can have policy to automatically update all packages on a machine. You might not want this enabled at all times, but temporarily enabling it for specific hosts is very useful. The new UI can be found in the host info page, and the CMDB can also be controlled [using a REST API](https://docs.cfengine.com/docs/3.18/reference-enterprise-api-ref-cmdb-api.html). ### Trigger agent runs and report collection The host info page in Mission Portal has new buttons for triggering agent runs and report collection for the host. Combined with CMDB, this enables a powerful workflow, allowing you to very quickly make changes and see the result. You can change a variable in CMDB, and click the "Play" button in the host info page, causing the agent on that host to fetch the latest policy, evaluate it and report back on the results. This developer demo video shows the new agent run and report collection buttons in action: {{< youtube -ZAMFb2msgA >}} ### Custom promise types [CFEngine 3.17](/blog/2020/cfengine-3-17-flexibility) introduced custom promise types. Since then, we've fixed several bugs and added more functionality to make the feature ready for production use. Using modules, you can add new promise types to CFEngine, allowing you to manage more complex resources with ease. As an example, a promise to keep a git repo up to date, looks like this: ```cf3 bundle agent main { git: "/home/tom/code/kubernetes/" repository => "https://github.com/kubernetes/kubernetes", version => "master"; } ``` Going over all of the capabilities of custom promise types would make this blog post too long, but [our documentation has a good introduction, links to tutorials as well as complete specification of how they work](https://docs.cfengine.com/docs/3.18/reference-promise-types-custom.html). So far, we have 4 modules implementing new promise types: - [`git`](https://build.cfengine.com/modules/promise-type-git) - Manage local checkouts of git repos - [`systemd`](https://build.cfengine.com/modules/promise-type-systemd) - Manage systemd services - [`ansible`](https://build.cfengine.com/modules/promise-type-ansible) - Run Ansible playbooks on individual hosts - [`groups`](https://build.cfengine.com/modules/promise-type-groups) - Create and manage local user groups Later this year, [we will make it easier for you to incorporate these modules with your policy and keep them up to date](/blog/2021/collaboration-and-sharing/). ### Compliance Reports [![](images/compliance.png)](images/compliance.png) [CFEngine 3.16](/blog/2020/cfengine-3-16-compliance) introduced Compliance Reports, a new way to specify compliance requirements as checks, and group them all into 1 high level report. With it came some very useful inventory conditions; `regex matches`, `regex doesn't match`, `is reported` and `is not reported`. You can create checks to ensure you are running supported operating systems, that specific packages are up to date, or anything you can imagine using CFEngine's extensible inventory system. Since their introduction, we've made countless usability and quality of life improvements to Compliance Reports. If you haven't seen compliance reports in action, take a look at this video: {{< youtube 1jNxP0WVEyI >}} ### OS Inventory [![](images/os_inventory.png)](images/os_inventory.png) We've cleaned up the OS Inventory strings in Mission Portal. Previously, this was based on policy attempting to use many different sources, including `lsb-release`, if installed. The new OS string is based on 2 new policy variables; `$(sys.os_name_human)` and `$(sys.os_version_major)`. This makes it a lot easier to read, and much more predictable for filtering or compliance reports. ### New policy language functions In CFEngine policy language, several functions have been added since 3.15: - [`cf_version_after()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_after.html) - [`cf_version_at()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_at.html) - [`cf_version_before()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_before.html) - [`cf_version_between()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_between.html) - [`cf_version_maximum()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_maximum.html) - [`cf_version_minimum()`](https://docs.cfengine.com/docs/3.18/reference-functions-cf_version_minimum.html) - [`execresult_as_data()`](https://docs.cfengine.com/docs/3.18/reference-functions-execresult_as_data.html) - [`findfiles_up()`](https://docs.cfengine.com/docs/3.18/reference-functions-findfiles_up.html) - [`int()`](https://docs.cfengine.com/docs/3.18/reference-functions-int.html) - [`string()`](https://docs.cfengine.com/docs/3.18/reference-functions-string.html) - [`string_trim()`](https://docs.cfengine.com/docs/3.18/reference-functions-string_trim.html) - [`type()`](https://docs.cfengine.com/docs/3.18/reference-functions-type.html) - [`validdata()`](https://docs.cfengine.com/docs/3.18/reference-functions-validdata.html) - [`validjson()`](https://docs.cfengine.com/docs/3.18/reference-functions-validjson.html) ### Other changes There are other big changes introduced in the past 1.5 years, but not shown here. Automatic synchronization of ActiveDirectory and Mission Portal roles, end to end encryption of secrets, simulating and visualizing changes `cf-agent` would make, customizing information shown on host info page, Ansible compatible hosts API, are just some of the features we've introduced. We encourage you to read our previous release blog posts to see more of these changes: - [CFEngine 3.16 - Compliance](/blog/2020/cfengine-3-16-compliance) - [New features of Mission Portal in 3.15.3](/blog/2020/cfengine-3-12-6-and-3-15-3-released/) - [CFEngine 3.17 - Flexibility](/blog/2020/cfengine-3-17-flexibility) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.18.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-core.html) - [3.18.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.18.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.18/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) If you are upgrading from the 3.15 LTS series, scroll down in the changelog to find changes made in 3.17 and 3.16 to see the older changes. Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates Compared to the recently released [3.15.4](/blog/2021/cfengine-3-12-7-and-3-15-4-released), these dependencies have been updated: | CFEngine | 3.15.4 | **3.18.0** | | :-------------------------------------------------------------------------------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.47 | **2.4.48** | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.31.1 | **2.32.0** | | [libcurl](https://curl.haxx.se/download.html) | 7.75.0 | **7.77.0** | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.4.58 | **2.5.5** | | [PHP](https://php.net/) | 7.4.19 | **8.0.7** | | [PostgreSQL](https://www.postgresql.org/) | 12.7 | **13.3** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/free-download/) to go to the download pages with new packages. If you are using `cf-remote`, it will now default to 3.18.0, since this is the latest LTS release available. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "The CFEngine UI gets a fresh look" url: /blog/2021/the-cfengine-ui-gets-a-fresh-look/index.md date: "2021-06-17" -------------------------------------------------------------------------------- In the past 2 and half months, we have refreshed the visual design of the CFEngine UI (Mission Portal). As part of the UX/UI design team, our goal is to provide an up-to-date design as well as an easy-to-use user interface. In this blog post, I will showcase the changes we made, the process behind the redesign, and the challenges encountered along the way. I am sharing the journey with you and hope to inspire the people who are currently planning to take a similar path. ![](images/dark_mode_dashboard.gif "New light and dark mode") ## Kickstart the design system and improve as we go After the launch of the new website in March this year, we knew that updating the look and feel to reflect the website design would be our next project. The first step was to have a similar brand design and identical color set for the Mission Portal. However, I soon realized that the design materials and resources were not enough to start the redesigning tasks. Therefore, I decided to start a design system for the Mission Portal and improve it as we go. For example, if we work on a navigation bar, the components would be documented after creating it and thus we can reuse them afterwards. This effectively saves time on the actual design work and also gives us a systematic overview of the current design assets. Besides, I believe having a design system in place would be worthwhile in the longer-term perspective. Below is a quick example of our design system. During the process, I also put Legibility & Accessibility into consideration. We chose a new set of colors for the dashboard charts and widgets and made sure they had enough contrast and reliable readability for all our users. [![](images/design_system.png)](images/design_system.png) ## One iteration at a time We also decided to focus on improving the UI surface as the starting point for the project. This would enable us to deliver a good amount of value to the user without spending months on design and implementation. With this in mind, I began dividing the design tasks into small and digestible sizes. The reasons behind it are; First, it helps us to make continuous small progress on the project as each task is manageable for each individual designer, second, it allows me to develop the design system step by step. ### Top navigation bar Here we refreshed the look and feel of the menu items and improved the dropdown behavior. ![](images/popdowns.gif "Top navigation bar user behavior showcase.") ### Left navigation bar For this part, we repainted the colors used for the navigation and redefined the borderline for the session. This gives a fresh look to the interface as well as a better layout transition to other sessions of the page. ![](images/left_navbar_before_after.png) ### Links and buttons I defined the primary and secondary buttons together with their different states' look and feel. The goal is to apply the consistent button looks across the pages. ![](images/buttons_links_before_after.png) ### Widgets refreshed I adopted the new color palette which is color blindness friendly into the widget design to ensure we have good coverage on accessibility and achieve a consistent look and feel of the dashboard page. [![](images/frame-5516.png)](images/frame-5516.png) ### Tables in Inventory reports The Tables were redesigned, improving the buttons, colors, borders, and text within. They are now easier to read and the primary buttons are relocated to the top of the page so the user can easily access them. [![](images/table_before_after.png)](images/table_before_after.png) ## Continued iteration and improvement We decided to improve the user interface in several stages. By doing this, we can quickly deliver the first set of changes, providing immediate value to our users, and gradually and persistently improve the product and provide our users with a satisfying user experience. This approach also comes with a downside: if we apply changes to one page it will not be consistent with the remaining "old" design. However, the advantages of this approach outweigh the drawbacks, which is why we decided to go down this path. ## What's next? We are planning more improvements for our web UI as well as continuing to optimize the user experience of the software. You can help us to improve by giving us your feedback. Please tell us your ideas, thoughts, and feedback in the community discussion [HERE](https://github.com/cfengine/core/discussions/4716). I hope you enjoy the new look of Mission Portal :) Note: Many of these changes will be in the upcoming 3.18.0 release, but we will continue working on our UI, and making improvements in subsequent releases as well. If you want to test the latest changes, nightly builds are available; See [this video](https://www.youtube.com/watch?v=mBX-Fy6cXH0), to learn how to run CFEngine nightly builds locally. -------------------------------------------------------------------------------- title: "CFEngine 3.12.7 and 3.15.4 released" url: /blog/2021/cfengine-3-12-7-and-3-15-4-released/index.md date: "2021-06-08" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.12.7 and 3.15.4\! These releases mainly contain bug fixes and dependency updates. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.12.7 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html) - [3.12.7 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.12.7 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.15.4 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [3.15.4 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.15.4 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. | CFEngine | 3.12.6 | **3.12.7** | 3.15.3 | **3.15.4** | | :---------------------------------------------------------- | :----- | :--------- | :----- | :--------- | | [Apache](https://httpd.apache.org/) | 2.4.46 | **2.4.47** | 2.4.46 | **2.4.47** | | [Git](https://git-scm.com/) | 2.28.0 | **2.31.1** | 2.28.0 | **2.31.1** | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.2.53 | **2.3.1** | 2.2.53 | **2.3.1** | | [libattr](https://download.savannah.gnu.org/releases/attr/) | 2.4.48 | **2.5.1** | 2.4.48 | **2.5.1** | | [libcurl](https://curl.haxx.se/libcurl/) | 7.72.0 | **7.75.0** | 7.72.0 | **7.75.0** | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.9.10 | **2.9.12** | 2.9.10 | **2.9.12** | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.26 | **0.9.29** | 0.9.26 | **0.9.29** | | [OpenLDAP](https://www.openldap.org/) | 2.4.53 | **2.4.58** | 2.4.53 | **2.4.58** | | [OpenSSL](https://www.openssl.org/) | 1.1.1g | **1.1.1k** | 1.1.1g | **1.1.1k** | | [PHP](https://php.net/) | 7.2.34 | **7.2.34** | 7.4.10 | **7.4.19** | | [PostgreSQL](https://www.postgresql.org/) | 10.14 | **10.14** | 12.4 | **12.7** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Downloads CFEngine Enterprise is free for up to 25 hosts, click [here](https://cfengine.com/free-download/) to go to the download pages with new packages. If you are using `cf-remote`, you can target the new versions: ```command cf-remote --version 3.12.7 list ``` ```command cf-remote --version 3.15.4 list ``` Version selection works for `list`, `download`, and `install` commands. [Read more about `cf-remote` here.](https://pypi.org/project/cf-remote/) ## CFEngine Enterprise 3.12 LTS end of life 3.12.7 is the last release of the 3.12 LTS series, as it is approaching end of life. All versions of CFEngine 3.12 are supported until August (2021). We strongly encourage anyone who is still using 3.12 to [contact us](https://cfengine.com/contact/) for help with upgrading, or to arrange an extended support agreement. ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help, share an idea, or start a discussion on [GitHub Discussions](https://github.com/cfengine/core/discussions) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Show notes: The agent is in - Episode 1 - Debugging with CFEngine policy analyzer" url: /blog/2021/the-agent-is-in-episode-1-debugging-with-cfengine-policy-analyzer/index.md date: "2021-05-27" -------------------------------------------------------------------------------- Interested in seeing promise results (KEPT, REPAIRED, NOTKEPT) overlaid on top of the policy itself? Craig (Digger) and Nick (Doer of Things) kick off the new series, "The agent is in" and take a look at the policy analyzer in CFEngine Enterprise Mission Portal. ## Video {#video} The video recording is available on YouTube: {{< youtube YYS1CUggomU >}} At the end of every webinar, we stop the recording for a nice and relaxed, off-the-record chat with attendees. [Join the next webinar to not miss this discussion](https://us02web.zoom.us/webinar/register/WN_JPz9ADvBRS2Ih8SaT0z2Uw). ## Links {#links} - [Using Policy Analyzer to develop and debug CFEngine Policy](/blog/2021/using-policy-analyzer-to-develop-and-debug-cfengine-policy/) - Connect w/ [Cody](https://www.linkedin.com/in/codyvalle/), [Craig](/author/craig-comstock/), or [Nick](https://www.linkedin.com/in/hithisisnick/) - [All Episodes](/webinars) -------------------------------------------------------------------------------- title: "Collaboration and sharing - Our plans for 2021" url: /blog/2021/collaboration-and-sharing/index.md date: "2021-04-15" -------------------------------------------------------------------------------- As we've hinted at before, 2021 will be a big year for CFEngine. In the summer, we will release CFEngine 3.18 LTS. This is the first LTS release with [Compliance Reports](/videos/?videoID=1jNxP0WVEyI), [Custom Promise types](/blog/2020/introducing-cfengine-custom-promise-types/), and [all of the other improvements we've made over the past year](/blog/2020/2020-retrospective/). ## Collaboration In addition to implementing valuable functionality for our users, we are focusing on better ways of interacting with them, and more opportunities for contribution, collaboration and sharing. The beginning of this was [the introduction of GitHub Discussions](/blog/2021/introducing-github-discussions-for-cfengine/) - a platform where users can ask questions, submit ideas, or show off their CFEngine-related creations. One month later, in March, [we launched our new website](/blog/2021/new-website-and-swag-giveaway/). The new website should make it easier for users to find what they're looking for, and also has some sections with great content, such as [videos](/videos), [case studies](/case-studies), and [white papers](/white-papers). Now, we are excited to share our plans for the rest of the year. ## Sharing We've gotten consistent feedback that users want more examples and places to find policy that is easy to reuse. This is especially important for new people who are just beginning to use CFEngine. They want to solve their first use case, deploy their first application, or get ideas for what is possible. You can automate anything in CFEngine policy language, but it's not always obvious how, especially to inexperienced users. Modules, as well as bundles and bodies, can be imported and used in policy language, to abstract away low level details. But there is no obvious place to look for what modules and bundles you should use. Additionally, if you download / copy-paste someone's code to reuse, there are no good mechanisms for keeping these dependencies up to date. In 2021, we will solve this. By the end of the year, we are launching tooling and a new website for sharing CFEngine modules and policy. The tooling will allow you to download dependencies, combine them into your policy set, and keep them up to date. Your data, your policy, and your dependencies will be separate. This is important as it makes updates easy, and allows policy and modules to be reused by different users, and in different environments, without editing the source files or manually copy-pasting anything. On the website you will find modules made by the CFEngine team, as well as contributors, with easy instructions on how to get started. There will also be categories for the most popular content as well as our recommendations. All of this will be available to both open source and enterprise users, allowing everyone to share and collaborate. ## Community feedback In order to make the content as appealing and useful as possible, we are looking for your ideas. Please tell us what you first wanted to automate, or what promise types you think we should add, in this discussion: https://github.com/cfengine/core/discussions/4593 We intend to collaborate with the community throughout the year, sharing our design, technical details, and asking for your opinions. Stay tuned! -------------------------------------------------------------------------------- title: "Feature preview: Trigger agent runs and report collection from Mission Portal" url: /blog/2021/trigger-agent-runs-and-report-collection/index.md date: "2021-03-31" -------------------------------------------------------------------------------- If you are debugging issues with a host, it is quite common to want to make changes to CFEngine policy, and speed up the process of fetching, evaluating and reporting for that host. You can do this by running `cf-runagent` and `cf-hub` from the command line, now we've brought this functionality into Mission Portal: [![UI Screenshot showing play and update buttons, with a tooltip saying Trigger Agent Run.](images/trigger.png)](images/trigger.png) You can see the feature in action, here: {{< youtube wMm0o_-xvrU >}} This feature will be part of CFEngine Enterprise 3.18 LTS, releasing this summer. To test it today, [use our nightly builds](/blog/2020/cfengine-nightlies/). For more CFEngine videos, check out [our video library](/videos/). **Edit (June 28th 2021):** [3.18.0 is now released](/blog/2021/cfengine-3-18-released-extensibility), with a new and refreshed UI. There is an updated video for this feature, which shows it in the new UI: {{< youtube -ZAMFb2msgA >}} -------------------------------------------------------------------------------- title: "Using Policy Analyzer to develop and debug CFEngine policy" url: /blog/2021/using-policy-analyzer-to-develop-and-debug-cfengine-policy/index.md date: "2021-03-29" -------------------------------------------------------------------------------- I have a [setup at home](./thinking-at-a-different-level-policy-changes-via-git-repo/) where I keep a local git server running on a Raspberry Pi 3 which contains personal/work journal, dotfiles and a personal policy repository. It was set up manually so before adding a new git repository for a family [password store](https://www.passwordstore.org/) I set about retrofitting the configuration in CFEngine. The goal in this blog is to ensure that what I have already is managed by CFEngine and that what I want to add, `/srv/git/passwords.git`, is created. My policy buddy, [Nick Anderson](https://www.linkedin.com/in/hithisisnick/), mentioned a strategy that he liked for working with CFEngine and policy: Inventory, Classify, then Promise. ## Setup For this setup I already had two physical servers: hub and git to work on. In order to make the steps easier to reproduce and follow along I will use our [Vagrant Environment](https://docs.cfengine.com/docs/3.15/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) and use the `host001` VM provided as the git server. After running `vagrant up` you should have Mission Portal available with two hosts bootstrapped: `hub` and `host001`. [![Mission Portal Inventory Report shows 2 hosts bootstrapped and reporting.](images/pa-two-hosts-present.png)](/blog/2021/images/pa-two-hosts-present.png) If you are impatient (and I suspect most of us are when playing around) you can shorten the feedback time for policy changes by configuring the agent schedule to every minute instead of the default of every 5 minutes. In [Augments](https://docs.cfengine.com/lts/reference-language-concepts-augments.html) (`/var/cfengine/masterfiles/def.json`) on hub add the following: ```json { "vars": { "control_executor_splaytime": "1", "control_executor_schedule": ["any::"], "control_hub_hub_schedule": ["any::"] } } ``` The above is a complete and working augments file. If you have other customizations in there you will have to adjust accordingly. After making this change, update the running policy as root with ```bash /var/cfengine/bin/cf-agent -KIf update.cf ``` After installing and logging in to Mission Portal I visit the Policy Analyzer app tab on the lower-left and enable Policy Analyzer. [![Mission Portal UI with Policy Analyzer Tab selected and Enable button present](images/pa-before-enable.png)](/blog/2021/images/pa-before-enable.png) Then wait 1-2 minutes and refresh the page. You should see a tree of your policy similar to this: [![Policy Analyzer UI with policy tree view](images/pa-enabled.png)](/blog/2021/images/pa-enabled.png) I also need to setup my initial state on `host001` with some directories. This is what I already have on my real git server. ```bash mkdir -p /srv/git/dotfiles.git mkdir -p /srv/git/journal.git mkdir -p /srv/git/personal-policy.git ``` ## Inventory Let's write the policy to promise a set of sub-dirs in `/srv/git`, or rather sort of "inventory" them the temporary way. :) Find out what is on the system(s) that is interesting to me. In this case I want to know a few things about what I setup manually: - Which repo directories are setup in `/srv/git`? - Is there a git user and group on the system? - Are these directories owned by the git user and group? - Are the permissions as I want them on `/srv/git` directories? In some cases I just want to know for a while what is going on and not monitor it over time. In this case I make promises for things and mark them as `action_policy => "warn"` in an `action body` so that I can see them as NOT KEPT in Policy Analyzer. At a later date I can change them to actuate their effects after I am confident they will "do the right thing". Assuming you are using the Vagrant environment, let's set up git version control. This allows us to track our changes, and revert (undo) if necessary. ```bash cd /var/cfengine/masterfiles git init git add . git commit -m initial ``` Now let's set up a minimal `def.json` to enable _autorun_ services, the most common way to work on specific use cases in policy. (This will also include the "update fast" bits we added earlier) ```json { "classes": { "services_autorun": ["any::"], "cfengine_internal_purge_policies": ["any::"], }, "vars": { "control_executor_splaytime": "1", "control_executor_schedule": ["any::"], "control_hub_hub_schedule": ["any::"] } } ``` And add the `git_server` policy at `/var/cfengine/masterfiles/services/autorun/git_server.cf`: ```cf3 bundle agent git_server { meta: "tags" slist => { "autorun" }; vars: "_repos" slist => { "dotfiles", "journal", "personal-policy" }; "_basedir" string => "/srv/git"; files: "$(_basedir)$(const.dirsep)." create => "true", action => warn; "$(_basedir)$(const.dirsep)$(_repos).git$(const.dirsep)." create => "true", action => warn; } body action warn { action_policy => "warn"; } ``` Wait for a few minutes for the policy to be updated both on `hub` and `host001` and refresh Policy Analyzer. The red box with `5` indicates that there are 5 promises which are not kept (errors): [![Policy Analyzer UI shows 5 not kept promises inside tree view.](images/pa-red5.png)](/blog/2021/images/pa-red5.png) One not kept promise is the methods promise which calls the `git_server` bundle. Since we are only calling the `git_server` bundle from this one methods promise we can ignore this not kept promise and focus on the underlying not kept promises in the `git_server` bundle. Another not kept promise is for the `/srv/git/` and sub-directories not existing on the hub. This is because so far we have not classified this policy to only apply to `host001` our "git" server. [![Policy Analyzer UI shows details in a table for 5 not kept promises.](images/pa-red5-table.png)](/blog/2021/images/pa-red5-table.png) ## Classify Now we will classify this policy so that it only applies to `host001` by adding a class-guard to the meta promise which activates `git_server`. ```cf3 meta: host001:: "tags" slist => { "autorun" }; ``` Wait a few minutes and see that Policy Analyzer is once again clean and empty of not kept promises (no red warning counter box!). Navigate to `services/autorun/git_server.cf` and select the _Kept_ checkbox at the top of Policy Analyzer to see which promises are kept. [![Policy Analyzer UI shows promises kept in tree view.](images/pa-kept-tree.png)](/blog/2021/images/pa-kept-tree.png) The table shows what we expect for the current state of `host001`: [![Policy Analyzer UI shows details in a table for kept promises.](images/pa-kept-table.png)](/blog/2021/images/pa-kept-table.png) Now let's add "passwords" to our list of git repos: ```cf3 "_repos" slist => { "dotfiles", "journal", "personal-policy", "passwords" }; ``` Save that and wait a minute or two and refresh Policy Analyzer to see that we have Not Kept promises again. [![Policy Analyzer UI shows 2 not kept promises inside tree view, 1 in lib and 1 in services.](images/pa-missing-passwords-tree.png)](/blog/2021/images/pa-missing-passwords-tree.png) Navigate again to `services/autorun/git_server.cf` in the tree. The table shows the specific promises: [![Below the Policy Analyzer tree view and code view there is a table of results for the selected file. It shows 1 promise not kept - the passwords.git promise.](images/pa-missing-passwords-table.png)](/blog/2021/images/pa-missing-passwords-table.png) ## Promise Now that we see clearly what is not kept and have checked that it's what we want, we remove the `action => warn` in our policy and let CFEngine fix the situation. ```cf3 files: "$(_basedir)$(const.dirsep)." create => "true"; "$(_basedir)$(const.dirsep)$(_repos).git$(const.dirsep)." create => "true"; ``` Save that change, wait a few minutes (stand up! stretch! do a pushup or wiggle your ears!) and refresh Policy Analyzer. Check the _Repaired_ checkbox to see that the `/srv/git/passwords.git` repo directory has been created. [![Policy Analyzer UI shows 2 not kept promises inside tree view, 1 in lib and 1 in services.](images/pa-repaired-tree.png)](/blog/2021/images/pa-repaired-tree.png) ## To Review Policy Analyzer is a handy tool you can use to visualize your policy alongside the current last outcome of each promise. While developing policy you can add an action body to any type of promise to see what would be broken and where so you can classify your promises properly and only make changes where appropriate: ```cf3 bundle agent main { files: "/my/dir/." create => "true", action => warn; } body action warn { action_policy => "warn"; } ``` ## Try it! Download [CFEngine Enterprise](/free-download) free for up to 25 hosts and try out Policy Analyzer! We'll be working on more improvements as we go and would love to get your feedback as well. -------------------------------------------------------------------------------- title: "Customer highlight: HIPAA Vault" url: /blog/2021/customer-highlight-hipaa-vault/index.md date: "2021-03-25" -------------------------------------------------------------------------------- As an MSSP (Managed Security Service Provider), HIPAA Vault relies on CFEngine to automate & secure their infrastructure on behalf of their customers. HIPAA Vault has been a longtime CFEngine Community user since 2012 and recently upgraded to Enterprise in order to boost their infrastructure visibility through Mission Portal and features like Compliance Reporting that help them provide a more secure & compliant cloud hosting solution. >We chose CFEngine over others because it is the most trusted and reliable solution in the market. We are more interested in dependability than having the new shiny tool that just hit the market. says Gil Vidals, Founder & CEO of HIPAA Vault Prior to CFEngine, HIPAA Vault would use various scripts to manage their infrastructure that eventually caused too many headaches to manage and ultimately slowed them down. Security & compliance are paramount to the business, so automating their infrastructure wasn't just about making day-to-day management quicker & easier, but also about reducing the aspect of human error, which is one of the top contributors to cybersecurity breaches. >Gil continues, It used to take us several days to setup a new customer environment manually. This includes installing things like the backup agent, monitoring & IDS, remote management software, etc. By leveraging CFEngine, we were able to automate these tasks without human intervention while avoiding the element of human error. [**Click here**](/case-studies/hipaa-vault/) **to download the case study and learn more about how HIPAA Vault leverages CFEngine Enterprise.** -------------------------------------------------------------------------------- title: "New website & SWAG giveaway" url: /blog/2021/new-website-and-swag-giveaway/index.md date: "2021-03-18" -------------------------------------------------------------------------------- ## New CFEngine Website Today we're excited to announce the release of our new website. This has been several months in the making and just one of many big announcements we have planned for 2021. Our goal in redesigning the website is simply to enhance your experience when visiting. Whether you're an IT Manager evaluating our Enterprise feature set, a Systems Administrator interested in trying CFEngine for the first time or a long time community member, we hope our new website helps you find exactly what you're looking for quickly and easily. We hope you enjoy the improved experience! ## SWAG Giveaway We recently introduced [GitHub Discussions](https://github.com/cfengine/core/discussions) as another way to stay connected to the CFEngine community. If you haven't had the opportunity to join, now is a great time. Come hang out, [share your CFEngine story](https://github.com/cfengine/core/discussions/4546) and we will send you some new CFEngine SWAG as a thank you. We look forward to seeing you there! -------------------------------------------------------------------------------- title: "Comparing Ansible and CFEngine" url: /blog/2021/comparing-ansible-and-cfengine/index.md date: "2021-02-25" -------------------------------------------------------------------------------- Generally speaking, CFEngine and Ansible can be used to solve the same problems, but their approaches are different. In this blog post I'd like to discuss the different approaches, their consequences, some advantages of each tool, and even using them together. ## CFEngines autonomous agents CFEngine works by installing and running an agent on every host of your infrastructure. It is *distributed*, each CFEngine agent will evaluate its policy periodically and independently. They rely on a centralized hub for refreshing policy and reporting. Updating the policy, enforcing it, and reporting on the results are decoupled - each of these 3 steps can happen with different configurations / schedules. The policy **update** is only needed when there is new policy available, so the agents won't re-transfer all the policy files they already have. Comparing the versions of policy on the machine and avoiding re-transfers can save a lot of bandwidth on large infrastructures. If the CFEngine hub is unavailable, agents can be configured to copy policy from a backup server, or the update can be postponed until the hub is available. The actual **enforcement** of the policy, checking the state of the system and making changes, does not rely on the centralized hub. Enforcement can happen if the host is offline or the hub is unavailable. Frequent policy enforcement is possible, even if reporting and policy update is less frequent (to save network traffic). **Reporting** necessarily involves the network and getting reporting data from hosts to the centralized hub. To reduce bandwidth, CFEngine only transmits the differences during each report collection, so if no variables change, the necessary data transfer is minimal. For the reporting aspect, the hub is a single point of failure, but this can be solved with [the High Availability (HA) setup](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-high-availability.html). Using HA, reporting will still work if the hub goes offline - another one will take over. ## Ansibles orchestration Ansible lets you *orchestrate* - tell it to do something and watch it go out to all hosts and make it happen. You don't have to install an agent \- as long your hosts have ssh and python (with some dependencies) you are good to go. It uses `ssh` to connect to all your hosts, and run python code there to perform the changes you want. Like `ssh`, you can use it from the command line, and see the results as they happen. When working with multiple hosts, parallelization can speed up the process, [this can be configured with an option for number of forks](https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html#setting-the-number-of-forks). AWX / Ansible Tower, gives you a nice GUI, and allows you to run your ansible playbooks from a dedicated machine. In the GUI, you can schedule runs periodically, and access the results whenever you want. Whether you are using the CLI, or the GUI, **playbooks** let you perform common tasks by only specifying the necessary bits in a YAML file. There's a great community of users writing them, so chances are somebody has already solved the problem you are facing and you can reuse their playbook. ## Scalability comparison We wanted to compare the performance of both tools, at various scales (number of hosts). The example use case is to install and configure web servers (nginx on Google Cloud VMs), with a load balancer running HAProxy. To make it interesting, the webservers are a mix of Ubuntu 18 and CentOS 7. How long does it take to run the same Ansible playbook / CFEngine policy? ### Setup We tried to make the comparison as fair as possible - in both cases, using the same infrastructure in Google Cloud, comparing the same host counts, and sticking to defaults. The previously mentioned `fork` option for Ansible was the only exception we made, as Ansible would be very slow with the default (5). Testing different values up to 20 gave better performance, beyond that it did not improve, on our setup. The setup is open source and can be found [on GitLab](https://gitlab.com/Northern.tech/CFEngine/web-cluster). Anyone interested can reproduce the setup, and we encourage you to try it and make suggestions for improvements. ### Results [![](images/cfengine_ansible_scale.png)](/blog/2021/images/cfengine_ansible_scale.png) Starting with just a few hosts, Ansible finishes quickly, within seconds. It is quite easy to see a linear relationship between the time and number of hosts. On our setup it looks like each host adds approximately 4 seconds. We stopped at **500 hosts** as the relationship up to that point was very linear, and each test run would exceed 30 minutes at that point. Based on the linear relationship, we projected numbers between each test and from 500-1000, as can be seen in the graph above. In CFEngine, you don't schedule a specific job, but rather update the policy and wait for hosts to fetch it and report the results back. For this reason, we observed the load of the hub to confirm whether CFEngine could finish the entire batch within the 5 minute interval. (A CPU load graph is shown on page 4 of [this white paper](/white-papers/ansible-and-cfengine-scalability)). The default schedule is that policy is updated and enforced every 5 minutes, and the hub collects reports every 5 minutes (independently). This gives us a worst case round trip time of 10 minutes, which we've plotted in the graph. We stopped at **3500 hosts** as this was our quota in Google Cloud. The CFEngine hub still had an idle period with capacity for more hosts within the 5 minute interval. The trend here shows a clear pattern - adding more for Ansible to do (more hosts or playbooks) increases the total runtime. The Ansible approach is convenient and easy to understand - but it does add some overhead as there is no agent, ssh connections must be established, files must be copied over and commands must be run, while the connection is held open and the Tower / CLI is waiting. All of this overhead does add up, especially at large scale, or if you want to run many of these on a regular schedule. For a small number of hosts, or one-off changes, the overhead might be worth it, and the convenience of direct (imperative) control and immediate feedback is more valuable. ### Notes The exact numbers here are not that important. They depend a lot on the use case, your hardware and network, and how you've configured the tool. There are many improvements that could be made, and a second version, an optimized comparison, would be interesting. As an example, we could easily set CFEngine's schedules to 1 minute, bringing the roundtrip time down to 2 minutes. Or speeding it up even further by requesting more agent runs with the cf-runagent functionality. Similarly, there are many strategies for [improving the performance of Ansible](https://cdn2.hubspot.net/hub/330046/file-476015569-pdf/pdf_content/Scaling_and_Performance_of_the_Ansible_Management_Toolchain.pdf). ## Conclusions Although CFEngine and Ansible *can* be used for the same things, which is the better option depends a lot on the situation. For most organizations, it makes sense to have both solutions available, and use the most appropriate one depending on the context. ### When to use Ansible In any situation where you'd SSH into a single machine (or perhaps a dozen with `parallel ssh`) and run a command, Ansible is great: - ssh / shell commands translate well into ansible. - Ansible essentially automates the `parallel ssh` workflow for you. - As long as you are working with a limited number of hosts, your job will finish quickly and you get immediate feedback. - There's a great library of playbooks available to perform common tasks. ### When to use CFEngine In some situations, you might have security / compliance requirements, or scale / complexity factors which make CFEngine a better option. For example, if you: - Want to make a change to a large part of your infrastructure - without being limited by the rate your Ansible Tower can connect to your hosts with SSH. - Need the policy to be continuously enforced and have up to date [compliance reports](https://www.youtube.com/watch?v=1jNxP0WVEyI) to prove it. - Find that some things are just too slow to do in the orchestrated Ansible way - either due to a large number of hosts or complex playbooks. ### Using both, together Perhaps even more interestingly, there are situations where you can use both, together: - [Using Ansible to install CFEngine](/blog/2022/cfengine-ansible-module/) - Using CFEngine to install Ansible dependencies - [Using CFEngine's reporting data as variables inside ansible playbooks](https://www.youtube.com/watch?v=VAOAtWy-das) - [Using CFEngine's inventory to target machines for ansible jobs:](/blog/2021/cfengine-inventory-to-ansible/) [![](images/cfengine_ansible_targeting.png)](/blog/2021/cfengine-inventory-to-ansible/) Aleksey Tsalolikhin at Vertical Sysadmin has some examples of how they use CFEngine and Ansible in a complementary fashion, in [this blog post](https://verticalsysadmin.com/blog/cfengine-and-ansible-are-complementary/). ## Additional reading We have published two white papers which discuss CFEngine and Ansible: 1. [Ansible|CFEngine - Analysis and methodologies](https://cfengine.com/company/blog-detail/ansiblecfengine/) 2. [Ansible and CFEngine Scalability](https://cfengine.com/company/blog-detail/ansible-and-cfengine-scalability-white-paper/) If you have any questions or comments about the white papers, [we'd love to have your feedback](https://github.com/cfengine/core/discussions/4488). -------------------------------------------------------------------------------- title: "cf-remote is now available via pip" url: /blog/2021/cf-remote-is-now-available-via-pip/index.md date: "2021-02-11" -------------------------------------------------------------------------------- `cf-remote` is a tool for downloading and installing/deploying CFEngine. It automates a lot of the things you have to do before CFEngine is actually installed on your infrastructure, such as provisioning cloud instances, downloading CFEngine installers, copying them to remote hosts and installing / bootstrapping. To make it as easy as possible to get started with `cf-remote` and CFEngine, it is now [available on pypi](https://pypi.org/project/cf-remote/). ## Getting started Installing cf-remote is as easy as: pip install cf-remote Use `pip3` if that is your `pip` for python 3. Requires python 3.6 or newer, but only on the one machine where you run `cf-remote`. Adding `sudo` to the beginning of that command might be necessary, depending on your operating system and how you installed python. Hosts are managed using SSH, and simple shell commands (no python on the remote machine). Provided you already have passwordless SSH access, installing CFEngine is as simple as: cf-remote install --hub user@1.2.3.4 --bootstrap 4.3.2.1 In this example, `user@1.2.3.4` is where we want to SSH, and `4.3.2.1`, is the IP we want CFEngine bootstrapped to. (It might not be the same public IP address / interface that you use for SSH). The latest LTS release will be automatically downloaded and installed. ## Downloading nightly CFEngine packages `cf-remote` can do much more. For example, if you want to test the latest features, you can download our nightly packages from the master branch: cf-remote --version master download hub deb The tags, `hub` and `deb`, are optional, and mean we will only download `.deb` packages for CFEngine Enterprise Hub. ## Provisioning instances in AWS `cf-remote spawn` can create cloud instances on demand, for example in AWS EC2, but you'll have to add some credentials and settings. To see examples of how to do this and other things, like deploying new versions of policy, look at the README file: ## Getting help If you run into any issues, have any questions, or need help, feel free to start a thread on our GitHub Discussions: ## How we make new releases of cf-remote We are using GitHub Actions, triggered by GitHub releases, to make releases entirely from the GitHub UI, and publish to [pypi.org](https://pypi.org/), without committing or tagging anything from the command line. If you are interested in how this works or have some python projects you'd like to publish, see this video: {{< youtube U-aIPTS580s >}} -------------------------------------------------------------------------------- title: "Introducing GitHub Discussions for CFEngine" url: /blog/2021/introducing-github-discussions-for-cfengine/index.md date: "2021-02-01" -------------------------------------------------------------------------------- We are excited to announce that CFEngine is now using [GitHub Discussions](https://github.blog/2020-05-06-new-from-satellite-2020-github-codespaces-github-discussions-securing-code-in-private-repositories-and-more/). GitHub Discussions is a feature of GitHub repos, and similar to Q\&A platforms like Stack Overflow, and other online forums. After testing it out for a few weeks we are pleased with how it works and want to encourage all our users to try it. We hope this fuels more discussion and sharing among CFEngine users - it is easy to discover on GitHub, many of you already have GitHub users, the UI is nice, and so the barrier to entry should be very low. ## What to post in GitHub Discussions [Our GitHub Discussions](https://github.com/cfengine/core/discussions) are inside the [`cfengine/core`](https://github.com/cfengine/core) repository, but can be used for anything related to CFEngine. (Including CFEngine policy language, CFEngine Enterprise, [cf-remote](https://github.com/cfengine/cf-remote), and community projects). [CFEngine Enterprise is free for up to 25 hosts](https://cfengine.com/download/), but only paying customers get access to professional support. This provides a nice alternative for free users of both CFEngine Community and CFEngine Enterprise; community-powered support. There are 4 categories, here with some examples of the content we'd like to see: - [General](https://github.com/cfengine/core/discussions/categories/general) - General open-ended discussion about CFEngine - Anything which doesn't fit in the other categories - [Ideas](https://github.com/cfengine/core/discussions/categories/ideas) - Suggestions for new features in CFEngine - Suggestions for what we could do differently around CFEngine (docs, websites, blog posts, etc.) - [Q\&A](https://github.com/cfengine/core/discussions/categories/q-a) - Ask a question about how to do something in CFEngine Policy language - Ask a question about how some part of CFEngine works - Ask for help troubleshooting a problem or understanding some logs / error messages - Similar to Stack Overflow, the author of a post should choose an accepted answer - [Show and tell](https://github.com/cfengine/core/discussions/categories/show-and-tell) - Share a blog post, video, book, or other content you made about CFEngine - Post and discuss a CFEngine policy you wrote - Show off a Dashboard or Report you are using in CFEngine Mission Portal - Announce releases of your CFEngine module or policy framework - To keep things organized, we ask that you post subsequent updates for the same project as comments in the thread (not as separate discussions) Note that in many cases we might create and/or link to JIRA tickets, since that is our primary ticketing system, for both bugs and features. ## What happens with existing channels In short: nothing. The existing channels remain as-is: - [The mailing list](https://groups.google.com/forum/#!forum/help-cfengine) is the most similar to GitHub Discussions - a place to ask questions or share content about CFEngine - We are very happy with the mailing list and think it's a valuable resource for those who use it, we don't want to disturb that - [The \#cfengine IRC channel on libera.chat](https://web.libera.chat/?channel=#cfengine) is for real-time chat ([also available via Gitter.im](https://gitter.im/cfengine/core)) - [The bug tracker (JIRA)](https://northerntech.atlassian.net/) is still the place for bug reports, and where we organize all the tickets we are working on - In many cases we will end up creating tickets from GitHub Discussions, for example if we see something that is a bug ## Get started To begin, we encourage everyone to post an introduction in the welcome thread: -------------------------------------------------------------------------------- title: "How to implement CFEngine custom promise types in bash" url: /blog/2021/how-to-implement-cfengine-custom-promise-types-in-bash/index.md date: "2021-01-29" -------------------------------------------------------------------------------- This blog post will focus on the bash programming part of implementing a promise type. To understand what custom promise types are, and how to use them, [you should read the introduction first](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/). To implement a custom promise type in CFEngine, you need a promise module. The module is an executable, and can be written in any language. It's possible to write one from scratch, but to make it as easy as possible, we decided to provide libraries for common programming languages. In [our previous blog post](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/), we showed how to implement modules in Python. That's great, it's a powerful, expressive and readable programming language, however there is one drawback; installing python. Many systems don't have python already, or have a version which is too old. So you will need to add policy to install / update Python, to make sure modules work correctly everywhere. CFEngine runs on a wide variety of platforms; Linux, BSD, Windows, Mac, HP-UX, IBM AIX, Solaris, etc. Most of these platforms have some version of bash already installed (except for Windows). Thus, by writing promise modules in bash we can make them very portable, and easy to use everywhere without additional dependencies. ## Using the CFEngine promise module bash library [The library](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/bash/cfengine.sh) implements the protocol and the main loop of the program, receiving and answering requests from `cf-agent`. The following examples will be based on it, but authors of shell-based custom promise modules are welcome to look through [the source code](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/bash/cfengine.sh) and use it as they find fitting. To use the library, the module script should set environment variables and define functions, then source the library script and call `module_main` function with two arguments: The module name and version. The most trivial example, which does nothing, looks like this: ```bash do_evaluate() { response_result="kept" } . "$(dirname "$0")/cfengine.sh" module_main "trivial_example_module" "1.0" ``` The function `do_evaluate` is called during policy evaluation. It must set `response_result` variable, otherwise policy evaluation will fail. After sourcing the library, `module_main` is called with module name and version as two arguments. **Note:** Generally, you cannot add `echo` or `printf` statements like you would in a normal shell script, to print error messages or log what the module is doing. Your module code should be _silent_ (not print to stdout). This is because the module is communicating with cf-agent on standard output, so its output needs to follow the protocol. ### Logging To send log messages to the agent (which will in the end be printed to the user), follow one of these examples: ```bash log error "Could not create file 'blah'" log info "Created file 'blah'" log verbose "File 'blah' already exists, skipping" log debug "Module 'create_file_module' received request to validate 'blah'" ``` Logging should only be added inside `do_evaluate` and `do_validate` functions. What to log and what the different levels mean is [explained in the specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html#logging). If you need additional logging, which will not be printed to CFEngine users, write to a file instead: ```bash echo "Example promise module starting" > example_module.log ``` ### Variables Currently, there are three optional variables, which instruct the library to check validity of promise attributes. `required_attributes` - space-separated list of attributes which must be presented in each promise. If provided - each promise is checked for presence of these attributes, and fails validation if at least one of them is not present. If not provided, no such check is performed. `optional_attributes` - space-separated list of attributes which are allowed, but not required, in each promise. Example: `optional_attributes="this that"`. `all_attributes_are_valid` - `yes` or `no` (default: `no`). By default, any attribute which is not in list of required or optional attributes is considered invalid and renders promise invalid. If set to `yes`, then no extra check for optional attributes is performed - all attributes are considered valid. ### Functions Currently, there is only one required function - which performs evaluation (the step which makes changes to the system), and three optional ones: to validate and to do setup / teardown if needed. `do_evaluate` - required, the function which does promise evaluation. As inputs it can use request*promiser variable and a set of `request_attribute*\*`variables - one for each specified attribute. As outputs, it must set`response_result`variable to one of`kept`, `not_kept`, or `repaired` strings. `do_validate` - optional, called to validate promise. Uses same inputs as `do_evaluate`, as output it can change `response_result` to `valid` or `invalid`. Note that the library itself checks presence of `required_attributes` and absence of invalid attributes (if requested), and sets `response_result` accordingly before calling this function. This function can change value of this variable. Also, you can check, for example, that `request_promiser` is a valid absolute pathname (that it starts with `/`). `do_initialize` - called once at the very beginning of module lifecycle, before any other function. Can be used for initialization / set up, if needed. `do_terminate` - called once at the very end of module lifecycle, after any other function. Can be used for cleaning up, if needed. ## Copy promises Let's have a look at a less trivial example - copying files: ```bash required_attributes="from" optional_attributes="" all_attributes_are_valid="no" do_evaluate() { if diff -q "$request_attribute_from" "$request_promiser" 2>/dev/null; then response_result="kept" return 0 fi if cp -a "$request_attribute_from" "$request_promiser"; then response_result="repaired" else response_result="not_kept" fi } . "$(dirname "$0")/cfengine.sh" module_main "cp" "1.0" ``` This declares `cp` custom promise type, which has one required attribute: `from`, and no optional attributes. In `do_evaluate`, value of this attribute is available in `request_attribute_from` variable. Also note that instead of overwriting the target file every time, this promise module first compares source and destination, and does the copying only if they differ. Promise result reflects that: the promise will be "kept" if source and destination were the same before policy evaluation, "repaired" if they were different, but `cp` command succeeded, and "not_kept" if `cp` command failed. In the policy it can be used like this: ```cf3 promise agent cp { interpreter => "/bin/bash"; path => "$(sys.inputdir)/cp.sh"; } bundle agent main { cp: "/tmp/dst" from => "/tmp/src"; } ``` When evaluating above policy, `do_evaluate` will be called with variables like this: ```bash request_attribute_from="/tmp/src" request_promiser="/tmp/dst" ``` ## Backup promises As a final example, I've added my pet project, namely [backup3](https://github.com/Lex-2008/backup3), as a CFEngine custom promise module: ```bash required_attributes="from" optional_attributes="root" all_attributes_are_valid="no" do_evaluate() { # Prepare environment for wrapped script BACKUP_ROOT="$request_attribute_root" BACKUP_BIN="$(dirname "$0")/backup3" BACKUP_KEEP_TMP="1" run_this() { run_rsync always "$request_promiser" "$request_attribute_from" } # call the script . "$BACKUP_BIN/backup.sh" # Analyse its results if [ "$?" != 0 -o ! -f "$BACKUP_TMP".files ]; then response_result="not_kept" rm -f "$BACKUP_TMP".sql "$BACKUP_TMP".files return 0 fi # Note: $BACKUP_TMP".files file contains lists of created and deleted files, # one file per line, separated by line saying "separator", and first line # being literally "first line". So if script succeeded, but there were no # changed files, then this file has only 2 lines. local filelist_lines="$(cat "$BACKUP_TMP".files | wc -l)" if [ "$filelist_lines" -gt 2 ]; then response_result="repaired" else response_result="kept" fi # Clean up rm -f "$BACKUP_TMP".sql "$BACKUP_TMP".files } . "$(dirname "$0")/cfengine.sh" module_main "backup" "1.0" ``` From first three lines we see that promises for this module must always have a "from" attribute, and, optionally, a "root" attribute (if not provided, it's expected that the `backup3/backup.sh` script called in `do_evaluate` will figure it out). In the `do_evaluate` function we see a different order of operations compared to the "copy promise": instead of checking if something needs to be done first, promise module always first calls the `backup3/backup.sh` script, and only then checks if it did anything. That's due to how the backup3 is written: it always first runs `rsync`, and then checks if it did anything. The promise module script analyzes one of its temporary files and concludes if there were any changes or not, based on its contents. ## Summary That's it. Whether you're using bash, python, or another programming language, extending CFEngine policy language has never been easier. If you encounter any issues or have any questions when working with promise modules, we encourage you to [post them here](https://github.com/cfengine/core/discussions/categories/q-a). We are excited to see what modules the community creates, and to share more than just example modules soon. If you want to learn more, here are some useful resources: - [Introducing CFEngine custom promise types](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/) - [How to implement CFEngine custom promise types in Python](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/) - [Custom promise outcomes in Mission Portal](https://cfengine.com/company/blog-detail/custom-promise-outcomes-in-mission-portal/) - [Promise module specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html) - [Copy files promise example](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/cp/cp.sh) - [Backup promise example](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/backup/backup.sh) - [CFEngine promise module bash library](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/bash/cfengine.sh) -------------------------------------------------------------------------------- title: "Custom promise outcomes in Mission Portal" url: /blog/2021/custom-promise-outcomes-in-mission-portal/index.md date: "2021-01-21" -------------------------------------------------------------------------------- [CFEngine 3.17.0](https://cfengine.com/blog/2020/cfengine-3-17-released-flexibility/) introduced [custom promise types](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/), which enable CFEngine users to extend core functionality and policy language in a simple way. As an example of the power and simplicity of this new feature, I will show a promise type that helps to observe a website's status. The module which implements this promise type was written in a couple of hours. ## Creating a promise type for whether a site is up We will use Python and the [CFEngine library](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) to implement a promise module. Our previous blog post, ["How to implement CFEngine custom promise types in Python"](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/), explains this in detail. First, we import dependencies, `re` for regex matching, `ssl` to configure ssl verification, `urllib` for making HTTP requests, and finally `PromiseModule, ValidationError, and Result` from the `cfengine` module which implements the necessary protocol, results, logging and error handling. ```python import re import ssl import urllib.request import urllib.error from cfengine import PromiseModule, ValidationError, Result ``` We then create a subclass of `PromiseModule` and use the constructor to define the promise module name and version: ```python class SiteUpPromiseTypeModule(PromiseModule): def __init__(self): super().__init__("site_up_promise_module", "0.0.3") ``` After that, we can implement validation under the `validate_promise` method. In my case, I check if the URL is valid, for that I'm performing pattern matching and if the URL isn't valid then I raise a `ValidationError`. ```python def is_url_valid(self, url): regex = re.compile( r"https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()!@:%_\+.~#?&\/\/=]*)", re.IGNORECASE, ) return re.match(regex, url) is not None def validate_promise(self, promiser, attributes, metadata): if not self.is_url_valid(promiser): raise ValidationError(f"URL '{promiser}' is invalid") ``` Finally, we need to implement `evaluate_promise` a method that performs the main business logic. Here I check if `skip_ssl_verification` exists in attributes and if so create unverified SSL context to avoid SSL verification. This can be useful if you are setting up a web server with a self signed certificate, for example for testing. Then I do HTTP requests and if there are no errors I return the promise result as kept: ```python def evaluate_promise(self, promiser, attributes, metadata): url = promiser ssl_ctx = ssl.create_default_context() if ( "skip_ssl_verification" in attributes and attributes["skip_ssl_verification"] == "true" ): ssl_ctx = ssl._create_unverified_context() error = None try: code = urllib.request.urlopen(url, context=ssl_ctx).getcode() self.log_verbose(f"Site '{url}' is UP!") return Result.KEPT except urllib.error.HTTPError as e: # HTTPError exception returns response code and useful when handling exotic HTTP errors error = f"Site '{url}' is DOWN! Response code: '{e.code}'" except urllib.error.URLError as e: # URLError is the base exception class that returns generic info error = f"Site '{url}' is DOWN! Reason: '{e.reason}'" except Exception as e: error = str(e) assert error is not None self.log_error(error) return Result.NOT_KEPT ``` Download the [full module here](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/site-up/site_up.py). ## Using the new custom promise type Now that we're done with the custom promise type module we need to write the CFEngine policy to use it. First, we define the custom promise type specifying the path to the module that implements the custom promise type as well as the interpreter that should be used. ```cf3 promise agent site_up { path => "/var/cfengine/modules/promises/site_up.py"; interpreter => "/usr/bin/python3.8"; } ``` After that, we can define the bundle as we usually do in CFEngine. The newly added promise type `site_up` is available and we set a URL as the promiser. ```cf3 bundle agent site_up_autorun { meta: "tags" slist => { "autorun" }; site_up: "http://172.28.128.10"; "https://cfengine.com/"; "https://cfengine2.com/"; "https://unavailable.com" skip_ssl_verification => "true"; } ``` Download the [policy here](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/site-up/site_up.cf). ## The result ### Policy analyzer Let's check the result in Mission Portal's [Policy analyzer](https://cfengine.com/company/blog-detail/from-black-box-magic-to-automation-transparency/) to see what's happening in our CFEngine policy: [![](/blog/2021/images/policy_analyzer_site_up.png)](/blog/2021/images/policy_analyzer_site_up.png) On the left side, policy files are listed in the file manager. On the right side, we see the content of the selected policy file, in my case [`site_up.cf`](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/site-up/site_up.cf). Not kept promises (inaccessible sites in other words) are highlighted in red. Below you can see all information presented in the table with log messages that fully describe the root cause. ### Email notification Additionally, we can create an alert to get an email notification once a site is down: [![](images/create_alert_site_up.png)](images/create_alert_site_up.png) ### Compliance report Finally, we can use the conditions from the alerts as part of a compliance report (introduced in [CFEngine 3.17.0](https://cfengine.com/blog/2020/cfengine-3-17-released-flexibility/)): [![](images/compliace_report_site_up.png)](images/compliace_report_site_up.png) Watch the [Compliance Reports video](https://www.youtube.com/watch?v=1jNxP0WVEyI) for more details about this new type of report: {{< youtube 1jNxP0WVEyI >}} ## Summary To summarize, we've gone through a few steps: 1. Writing a promise module (about 40 lines of Python code) 2. Defining the promise type in policy (pointing to the module) 3. Adding policy which uses the promise type, `site_up`, to monitor some websites 4. Seeing whether the sites are up in mission portal, by looking at the promise outcomes in Policy Analyzer 5. Creating alerts for when sites go down 6. Creating a compliance report, a checklist of all the websites we need to stay up This is just a small example, but it shows how easy it is to build a new promise type in CFEngine and see the results of those promise evaluations in Mission Portal. Here, we made a simple promise type that primarily provides reporting data, it doesn't change anything on the system, but custom promise types are not limited to reporting, they can be used to manage new types of resources as well. To achieve this, we'd just expand the `evaluate_promise` method, with more code for what we want to check and change on the system. For an idea of how to do this, see the example in [our previous blog post](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/). -------------------------------------------------------------------------------- title: "How to serve policy from a local git server" url: /blog/2021/thinking-at-a-different-level-policy-changes-via-git-repo/index.md date: "2021-01-19" -------------------------------------------------------------------------------- Several months ago I started the practice of using CFEngine Enterprise and its Mission Portal UI on a daily basis to manage the connected devices in my home. To start, I brought up an old desktop machine, `cfengine-hub`, to use as my hub and [downloaded Enterprise](https://cfengine.com/download/), which is free for use up to 25 hosts. The next step in using [best practices](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-best-practices.html) is to deploy policy from a version control repository. I use a local git server named `git-server-zero` instead of GitHub or GitLab as I like to be independent of the cloud when possible due to privacy and [environmental](https://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of-energy-belying-industry-image.html) concerns. ![raspberry pi zero device plugged in via usb cable to back of wifi router, low power, local control, private cloud](images/IMG_20210106_111241.jpg) I will use the Mission Portal [Version Control Repository](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-hub_administration-policy-deployment.html#configure-the-upstream-vcs) settings section to setup this repo as the source of policy for `cfengine-hub`. ## Configuring the git server You can choose whether to use a username/password combination for your git URL or a passphrase-less key. I am choosing the latter. root@cfengine-hub:~# ssh-keygen -o Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/hub_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/hub_rsa. Your public key has been saved in /root/.ssh/hub_rsa.pub. The key fingerprint is: SHA256:[a-z] root@cfengine-hub In case the account on the hub gets compromised I want to prevent writes from this user. I created a new account called `hubgituser` on `git-server-zero` and ensured that `/srv/git` on `git-server-zero` has only read permissions for this user. Afterwards, I added the public key generated on `cfengine-hub` to the `hubgituser`'s authorized keys. root@git-server-zero:/home/hubgituser/.ssh# cat /root/.ssh/hub_rsa.pub >> authorized_keys Now check that the connection works between my hub and the git server, while ensuring that I have no saved keys in my ssh-agent. root@cfengine-hub:~# eval $(ssh-agent -s) Agent pid 6334 root@cfengine-hub:~# ssh-add -D All identities removed. root@cfengine-hub:~# ssh-add .ssh/hub_rsa Identity added: .ssh/hub_rsa (root@cfengine-hub) root@cfengine-hub:~# ssh hubgituser@git-server-zero Linux git-server-zero 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Wed Jul 15 16:14:33 2020 from 192.168.1.2 hubgituser@git-server-zero:~ $ Ok\! Confirmed\! ## Configuring Mission Portal Version Control Repository Now to Mission Portal and entering the details as mentioned in [Configuring Upstream VCS](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-hub_administration-policy-deployment.html#configure-the-upstream-vcs). Login and navigate to Settings and then Version control repository: ![](images/dark-mode-vcs-settings.png) Enter details, select the private key file and click Apply. ## Initial Sync of Policy If you use Mission Portal to configure VCS settings then the initial sync will occur automatically after a little while. To test things out right away or if you configured manually then run the following: root@cfengine-hub:/opt/cfengine/dc-scripts# cf-agent -KIf update.cf --define cfengine_internal_masterfiles_update info: Executing 'no timeout' ... '/var/cfengine/httpd/htdocs/api/dc-scripts/masterfiles-stage.sh' info: Command related to promiser '/var/cfengine/httpd/htdocs/api/dc-scripts/masterfiles-stage.sh' returned code defined as promise kept 0 info: Completed execution of '/var/cfengine/httpd/htdocs/api/dc-scripts/masterfiles-stage.sh' Any troubles? Run the `masterfiles-stage.sh` script in debug mode with `-D` or also with `bash -x`. For example, once I tried these steps and got a password prompt which I should not: root@cfengine-hub:~# bash -x /var/cfengine/httpd/htdocs/api/dc-scripts/masterfiles-stage.sh -D Option Deploy Dir: /var/cfengine/masterfiles Option Params File: /opt/cfengine/dc-scripts/params.sh Cloning into bare repository '/opt/cfengine/git_git_server_zero__srv_git_masterfiles'... hubgituser@git-server-zero's password: ## Test Policy Deployment Now let's try making a change to policy and see if it goes through. I push a commit to add a hashed password for a user promise in a file called `personal.yaml`: craig@laptop:~/src/masterfiles$ git push Enumerating objects: 9, done. Counting objects: 100% (9/9), done. Delta compression using up to 4 threads Compressing objects: 100% (5/5), done. Writing objects: 100% (5/5), 1.02 KiB | 1.02 MiB/s, done. Total 5 (delta 3), reused 0 (delta 0) To git-server-zero:/srv/git/masterfiles.git 265bb3b..292ee56 master -> master And then ran the agent asking for updates again: info: Copied file '/var/cfengine/masterfiles/services/autorun/personal.yaml' to '/var/cfengine/inputs/services/autorun/personal.yaml.cfnew' (mode '600') info: Backed up '/var/cfengine/inputs/services/autorun/personal.yaml' as 'var/cfengine/inputs/services/autorun/personal.yaml.cfsaved' info: Moved '/var/cfengine/inputs/services/autorun/personal.yaml.cfnew' to '/var/cfengine/inputs/services/autorun/personal.yaml' info: Updated '/var/cfengine/inputs/services/autorun/personal.yaml' from source '/var/cfengine/masterfiles/services/autorun/personal.yaml' on 'localhost' info: Purged '/var/cfengine/inputs/services/autorun/personal.yaml.cfsaved' copy dest directory info: Purged '/var/cfengine/inputs/cf_promises_release_id.cfsaved' copy dest directory info: files promise '/var/cfengine/inputs' repaired The change is present in `/var/cfengine/inputs`\! # foobarbaz demo password - hashed_password: $6$hckmJCcp/vOWNjeH$s3FsHKoBuoVpKJW0XhuojyDK5yp1FqixtX5RE.kP8cwWpphmbIpeJ5lvsaKjDPp9I5FBRpBnXOHfrHW2nIM4k1 So let's see if it takes effect on `cfengine-hub` by trying to login as agent. I can login with `foobarbaz`\! Success\! $ hostname cfengine-hub $ whoami agent Now I can make changes to policy, push the changes to my git server and the change will roll out to all my agents/hosts automatically. -------------------------------------------------------------------------------- title: "Ansible and CFEngine scalability white paper" url: /blog/2021/ansible-and-cfengine-scalability-white-paper/index.md date: "2021-01-12" -------------------------------------------------------------------------------- Scalability is an important feature of any infrastructure management solution. Either the to-be-managed infrastructure is big already or it is expected to grow as the business grows. Over time more and more resources are needed for CI/CD pipelines and more customers use the product(s). Generally, growing a business means more traffic and requests need to be handled by the infrastructure. Hence, scalability is an important metric for comparing infrastructure management tools when deciding which one to use. Or which *ones*. Read [our latest white paper](/white-papers/ansible-and-cfengine-scalability), benchmarking and comparing the scalability of Ansible and CFEngine for large scale infrastructure management: {{< pdfpreview "/white-papers/Ansible_CFEngine_White_Paper_2_Scalability_preview.png" "Preview of the first page of the white paper fading into the background." "/white-papers/ansible-and-cfengine-scalability">}} -------------------------------------------------------------------------------- title: "CFEngine 2020 Retrospective" url: /blog/2020/2020-retrospective/index.md date: "2020-12-16" -------------------------------------------------------------------------------- 2020 is nearly over, and we'd like to take a couple of minutes to reflect on our year as well as provide a sneak peek into what you can expect from us in 2021. Although it has been a year full of distractions, the CFEngine team has continued to make significant strides when it comes to product improvements and new features that help our users. ## Build powerful compliance reports based on important inventory data Compliance reports are high level reports, allowing you to see how compliant your infrastructure is. Checks are run against reporting data from all of your hosts, or a filtered subset, to find how many of them meet a certain compliance requirement. They are easy to build and use, entirely from the UI, with no programming needed, but flexible and powerful, allowing you to use package version information, custom inventory from your policy or even SQL queries if you need it. Compliance reports are not limited to one regulation or framework, like [CIS](https://www.cisecurity.org/cis-benchmarks/) or [HIPAA](https://www.hhs.gov/hipaa/for-professionals/index.html), but allow you to build your own checklists, based on your organization's requirements and compliance goals. See Compliance Reports in this video: {{< youtube 1jNxP0WVEyI >}} A beta version was [introduced in CFEngine 3.16](https://cfengine.com/blog/2020/cfengine-3-16-released-compliance/), and we keep working on it to make it production ready for CFEngine 3.18 LTS. We encourage everyone to try it out, [using nightly packages](https://cfengine.com/company/blog-detail/cfengine_nightlies/), and send us feedback/suggestions via [the public issue tracker](https://northerntech.atlassian.net), or the [customer support portal](https://support.northern.tech). ## Ansible and CFEngine white paper In September, the team posted a white paper, discussing how [Ansible](https://www.ansible.com/) and CFEngine can solve problems together. Read the full report: {{< pdfpreview "/white-papers/Ansible_CFEngine_White_Paper_1_preview.png" "Preview of the first page of the white paper fading into the background." "/white-papers/ansible-and-cfengine/">}} ## Extend CFEngine policy language and manage new resources [Custom promise types](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/) allow you to extend CFEngine policy language, and manage more complex resources, software or services with ease. Details of what files to edit, commands to run, etc. can be abstracted away in a promise module, and the policy file can focus on the high level business logic. It's now possible to implement promise types to manage git repos, AWS instances, kubernetes pods, docker containers, software firewalls, etc. Previously, you would have to know C, edit the CFEngine source code, compile and deploy new binaries to do this. Now you can add new promise types as modules, written in any programming language, for example Python. Custom promises are easy to use, and once added, the syntax is the same as built in promises: ```cf3 promise agent git { path => "/var/cfengine/modules/promises/git.py"; interpreter => "/usr/bin/python3"; } bundle agent __main__ { git: "/opt/cfengine/masterfiles" repository => "https://github.com/cfengine/masterfiles"; } ``` If you are interested in contributing new promise types to CFEngine, by writing your own modules, take a look at [our latest blog post](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/). ## Refresh reporting data with the click of a button You can now get fresh reporting data from a host, by clicking the refresh button in its host info page: ![](images/trigger_report_collection.png) This is very useful if you are debugging issues with a given host, or want to see how it's affected by a change you just made. ## Simulate changes made by CFEngine Understanding what CFEngine is doing is crucial, especially for complex environments and larger policy sets. It can be a challenge to know exactly what changes will be made after a policy change, upgrade or on a brand new machine. The new `--simulate` modes address this, expanding existing dry-run functionality to simulate file changes and show you manifests or diffs once done. You can run a new policy set, or new CFEngine binary, and see exactly what the difference will be: $ /var/cfengine/bin/cf-agent -KI test.cf --simulate=diff warning: All changes in files will be made in the '/var/cfengine/state/29830.changes' chroot info: Updated content of '/tmp/hello-world' with content 'Hello, world! ' info: files promise '/tmp/hello-world' repaired info: Showing diff for changed files (in the changes chroot) =========================================================================== --- original /tmp/hello-world +++ changed /tmp/hello-world @@ -1 +1 @@ -Some data +Hello, world! ## Distribute secrets with end-to-end file encryption We also added a new binary, `cf-secret` which allows you to encrypt secrets for specific hosts. See it in action: {{< youtube lJ7UymB6G-k >}} `cf-secret` makes it easier to distribute things like API tokens and encryption keys between hosts, using [end-to-end encryption](https://en.wikipedia.org/wiki/End-to-end_encryption). While originally released in CFEngine [3.16.0](https://cfengine.com/blog/2020/cfengine-3-16-released-compliance/), we decided to include the binary in version [3.15.3](https://cfengine.com/company/blog-detail/cfengine-3-12-6-and-3-15-3-released/) as well, to give it to our LTS-users sooner. ## Contributions and statistics Looking at our issue tracker, we find that across CFEngine Community and Enterprise, there were 208 bugs fixed and 294 tasks done. That gives us a total of 502 tickets, [122 of which relate to the open source project and are publicly accessible](). Across our 5 [Open Source](https://northern.tech/blog/open-source-at-northern-tech) repos there were 25 unique contributors, here are their commit counts: | [libntech](https://github.com/cfengine/libntech) | [core](https://github.com/cfengine/core) | [masterfiles](https://github.com/cfengine/masterfiles) | [buildscripts](https://github.com/cfengine/buildscripts) | [documentation](https://github.com/cfengine/documentation) | Committer | | ------------------------------------------------ | ---------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------- | | | 8 | 7 | 63 | 4 | [Aleksei Shpakovskii](https://github.com/Lex-2008) | | | | 1 | | | [Aleksey Tsalolikhin](https://github.com/atsaloli) | | | | | | 1 | [Alexis Mousset](https://github.com/amousset) | | | | 3 | | | [Bas van der Vlies](https://github.com/basvandervlies) | | 1 | | | | | [Chris Rorvick](https://github.com/crorvick) | | 6 | 42 | 17 | 4 | 12 | [Craig Comstock](https://github.com/craigcomstock) | | 1 | 1 | | | | [David Lee](https://github.com/tdlee) | | | 1 | | | | [dependabot](https://github.com/dependabot) | | | 1 | 6 | 14 | 9 | [Ihor Aleksandrychiev](https://github.com/aleksandrychev) | | | 1 | | | | [Jon Henrik Bjørnstad](https://github.com/jonhenrik13) | | 3 | 9 | | | 2 | [Karl Hole Totland](https://github.com/karlhto) | | | | | | 1 | [Klaus Kämpf](https://github.com/kkaempf) | | | 1 | | | | [Kristian Amlie](https://github.com/kacf) | | | 1 | | | | [Lars Erik Wik](https://github.com/larsewi) | | 4 | 5 | | | 7 | [Lluis Campos](https://github.com/lluiscampos) | | | | 4 | | | [Mike Weilgart](https://github.com/mikeweilgart) | | | 32 | 57 | 1 | 44 | [Nick Anderson](https://github.com/nickanderson) | | 3 | | | | | [odidev](https://github.com/odidev) | | 48 | 274 | 9 | 7 | 21 | [Ole Herman Schumacher Elgesem](https://github.com/olehermanse) | | | 1 | 1 | | | [Ole Petter](https://github.com/oleorhagen) | | | | 1 | | | [Oliver Kett](https://github.com/systemheld) | | | 1 | | | | [Patrick Steinhardt](https://github.com/pks-t) | | | | 2 | | | [Ted Zlatanov](https://github.com/tzz) | | | | 1 | | | [troymuller](https://github.com/troymuller) | | 10 | 211 | 11 | 22 | 4 | [Vratislav Podzimek](https://github.com/vpodzime) | We'd like to thank all our contributors, including those not listed, who contributed through other channels, like the [issue tracker](https://northerntech.atlassian.net/), [IRC](https://web.libera.chat/?channel=#cfengine), or [mailing list](https://groups.google.com/g/help-cfengine). (Commit stats generated on 2020-12-16, using command: `git shortlog --summary --since 2020-01-01 --no-merges`). ## Next year As we wrap up 2020, the team is excited for what we have in store for 2021. We won't spoil everything now, but there will be some big announcements in the first half of 2021. Here are a few things you can look forward to: - Innovative promise types to manage modern resources or services - More power within the Mission Portal Web UI to perform actions and make changes to your infrastructure - Better integrations with cloud providers and other devops software you might already be using - New ways to contribute to CFEngine and communicate with other CFEngine users - CFEngine 3.18 LTS - all the features of [3.16](https://cfengine.com/blog/2020/cfengine-3-16-released-compliance/) and [3.17](https://cfengine.com/blog/2020/cfengine-3-17-released-flexibility/) in a Long Term Supported release We'd like to say thank you to all the dedicated CFEngine users out there. Your continued feedback & enthusiasm keeps us motivated and focused on making meaningful improvements for CFEngine. We wish you all a healthy & prosperous rest of the year and we'll see you in 2021\! -------------------------------------------------------------------------------- title: "How to implement CFEngine custom promise types in Python" url: /blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/index.md date: "2020-12-08" -------------------------------------------------------------------------------- This tutorial focuses on how to write a promise module, implementing a new CFEngine promise type. It assumes you already know how to install promise modules and use custom promise types, as shown in [the previous blog post](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/). ## Why Python? Promise modules can be written in any programming language, but there are some advantages of using python: - Readable and beginner friendly language / syntax - Popular and familiar to a lot of people, also used in some CFEngine package modules - Big standard library, allowing you to reuse data structures, parsers, etc. without reinventing the wheel or adding dependencies - [Official CFEngine promise module library](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) - Most of the code needed is already done (protocol, parsing, etc.) - You can focus on only the business logic, what is unique to your new promise type With that said, there are some reasons why you might not always want to use python: - 1 extra dependency (Must install python on all hosts which use module) - Relatively slow, in some situations (slow to start, slow for CPU-bound workloads) For these reasons we've ensured the implementation of custom promise types is language-agnostic, and we will post more tutorials for different programming languages soon. ## The git promise type As an example, we'll start writing a promise type to manage `git` repos. It won't be complete in a short tutorial, but the aim is to show you how to get started implementing your own promise types. ### Doing nothing, successfully Before we do anything else, we should make a module which speaks the protocol and doesn't cause any errors, even if it doesn't do anything. We'll start by importing the necessary parts of the library: ```python import os from cfengine import PromiseModule, ValidationError, Result ``` The 3 parts we need from the library are: - `PromiseModule` - The class which implements the module, we need to subclass this - `ValidationError` - Exception to raise when there are problems with validating policy - `Result` - Enumeration of the different results a promise evaluation can have Now, we simply make a subclass of `PromiseModule`, give our module a name and version number, and specify what should be done on validate and evaluate requests: ```python class GitPromiseTypeModule(PromiseModule): def __init__(self): super().__init__("git_promise_module", "0.0.1") def validate_promise(self, promiser, attributes, metadata): pass def evaluate_promise(self, promiser, attributes, metadata): return Result.KEPT ``` This module is now ready to use, it's called `git_promise_module`, version `0.0.1`, it performs no validation, and always returns promise result kept (success). `validate_promise` should raise an exception if validation fails, for example if the policy is using an attribute name which does not exist. If no exception is raised, the promise is considered valid and ready to be evaluated. `pass` is a no-op in python, it does nothing except indicate that this method is empty. The evaluate method should return one of 3 promise results: - `Result.KEPT` - No changes were necessary, everything was already as promised - There should be a verbose log message which explains why nothing was done - `Result.REPAIRED` - Changes were made to the system, and they were successful, the system state is now as promised - There should be an info log message which explains what was changed on the system - `Result.NOT_KEPT` - Changes were necessary, but they failed - There should be an error log message explaining the failure The last thing we need to do in the python code is to start the module: ```python if __name__ == "__main__": GitPromiseTypeModule().start() ``` The full module now looks like this: ```python import os from cfengine import PromiseModule, ValidationError, Result class GitPromiseTypeModule(PromiseModule): def __init__(self): super().__init__("git_promise_module", "0.0.1") def validate_promise(self, promiser, attributes, metadata): pass def evaluate_promise(self, promiser, attributes, metadata): return Result.KEPT if __name__ == "__main__": GitPromiseTypeModule().start() ``` [As explained previously](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/), the recommended way to install the module is to put it in `masterfiles/modules/promises`: ```bash $ cp *.py /var/cfengine/masterfiles/modules/promises/ && cf-agent -KIf update.cf && cf-agent -KI ``` Assuming that you have `cfengine.py` and `git.py` in the current working directory. Remember to run this command again after making changes to the module. It is now ready to be used from policy: ```cf3 promise agent git { path => "/var/cfengine/modules/promises/git.py"; interpreter => "/usr/bin/python3"; } bundle agent __main__ { git: "/opt/cfengine/masterfiles" repository => "https://github.com/cfengine/masterfiles"; } ``` If we run this policy, nothing happens, successfully: $ cf-agent -KI ./test.cf $ echo $? 0 We can see some interesting details from the debug logs: $ cf-agent -K --debug ./test.cf | grep git debug: Received header from promise module: 'git_promise_module 0.0.1 v1 json_based' debug: Received line from module: '{"operation": "validate_promise", "promiser": "/opt/cfengine/masterfiles", "attributes": {"repo": "https://github.com/cfengine/masterfiles"}, "result": "valid"}' debug: Received line from module: '{"operation": "evaluate_promise", "promiser": "/opt/cfengine/masterfiles", "attributes": {"repo": "https://github.com/cfengine/masterfiles"}, "result": "kept"}' verbose: Promise with promiser '/opt/cfengine/masterfiles' was kept by promise module '/var/cfengine/modules/promises/git.py' verbose: P: END git promise (/opt/cfengine/masterfiles) ### Minimum Viable Promise type In the methods we receive `promiser` (string) and `attributes` (dict) as parameters. Adding some validation is quite straight forward: ```python def validate_promise(self, promiser, attributes, metadata): if not promiser.startswith("/"): raise ValidationError(f"File path '{promiser}' must be absolute") if "repository" not in attributes: raise ValidationError(f"Attribute 'repository' is required") ``` We can also make our module do something, like running git clone: ```python def evaluate_promise(self, promiser, attributes, metadata): url = attributes["repository"] folder = promiser if os.path.exists(folder): self.log_verbose(f"'{folder}' already exists, nothing to do") return Result.KEPT self.log_info(f"Cloning '{url}' -> '{folder}'...") os.system(f"git clone {url} {folder} 2>/dev/null") if os.path.exists(folder): self.log_info(f"Successfully cloned '{url}' -> '{folder}'") return Result.REPAIRED else: self.log_error(f"Failed to clone '{url}' -> '{folder}'") return Result.NOT_KEPT ``` Here we see that the promise evaluation has 3 possible outcomes, along with appropriate log messages: - _Kept_ - The folder already existed, and nothing was done, this is explained in verbose output - _Repaired_ - The git clone command was run, this is shown in info output, since it changes the system - _Not kept_ - The git clone command was run, but it failed, this deserves an error message This is a very simplified implementation. It assumes that if the folder exists, it is a correct clone, but we can improve it later. Here is the full module, again: ```python import os from cfengine import PromiseModule, ValidationError, Result class GitPromiseTypeModule(PromiseModule): def __init__(self): super().__init__("git_promise_module", "0.0.1") def validate_promise(self, promiser, attributes, metadata): if not promiser.startswith("/"): raise ValidationError(f"File path '{promiser}' must be absolute") if "repository" not in attributes: raise ValidationError(f"Attribute 'repository' is required") def evaluate_promise(self, promiser, attributes, metadata): url = attributes["repository"] folder = promiser if os.path.exists(folder): self.log_verbose(f"'{folder}' already exists, nothing to do") return Result.KEPT self.log_info(f"Cloning '{url}' -> '{folder}'...") os.system(f"git clone {url} {folder} 2>/dev/null") if os.path.exists(folder): self.log_info(f"Successfully cloned '{url}' -> '{folder}'") return Result.REPAIRED else: self.log_error(f"Failed to clone '{url}' -> '{folder}'") return Result.NOT_KEPT if __name__ == "__main__": GitPromiseTypeModule().start() ``` ### Adding more useful functionality There are some obvious areas of improvement: - We want to be able to specify a branch other than the default - We want the module to be able to `git pull` new commits - We want the module to clone recursively (submodules) Adding these are left as an exercise for the reader. ## Additional resources If you'd like to learn more, here are a few useful resources: - [Installing and using promise modules](https://cfengine.com/blog/2020/introducing-cfengine-custom-promise-types/) - [Custom promise type specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html) - [Minimal example module in python - git promise type](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/git-using-lib/git_using_lib.py) - [CFEngine promise module library for python](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) - [How to write a custom promise type in Bash](https://cfengine.com/company/blog-detail/how-to-implement-cfengine-custom-promise-types-in-bash/) -------------------------------------------------------------------------------- title: "Introducing CFEngine custom promise types" url: /blog/2020/introducing-cfengine-custom-promise-types/index.md date: "2020-12-03" -------------------------------------------------------------------------------- In [CFEngine 3.17](https://cfengine.com/blog/2020/cfengine-3-17-released-flexibility/), custom promise types were introduced. This allows you to extend policy language, managing resources which don't have built in promise types. The implementation of custom promise types is open source, and available in both CFEngine Enterprise and CFEngine Community. To implement a new custom promise type, you need a promise module. (The promise type is what you use in policy language (the concept), while the module is the underlying implementation - can be a python script, compiled executable or similar). ## Promise modules can be written in any programming language The only requirement is that it communicates with the CFEngine agent on standard output and standard input. CFEngine (cf-agent) spawns a subprocess with the module, and sends it requests to validate and evaluate promises. In this blog post, we will show how to install and use an example module. This example module is written in python, and uses our [cfengine python library](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) to simplify the implementation a lot. [The module itself is just 39 lines of python code](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/git-using-lib/git_using_lib.py). Please note that it is just a minimal example, it does not implement everything you would expect from a complete, production ready, promise type. If you are interested in more technical details, you can [read the specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html#protocol). ## The limitations of built in promise types Anything can be managed using a combination of `commands`, `files`, `users`, and `packages` promises, but this can be tedious and error prone. As an example, if you want to use `git` to clone a repo, and then keep it up to date, you might end up doing something like this: - A `packages` promise to install `git` - A `users` promise to create the user which will clone `git` repositories - Some `files` promises to create folders, git config files and enforce permissions - Some `commands` to do what you want; `git clone`, `git pull`, `git config`, `git remote`, `git checkout` The `commands` promises can be troublesome; the order matters, they have dependencies and they are not always idempotent. To do this well, you should use some combination of promise handles, `if` attributes, and classes to control which commands to run, and when. With a custom promise type, you can abstract things away from the policy writer. When writing policy, they can focus on what `git` repo to clone where, and write clean, readable policy, while leaving the technical details up to the module. ## Installing a Promise Module Using Masterfiles Policy Framework, adding modules is quite straight forward, you just copy the files into the right folder. Let's use the example git promise type: ```bash $ git clone --recursive https://github.com/cfengine/core $ git clone --recursive https://github.com/cfengine/modules $ mkdir /var/cfengine/masterfiles/modules/promises $ cp core/misc/custom_promise_types/cfengine.py /var/cfengine/masterfiles/modules/promises/ $ cp modules/promise-types/git/git.py /var/cfengine/masterfiles/modules/promises/ ``` (It is recommended to keep your masterfiles in a git repo, and you should commit your modules to it). A normal policy run copies the modules to the right locations, and adjusts permissions: ```bash $ cf-agent -Kf update.cf && cf-agent -K $ ls -l /var/cfengine/inputs/modules/promises/ total 12 -rw------- 1 root root 5416 Dec 3 14:50 cfengine.py -rw------- 1 root root 1352 Dec 3 14:50 git_using_lib.py $ ls -l /var/cfengine/modules/promises/ total 12 -rwxr-xr-x 1 root root 5416 Dec 3 14:50 cfengine.py -rwxr-xr-x 1 root root 1352 Dec 3 14:50 git_using_lib.py ``` This is identical to how package modules are handled. The update policy copies `/var/cfengine/masterfiles` from the policy server to `/var/cfengine/inputs` on all hosts. Modules are then placed in `/var/cfengine/modules/promises` with correct permissions, which is the preferred location to run them from. ## Defining and using custom promise types in policy To define a new promise type in policy, we use the `promise` block: ```cf3 promise agent git { path => "/var/cfengine/modules/promises/git_using_lib.py"; interpreter => "/usr/bin/python3"; } ``` For now, you'll have to do this in any policy file where you use the promise type. In the future we'll let you define all your custom promise types in one place. After that, you can use the `git` promise type just like any other promise type: ```cf3 bundle agent main { git: "/home/ubuntu/kubernetes" repository => "https://github.com/kubernetes/kubernetes"; } ``` Here is the complete example policy (`/var/cfengine/masterfiles/services/autorun/git_clone_kubernetes.cf`): ```cf3 promise agent git { path => "/var/cfengine/modules/promises/git_using_lib.py"; interpreter => "/usr/bin/python3"; } bundle agent git_clone_kubernetes_autorun { meta: "tags" slist => { "autorun" }; git: "/home/ubuntu/kubernetes" repository => "https://github.com/kubernetes/kubernetes"; } ``` The example uses `autorun` functionality, to be automatically added to inputs and bundlesequence. You can use `/var/cfengine/masterfiles/def.json` to enable `autorun`: ```json { "classes": { "services_autorun": ["any"] } } ``` ## The result To summarize, at this point you should have added 4 files: 1. `/var/cfengine/masterfiles/modules/promises/cfengine.py` - The CFEngine promise module library 2. `/var/cfengine/masterfiles/modules/promises/git_using_lib.py` - The git example module 3. `/var/cfengine/masterfiles/services/autorun/git_clone_kubernetes.cf` - Our policy which uses the `git` promise type to clone the `kubernetes` repository 4. `/var/cfengine/masterfiles/def.json` - Augments file to enable `autorun` Remember to commit these files to your masterfiles git repo. We can do a full agent run again, and observe the result: ```bash $ cf-agent -Kf update.cf && cf-agent -KI info: Cloning 'https://github.com/kubernetes/kubernetes' -> '/home/ubuntu/kubernetes'... info: Successfully cloned 'https://github.com/kubernetes/kubernetes' -> '/home/ubuntu/kubernetes' [...] $ ls /home/ubuntu/kubernetes/ BUILD.bazel SECURITY_CONTACTS hack CHANGELOG SUPPORT.md logo CHANGELOG.md WORKSPACE pkg CONTRIBUTING.md api plugin LICENSE build staging LICENSES cluster test Makefile cmd third_party Makefile.generated_files code-of-conduct.md translations OWNERS docs vendor OWNERS_ALIASES go.mod README.md go.sum ``` Cloning kubernetes or any big git repo for the first time can take a few minutes, be patient. If you run the policy again, the module will see that the repo is already cloned, and will not do anything. A real git promise module should, in that case, go inside the folder and run more git commands, for example `git pull`. ## Additional resources If you'd like to learn more, here are a few useful resources: - [How to write a custom promise type in Python](https://cfengine.com/blog/2020/how-to-implement-cfengine-custom-promise-types-in-python/) - [Custom promise type specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html) - [Minimal example module in python - git promise type](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/examples/git-using-lib/git_using_lib.py) - [CFEngine promise module library for python](https://github.com/cfengine/modules/blob/54a41ff399e0f918fa5cf50f4e1e4b858d678915/libraries/python/cfengine.py) - [How to write a custom promise type in Bash](https://cfengine.com/company/blog-detail/how-to-implement-cfengine-custom-promise-types-in-bash/) -------------------------------------------------------------------------------- title: "CFEngine 3.17 released - Flexibility" url: /blog/2020/cfengine-3-17-released-flexibility/index.md date: "2020-11-18" -------------------------------------------------------------------------------- We are pleased to announce the release of CFEngine 3.17.0, with the theme _Flexibility_\! This is a non-LTS release and allows the CFEngine community to test the features which will be in CFEngine 3.18.0 LTS (Summer 2021). ## What's new? ### A new look - Mission Portal Dark Mode Mission portal now gives you the option of switching to an alternate color theme, dark mode: ![](images/dark_mode.png) ### Trigger report collection from Host Info Page You no longer have to wait for the next reporting interval, or use the command line to get updated reports. Click the button on the host info page to trigger a report collection: ![](images/trigger_report_collection.png) (cf-hub will collect new reports from the host and the host info page will be refreshed in a few seconds). ### Simulate changes made by CFEngine before performing them The new `--simulate` option allows you to test your policy and see what changes it would make. Here is a basic example policy: ```cf3 bundle agent main { files: "/tmp/hello-world" content => "Hello, world!$(const.n)"; } ``` Running this with `--simulate=diff` we can see the differences, before and after the files promise changed the file: ``` $ echo "Some data" > /tmp/hello-world $ /var/cfengine/bin/cf-agent -KI test.cf --simulate=diff warning: All changes in files will be made in the '/var/cfengine/state/29830.changes' chroot info: Updated content of '/tmp/hello-world' with content 'Hello, world! ' info: files promise '/tmp/hello-world' repaired info: Showing diff for changed files (in the changes chroot) =========================================================================== --- original /tmp/hello-world +++ changed /tmp/hello-world @@ -1 +1 @@ -Some data +Hello, world! ``` The files promise was simulated in a _change-root_ which is cleaned up afterwards, so no changes were made to the system. This allows you to easily audit the impact of a new CFEngine policy or upgrade. ### Manage new types of resources with custom promise types The CFEngine agent can now interact with promise modules, allowing users to implement and share new promise types. You add a promise type using a `promise` block, giving paths to the module and interpreter to run it. Once added, custom promises are straight forward to use, there is no special syntax: ```cf3 promise agent git { path => "/var/cfengine/modules/promises/git_using_lib.py"; interpreter => "/usr/bin/python3"; } bundle agent __main__ { git: "/opt/cfengine/masterfiles" repository => "https://github.com/cfengine/masterfiles"; } ``` Modules can be implemented in any programming language, for example bash or python. You can get started adding your own promise types now, see [the specification](https://docs.cfengine.com/docs/3.21/reference-promise-types-custom.html) for more information. We will soon share more examples and details in a separate blog post. ### Expressive new conditions using regular expressions We've introduced new conditions which utilize regular expressions. As an example, filtering a few different versions of an OS becomes much easier: ![](images/regex.png) Additionally, we've introduced conditions for whether inventory attributes are reported or not: ![](images/is_reported.png) All of the new conditions can be used in alerts, host filters, and compliance reports. ### Features already released in 3.15.3 3.17.0 also includes the improvements mentioned in the 3.15.3 release announcement: - [Synchronizing roles between Mission Portal and Active Directory](https://cfengine.com/company/blog-detail/cfengine-3-12-6-and-3-15-3-released/#active-directory-role-sync) - [Ansible compatible hosts API](https://cfengine.com/company/blog-detail/cfengine-3-12-6-and-3-15-3-released/#ansible-compatible-hosts-api) - [Pinning inventory and reporting data to customize the host info page](https://cfengine.com/company/blog-detail/cfengine-3-12-6-and-3-15-3-released/#copy-to-clipboard-buttons) - [Copy-to-clipboard buttons](https://cfengine.com/company/blog-detail/cfengine-3-12-6-and-3-15-3-released/#pin-variables-and-classes-to-host-info-page) - [Host-specific end-to-end encryption of files using cf-secret](https://cfengine.com/company/blog-detail/cf-secret/) ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.17.0 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.17/guide-latest-release-whatsnew-changelog-core.html) - [3.17.0 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.17/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.17.0 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.17/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs. ## Dependencies The table below shows versions of dependencies used in 3.17.0, compared to the latest LTS releases: | CFEngine version | 3.12.6 | 3.15.3 | 3.17.0 | Notes | | -------------------------------------------------------------------------------- | ------ | ------ | ------ | -------------------- | | [Apache](https://httpd.apache.org/) | 2.4.43 | 2.4.46 | 2.4.46 | Hub only | | [apr-util](https://apr.apache.org/) | 1.6.1 | 1.6.1 | 1.6.1 | Hub only | | [APR](https://apr.apache.org/) | 1.7.0 | 1.7.0 | 1.7.0 | Hub only | | [diffutils](https://ftpmirror.gnu.org/diffutils/) | | | 3.7 | Introduced in 3.17.0 | | [Git](https://www.kernel.org/pub/software/scm/git/) | 2.28.0 | 2.28.0 | 2.29.2 | Hub only | | [libacl](https://download.savannah.gnu.org/releases/acl/) | 2.2.53 | 2.2.53 | 2.2.53 | | | [libattr](https://download.savannah.gnu.org/releases/attr/) | 2.4.48 | 2.4.48 | 2.4.48 | | | [libcurl](https://curl.haxx.se/download.html) | 7.72.0 | 7.72.0 | 7.73.0 | | | [libgnurx](https://www.gnu.org/software/rx/rx.html) | 2.5.1 | 2.5.1 | 2.5.1 | Windows only | | [libiconv](https://ftp.gnu.org/gnu/libiconv/) | 1.16 | 1.16 | 1.16 | | | [libxml2](https://gitlab.gnome.org/GNOME/libxml2) | 2.9.10 | 2.9.10 | 2.9.10 | | | [libyaml](https://pyyaml.org/wiki/LibYAML) | 0.2.5 | 0.2.5 | 0.2.5 | | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.26 | 0.9.26 | 0.9.27 | | | [OpenLDAP](https://www.openldap.org/software/download/OpenLDAP/openldap-release/) | 2.4.53 | 2.4.53 | 2.4.56 | | | [OpenSSL](https://openssl.org/) | 1.1.1g | 1.1.1g | 1.1.1h | | | [PCRE](https://www.pcre.org/) | 8.44 | 8.44 | 8.44 | | | [PHP](https://php.net/) | 7.2.34 | 7.4.10 | 7.4.12 | Hub only | | [PostgreSQL](https://www.postgresql.org/) | 10.14 | 12.4 | 13.0 | Hub only | | [pthreads-w32](ftp://sourceware.org/pub/pthreads-win32/) | 2-9-1 | 2-9-1 | 2-9-1 | Windows only | | [rsync](https://download.samba.org/pub/rsync/) | 3.2.3 | 3.2.3 | 3.2.3 | Hub only | | [SASL2](https://cyrusimap.org/mediawiki/index.php/Downloads) | 2.1.27 | 2.1.27 | 2.1.27 | Solaris only | | [zlib](https://www.zlib.net/) | 1.2.11 | 1.2.11 | 1.2.11 | | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult - Chat with the developers and other users [on IRC](https://web.libera.chat/?channel=#cfengine) - Ask for help or start a discussion on [the mailing list](https://groups.google.com/forum/#!forum/help-cfengine) -------------------------------------------------------------------------------- title: "CFEngine 3.12.6 and 3.15.3 released" url: /blog/2020/cfengine-3-12-6-and-3-15-3-released/index.md date: "2020-11-10" -------------------------------------------------------------------------------- We are pleased to announce two new patch releases for CFEngine, version 3.12.6 and 3.15.3\! These releases mainly contain bug fixes and dependency updates, but in 3.15.3 there are also some new enhancements in Mission Portal. [The new cf-secret binary](https://cfengine.com/company/blog-detail/cf-secret/) is also included in 3.15.3 packages. ## New in Mission Portal 3.15.3 ### Synchronizing roles between Mission Portal and Active Directory When using LDAP for authentication, Mission Portal can now automatically grant roles based on the tags received from your LDAP server (for example Active Directory). This means that new users can start using Mission Portal immediately, without having to wait for an administrator to grant the appropriate roles manually. Enable this in Mission Portal Authentication Settings: [![](images/ldap_3_15_3.png)](/blog/2020/images/ldap_3_15_3.png) ### Ansible compatible hosts API [The new hosts-by-class API](https://docs.cfengine.com/docs/3.15/reference-enterprise-api-ref-host.html#hosts-list-grouped-by-hard-classes) provides lists of hosts, grouped by classes. This allows you to use CFEngine reporting data within an Ansible playbook. [Nick Anderson](https://twitter.com/cmdln_/) shows how in this demonstration: {{< youtube VAOAtWy-das >}} ### Extending the information shown on the Host Info Page The Host Info Page shows a lot of useful information out of the box, like OS, hostname, IP address, CFEngine version, etc. You can now extend what is shown, by pinning CFEngine inventory, variables, or classes. Pinned items will show up on all host info pages: [![](images/pinning_3_15_3.png)](/blog/2020/images/pinning_3_15_3.png) ### Copy-to-clipboard buttons New clipboard buttons in the host info page allow you to easily copy things like IP addresses or hostnames: [![](images/clipboard_3_15_3.png)](/blog/2020/images/clipboard_3_15_3.png) ## CFEngine 3.12 LTS end of life CFEngine 3.12 series is scheduled to reach end of life in approximately 1 year (winter 2021). There will be 1 more release before then (CFEngine 3.12.7 in summer 2021). If you are still using CFEngine 3.12, you should consider upgrading to the 3.15 series. Please reach out to [CFEngine Enterprise Support](https://support.northern.tech/) if you have any questions about upgrading, or need assistance with the process. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs: - [3.12.6 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html) - [3.12.6 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.12.6 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) - [3.15.3 Changelog for CFEngine Community](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [3.15.3 Changelog for CFEngine Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [3.15.3 Changelog for Masterfiles Policy Framework](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) (Please note that the Enterprise changelogs contain only changes specific to enterprise. To get a full overview of all changes in a version, read all 3 changelogs.) ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. | CFEngine | 3.12.5 | **3.12.6** | 3.15.2 | **3.15.3** | | ---------------------------------------------- | ------ | ---------- | ------ | ---------- | | [Apache](https://httpd.apache.org/) | 2.4.43 | **2.4.46** | 2.4.43 | **2.4.46** | | [Git](https://git-scm.com/) | 2.26.2 | **2.28.0** | 2.26.2 | **2.28.0** | | [libcurl](https://curl.haxx.se/libcurl/) | 7.70.0 | **7.72.0** | 7.70.0 | **7.72.0** | | [libyaml](https://pyyaml.org/wiki/LibYAML) | 0.2.4 | **0.2.5** | 0.2.4 | **0.2.5** | | [LMDB](https://github.com/LMDB/lmdb/) | 0.9.24 | **0.9.26** | 0.9.24 | **0.9.26** | | [OpenLDAP](https://www.openldap.org/) | 2.4.50 | **2.4.53** | 2.4.50 | **2.4.53** | | [PHP](https://php.net/) | 7.2.30 | **7.2.34** | 7.4.5 | **7.4.10** | | [PostgreSQL](https://www.postgresql.org/) | 10.12 | **10.14** | 12.2 | **12.4** | | [rsync](https://download.samba.org/pub/rsync/) | 3.1.3 | **3.2.3** | 3.1.3 | **3.2.3** | Thank you to all the developers and maintainers of Open Source Software which make CFEngine possible\! ## Contributions We encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help or start a discussion on [the mailing list](https://groups.google.com/forum/#!forum/help-cfengine) - Chat with the developers and other users [on IRC](https://web.libera.chat/?channel=#cfengine) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Ansible|CFEngine white paper" url: /blog/2020/ansiblecfengine/index.md date: "2020-09-09" -------------------------------------------------------------------------------- *Ansible* and *CFEngine* are two configuration management tools and at first glance they look like competitors - two tools dealing with the same problem, in very different ways. But are they? Maybe they are actually not dealing with the same problem and are not as incompatible as it seems. Read our [Ansible|CFEngine white paper](/white-papers/ansible-and-cfengine) providing an analysis of this area to learn more: {{< pdfpreview "/white-papers/Ansible_CFEngine_White_Paper_1_preview.png" "Preview of the first page of the white paper fading into the background." "/white-papers/ansible-and-cfengine/">}} -------------------------------------------------------------------------------- title: "Installing CFEngine Nightlies using cf-remote" url: /blog/2020/cfengine-nightlies/index.md date: "2020-08-28" -------------------------------------------------------------------------------- Nightly packages are very useful for testing new features of CFEngine. Right now (as of August 2020), nightly packages can be used to test out these new features: - [Compliance Reports.](https://www.youtube.com/watch?v=1jNxP0WVEyI) - Mission Portal Dark Mode. - New host info page with variable pinning and copy buttons. Note that these features are in development, some parts may be unfinished or buggy. Nightly packages are not supported and should not be used in production environments. `cf-remote` is a tool for automating the setup and installation of CFEngine. It allows you to easily install CFEngine packages over SSH. The tool is open source, and installation instructions are available on GitHub: ## Downloading nightlies Downloading packages using `cf-remote` is easy: $ cf-remote --version master packages hub ubuntu Available releases: 3.16.0, 3.15.x, 3.15.2, 3.15.1, 3.15.0, 3.15.0b1, 3.12.x, 3.12.5, 3.12.4, 3.12.3, 3.12.2, 3.12.1, 3.12.0 Using master: Downloading package: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.17.0a.db4bd2209~10418.ubuntu14_amd64.deb' Downloading package: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.17.0a.db4bd2209~10413.ubuntu16_amd64.deb' Downloading package: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.17.0a.db4bd2209~10418.ubuntu18_amd64.deb' $ `cf-remote` defaults to the latest LTS release, so we specify `--version master` to get the latest nightly build from the master branch. The `packages` command downloads packages. The two tags `ubuntu` and `hub` are optional, and used to filter packages. Without them, `cf-remote` would download packages for all the supported platforms. You don't have to use cf-remote to install these packages, they are just like our normal packages. ## Installing nightlies If you have ssh access to a host, with password-less sudo, you can use cf-remote to install packages: $ cf-remote --version master install --hub ubuntu@54.217.161.96 --bootstrap 172.31.18.204 ubuntu@54.217.161.96 OS : ubuntu (debian) Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : dpkg, apt Package already downloaded: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.17.0a.db4bd2209~10413.ubuntu16_amd64.deb' Copying: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.17.0a.db4bd2209~10413.ubuntu16_amd64.deb' to 'ubuntu@54.217.161.96' Installing: 'cfengine-nova-hub_3.17.0a.db4bd2209~10413.ubuntu16_amd64.deb' on 'ubuntu@54.217.161.96' CFEngine 3.17.0a.db4bd2209 (Enterprise) was successfully installed on 'ubuntu@54.217.161.96' Bootstrapping: '54.217.161.96' -> '172.31.18.204' Bootstrap successful: '54.217.161.96' -> '172.31.18.204' The `install` command will find the same packages as `packages` did. If they are already downloaded, they will be reused. That's it, the host has been bootstrapped, and Mission Portal is running. You can open Mission Portal in your browser and start exploring. (In my case above, it would be `https://54.217.161.96`). -------------------------------------------------------------------------------- title: "CFEngine 3.17.0a1-termux - better Android Termux Support" url: /blog/2020/cfengine-3-17-0a1-termux-better-android-termux-support/index.md date: "2020-08-26" -------------------------------------------------------------------------------- As a follow up to my previous "personal policy" blog I have exciting news: ### An improved CFEngine is available for [Termux](https://termux.com/)\! This provides a way to play with policy and implement policy on your non-rooted Android phone\! Version 3.17.0a1-termux is an alpha release so understand it's not heavily tested. That said, CFEngine for Termux is looking pretty awesome and useful. Highlights of features: - allow self-bootstrap to loopback since Android devices often change their IP address and bootstrapping locally seems to make some sense for a developer device and ability to play around, this is just as helpful on the desktop for that matter. - packages promises work with local masterfiles or with patched policy server masterfiles (pkg uses apt\_get which CFE supports) - since Termux supports "real" versions of commands and doesn't rely exclusively on busybox, CFEngine considers a Termux environment as a fairly full featured linux box in terms of commands and features - runs as un-privileged account, CFEngine for Termux does NOT require root - files promises work inside the `/data/data/com.termux/files` scope, not outside (unless possibly you have a rooted device, which is completely untested) - masterfiles policy framework works well, paths for common commands are modified to adjust to termux's prefix `$PREFIX` being `/data/data/com.termux/files/usr`. Some common paths are setup for creating policy that works on Termux and other unices (`etc_path, tmp_path, bin_path, var_path`). Not supported (yet): - [Termux Services](https://wiki.termux.com/wiki/Termux-services) - user promises, on Android this doesn't make much sense unless rooted and even then it is arguable - full CI/test suite passing - no allowances/testing of sleep states and termux wake lock affects on CFEngine - report collection for enterprise hub ### Installation The CFEngine package is available for Termux installations with Android 7.0 and higher (see this ticket about older Android support: https://github.com/termux/termux-packages/issues/4467). pkg install cfengine cf-agent --bootstrap 127.0.0.1 --inform | tee bootstrap.log cf-agent -KIf update.cf cf-agent -KI At that point you should have the masterfiles policy framework in place at `$PREFIX/var/lib/cfengine/inputs` and changes made to `$PREFIX/var/lib/cfengine/masterfiles` can be updated with the following sequence which I made into an alias for ease-of-use while testing/developing: alias cf='cf-agent -KIf update.cf && cf-agent -KI' And since the masterfiles directory has a rather long path I created a symlink in my home dir: cd ~ ln -s $PREFIX/var/lib/cfengine/masterfiles . Now we can play around with some features. Let's use autorun services by adding a `def.json` file at `$PREFIX/var/lib/cfengine/masterfiles` ```json { "classes": { "services_autorun": [ "any::" ], "cfengine_internal_purge_policies": [ "any::" ] } } ``` So create a small service to show off some features, create at `$PREFIX/var/lib/cfengine/masterfiles/services/autorun/termux_demo.cf` ```cf3 bundle agent termux_demo { meta: "tags" slist => { "autorun" }; classes: "in_termux" expression => "termux"; "in_linux" expression => "linux"; commands: _stdlib_path_exists_env:: "$(paths.env)"; files: "$(paths.etc_path)/ssh/sshd_config" changes => detect_all_change; # from masterfiles/lib/files.cf (stdlib) reports: "Hello from $(this.bundle)"; "paths.bin_path is $(paths.bin_path)"; termux:: "Hello, only from termux platform agents"; # users promises are not supported, this will generate an error # but not prevent policy from being evaluated in general users: "agent" policy => "present"; } ``` And run the update and agent run: $ cf error: Method 'personal' failed in some repairs error: Users promise type is not supported on this OS info: Executing 'no timeout' ... '/data/data/com.termux/files/usr/bin/env' notice: Q: "...sr/bin/env": SHELL=/data/data/com.termux/files/usr/bin/bash Q: "...sr/bin/env": PREFIX=/data/data/com.termux/files/usr Q: "...sr/bin/env": PWD=/data/data/com.termux/files/home Q: "...sr/bin/env": LOGNAME=u0_a138 Q: "...sr/bin/env": EXTERNAL_STORAGE=/sdcard Q: "...sr/bin/env": LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so Q: "...sr/bin/env": HOME=/data/data/com.termux/files/home Q: "...sr/bin/env": TMPDIR=/data/data/com.termux/files/usr/tmp Q: "...sr/bin/env": SSH_CONNECTION=127.0.0.1 42047 127.0.0.1 8022 Q: "...sr/bin/env": ANDROID_DATA=/data Q: "...sr/bin/env": TERM=xterm-256color Q: "...sr/bin/env": USER=u0_a138 Q: "...sr/bin/env": SHLVL=1 Q: "...sr/bin/env": ANDROID_ROOT=/system Q: "...sr/bin/env": BOOTCLASSPATH=/system/framework/prcui-config.jar:/system/framework/core-oj.jar:/system/framework/core-libart.jar:/system/framework/conscrypt.jar:/system/framework/okhttp.jar:/system/framework/core-junit.jar:/system/framework/bouncycastle.jar:/system/framework/ext.jar:/system/framework/framework.jar:/system/framework/telephony-common.jar:/system/framework/voip-common.jar:/system/framework/ims-common.jar:/system/framework/apache-xml.jar:/system/framework/org.apache.http.legacy.boot.jar:/system/framework/qcom.fmradio.jar:/system/framework/oem-services.jar:/system/framework/tcmiface.jar:/system/framework/telephony-ext.jar Q: "...sr/bin/env": SSH_CLIENT=127.0.0.1 42047 8022 Q: "...sr/bin/env": PATH=/data/data/com.termux/files/usr/bin:/data/data/com.termux/files/usr/bin/applets Q: "...sr/bin/env": SSH_TTY=/dev/pts/1 Q: "...sr/bin/env": OLDPWD=/data/data/com.termux/files/usr/etc Q: "...sr/bin/env": _=/data/data/com.termux/files/usr/bin/cf-agent info: Last 20 quoted lines were generated by promiser '/data/data/com.termux/files/usr/bin/env' info: Completed execution of '/data/data/com.termux/files/usr/bin/env' R: Hello from termux_demo R: paths.prefix is /data/data/com.termux/files/usr R: Hello, only from termux platform agents error: Method 'termux_demo' failed in some repairs In order to support packages promise we must manually install python, because the package modules are written in python and we don't typically have to bootstrap this installation as most distributions have python available. pkg install python MPF had to be adjusted as well so if you bootstrap to an existing server make sure you have the most recent masterfiles or \`git cherry-pick\` [this commit](https://github.com/cfengine/masterfiles/commit/83c4e0cf9dddffdd192d49f98a30a9bb705edfc4) into your policy. Create an auto run service to install some software at `$PREFIX/var/lib/cfengine/masterfiles/services/autorun/software.cf` ```cf3 bundle agent software { meta: "tags" slist => { "autorun" }; packages: "pass" policy => "present"; } ``` And with an agent run I see that pass is installed\! ``` info: Successfully installed package 'pass' ``` $ pass --version ============================================ = pass: the standard unix password manager = = = = v1.7.3 = = = = Jason A. Donenfeld = = Jason@zx2c4.com = = = = http://www.passwordstore.org/ = ============================================ If you have any ideas on what you'd like to see with CFEngine on Termux, contribute your ideas and thoughts to the [Termux Support Iteration 2](https://northerntech.atlassian.net/browse/CFE-3380) epic in our JIRA system. -------------------------------------------------------------------------------- title: "COVID-19's impact on infrastructure security" url: /blog/2020/covid-19s-impact-on-infrastructure-security/index.md date: "2020-07-14" -------------------------------------------------------------------------------- It's no secret that COVID-19 is negatively impacting businesses of all sizes in a number of ways. Some more obvious than others. Unless you are in IT, you're probably not thinking of how COVID-19 can affect the infrastructure security of your organization, but the truth is that as businesses make the tough decision to layoff employees in order to stay in business, basic security hygiene can easily be overlooked. Even organizations that are fortunate enough to not have to make cuts are still impacted in the form of needing access to specialized tools that allow IT & Security teams to enforce infrastructure changes remotely, efficiently, and at scale. If you're looking to implement a configuration management tool to improve infrastructure security, such as CFEngine, it can be a little overwhelming to understand what types of questions to ask and criteria to consider. To help you brainstorm and prioritize, I'd like to cover what I believe are the top 3 most important criteria to consider during your evaluation. ## Security / stability One of the most important use cases of configuration management tools is to help harden your infrastructure assets. Because of this it is important that as you look to introduce new tools or modify existing ones, that you keep in mind how it can improve or harm your overall security posture. - What types of dependencies does this tool rely on? Python, Ruby, etc. - Are these dependencies increasing my attack surface? - Can these dependencies cause frequent disruptions to my service? - Am I able to implement role-based access controls? - Can this tool help me enforce & verify security compliance? ## Speed / scalability These days speed & efficiency are critical to survival. One saying that holds true, especially within configuration management, is "Organizations don't change for 1x improvement. They change for 10x." - How quickly can I execute changes across my entire infrastructure? Second, Hours, Days, Weeks? - Can most of the desired and required changes be automated? - Will this solution help me avoid configuration drift without manual intervention? - Will this solution scale as my business continues to grow? ## Multi-purpose The market drowns in IT tools aimed at increasing efficiency. While many of these tools may serve their intended purpose, the challenge easily becomes that IT teams end up managing too many different tools that each only serve an individual purpose. This not only becomes a headache to manage on a daily basis, but if you're not careful, can also create many performance issues. Finding a tool that can become more of a Swiss Army Knife for your team can be the simplest and best solution. Less to manage. Less to learn. Happier IT & Security teams. Obviously there are many more questions and criteria to consider when looking to implement new IT tools, but in a time of uncertainty it is important that you don't sacrifice one problem only to gain multiple others. This is an opportunity to re-evaluate your current set of tools and ensure that moving forward you're getting the most from them. -------------------------------------------------------------------------------- title: "Personal Policy" url: /blog/2020/personal-policy/index.md date: "2020-07-06" -------------------------------------------------------------------------------- ## My laptop was getting stale… I've been using it every work day for about 2.5 years now and so much software is installed it just boggles my mind. I really love it otherwise, open source, trying to be transparent, generally has worked amazingly\! I have a [Librem 15v3](https://puri.sm/products/librem-15/) from [Purism](https://puri.sm/). ![librem15v3 laptop](images/l15-v3-turns97.png) My home dir is a maze of old and new directories, odd files, tons of \~/Downloads junk. And the real kicker? I can't build CFEngine core anymore\! :( I tried to fix the situation but just couldn't quite fix it. So the solution? Well reinstall PureOS of course and see if that helps things out. ## Start fresh\!\!\! My first try at finding instructions failed a bit: says "TODO: write some instructions". ### (oops) So a bit more poking around in the [forums](https://forums.puri.sm/t/reinstalling-pureos/8734/2) led me to a proper [Live System Installation guide](https://tracker.pureos.net/w/installation_guide/live_system_installation/). I also found some more details about preparation and maintenance at the [General Recommendations](https://tracker.pureos.net/w/recommendations/). ### (instructions were great, live install was easy\!) That working well enough I decided to finally get serious about something I've been playing with for over a year: a "personal" policy. ## The Goal: Use CFEngine for everything\! Ideally I'd like to manage all aspects of customization with CFEngine policy. Just to get a good understanding of how the policy language works and where things are difficult and how we might improve CFEngine. ## Install CFEngine If you have a typical OS you can just use a one-liner quick install provided for [Enterprise](https://docs.cfengine.com/docs/3.15/guide-installation-and-configuration-general-installation-installation-enterprise-free.html) or [Community](https://docs.cfengine.com/docs/3.15/guide-installation-and-configuration-general-installation-installation-community.html) versions. For this work I am going to use community. Sadly our quick install script doesn't understand Pure OS 9 as a version. I also tried our rather helpful tool [cf-remote](https://github.com/cfengine/cf-remote). Sadly it doesn't understand Pure OS either and tries to install an ubuntu package, which fails. So I had to download the [Debian 10 3.15.2 package](https://cfengine-package-repos.s3.amazonaws.com/community_binaries/Community-3.15.2/agent_debian10_x86_64/cfengine-community_3.15.2-1.debian10_amd64.deb) manually from this [page](https://cfengine.com/product/community/). ## Bootstrap I am just working locally on this policy and on WiFi so the IP could change. This could be a problem later but I'm just working locally for now so I will deal with that later by setting up a policy server that has a stable IP. The first step after installing is getting our Policy framework in place. This is called bootstrapping. As root: # cf-agent --bootstrap 192.168.43.233 R: Bootstrapping from host '192.168.43.233' via built-in policy '/var/cfengine/inputs/failsafe.cf' R: This host assumes the role of policy server R: Updated local policy from policy server R: Triggered an initial run of the policy notice: Bootstrap to '192.168.43.233' completed successfully! Now my system is ready for experimentation/development. Generally you should develop on something that is not your primary workstation just to be safe. But I wanted to live on the edge and eat my own dog food. So I will proceed. ## Track changes I want to be sane about knowing what I have changed in the policy so will initialize a git repo in `/var/cfengine/masterfiles` # cd /var/cfengine/masterfiles # git init # cat > .gitignore cf_promises_release_id cf_promises_validated (ctrl-d) # git add .gitignore # git add . # git commit -m initial ## Update Policy and Evaluate Running the agent on every change is necessary when developing policy. Normally `cf-execd` takes care of updating policy from the policy server (in this case localhost) and the evaluating the policy. This is done in two steps: `cf-agent -KIf update.cf` which updates policy, in this case copying files from `/var/cfengine/masterfiles` to `/var/cfengine/inputs` where it will be evaluated by the command `cf-agent -KI`. A bit of explanation about the options/commands above: `-K` or `--no-lock` Normally cf-agent won't evaluate the same promise more often than 1 minute due to [Promise Locking](https://docs.cfengine.com/docs/3.15/reference-language-concepts-promises.html#promise-locking). `-K` will ignore these locks making development easier. `-I` or `--inform` This gives a bit more verbose output than normal and will generally tell you what cf-engine has done to your system during each run. `-f` or `--file` This specifies the first policy file to read and evaluate. If a relative path this will refer to a path relative to `/var/cfengine/inputs`. It is useful sometimes to specify a local/absolute path such as `./test.cf`. To make this go a little quicker I made a `cf` script in `/usr/local/sbin/cf` # cat /usr/local/sbin/cf cf-agent -KIf update.cf && cf-agent -KI ## Policy Organization I haven't written or managed large bodies of policy just yet. But I do know that our [autorun services](https://docs.cfengine.com/docs/3.15/reference-masterfiles-policy-framework.html#services_autorun) help when adding custom policy to our policy framework called [masterfiles](https://github.com/cfengine/masterfiles/). To enable autorun services we should add a few things via our [Augments](https://docs.cfengine.com/docs/3.15/reference-language-concepts-augments.html) feature to a file at `/var/cfengine/masterfiles/def.json`. First, if we might rename our services cf files then by default any agents would end up with both the old and new files. So we define a class which purges files on the agent which are not present on the policy server: [`cfengine_internal_purge_policies`](https://docs.cfengine.com/docs/3.15/reference-masterfiles-policy-framework.html#automatically-remove-files-not-present-upstream-sync-masterfiles). Next, if we make any adjustments to the way CFEngine services are configured in our policy we will need to force these components to restart with [`mpf_augments_control_enabled`](https://docs.cfengine.com/docs/3.15/reference-masterfiles-policy-framework.html#configure-mpf-to-automatically-restart-components-on-relevant-data-change). ```json { "classes": { "services_autorun": [ "any::" ], "cfengine_internal_purge_policies": [ "any::" ] } } ``` Note the use of `any::`. "any" is a [hard class](https://docs.cfengine.com/docs/3.15/reference-language-concepts-classes.html#hard-classes) which is always defined so these two classes in the augments file (def.json) will always be defined. The two semi-colons afterward cause the string to be considered a class expression instead of a regex. See the details about how these JSON strings are handled [here](https://docs.cfengine.com/docs/3.15/reference-language-concepts-augments.html#classes). And now on to the basic things I need to have a usable computer. ## User accounts When installing linux I created an account so I'll leave that one alone for now. Just for fun I'll create two accounts: `me` and `too`. First I make a json file to hold variable data so that others can re-use this policy without so much trouble, at `/var/cfengine/masterfiles/services/autorun/personal.json`: ```json { "users": [ "me", "too" ] } ``` Then the policy will load that json file and I make sure that works by printing out a diagnostic with a reports promise. The autorun service policy files MUST be in the `/var/cfengine/masterfiles/services/autorun` directory to be picked up automatically. There are other ways but this is the easiest for autorun services. So I create `users.cf` in that directory. ```cf3 bundle agent users { meta: "tags" slist => { "autorun" }; vars: "personal" data => readjson("$(this.promise_dirname)/personal.json"); reports: "$(personal[users])"; } ``` Running my `cf` script I get: ``` R: me R: too ``` Next I add a users promise to ensure these users exist ```cf3 users: "$(personal[users])" policy => "present"; ``` And with another run of the agent I have users\! info: User promise repaired info: User promise repaired And inspection of `/etc/passwd` shows the users are there\! me:x:1001:1001::/home/me:/bin/sh too:x:1002:1002::/home/too:/bin/sh Wait, where's the home dir? I have a user but no home dir for it to use\! So let's back up, remove the users (with policy) and add the homedir and some basic files from `/etc/skel` with the [home\_bundle](https://docs.cfengine.com/docs/3.15/reference-promise-types-users.html#home_bundle) attribute of the users promise. In the policy I change "present" to "absent" ```cf3 "$(personal[users])" policy => "absent"; ``` The users are removed, though the inform output doesn't say "removed" but rather "repaired". That's because promises can promise many different things and in this case the removal was done and so is repaired. info: User promise repaired info: User promise repaired Checking `/etc/passwd`, `me` and `too` are gone. :) I refactor to get the users into a [list](https://docs.cfengine.com/docs/3.15/reference-language-concepts-variables.html#lists) data variable so I don't have to keep typing personal\[users\] ```cf3 "users" slist => getvalues("personal[users]"); ``` I take the `skel` var from the `home_bundle` example: ```cf3 "skel" string => "/etc/skel"; ``` Rewrite the users promise to use home\_dir and home\_bundle: ```cf3 users: "$(users)" policy => "present", home_dir => "/home/$(users)", home_bundle => home_skel($(users), $(skel)); ``` and add the `home_skel` bundle: ```cf3 bundle agent home_skel(user, skel) { files: "/home/$(user)/." create => "true", copy_from => seed_cp($(skel)), depth_search => recurse("inf"); } ``` and the result from an agent run: ``` info: Created directory '/home/me/.' info: Copying from 'localhost:/etc/skel/.bash_logout' info: Copying from 'localhost:/etc/skel/.profile' info: Copying from 'localhost:/etc/skel/.bashrc' info: User promise repaired info: Created directory '/home/too/.' info: Copying from 'localhost:/etc/skel/.bash_logout' info: Copying from 'localhost:/etc/skel/.profile' info: Copying from 'localhost:/etc/skel/.bashrc' info: User promise repaired ``` ## Whose stuff is this anyway? Uh oh, these dirs aren't very usable for our new users. They are owned by `root`\! :( ``` drwxr-xr-x 2 root root 4096 Jun 12 12:34 me drwxr-xr-x 2 root root 4096 Jun 12 12:34 too ``` So we add a files promise to take care of that. I could have added a few attributes in the home\_bundle body but that only runs on initial user creation and I'd like to enforce these permissions all the time. Notice that I use the body [og()](https://docs.cfengine.com/docs/3.15/reference-masterfiles-policy-framework-lib-files.html#og) from the policy framework to enforce the owner and group for the user. ```cf3 files: "/home/$(users)/." perms => og("$(users)","$(users)"), depth_search => recurse("inf"); ``` agent output: ``` info: Owner of '/home/me/./.bash_logout' was 0, setting to 1001 info: Group of '/home/me/./.bash_logout' was 0, setting to 1001 info: Owner of '/home/me/./.profile' was 0, setting to 1001 info: Group of '/home/me/./.profile' was 0, setting to 1001 info: Owner of '/home/me/./.bashrc' was 0, setting to 1001 info: Group of '/home/me/./.bashrc' was 0, setting to 1001 info: Owner of '/home/too/./.bash_logout' was 0, setting to 1002 info: Group of '/home/too/./.bash_logout' was 0, setting to 1002 info: Owner of '/home/too/./.profile' was 0, setting to 1002 info: Group of '/home/too/./.profile' was 0, setting to 1002 info: Owner of '/home/too/./.bashrc' was 0, setting to 1002 info: Group of '/home/too/./.bashrc' was 0, setting to 1002 ``` ## Uh, what's the password? I manually set the password for the `me` user and then check /etc/shadow for the hash. Then use that in the policy. For now I will just use the same password for both me and too since I want to keep my json simple and for the near future I will really only be working with one user: `craig` ;) I use another policy framework body called [hashed\_password()](https://docs.cfengine.com/docs/3.15/reference-masterfiles-policy-framework-lib-users.html#hashed_password) to make things a bit easier. ```cf3 password => hashed_password("$6$IvdxUGIwiwcDYPaH$MALkFPHXtbt3lu1WzJSoHZ27zoHV/ow553.0nxARTLktL.xKuDRamUcaj2SIqmsMohPknPWqsZinMGDR1vkKy0"), ``` agent output: ``` info: User promise repaired ``` And try to login\! ``` craig@other:~$ su - me Password: ******** $ whoami me ``` ## точно, молодец I studied Russian for several years in college. That means "perfect, you're a genius" or something like that. (google says "accurate, well done", that's pretty close). Here is the final policy. ```cf3 bundle agent users { meta: "tags" slist => { "autorun" }; vars: "personal" data => readjson("$(this.promise_dirname)/personal.json"); "skel" string => "/etc/skel"; "users" slist => getvalues("personal[users]"); users: "$(users)" policy => "present", password => hashed_password("$6$IvdxUGIwiwcDYPaH$MALkFPHXtbt3lu1WzJSoHZ27zoHV/ow553.0nxARTLktL.xKuDRamUcaj2SIqmsMohPknPWqsZinMGDR1vkKy0"), home_dir => "/home/$(users)", home_bundle => home_skel($(users), $(skel)); files: "/home/$(users)/." perms => og("$(users)","$(users)"), depth_search => recurse("inf"); reports: "$(users)"; } bundle agent home_skel(user, skel) { files: "/home/$(user)/." create => "true", copy_from => seed_cp($(skel)), depth_search => recurse("inf"); } ``` ## More later... I plan on continuing to refine and expand this personal policy of mine and blog about the steps along the way. The most pressing things to handle are: - [password store](https://www.passwordstore.org/) software install and git pull/push to keep things up-to-date - synchronize important data from device to device so it's safe and available wherever I want it, such as gpg, config files, music, pictures, email, documents - `/etc/hosts` config/distribution for all the devices on my home wifi so that when IPs change I can still just `ssh hostname` and not worry about it. -------------------------------------------------------------------------------- title: "CFEngine 3.16 released - Compliance" url: /blog/2020/cfengine-3-16-released-compliance/index.md date: "2020-06-25" -------------------------------------------------------------------------------- Today we announce the newest additions to CFEngine. CFEngine 3.16 brings several improvements, bug fixes, and new features. The theme for this release has been compliance, and it notably includes a new category of reports for proving compliance to regulation and other compliance frameworks in high level, easy to read reports. If you are interested to learn more about CFEngine, schedule training, or hear about pricing options, feel free to [reach out to us\!](https://cfengine.com/engage/contact/) ## What's new ### Compliance reports (Beta) Compliance reports allow you to implement internal or regulatory frameworks as a list of checks which have to pass on all or a subset of your hosts. The system is very flexible, allowing you to implement checks using Inventory values, software updates, or even custom SQL queries. Once the report is completed, the results can be exported to PDF, or scheduled to run at a fixed interval, for example every week. Different compliance reports can be set up for various stakeholders in your organization, and automated emails with PDFs can bring this valuable information to people/departments which don't use Mission Portal directly. This feature is still in beta, and a lot of the UI will change, but we are excited to show it to you and start getting some feedback on it, so it will be in a great shape for the next LTS release. ![](images/Mission_Portal_Compliance-Infrastructure-compliance-564x800.png) ### Secret Encryption A new tool has been introduced, `cf-secret`, which allows you to encrypt files for specific hosts in your infrastructure. It is open-source, available in both community and enterprise packages, and can be used from policy or as a stand-alone tool. ![](images/Screenshot-2020-06-25-at-09.18.17.png) [Read more about `cf-secret` here.](https://cfengine.com/company/blog-detail/cf-secret/) ### Detailed host and license counts Mission Portal now gives you a detailed overview of your hosts, including the license count, number of recently deleted hosts, as well as the total number of hosts before RBAC filtering is applied. This should clear up some confusion in large infrastructures, especially when you are approaching the license count, and if you decommission hosts frequently. (Unfiltered host count is a separate control in our Role Based Access Control (RBAC) system, so you can grant or remove access to it for any user/role). ![](images/Screenshot-2020-06-24-at-12.10.23.png) ### New version macros From CFEngine 3.16 and onward, you can more easily target specific CFEngine versions within the policy language. Previously, you could use the `minimum_version` macro to avoid using newer syntax and features on older versions that don't support it. However, there was no obvious approach to where you would put the alternative (the policy which you want to run on older versions only). For this reason and to allow greater flexibility to policy writers, we have introduced a few new macros: - `maximum_version` - `else` - `at_version` - `between_versions` - `before_version` - `after_version` [Read more about all these macros in the documentation](https://docs.cfengine.com/docs/3.16/reference-macros.html). ### New functions CFEngine 3.16.0 introduces 9 new functions. To complement the version macros, which function at a parsing level, we also now have functions for making decisions based on the version of the agent from the policy. - `cf_version_after()` - `cf_version_at()` - `cf_version_before()` - `cf_version_between()` - `cf_version_maximum()` - `cf_version_minimum()` A new utility function to strip leading and trailing whitespace was added. - [`string_trim()`](https://docs.cfengine.com/docs/3.16/reference-functions-string_trim.html) And two new data validation functions were added for making decisions based on data validity. - [`validdata()`](https://docs.cfengine.com/docs/3.16/reference-functions-validdata.html) - [`validjson()`](https://docs.cfengine.com/docs/3.16/reference-functions-validjson.html) ### New files promise type attribute The new content attribute for `files` type promises makes it more clear and concise to promise the full content of a file. An example illustrating files promise content attribute: ```cf3 bundle agent main { files: "/tmp/hello_content" create => "true", content => "Hello, content!$(const.n)"; } ``` Here we can see the results of executing the example above. ```example # cf-agent --no-lock example_file_content.cf # cat /tmp/hello_content Hello, content! ``` You can find the [documentation about the content attribute here](https://docs.cfengine.com/docs/3.16/reference-promise-types-files.html#content). ## Changelogs We have made a lot of smaller improvements, dependency updates, and bug fixes which would not fit into a single blog post. Some of these are brand new to CFEngine 3.16, while some were also included in the [recent 3.12.5 and 3.15.2 releases](https://cfengine.com/company/blog-detail/cfengine-3-12-5-and-3-15-2-released/). You can read all the details in our changelogs: - [Core](https://docs.cfengine.com/docs/3.16/guide-latest-release-whatsnew-changelog-core.html) - [Enterprise](https://docs.cfengine.com/docs/3.16/guide-latest-release-whatsnew-changelog-enterprise.html) - [Masterfiles](https://docs.cfengine.com/docs/3.16/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ## Contributing We are always looking for new **contributions to CFEngine\!** Are you unsure how to get started? Please check out our [contributing guide](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) in addition to the following suggestions. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO]() that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Fix issues pointed out by [code analysis](https://lgtm.com/projects/g/cfengine/core/alerts/). -------------------------------------------------------------------------------- title: "How CFEngine stays ahead of the pack" url: /blog/2020/how-cfengine-stays-ahead-of-the-pack/index.md date: "2020-06-23" -------------------------------------------------------------------------------- ## Blazing the trail CFEngine was the first Configuration Management solution on the market, and while we have made many and significant changes and improvements to CFEngine in that time, we stay true to the principles that make it such a great product and technology. There are many things that have changed in the market, not at least the competitive situation, we believe that fundamentally many of the challenges stay the same. It then follows that good architecture should not be sacrificed for short term hype. In this short blog post, I will go over a few of the items that lead to CFEngine's excellence, longevity in the market, and current strong position. ### What is a promise? Let's start with the basics of what makes CFEngine tick\! CFEngine provides a declarative policy language that focuses on the outcomes you want to achieve, not the instructions or steps needed to get there. While some other automation solutions only provide a set of instructions or workflows, CFEngine does not take this approach. Why not? Because a sequence of instructions will only lead to the correct outcome if we truly know where we are starting from. We cannot plan for the specific error state or misconfiguration we want to mitigate, so a declarative approach is the most stable approach to large scale automation. Over time, any given state can occur that we never planned for. In order to consistently achieve a desired state or outcome, a promise is what you need. When using CFEngine you will find yourself writing policy that only promises something about the outcome, not focusing on the process. This makes the architecture of CFEngine highly effective, provides great cross-platform support and makes it very effective at achieving the desired state. The agent will then act on the promise, sometimes taking different paths but always leading to a predictable outcome. We call this convergence. ### Promise theory When we use the word "promise" in CFEngine, it comes from Promise Theory, an area of research pioneered by Mark Burgess, the original creator of CFEngine. Maybe you have never heard of Promise Theory, maybe you are a big fan. Either way, using CFEngine is unrelated to Promise Theory today. So what does Promise Theory mean for CFEngine? In short, it means that CFEngine is underpinned by a rigorous scientific framework, and we know that the promises you implement using our Policy language will be kept, and behave in a certain way. There are many great resources available on Promise Theory [here](https://www.amazon.com/Thinking-Promises-Designing-Systems-Cooperation-ebook/dp/B01092PYG8), [here](https://www.amazon.com/gp/product/B00WL6SPR6), [here](https://www.linuxjournal.com/content/promise-theory%E2%80%94what-it) and [here](https://www.youtube.com/watch?v=2TPsB5WuZgk). Learning promise theory, or even knowing what it is, is not a requirement or advantage when using CFEngine, but could be interesting to have a look at for those of you that are curious. ## Best practices With the promise as our base "unit", it enables us to help you solve some of the hardest problems you have. Here we present some best-practices and ways of thinking to get the most out of your infrastructure, and of CFEngine. ### Infrastructure as code Infrastructure as code has long been a goal for many organizations. It should not be a pipe dream, indeed it is exactly what you can easily achieve with CFEngine. Any change to your infrastructure can easily be made by modifying your CFEngine Policy, or better yet the data that your Policy leverages. Do you want to switch a machine over from Testing to Prod, you can change one variable, and CFEngine will take care of the rest. Do you need to upgrade a software version, a database type or any default settings across 50 or 50 000 machines, it is equally simple using CFEngine. The more you can rely on code to drive your infrastructure the better. And CFEngine makes that simple. ### Source data externally To simplify things further; build a Policy for your framework and use external data sources for your quickly changing variables. One key to success with CFEngine is to work in a dual mindset. Let Policy define your architecture, and the variables should all be sourced from external data files like an MDR or CMDB system. In this way, anyone with the appropriate access rights can easily modify for example an application version across the entire infrastructure without modifying the underlying Policy. A great example of this way of working is our Success Story LinkedIn that you can read more about in [this whitepaper](https://cfengine.com/case-studies/linkedin-case-study/). ### Ad-hoc changes Making quick fixes and fast ad-hoc changes is not the design pattern of CFEngine. While we do not recommend doing this, we provide simple ways to make sure you can put out a fire, before making the needed changes that can avoid such fires in the first place. With the powerful Mission Portal UI, you can find issues and drill down to any host in your infrastructure. The best solution if something is not correctly configured or implemented, is usually to fix the policy so that the problem never repeats neither on that one server nor anywhere else. Before you have an opportunity to do that, CFEngine makes it simple to integrate with Ansible or other tools that focus on small scale infrastructure and ad-hoc, manual processes for firefighting. You should always use the tool that is best suited for the task. ### Scale effortlessly CFEngine is the best choice you can make, not only for large scale infrastructure but complex and hard to manage systems. CFEngine thrives with complexity, many operating systems, complicated timescales, and many users with different needs. The main idea behind CFEngine is to make the infrastructure work for you, not the other way around. Define policy once, and you are good to go. No need to babysit or watch over your configuration like a hawk. ### Best in class reporting CFEngine Enterprise comes with the best and most effortless large-scale reporting capabilities in the market. Set up custom reports, email them automatically to management, get in-depth compliance data, and much more through the simple to use Mission Portal UI. Each host in your infrastructure will gather a lot of information, some of it is configured out-of-the-box, but the options are limitless. The architecture is highly optimized, so it consumes a minimal amount of capacity on your nodes. The cost of having to run an agent that requires several hundred MB of memory vs an agent that requires less than 10MB of memory quickly adds up. This is then all propagated to a reporting hub, where it's available for analysis. Unlike many others, we make all the data immediately available, and you can query our database for anything you want. ## Summary In short, we believe that CFEngine is the best tool available for a range of automation, configuration, security, and report collection tasks. CFEngine was the first successful tool in configuration management and has been blazing the trail of this field since. You should always choose the best tool for the job you need to get done. You should choose CFEngine\! -------------------------------------------------------------------------------- title: "CFEngine in a High Performance Computing environment" url: /blog/2020/cfengine-in-a-high-performance-computing-environment/index.md date: "2020-06-11" -------------------------------------------------------------------------------- # CFEngine in HPC In High-Performance Computing (HPC) uptime and performance are very important. HPC is an area of computing that often focuses on research and development, supporting teams with extremely complex problems they need to solve, and heavy computation mathematical problems, such as protein folding for vaccine development. To achieve this, HPC systems rely on high performance, the equipment is expensive, and the average customer has very high demands. Any downtime, performance degradation, misconfiguration, or unexpected behavior will be a financial cost and will reduce the customers' trust in the HPC provider. CFEngine is a configuration management tool that is created to manage such environments and truly excels at its tasks in an environment like this. The modular architecture, the small size, fast executables, self-healing properties, and autonomous execution are what make CFEngine ideal for these tasks. In this short blog post, we will look at these aspects of CFEngine, and how CFEngine users are saving money and improving the service they offer their users by leveraging CFEngine. ## Minimal agent CFEngine is an agent-based technology that uses a minimal agent deployed to all hosts in the infrastructure. The significance of the performance of the CFEngine agent compared to any competing solution cannot be underestimated. The agent only uses a minimal amount of resources, due to the architecture and implementation. The agent is written in C and is highly configurable to minimize footprint. CFEngine has several daemons that can be running, but depending on the level of functionality most or even all of them can be turned off to reduce the footprint of CFEngine on the hosts. What is also possible is to configure the agent to only run during specific periods, not to impact the performance of the environment during computation. This thinking of "time scales" is one of the things that can make CFEngine even more efficient. The agent can run often to execute tiny policy sets and only do operations that are more demanding at specific times or intervals. ## Minimal network dependency There are "agent-less" approaches to configuration management available, but in the context of HPC, these typically do not come near meeting the requirements of a high-performance environment. For each run or each invocation of such a technology, the node in question will need to open a network connection to the server, transfer the agent binary (it is not actually agentless, it is just not permanently installed) all the needed policy data and potentially any dependencies, before it can take any action. If the server is unavailable for any reason the policy execution will not happen, which is clearly unacceptable. With an "agent-less" approach, if the admins or policy server is distributing or forcing a change, this will not be applied to any node not available on the network at that specific time. This can cause serious deviations in the infrastructure, and it makes it hard to know what servers have actually had a change applied. With CFEngine, a given node does not need to have network access to the policy hub (or at all for that matter) in order for the agent to execute the given policy. The policy is stored locally, and the node will check for updates to the policy at a set interval - the default is every five minutes, but this can be changed. Both in distributing policy updates as well as reporting back, the protocol defaults to delta information in order to reduce bandwidth. This approach significantly reduces the network traffic and the dependence on network availability. Regardless of the network status or connection between the Hub and the Host, the agent will continue to function independently. ## Large variety of hardware It's common for HPC systems to have several generations of hardware. However, to the users of the system, this should largely not be a concern. The management team needs to make sure that each generation of hardware has the correct configuration, software versions, and firmware. CFEngine excels at managing such a complex configuration, down to the smallest detail of these differences. Examples of this can be to set a specific sysctl value based on the video card that is found. ## Self-healing infrastructure A key feature of CFEngine is the self-healing capacity. The agent will react to any change in the system by converging all changes to the desired state. The agent itself will execute its policy, so this self-healing property is not dependent on any network connection or network traffic to be available. Typically, as time passes, configuration changes, sysadmins perform needed changes manually or just by logging in to check on the status of a machine in the system, system administrators are prone to change or break configurations or other parameters. Like installing and setting a new shell as the default, changing who is in the sudoers list, or other actions that lead to deviations from the desired state. This can have severe consequences over time, and CFEngine makes sure all resources are configured correctly. ## Reporting Knowledge is power, and with CFEngine you have the most knowledge about your system available, at all times. Reporting is key to knowing that the changes you roll out are applied correctly, throughout the infrastructure. CFEngine can now collect reporting data from multiple Hubs, in different configurations, security zones or locations, to provide a single pane of glass into the entire infrastructure. ## Integrations CFEngine reporting can easily be extended to integrate with a multitude of other tools. One of the important features to point out that make CFEngine invaluable in such an environment is the [Jira integration](https://cfengine.com/learn/ticketing-system-integration-jira/). CFEngine can automatically open a ticket in Jira (or other issue trackers) once a certain condition is observed in the infrastructure. An example of this can be the size of a certain database or the permissions of a user being incorrect. At such time that this condition is no longer observed CFEngine can automatically close the ticket again. This enables a rapid response model that can enable system engineers to work much more efficiently. ## The value of CFEngine One of our customers is a great showcase of the value of CFEngine in an HPC environment. They have two data centers, one consisting of around 10 000 servers, the other is approximately double that. The larger one is managed without CFEngine, but using various ad-hoc tools and processes. The team size for the non-automated datacenter is several times larger. Despite having a much larger team, they spend significantly more time deploying new software, struggling with much more sprawl. They struggle to correctly report on the state of their infrastructure. Neither of these are problems that the CFEngine based team faces. In the last year, there were no outages or severe issues with the datacenter automated using CFEngine, while there were three major outages caused by incorrect patching attempts in the non-managed datacenter. The costs of these outages are significant and help underscore not just the value in operations of using CFEngine, but the direct monetary impact of not doing it. That team has since also started to look at a path to start using CFEngine, and we are excited to bring them on board. -------------------------------------------------------------------------------- title: "Getting out from under a SIGBUS BUS_ADRALN on Solaris/HP-UX" url: /blog/2020/getting-from-under-a-sigbus-bus_adraln-on-solaris-hp-ux/index.md date: "2020-06-08" -------------------------------------------------------------------------------- # Introduction In the CFEngine Core team, we have recently been working on a fix for our `WaitForCriticalSection()` function. In short, the function checks a timestamp in a chunk of (lock) data stored in a local LMDB database and if the timestamp is too old, it writes a new chunk of (lock) data with the new timestamp. However, this used to be done in separate steps - read the data from the DB and close DB, check the data and potentially write the new data into the DB. So, there was a race condition because if multiple processes did the same steps at the same time, they could have read and checked the same timestamp value and then write their own data with their new timestamps one after another. On the high-level perspective that meant multiple processes could have entered the critical section at the same time. To fix the issue, we needed a function that would read, check and potentially write data into an LMDB database **in one transaction**. That way, multiple processes trying to enter the critical section at the same time would do so one after another. To decide whether the current value should be overwritten with a new value, the new function, called `OverwriteDB()` (complementary to `ReadDB()` and `WriteDB()`), takes as one of its arguments a pointer to a function that gets the current value and returns `true` (overwrite) or `false` (pass). Everything was working fine with this new function. And then, in our CI, we started getting hit by a (SIG)BUS. Tests started failing and hanging on our Solaris SPARC64 and HP-UX IA64 machines and investigation revealed that *cf-agent* processes in the tests were sometimes killed with the *SIGBUS* signal when there were multiple such processes running at the same time. And fighting over the critical section, we figured. # Bad alignment When we tried to debug the issue using *truss* (Solaris alternative of *strace*), we saw this: ``` example 16422: open("/var/cfengine/state/cf_lock.lmdb", O_RDWR|O_CREAT, 0644) = 10 16422: pread(10, "\0\0\0\0\0\0\0\b\0\0\0\0".., 92, 0) = 92 16422: pread(10, "\0\0\001\0\0\0\b\0\0\0\0".., 92, 8192) = 92 16422: mmap(0x00000000, 104857600, PROT_READ, MAP_SHARED, 10, 0) = 0xF8000000 16422: open("/var/cfengine/state/cf_lock.lmdb", O_WRONLY|O_DSYNC) = 11 16422: fcntl(11, F_GETFD, 0x00000000) = 0 16422: fcntl(11, F_SETFD, 0x00000001) = 0 16422: fcntl(9, F_SETLK, 0xFFBFA63C) = 0 16422: close(8) = 0 16422: time() = 1591194272 16422: getpid() = 16422 [16343] 16422: time() = 1591194272 16422: time() = 1591194272 16422: lwp_mutex_timedlock(0xFE860040, 0x00000000) = 0 16422: time() = 1591194272 16422: Incurred fault #5, FLTACCESS %pc = 0xFF28D24C 16422: siginfo: SIGBUS BUS_ADRALN addr=0xF800B426 16422: Received signal #10, SIGBUS [caught] 16422: siginfo: SIGBUS BUS_ADRALN addr=0xF800B426 ``` So the process used `mmap()` to map a file to its memory address space starting at the address `0xF8000000` and then when it tried to access the address `0xF800B426` it was killed with the *SIGBUS* signal indicating that there was an invalid address alignment (*BUS\_ADRALN*). This was the first time we saw an issue like this. Memories from the university had some traces of alignment and alignment issues, but not too strong. And actually, it turns out that you will not get this error on *x86* CPUs and some other architectures, at least not in a similar scenario. That was why everything worked on most of our machines except for Solaris 10 and 11 and HP-UX machines which use SPARC64 and IA64 CPUs, respectively. # Alignment error example It is simple to write code that triggers this behavior: ```c #include int main() { int nums[2] = {1, 2}; unsigned char *bytes = (unsigned char *) nums; bytes++; bytes++; int wrong = *((int *)bytes); printf ("wrong number: %d\n", wrong); return 0; } ``` When compiled and run on an x86\_64 system, it produces this output: ``` example wrong number: 131072 ``` but on our Solaris 10 SPARC64 machine, it's not so lucky: ``` example Bus Error (core dumped) ``` The problem is that when trying to `load` an integer from an address that is 2 bytes after where the `nums` array starts, there is an alignment problem. The `nums` array is properly aligned (otherwise it wouldn't be accessible at all\!) and thus its address incremented by 2 is not divisible by 4 (`sizeof(int)`). On x86\_64, this is not a problem, the address is in the address space of the process and so the 4 bytes from the middle of the `nums` array are interpreted as an `int` (why 131072 is left as an exercise to the reader). But wait\! CFEngine code is (hopefully) not doing anything crazy and weird like that, right?\! It's not. However, that's where `mmap()` comes into the game. # Alignment and mmap() Let's have a look at another, this time a bit longer, example: ```c #include #include #include #include #include #include #include struct Data { int num; char code[2]; }; int main () { char chars[2] = {'a', 'b'}; struct Data data = { 1, { 'c', 'd' }}; printf("chars (%p): %c, %c\n", chars, chars[0], chars[1]); printf("data (%p): {%d, {%c, %c}\n", &data, data.num, data.code[0], data.code[1]); int fd = open("/tmp/file.dat", O_CREAT|O_TRUNC|O_WRONLY, S_IRUSR|S_IWUSR); ssize_t written = write(fd, chars, sizeof(chars)); if (written != sizeof(chars)) { printf("Failed to write 'chars': %m\n"); return 1; } written = write(fd, &data, sizeof(data)); if (written != sizeof(data)) { printf("Failed to write 'chars: %m'\n"); return 1; } close(fd); fd = open("/tmp/file.dat", O_RDONLY); const size_t total_size = sizeof(chars) + sizeof(data); unsigned char buf[total_size]; ssize_t n_read = read(fd, buf, total_size); if (n_read != total_size) { printf("Failed to read data back: %m\n"); return 1; } close(fd); printf("File contents:"); for (size_t i = 0; i < total_size; i++) printf(" %x", buf[i]); printf("\n"); fd = open("/tmp/file.dat", O_RDONLY); struct stat sb = {0}; fstat(fd, &sb); unsigned char *addr = mmap(NULL, sb.st_size, PROT_READ, MAP_SHARED, fd, 0); char *chars_in_file = addr; struct Data *data_in_file = (struct Data *) (addr + sizeof(chars)); printf("chars_in_file (%p): %c, %c\n", chars_in_file, chars_in_file[0], chars_in_file[1]); printf("data_in_file (%p): {%d, {%c, %c}\n", data_in_file, data_in_file->num, data_in_file->code[0], data_in_file->code[1]); return 0; } ``` The output on my x86\_64 system is as one would expect: ``` example chars (0x7ffd0f83a0de): a, b data (0x7ffd0f83a0d4): {1, {c, d} File contents: 61 62 1 0 0 0 63 64 0 0 chars_in_file (0x7f0665784000): a, b data_in_file (0x7f0665784002): {1, {c, d} ``` and we can see multiple interesting things. `chars` and `data` are stored on stack which, of course, grows down (in the address space). So the address of `data` is lower than the address of `chars`. We can also check that the address of `chars` (0x7ffd0f83a0de) is divisible by 2, but not by 4. The address of `data` (0x7ffd0f83a0d4), however, is divisible by 4. Which is good because it is a `struct Data` variable which starts with an `int` that is 4 bytes big. The difference between the addresses is 10 so there is padding between the two variables on the stack (`data` is only 6 bytes big). If we write both `chars` and `data` into a file one right after the other and then read the bytes back, we can see that there are the two `char` bytes for `'a'` (61) and `'b'` (62) followed by 4 bytes of the little-endian representation of `int` 1 followed by `char` bytes for `'b'` and `'c'` followed by two zero bytes. Where do these two zero bytes come from? They are padding, of course. The `struct Data` structure is 6 bytes big (4 bytes for the `int num` + 2 bytes for `char code[2]`) so it is padded with 2 zero bytes to be 8 bytes, which is a round (binary) number. When the file is mapped to the address space of the process using `mmap()`, pointers to the particular offsets (0 and 2) of the address segment can be created to get access to the copies of `chars` and `data` **in the file**. This time, however, the copy of `data` (at address 0x7f0665784002) follows immediately after the copy of `chars` (at address 0x7f0665784000). Any guesses what happens with the above code on SPARC64 Solaris 10 machine? ``` example chars (ffbffbc0): a, b data (ffbffbb8): {1, {c, d} File contents: 61 62 0 0 0 1 63 64 0 0 chars_in_file (ff1e0000): a, b Bus Error (core dumped) ``` (Also note that SPARC64 is big-endian, the 4B `int` 1 is stored as `0 0 0 1`.) And the output from *truss* sheds some more light: ``` example Received signal #10, SIGBUS [default] siginfo: SIGBUS BUS_ADRALN addr=0xFF1E0002 ``` Trying to access the copy of `data` in the file results in the process being killed with *SIGBUS* due to an invalid alignment. Of course, the address `0xFF1E0002` is not divisible by 4 and so trying to `load` the `int` in the struct attempts to do a non-aligned access. # Getting from under the (SIG)BUS And the above is what was happening in our tests with the changes that were fixing the `WaitForCriticalSection()` function. Or, to be more precise, in the `OverwriteDB()` function, where the data from the `mdb_get()` function (getting a value from LMDB) was directly passed to the function making the decision about whether to overwrite the data with the new value or not. So while making the decision the function did an unaligned access to the data passed to it which was actually a pointer to a memory segment where the LMDB DB file was mapped before with `mmap()`. And of course, this didn't happen always, because the position of the data depended on the contents of the DB file. The fix was, as usually, almost trivial compared to the complexity of the investigation and debugging. All that was needed was to create a local copy of the data from the mapped segment using the `memcpy()` function which makes sure the copied data is accessed using aligned addresses. And the destination was a local variable on the stack, which is always properly aligned (by the compiler) for direct access. -------------------------------------------------------------------------------- title: "CFEngine 3.12.5 and 3.15.2 released" url: /blog/2020/cfengine-3-12-5-and-3-15-2-released/index.md date: "2020-06-03" -------------------------------------------------------------------------------- Yesterday, packages for CFEngine 3.12.5 and 3.15.2 were made available. This release announcement was delayed in support of Blackout Tuesday in the United States of America. We are pleased to announce two new patch releases for CFEngine, version 3.12.5 and 3.15.2\! These patch releases don't contain major changes or new features, but rather fix important bugs reported by our community of users and customers. Some users reported issues with federated reporting, specifically related to non-reporting hosts. These issues are fixed in policy in the newest releases, upgrading to the latest version of Masterfiles Policy Framework (MPF) resolves them. A race condition during report collection was mitigated. This could in some cases cause a failed report collection, scheduling a retry (rebase), and emitting errors in syslog. The race condition only happened with long running agents, or agents running at the same time as report collection. It only caused errors in some very specific situations, so this was not caught by our tests, and most users did not experience the issue. Also, in relation to reporting, the hub can now query itself over the IPv6 loopback address, `::1`, similar to `127.0.0.1`. ## Changelogs As always, you can see a full list of changes and improvements in our changelogs 3.12.5: Changelogs for [Core](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html), [Masterfiles](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [Enterprise](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-enterprise.html). 3.15.2: Changelogs for [Core](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html), [Masterfiles](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html). ## Dependency updates The following dependencies were updated, see respective project websites for more information about the changes in those projects. ### 3.12.5 | | | | | -------- | ------ | ---------- | | apache | 2.4.41 | **2.4.43** | | git | 2.25.1 | **2.26.2** | | libcurl | 7.69.0 | **7.70.0** | | libyaml | 0.2.2 | **0.2.4** | | OpenLDAP | 2.4.49 | **2.4.50** | | OpenSSL | 1.1.1f | **1.1.1g** | | PHP | 7.2.28 | **7.2.30** | ### 3.15.2 | | | | | -------- | ------ | ---------- | | apache | 2.4.41 | **2.4.43** | | git | 2.25.1 | **2.26.2** | | libcurl | 7.69.0 | **7.70.0** | | libyaml | 0.2.2 | **0.2.4** | | OpenLDAP | 2.4.49 | **2.4.50** | | OpenSSL | 1.1.1f | **1.1.1g** | | PHP | 7.4.3 | **7.4.5** | We'd like to take this opportunity to thank the authors and maintainers of the Open Source projects we rely on to make CFEngine. ## Contributions CFEngine is an open source project, and we encourage all of our users to get involved in the community and contribute. Feel free to use one of the following channels: - Ask for help or start a discussion on [the mailing list](https://groups.google.com/forum/#!forum/help-cfengine) - Chat with the developers and other users [on IRC](https://web.libera.chat/?channel=#cfengine) - Submit a bug report or feature request in [our issue tracker](https://northerntech.atlassian.net/) - Look through [our curated list of issues for new contributors](https://northerntech.atlassian.net/issues/?filter=10073) - [Browse the code or submit a pull request through GitHub](https://github.com/cfengine/core) - Improve [the documentation](https://github.com/cfengine/documentation) by fixing typos, adding examples, or explaining things you found difficult -------------------------------------------------------------------------------- title: "Introducing cf-secret - Secret encryption in CFEngine" url: /blog/2020/cf-secret/index.md date: "2020-05-30" -------------------------------------------------------------------------------- Contributor and [CFEngine Champion](https://cfengine.com/engage/cfengine-champions/), [Jon Henrik Bjørnstad](https://github.com/jonhenrik13), developed a tool for encrypting files using CFEngine host keys, called `cf-keycrypt`. Thank you to Jon Henrik and all of our contributors for helping improve the CFEngine project. Our developer, [Vratislav Podzimek](https://github.com/vpodzime), recently took some time to review the cf-keycrypt code, and made many improvements and fixes. The most notable changes were: - Switched to hybrid encryption (payload is encrypted with randomly generated AES key, AES key is encrypted with RSA key). - Added file format, with HTTP-like headers for metadata - Files can be encrypted for multiple hosts (host keys) - Name changed to `cf-secret` `cf-secret` is now merged and will be a part of the upcoming 3.16 release. ## Encrypting a file Use the encrypt command to encrypt a file: $ echo "Hello, secret!" > message $ cf-secret encrypt -H 172.31.38.30 -o message.secret message You can also specify a comma separated list of IPs, host names, or host keys to the `-H` option to encrypt for multiple hosts. `cf-secret` uses the local `cf_lastseen.lmdb` database to find the corresponding host key for a given IP or host name, and then loads the RSA public key for that host from the `ppkeys` directory. ## Showing cf-secret metadata The `print-headers` command can be used to show metadata about an encrypted file: $ cf-secret print-headers message.secret Version: 1.0 Encrypted-for: SHA=08582c4100dfda8db6a4bd7e28d1de4bdac0f5303dc192b51e672c06f4ea2fb1 ## Decrypting files Finally, the `decrypt` command can restore the original message: $ cf-secret decrypt message.secret -o message.decrypted $ cat message.decrypted Hello, secret! (Must be run on the host that the file was encrypted for). ## Video Watch the video below to see cf-secret in action: {{< youtube lJ7UymB6G-k >}} ## Feedback If you have any features, ideas, or suggestions for how to expand secret management in CFEngine, feel free to contact us through one of these channels: - [Twitter](https://twitter.com/cfengine) - [Bug tracker](https://northerntech.atlassian.net/) - [Mailing list](https://groups.google.com/forum/#!forum/help-cfengine) - [IRC](https://web.libera.chat/?channel=#cfengine) -------------------------------------------------------------------------------- title: "CVE-2019-19394 - Mission Portal JavaScript injection vulnerability" url: /blog/2020/cve-2019-19394/index.md date: "2020-04-16" -------------------------------------------------------------------------------- A vulnerability was recently discovered in CFEngine Mission Portal and has now been fixed. Under certain circumstances, it was possible to inject JavaScript code into data presented in Mission Portal, that would be run in the user's browser. This security issue was fixed in CFEngine 3.10.7, 3.12.3, and 3.15.0, and will be mitigated by [upgrading your hub](https://docs.cfengine.com/docs/3.15/guide-installation-and-configuration-upgrading.html) to one of these versions (or later). No other action is required than upgrading the Hub. This issue is present in CFEngine Enterprise 3.7 versions, 3.10.0 through 3.10.6, as well as 3.12.0, 3.12.1, and 3.12.2. All customers have been notified prior to this announcement and had time to address the issue. Any community users who use CFEngine Enterprise Free 25 should upgrade immediately. Open source versions of CFEngine (CFEngine Community) are not affected, as they do not include the Mission Portal Web UI. The security of the CFEngine product and our users is something we take very seriously, and we will continue to look for, fix and responsibly disclose serious weaknesses in our product(s). This issue has been registered as CVE-2019-19394 in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19394). If you have any questions or concerns please contact CFEngine support if you have a support contract or email -------------------------------------------------------------------------------- title: "CFEngine 3.12.4-2 released" url: /blog/2020/cfengine-3-12-4-2-released/index.md date: "2020-04-06" -------------------------------------------------------------------------------- Today we released 3.12.4-2. Shortly after releasing 3.12.4-1, we identified a permissions problem that prevents 3.12.4-1 from contributing data to a 3.15 hub setup for federated reporting; this release fixes that permission issue. As always, you can find Enterprise packages on our [Enterprise downloads page](https://cfengine.com/download/) and Community packages can be found in [our public repositories](https://cfengine.com/product/community/cfengine-linux-distros/) and on our [Community downloads page](https://cfengine.com/product/community/cfengine-linux-distros/). Additionally, please note, [cf-remote](https://github.com/cfengine/cf-remote) can be used to install our released Enterprise or Community packages. -------------------------------------------------------------------------------- title: "CFEngine migrates to FORTH language for policy in version 4.0.-4 (non-breaking change!)" url: /blog/2020/cfengine-migrates-to-forth-language-for-policy-in-version-4-0-4-non-breaking-change/index.md date: "2020-04-01" -------------------------------------------------------------------------------- ## Announcing CF4\! (or is it CF-FORTH?\!) I imagine you didn't expect such a big release so soon after our most recent release of [3.12.4 and 3.15.1 on March 26](https://cfengine.com/company/blog-detail/cfengine-3-12-4-and-3-15-1-released/), but here it is: our alpha-release. Thus the reason for the .-4 in the version number. Of course choosing -4 has something to do with the fun of spelling FORTH without the 'U'. Also, it's nearly a palindrome and I imagine we'll have a few alphas/betas before the final release is finished. (a good palindrome: a man a plan a canal panama) We learned our lesson from the CFE2 to CFE3 transition and have completely preserved backwards compatibility while adding some amazing new language and runtime features. ## A brand new(old) language: FORTH\! With CF4 we use the FORTH language invented by [Charles H. Moore](https://en.wikipedia.org/wiki/Charles_H._Moore) to really bring a lot of power to the policy language. Define your own words to create promises, bundles, variables, you name it\! - Interactive REPL/debugger - Introspection of vars/classes/promises - Easy-to-construct DSL (domain specific language) for your policy. Create forth words to represent important aspects of your policy\! - New language constructs provided by our transition to a Forth-based policy language - Control structures like IF THEN ENDIF, IF THEN, BEGIN UNTIL, REPEAT - Exception handling: CATCH THROW - Extensive arithmetics operations - Recursion ## Let's get started\! Download cf4 and run it like this: ``` wp-block-code $ cf4 ``` This will present you with the following information and be ready for your input: ``` wp-block-code CFEngine Core 4.0.-4 ``` (You might be confused by the lack of a prompt, but we feel keeping things simple is important, just as with my favorite text editor ed(1)). You can add a prompt if you wish (see the `info` pages). At this point, you can start entering policy manually or loading policy from files, either directly including them or by requiring them. ``` wp-block-code include promises.cf4 S" promises.cf4" included require my-library.cf4 S" promises.cf4" required ``` Here is a simple hello world example in CFE3: ``` wp-block-code bundle agent main { reports: "Hello world!"; } ``` The equivalent CF4 policy is much more succinct and clear: ``` wp-block-code S" Hello world!" reports promise ``` By default in CF4, there is an assumed `bundle agent main` present. If you want to specify a different name simply add it to the end\! ``` wp-block-code S" Hello world!" reports promise S" main" agent bundle ``` Now that it is entered into the system you can evaluate the promises with the word `keep_promises`: ``` wp-block-code % keep_promises R: Hello world! ``` ## A more advanced example Now let's try a slightly more complicated example from Diego Zamboni's excellent book: [Learning CFEngine](https://cf-learn.info/). ``` wp-block-code bundle agent edit_motd { vars: "motd" string => "/etc/motd"; files: "$(motd)" create => "true", edit_line => addmessage; } bundle edit_line addmessage { insert_lines: "This system is managed by CFEngine 4!"; } ``` In CF4 you would first define a few words for the `motd` path and your `edit_line` bundle details: ``` wp-block-code : motd S" /etc/motd" ; : addmessage S" This system is managed by CFEngine 4!" insert_lines edit_line bundle ; ``` Next, you would use those words to express the files promise: ``` wp-block-code motd files ``` The word `files` takes the string returned by the word `motd` and starts formulating a new promise. At this point, we can add any number of attributes before we "close out" the current promise. ``` wp-block-code S" true" create addmessage edit_line ``` And finally, we close out the promise with the `promise` word ``` wp-block-code promise ``` And execute/evaluate the policy with `keep_promises` ``` wp-block-code keep_promises info: Created file '/etc/motd', mode 0600 info: Edit file '/etc/motd' ``` Compare that CFE3 policy above with the precision of the equivalent CF4 policy we have built up\! (this one includes the `bundle` and `policy` words for fun, not required) ``` wp-block-code : motd S" /etc/motd" ; : addmessage S" This system is managed by CFEngine 4!" insert_lines edit_line bundle ; motd files true create addmessage edit_line promise S" edit_motd" agent bundle policy ``` At this point, you can examine vars, classes, promise outcomes and various other aspects of the system in real-time. When you are done simply type `bye` and you are done\! There is so much more to explore with CF4, but we'll leave that for another time. If you'd like to hack-a-long (and trust me, this is a SERIOUS hack at this point, not all the build steps are even documented right now\! :O (I really need to learn autotools better)) ``` wp-block-code git clone -b cf-forth --depth 1 --recursive git@github.com:craigcomstock/core.git ``` Cheers, be well, and trust in your promises. This April Fools joke brought to you by the CFEngine Forth fan club\! ![](images/3b5c6eeec85b1dd7.gif) -------------------------------------------------------------------------------- title: "CFEngine 3.12.4 and 3.15.1 released" url: /blog/2020/cfengine-3-12-4-and-3-15-1-released/index.md date: "2020-03-26" -------------------------------------------------------------------------------- We are today very excited to bring you new updates to CFEngine. This is a set of patch releases for the CFEngine 3.12 LTS and 3.15 LTS series. We usually release new patch releases every 6 months, but we want to bring new features and all improvements and bug fixes to our users as soon as possible. Hence these early releases. In CFEngine 3.15 LTS we introduced Federated Reporting, our single pane of glass reporting architecture. This is a great new feature that allows you to set up a dedicated Hub that collects all reporting data from your entire infrastructure to really provide a single pane of glass into all your operations. In this patch release, we have included several performance improvements and bug fixes. There are no new features or larger changes in these patch releases. We focus on stability, improving performance, fixing bugs and are actively listening to open source users and customers alike when planning what to fix. We hope you enjoy the faster release this time and benefit from some of the improvements we have made. ## New language macros We have now introduced a new set of binary macros to the policy language. This greatly simplifies the process of being in control of what policy is executed in what parts of your infrastructure. You can more safely introduce new features and control how they are used. The versions macros now also accept single-digit input. The new macros are the following - `@if maximum_version(x)` - `@else` - `@if between_versions(x, y)` - `@if before_version(x)` - `@if at_version(x)` - `@if after_version(x)` They are available in both 3.12.4 as well as 3.15.1. These new macros are specifically intended to further improve forward and backward compatibility. Usually, we do not introduce features in a patch release. However, we evaluated the risk of introducing these new features in a patch release to be very small compared to the value they bring. ## Improvements and valuable changes in CFEngine We have fixed over 20 different issues relating to the open-source part of CFEngine in these releases, and more than 50 bugs in total. Community member Aleksey Tsalolikhin has made a contribution to inventory NFS servers, that we have included in both CFEngine 3.12.4 and 3.15.1, after a great discussion about this on the mailing list. We have also fixed several issues, making CFEngine more stable, and faster. Such as an issue causing duplicate entries in `sys.interfaces`, and `sys.hardware`, both a crash, as well as memory leaks in JSON and mustache code, we have made `classfiltercsv()` fail properly on an invalid class expression index, using variable references with nested parentheses no longer cause errors. As well as many more improvements. We have improved the SUSE and OpenSUSE platform support, by for example adding Zypper as the default package manager on OpenSUSE, added the correct symlink to Python in SUSE, made sure that OpenSUSE uses the same paths as SUSE. ## Red Hat Enterprise Linux 8 support CFEngine 3.15.0 already supported RHEL 8, but we have now also added RHEL 8 support to CFEngine 3.12.4. Both versions only support RHEL 8.1 and higher, as there are incompatible changes in SELinux implementation between RHEL 8.0 and 8.1. ## Federated Reporting Federated reporting comes with a lot of benefits. We have focused a lot on the very large scale customers, that needed several CFEngine hubs to manage their massive infrastructure. However, Federated Reporting also makes it easy for smaller organizations to now use multiple Hubs to create different infrastructure segments, while still maintaining all reporting data in a unified place. In CFEngine 3.15.1 we have only made small improvements to Federated Reporting. Mainly, after seeing more heavy use we have fixed some corner-case bugs and sped up the import process. ## Mission Portal In Mission Portal, we have made some very useful improvements in these releases. Most of them are backported to 3.12.4 as well, but we recommend upgrading to 3.15 LTS series soon, in order to get all the benefits. ### Time zone management We have now introduced much improved time zone management so that users can control what time zone reports should be sent out or schedules in. This greatly improves user experience in global infrastructures where Hubs can be located in multiple time zones, or when you are traveling the world for work. ![](images/Screenshot-2020-03-26-at-07.46.57-1.png) Changing the timezone settings of each Mission Portal user is now simple and can be done automatically as you travel. ### License management We have now introduced easier to use and understand license overview in Mission Portal. There is a simple warning as you approach your license allocation. Now, there is also a new inventory report on license allocation, to make this easier to track. This provides great value for the user of Mission Portal, but also makes it simple to schedule a report to the correct authority whenever license usage is running high, so you can give us a call\! ### Dashboard sharing We have made it simpler to share Dashboards in Mission Portal. You can now share a Dashboard, either with a specific user, or set of users, or also a set of roles. Anyone you share a Dashboard with will now see who created it. Any user can now also clone a shared Dashboard in order to make their own modifications to it. Dashboards are now no longer required to have a unique screen name, so this makes Dashboard creation much more flexible, especially on systems with multiple users. ![](images/Screenshot-2020-03-26-at-08.01.39.png) ### Improved Usability The Inventory attributes list you see when adding a new Widget to a Dashboard was improved to be a scrollable view, include a search box, and is now alphabetized as well. ![](images/Screenshot-2020-03-26-at-08.11.36.png) ![](images/Screenshot-2020-03-26-at-08.13.37.png) We have fixed a bug in the Host Search, where it returned a "500 - Internal Server Error", a bug that caused false positives in the "Last agent run unsuccessful" Health Check. ## High Availability support We have enabled High Availability support in CFEngine 3.15.1. This was not supported in 3.15.0 after we moved to use PostgreSQL 12 in CFEngine 3.15 series. We have now made it work, and continue to support it on RHEL 6. If you are interested in upgrading your HA setup to a later version of RHEL or a different OS, please get in touch with us. You can watch a [short video about our High Availability](https://www.youtube.com/watch?v=BP15M9Mndo4) functionality here. ## Other improvements We have improved the logging of users, so each login to the Hub will now be registered in the logs with more details, including the username. We have fixed a number of smaller issues. If you upgraded from 3.12.1 to 3.12.2 a bug in our database management code would make a subsequent upgrade to 3.15.0 break. This now works as expected when you upgrade to 3.15.1 instead. In 3.12.3 we made a small change that broke the Mission Portal Host Search functionality, this has been fixed. As always, you can see a full list of changes and improvements in our changelogs 3.12.4: Changelogs for [Core](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html), [Masterfiles](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [Enterprise](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-enterprise.html). 3.15.1: Changelogs for [Core](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html), [Masterfiles](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html). ## Dependency updates We have now updated the following dependencies, in order to get the latest security, performance and reliability improvements. ### 3.12.4 | | | | | ----------- | ------ | ------ | | pcre | 8.43 | 8.44 | | openLDAP | 2.4.48 | 2.4.49 | | libcurl-hub | 7.67.0 | 7.69.0 | | git | 2.24.0 | 2.25.1 | | PostgreSQL | 10.11 | 10.12 | | PHP | 7.2.24 | 7.2.28 | | libcurl | 7.67.0 | 7.69.0 | ### 3.15.1 | | | | | ----------- | ------ | ------ | | pcre | 8.43 | 8.44 | | openLDAP | 2.4.48 | 2.4.49 | | libcurl-hub | 7.67.0 | 7.69.0 | | git | 2.24.0 | 2.25.1 | | PostgreSQL | 12.1 | 12.2 | | PHP | 7.4.0 | 7.4.3 | | libcurl | 7.67.0 | 7.69.0 | ## Contributions We highly encourage you to start **contributing to CFEngine\!** We appreciate all new members of the community. Please check out our [contributing guide](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) in addition to the following suggestions. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Fix issues pointed out by code analysis: ([We recently added](https://github.com/cfengine/core/pull/3631) [some custom rules](https://github.com/cfengine/core/pull/3644), so there are many alerts to fix) And if there is anything, in particular, you want us to fix then please shout out\! Sharing your priorities is also a way of contributing. -------------------------------------------------------------------------------- title: "CFEngine and Covid-19" url: /blog/2020/cfengine-covid19/index.md date: "2020-03-16" -------------------------------------------------------------------------------- The ongoing COVID-19 pandemic brings challenging times for many countries, companies, families, and individuals. Therefore we wanted to make a brief statement about the state of our operations. The CFEngine team has offices in Norway and the USA, as well as remote workers in Italy and other European countries that are all currently experiencing various levels of lock-down. We made a decision last week that we would encourage all our employees to work from home and our offices are now temporarily closed. Our parent company, Northern.tech, has focused on autonomy and being a remote-friendly organization for a long time. This focus and experience, fortunately, makes the current situation easier to manage. We do not anticipate a large disruption in our operations due to this decision. You can read more about our view on remote work on our [company website](https://northern.tech/careers/working-remotely). We continue to develop our products, create new releases and support our customers as usual. New releases are imminent, and there is much to look forward to. If your operations are affected by the current situation, please let us know if there is anything we can assist you with during this time. Lastly, we will not participate in any physical meet-ups, we will not attend any conferences or host any training on-premises in the immediate future. Please reach out to us if you would like an online training, meeting or another contact point. We encourage everyone to listen to their government's advice, take all needed precautions, and stay safe and healthy through this challenging time. -------------------------------------------------------------------------------- title: "Welcoming Dimitrios Apostolou as a CFEngine Champion" url: /blog/2020/welcoming-dimitrios-apostolou-as-a-cfengine-champion/index.md date: "2020-02-14" -------------------------------------------------------------------------------- As we enter 2020 and reflect on the various contributions the project has received we want to take a moment to recognize one of the more prolific contributors as a [CFEngine Champion](https://cfengine.com/engage/cfengine-champions/). ![Jimis](images/dimitrios-apostolou-150x150.jpg) It's my honor to announce and welcome Dimitrios Apostolou as the latest CFEngine Champion. At the time of this writing, he is the fourth most prolific committer in [cfengine/core](https://github.com/cfengine/core) with 1101 commits. ``` 2584 Mikhail Gusarov 2045 Mark Burgess 1430 Sigurd Teigen 1101 Dimitrios Apostolou 825 Kristian Amlie ``` Notably, as an employee of CFEngine AS and Northern.tech AS Dimitrios was responsible for introducing Protocol 2 (TLS), greatly improving the performance of `cf-serverd`, and refactoring the policy evaluation to improve the speed of iteration over large and complex lists and data structures. As a community member, Dimitrios worked in his spare time to open source `cf-monitord` bringing the ability to [write custom measurement promises](https://cfengine.com/company/blog-detail/measuring-values-extracted-from-a-running-log/) to the Community edition. Thank you, Jimis. You have significantly enhanced the CFEngine community through your individual efforts. Know someone that has significantly enhanced CFEngine through their contributions to the project itself or via their engagement with the community? [Nominate them](https://goo.gl/forms/GLJ0lgTX6k)\! -------------------------------------------------------------------------------- title: "CFEngine 3.12.3-2 and 3.15.0-2 released" url: /blog/2020/cfengine-3-12-3-2-and-3-15-0-2-released/index.md date: "2020-02-07" -------------------------------------------------------------------------------- We recently released new builds for our Enterprise and Community packages. This release fixes an issue causing Enterprise Hub packages to fail upgrade in some cases. As part of this release, we also made changes to package names to ensure consistent naming that also includes the target platform in the filename. As always, you can find Enterprise packages on our [Enterprise downloads page](https://cfengine.com/download/) and Community packages can be found in [our public repositories](https://cfengine.com/product/community/cfengine-linux-distros/) and on our [Community downloads page](https://cfengine.com/product/community/cfengine-linux-distros/). Additionally, please note, [cf-remote](https://github.com/cfengine/cf-remote) can be used to install our released Enterprise or Community packages. -------------------------------------------------------------------------------- title: "Upgrading from CFEngine 2 to 3: running the 2 agents side by side with 3" url: /blog/2020/upgrading-from-cfengine-2-to-3-running-the-2-agents-side-by-side-with-3/index.md date: "2020-01-28" -------------------------------------------------------------------------------- CFEngine 2 network communication is insecure by today's standards. ### CFEngine 2 - [CVE-2016-6329](https://nvd.nist.gov/vuln/detail/CVE-2016-6329): CFEngine 2 uses Blowfish cipher (1993) which today is considered: - Weak - Deprecated - Subject to [key recovery attack](https://sweet32.info/) - No security fixes since 2008. - Protocol communications not encrypted; only data transfer (which facilitates attack). - Encryption is off by default. ### CFEngine 3 - All communication is encrypted - Uses TLS 1.3 (current state of the art) - Up to date, maintained, secure from the software vendor - Full Enterprise support, with SLA. ### Solution CFEngine 3 was intentionally designed so that you can install it side by side with 2, so you have time to migrate your policies from 2 to 3. If you are running a large CFEngine 2 installation, you can increase the security of your site by installing CFEngine 3 side by side with CFEngine 2 and using CFEngine 3 for network transport. The following demonstration of running 2 and 3 side by side was done in the [CFEngine Vagrant](https://docs.cfengine.com/docs/3.15/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) sandbox environment, which provides two CentOS 6.5 VMs, called "hub" and "host001", to represent a CFEngine policy distribution server and a managed host. The rest of this write-up documents my steps for building and installing CFEngine 2 and configuring CFEngine 3 to run the version 2 agent alongside the version 3 agent. The idea being, once you've got 3 running side by side with 2, you can take your time to transition your CFEngine policy code base from 2 to 3. #### Procedure Get the latest [CFEngine 2 source code](https://cfengine-package-repos.s3.amazonaws.com/tarballs/cfengine-2.2.9.tar.gz) (2008). Install build tool prerequisites: ```bash yum install -y byacc flex ``` Install library prerequisites: ```bash yum install -y db4-devel openssl-devel ``` Build and install CFE2: ```bash ./configure --prefix=/opt/cfengine-2.2.9 make install ``` Create a stub CFEngine 2 file, `/var/cfengine/masterfiles/cfagent.conf` on the Hub. ``` src src-cfengine2 control: actionsequence = ( shellcommands ) shellcommands: any:: '/bin/echo $(version) agent says "Hello world"' ``` Make sure it works: ``` example [root@hub ~]# /opt/cfengine-2.2.9/sbin/cfagent cfengine:hub:/bin/echo 2.2.9: 2.2.9 agent says Hello world [root@hub ~]# ``` Add a comparable promise to the CFE3 code base, `/var/cfengine/masterfiles/services/main.cf`: ```cf3 ############################################################################### # # bundle agent main # - User/Site policy entry # ############################################################################### bundle agent main # User Defined Service Catalogue { commands: any:: "/bin/echo $(sys.cf_version) agent says 'Hello world'"; } ``` Update CFE3 controls to run CFE2 agent after CFE3 agent: ``` example [root@hub masterfiles]# git diff --color=never controls/cf_execd.cf diff --git a/controls/cf_execd.cf b/controls/cf_execd.cf index 74e5c4b..6d46080 100644 --- a/controls/cf_execd.cf +++ b/controls/cf_execd.cf @@ -51,6 +51,6 @@ body executor control exec_command => "$(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated -f \"$(sys.update_policy_path)\" & $(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated"; !windows:: - exec_command => "$(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated -f \"$(sys.update_policy_path)\" ; $(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated"; + exec_command => "$(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated -f \"$(sys.update_policy_path)\" ; $(sys.cf_agent) -Dfrom_cfexecd,cf_execd_initiated ; /opt/cfengine-2.2.9/sbin/cfagent"; } [root@hub masterfiles]# ``` Allow some time for the host to pick up the changes from the hub (it should happen within 5-10 minutes). Observe the host (host001) is now running CFEngine 3 and 2 agents: `/var/cfengine/outputs/previous`: ``` example [root@host001 outputs]# cat previous notice: Q: ".../bin/echo 3.15.": 3.15.0 agent says Hello world cfengine:host001:/bin/echo 2.2.9: 2.2.9 agent says Hello world [root@host001 outputs]# ``` The first line of output is from CFEngine 3, and the second from CFEngine 2. ### Credits I want to acknowledge the team at [Northern.tech](https://northern.tech), especially Nils Christian Flinder Roscher-Nielsen and Nick Anderson, who helped with this research and reviewed this article. -------------------------------------------------------------------------------- title: "CFEngine 3.15 LTS released" url: /blog/2019/cfengine-3-15-lts-released/index.md date: "2019-12-19" -------------------------------------------------------------------------------- Today marks a new milestone for CFEngine, with the release of the new CFEngine 3.15.0 LTS. This is the newest Long Term Supported CFEngine series, introducing a lot of great stuff. The biggest new feature in CFEngine 3.15 is Federated Reporting, which we will cover later in this blog post, but there are many other new improvements as well. If you are interested to learn more, schedule training, or hear about pricing options, feel free to [reach out to us\!](https://cfengine.com/engage/contact/) Last week, we launched the last release of the CFEngine 3.10 LTS series, and support for 3.10 is coming to an end at the end of this year. CFEngine 3.12 LTS is still under standard support for another 18 months, and CFEngine 3.15 will receive standard support for the next 3 years. This is all described in the [CFEngine release schedule](https://cfengine.com/product/supported-versions/). We are always looking for new **contributions to CFEngine\!** Are you unsure how to get started? Please check out our [contributing guide](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) in addition to the following suggestions. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Fix issues pointed out by code analysis: ([We recently added](https://github.com/cfengine/core/pull/3631) [some custom rules](https://github.com/cfengine/core/pull/3644), so there are many alerts to fix) ## New in CFEngine Core We have made a lot of improvements to CFEngine Core. Some of these are brand new to CFEngine 3.15, while some we were also able to backport to the 3.12 series. You can read all the details in our changelogs: - [Core](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-core.html) - [Enterprise](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-enterprise.html) - [Masterfiles](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ### Platform Support In CFEngine 3.15 we have made a few changes to the supported platforms, based on customer conversations, surveys, and as we have earlier announced. For the Enterprise Hub, we have dropped support for - **Ubuntu 14.04** - **Debian 7** For the agent, we have dropped standard support for the following platforms ([contact us](https://cfengine.com/engage/contact/) for [extended support](https://cfengine.com/extended-support/)): - **CentOS/RHEL 5** - **Debian 7** CFEngine 3.15 LTS adds agent support for **CentOS/RHEL 8**, and also provides a new **file system image for use in container environments**. In our documentation, you can see the [full list of supported platforms](https://docs.cfengine.com/docs/3.15/guide-latest-release-supported-platforms.html). If you have any questions or would like extended support for additional platforms, please let us know\! ### Improved database stability We have focused a lot on the stability of the LMDB databases used by CFEngine on the hosts lately. If these end up in a confused state, it can cause problems for the CFEngine agent. We have made several changes to improve the overall stability to avoid this from happening. Also, in CFEngine we now have a tool called `cf-check` that is used to check and manage LMDB database consistency. This tool has received significant improvements and is also available for direct utilization when needed. CFEngine will itself use this to automatically mitigate issues where the LMDB data is not in a consistent state. ## New features in Enterprise Many of the new features we have worked on will greatly benefit the reporting capabilities, Mission Portal and other aspects of the Enterprise version of CFEngine. ### Federated Reporting Federated reporting is a major new feature in CFEngine. It was introduced in the CFEngine 3.14 release earlier this year, and we have received a lot of really good feedback since then. Thank you to all who provided feedback and opinions. Federated Reporting is a fairly simple addition to our already flexible reporting scheme. Currently, we support up to 5000 reporting hosts per CFEngine hub. This enables a single pane of glass for all your reporting, across sites, security zones, datacenters, or however you wish to set up your infrastructure. It simplifies creating and sharing reports on the entire infrastructure you have running, for compliance monitoring or whatever other needs your organization has. ![](images/Screenshot-2019-12-19-at-16.45.35.png) Using the new Hub Management app in Mission Portal, you can control the settings of the Federated Reporting feature. The new host info page has a lot more in-depth information immediately available. ### File Integrity Monitoring In CFEngine 3.15 we have significantly improved the File Integrity Monitoring (FIM) capabilities. We have added a range of new features in Mission Portal, as well as on the back-end. ![](images/file-changes-widget-2019-12-12-38-20.png) There is now a _file changed_ alert condition, and you can trigger events in Mission Portal based on this. There is also a widget in Mission Portal you can use to track changes and In reports, you can see all the file diffs directly in Mission Portal. ![](images/file-diff-2019-12-12-41-26.png) ### Improved Query Builder In Mission Portal, we have a UI based query builder, to enable anyone to build their own SQL queries to all the data in CFEngine without actually writing complex SQL themselves. This has now been significantly improved. The UI experience has gotten a touch-up and feels more intuitive and straight forward to use. The query builder now also has the opportunity to add custom schemas to your queries, not just the data that comes out of the box with Mission Portal. In addition to that, we have now improved the flexibility of the available Join types. Our goal is to make your data more accessible and understandable so you can make the right choices for your infrastructure. ![](images/Screenshot-2019-12-19-at-16.37.13-757x800.png) Query Builder in action. Now with more flexible ways to construct your SQL queries. ### Improved Role-Based Access Control (RBAC) features in Mission Portal We have now improved the RBAC control potential in Mission Portal. This makes it simpler for Admins to make sure that all the Mission Portal users have the correct access rights, and to make it simpler to create new Roles.![](images/2019-12-12-52-25-800x454.png) We have added a powerful UI in Mission Portal to enable the admin role in Mission Portal to assign access to features to roles and groups as they want to. An admin can also easily create new roles. In association with this, we have now also optimized the default RBAC settings of new roles to be more conservative. Previously, RBAC settings were limiting even the Admins' possibility to delete hosts that had never reported. This issue has now been fixed. ### Policy Analyzer In CFEngine 3.13 we introduced the Policy Analyzer for the first time. This is now finally released in a Long Term Supported release. This is a very helpful feature that makes the work of policy writers simpler and more fun. The Policy Analyzer provides a UI to see the status of a policy run, and directly correlate the actual policy and the outcome on the host. Easily filter on policy outcome, dig into each policy file, and understand the outcome of the policy. ![](images/Screenshot-2019-12-19-at-16.31.02-800x620.png) The new policy Analyzer provides in-depth details about your policy and its status. ### Improved Host Info page Several very valuable improvements have been made to the Host Info page, making all the information about any host in your infrastructure immediately available in one place. We have added a long list of new data to the Host Info page, that was previously available but much harder to find. This includes. - Average agent execution interval - Average agent execution time for each policy entry, - First report collection time, - Host bootstrapped time - Last agent execution time - Inventory attributes and corresponding values - All defined classes - All defined variables ![](images/hostInfo-657x800.png) This improved Host Info page has been requested by several customers, and we hope this can significantly improve your workflow with Mission Portal. ### Report on Promises NOT KEPT In CFEngine, a promise can have three outcomes, KEPT, REPAIRED or NOT KEPT. KEPT means that everything was as it should be. REPAIRED means that something was not as it should be, but CFEngine fixed it, while NOT KEPT are the issues that CFEngine for whatever reason was not able to repair. For every policy run, we update the information about the outcome of all promises that are executed. In some cases though, we have seen that infrequently-run promises that are NOT KEPT are easy to lose track of. We have therefore added a new widget to Mission Portal that keeps track of all promises that were NOT KEPT and have not subsequently been KEPT or REPAIRED. ### Faster Reporting after bootstrapping The reporting toolchain of CFEngine is very efficient - but it is not optimized to be immediate - because CFEngine is optimized for managing large scale infrastructure, not small-scale demo environments. However, in 3.15 we have now made several improvements to make sure that bootstrapped hosts show up in Mission Portal and report almost immediately. This improves the workflow of bootstrapping new hosts, as you will more quickly be able to verify they are all there, and reporting as they should be. ### High Availability CFEngine supports a [High Availability](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-high-availability.html) (HA) setup, where multiple Hubs are available to ensure no disruption to service, even if one of the hubs should go down for whatever reason. This is not supported in the 3.15.0 release of CFEngine. We have upgraded to PostgreSQL 12, to get the benefit of many improvements. The drawback is that there is currently no support for PostgreSQL 12 in a part of the setup we use for HA. We will support HA from 3.15.1, so please wait for that release if you depend on an HA setup. ### Miscellaneous We have fixed a myriad of large and small issues, made improvements and added small features. Some examples include - Improved all password fields to not show characters unless you ask for it - Fixed an issue where certain IPv6 addresses showed as IPv4 - Improved discovery of license information. `cf-hub --show-license` will now show you the up to date license information on your CFEngine hub. You can use this to see how the license utilization of each Hub in your infrastructure is. ## Updated dependencies As always, CFEngine bundles all dependencies in our product, so there is no need to worry about meeting these, or managing them yourself. We have updated some dependencies since CFEngine 3.14, and here is a list of the new requirements. | | | | | ----------- | ------ | ------ | | LMDB | 0.9.23 | 0.9.24 | | openSSL | 1.1.1b | 1.1.1d | | openLDAP | 2.4.47 | 2.4.48 | | libcurl | 7.64.1 | 7.67.0 | | libcurl-hub | 7.64.1 | 7.67.0 | | apache | 2.4.39 | 2.4.41 | | postgresql | 10.7 | 12.1 | | php | 7.3.5 | 7.4.0 | | git | 2.21.0 | 2.24.0 | For a full list of dependencies, you can see the [changelogs in our documentation.](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew.html) ### Feedback We would love to hear what you think about this release, and what features you are looking forward to the most. What more do you want? We are still collecting data on what our customers and users are most interested in, and would greatly appreciate more feedback on this short survey\! -------------------------------------------------------------------------------- title: "CFEngine 3.10.7 LTS and 3.12.3 LTS released" url: /blog/2019/cfengine-3-10-7-lts-and-3-12-3-lts-released/index.md date: "2019-12-06" -------------------------------------------------------------------------------- We are now happy to release two new LTS versions of CFEngine, 3.10.7 LTS, and 3.12.3 LTS. ## CFEngine 3.10.7 - end of life This will be the last release of the CFEngine 3.10 LTS series. Standard Support of CFEngine 3.10 LTS ends end of this year. If you would like [extended support](https://cfengine.com/extended-support/), please [contact us](https://cfengine.com/engage/contact/). From the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we see that **CFEngine 3.10 LTS** is maintained and supported until **December 28th, 2019**. That is the end of this year, so you should start planning on upgrading to CFEngine 3.12 LTS, or the soon to be released CFEngine 3.15.0 LTS that is scheduled to be released in the next few weeks. 3.10.7 LTS is the last maintenance release (patch release) of the CFEngine 3.10 LTS series. The goal of this release is to make sure that the stability and reliability for CFEngine users that cannot immediately upgrade to 3.12, and enable a safe upgrade path. As such, this release includes bug fixes and low-risk changes that do not impact the compatibility between previous patch releases. ## CFEngine 3.12.3 LTS This update comes with many important stability and performance improvements and is thus well worth the upgrade from an older version of 3.12 LTS. CFEngine 3.12 LTS brings a lot of innovation, new features and improved performance to CFEngine, and allows you to make the most efficient use of your time. We are looking forward to your feedback on this release. From the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see that **CFEngine 3.12 LTS** is maintained and supported until **June 2021**. The goal of 3.12.3 LTS is to increase the stability and reliability of CFEngine for users and enable a safe upgrade path. As such, this release primarily includes bug fixes and low-risk changes that do not impact the compatibility with earlier patch releases. Do you want to start **contributing to CFEngine**, but are unsure how? Please check out our [contributing guide](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) in addition to the following suggestions. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Fix issues pointed out by code analysis: ([We recently added](https://github.com/cfengine/core/pull/3631) [some custom rules](https://github.com/cfengine/core/pull/3644), so there are many alerts to fix) ## Improvements to CFEngine 3.10.7 In 3.10.7 we have made a series of small improvements. This will be the last update to the 3.10 LTS series, so if you depend on further improvements, please consider upgrading. ### Core We have fixed a bug in `ps` parsing on OpenBSD / NetBSD causing bootstrap to fail. A crash that was caused by Zero-bytes in class guards is no longer causing crashes. Fixed promise results when using `process_stop` in processes type promises. The package modules now hit the network when the package cache is first initialized. The `@` character is now allowed in the key of classic arrays defined by the module protocol. Added derived-from-file tag on hard classes based on the content of `/etc/redhat-release`. ### Enterprise Version specific distro classes are now collected by default in Enterprise (ENT-4752) We have set create permissions for `cf-monitord` files in state directory to `0600`. This now matches the permissions enforced by policy. The affected files are: - `state/cf_incoming.*` - `state/cf_outgoing.*` - `state/cf_users` - `state/env_data` Key rotation now waits for PostgreSQL to be available when starting or restarting the service. ### Masterfiles We have added the ability to avoid limiting robot agents, added and transitioned to using the `master_software_updates` shortcut, added continual checking for `policy_server` state and added documentation on how to enable systemd unit management and disable agents on all hosts. Also, a new `package_module` for `snap` packages has been added. We have made a change to always set `files_single_copy` from augments if it is available, and fixed cleanup of future timestamps from the status table. There are also many other fixes and improvements. You can see the full changelogs for [Core](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [Masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [Enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html) here. ## Changes in CFEngine 3.12.3 There are many improvements to CFEngine 3.12 in addition to the fixes made for 3.10. In addition to that, there are many other fixes details below. You can also see the documentation for the latest release of 3.12 LTS that includes [changelogs for Core, Enterprise, and the MPF (Masterfiles Policy Framework)](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew.html). ### Platform Support We have implemented a change in how we build CFEngine packages from 3.12.3. We now build on all the platforms we support, as opposed to a single older platform. This means that there are now more packages to download, and while all the packages should work on the platform they are built on and newer, we now only test packages on the platform they are intended to work on. To clarify this new policy as much as possible. While we officially support these platforms (and more): - RHEL/CentOS 5,6,7 - Debian 7,8,9 - Ubuntu 14,16,18 To support these platforms we used to build only on: - RHEL/CentOS 4 and 6 - Debian 4 and 7 Now we're building on: - RHEL/CentOS 5, 6, 7 - Debian 7, 8, 9 - Ubuntu 14, 16, 18 ### Containers We have also improved the support for running CFEngine in a containerized environment. While we have for a long time supported running CFEngine in a CoreOS environment, we now provide a much better way of doing this. We have simplified the packaging and management of CFEngine for container hosts by packaging CFEngine as a file system image, you can easily install, and uninstall to upgrade. You can download that image from our [downloads page](https://cfengine.com/product/cfengine-enterprise-free-25). ### Core Abortclasses cause the agent to terminate when a matching class is defined. However, in the past it was terminating too fast, not saving the last recorded values properly. Agent runs that hit abortclasses now record the results. We have added a `newline` to API error responses, and changed response codes in the User API from `204 No Content` to `202 Accepted` in case of _update_ or _delete_ requests. In this version of CFEngine, with the help of community member Joseph Holsten, we have added a [snapcraft package module](https://docs.cfengine.com/docs/3.12/reference-promise-types-packages.html#snap). Thank you very much for your contribution\! ### Enterprise To make managing the utilization of licenses a bit easier, the Hub now properly logs an error if license counts are exceeded. Several issues around this have been fixed and improved. We have made many improvements to the reporting capabilities. We have fixed a SQL schema error during the upgrade, improved logging of reporting patch failures, and turned on verbose logging to see more in-depth information when patch failure errors show up. ### Improved database consistency We have done a lot of work in CFEngine 3.12.3 to make LMDB behave better. We have added several capabilities that make it more self-healing. Corrupt databases will now automatically be backed up, deleted, and if the backup contains usable information, CFEngine will copy that back, to ensure that as much information is kept as possible. We have also changed some time dependant values that caused some databases to change state a lot, to no longer trigger a change. All in all, these changes will make CFEngine 3.13.3 more stable. We have also improved the tool, `cf-check` that does these operations. This tool has gotten a number of improvements in this version of CFEngine. - directories can now be controlled from ENV vars - Added the `--no-fork` to diagnose command - Added the `-M` manpage option and other common options - The `dump` command now dumps DB contents to JSON5, and print structs as JSON objects - The `help` command can now take a topic as an argument - `--dump` option was added to the `backup` command - The `repair` command now preserves readable data in corrupted LMDB files - Errors are now printed when there are no DB files in the state directory ### Mission Portal In Mission Portal, we have added a lot of new features in 3.12.3. In the Host Info page, we have added a lot more information out-of-the-box. You can now find all the details about the host in question in one place, such as the average agent execution interval, the average agent execution time for each policy entry, first report collection time, host bootstrapped time, last agent execution time, and inventory attributes and values on the Host page. ![](images/hostInfo-657x800.png) New improved Host Info page. Here, you can also see a list of all the classes and variables that are defined on this host. You can also directly access measurements taken by CFEngine, such as CPU load or memory usage. We have also made the list of Inventory attributes scrollable, so you don't need to scroll the whole page to find a given value. Admin users of Mission Portal are now allowed to delete hosts that have no classes currently reported. This fixes an issue that made non-reporting hosts difficult to manage. We have also fixed several issues around Scheduled Reports, among others an issue where scheduled reports were not saved properly. In order to search for specific package versions, we have added an _exact match_ option to the `Software Update` Alert type. We have also added a number of new ways to customize Mission Portal. You can now add a company logo, and customize the text on the login page, as well as customize the color scheme of Mission Portal. We have made changes to how the widgets on the Mission Portal Dashboard display information. That they are now quite a bit faster than they used to be. We have added a 10 minutes threshold to "Agent not run recently" health diagnostics category to avoid showing false-positive warnings in case of manual cf-agent execution. We have also fixed another issue with the health diagnostics, where the "Hosts never collected from" was erroneously empty. The _Host count_ widget has been renamed to _Newly bootstrapped hosts_ ## Dependency updates - 3.10.7 In CFEngine 3.10.7 we have updated the following dependencies. As usual, we have updated dependencies in order to get the latest security, performance and reliability improvements. | | | | | -------------- | ------ | ------ | | LMDB | 0.9.23 | 0.9.24 | | openSSL | 1.0.2r | 1.0.2t | | sasl2 | 2.1.26 | 2.1.27 | | libiconv | 1.15 | 1.16 | | libxml2 | 2.9.8 | 2.9.10 | | openLDAP | 2.4.47 | 2.4.48 | | libcurl | 7.64.1 | 7.66.0 | | libcurl-hub | 7.64.1 | 7.66.0 | | apache | 2.4.39 | 2.4.41 | | postgresql-hub | 9.6.12 | 9.6.15 | ## Dependency updates - 3.12.3 In CFEngine 3.12.3 we have updated the following dependencies. As usual, we have updated dependencies in order to get the latest security, performance and reliability improvements. | | | | | ----------- | ------ | ------ | | LMDB | 0.9.23 | 0.9.24 | | openSSL | 1.1.1b | 1.1.1d | | sasl2 | 2.1.26 | 2.1.27 | | libiconv | 1.15 | 1.16 | | libxml2 | 2.9.9 | 2.9.10 | | openLDAP | 2.4.47 | 2.4.48 | | libcurl | 7.64.1 | 7.67.0 | | libcurl-hub | 7.64.1 | 7.67.0 | | apache | 2.4.39 | 2.4.41 | | postgresql | 10.7 | 10.11 | | php | 7.2.18 | 7.2.24 | | git | 2.21.0 | 2.24.0 | **Upgrading?** If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are always happy to assist our customers with upgrading\! You can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure, and get more out of CFEngine\! ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [CFEngine 3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new release, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! ## Brush up your CFEngine knowledge\! If you would like to refresh your CFEngine knowledge or are learning it from scratch, you can attend one of our training sessions. Check the [event calendar](https://cfengine.com/engage/events/) on our website, or get in touch with us to see what the best option in your area is\! There is also an updated version of the [Learning CFEngine](https://leanpub.com/learning-cfengine/) book by Diego Zamboni now available on LeanPub. -------------------------------------------------------------------------------- title: "New CFEngine hub packages released" url: /blog/2019/new-cfengine-hub-packages-released/index.md date: "2019-10-03" -------------------------------------------------------------------------------- Due to a number of vulnerabilities found in the version of Apache we bundle with CFEngine hub, we have upgraded the CFEngine hub packages to use an updated version of Apache. We upgrade from Apache 2.4.39 to Apache 2.4.41. We are now releasing a new version, [CFEngine Hub 3.12.2-5](https://cfengine.com/product/cfengine-enterprise-free-25#3.12). Only new Hub packages are being released, as no other packages are affected by these vulnerabilities. ## The issues fixed There are several issues that have been fixed with this new version of Apache. Out of these, only CVE-2019-10098 should affect CFEngine and is the one we were most concerned with. **low: mod\_rewrite potential open redirect** ([CVE-2019-10098](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10098)) Redirects configured with mod\_rewrite that were intended to be self-referential might be fooled by encoded newlines and redirect instead to an unexpected URL within the request URL. You can see the full list of issues fixed in Apache 2.4.41 here: [https://httpd.apache.org/security/vulnerabilities\_24.html](https://httpd.apache.org/security/vulnerabilities_24.html "Follow link") This dependency upgrade is the only change we have made. So please upgrade your CFEngine hub today. -------------------------------------------------------------------------------- title: "Speeding up PostgreSQL ETL pipeline with the help of GODS" url: /blog/2019/speeding-up-postgresql-etl-pipeline-with-the-help-of-gods/index.md date: "2019-09-30" -------------------------------------------------------------------------------- # Problem to solve When working on the new [Federated Reporting](https://docs.cfengine.com/docs/master/enterprise-cfengine-guide-federated-reporting.html) feature for CFEngine we had to solve the problem of collecting data from multiple CFEngine hubs (feeders) on a single hub (superhub). CFEngine hubs are using PostgreSQL to store data, so, more specifically, the problem was how to collect data from multiple PostgreSQL databases in one PostgreSQL database. And because we are talking about \~1 GiB of SQL data per feeder hub and for example 10 feeders connected to a superhub here, the initial and trivial solution using basically this ETL (Extract Transform Load) pipeline - `pg_dump | gz | ssh | gunzip | psql` - provided really poor performance. The problem was in the last part of the pipeline - importing data using `psql`. Reading and writing 10 GiB of data of course takes a while, but we soon realized that I/O speed was not the bottleneck in this case. Part of the problem is that the feeder hubs and the superhub are potentially using different versions of CFEngine and thus PostgreSQL. Their schemas can also be a little different, with the differences being backwards compatible (new columns having default values, etc.). This means, that the `--inserts` option has to be used when `pg_dump` is run. Worse, the `--column-inserts` have to be used because columns could be in different orders on the feeders and the superhub. And this is what `man pg_dump` has to say about these two options: ``` example --inserts Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can be loaded into non-PostgreSQL databases. However, since this option generates a separate command for each row, an error in reloading a row causes only that row to be lost rather than the entire table contents. Note that the restore might fail altogether if you have rearranged column order. The --column-inserts option is safe against column order changes, though even slower. ``` Given the amount of data, *"generates a separate command for each row"* in our case actually means \~40 millions of commands/rows. And that is **a lot** of processing PostgreSQL has to do. # Solution with the help of GODS When in troubles, GODS, or [Good Old Damn Scripts](https://gitlab.com/Northern.tech/OpenSource/gods), come to help. ## Go parallel The first strategy to make things faster we chose was to run the import (`gunzip | psql`) in parallel. Because I/O was not the bottleneck, utilizing multiple cores to handle the processing of all the PSQL commands seemed promising. First we had to change the superhub's database schema to make it possible to import the new (fresh) data from a feeder next to the older (previous) data from the same feeder because running `DELETE FROM some_table WHERE source = the_feeder` in a transaction together with the data import is much slower than just importing the data into a new table, then running `TRUNCATE old_table` and replacing the references to the old table by references to the new table. And we wanted to avoid the case where a failed import of the fresh data would delete the old data. We made use of [PostgreSQL partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html) to put together data from individual feeders' tables and this also allowed us to swap the old and new data behind the scenes atomically. The next question was how to run things in parallel in a shell. Probably the best choice is to use the [parallel](https://www.gnu.org/software/parallel/) utility which takes care of many issues related to running things in parallel like interleaving output etc. But we didn't want to force CFEngine users to install *GNU Parallel* if they wanted to use Federated Reporting. Another option, a bit worse, is `xargs -P`, but even `xargs` is not available by default on some systems. Last resort - just use a `for` loop in shell backgrounding the individual tasks and keeping track of their numbers and statuses. We could have just chosen the last option which provides worst results (interleaved outputs from the parallel tasks/commands), but since the other two options are better, we wrote a small Bash "library" - [parallel.sh](https://gitlab.com/Northern.tech/OpenSource/gods/blob/master/parallel.sh) - providing the `run_in_parallel()` function which automatically chooses the best possible way to run a given command on given inputs in parallel with up to a given number of concurrent processes. With this we can simply do echo "$dump_files" | run_in_parallel "$(dirname "$0")/import_file.sh" - $CFE_FR_IMPORT_NJOBS and we know that the best possible option to run `import_file.sh` on all the dump files in parallel will be chosen and the result (exit code, `$?`) will always be `0` in case of success and `1` otherwise. ## Reduce the number of SQL commands The next step was to reduce the number of SQL commands (statements) PostgreSQL had to execute. As the documentation of the `--inserts` option says, inserting data as one row per SQL command is very slow and when the command specifies column order (`--column-inserts`), it is even slower. However, as explained above, we cannot drop the `--column-insert` option or use `--insert` instead. Fortunately, PostgreSQL supports a little trick - multiple `VALUES` tuples (rows) can be specified with a single `INSERT INTO` statement which can significantly speed up the import process. The upcoming new major release of PostgreSQL, version 12, supports a new option for the `pg_dump` command - `--rows-per-insert` - but older versions of `pg_dump` have no such option and thus we cannot rely on it. We need something that would transform multiple lines inserting data into **the same table** using one `INSERT INTO` command per row into multiple lines starting with `INSERT INTO` followed by multiple `VALUES` rows. For example: ``` example INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=701e89c9db156196ef97a65189f575ca6ff36ecf1d373b7df128567744bb8265', 325, '2019-07-02 10:54:32+00', 'OK'); INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=5963e41506ec205fe877ea5acbc5db728af41a601b0bcc45c358d00ceea11fb0', 315, '2019-07-02 10:52:05+00', 'OK'); INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=4a2de5b0ac9f91de4899f1a65508e6408ddc773f3adbc981ff9fc08f0fe7eea8', 310, '2019-07-02 10:52:05+00', 'OK'); INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=8818e5251f608b06804b6862fe601b0fa3d9fa57d3be8c00b92f14c8ab783631', 281, '2019-07-02 10:53:49+00', 'OK'); INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=a51bfbd7e062205766b8fb682269345c0d9282b590557bfdf29a3ae138e504bf', 310, '2019-07-02 10:50:57+00', 'OK'); ``` should be transformed into: ``` example INSERT INTO public.__agentstatus (hostkey, agentexecutioninterval, lastagentlocalexecutiontimestamp, lastagentexecutionstatus) VALUES ('SHA=701e89c9db156196ef97a65189f575ca6ff36ecf1d373b7df128567744bb8265', 325, '2019-07-02 10:54:32+00', 'OK'), ('SHA=5963e41506ec205fe877ea5acbc5db728af41a601b0bcc45c358d00ceea11fb0', 315, '2019-07-02 10:52:05+00', 'OK'), ('SHA=4a2de5b0ac9f91de4899f1a65508e6408ddc773f3adbc981ff9fc08f0fe7eea8', 310, '2019-07-02 10:52:05+00', 'OK'), ('SHA=8818e5251f608b06804b6862fe601b0fa3d9fa57d3be8c00b92f14c8ab783631', 281, '2019-07-02 10:53:49+00', 'OK'), ('SHA=a51bfbd7e062205766b8fb682269345c0d9282b590557bfdf29a3ae138e504bf', 310, '2019-07-02 10:50:57+00', 'OK'); ``` which is easier to parse even for humans. We want to do this using stream processing instead of dumping data into a temporary file and then reading the file to do the transformation because we want to keep the whole pipeline busy to get the best results (shortest times). And similarly to running things in parallel in this case we also don't want to force users to install any extra dependencies. Again, GODS are here to help. In particular, the [`merge_sql_inserts.awk`](https://gitlab.com/Northern.tech/OpenSource/gods/blob/master/merge_sql_inserts.awk) *AWK* script which does exactly the transformation described above. Running 10 GiB of data through *AWK* of course takes a while, 100% utilizing 1 CPU core. However, the transformation is not specific to a particular version of PostgreSQL or anything else specific to the superhub and thus can be done on the feeder hubs in parallel, each processing "only" 1 GiB of data. Plus, it is the total time that matters. And when running data through a pipeline, the effect of adding or slowing down one step can range from slowing the whole pipeline down to speeding it up depending on what the bottleneck (or resource shortage) is. For example, when reading data from disk and sending it over network, adding compression (on the sender) and decompression (on the target) can speed things up if network speed is the limiting factor and the compression is effective and fast enough. Which is something our pipeline is doing. On the other hand, if the compression throughput is much lower than the network speed and/or the compression ratio is low (1:1), the effect of adding it to the pipeline is negative (or negligible at best). In our case, the throughput of *GNU AWK* processing data with `merge_sql_inserts.awk` is only a little bit lower than the throughput of *GZIP* following it in the pipeline so on a multi-core system (recommended for a CFEngine hub) the negative impact of this extra step in the pipeline is very small. In contrast to a very big positive impact on the throughput of PostgreSQL importing the data with much fewer commands/statements. For example, on a smaller data set (\~2 GiB) and a low-end VM, the overhead of adding `merge_sql_inserts.awk` into the dumping part of the pipeline makes it take **14 seconds** instead of **8 seconds**. The importing part of the pipeline, however, goes from **9 minutes** to **1.5 minutes**\! # Good Old Damn Scripts (GODS) Sometimes, actually quite often, some may even say always, simple solutions using good old technologies can be very helpful and very powerful. In our particular case on the setup we use for testing, utilizing *GNU Bash*, *GNU AWK* and optionally *GNU Parallel* resulted in a speedup of the ETL (Extract Transform Load) pipeline from **\~20 minutes** to **\~5 minutes**. From the user's perspective this means that the data they are looking at on the superhub is 5 minutes old at worst instead of 20 minutes old at worst which is quite a difference. If you want some more examples of how good old technologies can help solve modern problems, search for things like *"bash big data"*, *"awk big data"* with your favorite search engine. And because Open Source and UNIX principles are [part of our nature at Northern.tech](https://northern.tech/blog/open-source-at-northern-tech), we are proud of using such simple yet powerful solutions and want to make them available for others. This is how the new [*GODS*](https://gitlab.com/Northern.tech/OpenSource/gods/tree/master) repository was born. At the time of writing this blog post, the repository only contains the two scripts mentioned here, but more are coming soon. They are licensed under the MIT license and we encourage everybody to use them, report any kinds of issues and ideally, submit improvements or even new GOD-grade scripts. -------------------------------------------------------------------------------- title: "CFEngine priorities for 2020" url: /blog/2019/cfengine-priorities-for-2020/index.md date: "2019-09-25" -------------------------------------------------------------------------------- # Where we are This is a very exciting time for the CFEngine product team, the community, and the user base. As we are getting closer to the release of CFEngine 3.15 LTS, we can look back at some great improvements in the last few years. CFEngine was the first product to tackle the challenge of managing large scale infrastructure. While there are now many other solutions in the market, CFEngine is still solving the hardest problems. Over the past few weeks, we met with customers and users that are managing infrastructures of 250 000 to around 1 million servers using CFEngine. These teams are just a handful of people, clearly showing how efficient CFEngine can be when it is coupled with a good strategy and has good alignment throughout the organization. CFEngine really thrives at scale\! Other areas where we see CFEngine used widely, are several highly regulated or high-risk industries, like banking, finance, automotive, and so forth. We know, and our users know, that CFEngine is a stable, reliable and secure automation platform. This is a core value for us, something we cherish and will make sure stays true in future versions of CFEngine. # Where we are going Moving ahead, making sure that these infrastructures are well supported and keep working effortlessly is paramount to us. But we also keep on innovating, for the biggest, for the smaller, for the fastest moving and the extremely security-obsessed. We welcome all of our users to contribute with your perspective and ideas to help us shape the roadmap for 2020 and beyond. Please [share your feedback in this survey](https://www.surveymonkey.com/r/Z8TDRGL). Your thoughts, ideas and great suggestions are what can make CFEngine improve. A key to success for some of the largest deployments of CFEngine so far has been the ability to drive policy from external data sources. Would you like to connect a CMDB or other data sources to CFEngine? We want to make this best-in-class way of working more easily attainable for anyone. Our goal is that CFEngine should continue to be the main choice for any organization concerned with security. CFEngine has always been highly cross-platform, and this is something we continue to deliver on. [CFEngine supports](https://docs.cfengine.com/docs/3.12/guide-latest-release-supported-platforms.html) AIX, Solaris, HP-UX, CoreOS, Linux, and Windows. Community members use it on Mac, FreeBSD, and other platforms. What more platforms do you want support for? Please let us know in the survey. What other aspects are important to you? We always want to listen to you, but now we have a great opportunity to collect feedback and make it impact next year's priorities. So [make your voice heard\!](https://www.surveymonkey.com/r/Z8TDRGL) -------------------------------------------------------------------------------- title: "CFEngine 3.15 LTS beta released" url: /blog/2019/cfengine-3-15-lts-beta-released/index.md date: "2019-09-18" -------------------------------------------------------------------------------- Today we are happy to announce the general availability of CFEngine 3.15.0 beta. CFEngine 3.15 is our upcoming LTS (Long Term Support) release. The main focus of this release has been the new Federated Reporting feature. It also contains a lot of performance work and stability improvements. You can download [CFEngine 3.15 LTS beta here](https://cfengine.com/product/cfengine-enterprise-free-25#3.15.0b1). ## Beta program CFEngine 3.15 is a beta release that is not generally supported, however, the quality is good and interesting new features are available. So, in order for all the new features to be of the best quality, we make it available to you to test already now. We appreciate all the feedback we can get on this beta release. If you test it, you can provide any and all feedback through a quick survey [here](https://www.surveymonkey.com/r/VCC99TD). We are eagerly awaiting your feedback. You can also [email us](mailto:contact@cfengine.com), or [contact us through our webpage](https://cfengine.com/engage/contact/). Feel free to read through the changelogs, and the release announcement to get a feel for it, and provide us with whatever feedback you have. ## Notable changes in CFEngine 3.15.0 beta A more in-depth description of what has changed in 3.15.0 beta can be found in the [3.15 changelogs](https://docs.cfengine.com/docs/3.15/guide-latest-release-whatsnew.html). ### Supported platforms We recently conducted a Platform Survey ([contribute here](https://www.surveymonkey.com/r/CY9KWJG)), and continuously analyze the download numbers to see what platforms are still being widely used and if some platforms are no longer as relevant to our users as they once were. We want to support as many platforms as possible, and try to keep all changes and new features compatible both with all supported and unsupported platforms. But maintaining support involves both a long term commitment as well as investments in HW, SW and time. For the 3.15 LTS series we, therefore, suggest dropping the support for the following platforms. For the Enterprise Hub, we will drop support for - **Ubuntu 14.04** - **Debian 7** - **RHEL 6** For the agent, we propose to drop the following platforms: - **CentOS/RHEL 5** - **Debian 7** If you need continued support, please let us know, so we can discuss potential solutions or change our plans. We will add support for **RHEL 8** and improve support for **CoreOS. **More information on the CFEngine 3.15 [supported platforms page](https://docs.cfengine.com/docs/3.15/guide-latest-release-supported-platforms.html). #### RHEL 8 support We are happy to introduce proper support for Red Hat Enterprise Linux 8 already now. We worked with the Red Hat team and made sure that the SELinux settings make sense for CFEngine so that they will play well together. CentOS 8 is scheduled to be released during September, and we will make sure this also works well with CFEngine before the final release of CFEngine 3.15 LTS. #### CoreOS support While CFEngine has supported CoreOS for a while, we are now introducing a new and improved workflow. By packaging CFEngine as a file system image, we make it easier to install CFEngine onto the base operating system and maintain the CFEngine instance in CoreOS over time. ### Mission Portal Performance In 3.15 we bring a large set of performance improvements to the Mission Portal UI. The structure of many queries has been improved, as well as the loading of many of the default widgets. We have updated and refactored much of the underlying framework code, to make the dashboards a lot more responsive. This will benefit all the users of the CFEngine Mission Portal. We have been doing extensive scale testing, as well as working with customers to identify bottlenecks and improve workflows. This means a lot, from the smaller architectures of only a few hundred hosts all the way to the largest ones that will use Mission Portal's new Federated Reporting mode. ### Federated Reporting We introduced a preview of Federated Reporting in the previous non-LTS release, 3.14. Based on customer feedback, this has been improved and is now available in the CFEngine 3.15 LTS beta. This is a major new feature in CFEngine that will make consolidated reporting and management of a large scale infrastructure much simpler than it is today. The size of the infrastructure under management with CFEngine varies from the quite small to the very large, and is commonly in the 10's of thousands of servers and above. We keep providing and improving the best tools for this particular market. Managing a quarter of a million servers with only one CFEngine policy writer, or managing around a million servers using CFEngine with a very minimal team of around 5-6 people are use cases that show the capability of CFEngine today. We are working to ensure Mission Portal scales to their needs, and that they can fully utilize our reporting and inventory capabilities across the entire infrastructure, through a Single Pane of Glass. The documentation ([available here](https://docs.cfengine.com/docs/3.15/enterprise-cfengine-guide-federated-reporting.html)) will easily help you understand how to set it up and get started. If anything is unclear, please do not hesitate to get in touch with us, and we will explain it, as well as improve the documentation. You can also read more about this feature in the [CFEngine 3.14 release blog post](https://cfengine.com/company/blog-detail/cfengine-3-14-0-released/). ### Stability and performance As a Long Term Support release, we see stability as one of the main features. CFEngine is trusted by the community and our customers alike. In CFEngine 3.15 we are focusing on the code quality and stability of the product. We are working on tooling to better support issues relating to our state databases. Tools like the very helpful cf-check will make it easier to mitigate issues concerning state databases, as well as improve the ability of support to get to the root cause of issues more easily. We continue to improve the score of CFEngine with the [LGTM](https://lgtm.com/projects/g/cfengine/core/context:cpp) code quality database, to make sure the great quality stays high. ## Timeline We are eager to get the final version of this release into the hands of our customers. We are aiming for a release of the final version of CFEngine 3.15.0 LTS to happen during Q4 2019. As always, our LTS releases are supported for 3 years. If you require a longer support timeframe, please get in touch with us, to discuss your options. ## Improve CFEngine Do you want to help improve CFEngine? Are you intrigued by how CFEngine achieves everything, what is going on under the hood? If you want to contribute features or fixes to the next set of releases we would warmly welcome that\! The window for features is not closed for 3.15 yet, so don't hesitate. We appreciate help with fixing potential bugs in the beta, improving the performance if there are any identified issues, or generally implementations of suggestions moving forward. If you want to start contributing but are unsure how: - Send documentation [updates](https://docs.cfengine.com/) as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy]() that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core). You can also [read through our contribution guidelines](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) before you get started. -------------------------------------------------------------------------------- title: "Updated Windows packages" url: /blog/2019/dependency-vulnerabilities-make-us-re-release/index.md date: "2019-08-26" -------------------------------------------------------------------------------- Recently a security flaw, [CVE-2019-1552](https://www.openssl.org/news/vulnerabilities.html#CVE-2019-1552), has been discovered in OpenSSL. This vulnerability affects the **Windows Enterprise** agent packages. To mitigate this security vulnerability we have rebuilt CFEngine with the fix to this issue. These packages have been re-released with the version number CFEngine 3.12.2-4. As always, you can download CFEngine Enterprise packages from [the download page](https://cfengine.com/download/). Note that only the affected packages have been re-released. CFEngine Community wasn't affected at all, due to lack of affected feature. Upgrade today, and make your automation even more secure\! -------------------------------------------------------------------------------- title: "Measuring values extracted from a running log" url: /blog/2019/measuring-values-extracted-from-a-running-log/index.md date: "2019-08-13" -------------------------------------------------------------------------------- Recently I wanted to start measuring the length of time it took for PostgreSQL to acquire a lock so that I could keep an eye on how it changes over time. My PostgreSQL logs contain entries like the following that record the amount of time in ms it took to acquire a lock. ``` example 2019-06-11 18:49:39 GMT LOG: process 10427 acquired AccessShareLock on relation 17949 of database 16384 after 1118.396 ms at character 269 ``` Measurement promises store and track values. Values can be sampled from either a *pipe* (command output) or a *file* as defined by the promises `stream_type`. Values are stored according to the promises `history_type`. When a measured value is not expected to change frequently a `history_type` of *scalar* or *static* will result in the measurement being sampled less frequently and the single value with compressed statistics will be stored. A `history_type` of *log* results in the measured value being logged as an infinite time-series in `$(sys.statedir)/_measure.log`. A `history_type` of *weekly* results in the storing of a two-dimensional time average over a weekly period. Measurements with `history_type` of *weekly* are automatically graphed in Mission Portal if they are collected. Measurements in CFEngine have been around for a long time, but 3.10 brought custom measurements from the closed source enterprise code base to the open source core code base. Let's review this policy to extract and measure ms PostgreSQL had to wait before acquiring the lock. ```cf3 bundle monitor measure_psql { measurements: am_policy_server:: "/var/log/postgresql.log" handle => "psql_lock_wait_before_acquisition", comment => "How long psql had to wait before it acquired a lock", stream_type => "file", match_value => psql_acquired_AccessShareLock, data_type => "int", units => "ms", history_type => "weekly", if => fileexists( "/var/log/postgresql.log" ); } ``` Measurement promises are evaluated by `cf-monitord` and must reside within *monitor* type bundles. Hosts that have the `am_policy_server` class defined will scan the postgresql.log file, extract the value to measure as defined by `body match_value psql_acquired_AccessShareLock` and store it in a two-dimensional time average if the file exists. The values related to this measurement are stored in the `mon` bundle scope, with names based on the promise *handle*. - **mon.value\_psql\_lock\_wait\_before\_acquisition** The last sampled value for the promise with handle `psql_lock_wait_before_acquisition`. - **mon.av\_psql\_lock\_wait\_before\_acquisition** The average sampled value for the promise with handle `psql_lock_wait_before_acquisition` over the stored weekly history. - **mon.dev\_psql\_lock\_wait\_before\_acquisition** The standard deviation sampled value for the promise with handle `psql_lock_wait_before_acquisition` over the stored weekly history. Next, lets review `body match_value psql_acquired_AccessShareLock` where the details of how the sampled value is extracted from the log file reside. ```cf3 body match_value psql_acquired_AccessShareLock # @brief Average time to successfully acquire AccessShareLock from log file { select_line_matching => ".*acquired AccessShareLock.*"; extraction_regex => ".*after (\d+)"; track_growing_file => "true"; select_multiline_policy => "average"; # average, sun, first, last } ``` `select_line_matching` takes an *anchored* regular expression matching lines that we should consider. `extraction_regex` is an *unanchored* regular expression that should contain a single back-reference for extracting the value that should be sampled. `track_growing_file` allows cf-monitord to remember the position from which it last read in the file so that the same log entries are not counted multiple times. Finally, during each sample there may be multiple log entries recording that a lock was acquired. We use `select_multiline_policy` to average the extracted values together. Once the policy has been running I can see the last recorded, average, and standard deviation by running `cf-promises --show-vars=psql`. ``` example [root@demohub ~]# cf-promises --show-vars=psql Variable name Variable value Meta tags default:mon.av_psql_lock_wait_before_acquisition 1640.40 monitoring,source=environment default:mon.dev_psql_lock_wait_before_acquisition 0.00 monitoring,source=environment default:mon.value_psql_lock_wait_before_acquisition 5468.00 monitoring,source=environment ``` In Mission Portal of CFEngine Enterprise the measurements are graphed so they can be visualized over time. ![](images/measurement-preview-mission-portal-Screenshot_20190626_124132.png) ![](images/measurement-past-4-hours-Screenshot_20190626_124228-800x310.png) ![](images/measurement-past-week-Screenshot_20190626_124312-800x294.png) -------------------------------------------------------------------------------- title: "CFEngine 3.12.2-3, 3.14.0-2 released (mitigating PostgreSQL CVE-2019-10164)" url: /blog/2019/cfengine-3-12-2-3-3-14-0-2-released-mitigating-cve-2019-10164/index.md date: "2019-08-06" -------------------------------------------------------------------------------- On \[2019-07-29 Mon\] we released new builds of our Enterprise Hub packages for 3.12.2 and 3.14.0. This release addresses [CVE-2019-10164](https://nvd.nist.gov/vuln/detail/CVE-2019-10164). > PostgreSQL versions 10.x before 10.9 and versions 11.x before 11.4 are > vulnerable to a stack-based buffer overflow. Any authenticated user > can overflow a stack-based buffer by changing the user's own password > to a purpose-crafted value. This often suffices to execute arbitrary > code as the PostgreSQL operating system account. CFEngine Enterprise LTS versions 3.12.0, 3.12.1, 3.12.2-1, 3.12.2-2, and non-LTS version 3.14.0 vendor PostgreSQL versions affected by this vulnerability. In the default configuration as access to `root` or `cfpostgres` local users must be achieved first. The replacement hub packages can be found on the [downloads page](https://cfengine.com/download/). If you have any questions about this, please email [contact@northern.tech](mailto:contact@northern.tech?subject=Question%20about%20mitigating%20CVE-2019-10164). -------------------------------------------------------------------------------- title: "CFEngine 3.14.0 released" url: /blog/2019/cfengine-3-14-0-released/index.md date: "2019-07-01" -------------------------------------------------------------------------------- Today we are very proud and happy to launch our latest non-supported release, CFEngine 3.14.0. 3.14 is a great number, being the closest we will get to π, we also wanted to introduce something very special this time around, and we did\! # New features Let's start with an overview of some new changes debuting in CFEngine 3.14. ## Improved Role Based Access Control (RBAC) In CFEngine 3.14 we have introduced a new backend for managing RBAC settings, as well as a whole new UI in the Mission Portal to manage this. This allows for more granular RBAC settings and makes it simple to set up roles with very limited and specific access. This new Mission Portal & API RBAC is based on existing roles. RBAC is a tricky topic, and we advise to create specific roles when users should have specific access. The permissions are purely additive, i.e. they give permission to access something. Every role has a set of permissions, and in the case where a user has more than one role, she has access to all the permissions of those roles. ## Other new additions In addition to the major news about introducing a Single Pane of Glass for all your reporting needs, there are also a number of small new features and bug fixes in 3.14. We have done a lot of work on the Mission Portal UI. We have for example made an improvement in Mission Portal, that makes it easier to see when a specific package was actually installed, and last modified, making software updating easier. For overall faster reporting, we have spent a significant amount of time improving the CFEngine hub main loop, to make it both more robust and faster. We have made several improvements to the Policy Analyser introduced in CFEngine 3.13, amongst others it will now also show the content of def.json and all other files that are a part of the distribution, not only .cf files. For those customers using the CFEngine High Availability mode, we have improved the setup documentation significantly, so this is now simpler than ever to use. Also, a bug that prevented files_single_copy from being defined from Augments has been fixed. # Introducing Federated Reporting In this release, we launch the beta version of CFEngine Federated reporting. This is a new feature that we have worked on for a while, that makes our reporting capabilities even better, and allows you to get a full 360˚ view of your entire infrastructure. Each Policy Hub will now be able, in addition to hosting its own Mission Portal, also to report all its data to a central Reporting Hub. This single hub runs a Mission Portal instance that will provide you with all the data from your entire infrastructure. Aggregated and presented just as you want it. ## Why use multiple policy hubs? CFEngine supports a large number of hosts per hub, usually in the realm of 5000 hosts per hub, but this will depend on many factors, and can be both higher or lower. There are many good reasons to set up an architecture with multiple hubs, and we will quickly go through some of them here. ### Global Diversity Servers might reside in many different locations. Due to issues like network latency, firewall configurations, etc. But also who is managing that datacenter, and issues like when servers are taken offline for maintenance, you want a local Policy Hub in the datacenter, to manage the infrastructure. ### Multiple Security Zones Your infrastructure might reside in multiple different security zones, where the end nodes - the hosts - might not all be able to connect to the same Hub. It is then a great choice to set up a separate Hub in each of these security zones to reduce the traffic between them. ### Different network availability We have customers that are running CFEngine in remote devices, that only have internet connection at a certain, sometimes even unpredictable, times. For this infrastructure, they often set up a separate Hub to deal with these types of devices. This includes CFEngine running in what we typically think of IoT devices, vehicles and so on. CFEngine is an excellent configuration management solution for such infrastructure. ### Different Timescales We have many CFEngine customers using CFEngine for infrastructures with inherently different timescales for different aspects of their infrastructure. Some parts of their infrastructure can work at an entirely different time scale than other parts, and you can save a lot of work by allocating for an appropriately scaled Hub. ### Large Scale Infrastructure CFEngine is the only automation solution that truly supports managing a large scale infrastructure. Other solutions can be great for a few hundred hosts under management, but CFEngine thrives with thousands of servers. However, at a certain scale, a single Hub might not be able to handle your entire infrastructure. We support 5000 hosts per Hub in a standard configuration, but through the help and assistance of our excellent support team, we can also go way beyond that. But when you manage many close to 100 000 servers or more, like several CFEngine users do, you need multiple Hubs. With Federated Reporting, however, you can connect them all together to have a single pane of glass to view reporting from your entire infrastructure. ## How does it work? Importing large scale data sets can be a challenge, and with very large scale infrastructure, accurate and detailed reporting and inventory data, this is a challenging, yet important, problem to solve. We have long been working with a client on a similar solution. While they were only able to do a few rounds of importing data to their central server every day, we have reduced the full round time with their data set to around 20 minutes. This means we can go from 5-6 updates of the data each day to a complete refresh twice an hour. The architecture we here introduce works in a 3 layered way. First, all of the hosts, hundreds or more often thousands or tens of thousands that are under management are all running the light-weight CFEngine agent. These hosts are each bootstrapped to one CFEngine hub. A system can often have several Hubs, and these Hubs do not traditionally talk to each other or share information. We now introduce the third layer, the Single Pane of Glass Reporting Hub, often referred to as a Superhub. This Hub collects and collates all the reporting data from each of the Hubs. This allows you to use a single interface to access the full reporting capabilities and inventory data from your entire infrastructure. Our main priority for the first release is that this new SPOG Hub will work with existing installations of CFEngine, without interfering with the existing setup. Connect the existing Hubs to the newly set up SPOG Hub. The collection mechanism will depend on the CFEngine version of the Hub in question. With an existing CFEngine setup, we use a simple ETL (Extract, Transform, Load) process, copying data from the CFEngine hub and integrating it to the SPOG Hub at a given interval. When you also upgrade the other components in your infrastructure to CFEngine 3.14 or higher more opportunities will open. Newer versions of the CFEngine hub will support more advanced data synchronization mechanisms. This will allow for simplified setup, even faster synchronization of data and reduced bandwidth consumption. If there are other features or ideas you have, please share them with us, we are happy to work your priorities into our roadmap. ### Hub Management ### ![](images/Screenshot-2019-06-27-at-13.24.57-800x523.png) CFEngine 3.14 introduces a new Hub Management module. This allows you to set up a new Hub as the Single Pane of Glass Reporting Server - aka a Superhub, as well as enable a regular Hub to be a Feeder Hub, that will feed data to the Superhub. In the Hub Management module, you will also be able to tweak other settings such as the refresh rate of the data. ![](images/Screenshot-2019-06-27-at-13.46.14.png) Depending on how many Hubs, and how many hosts in total you want to show reports from, and how often you want to update the reporting data on the Single Pane of Glass reporting Hub, the load on this will vary, and you should make sure to use a machine that can meet the requirements of your infrastructure. We will get back with more specific HW requirements once it is launched as a supported feature of CFEngine 3.15. ## The path to support We are hoping for as much customer feedback as possible. This will be sifted through, made sense of, and implemented to the best of our ability in the coming months. Towards the end of the year, we will be ready to launch our next LTS series, CFEngine 3.15 LTS, that will feature a fully supported version of CFEngine Federated Reporting. In order to get as much feedback as possible, we encourage all our enterprise users to test this, and let us know how it goes. Even if you only collect data from a single Hub, your feedback is valuable. ## Updated dependencies We have updated some dependencies since CFEngine 3.13. Here is a list of the new requirements. | | | | | -------------- | ------ | ------ | | lcov | 1.13 | 1.14 | | lmdb | 0.9.22 | 0.9.23 | | pcre | 8.42 | 8.43 | | sasl2 | 2.1.26 | 2.1.27 | | libiconv | 1.15 | 1.16 | | libxml2 | 2.9.8 | 2.9.9 | | libyaml | 0.2.1 | 0.2.2 | | openldap | 2.4.46 | 2.4.47 | | libcurl | 7.61.1 | 7.64.1 | | lubcurl-hub | 7.61.1 | 7.64.1 | | apr | 1.6.5 | 1.7.0 | | apache | 2.4.35 | 2.4.39 | | postgresql-hub | 11.0 | 11.3 | | php | 7.2.11 | 7.3.5 | | OpenSSL | 1.1.1 | 1.1.1b | | Git | 2.19.1 | 2.21.0 | ## Try CFEngine 3.14 today We have been working hard on CFEngine 3.14 so that it can work hard for you. Download [CFEngine Enterprise](https://cfengine.com/download/) or [CFEngine Community](https://cfengine.com/product/community/) today, and make your automation even more awesome\! -------------------------------------------------------------------------------- title: "Writing custom service_methods" url: /blog/2019/writing-custom-service_methods/index.md date: "2019-06-17" -------------------------------------------------------------------------------- This post has been [re-published](https://cmdln.org/2019/05/31/custom-services-in-cfengine-3/) with permission. CFEngine provides the [`services` promise type](https://docs.cfengine.com/docs/3.12/reference-promise-types-services.html) to manage the state of a given service. `services` type promises are an abstraction of agent bundles, they can be used to declare the desired state for a collection of things identified by a name. Most commonly `services` type promises are used to manage standard operating system services though they can be used for abstracting other logical states. By default, `bundle agent standard_services` is used for the `service_method` in promises that specify no specific `service_method`. > Why is CFEngine always or never issuing the command to start or stop a > service each agent run? For sysvinit (non-systemd) hosts, the `standard_services` bundle uses the `status` command and interprets the return codes according to the [Linux Standard Base init script actions](https://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html). Unfortunately some init scripts do not follow the standards. If the status command returns zero when a service is **not** running, cfengine will issue the commands necessary to stop the service every time the agent is run. Similarly, if the status command returns zero when the service is **not** running, CFEngine will never issue the commands necessary to start a service. > How can I handle a service status that does not comport with the > standard? One way to deal with misbehaving services is to implement a custom `service_method` that better understands the specifics for a given service. For example, the MacAfee Agent status command simply emits the current status for its services and does not use the expected return codes. ``` example [root@host ]# /etc/init.d/ma status McAfee agent service is not running. McAfee common services is not running. McAfee compat service is not running. [root@host ]# echo $? 0 ``` First, for testing, create a mock init script that matches the behavior seen in the output above. Listing 1: Mock /etc/init.d/ma ```bash case $1 in status) echo McAfee agent service is not running. echo McAfee common services is not running. echo McAfee compat service is not running. ;; start) echo Starting McAfee agent service. echo Starting McAfee common services. echo Starting McAfee compat service. ;; stop) echo Stoping McAfee agent service. echo Stoping McAfee common services. echo Stoping McAfee compat service. ;; *) echo "Unknnown action '$1'. Expecting status|start|stop." ;; esac return 0; # Return zero means service OK/Running per the LSB ``` Next you can write a custom bundle to implement services based on the McAfee behavior. Listing 2: bundle agent mcafee\_services\_handler in example\_custom\_services.cf ```cf3 bundle agent mcafee_services_handler( service_name, desired_state ) # @brief Works around init script not exiting with the appropriate return code for status # (per Linux Standard Base init script actions) { vars: "valid_states" slist => { "active", "inactive" }; "init_script" string => "/etc/init.d/ma"; "ma_status" string => execresult( "$(init_script) status", noshell ); classes: # Define a class for the desired state if the state is one we recognize "desired_$(desired_state)" expression => strcmp( $(desired_state), $(valid_states) ); # It's nice to know if someone is using the policy incorrectly "invalid_desired_state" not => some( $(desired_state), @(valid_states) ); "at_least_one_component_not_running" expression => regcmp( '.*not running.*', $(ma_status) ), comment => "If any component is not running, then we consider the whole service not running"; "some_component_running" expression => regcmp( '.*is running.*', $(ma_status) ); "all_components_running" expression => not( regcmp( '.*not running.*', $(ma_status) ) ), comment => "If no component is not running, then we consider the service running"; commands: desired_active.at_least_one_component_not_running:: "$(init_script) start"; desired_inactive.some_component_running:: "$(init_script) stop"; reports: "Something is wrong, I only expect to be used with 'ma' aka 'MacAfee Agent' but I am being used with '$(this.service_name)'" if => not( strcmp( $(service_name), 'ma' ) ); @if minimum_version(3.11) # This debug message levarages the with attribute to avoid building a # intermediary variable (joined string) that was only useful in a single report "Invalid service state. Selected '$(desired_state)'. Valid states: '$(with)'" with => join( ", ", @(valid_states) ), if => "invalid_desired_state"; @endif } ``` Then you can expose it as a `service_method` attribute value by defining a `service_method` body. Listing 3: body service\_method macafee in example\_custom\_services.cf ```cf3 body service_method macafee { linux:: service_bundle => mcafee_services_handler( $(this.promiser), # The services promiser $(this.service_policy) # The value of service_policy ); } ``` And then you could specify use the services promise Listing 4: bundle agent example\_custom\_services in example\_custom\_services.cf ```cf3 bundle agent example_custom_services { services: "ma" service_policy => "active", # you get to choose the word describing the state service_method => macafee; "my custom service" service_policy => "kablewy", service_method => macafee; } bundle agent __main__ { methods: "example_custom_services"; } ``` Execute the policy to see it in action: Listing 5: Running the policy ```bash cf-agent -KIf /tmp/example_custom_services.cf ``` Reviewing the output we can see that the service was correctly seen to not be running, so the start command was issued. ``` example info: Executing 'no timeout' ... '/etc/init.d/ma start' notice: Q: ".../init.d/ma star": Starting McAfee agent service. Q: ".../init.d/ma star": Starting McAfee common services. Q: ".../init.d/ma star": Starting McAfee compat service. info: Last 3 quoted lines were generated by promiser '/etc/init.d/ma start' info: Completed execution of '/etc/init.d/ma start' R: Something is wrong, I only expect to be used with 'ma' aka 'MacAfee Agent' but I am being used with 'my custom service' R: Invalid service state. Selected 'kablewy'. Valid states: 'active, inactive' ``` -------------------------------------------------------------------------------- title: "CVE-2019-9929 - Internal authentication secrets leaked in logs" url: /blog/2019/cve-2019-9929/index.md date: "2019-05-28" -------------------------------------------------------------------------------- # Description The CFEngine engineering team has recently discovered a severe security issue in the *CFEngine Enterprise* product. CFEngine is using some internal secrets for authentication to the Mission Portal API and the PostgreSQL database when running background maintenance tasks. These internal secrets are randomly generated during the installation process and stored in files which only the *root* user has access to. Unfortunately, the commands that generate and store the secrets were being logged to the `/var/log/CFEngineHub-Install.log` installation log which was world-readable and thus accessible for any user logged in to the system (on the hub machine). *Please note that this only affects the hub hosts, agent hosts don't generate and use such internal secrets.* # Detection To find out if a system is affected, check the permissions of the `/var/log/CFEngineHub-Install.log` file and check if the secrets are there. This is an easy way to perform such check: ``` example $ ll -d /var/log/CFEngineHub-Install.log -rw-r--r--. 1 root root 36410 May 23 11:27 /var/log/CFEngineHub-Install.log $ grep -P '(SECRET|PW)' /var/log/CFEngineHub-Install.log + '[' -f /var/cfengine/CF_CLIENT_SECRET_KEY.tmp ']' + sed -i s/CFE_CLIENT_SECRET_KEY/wBosnkNR247KLVJpIvJYXSAp8zJv1259/ /var/cfengine/share/GUI/application/config/appsettings.php + sed -i s/CFE_CLIENT_SECRET_KEY/wBosnkNR247KLVJpIvJYXSAp8zJv1259/ /var/cfengine/share/db/ootb_settings.sql + sed -i s/LDAP_API_SECRET_KEY/wBosnkNR247KLVJpIvJYXSAp8zJv1259/ /var/cfengine/share/GUI/application/config/appsettings.php + sed -i s/LDAP_API_SECRET_KEY/wBosnkNR247KLVJpIvJYXSAp8zJv1259/ /var/cfengine/share/GUI/ldap/config/settings.php + sed -i '/LDAP_API_SECRET_KEY/s/'\'''\''/'\''wBosnkNR247KLVJpIvJYXSAp8zJv1259'\''/' /var/cfengine/share/GUI/api/config/config.php + CFE_ROBOT_PWD=eFKzEV7yr5Xv0FiYRwLZxIJA7DLT6OXS + rm -f /var/cfengine/CF_CLIENT_SECRET_KEY.tmp ``` The above results are from an affected host, the log file is world-readable (the third `r` in the permissions) and it contains the secrets. # Impact The internal `CFE_ROBOT` Mission Portal user has the *admin* role. Logging in to Mission Portal or authenticating to the API as this user would allow the attacker to change any configurable option in Mission Portal, add, modify or delete user accounts or even change the version control system configuration to distribute policy of their choice to all hosts bootstrapped to an affected hub (including the hub itself). Any user logged in to an affected hub's operating system can read the authentication secrets from the `/var/log/CFEngineHub-Install.log` log file. # Mitigation In order to mitigate the issue, the secrets need to be rotated (regenerated). Care must be taken to avoid logging the new secrets in some place accessible to non-root users. The CFEngine team provides two options to rotate the secrets: - [a shell script](https://cfengine-package-repos.s3.amazonaws.com/enterprise/CVE_2019_9929.sh) ([checksum](https://cfengine-package-repos.s3.amazonaws.com/enterprise/CVE_2019_9929.sh.sha256sum), `63b7da16850858944afd80bf7dfc04edfb8267dc9607809e80ca90c2edb44c89`) and - [a policy](https://cfengine-package-repos.s3.amazonaws.com/enterprise/CVE_2019_9929.cf) ([checksum](https://cfengine-package-repos.s3.amazonaws.com/enterprise/CVE_2019_9929.cf.sha256sum), `ac6e5bfcc309d7c2f029d48ccc6b81f7757833db94b1186bb99306190287915e`). Administrators of affected systems are advised to download and run the shell script or the policy, or to manually execute the steps based on the contents of the shell script. The CFEngine team has also patched the installation scripts to: 1. set the correct permissions for the `/var/log/CFEngineHub-Install.log` installation log file, 2. avoid leaking internal authentication secrets into the log file and to 3. rotate the internal authentication secrets when upgrading CFEngine. These patches are included in both of the recent LTS releases - [3.10.6](https://cfengine.com/company/blog-detail/cfengine-3-10-6-lts-released/) and [3.12.2](https://cfengine.com/company/blog-detail/cfengine-3-12-2-lts-released/) - and will also be included in the upcoming non-LTS 3.14.0 release. In the 3.14.0 release the `CFE_ROBOT` user will no longer have the *admin* role, but a more restricted access. This issue has been registered as CVE-2019-9929 in [the official public CVE registry](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-9929). We have been in contact with and provided a solution to all our customers in advance of publishing this. In case of any questions, feel free to [contact us](https://cfengine.com/support/). -------------------------------------------------------------------------------- title: "CFEngine 3.12.2 LTS Released" url: /blog/2019/cfengine-3-12-2-lts-released/index.md date: "2019-05-20" -------------------------------------------------------------------------------- We are happy to release the 2nd update to the CFEngine 3.12 LTS series. This update comes with many important stability and performance improvements and is thus well worth the upgrade from an older version of 3.12 LTS. CFEngine 3.12 LTS brings a lot of innovation, new features and improved performance to CFEngine, and allows you to make the most efficient use of your time. We are looking forward to your feedback on this release. Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see that **CFEngine 3.12 LTS** is maintained and supported until **June, 2021** 3.12.2 LTS is a maintenance release (also known as a patch release), with the goal to increase the stability and reliability for CFEngine users and enable a safe upgrade path. As such, this release primarily includes bug fixes and low-risk changes that do not impact the compatibility between previous patch releases. Do you want to start **contributing to CFEngine**, but are unsure how? Please check out our [contributing guide](https://github.com/cfengine/core/blob/master/CONTRIBUTING.md) in addition to the following suggestions. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Fix issues pointed out by code analysis: ([We recently added](https://github.com/cfengine/core/pull/3631) [some custom rules](https://github.com/cfengine/core/pull/3644), so there are many alerts to fix) ## Improvements to CFEngine Core The core of CFEngine is Open Source, and in this codebase, we have done multiple improvements to CFEngine Core and the open source part of CFEngine. ### Platform Support With the 3.12.2 release, CFEngine now officially supports Ubuntu 18.10. Ubuntu continues to gain popularity and many CFEngine users rely on it daily. CFEngine provides a wide set of supported platforms, and we are proud to keep support for older, yet widely used platforms, as well as very recent ones. This enables a diverse group of users and customer to get the most out of CFEngine across very different infrastructures. ### **Usability** For improved flexibility, we have added support for setting the package module interpreter more flexibly, and we have made it possible to override the package module's path. We have improved _cf-key_ by adding the _--no-truncate_ option. This option, when used with _--show-hosts_ changes the formatting of the output to be printed in full, with no padding, separated by a single tab character. The output is more useful for parsing by other scripts and tooling. As in the recently released 3.10.6 _cf-key help_ properly notes the ability to delete by key digest. CFEngine has added a new option _--skip-db-check_ to the _agent_ and _execd_. This option allows you to enable/disable database (LMDB) consistency checks. For now, this is disabled by default, but this will possibly change in the future. On AIX the sys.flavor variable has been fixed to show more accurate information, also an issue where CFEngine was not able to correctly parse the output of '_ps_' as some lines were not of the length CFEngine expected has been fixed on AIX. When it comes to '_ps_' we have also improved it so that the '_ps_' command used internally can now handle longer usernames. Unresolved function calls in process_select body are now skipped. Function calls which always fail, like _getuid("nosuchuser")_, are never resolved. Previously this would cause a programming error, since the body is expected to have a list of strings, not unresolved function calls. The function calls are silently skipped (with a verbose message) as this matches the behavior of calling the functions in a vars promise, and using that as a body parameter. We have improved the "_failsafe_" policy run, by setting the _policy release_id_ to "failsafe"/"bootstrap" when running failsafe.cf The CFEngine CSV parser now supports the CRLF line break inside double quotes. This was done as some policy did not run as expected while using this line break, but now it works well. CFEngine has introduced an error when a function defining a variable still fails at the third evaluation pass. Messages about invalid class characters from module protocol have been moved to VERBOSE to make logs less noisy, and _copylink_pattern_ now honors '/../' in copy source and forward slashes are now allowed in module protocol commands. ### **Documentation ** The documentation of how _cf-execd_ and _cf-serverd_ responds to SIGHUP has been added to the respective manpages, as we know that the online documentation is not always the most convenient option. ### **Improved Quality** In this release we have fixed a segfault in '_cf-promises -p json-full_', as well as fixed the parsing of YAML values starting with numbers, fixed a memory leak in '_filesexist()_' function and fixed a segfault in the policy parser. This all improves the stability and quality of CFEngine. All of this makes CFEngine easier to use, and more consistent. Documentation for the latest release of 3.12 LTS includes [changelogs for Core, Enterprise, and the MPF (Masterfiles Policy Framework)](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew.html). ## CFEngine Enterprise For use in larger infrastructure, where you have the need for accurate inventory and reporting, CFEngine Enterprise is the preferred option. These features have received significant improvements in the 3.12.2 release. ### **New and improved APIs** In CFEngine 3.12.2 we have added API support (and documentation of them) for the Health Diagnostics feature. As a part of this, we have also expanded the capabilities of the Health Diagnostics feature, so you can dismiss warnings more easily, and also through the API. We have made the error log message for failed user authentication more specific. Fixed Host API deletion status codes and added _cleanup_historical_data()_ function to our SQL schema. With this release, we have also made sure that our detection of recent Windows versions is working correctly. ### **Reporting** An issue was discovered, where the generation of a report with no errors in the log would silently hang, and not be generated. This has been rectified. In the Inventory reports, we have now improved the filters. This includes fixing a bug where multiple filters on the same attribute, would only result in filtering on the first instance. Now the report will honor all the filters, also on the same attribute. An issue with error handling in _cf-hub_ delta queries has been fixed so that _cf-hub_ will now attempt to do a rebase if it cannot obtain the last report timestamp. This change only affects scheduled runs, not when invoked with \`_--query delta_\` from the command line. This also means that hosts which are not present in the status table will be rebased instead of generated from a delta from 0. We have now also stopped logging it when we successfully find no pending alerts ### **Code Quality** As part of our continuous focus on code quality, we have discovered and fixed memory leaks in both cf-hub and in the enterprise agent. We have also discovered an issue where a few data structures were not correctly managed, and slowly kept growing, but were not detected as memory leaks as they were technically not leaks. This, along with actual memory leaks, has been fixed to make the memory use of CFEngine 3.12.2 LTS much better than in the previous 3.12 releases. This has caused unstable behavior on certain platforms, we have specifically seen it on AIX. These fixes will improve the runtime performance and stability of CFEngine. ### **Mission Portal** In the Mission Portal in 3.12.2, we have added the ability to do a full export and import of settings from one instance to another. This includes dashboards, users and their access rights, scheduled reports, and much more. We have also made sure that new widgets in CFEngine are now added to the default dashboard after a CFEngine upgrade, so they are simpler to evaluate and learn how to use. ![](images/Screenshot-2019-05-16-at-12.20.14-800x450.png) The new Mission Portal Export/Import feature. CFEngine 3.12.2 also introduces a new out of the box widget for decommissioned hosts. Combined with the Added Hosts Widget, this gives you full insight into the number of hosts under management and a continuous view of the changes made to your system. In Mission Portal, we have also fixed an issue where custom categorized host list fails to show the correct host, but instead only shows an error. ### **Security** In CFEngine 3.12 we recently discovered a security vulnerability that qualifies as a CVE, in fact, the first one since the release of CFEngine 3. This is only relevant for the enterprise customers, as it relates to our reporting capabilities and Mission Portal. The vulnerability has been mitigated, all customers notified, and in this release, it is no longer a problem. More details around this will be published soon. ## Dependency updates In CFEngine 3.12.2 we have updated the following dependencies. As usual, we have updated dependencies in order to get the latest security, performance and reliability improvements. | | | | | -------------- | ------ | ------ | | lcov | 1.13 | 1.14 | | pcre | 8.42 | 8.43 | | libyaml | 0.2.1 | 0.2.2 | | openldap | 2.4.46 | 2.4.47 | | libcurl | 7.62.0 | 7.64.1 | | libcurl-hub | 7.62.0 | 7.64.1 | | apr | 1.6.5 | 1.7.0 | | apache | 2.4.35 | 2.4.39 | | git | 2.19.1 | 2.21.0 | | postgresql-hub | 10.6 | 10.7 | | php | 7.2.12 | 7.2.18 | | libxml2 | 2.9.8 | 2.9.9 | | OpenSSL | 1.1.0i | 1.1.1b | | sasl2 | 2.1.26 | 2.1.27 | | libiconv | 1.15 | 1.16 | | CodeIgniter | 3.1.3 | 3.1.10 | OpenSSL 1.1.1b provides support for TLS 1.3, a new major update to the TLS protocol family. It will be used whenever both hub and agent support it, unless explicitly disabled in the policy. **Upgrading?** If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are always happy to assist our customers with upgrading\! You can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure, and get more out of CFEngine\! ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [CFEngine 3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new release, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! ## Brush up your CFEngine knowledge\! If you would like to refresh your CFEngine knowledge or are learning it from scratch, you can attend one of our training sessions. Check the [event calendar](https://cfengine.com/engage/events/) on our website, or get in touch with us to see what the best option in your area is\! There is also an updated version of the [Learning CFEngine](https://leanpub.com/learning-cfengine/) book by Diego Zamboni now available on LeanPub. -------------------------------------------------------------------------------- title: "How can I execute a command that uses command substitution in CFEngine?" url: /blog/2019/how-can-i-execute-a-command-that-uses-command-substitution-in-cfengine/index.md date: "2019-05-13" -------------------------------------------------------------------------------- This was [originally published here](https://cmdln.org/2019/03/08/executing-commands-with-substitution-from-cfengine/), it has been re-published with permission. > How can I execute a command that uses command substitution in > CFEngine? On the console I might execute something like this: Listing 1: Example command substitution ```bash touch /tmp/file-$(date --iso-8601) ls /tmp/file-* ``` ``` example /tmp/file-2019-03-08 ``` I recommend not executing commands using substitution. Instead, prepare all that you need up front. Get the result of the data command and put it into a CFEngine variable, then use the CFEngine variable directly. ```cf3 bundle agent main { vars: "date" string => execresult( "date --iso-8601=second", useshell ); "command" string => "/usr/bin/touch /tmp/file-$(date)"; "result" string => execresult( $(command), useshell ); reports: "CFEngine $(sys.cf_version)"; "Executed $(command)" if => isvariable( result ); "Files:$(const.n)$(with)" with => join( "$(const.n)", lsdir( "/tmp/", "file-.*", false )); } ``` ``` example R: CFEngine 3.13.0 R: Executed /usr/bin/touch /tmp/file-2019-03-08T11:34:37-06:00 R: Files: file-2019-03-08T11:34:21-06:00 file-2019-03-08T11:34:34-06:00 file-2019-03-08T11:34:37-06:00 file-2019-03-08T11:34:24-06:00 ``` But, if you really want to use command substitution you can use backticks directly. ```cf3 bundle agent main { vars: "command" string => "/usr/bin/touch /tmp/file2-`date --iso-8601=second`"; "result" string => execresult( $(command), useshell ); reports: "CFEngine $(sys.cf_version)"; "Executed $(command)" if => isvariable( result ); "Files:$(const.n)$(with)" with => join( "$(const.n)", lsdir( "/tmp/", "file2-.*", false )); } ``` ``` example R: CFEngine 3.13.0 R: Executed /usr/bin/touch /tmp/file2-`date --iso-8601=second` R: Files: file2-2019-03-08T11:36:30-06:00 file2-2019-03-08T11:36:35-06:00 ``` > Why is it so? [CFEngine expands variables wrapped in `$()` and `${}`](https://docs.cfengine.com/docs/3.13/reference-language-concepts-variables.html#scalar-referencing-and-expansion) and [functions are skipped](https://docs.cfengine.com/docs/3.12/reference-functions.html#function-skipping) (e.g. [execresult()](https://docs.cfengine.com/docs/3.12/reference-functions-execresult.html)) if they have parameters that are variables that do not dereference. -------------------------------------------------------------------------------- title: "CFEngine 3.10.6 LTS Released" url: /blog/2019/cfengine-3-10-6-lts-released/index.md date: "2019-05-10" -------------------------------------------------------------------------------- We are now happy to release the 6th update to the CFEngine 3.10 LTS series. This update comes with many important stability and performance improvements and is thus well worth the upgrade from an older version of 3.10 LTS. Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see that **CFEngine 3.10 LTS** is maintained and supported until **December 27th, 2019**. That is the end of this year, so you should start planning on upgrading to CFEngine 3.12 LTS, or the upcoming 3.15.0 LTS that is scheduled to be released around the same time as 3.10 reaches its end of life. 3.10.6 LTS is a maintenance release (also known as a patch release), with the goal to increase the stability and reliability for CFEngine users and enable a safe upgrade path. As such, this release primarily includes bug fixes and low-risk changes that do not impact the compatibility between previous patch releases. Do you want to start **contributing to CFEngine**, but are unsure how? Here are some nifty tricks. - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Contribute with code improvements by checking out the LGTM framework: ## Improvements to CFEngine Core The core of CFEngine is Open Source, and in this codebase we have done multiple improvements to CFEngine Core and the open source parts of CFEngine. ### Usability We have made several improvements to the usability of CFEngine in this release, for example, we have made a change to _cf-key help_ so that the output will indicate the ability to delete by key digest. On AIX we have fixed sys.flavor variable on AIX to show more correct information, also an issue where CFEngine was not able to correctly parse the output of '_ps_' as some lines were not of the length CFEngine expected has been fixed on AIX. When it comes to '_ps_' we have also improved it so that the '_ps_' command used internally can now handle longer usernames. We have improved the "_failsafe_" policy run, by setting the _policy release_id_ to "failsafe"/"bootstrap" when running failsafe.cf The CFEngine CSV parser now supports the CRLF line break inside double quotes. This was done as some policy did not run as expected while using this line break, but now it works well. CFEngine has introduced an error when a function defining a variable still fails at the third evaluation pass. Fixed log message about setting collect_window. Messages about invalid class characters from module protocol have been moved to VERBOSE to make logs less noisy, and _copylink_pattern_ now honors '/../' in copy source. Forward slashes are now allowed in module protocol commands ### Documentation The documentation of how _cf-execd_ and _cf-serverd_ responds to SIGHUP has been added to the respective manpages, as we know that the online documentation is not always the most convenient option. ### Improved Quality In this release we have fixed a segfault in '_cf-promises -p json-full_', as well as fixed the parsing of YAML values starting with numbers, fixed a memory leak in '_filesexist()_' function and fixed a segfault in policy parser. This all improves the stability and quality of CFEngine. All of this makes CFEngine easier to use, and more consistent. Documentation for the latest release of 3.10 LTS includes [changelogs for Core, Enterprise, and the MPF (Masterfiles Policy Framework)](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew.html). ## CFEngine Enterprise For use in larger infrastructure, where you have the need for accurate inventory and reporting, CFEngine Enterprise is the preferred option. These features have also received significant improvements in this release. ### New and improved APIs We have made the error log message for failed user authentication more specific. Fixed Host API deletion status codes and added _cleanup_historical_data()_ function to SQL schema. With this release, we have also made sure that our detection of recent Windows versions is working correctly, also beyond the versions of Windows that are [officially supported](https://docs.cfengine.com/docs/3.10/guide-latest-release-supported-platforms.html) by this version of CFEngine. ### Reporting An issue was discovered, where the generation of a report with no errors in the log would silently hang, and not be generated. This has been rectified. An issue with error handling in _cf-hub_ delta queries has been fixed so that _cf-hub_ will now attempt a rebase if it cannot obtain the last report timestamp. This change only affects scheduled runs, not when invoked with \`_--query delta_\` from the command line. This also means that hosts which are not present in the status table will be rebased instead of generated from a delta from 0. We have now also stopped logging it when we successfully find no pending alerts, to make that log a bit cleaner. ### Code Quality As part of our continuous focus on code quality, we have discovered and fixed memory leaks in both cf-hub and in the enterprise agent. These fixes will improve the runtime performance and stability of CFEngine. ### Mission Portal There are several improvements to the CFEngine Mission Portal frontend. We have improved the readability of reporting information on the "host info" page, as well as renamed \`IP Naming\` to \`Report Collection\` to make it more clear what data we are actually referring to. We have also fixed an error that caused wrong license expiration information to be shown in Mission Portal. ### Security In CFEngine 3.10 we recently discovered a security vulnerability that qualifies as a CVE, in fact, the first one since the release of CFEngine 3. This is only relevant for the enterprise customers, as it relates to our reporting capabilities and Mission Portal. The vulnerability has been mitigated, all customers notified, and in this release, it is no longer a problem. More details around this will be published soon. ## Dependency updates In CFEngine 3.10.6 we have updated the following dependencies. As usual, we have updated dependencies in order to get the latest security, performance and reliability improvements. | | | | | -------------- | ------ | ------ | | lcov | 1.13 | 1.14 | | lmdb | 0.9.22 | 0.9.23 | | pcre | 8.42 | 8.43 | | libyaml | 0.2.1 | 0.2.2 | | openldap | 2.4.46 | 2.4.47 | | libcurl | 7.61.1 | 7.64.1 | | libcurl-hub | 7.61.1 | 7.64.1 | | apr | 1.6.5 | 1.7.0 | | apache | 2.4.35 | 2.4.39 | | git | 2.13.6 | 2.13.7 | | postgresql-hub | 9.6.10 | 9.6.12 | | php | 5.6.38 | 5.6.40 | | redis | 3.2.12 | 3.2.13 | | OpenSSL | 1.0.2p | 1.0.2r | ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are always happy to assist our customers with upgrading\! You can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure, and get more out of CFEngine\! ## Get it now\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [CFEngine 3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new release, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! ## Brush up your CFEngine knowledge\! If you would like to refresh your CFEngine knowledge or are learning it from scratch, you can attend one of our training sessions. Check the [event calendar](https://cfengine.com/engage/events/) on our website, or get in touch with us to see what the best option in your area is\! There is also an updated version of the [Learning CFEngine](https://leanpub.com/learning-cfengine/) book by Diego Zamboni now available on LeanPub. -------------------------------------------------------------------------------- title: "Introducing cf-remote: Tooling to deploy CFEngine" url: /blog/2019/cf-remote/index.md date: "2019-04-30" -------------------------------------------------------------------------------- About a year ago, I wrote a small python script to automate installing and bootstrapping CFEngine on virtual machines in AWS. It had some hard coded IP addresses that I needed to update when I spawned new hosts, but other than that, it worked well. During manual testing, it saved me a lot of time instead of having to do things manually. Deploying CFEngine normally consists of these steps: 1. Determine what CFEngine package to use. 2. Download appropriate package if you haven't already - `curl`. 3. Copy the package to the host - `scp`. 4. Log into the host - `ssh`. 5. Install the package - `rpm` / `dpkg`. 6. Bootstrap CFEngine - `cf-agent -B`. At a company hackathon I decided to make my script into something better, something that would be useful to my colleagues, and maybe even CFEngine users in general. Enter `cf-remote`. ## Info `cf-remote` can be used to show information about a system before installing CFEngine. The `info` command logs into the system, runs a few commands and parses [`/etc/os-release`](https://www.freedesktop.org/software/systemd/man/os-release.html) to present relevant information: $ cf-remote info -H 34.252.28.73 ec2-user@34.252.28.73 OS : rhel (fedora) Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : rpm, yum The `cf-remote` command line tool is written in Python, and uses [Fabric](https://www.fabfile.org/) to log into the system via SSH. (Add `--log-level debug` to see all commands `cf-remote` runs). Note that there are almost no dependencies on the target system. You only need ssh access and a shell which can perform basic UNIX commands like `ls`, `cat`, `which`. The printout shows that it's a Red Hat machine, and CFEngine is not installed yet. ## Install and bootstrap `cf-remote` can install CFEngine on the system above. The only thing you really need to specify is IP address(es): $ cf-remote install --hub 34.252.28.73 --bootstrap 172.31.30.237 Here is the output from the example above: $ cf-remote install --hub 34.252.28.73 --bootstrap 172.31.30.237 ec2-user@34.252.28.73 OS : rhel (fedora) Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : rpm, yum Package already downloaded: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub-3.12.1-1.x86_64.rpm' Copying: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub-3.12.1-1.x86_64.rpm' to '34.252.28.73' Installing: 'cfengine-nova-hub-3.12.1-1.x86_64.rpm' on '34.252.28.73' CFEngine 3.12.1 was successfully installed on '34.252.28.73' Bootstrapping: '34.252.28.73' -> '172.31.30.237' Bootstrap successful: '34.252.28.73' -> '172.31.30.237' We can guess the username and CFEngine version if not specified. Many hosts can be specified in a single command: $ cf-remote install --hub 34.252.28.73 --bootstrap 172.31.30.237 --clients 52.212.51.201,52.212.51.202,52.212.51.203 Or using a newline delimited file: $ cat ./clients 52.212.51.201 52.212.51.202 52.212.51.203 $ cf-remote install --hub 34.252.28.73 --bootstrap 172.31.30.237 --clients ./clients ## Other useful commands The tooling includes a few other useful utilities. Some of these things can be done in CFEngine policy, but `cf-remote` doesn't assume that CFEngine is installed. Thus, these commands can be useful both before and after installing CFEngine. ### File copy The `scp` command can be used to transfer files to the host. $ echo "Hello, world" > txt $ cf-remote -H 34.252.28.73 scp txt Copying: 'txt' to '34.252.28.73' The destination defaults to the home folder of the SSH user (`/home/ec2-user/txt` in this example). The command doesn't actually use an `scp` executable, but Fabric's `Connection.put()`. ### Arbitrary command execution We can also run commands as if we were logged in via ssh: $ cf-remote -H 34.252.28.73 run "cat txt" 34.252.28.73: 'cat txt' -> 'Hello, world' This is especially useful to run commands on many hosts. `-H` accepts a comma separated list or a path to a file (beginning with `./`, `/`, `../` or `~/`). ## Videos `cf-remote` was introduced, with some live demos, in my talk at [CfgMgmtCamp](https://cfgmgmtcamp.eu/) 2019. The video recording is available on YouTube (`cf-remote` part starts at [20:51](https://www.youtube.com/watch?v=q50_QTd1AN4&t=20m51s)): {{< youtube q50_QTd1AN4 >}} We also made some videos showing how to install and use `cf-remote`: [Install cf-remote](https://youtu.be/cEipwS4SePc) [Deploying CFEngine using cf-remote](https://youtu.be/l4kJ4qAvT1I) ## Getting started The tool is completely open source, and available on GitHub: https://github.com/cfengine/cf-remote The README has installation instructions. It should work on any system with curl, Python(3), [Fabric](https://www.fabfile.org/), and [Requests](https://docs.python-requests.org/en/master/). We've tested it on Linux(Fedora) and OS X. Currently, RHEL and Ubuntu targets are supported. Debian, Fedora, and CentOS should also work as they use `dpkg` / `rpm`. The target must be running an SSH server with your key in `authorized_keys`, so Fabric can log in. ### Contributing Anyone can contribute to `cf-remote`, we have a curated list of tickets to pick up here: Additional platform support (beyond dpkg and rpm) and parallelized installation are good next features. -------------------------------------------------------------------------------- title: "Using cf-runagent as non-root" url: /blog/2019/unpriviledged_cf-runagent/index.md date: "2019-04-12" -------------------------------------------------------------------------------- `cf-runagent` is a component for triggering remote agent runs using the CFEngine network protocol. It does not allow for arbitrary commands to be executed, but rather asks the remote host to run the policy it already has. To trigger `cf-runagent` from other systems or web interfaces, you want to be able to run it as non-root. ## Install and bootstrap I will use [cf-remote](https://pypi.org/project/cf-remote/) to set up a demo hub running CFEngine Enterprise 3.12.1: ``` olehermanse@OHMBP-2869 ~ $ cf-remote --version 3.12.1 install --hub 34.251.155.150 --bootstrap 172.31.2.67 --demo ubuntu@34.251.155.150 OS : ubuntu (debian) Architecture : x86_64 CFEngine : Not installed Policy server : None Binaries : dpkg, apt Package already downloaded: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.12.1-1_amd64.deb' Copying: '/Users/olehermanse/.cfengine/cf-remote/packages/cfengine-nova-hub_3.12.1-1_amd64.deb' to '34.251.155.150' Installing: 'cfengine-nova-hub_3.12.1-1_amd64.deb' on '34.251.155.150' CFEngine 3.12.1 was successfully installed on '34.251.155.150' Bootstrapping: '34.251.155.150' -> '172.31.2.67' Bootstrap successful: '34.251.155.150' -> '172.31.2.67' Transferring def.json to hub: '34.251.155.150' Copying: '/Users/olehermanse/.cfengine/cf-remote/json/def.json' to '34.251.155.150' Triggering an agent run on: '34.251.155.150' Disabling password change on hub: '34.251.155.150' Triggering an agent run on: '34.251.155.150' Your demo hub is ready: https://34.251.155.150/ (Username: admin, Password: password) ``` ## SSH `cf-remote` output shows the username and IP we can ssh to: ``` olehermanse@OHMBP-2869 ~ $ ssh ubuntu@34.251.155.150 The authenticity of host '34.251.155.150 (34.251.155.150)' can't be established. ECDSA key fingerprint is SHA256:J2rxvibDSb8KCoz3xdfixHLdkt+znmwt/qaVHXFVRns. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '34.251.155.150' (ECDSA) to the list of known hosts. Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1032-aws x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Wed Apr 10 08:01:30 UTC 2019 System load: 0.16 Processes: 214 Usage of /: 18.2% of 7.69GB Users logged in: 0 Memory usage: 1% IP address for eth0: 172.31.2.67 Swap usage: 0% Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 0 packages can be updated. 0 updates are security updates. ubuntu@ip-172-31-2-67:~$ sudo bash root@ip-172-31-2-67:~# ``` (You will need a text editor, so feel free to install `emacs` if you want). ## Create user ``` root@ip-172-31-2-67:~# adduser cf-runner --disabled-password Adding user `cf-runner' ... Adding new group `cf-runner' (1001) ... Adding new user `cf-runner' (1001) with group `cf-runner' ... The home directory `/home/cf-runner' already exists. Not copying from `/etc/skel'. Changing the user information for cf-runner Enter the new value, or press ENTER for the default Full Name []: Room Number []: Work Phone []: Home Phone []: Other []: Is the information correct? [Y/n] Y root@ip-172-31-2-67:~# sudo -u cf-runner bash cf-runner@ip-172-31-2-67:~$ ``` ## Add policy file The CFEngine components expect a `promises.cf` policy entry point in the `inputs` directory. For a non-root user, this is `~/.cfagent/inputs/promises.cf`. For the purposes of using `cf-runagent` it can be an empty file: ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 error: Can't stat file '/home/cf-runner/.cfagent/inputs/promises.cf' for parsing. (stat: No such file or directory) cf-runner@ip-172-31-2-67:~$ touch /home/cf-runner/.cfagent/inputs/promises.cf && chmod ugo-w /home/cf-runner/.cfagent/inputs/promises.cf ``` ## Generate a key pair CFEngine stores public and private keys in `/var/cfengine/ppkeys/`(root) and `~/.cfagent/ppkeys/`(non-root). We can generate a new key specific to this user: ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 error: No public/private key pair is loaded, please create one using cf-key error: Failed to connect to host: 172.31.2.67 cf-runner@ip-172-31-2-67:~$ cf-key cf-runner@ip-172-31-2-67:~$ ``` ## Trust the server key `cf-runagent` will now trust public keys in `~/.cfagent/ppkeys/`, so we will have to add the server's public key: ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 error: TRUST FAILED, server presented untrusted key: SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6 error: Failed to connect to host: 172.31.2.67 ``` ### Interactively trusting keys The easiest way to trust the key is to run with the `--interactive` option: ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 --interactive WARNING - You do not have a public key from host 172.31.2.67 = 172.31.2.67 Do you want to accept one on trust? (yes/no) --> yes Will trust the key... notice: Trusting new key: SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6 172.31.2.67> !! Unspecified server refusal (see verbose server output) cf-runner@ip-172-31-2-67:~$ ``` ### Manually trusting keys Interactive dialogs can be awkward to script/automate, but we can achieve trust by copying the public key. We need to switch back to root for a second: ``` cf-runner@ip-172-31-2-67:~$ exit root@ip-172-31-2-67:~# cp /var/cfengine/ppkeys/localhost.pub /home/cf-runner/.cfagent/ppkeys/cf-runner-SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6.pub root@ip-172-31-2-67:~# ls -al /home/cf-runner/.cfagent/ppkeys/cf-runner-SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6.pub -rw------- 1 root root 426 Apr 10 08:21 '/home/cf-runner/.cfagent/ppkeys/cf-runner-SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6.pub' root@ip-172-31-2-67:~# chown cf-runner:cf-runner /home/cf-runner/.cfagent/ppkeys/cf-runner-SHA=af4c5dce82c29d18142cb3308f2086bfd0e1123fef948f73209c3053c2d6d7a6.pub root@ip-172-31-2-67:~# sudo -u cf-runner bash cf-runner@ip-172-31-2-67:~$ ``` Note the filename of the public key: ``` /home/cf-runner/.cfagent/ppkeys/-.pub ``` Where `` is the **local** user running `cf-runagent`, and `` is the key digest of the (remote) `cf-serverd` (most likely running as root). If you forget the `.pub` extension, or get the file name wrong in any way, it will not work. ## Trusting the client key In a new installation, `cf-serverd` automatically trusts new keys, placing them into `/var/cfengine/ppkeys`. This behavior is controlled by `trustkeysfrom` in the `cf-serverd` policy. It is recommended to disable or tighten this feature, so the server doesn't automatically trust new keys. If you have already disabled it, and need to manually trust the new client key we generated, you can do so by copying the public key file: ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 error: Connection unexpectedly closed (SSL_read): socket closed error: Connection was hung up while receiving line: error: Connection was hung up during identification! (3) error: Failed to connect to host: 172.31.2.67 cf-runner@ip-172-31-2-67:~$ exit root@ip-172-31-2-67:~# cf-key -p /home/cf-runner/.cfagent/ppkeys/localhost.pub SHA=c57b3c10ff9dd6b051b02157561de26cf11e357523cf39da6fdeb1dc4cfe790a root@ip-172-31-2-67:~# cp /home/cf-runner/.cfagent/ppkeys/localhost.pub /var/cfengine/ppkeys/cf-runner-SHA=c57b3c10ff9dd6b051b02157561de26cf11e357523cf39da6fdeb1dc4cfe790a.pub root@ip-172-31-2-67:~# cf-agent -Kf update.cf && cf-agent -K root@ip-172-31-2-67:~# systemctl restart cfengine3 root@ip-172-31-2-67:~# cf-agent -Kf update.cf && cf-agent -K root@ip-172-31-2-67:~# sudo -u cf-runner bash cf-runner@ip-172-31-2-67:~$ ``` (If you are using the default policy, without altering `trustkeysfrom`, this step is not necessary). ## Verbose server output ``` cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 172.31.2.67> !! Unspecified server refusal (see verbose server output) cf-runner@ip-172-31-2-67:~$ ``` The server is still rejecting cf-runagent. Let's start it in verbose mode to find out why. ``` cf-runner@ip-172-31-2-67:~$ exit root@ip-172-31-2-67:~# systemctl stop cf-serverd root@ip-172-31-2-67:~# /var/cfengine/bin/cf-serverd --no-fork --verbose 2>&1 > /server.log & [1] 7804 root@ip-172-31-2-67:~# tail /server.log verbose: Object delta verbose: Object rebase verbose: Object full verbose: === END summary of access promises === verbose: Setting minimum acceptable TLS version: 1.0 verbose: Setting cipher list for incoming TLS connections to: AES256-GCM-SHA384:AES256-SHA verbose: Listening for connections on socket descriptor 6 ... notice: Server is starting... verbose: CollectCallWorker: no interval specified. Not starting verbose: CollectCallHasPending: false root@ip-172-31-2-67:~# sudo -u cf-runner bash cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 172.31.2.67> !! Unspecified server refusal (see verbose server output) cf-runner@ip-172-31-2-67:~$ exit root@ip-172-31-2-67:~# tail /server.log verbose: 172.31.2.67> Peeked nothing important in TCP stream, considering the protocol as TLS verbose: 172.31.2.67> TLS version negotiated: TLSv1.2; Cipher: AES256-GCM-SHA384,TLSv1.2 verbose: 172.31.2.67> TLS session established, checking trust... verbose: 172.31.2.67> Setting IDENTITY: USERNAME=cf-runner verbose: 172.31.2.67> Received public key compares equal to the one we have stored verbose: 172.31.2.67> SHA=c57b3c10ff9dd6b051b02157561de26cf11e357523cf39da6fdeb1dc4cfe790a: Client is TRUSTED, public key MATCHES stored one. verbose: 172.31.2.67> Received: EXEC info: 172.31.2.67> EXEC denied due to not allowed user: cf-runner verbose: 172.31.2.67> REFUSAL to user='cf-runner' of request: EXEC info: 172.31.2.67> Closing connection, terminating thread root@ip-172-31-2-67:~# ``` The output indicates that `cf-runner` is not an allowed user for `EXEC` requests (cf-runagent). (In CFEngine 3.7 there is no `cf-serverd` unit, stop the entire `cfengine3` unit instead). ## Allow the user to make EXEC requests ### Via augments If you are running masterfiles 3.12.1 or newer, allowed users can easily be controlled via augments. Use your favorite editor to alter `def.json`: ``` root@ip-172-31-2-67:~# cat > /var/cfengine/masterfiles/def.json { "vars": { "control_server_allowusers_non_policy_server": [ "cf-runner" ], "control_server_allowusers_policy_server": [ "cf-runner" ] } } root@ip-172-31-2-67:~# ``` `cat` will overwrite `def.json`, so you might want to use a more advanced editor. Note that the usernames in `allowusers` are self reported, and could be spoofed. Use other ACL's and `allowconnects` as security measures, not `allowusers`. ### Via policy For older versions of the policy framework, you will have to edit the policy manually. `cf-serverd` policy is at `/var/cfengine/masterfiles/controls/cf_serverd.cf`. (In 3.7: `/var/cfengine/masterfiles/controls/3.7/cf_serverd.cf`). You need to add a list of allowed users to `body server control`: ```cf3 body server control { [...] any:: allowusers => { "cf-runner" }; } ``` Replace the `any::` class guard with one describing the hosts where you want to allow `cf-serverd` to accept remote `cf-runagent` requests. ## Update policy and restart server ``` root@ip-172-31-2-67:~# pkill -f cf-serverd root@ip-172-31-2-67:~# cf-agent -Kf update.cf && cf-agent -K root@ip-172-31-2-67:~# systemctl start cf-serverd root@ip-172-31-2-67:~# systemctl restart cfengine3 root@ip-172-31-2-67:~# cf-agent -Kf update.cf && cf-agent -K ``` ## Voilà ``` root@ip-172-31-2-67:~# sudo -u cf-runner bash cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 172.31.2.67> cf-serverd executing cfruncommand: /bin/sh -c ' "/var/cfengine/bin/cf-agent" -I -D cf_runagent_initiated -f /var/cfengine/inputs/update.cf ; "/var/cfengine/bin/cf-agent" -I -D cf_runagent_initiated 172.31.2.67> -> info: Executing 'no timeout' ... '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks clearLastSeenHostsLogs' 172.31.2.67> -> info: Completed execution of '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks clearLastSeenHostsLogs' 172.31.2.67> -> info: Executing 'no timeout' ... '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks process_api_events' 172.31.2.67> -> info: Command related to promiser '/var/cfengine/httpd/php/bin/php' returned code defined as promise kept 0 172.31.2.67> -> info: Completed execution of '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks process_api_events' 172.31.2.67> -> info: Executing 'no timeout' ... '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks materialized_hosts_view_refresh' 172.31.2.67> -> info: Completed execution of '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks materialized_hosts_view_refresh' 172.31.2.67> -> info: Executing 'no timeout' ... '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks inventory_refresh' 172.31.2.67> -> info: Command related to promiser '/var/cfengine/httpd/php/bin/php' returned code defined as promise kept 0 172.31.2.67> -> info: Completed execution of '/var/cfengine/httpd/php/bin/php /var/cfengine/httpd/htdocs/index.php cli_tasks inventory_refresh' cf-runner@ip-172-31-2-67:~$ cf-runagent -H 172.31.2.67 172.31.2.67> cf-serverd executing cfruncommand: /bin/sh -c ' "/var/cfengine/bin/cf-agent" -I -D cf_runagent_initiated -f /var/cfengine/inputs/update.cf ; "/var/cfengine/bin/cf-agent" -I -D cf_runagent_initiated cf-runner@ip-172-31-2-67:~$ ``` Locks are respected, so some runs have more output than others. ## Summary In short these things are needed: - Bootstrapped CFEngine installation working and running `cf-serverd` (Preferably 3.12.1+). - A new user to run `cf-runagent` - Created using `adduser`. - A new key pair for the `cf-runagent` user - Created by running `cf-key` as that user. - There must be a policy entry point with correct permissions at `~/.cfagent/inputs/promises.cf`. (It can be an empty file). - The server must trust the `cf-runagent` client key - must be present in `/var/cfengine/ppkeys/` with correct name. - The `cf-runagent` client must trust the server key - must be present in `~/.cfagent/ppkeys/` with correct name. - The `cf-runagent` user must be added to `allowusers` in the server policy - either directly or through `def.json` (3.12.1+). In some cases/versions, you will have to run update policy and/or restart the `cfengine3` service for changes to take effect. Tested on CFEngine Enterprise Hub, version 3.12.1, 3.10.5, and 3.7.8. -------------------------------------------------------------------------------- title: "Context-specific Security Settings" url: /blog/2019/context-specific-security-settings/index.md date: "2019-02-12" -------------------------------------------------------------------------------- CFEngine is very simple to set up and use, especially if all of the clients and the hub are going to be using the same promises. But what if there are certain things you want to enforce on a hub and not a client? What if there are certain things you want to enforce on a client but not on a hub? For example, if you are using the [Git Setup](https://docs.cfengine.com/docs/3.12/guide-installation-and-configuration-version-control.html), you want the hub to pull from the Git repository, but you don't want the clients to do this. You want the hub to make those changes available to the clients only after it's verified them. So how do you have a promise that only enforces on the hub, and not on a client? A simple solution is to use the `am_policy_hub` class to conditionally pull from Git if the server is a hub: ```cfengine3 # Very over-simplified example of a git pull for a policy hub only bundle agent git_update_masterfiles { vars: "git_branch" string => "branch_name"; commands: am_policy_hub:: "/usr/bin/git pull origin $(git_branch)"; } ``` This `am_policy_hub` class is great if you only plan to have a couple of promises that are different between the hub and the client. It becomes more tedious if you have many promises that differ between the hub and the client. You'd have to remember to conditionally check each promise to make sure a promise wasn't inadvertently applied to the wrong environment. Now, imagine you work for an organization that has thousands of servers. Imagine you are tasked to enforce hundreds of security policies on these servers. The problem is, the hubs need to enforce different security policies than the clients. Some clients should enforce all policies, while other clients (like the ones already deployed in production) should have the option to select which policies are enforced and which are only checked. Oh, and in case you can think of some sophisticated ways this might be possible with some of the new features of CFEngine Community 3.12, this solution has to be backward compatible with CFEngine Community 3.10.4. This is the scenario I found myself in while working for a client. As you can imagine, a simple conditional class like `am_policy_hub` is not enough for this kind of challenge. Creating additional classes that need to be checked in each promise would also not be very practical. I needed something a little more robust and flexible that wouldn't make each promise look so cluttered with conditionals. Below, I describe the approach I used. ## Contexts First, we implement the concept of contexts. We define categories (i.e. contexts) for the servers that need special handling, create directories for each of those contexts, then place promises inside those directories that only apply to that context. This also allows us to have context-specific settings, which we will see below. Then we just need to instruct CFEngine to only run the policies that apply to a particular context. Here are the contexts I needed for my client: `context_clients`, `context_clients_enforced`, `context_hubs`, `context_shared`, and `context_unknown`. The `context_shared` context contains all promises that apply to every context. This promotes the DRY (don't repeat yourself) principle. The security policies are in this context. They will be run in every context. However, every context contains settings to determine how the security policies will work, as described below. The `context_clients` context is the default context for production clients. It contains some promises that only apply to clients, and also contains a setting to indicate which security policies to enforce and which to only check. Initially, it doesn't enforce any security policies, but only checks them. The `context_clients_enforced` context is for a separate set of clients that are not yet in production. Similar to the context_clients context, it contains some client-specific promises as well as the setting to indicate which security policies will be enforced. This context is set to enforce all by default. The `context_hubs` context contains hub-specific promises, such as pulling the most recent masterfiles from Git. It also contains the setting to determine which security policies to enforce. Because these are also production servers, this context also doesn't enforce any security policies initially, but only checks them. The `context_unknown` context is a "catch-all" when the context cannot be determined. Nothing happens here. ## The Masterfiles Structure One of the goals of this implementation is to not modify any of the CFEngine policy files that ship with cfengine-community. This will make upgrades easier. In order to do that, we take advantage of [the augments file](https://docs.cfengine.com/docs/3.12/reference-language-concepts-augments.html) to add a custom policy file called `promises_company_name.cf` to inputs that will be run at the end of the bundlesequence as defined in body common control of `promises.cf`. This custom policy file will determine the context and then instruct CFEngine to run whatever additional promises we want for that context. Here is a list of all of the files required for a basic example of this solution: All of these files will be described below. They have also been made [available on GitHub](https://github.com/eli-taft-quoin/cfengine-context-specific). ## Step-by-step Instructions #### The Augments file: Create the `def.json` file in the root of the masterfiles directory, and insert these contents: ```json {file="/var/cfengine/masterfiles/def.json"} { "inputs": ["promises_company_name.cf"], "vars": { "control_common_bundlesequence_end": ["company_name_main"] } } ``` #### The Custom Promises File: Create the promises_company_name.cf file in the root of the masterfiles directory, and insert these contents: ```cfengine3 {file="/var/cfengine/masterfiles/promises_company_name.cf"} body file control { # Load whatever inputs are determined by the contexts bundle below inputs => { @(contexts.inputs), @(context_shared.inputs), }; } bundle common contexts { classes: # CFEngine creates classes for the hostname and IP address of # each server. This aproach assumes that you have a way of # separating client servers into various contexts based on one # of those classes. "context_hubs" expression => classmatch("am_policy_hub"); "context_clients" expression => classmatch("^reg_expression_match$"); "context_clients_enforce" expression => classmatch("^reg_expression_match$"); vars: any:: # Default is unknown, but overridden below "active" string => "unknown", policy => "overridable"; "inputs" slist => { "context_unknown/context_unknown.cf" }; "bundles" slist => { "context_unknown" }; context_hubs:: "active" string => "hubs", policy => "overridable"; "inputs" slist => { "context_hubs/context_hubs.cf" }; "bundles" slist => { "context_hubs" }; context_clients:: "active" string => "clients", policy => "overridable"; "inputs" slist => { "context_clients/context_clients.cf" }; "bundles" slist => { "context_clients" }; context_clients_enforce:: "active" string => "clients_enforce", policy => "overridable"; "inputs" slist => { "context_clients_enforce/context_clients_enforce.cf" }; "bundles" slist => { "context_clients_enforce" }; } bundle common context_shared { vars: any:: "inputs" slist => { "context_shared/context_shared.cf" }; "bundles" slist => { "context_shared"}; # Here we get the "check_only" array as it is defined for the # specific context we are in. This will be used in the shared # security policies to determine whether to check only or enforce. context_hubs:: "check_only" slist => { @(context_hubs.check_only) }; context_clients:: "check_only" slist => { @(context_clients.check_only) }; context_clients_enforce:: "check_only" slist => { @(context_clients_enforce.check_only) }; } bundle agent company_name_main { methods: # Now that we have determined the context and the context-specific # security policy settings, run the shared promises and then the # context-specific promises. Remember, the shared promises contain # the security policies. # Promises shared by all contexts "$(context_shared.bundles)" usebundle => $(context_shared.bundles); # Context-specific promises "$(contexts.bundles)" usebundle => $(contexts.bundles); } ``` #### The Additional Files: Now we will create all of the additional files needed to support this structure. Each file is listed below with the path relative to the masterfiles root directory. Create these files and insert the corresponding contents into each of them. ```cfengine3 {file="context_clients/context_clients.cf"} body file control { # Defines the input files to load for this particular context inputs => { "context_$(contexts.active)/sample_promises.cf", }; } bundle common context_clients { vars: any:: # This array is loaded by the context_shared promise in the # promises_company_name.cf file if this is the context to be used. # Potentially hundreds more of these; this is just a sample "check_only" slist => { "cis_01_01_01_01", "cis_01_01_01_02" }; } bundle agent context_clients { # This is where we tell CFEngine which bundles to run for this # context methods: "any" usebundle => sample_bundle; } ``` ```cfengine3 {file="context_clients/sample_promises.cf"} bundle agent sample_bundle { reports: any:: "This is a sample bundle that only runs in the context_clients context."; } ``` ```cfengine3 {file="context_clients_enforce/context_clients_enforce.cf"} body file control { # Defines the input files to load for this particular context inputs => { "context_$(contexts.active)/sample_promises.cf", }; } bundle common context_clients_enforce { vars: any:: # This array is loaded by the context_shared promise in the # promises_company_name.cf file if this is the context to be used. # None are set to check_only, meaning ALL will be enforced. "check_only" slist => { }; } bundle agent context_clients_enforce { # This is where we tell CFEngine which bundles to run for this # context methods: "any" usebundle => sample_bundle; } ``` ```cfengine3 {file="context_clients_enforce/sample_promises.cf"} bundle agent sample_bundle { reports: any:: "This is a sample bundle that only runs in the context_clients_enforce context."; } ``` ```cfengine3 {file="context_hubs/context_hubs.cf"} body file control { # Defines the input files to load for this particular context inputs => { "context_$(contexts.active)/sample_promises.cf", }; } bundle common context_hubs { vars: any:: # This array is loaded by the context_shared promise in the # promises_company_name.cf file if this is the context to be used. # Potentially hundreds more of these; this is just a sample "check_only" slist => { "cis_01_01_01_01", "cis_01_01_01_02" }; } bundle agent context_hubs { # This is where we tell CFEngine which bundles to run for this # context methods: "any" usebundle => git_update_masterfiles; } ``` ```cfengine3 {file="context_hubs/sample_promises.cf"} bundle agent git_update_masterfiles { reports: any:: "This is a sample promise that only runs in the context_hubs context."; } ``` ```cfengine3 {file="context_shared/context_shared.cf"} body file control { # Defines the input files to load for this particular context inputs => { # Potentially hundreds more of these; this is just a sample "context_shared/security_promises.cf", }; } bundle agent context_shared { # This is where we tell CFEngine which bundles to run for this # context methods: # Potentially hundreds more of these; this is just a sample "any" usebundle => cis_01_01_01_01; "any" usebundle => cis_01_01_01_02; } ``` ```cfengine3 {file="context_shared/security_promises.cf"} bundle agent cis_01_01_01_01 { classes: # This variable will be true if this bundle name is found in the # check_only array. "check_only" expression => strcmp("$(context_shared.check_only)", "$(this.bundle)"); reports: check_only:: "This promise only checks CIS security policy 1.1.1.1"; !check_only:: "This promise enforces CIS security policy 1.1.1.1"; } bundle agent cis_01_01_01_02 { classes: # This variable will be true if this bundle name is found in the # check_only array. "check_only" expression => strcmp("$(context_shared.check_only)", "$(this.bundle)"); reports: check_only:: "This promise only checks CIS security policy 1.1.1.2"; !check_only:: "This promise enforces CIS security policy 1.1.1.2"; } ``` ```cfengine3 {file="context_unknown/context_unknown.cf"} bundle agent context_unknown { # This is where we tell CFEngine which bundles to run for this # context reports: any:: "Could not determine the context. No promises to run."; } ``` Eli Taft Senior Solutions Architect Quoin, Inc. [www.quoininc.com](https://www.quoininc.com/) [![Quoin Logo](images/quoin.jpg)](https://www.quoininc.com/) -------------------------------------------------------------------------------- title: "From black box magic to automation transparency" url: /blog/2019/from-black-box-magic-to-automation-transparency/index.md date: "2019-01-17" -------------------------------------------------------------------------------- The CFEngine policy analyzer is an awesome new service introduced in CFEngine 3.13. The policy analyzer allows you to quickly debug policies and inspect what is going on under the hood of CFEngine. A known challenge with CFEngine, and most DSL based automation tools, relates to understanding what is actually going on during live operations. Many users view it as "black-box magic". Unfortunately, the amount of magic and the size of the black box increases with the level of automation. This is undesirable. Enter the policy analyzer. ## The Policy Analyzer The policy analyzer allows you to easily debug out-of-compliance settings (promise not-kept) and quickly navigate to the relevant places in your CFEngine policies so you can fix it. If you want to understand why there are thousands of "promises repaired" states, where CFEngine continuously fixes out-of-compliance events, you can use the policy analyzer to inspect the relevant policies and get a better understanding on what is really going on. The CFEngine policy analyzer comes in the form of a file browser that allows you to easily browse and filter all of your CFEngine policies. You can now, with a few clicks get to the exact line in a policy that is causing an out-of-compliance event. Thanks to the intuitive filtering capabilities, it is extremely intuitive to use. ## Apply the broken windows theory An inherent beauty of automation is to watch machines do a massive amount of work. CFEngine is often used at large-scale. Highly automated IT-infrastructures enjoy seeing tens and hundreds of thousands of rules being automatically enforced and, if needed, fixed by CFEngine. However, if the amount of out-of-compliance incidents grows too big, fatigue enters the scene. At this stage one typically starts to prioritize, and accept a certain level of out-of-compliance states. This is of course highly undesirable. The best-of-the-best IT-operations apply the [broken windows theory](https://en.wikipedia.org/wiki/Broken_windows_theory), and do not accept any degradation. With the policy analyzer we hope that many more CFEngine users can apply the same principle and end up with an IT-infrastructure where the part under CFEngine management stays at **100% compliance**. This is absolutely feasible, and we are confident that the policy analyzer will help you greatly on this journey. The carrot waiting for you will be a life of proactive and creative work, as opposed to reactive and boring. ## Check out the demo We have made an introductory video to the policy analyzer below. Please check it out. Remember, the first 25 hosts are always free with CFEngine\! Try it out now. [Download](https://cfengine.com/product/cfengine-enterprise-free-25/). {{< youtube gGqZVN5NC1E >}} -------------------------------------------------------------------------------- title: "Hacking custom variables for additional augments in CFEngine" url: /blog/2018/hacking-custom-variables-for-additional-augments-in-cfengine/index.md date: "2018-12-17" --------------------------------------------------------------------------------
This post was syndicated with permission from the [original source](https://cmdln.org/2018/11/13/hacking-custom-variables-for-additional-augments-in-cfengine/). CFEngine 3.12.0 introduced the [*augments* key](https://docs.cfengine.com/docs/3.12/reference-language-concepts-augments.html#augments) to the [Augments file format](https://docs.cfengine.com/docs/3.12/reference-language-concepts-augments.html#augments). If you are not already familiar with Augments, check it out. It's a very easy way to define classes and variables very early during agent execution, before policy. The new augments key allows you to merge additional data in the augments format on top of the base augments. However, there is, I think, still a simple way to accomplish this. This can provide a flexible way of providing different data to different sets of machines. Since augments are loaded very early during policy evaluation. Currently, only [special variables](https://docs.cfengine.com/docs/3.12/reference-special-variables.html) like the [`sys` vars](https://docs.cfengine.com/docs/3.12/reference-special-variables-sys.html) can be used to select additional augments. The question arises … > How can I extend augments from custom policy? The short answer is that you can't, at least not directly, but there are a couple of things that you can do to achieve the same result.
## Customize /etc/os-release
In 3.11.0 CFEngine began parsing `/etc/os-release` during system discovery into [sys.os\_release](https://docs.cfengine.com/docs/master/reference-special-variables-sys.html#sys-os_release). I think that `/etc/os-release` is perhaps a good place for you to encode this `DEVICE_TYPE`. It's a simple `key=value` format, and according to the docs on freedesktop.org [the file format may be extended by vendors](https://www.freedesktop.org/software/systemd/man/os-release.html). It's highly recommended to prefix new fields with an OS specific name in order to avoid name clashes. So, let's edit `/etc/os-release` and define `CUSTOM_DEVICE_TYPE`. This is `/etc/os-release` on my workstation, after I have edited it to add my new key value. ```bash NAME="Ubuntu" VERSION="18.10 (Cosmic Cuttlefish)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.10" VERSION_ID="18.10" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=cosmic UBUNTU_CODENAME=cosmic CUSTOM_DEVICE_TYPE=laptop ``` This way, when the agent initializes, it will have `sys.os_release[CUSTOM_DEVICE_TYPE]` available for use. This could be set during initial provisioning, or of course CFEngine policy could manage the extra file content. Here is a small example: **def.json:**
```json { "vars": { "my_var": "I am RED" }, "augments": [ "$(sys.os_release[CUSTOM_DEVICE_TYPE]).json" ] } ```
Here is my data file that is used when `CUSTOM_DEVICE_TYPE` is set to `laptop`. **laptop.json:**
```json { "vars": { "my_var": "I am BLUE" } } ```
Here is the simple policy to illustrate the point. **promises.cf:**
```cf3 bundle agent main { reports: "def.my_var is $(def.my_var)"; "My device type is $(sys.os_release[CUSTOM_DEVICE_TYPE])"; } ```
**Output:** ``` example > $ cf-agent -KIf ./promises.cf R: def.my_var is I am BLUE R: My device type is laptop ``` In the above output, note that `def.my_var` is `BLUE`, what happens if you change the `CUSTOM_DEVICE_TYPE`?
## Download device type specific data to a stable location
An alternate methodology to achieve similar results would be to reference a stable location, but use a files promise to ensure that the proper device type specific data is there. For example, instead of shipping `device_type1.json` and `devices_type2.json` as part of the policy, you can download it to `device.json`. This copy\_from files promise illustrates what I mean:
```cf3 bundle agent device_type_data { files: "/var/cfengine/inputs/device.json" copy_from => remote_dcp( "/var/cfengine/data/devices/$(my_common.device_type).json", $(sys.policy_hub) ); } ```
Then the base augments would look like this: **def.json:**
```json { "vars": { "my_var": "I am RED" }, "augments": [ "device.json" ] } ```
-------------------------------------------------------------------------------- title: "CFEngine 3.12.1 LTS Released" url: /blog/2018/cfengine-3-12-1-lts-released/index.md date: "2018-11-30" -------------------------------------------------------------------------------- CFEngine 3.12.1 LTS has now been released. This release brings many stability and performance improvements to the 3.12 LTS series. It is a stable and well-tested version of CFEngine. We wish to extend a big thanks to the ecosystem that helps make CFEngine great by reporting bugs, contributing fixes and suggesting new and improved functionality. Without you, CFEngine would not be the powerful, high performance, widely used product we all appreciate today\! We hope and think this release meets the high standards we know all our users have. That is why you chose CFEngine in the first place\! This is a good time to start thinking about updating to 3.12, as this is the best and most long-term solution available. You can read more about our supported versions [here](https://cfengine.com/product/supported-versions/), but in short, we can highlight that: - CFEngine 3.10 will be supported until December 2019. - CFEngine 3.12 will be supported until June 2021. # Improvements CFEngine 3.12.1 has received all of the bug fixes and improvements that have been applied to CFEngine since 3.12.0 was released, and is as such a great release, with a lot of improvements. We highlight a few changes here, but you can find the full list of changes in the ChangeLog entries linked below. ## Changes in CFEngine Core In CFEngine 3.12.1 we have added support for Ubuntu 18.04. It is nice to see old and new software working together as when we are using Ubuntu 18 with Windows server 2008, and the support for more than decades-old infrastructure is one of the benefits of such a wide platform support as what CFEngine provides. You can see all the supported platforms for 3.12 [here](https://docs.cfengine.com/docs/3.12/guide-latest-release-supported-platforms.html). We have added a new binary, `cf-check` will be used to diagnose and remediate problems with corrupt LMDB databases. As we have seen several issues with LMDB corruption lately, this tool will make those issues a lot simpler to debug and mitigate. A community user made us aware of a [memory leak](https://northerntech.atlassian.net/browse/CFE-2913) in CFEngine 3.12.0, that we have now fixed in 3.12.1. To make sure this does not happen again, we have introduced several new Valgrind and AddressSanitizer (ASAN) tests as part of our CI process. When we introduced this, we also found and fixed several other memory issues. On Windows, we have also started using a custom cleanup function instead of `atexit` due to a race condition when unloading DLLs on Windows. ## Changes in CFEngine Enterprise In CFEngine Enterprise, we have done a lot of quality and stability work to make it the best possible enterprise tool. Several of the major changes we introduced in 3.12.0 have been thoroughly tested and further stabilized. We have resolved an issue that caused problems with our [High Availability](https://docs.cfengine.com/docs/3.12/enterprise-cfengine-guide-high-availability.html) functionality, so this is now working very well. We have also updated the HA documentation, to make the steps for enabling and setting this up a bit simpler to follow. This is a very useful feature for any Enterprise-grade system, and we would be happy to talk more about how it can benefit you. When it comes to Mission Portal, we have spent a lot of time improving the reporting interfaces and making them more efficient. We have solved an error where sometimes a scheduled report was sent as a PDF instead of a CSV. We have also made it simpler to navigate from the Inventory view to the relevant host info page. We also solved an issue that caused some custom notification scripts not to fire, fixed a bug that caused alerts not to show up in the Mission Portal Events Log on some platforms and made it possible for multiple users to create independent reports with the same name, something that was causing issues previously. You can see a list of the issues fixed in our public Jira for 3.12.1 [here](https://northerntech.atlassian.net/issues/?jql=project%20%3D%20CFE%20AND%20fixVersion%20%3D%203.12.1). And for more details, please see the relevant ChangeLog entries: - CFEngine 3.12.1 [community](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html) - CFEngine 3.12.1 [enterprise](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-enterprise.html) - CFEngine 3.12.1 [masterfiles](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ## Quality We wrote extensively about our quality efforts in last week's [3.13 release blog post](https://cfengine.com/company/blog-detail/cfengine-3-13-0-released/), but it bears repeating here. CFEngine is an enterprise product, and we have always been proud of our code quality, both with regards to the languages and patterns we use, the development team, and the end product we provide for our customers at the end of a development cycle. We have a lot of tests covering the CFEngine source code, and we think the time has come to share some of the results that we see, that make us confident in the quality of CFEngine. We are using both Jenkins and Travis to run automated tests on all of our builds, to ensure that the quality is always high. And as we are adding new features or improving existing code, we are always adding new test cases to make sure we keep a close eye on the performance of our product. For 3.12 we are running a base set of more than 2 000 tests on each build, resulting in 23 000 tests across all the platform combinations we support. At the time of releasing 3.12.1, all the tests we run are passing, giving us a good indication of our quality. We are fully aware that automated tests are not the only way to ensure quality, so we also spend a lot of time using our own software, both on a day to day basis, as well as executing a rigorous manual testing scheme before the release of each version of CFEngine. We will continue to improve our test coverage and code quality and keep the community up to date with the quality metrics we are introducing. ## Dependencies CFEngine is not depending on many external libraries, but we do depend on a few. With 3.12.1 we have updated a few dependencies from what we used in 3.12.0 in order to get the latest security, performance and reliability improvements. | | | | | ---------- | ---------- | ---------- | | | **3.12.0** | **3.12.1** | | libacl | 2.2.52 | 2.2.53 | | libattr | 2.4.47 | 2.4.48 | | libcurl | 7.59.0 | 7.62.0 | | php | 7.2.6 | 7.2.12 | | postgresql | 10.4 | 10.6 | | git | 2.17.1 | 2.19.1 | | apache | 2.4.33 | 2.4.35 | | apr | 1.6.3 | 1.6.5 | | libyaml | 0.1.7 | 0.2.1 | | openssl | 1.1.0h | 1.1.0i | ## Upgrading? If you are upgrading an existing CFEngine Enterprise installation, you can check out the upgrade documentation for [3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are of course always happy to assist customers with upgrading\! If you are a community user, you can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure to the Enterprise version of CFEngine. We are here to help you get more out of CFEngine\! # Get CFEngine 3.12.1 LTS now\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [3.12](https://cfengine.com/docs/3.12/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy this new release, and we are looking forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! # Contribute to CFEngine We are always interested in new contributors, and in growing the community around CFEngine. We encourage you to contribute and participate in the fun and interesting development of the CFEngine [ecosystem](https://cfengine.com/engage/)\! Do you want to start contributing but are unsure how? - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). - Or you can [suggest your improvements](https://northerntech.atlassian.net/secure/CreateIssue!default.jspa) or report bugs that you find\! No matter how you do it, we hope you get the most out of this new version of CFEngine\! -------------------------------------------------------------------------------- title: "CFEngine 3.13.0 released" url: /blog/2018/cfengine-3-13-0-released/index.md date: "2018-11-23" -------------------------------------------------------------------------------- Today we are very happy to announce the release of CFEngine 3.13.0. This is a non-LTS release, introducing new features and functionality. There is a lot happening with CFEngine these days\! This release is closely following last weeks release of [CFEngine 3.10.5 LTS](https://cfengine.com/company/blog-detail/cfengine-lts-3-10-5-released/), and soon we will also release the next patch version of our 3.12 LTS series. So keep following our updates\! ## Contribute to CFEngine Did you know that CFEngine is a dual license open source project? And not only that, we are encouraging community contributions, and are always looking for ways to improve and grow our [ecosystem](https://cfengine.com/engage/). We encourage you to contribute and participate in the fun development of CFEngine\! Do you want to start contributing but are unsure how? - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). # New Features CFEngine 3.13.0 has received all of the bug fixes and improvements that have been applied to CFEngine since 3.12.0 was released, and is as such a great release, with a lot of improvements. In addition, there are some new features we are very excited about. ## Policy Analyzer Perhaps the most prominent feature we have introduced in CFEngine Enterprise 3.13 is the new Policy Analyzer. This allows you to visually inspect your policy, and understand what parts of it are kept, not kept or repaired - and at what hosts this is happening, directly from the Mission Portal. This new feature enables you to drill down into policy and see inline in the code viewer embedded in Mission Portal how the policy works on each host and what the outcome is. You can filter on promises status, specific hosts or classes to get insight into all policy evaluation there. The resulting promise outcomes can be sorted by promise outcome, promise type, hostname or other properties. It will also tell you if the version of policy executing on the host is up to date with the latest version from the host. In order to enable the Policy Analyzer, you will need to consent to the copying of Masterfiles to a location readable by Mission Portal. Out of the box, Mission Portal does not have this ability and we have decided not to change the default behavior. For this reason, the Policy Analyzer tool is not enabled by default, so please go ahead and enable it - go to Mission Portal, and to the Policy Analyzer tab, and follow the instructions. But don't worry, you are just one click away\! ![A new visual Policy Language inspection tool for use in Mission Portal](images/screencapture-188-166-113-224-index-php-policy-analyzer-2018-11-19-20_50_12-756x800.png) We are very interested in more feedback on this new feature and hope that many people would like to use it. We have a long list of features and value we will add to this tool in the next few versions of CFEngine, but are excited to hear more about what you want to use it for and what can make it better for you. Here is a small screencast of how the Policy Analyser can show you the policy that does not work as expected. The correct package name on Ubuntu should not be `httpd` but `apache2`. {{< youtube gGqZVN5NC1E >}} ## Changes in CFEngine Core In 3.13.0 we have introduced a number of interesting new features and fixes to **CFEngine Core**. All of the bug fixes and improvements that have been fixed for [3.10.5 LTS](https://cfengine.com/company/blog-detail/cfengine-lts-3-10-5-released/) is also included in 3.13. In August, OpenSSL released version 1.1.1 and we have upgraded to this in 3.13. This brings the new TLS 1.3 that provides significant security and performance benefits. It will be used by default whenever both hub and agent are running 3.13. We have introduced a new inventory policy that will gather metadata from Amazon EC2 instances, improving the user experience and functionality in that cloud environment. This is a part of our ongoing push to improve the CFEngine cloud experience. We have introduced a new class for forcing the policy to be requested from the network. Running update policy with `mpf_skip_local_copy_optimization` results in policy update connecting to cf-serverd instead of using a local copy. This can be valuable in both testing and real-life examples. Also worth mentioning is that we have added support for Ubuntu 18.04 to CFEngine 3.13.0. It is nice to see old and new software working together as when we are using Ubuntu 18 with Windows server 2008, and the support for more than decades-old infrastructure is one of the benefits of such a wide platform support as what CFEngine provides. You can see all the supported platforms for 3.13 [here](https://docs.cfengine.com/docs/3.13/guide-latest-release-supported-platforms.html). ## New in CFEngine Enterprise In **CFEngine Enterprise**, we have done a lot of work\! A lot of it is internal, such as improving our multi-threaded code for performance and stability. As mentioned above, we have the new Policy Analyzer. We have also implemented a new way to get your own documentation displayed in Mission Portal so that you can document e.g. your own policy, and make that documentation available in a simple and RBAC-protected way to the users of your system. This has also been backported to 3.10.5 and the upcoming version of 3.12. In Mission Portal, we have made several improvements. We have resolved an issue that prevented some users from exporting reports as CSV, we have fixed a bug that caused some custom notification scripts not to fire, fixed a bug that caused alerts not to show up in the Mission Portal Events Log on some platforms, and made it possible for multiple users to create independent reports with the same name, something that was causing issues previously. In 3.13 we have changed the old Design Center API for version control, into a new VCS API (Version Control System) that makes it easier and more intuitive to maintain your policy in a Git repository, and check it out for use in CFEngine. You can read more about this API in our documentation [here](https://docs.cfengine.com/docs/3.13/reference-enterprise-api-ref-vcs-settings.html). You can see a list of the issues fixed in our public Jira for 3.13.0 [here](https://northerntech.atlassian.net/issues/?jql=project%20%3D%20CFE%20AND%20fixVersion%20%3D%203.13.0). And for more details, please see the relevant ChangeLog entries: - CFEngine 3.13.0 [community](https://docs.cfengine.com/docs/3.13/guide-latest-release-whatsnew-changelog-core.html) - CFEngine 3.13.0 [enterprise](https://docs.cfengine.com/docs/3.13/guide-latest-release-whatsnew-changelog-enterprise.html) - CFEngine 3.13.0 [masterfiles](https://docs.cfengine.com/docs/3.13/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ## Quality CFEngine is an enterprise product, and we have always been proud of our code quality, both with regards to the languages and patterns we use, the development team, and the end product we provide for our customers at the end of a development cycle. We have a lot of tests covering the CFEngine source code, and we think the time has come to share some of the results that we see, that make us confident in the quality of CFEngine. We are using both Jenkins and Travis to run automated tests on all of our builds, to ensure that the quality is always high. And as we are adding new features or improving existing code, we are always adding new test cases to make sure we keep a close eye on the performance of our product. For 3.13 we are running a base set of more than **2 000** tests on each build, resulting in **23 000** tests across all the platform combinations we support. At the time of releasing 3.13, all the tests we run are passing, giving us a good indication of our quality. We are fully aware that automated tests are not the only way to ensure quality, so we also spend a lot of time using our own software, both on a day to day basis, as well as executing a rigorous manual testing scheme before the release of each version of CFEngine. We will continue to improve our test coverage and code quality and keep the community up to date with the quality metrics we are introducing. For 3.13 we have also enabled a tool called [LGTM](https://lgtm.com/projects/g/cfengine/core/overview/) to track several quality metrics of our source code. We have spent quite a bit of time resolving all the issues reported and have significantly improved with this tool. For now, we have only subjected CFEngine Core to this tool, but we are looking for ways to add other parts of CFEngine as well. ![](images/LGTM-800x507.png) One view of how LGTM is scoring the quality of CFEngine Core. Note that the large project to the right is the Linux kernel. ## Dependencies CFEngine is not depending on many external libraries, but we do depend on a few. With 3.13 we have updated some dependencies from what we used in 3.12 in order to get the latest security, performance and reliability improvements. | | | | | ---------- | ---------- | ---------- | | | **3.12.0** | **3.13.0** | | libacl | 2.2.52 | 2.2.53 | | libattr | 2.4.47 | 2.4.48 | | libcurl | 7.59.0 | 7.61.0 | | php | 7.2.6 | 7.2.11 | | postgresql | 10.4 | 11.0 | | git | 2.17.0 | 2.19.1 | | apache | 2.4.33 | 2.4.35 | | apr | 1.6.3 | 1.6.5 | | libyaml | 0.1.7 | 0.2.1 | | openssl | 1.0.0h | 1.1.1 | ## Upgrading? If you are upgrading an existing CFEngine Enterprise installation, you can check out the upgrade documentation for [3.13](https://cfengine.com/docs/3.13/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are of course always happy to assist customers with upgrading\! If you are a community user, you can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure to the Enterprise version of CFEngine. We are here to help you get more out of CFEngine\! # Get CFEngine 3.13 now\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [3.13](https://cfengine.com/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy this new release, and we are looking forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Three high level patterns in CFEngine" url: /blog/2018/three-high-level-patterns-in-cfengine-3/index.md date: "2018-11-19" -------------------------------------------------------------------------------- This post was syndicated with permission from the [original source](https://cmdln.org/2018/11/10/looking-at-three-high-level-patterns-in-cfengine-3/). How do you deal with config files that need different settings based on various services that are running on a host and cooperate with other teams? It's a common question, and it came up in \#cfengine on libera.chat recently. > The issue is that team A might be working on package A, which requires > some environment variables set. But team B might be working on a > totally different thing -- and want to achieve the same thing. I hoped > to give them a bit of 'library' code to take care of it, rather than > have them touch a centralized environment-setting policy file. Here, I look at three high level patterns I have seen. - Competitive management Each bundle that needs a specific config manages it by itself, typically by using a parameterized bundle. - Centralized management All config definition is in a single place, anyone needing something edits the global config. - Cooperative management Each bundle that needs a specific config, publishes the config it needs, and another policy handles consolidating the configs and edits the config as necessary. Let's take a look at an example of each of these patterns. I will start from an example recently shared in \#cfengine on libera.chat that manages =/etc/environment, `/etc/sysstemd.conf`, and `/etc/profile`. I'll first fix it, so that it works, then show several different patterns that achieve the same goal. ## The original example > Hello everyone. I am trying to use set\_line\_based to set some > "global" environment variables. I've created a bundle that I thought > might make it easier. However, using this bundle more than once only > applies the first 'invocation'. Code originally here (now expired): > https://pastebin.com/mbU1Bb6i **Example usage:** ```cf3 bundle agent vagrant_vm { meta: "tags" slist => { "autorun" }; methods: "unifi"; "any" usebundle => global_env("PAPERTRAIL_HOST", "xxx.papertrailapp.com"); "any" usebundle => global_env("PAPERTRAIL_PORT", "12345"); "any" usebundle => global_env("APPOPTICS_USER", "ops@whatever.com"); "any" usebundle => global_env("APPOPTICS_APIKEY", "cafebabedeadbeef"); "any" usebundle => global_env("CORE_JDBC_URL", "jdbc:postgresql://localhost/xxx"); "any" usebundle => global_env("CORE_JDBC_USER", "vagrant"); "any" usebundle => global_env("SMS_ENABLED", "false"); "any" usebundle => global_env("SMS_AWS_REGION", "eu-west-1"); } ``` **Implementation:** ```cf3 bundle agent global_env(name, value) { vars: "etcenv[$(name)]" string => "$(value)"; "systemconf[DefaultEnvironment=$(name)]" string => "$(value)"; "etcprofile[export $(name)]" string => "$(value)"; files: "/etc/environment" edit_line => set_line_based("global_env.etcenv", "=", "\s*=\s*", ".*", "\s*#\s*"); "/etc/systemd/system.conf" edit_line => set_line_based("global_env.systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), classes => if_repaired("system_conf_changed"); "/etc/profile" edit_line => set_line_based("global_env.etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"); commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"); } ``` The reason that only the first actuation of `global_env` alters the file is because the promise does not differ. Once a promise has been kept or repaired, it is not actuated again within the same agent run. The promise can be made unique by including the parameters which are different between actuation's in the promise. Here we add a handle using both parameters, so that the promise is unique across actuation's of the bundle for different values of `name` and `value`. ``` "/etc/environment" edit_line => set_line_based("global_env.etcenv", "=", "\s*=\s*", ".*", "\s*#\s*"), handle => "etc_environment_$(name)_$(value)"; ``` I will combine them together into the same policy file and make some minor changes. - I added handles using the parameters to make sure that the promises are unique across executions with different parameter values. For more on this, see the [language concepts for bundles](https://docs.cfengine.com/docs/3.12/reference-language-concepts-bundles.html "language concepts for bundles"), it discusses how bundles are not functions. - I added `bundle agent __main__` so that the `vagrant_vm` bundle will be actuated when this policy file is run directly. - I removed the `unifi` methods promise because it was not provided in the original example. - I added reports that show the complete content of the files for easy demonstration. - I include the stdlib for my executions, you don't see it, but know that it's in use. - I guard the `systemd daemon-reload` with a check that the executing user is root because I prototype policy as my own user, inside org-mode using [ob-cfengine3](https://github.com/nickanderson/ob-cfengine3 "ob-cfengine3") (shameless plug). - I add an init bundle to reset as if running from a clean state with no pre-existing configuration. - I added body contain `exec_owner()` because it isn't in the stdlib. - I added `create => "true"` to the files promises. - I prefixed the files promises with `/tmp`. ## Competitive management This is a common pattern. I think of this pattern as competitive because each service manages what it needs, multiple services may compete for control over the same resources like configuration files. Each service only worries about itself, this pattern allows for extra settings to exist in the configuration files which is beneficial in situations where control must be shared between multiple agents. This pattern results in files being edited multiple times within a single agent run which may be useful in ordering, but results in some overhead. Additionally, depending how the service restart on config change is managed, may lead to the same service re-starting multiple times in the same agent run. **Policy:** ```cf3 bundle agent global_env(name, value) { vars: "etcenv[$(name)]" string => "$(value)"; "systemconf[DefaultEnvironment=$(name)]" string => "$(value)"; "etcprofile[export $(name)]" string => "$(value)"; files: "/tmp/etc/environment" edit_line => set_line_based("global_env.etcenv", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_environment_$(name)_$(value)"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("global_env.systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf_$(name)_$(value)"; "/tmp/etc/profile" edit_line => set_line_based("global_env.etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_profile_$(name)_$(value)"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload_$(name)_$(value)"; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent vagrant_vm { meta: "tags" slist => { "autorun" }; methods: "any" usebundle => global_env("PAPERTRAIL_HOST", "xxx.papertrailapp.com"); "any" usebundle => global_env("PAPERTRAIL_PORT", "12345"); "any" usebundle => global_env("APPOPTICS_USER", "ops@whatever.com"); "any" usebundle => global_env("APPOPTICS_APIKEY", "cafebabedeadbeef"); "any" usebundle => global_env("CORE_JDBC_URL", "jdbc:postgresql://localhost/xxx"); "any" usebundle => global_env("CORE_JDBC_USER", "vagrant"); "any" usebundle => global_env("SMS_ENABLED", "false"); "any" usebundle => global_env("SMS_AWS_REGION", "eu-west-1"); } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { methods: "init"; "vagrant_vm"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: CFEngine 3.12.0 R: /tmp/etc/environment R: PAPERTRAIL_HOST=xxx.papertrailapp.com R: PAPERTRAIL_PORT=12345 R: APPOPTICS_USER=ops@whatever.com R: APPOPTICS_APIKEY=cafebabedeadbeef R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: CORE_JDBC_USER=vagrant R: SMS_ENABLED=false R: SMS_AWS_REGION=eu-west-1 R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.papertrailapp.com R: DefaultEnvironment=PAPERTRAIL_PORT=12345 R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=APPOPTICS_APIKEY=cafebabedeadbeef R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: /tmp/etc/profile R: export PAPERTRAIL_HOST=xxx.papertrailapp.com R: export PAPERTRAIL_PORT=12345 R: export APPOPTICS_USER=ops@whatever.com R: export APPOPTICS_APIKEY=cafebabedeadbeef R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export CORE_JDBC_USER=vagrant R: export SMS_ENABLED=false R: export SMS_AWS_REGION=eu-west-1 ``` A common evolution for this policy would be to condense the write operations by altering the parameter to pass a map of key value pairs. ### Condense the write operations by changing the parameters A quick change to the parameters allows multiple key values to be set at a single time, reducing IO. **Policy:** ```cf3 bundle agent global_env(map) { vars: # map is KEY=VAL "name" slist => getindices( map ); "etcenv[$(name)]" string => "$(map[$(name)])"; "systemconf[DefaultEnvironment=$(name)]" string => "$(map[$(name)])"; "etcprofile[export $(name)]" string => "$(map[$(name)])"; files: "/tmp/etc/environment" edit_line => set_line_based("global_env.etcenv", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_environment_$(name)_$(map[$(name)])"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("global_env.systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf_$(name)_$(map[$(name)])"; "/tmp/etc/profile" edit_line => set_line_based("global_env.etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_profile_$(name)_$(map[$(name)])"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload_$(name)_$(map[$(name)])"; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent vagrant_vm { meta: "tags" slist => { "autorun" }; vars: "d" data => '{ "PAPERTRAIL_HOST": "xxx.papertrailapp.com", "PAPERTRAIL_PORT": "12345", "APPOPTICS_USER": "ops@whatever.com", "APPOPTICS_APIKEY": "cafebabedeadbeef", "CORE_JDBC_URL": "jdbc:postgresql://localhost/xxx", "CORE_JDBC_USER": "vagrant", "SMS_ENABLED": "false", "SMS_AWS_REGION": "eu-west-1" }'; methods: "any" usebundle => global_env( @(d) ); } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { methods: "init"; "vagrant_vm"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: CFEngine 3.12.0 R: /tmp/etc/environment R: APPOPTICS_USER=ops@whatever.com R: PAPERTRAIL_PORT=12345 R: CORE_JDBC_USER=vagrant R: APPOPTICS_APIKEY=cafebabedeadbeef R: SMS_ENABLED=false R: PAPERTRAIL_HOST=xxx.papertrailapp.com R: SMS_AWS_REGION=eu-west-1 R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=APPOPTICS_APIKEY=cafebabedeadbeef R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.papertrailapp.com R: DefaultEnvironment=PAPERTRAIL_PORT=12345 R: /tmp/etc/profile R: export SMS_ENABLED=false R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export APPOPTICS_USER=ops@whatever.com R: export CORE_JDBC_USER=vagrant R: export SMS_AWS_REGION=eu-west-1 R: export PAPERTRAIL_PORT=12345 R: export APPOPTICS_APIKEY=cafebabedeadbeef R: export PAPERTRAIL_HOST=xxx.papertrailapp.com ``` ## Centralized management Let's take a look at what a centralized management pattern might look like. This is the simplest pattern, and many times how policies begin as a prototype, before evolving into another pattern. We removed the `vagrant_vm` bundle, and the uniqueness from the promise handles. Now, all the configuration data is right in the `global_env` bundle. We use a classic array as a generator to produce copies of the key values tailored for each config file. An alternative to creating different variables using a classic array generator we could copy `set_line_based` and customize it to allow a prefix to be specified. **Policy:** ```cf3 bundle agent global_env { vars: vagrant_vm:: "env[PAPERTRAIL_HOST]" string => "xxx.papertrailapp.com"; "env[PAPERTRAIL_PORT]" string => "12345"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "cafebabedeadbeef"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-west-1"; application_2:: "env[PAPERTRAIL_HOST]" string => "xxx.trailpaperapp.com"; "env[PAPERTRAIL_PORT]" string => "54321"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "xxxxiiiiiiixxixxix"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-east-2"; any:: "env_i" slist => getindices( env ); # Here we take the original simple key value pair and add different # prefixes for the differnt config file types using a classic array as a # generator. Alternatively, we could copy set_line_based and customize it # allowing for a prefix parameter. "systemconf[DefaultEnvironment=$(env_i)]" string => "$(env[$(env_i)])"; "etcprofile[export $(env_i)]" string => "$(env[$(env_i)])"; files: "/tmp/etc/environment" edit_line => set_line_based("global_env.env", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_environment"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("global_env.systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf"; "/tmp/etc/profile" edit_line => set_line_based("global_env.etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_profile"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload"; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { classes: "application_2" scope => "namespace"; methods: "init"; "global_env"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: CFEngine 3.12.0 R: /tmp/etc/environment R: SMS_AWS_REGION=eu-east-2 R: PAPERTRAIL_HOST=xxx.trailpaperapp.com R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: SMS_ENABLED=false R: APPOPTICS_APIKEY=xxxxiiiiiiixxixxix R: APPOPTICS_USER=ops@whatever.com R: PAPERTRAIL_PORT=54321 R: CORE_JDBC_USER=vagrant R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=SMS_AWS_REGION=eu-east-2 R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=APPOPTICS_APIKEY=xxxxiiiiiiixxixxix R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.trailpaperapp.com R: DefaultEnvironment=PAPERTRAIL_PORT=54321 R: /tmp/etc/profile R: export SMS_ENABLED=false R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export APPOPTICS_USER=ops@whatever.com R: export CORE_JDBC_USER=vagrant R: export SMS_AWS_REGION=eu-east-2 R: export PAPERTRAIL_PORT=54321 R: export APPOPTICS_APIKEY=xxxxiiiiiiixxixxix R: export PAPERTRAIL_HOST=xxx.trailpaperapp.com ``` A common evolution's on this policy include moving variable definition into its own bundle, separating the data into external files, and since all data is known at one time, full file management is an option. ### Centralized management separate data bundle Here I am sure to have the data bundle converge before the bundle that will use the data defined there. This separation enables more complex data convergence, delegation of control, and especially with more data can improve the readability of the policy. **Policy:** ```cf3 bundle agent global_env { vars: "env_i" slist => getindices( "app_data.env" ); # Here we take the original simple key value pair and add different prefixes # for the differnt config file types using a classic array as a generator. "systemconf[DefaultEnvironment=$(env_i)]" string => "$(app_data.env[$(env_i)])"; "etcprofile[export $(env_i)]" string => "$(app_data.env[$(env_i)])"; files: "/tmp/etc/environment" edit_line => set_line_based("app_data.env", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_environment"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("$(this.bundle).systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf"; "/tmp/etc/profile" edit_line => set_line_based("$(this.bundle).etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"), create => "true", handle => "_etc_profile"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload"; } bundle agent app_data { vars: vagrant_vm:: "env[PAPERTRAIL_HOST]" string => "xxx.papertrailapp.com"; "env[PAPERTRAIL_PORT]" string => "12345"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "cafebabedeadbeef"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-west-1"; application_2:: "env[PAPERTRAIL_HOST]" string => "xxx.trailpaperapp.com"; "env[PAPERTRAIL_PORT]" string => "54321"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "xxxxiiiiiiixxixxix"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-east-2"; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { classes: "vagrant_vm" expression => "any", scope => "namespace"; methods: "init"; "app_data"; "global_env"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: CFEngine 3.12.0 R: /tmp/etc/environment R: CORE_JDBC_USER=vagrant R: PAPERTRAIL_PORT=12345 R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: SMS_ENABLED=false R: SMS_AWS_REGION=eu-west-1 R: APPOPTICS_APIKEY=cafebabedeadbeef R: PAPERTRAIL_HOST=xxx.papertrailapp.com R: APPOPTICS_USER=ops@whatever.com R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=APPOPTICS_APIKEY=cafebabedeadbeef R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.papertrailapp.com R: DefaultEnvironment=PAPERTRAIL_PORT=12345 R: /tmp/etc/profile R: export SMS_ENABLED=false R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export APPOPTICS_USER=ops@whatever.com R: export CORE_JDBC_USER=vagrant R: export SMS_AWS_REGION=eu-west-1 R: export PAPERTRAIL_PORT=12345 R: export APPOPTICS_APIKEY=cafebabedeadbeef R: export PAPERTRAIL_HOST=xxx.papertrailapp.com ``` ### Centralized management separate data bundle with full file management Here we remove the init to reset the environment and add full file content management by attaching edit\_defaults empty to the files promises. This ensures no unknown content in the configuration file. **Policy:** ```cf3 bundle agent global_env { vars: "env_i" slist => getindices( "app_data.env" ); # Here we take the original simple key value pair and add different prefixes # for the differnt config file types using a classic array as a generator. "systemconf[DefaultEnvironment=$(env_i)]" string => "$(app_data.env[$(env_i)])"; "etcprofile[export $(env_i)]" string => "$(app_data.env[$(env_i)])"; files: "/tmp/etc/environment" edit_line => set_line_based("app_data.etcenv", "=", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, create => "true", handle => "_etc_environment"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("$(this.bundle).systemconf", "=", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf"; "/tmp/etc/profile" edit_line => set_line_based("$(this.bundle).etcprofile", "=", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, create => "true", handle => "_etc_profile"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload"; } bundle agent app_data { vars: vagrant_vm:: "env[PAPERTRAIL_HOST]" string => "xxx.papertrailapp.com"; "env[PAPERTRAIL_PORT]" string => "12345"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "cafebabedeadbeef"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-west-1"; application_2:: "env[PAPERTRAIL_HOST]" string => "xxx.trailpaperapp.com"; "env[PAPERTRAIL_PORT]" string => "54321"; "env[APPOPTICS_USER]" string => "ops@whatever.com"; "env[APPOPTICS_APIKEY]" string => "xxxxiiiiiiixxixxix"; "env[CORE_JDBC_URL]" string => "jdbc:postgresql://localhost/xxx"; "env[CORE_JDBC_USER]" string => "vagrant"; "env[SMS_ENABLED]" string => "false"; "env[SMS_AWS_REGION]" string => "eu-east-2"; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { classes: "vagrant_vm" expression => "any", scope => "namespace"; methods: #"init"; "app_data"; "global_env"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** R: CFEngine 3.12.0 R: /tmp/etc/environment R: PAPERTRAIL_HOST=xxx.trailpaperapp.com R: PAPERTRAIL_PORT=22222 R: APPOPTICS_USER=ops@cfengine.com R: APPOPTICS_APIKEY=toodledum R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: CORE_JDBC_USER=vagrant R: SMS_ENABLED=false R: SMS_AWS_REGION=eu-west-1 R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=PAPERTRAIL_PORT=22222 R: DefaultEnvironment=APPOPTICS_APIKEY=toodledum R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=APPOPTICS_USER=ops@cfengine.com R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.trailpaperapp.com R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: /tmp/etc/profile R: export SMS_ENABLED=false R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export APPOPTICS_USER=ops@whatever.com R: export CORE_JDBC_USER=vagrant R: export SMS_AWS_REGION=eu-west-1 R: export PAPERTRAIL_PORT=12345 R: export APPOPTICS_APIKEY=cafebabedeadbeef R: export PAPERTRAIL_HOST=xxx.papertrailapp.com ### Centralized management separate data bundle with external data files In addition to the benefits derived from separating data bundles external data files can further enable delegation of control working much more easily with external systems, reduces the size of the policy and improve policy readability, and eases testing of policy. We can use the json, yaml or env file formats interchangeably. CSV is also possible, but different parsing would be required. **/tmp/env.env:** ``` PAPERTRAIL_HOST="xxx.papertrailapp.com" PAPERTRAIL_PORT="11111" APPOPTICS_USER="ops@whatever.com" APPOPTICS_APIKEY="cafebabedeadbeef" CORE_JDBC_URL="jdbc:postgresql://localhost/xxx" CORE_JDBC_USER="vagrant" SMS_ENABLED="false" SMS_AWS_REGION="eu-west-1" ``` **/tmp/env.json:** ```json { "PAPERTRAIL_HOST": "xxx.papertrailapp.com", "PAPERTRAIL_PORT": "11111", "APPOPTICS_USER": "ops@whatever.com", "APPOPTICS_APIKEY": "cafebabedeadbeef", "CORE_JDBC_URL": "jdbc:postgresql://localhost/xxx", "CORE_JDBC_USER": "vagrant", "SMS_ENABLED": "false", "SMS_AWS_REGION": "eu-west-1" } ``` **/tmp/env.yaml:** ```yaml --- PAPERTRAIL_HOST: xxx.papertrailapp.com PAPERTRAIL_PORT: '11111' APPOPTICS_USER: ops@whatever.com APPOPTICS_APIKEY: cafebabedeadbeef CORE_JDBC_URL: jdbc:postgresql://localhost/xxx CORE_JDBC_USER: vagrant SMS_ENABLED: 'false' SMS_AWS_REGION: eu-west-1 ``` **Policy:** ```cf3 bundle agent global_env { vars: "env_i" slist => getindices( "app_data.env" ); # Here we take the original simple key value pair and add different prefixes # for the differnt config file types using a classic array as a generator. "etc_systemd_conf[DefaultEnvironment=$(env_i)]" string => "$(app_data.env[$(env_i)])"; "etc_profile[export $(env_i)]" string => "$(app_data.env[$(env_i)])"; files: "/tmp/etc/environment" edit_line => set_line_based("app_data.env", "=", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, create => "true", handle => "_etc_environment"; "/tmp/etc/systemd/system.conf" edit_line => set_line_based("$(this.bundle).etc_systemd_conf", "=", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, classes => if_repaired("system_conf_changed"), create => "true", handle => "_etc_systemd_system_conf"; "/tmp/etc/profile" edit_line => set_line_based("$(this.bundle).etc_profile", "export ", "\s*=\s*", ".*", "\s*#\s*"), edit_defaults => empty, create => "true", handle => "_etc_profile"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload"; } bundle agent app_data { vars: vagrant_vm:: "env" data => readdata( "/tmp/env.yaml", auto ); } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { classes: "vagrant_vm" expression => "any", scope => "namespace"; methods: #"init"; "app_data"; "global_env"; reports: "CFEngine $(sys.cf_version)"; "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: CFEngine 3.12.0 R: /tmp/etc/environment R: PAPERTRAIL_HOST=xxx.papertrailapp.com R: PAPERTRAIL_PORT=11111 R: APPOPTICS_USER=ops@whatever.com R: APPOPTICS_APIKEY=cafebabedeadbeef R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: CORE_JDBC_USER=vagrant R: SMS_ENABLED=false R: SMS_AWS_REGION=eu-west-1 R: /tmp/etc/systemd/system.conf R: DefaultEnvironment=PAPERTRAIL_PORT=11111 R: DefaultEnvironment=APPOPTICS_APIKEY=cafebabedeadbeef R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.papertrailapp.com R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: /tmp/etc/profile R: export PAPERTRAIL_HOSTexport xxx.papertrailapp.com R: export SMS_AWS_REGIONexport eu-west-1 R: export PAPERTRAIL_PORTexport 11111 R: export APPOPTICS_USERexport ops@whatever.com R: export SMS_ENABLEDexport false R: export CORE_JDBC_USERexport vagrant R: export CORE_JDBC_URLexport jdbc:postgresql://localhost/xxx R: export APPOPTICS_APIKEYexport cafebabedeadbeef ``` ## Cooperative management Here we allow each service to define the key values they need. They *advertise/ or /publish* their data by tagging it. In this iteration, since we have full knowledge of the desired state at one time we switched the file promises to be templates so that the full content is managed. This helps to avoid unknown settings. **Policy:** ```cf3 bundle agent global_env { vars: # Collect a datastrucutre from variables tagged as used_by this bundle "env" data => variablesmatching_as_data( ".*", "used_by=global_env" ); # Get the index of the collected data (the variable names) in lexically # sorted order so that we can iterate over them "i" slist => sort( getindices( env ), lex ); # Here we merge all the objects together by picking out each objects data # and merging on top. NOTE In event of conflict the last definition # alphabetically wins "c" data => '[]'; "c" data => mergedata( c, mergedata( "env[$(i)]")); files: "/tmp/etc/environment" create => "true", template_method => "inline_mustache", template_data => @(c), edit_template_string => "{{#-top-}}{{{@}}}={{{.}}}$(const.n){{/-top-}}"; "/tmp/etc/systemd/system.conf" create => "true", template_method => "inline_mustache", template_data => @(c), edit_template_string => "{{#-top-}}DefaultEnvironment={{{@}}}={{{.}}}$(const.n){{/-top-}}"; "/tmp/etc/profile" create => "true", template_method => "inline_mustache", template_data => @(c), edit_template_string => "{{#-top-}}export {{{@}}}={{{.}}}$(const.n){{/-top-}}"; commands: system_conf_changed:: "/sbin/systemctl daemon-reload" contain => exec_owner("root"), if => strcmp( "$(sys.user_data[username])", "root" ), handle => "_sbin_systemctl_daemon_reload_$(name)_$(value)"; } bundle agent vagrant_vm { vars: "v" data => '{ "PAPERTRAIL_HOST": "xxx.papertrailapp.com", "PAPERTRAIL_PORT": "11111", "APPOPTICS_USER": "ops@whatever.com", "APPOPTICS_APIKEY": "cafebabedeadbeef", "CORE_JDBC_URL": "jdbc:postgresql://localhost/xxx", "CORE_JDBC_USER": "vagrant", "SMS_ENABLED": "false", "SMS_AWS_REGION": "eu-west-1" }', meta => { "used_by=global_env" }; } bundle agent application_2 { vars: "v" data => '{ "PAPERTRAIL_HOST": "xxx.trailpaperapp.com", "PAPERTRAIL_PORT": "22222", "APPOPTICS_USER": "ops@cfengine.com", "APPOPTICS_APIKEY": "toodledum", "CORE_JDBC_URL": "jdbc:postgresql://localhost/xxx", "CORE_JDBC_USER": "vagrant", "SMS_ENABLED": "false", "SMS_AWS_REGION": "eu-west-1" }', meta => { "used_by=global_env" }; } body contain exec_owner(user) { exec_owner => "$(user)"; } bundle agent init { files: "/tmp/etc/environment" delete => tidy; "/tmp/etc/systemd/system.conf" delete => tidy; "/tmp/etc/profile" delete => tidy; } bundle agent __main__ { classes: "vagrant_vm" scope => "namespace"; methods: #"init"; "global_env"; reports: "/tmp/etc/environment" printfile => cat( "$(this.promiser)" ); "/tmp/etc/systemd/system.conf" printfile => cat( "$(this.promiser)" ); "/tmp/etc/profile" printfile => cat( "$(this.promiser)" ); } ``` **Output:** ``` R: /etc/environment R: PAPERTRAIL_HOST=xxx.papertrailapp.com R: PAPERTRAIL_PORT=11111 R: APPOPTICS_USER=ops@whatever.com R: APPOPTICS_APIKEY=cafebabedeadbeef R: CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: CORE_JDBC_USER=vagrant R: SMS_ENABLED=false R: SMS_AWS_REGION=eu-west-1 R: /etc/systemd/system.conf R: DefaultEnvironment=PAPERTRAIL_HOST=xxx.papertrailapp.com R: DefaultEnvironment=PAPERTRAIL_PORT=11111 R: DefaultEnvironment=APPOPTICS_USER=ops@whatever.com R: DefaultEnvironment=APPOPTICS_APIKEY=cafebabedeadbeef R: DefaultEnvironment=CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: DefaultEnvironment=CORE_JDBC_USER=vagrant R: DefaultEnvironment=SMS_ENABLED=false R: DefaultEnvironment=SMS_AWS_REGION=eu-west-1 R: /etc/profile R: export PAPERTRAIL_HOST=xxx.papertrailapp.com R: export PAPERTRAIL_PORT=11111 R: export APPOPTICS_USER=ops@whatever.com R: export APPOPTICS_APIKEY=cafebabedeadbeef R: export CORE_JDBC_URL=jdbc:postgresql://localhost/xxx R: export CORE_JDBC_USER=vagrant R: export SMS_ENABLED=false R: export SMS_AWS_REGION=eu-west-1 ``` As with other patterns moving the data into external files would improve delegation of control and tighter integration with other systems. -------------------------------------------------------------------------------- title: "CFEngine LTS 3.10.5 released" url: /blog/2018/cfengine-lts-3-10-5-released/index.md date: "2018-11-16" -------------------------------------------------------------------------------- Today we are very happy to announce the maintenance release of CFEngine 3.10.5. This is an update to the LTS 3.10 series, adding improved stability, several bug fixes and increased performance. 3.10 LTS is the successor of 3.7 LTS that, since August 2018, is no longer supported. We recommend everyone still using CFEngine 3.7 to upgrade to either 3.10 or 3.12. We are available to support you with such an upgrade if you need it. 3.10.5 LTS is a maintenance release (also known as a patch release), with the goal to increase the stability and reliability for CFEngine users and enable a safe upgrade path. As such, this release primarily includes bug fixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see that - **3.10** LTS is maintained and supported until **December 27th, 2019** Do you want to start contributing but are unsure how? - Send [documentation](https://docs.cfengine.com/) updates as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy or help_wanted that are OPEN or TODO](https://northerntech.atlassian.net/issues/?filter=10073) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core) or [cfengine/masterfiles](https://github.com/cfengine/masterfiles). ## Improvements and fixes We have introduced a number of smaller fixes to **CFEngine Core** in this release. Notably, we fixed a bug that has slightly changed how Mustache templates in policy files are being processed to adhere to the mustache specification, as detailed in [CFE-2125](https://northerntech.atlassian.net/browse/CFE-2125). This further improves the CFEngine mustache capability and makes it easier to create templates using Mustache in CFEngine. We have fixed an issue where in a few cases not all cf-serverd instances were restarting properly after upgrading from 3.7 to 3.10, we have fixed an issue where the queue size of cf-serverd listen() was too small, and we have fixed a potential security issue by falling back to denying traffic to any hostname if the reverse lookup for that hostname fails. As of late, we have put a lot of focus on code quality and fixed several memory leaks. We have enforced the stricter use of Valgrind to avoid introducing any new memory leaks. In [CFE-2896](https://northerntech.atlassian.net/browse/CFE-2896) we have added a new inventory attribute, namely the physical memory available. We also made it simpler to monitor more values from _dmidecode_ without modifying our Masterfiles, as well as added _UUID_ as a default value that CFEngine adds to the inventory. Also worth mentioning is that we have added support for Ubuntu 18.04 to CFEngine 3.10.5 LTS. In **CFEngine Enterprise**, we have also fixed several minor issues. Now only the specific CFEngine components that were running before an upgrade are restarted and run after the upgrade. In Mission Portal, we have made several improvements. We have resolved an issue that prevented some users from exporting reports as CSV, we have fixed a bug that caused some custom notification scripts not to fire, fixed a bug that caused alerts not to show up in the Mission Portal Events Log on some platforms and made it possible for multiple users to create independent reports with the same name, something that was causing some issues previously. On Windows, we have also solved a few specific issues, such as improved the newline sequence handling and sorted out a minor bug that caused installation problems under certain conditions. You can see a list of the issues fixed in our public Jira for 3.10.5 [here](https://northerntech.atlassian.net/issues/?jql=statusCategory%20%3D%20done%20AND%20project%20%3D%2010100%20AND%20fixVersion%20%3D%203.10.5%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC). For more details please see the relevant ChangeLog entries: - CFEngine 3.10.5 LTS [community](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) As usual, we have updated dependencies in order to get the latest security, performance and reliability improvements. | | | | | ---------- | ------ | ------ | | libacl | 2.2.52 | 2.2.53 | | libattr | 2.4.47 | 2.4.48 | | libcurl | 7.59.0 | 7.61.1 | | redis | 3.2.11 | 3.2.12 | | php | 5.6.35 | 5.6.38 | | postgresql | 9.6.8 | 9.6.10 | | apache | 2.4.33 | 2.4.35 | | apr-util | 1.5.4 | 1.6.1 | | apr | 1.5.2 | 1.6.5 | | libyaml | 0.1.7 | 0.2.1 | | openssl | 1.0.2o | 1.0.2p | ``` ``` ``` ``` ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are always happy to assist our customers with upgrading\! You can [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure, and get more out of CFEngine\! ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new release, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! ## Learn CFEngine\! If you would like to refresh your CFEngine knowledge or are learning it from scratch, you can attend one of our training sessions. Now, we have the [following event](https://cfengine.com/engage/events/) scheduled in Prague, Czech Republic, from January 14th. There is also an updated version of the [Learning CFEngine](https://leanpub.com/learning-cfengine/) book by Diego Zamboni available. -------------------------------------------------------------------------------- title: "missing_ok and multiple augments in 3.12.0" url: /blog/2018/missing_ok-and-multiple-augments-in-3-12-0/index.md date: "2018-09-20" -------------------------------------------------------------------------------- At [SURFsara](https://surfsara.nl/) we use CFEngine on our National Compute Cluster ([LISA](https://userinfo.surfsara.nl/systems/lisa/description)) and other systems as our configuration management tool. With the release of [CFEngine 3.12](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew-changelog-core.html) I want to highlight 2 new features, namely: - [`missing_ok`](https://docs.cfengine.com/docs/3.12/reference-promise-types-files.html#missing_ok) - `multiple augments` We use these 2 new features heavily in our framework in combination with my open source library [`cf_surfsara_lib`](https://github.com/basvandervlies/cf_surfsara_lib). This library aims to be a central repository for configuring services, eg: ssh. For configuring the services we use JSON as data format and it is easy to override the default values via JSON. Pre CFEngine 3.12 there is only one strategy possible: - def.json : Setting the initial value - def.cf : Override the initial value Due to this restriction we have a def.json for all hosts even when the difference between them is small. For minor tweaks we use def.cf with the aid of classes. As with CFEngine version 3.12 you can define multiple JSON files with the aid of the [multiple augments](https://docs.cfengine.com/docs/3.12/reference-language-concepts-augments.html#augments) feature. In our environment we have implemented the following strategy in def.json:
Listing 1: def.json ```json { "augments": [ "$(sys.inputdir)/json.d/global.json", "$(sys.inputdir)/json.d/os.json", "$(sys.inputdir)/json.d/key.json", "$(sys.inputdir)/json.d/ip.json" ] } ```
We only have these definitions in our def.json so we can easily create a local JSON file with local paths for testing and running automatic checks. Another issue is that all the JSON files must be present else an error is reported. This issue has been reported: - [CFE-2856](https://northerntech.atlassian.net/browse/CFE-2856) : suppress error message - [CFE-2573](https://northerntech.atlassian.net/browse/CFE-2573) : augments command line option Due to this restriction we must have a default JSON file for all defined JSON files in def.json. In our cluster we have defined roles for this we use the CFEngine key, eg: all compute nodes in our cluster share the same CFEngine key. That is the reason why key.json is not the last JSON file being used in our def.json. For copying the JSON files we use variables that are set by CFEngine, namely: - os.json : is copied via the CFEngine variable: sys.flavor - key.json : is copied via the CFEngine variable: connecton.id - ip.json : is copied via the CFEngine variable: connection.ip We do not care if the os.json, key.json or ip.json files are present on the policy server. For this I wrote a patch and introduced the [`missing_ok`](https://docs.cfengine.com/docs/3.12/reference-promise-types-files.html#missing_ok) feature. When this is set then CFEngine will consider the promise kept even if the file is missing. This is how we use these two new features in CFEngine at SURFsara. Below I will share the update.conf and cfserverd.conf code. It would be nice for the feature if we can specify another JSON merge option than override. The discussion for other merge options is in ticket: - [CFE-2742](https://northerntech.atlassian.net/browse/CFE-2742) : Augments merge strategy Another feature that I have proposed is [CFE-2790](https://northerntech.atlassian.net/browse/CFE-2790) (auto-loading JSON file(s)). With this feature I can just drop JSON file(s) in a directory and it will be loaded just like the autorun bundle feature.
Listing 2: Update policy ```cf3 bundle agent update_def_json() { methods: any:: "" usebundle => update_json_file("global.json", "global_def_json", "global_def_json"); "" usebundle => update_json_file("os.json", "$(sys.flavor)_def_json", "default_os_def_json"); "" usebundle => update_json_file("key.json", "key_def_json", "default_key_def_json"); "" usebundle => update_json_file("ip.json", "ip_def_json", "default_ip_def_json"); } bundle agent update_json_file(file, shortcut, failback_shortcut) { vars: "json_file" string => "$(update_init.json_dir)/$(file)"; files: any:: "$(json_file)" comment => "def json file copy for $(shortcut), missing_ok: true", copy_from => update_def_json("$(shortcut)", "true"), perms => update_perms("600"), move_obstructions => "true", action => update_immediate; "$(json_file)" comment => "default def json file copy for $(failback_shortcut), missing_ok: false", copy_from => update_def_json("$(failback_shortcut)", "false"), perms => update_perms("600"), move_obstructions => "true", action => update_immediate, classes => update_results("$(failback_shortcut)"); reports: any:: "$(this.bundle): Failed to fetch: $(json_file)" ifvarclass => or( canonify("$(failback_shortcut)_failed") ); } body copy_from update_def_json(from, missing) { source => "$(from)"; compare => "hash"; copy_backup => "false"; missing_ok => "$(missing)"; protocol_version => "2"; servers => { @(update_init.servers) }; } ```
Listing 3: cf-serverd policy ```cf3 bundle server access_rules() { ### Global.json "/data/cfengine3/cmdb/global.json" comment => "def.json for all hosts", shortcut => "global_def_json", admit_ips => { "@(g.acl_admit_ips)" }; ### Key def.json "/data/cfengine3/cmdb/key/$(connection.key).json" comment => "host key def.json", shortcut => "key_def_json", admit_keys => { "$(connection.key)" }; "/data/cfengine3/cmdb/key/default.json" comment => "default host key def.json, failback", shortcut => "default_key_def_json", admit_ips => { "@(g.acl_admit_ips)" }; ### End Key def.json ### IP def.json "/data/cfengine3/cmdb/ip/$(connection.ip).json" comment => "host ip def.json", shortcut => "ip_def_json", admit_ips => { "$(connection.ip)" }; "/data/cfengine3/cmdb/ip/default.json" comment => "default host ip def.json, failback", shortcut => "default_ip_def_json", admit_ips => { "@(g.acl_admit_ips)" }; ### End IP def.json ### OS def.json "/data/cfengine3/cmdb/os/debian_9.json" comment => "debian stretch def.json", shortcut => "debian_9_def_json", admit_ips => { "@(g.acl_admit_ips)" }; "/data/cfengine3/cmdb/os/debian_8.json" comment => "debian jessie def.json", shortcut => "debian_9_def_json", admit_ips => { "@(g.acl_admit_ips)" }; "/data/cfengine3/cmdb/os/debian_7.json" comment => "debian jessie def.json", shortcut => "debian_9_def_json", admit_ips => { "@(g.acl_admit_ips)" }; "/data/cfengine3/cmdb/os/centos_6.json" comment => "centos 6 def.json", shortcut => "centos_6_def_json", admit_ips => { "@(g.acl_admit_ips)" }; "/data/cfengine3/cmdb/os/default.json" comment => "default host os def.json, failback", shortcut => "default_os_def_json", admit_ips => { "@(g.acl_admit_ips)" }; ### End OS def.json } ``` This post is syndicated with permission from Bas van der Vlies blog post [CFEngine 3.12 New Features Missing OK and Multiple Augments](https://basvandervlies.blogspot.com/2018/09/cfengine-312-new-features-missingok-and.html).
-------------------------------------------------------------------------------- title: "Restricting CFEngine to one CPU core using Systemd" url: /blog/2018/restricting-cfengine-to-one-cpu-core/index.md date: "2018-08-29" -------------------------------------------------------------------------------- In some performance critical situations, it makes sense to limit management software to a single CPU (core). We can do this using systemd and cgroups. CFEngine already provides systemd units on relevant platforms, we just need to tweak them. I'm using CFEngine Enterprise 3.12 on CentOS 7, but the steps should be very similar on other platforms/versions. This post is based on an excellent article from Red Hat: ## Using ps to check what CPU core is utilized ### Listing all processes and their core We can use ps to check CPU core for desired processes: $ ps x -o pid,psr,comm PID PSR COMMAND 1 1 systemd 2 0 kthreadd 3 0 ksoftirqd/0 5 0 kworker/0:0H 6 1 kworker/u4:0 7 0 migration/0 8 0 rcu_bh 9 1 rcu_sched 10 0 lru-add-drain [...] (first column is PID, second is core, third is the process command) ### Filtering using grep $ ps x -o pid,psr,comm | grep [c]f- 10716 1 cf-serverd 10719 3 cf-execd 10723 3 cf-monitord 10791 0 cf-hub (The brackets in the grep command means it won't match its own process). ## Systemd services ### Location and editing CFEngine provides an umbrella service: $ systemctl status cfengine3 ● cfengine3.service - CFEngine 3 umbrella service Loaded: loaded (/usr/lib/systemd/system/cfengine3.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2018-07-27 13:23:43 UTC; 1min 21s ago Docs: https://docs.cfengine.com/ [...] The other services, like `cf-serverd` and `cf-execd` etc. are in the same folder: $ ls -al /usr/lib/systemd/system/ | grep [c]f -rw-r--r--. 1 root root 433 Jun 28 10:03 cf-apache.service -rw-r--r--. 1 root root 851 Jun 28 10:03 cfengine3.service -rw-r--r--. 1 root root 385 Jun 28 10:03 cf-execd.service -rw-r--r--. 1 root root 387 Jun 28 10:03 cf-hub.service -rw-r--r--. 1 root root 358 Jun 28 10:03 cf-monitord.service -rw-r--r--. 1 root root 1150 Jun 28 10:03 cf-postgres.service -rw-r--r--. 1 root root 638 Jun 28 10:03 cf-runalerts.service -rw-r--r--. 1 root root 437 Jun 28 10:03 cf-serverd.service To edit them, we should make copies in `/etc/systemd/system`: $ cp /usr/lib/systemd/system/cf* /etc/systemd/system/ $ ls -al /etc/systemd/system/ | grep [c]f -rw-r--r--. 1 root root 433 Jul 27 13:26 cf-apache.service -rw-r--r--. 1 root root 851 Jul 27 13:26 cfengine3.service drwxr-xr-x. 2 root root 185 Jul 27 13:22 cfengine3.service.wants -rw-r--r--. 1 root root 385 Jul 27 13:26 cf-execd.service -rw-r--r--. 1 root root 387 Jul 27 13:26 cf-hub.service -rw-r--r--. 1 root root 358 Jul 27 13:26 cf-monitord.service -rw-r--r--. 1 root root 1150 Jul 27 13:26 cf-postgres.service drwxr-xr-x. 2 root root 34 Jul 27 13:22 cf-postgres.service.wants -rw-r--r--. 1 root root 638 Jul 27 13:26 cf-runalerts.service -rw-r--r--. 1 root root 437 Jul 27 13:26 cf-serverd.service If we reload and restart, we should see that systemd is now using our new copies: $ systemctl daemon-reload $ systemctl restart cfengine3 $ systemctl status cfengine3 ● cfengine3.service - CFEngine 3 umbrella service Loaded: loaded (/etc/systemd/system/cfengine3.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2018-07-27 13:28:54 UTC; 4s ago Docs: https://docs.cfengine.com/ [...] ### Making a cgroup for the umbrella service `/etc/systemd/system/cfengine3.service` is the umbrella service which starts and stops all the other services. We will create one shared cgroup, called `cfe_group` for the different services. We want to add 4 lines to set up and tear down the cgroup: ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/cfe_group ExecStartPre=/bin/bash -c '/usr/bin/echo "1" > /sys/fs/cgroup/cpuset/cfe_group/cpuset.cpus' ExecStartPre=/bin/bash -c '/usr/bin/echo "0" > /sys/fs/cgroup/cpuset/cfe_group/cpuset.mems' ExecStopPost=/usr/bin/rmdir /sys/fs/cgroup/cpuset/cfe_group The *pre* steps create a cgroup and say that all processes within it should run on cpu `1`. The *post* step deletes the cgroup using `rmdir`. The final version should look like this: $ cat /etc/systemd/system/cfengine3.service [Unit] Description=CFEngine 3 umbrella service Documentation=https://docs.cfengine.com/ After=syslog.target [Install] WantedBy=multi-user.target [Service] Type=oneshot RemainAfterExit=yes # Creat cgroup to assign processes to CPU 1 ExecStartPre=/usr/bin/mkdir -p /sys/fs/cgroup/cpuset/cfe_group ExecStartPre=/bin/bash -c '/usr/bin/echo "1" > /sys/fs/cgroup/cpuset/cfe_group/cpuset.cpus' ExecStartPre=/bin/bash -c '/usr/bin/echo "0" > /sys/fs/cgroup/cpuset/cfe_group/cpuset.mems' # ENT-2841: Ensure synchronous start behavior ExecStart=/bin/systemctl start cf-serverd ExecStart=/bin/systemctl start cf-execd ExecStart=/bin/systemctl start cf-monitord ExecStart=/bin/systemctl start cf-postgres ExecStart=/bin/systemctl start cf-apache ExecStart=/bin/systemctl start cf-runalerts ExecStart=/bin/systemctl start cf-hub # ENT-2841: Ensure synchronous stop behavior ExecStop=/bin/systemctl stop cf-serverd ExecStop=/bin/systemctl stop cf-execd ExecStop=/bin/systemctl stop cf-monitord ExecStop=/bin/systemctl stop cf-hub ExecStop=/bin/systemctl stop cf-runalerts ExecStop=/bin/systemctl stop cf-apache ExecStop=/bin/systemctl stop cf-postgres # Tear down cgroup ExecStopPost=/usr/bin/rmdir /sys/fs/cgroup/cpuset/cfe_group There is no relevant PID in this service, so we will have to add the individual PID in the different services. For each of the services you want in the cgroup (running on CPU 1), you need to add this line to the service file: ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID >> /sys/fs/cgroup/cpuset/cfe_group/tasks' I added it to the 4 processes we saw earlier, `cf-hub`, `cf-serverd`, `cf-execd`, `cf-monitord`. `cf-execd` spawns (forks) `cf-agent`, so the agent will also be in the same cgroup. As an example, my cf-execd service looks like this: $ cat /etc/systemd/system/cf-execd.service [Unit] Description=CFEngine Enterprise Execution Scheduler After=syslog.target ConditionPathExists=/var/cfengine/bin/cf-execd ConditionPathExists=/var/cfengine/inputs/promises.cf PartOf=cfengine3.service [Service] Type=simple ExecStart=/var/cfengine/bin/cf-execd --no-fork Restart=always RestartSec=10 KillMode=process # Add PID to cgroup: ExecStartPost=/bin/bash -c '/usr/bin/echo $MAINPID >> /sys/fs/cgroup/cpuset/cfe_group/tasks' [Install] WantedBy=multi-user.target WantedBy=cfengine3.service ### Reloading and checking results: Again, we need to reload and restart to see results: $ systemctl daemon-reload $ systemctl restart cfengine3 $ ps ax -o pid,psr,comm | grep [c]f 11934 1 cf-serverd 11941 1 cf-execd 11947 1 cf-monitord 12002 1 cf-hub That's it\! Change the `1` in `cfengine3.service` to whatever cpu core you want to use, then reload and restart to check that it worked. A good next step is to automate this, make CFEngine set up these services with our new modifications. If you want to use multiple cores for CFEngine (but not all), there are two options: - Create more cgroups and assign the services how you want them manually - Instead of writing `"1"` to `cpuset.cpus` use a comma-separated list: ExecStartPre=/bin/bash -c '/usr/bin/echo "1,2,3" > /sys/fs/cgroup/cpuset/cfe_group/cpuset.cpus' -------------------------------------------------------------------------------- title: "CFEngine 3.12.0 LTS Released" url: /blog/2018/cfengine-3-12-0-lts-released/index.md date: "2018-06-28" -------------------------------------------------------------------------------- Today we are happy to announce the general availability of CFEngine 3.12.0 LTS\! This release has a lot of new features, and we are very excited about all the new possibilities you get with CFEngine 3.12.0 LTS. If you are using the previous LTS, 3.10 you will also benefit from all the new features, improvements and testing of the 3.11 release, which you can read more about in the [CFEngine 3.11](https://cfengine.com/company/blog-detail/cfengine-3-10-2-lts-and-3-11-0-non-lts-released/) release post. ## New Features in Core We have introduced a wide set of new features in CFEngine 3.12, making it an up to date and modern version. ### Supported platforms Based on the results from a wide platform survey, download statistics of CFEngine, and the feedback during the beta period of this release, we have ended up with a slight modification of our supported platforms compared to the previous LTS release of CFEngine. We want to support as many platforms as possible, and try to keep all changes and new features compatible with all both supported and unsupported platforms. Maintaining support involves both a long-term commitment as well as investments in hardware, software and time. For 3.12 LTS series have dropped the support for the following platforms. - Solaris 9 - AIX 5.3 In addition, we have added support for Ubuntu 18.04 in CFEngine 3.12 LTS series. If you have any questions or comments about this, please reach out to us. If you still need support for an older version of CFEngine or an unsupported platform, please look at our [Extended Lifetime Support](https://cfengine.com/extended-support/) pages, and get in touch with us for more information. ### Improved structure of augments files. In CFEngine we are working hard to make policy execution more robust, and make it simpler to upgrade from one version of CFEngine to the next. To this end, we have now expanded the use of augments files, that allow you to define policy sets so that you can make fewer changes directly to the masterfiles. This is an important step in improving how policy is implemented, and it is now easier than ever to maintain a large policy set in a simple and robust way. The support for multiple augments files is a much more robust way to organize your policy, and can make it a lot simpler to keep your policy in place, when upgrading from one version of CFEngine to another. You can learn all about it in our [documentation](https://docs.cfengine.com/docs/3.12/reference-language-concepts-augments.html). ### Improved classification for new Linux distributions Most new Linux distributions have standardized encoding release information in `/etc/os-release` per the [freedesktop.org standard](https://www.freedesktop.org/software/systemd/man/os-release.html). As of 3.12 CFEngine uses this information for operating system classification. ### Improved support for inline templates In CFEngine we have implemented support for rendering inline Mustache templates. This is an important feature for the many CFEngine users that heavily rely on small templates. A quick example of this is shown here: ```cf3 bundle agent main { vars: "d" data => '{ "hello": "world", "feature": [ "render", "inline", "mustache" ] }'; files: "/tmp/example.txt" create => "true", template_method => "inline_mustache", edit_template_string => "{{% print `{{%-top-}}` %}}", template_data => @(d); } ``` Results in `/tmp/example.txt` having this content. ```json { "feature": ["render", "inline", "mustache"], "hello": "world" } ``` We hope this can improve your workflow and your policy. ## New Enterprise Features Mission Portal makes using CFEngine easier and makes understanding the state of your system a lot easier. And with the CFEngine 3.12.0 LTS release, we bring a wide set of large and small improvements to Mission Portal to make it even more powerful and easier to use. ### Improved Mission Portal UI responsiveness at large scale With this release we have focused a lot on usability, and user experience. In a response to the request of several customers with large-scale environments, Mission Portal and APIs now support up to 50,000 hosts in a single hub, and the most common queries will respond within 10 seconds at this scale\! Note that these improvements are intended to be used with a dedicated "query hub" where the data sets from multiple hubs have been coalesced. ### New Inventory API In CFEngine 3.12 we have introduced a new Inventory REST API to augment the powerful Mission Portal Inventory reporting feature. This Mission Portal feature allows users to sort hosts on Operating System, or other attributes. You can now also use this powerful REST API to do the same. ### New LDAP settings API As most of the CFEngine users are using LDAP for authentication we have made this easier and more powerful to use. CFEngine Enterprise 3.12.0 LTS introduces a [new LDAP settings](https://docs.cfengine.com/docs/3.12/enterprise-cfengine-guide-settings.html#authentication-settings) implementation with [accompanying API](https://docs.cfengine.com/docs/3.12/reference-enterprise-api-ref-ldap-api.html) supporting new highly requested features, including custom LDAP ports and default roles. By using a default role you can create a role that new users of Mission Portal (and APIs) will get, both if they are created in the local database or come from the LDAP integration. This enables you to decide which hosts all new users will see upfront before they log in to their account for the first time. ### New email settings In mission portal, you can easily set up email notifications on report conditions, or warnings. In 3.12 we have improved the email settings, and you can now choose how you send emails from Mission Portal. CFEngine Enterprise 3.12.0 LTS beta brings new email settings that can easily be [configured in Mission Portal](https://docs.cfengine.com/docs/3.12/enterprise-cfengine-guide-settings.html#mail-settings). Authenticated SMTP support is added, and exported reports (CSV/PDF) can now be attached to outbound emails. There is also a configurable maximum attachment size to give you control over the attachments that are sent. The default max size is set to 10MB. ### New Host count dashboard widget We are making it simpler to manage your dashboard in Mission Portal, and have introduced many new features. One of them is a new Host count widget. This widget allows you to quickly see the volume of managed hosts over time. ![](images/2018-01-10_Selection_001.png) ### Better Health Diagnostics in Mission Portal In 3.12 we have put a lot of effort into making it easier to understand the overall health of your entire system. When there are reports with an unclear origin, this has caused some confusion in the past. To solve this we have had a Health indicator, however in 3.12.0 LTS we have now extended this, and clarified its usage and meaning. The current report categories that are available are - Hosts Never collected from - Hosts not recently collected from - Hosts likely using the same identity, and - Agents not recently run. Two of these have been renamed since their usage in the Health Indicator. - "Hosts not reporting" has been renamed to "Hosts not recently collected from" and - "Agents not running" has been renamed to "Agents not recently run" The categories are in priority order and non-overlapping, so any host can maximum be in one of these categories. For each category, there is a report of hosts in it, together with diagnostics information about what exactly it means and probable root causes with remediation recommendations. This should make reporting more reliable, and issues that arise should be simpler to fix. ### Improved Reporting Pipeline The reporting feature in CFEngine is one of the components that adds the most value for end users. Therefore it is also one of the components we care deeply about. We have removed the Redis database dependency we used to cache reports before committing them to the database, and simplified our architecture to use PostgreSQL only, instead of the two-stage approach we used to have. This will improve the stability of the reporting pipeline, decrease the overall complexity of the software, and decrease the load on the Hub. ## Other bug fixes Based on customer and community feedback, and with a lot of help from the community we have fixed a lot of small and large issues in the CFEngine codebase, making it a better, more productive tool for all our users. We appreciate the effort of those that contribute, and want to extend a big thank you to everyone who reported, contributed or even fixed bugs in CFEngine to help us make 3.12.0 LTS the best version of CFEngine so far. You can read through the whole list of issues fixed in this version of CFEngine in the [Change Logs](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew.html). ## Dependencies As can be seen from the lists below, we have spent a lot of time on updating the dependencies in CFEngine 3.12.0 LTS. The largest changes since 3.10 LTS are that we have moved from PHP 5 to PHP 7, OpenSSL 1.0 to OpenSSL 1.1, as mentioned above we have removed Redis and have moved from Postgres 9 to Postgres 10. This makes CFEngine a better, more stable product, and we have updated to newer versions of packages with a significant number of CVE's fixed. ### Core Agent Dependencies | Package | CFEngine 3.12.x | | ------------ | --------------- | | cyrus-sasl | 2.1.26 | | lcov | 1.13 | | libacl | 2.2.52 | | libattr | 2.4.47 | | libcurl | 7.59.0 | | libgnurx | 2.5.1 | | libiconv | 1.15 | | libxml2 | 2.9.8 | | libyaml | 0.1.7 | | libvirt | removed in 3.11 | | LMDB | 0.9.22 | | OpenSSL | 1.1.0h | | OpenLDAP | 2.4.46 | | PCRE | 8.42 | | pthreads-w32 | 2.9.1 | | zlib | 1.2.11 | | PostgreSQL | removed in 3.11 | ### Enterprise Hub Dependencies | Package | CFEngine 3.12.x | | ---------- | --------------- | | Apache | 2.4.33 | | APR | 1.6.3 | | apr-util | 1.6.1 | | Git | 2.17.1 | | libmcrypt | removed | | OpenLDAP | 2.4.46 | | PHP | 7.2.6 | | PostgreSQL | 10.4 | | Redis | removed | | rsync | 3.1.3 | ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the [CFEngine Enterprise 3.12 Vagrant environment](https://docs.cfengine.com/docs/3.12/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.12.0 LTS beta released!" url: /blog/2018/cfengine-3-12-0b1-released/index.md date: "2018-05-08" -------------------------------------------------------------------------------- Today we are happy to announce the general availability of CFEngine 3.12.0 LTS beta. The release of 3.12.0 beta took longer than expected. As a result we have decided to adjust our release schedule, and we are releasing 3.12.0 beta now and later this year 3.12.0 will be the next official LTS release. If you are planning to contribute features or fixes to the next set of releases (we warmly welcome that \!), we are always accepting those. The window for features is not closed for 3.12, but we appreciate help with fixing potential bugs in the beta, improving the performance if there are any identified issues, or generally implementations of suggestions moving forwards. If you want to start contributing but are unsure how? - Send documentation [updates](https://docs.cfengine.com) as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy]() that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core). # Beta Program We appreciate all feedback we can get on this beta release. If you test it, you can provide any and all feedback [here](https://www.surveymonkey.com/r/8Q5FPWH). # Notable changes in CFEngine 3.12.0 LTS beta A detailed description of what has changed in 3.12.0 LTS beta can be found in the [3.12 changelogs](https://docs.cfengine.com/docs/3.12/guide-latest-release-whatsnew.html). ## Supported platforms We have recently been conducting both a Platform Survey, as well as trying to analyse the download numbers and see what platforms are still being widely used and if some platforms are no longer as relevant to our users as they once were. We want to support as many platforms as possible, and try to keep all changes and new features compatible with all both supported and unsupported platforms. But maintaining support involves both a long term commitment as well as investments in HW, SW and time. For 3.12 LTS series we therefor suggest to drop the support for the following platforms. - Solaris 9 - AIX 5.3 We will add support for Ubuntu 18.04 in CFEngine 3.12 LTS series. We would want to hear directly from you if you have a different perspective on this. Are you using any of the platforms we are planning to drop support for, or if you want us to add support for another platform, now is the time to tell us. ## Multiple augments files can now be used **NOTE:** Discussion about current implementation in progress in [CFE-2741](https://northerntech.atlassian.net/browse/CFE-2741). Please consider contributing to the discussion. You merge more specific augments on top those set first from `def.json`. For example create `/tmp/example.cf` with the following content. ```cf3 bundle agent main { reports: "def.my_var == $(def.my_var)"; "def.my_other_var == $(def.my_other_var)"; "def.centos_6_var == $(def.centos_6_var)"; } ``` Create `/tmp/def.json` with the following content. Note the `augments` keys position currently affects how this functions. See [CFE-2741](https://northerntech.atlassian.net/browse/CFE-2741) for details. ```json { "vars": { "my_var": "defined in def.json", "my_other_var": "Defined ONLY in def.json" }, "augments": ["/tmp/$(sys.flavor).json"] } ``` Create an additional augments file to merge on top. `/tmp/$(sys.flavor).json` where `$(sys.flavor)` is the platform you are running. For example create `/tmp/centos_6.json` with the following content. ```json { "vars": { "my_var": "Overridden in centos_6.json", "centos_6_var": "Defined ONLY in centos_6.json" } } ``` Running the policy will produce output that looks like this: ```example [root@hub tmp]# cf-agent -KIf ./example.cf R: def.my_var == Overridden in centos_6.json R: def.my_other_var == Defined ONLY in def.json R: def.centos_6_var == Defined ONLY in centos_6.json ``` ## New template method for inline mustache You are no longer required to use an external file template. Now you can provide the template directly within the policy. For example: ```cf3 bundle agent main { vars: "d" data => '{ "hello": "world", "feature": [ "render", "inline", "mustache" ] }'; files: "/tmp/example.txt" create => "true", template_method => "inline_mustache", edit_template_string => "{{% print `{{%-top-}}` %}}", template_data => @(d); } ``` Results in `/tmp/example.txt` having this content. ```json { "feature": ["render", "inline", "mustache"], "hello": "world" } ``` ## Improved UI responsiveness at large scale In a response to customers of large scale environments, Mission Portal and APIs now support up to 50,000 hosts in a single hub, and the most common queries will respond within 10 seconds at this scale\! Note these improvements are intended to be used with a dedicated "query hub" where multiple hubs data sets have been coalesced. {{< youtube iim5axkF64s >}} {{< youtube Kdpjm95vGhE >}} {{< youtube zoMDV7Wd6t4 >}} ## New Inventory API Mission Portals Inventory reporting interface is popular for reporting standard inventory like OS distribution across hosts. However often these data needs to be integrated with external systems. 3.12.0 LTS beta brings [a new REST API](https://docs.cfengine.com/docs/3.12/reference-enterprise-api-ref-inventory.html) to better support extracting and using this information. ```bash curl --user admin -X POST \ -H 'content-type: application/json' \ https://hub/api/inventory -d '{ "select":[ "Host name", "OS type"]}' ``` ```json { "data": [ { "header": [ { "columnName": "Host name", "columnType": "STRING" }, { "columnName": "OS type", "columnType": "STRING" } ], "queryTimeMs": 11, "rowCount": 2, "rows": [ ["host001", "linux"], ["hub", "linux"] ] } ], "meta": { "count": 1, "page": 1, "timestamp": 1515607751, "total": 1 } } ``` ## New LDAP settings API [![](images/Authentication-settings-800x558.png)](images/Authentication-settings.png) CFEngine Enterprise 3.12.0 LTS beta also introduces a [new LDAP settings](https://docs.cfengine.com/docs/3.12/enterprise-cfengine-guide-settings.html#Authentication-settings) implementation with [accompanying API](https://docs.cfengine.com/docs/3.12/reference-enterprise-api-ref-ldap-api.html) supporting new highly requested features, including custom LDAP ports and default roles. By using a default role you can create a role that new users of Mission Portal (and APIs) will get, both if they are created in the local database or come from the LDAP integration. This enables you to decide which hosts all new users will see upfront, before they log in to their account for the first time. ## New email settings [![](images/mail-settings-800x419.png)](images/mail-settings.png) CFEngine Enterprise 3.12.0 LTS beta brings new email settings that can be [configured in Mission Portal](https://docs.cfengine.com/docs/3.12/enterprise-cfengine-guide-settings.html#mail-settings). Authenticated SMTP support is added, and exported reports (CSV/PDF) can now be attached to outbound emails if they do not exceed the configured maximum attachment size (default max size is 10MB). ## New Host count dashboard widget The new Host count widget allows you to visualize the number of agents reporting over a specified range of time. [![](images/2018-01-10_Selection_001.png)](/blog/2018/images/2018-01-10_Selection_001.png) Bootstrap and decommission events can now be found in the Event log. [![](images/2018-01-14_Selection_003-800x279.png)](images/2018-01-14_Selection_003.png) ## Improved Health Diagnostics in Mission Portal When there are reports with an unclear origin, this has caused some confusion in the past. We have continuously improved how reposting is done, and among other things we have had a Health indicator already, however we have now extended this, and clarified its usage and meaning. The current two report categories that are available, have been renamed as follows: - "Hosts not reporting" have been renamed "Hosts not recently collected from" and - "Agents not running" have been renamed "Agents not recently run" The categories are in priority order and non-overlapping, so any host can maximum be in one of these two categories. For each category there is a report of hosts in it, together with diagnostics information about what exactly it means and probable root causes with remediation recommendations. This should make reporting more reliable, and issues that arise should be simpler to fix. We have also added a new health category for hosts that are misconfigured so that they use same key and appear to be the same host. This has created some unreliable reporting in the past, that we now mitigate. ## New Host Exclusion List Many of our customers are using reporting extensively, and have many different reports. In mission portal it is now straight forward to create a report for a set of hosts, but add a list of hosts to be excluded from the search criteria directly from the Mission Portal UI. This makes it significantly simpler to create and maintain such exclusion lists on top of dynamic host groups. ## Removal of Redis database In 3.12.0 LTS beta we have removed the Redis database structure from our reporting pipeline, to make sure that we have a more robust, unified database structure. This is now only using Postgres. This ensures that the consistency of reports is higher, that all the reports have correct time stamps and arrive in the correct order. This is a significant improvement to our reporting, and will benefit all the customers of CFEngine that use reporting. This allows us to improve the performance of reporting, and is a key factor in enabling a single pane of glass from all systems in a large infrastructure. This change will allow many changes to be done faster in the lifetime of 3.12 and the subsequent non-LTS releases. # Dependencies As can be seen from the lists below, we have spent a lot of time on updating the dependencies. The largest changes are that we have moved from PHP 5 to PHP 7, OpenSSL 1.0 to OpenSSL 1.1, we have removed Redis and have moved from Postgres 9 to Postgres 10. We therefore encourage every user of CFEngine to test this beta release as much as possible to make sure we find all potential issues before the final release before the summer. ## Core Agent Dependencies | Package | CFEngine 3.12.x | | ------------ | --------------- | | cyrus-sasl | 2.1.26 | | lcov | 1.13 | | libacl | 2.2.52 | | libattr | 2.4.47 | | libcurl | 7.59.0 | | libgnurx | 2.5.1 | | libiconv | 1.15 | | libxml2 | 2.9.8 | | libyaml | 0.1.7 | | libvirt | removed | | LMDB | 0.9.22 | | OpenSSL | 1.1.0h | | OpenLDAP | 2.4.46 | | PCRE | 8.42 | | pthreads-w32 | 2.9.1 | | zlib | 1.2.11 | | PostgreSQL | removed | ## Enterprise Hub Dependencies | Package | CFEngine 3.12.x | | ---------- | --------------- | | Apache | 2.4.29 | | APR | 1.6.3 | | apr-util | 1.6.1 | | Git | 2.17.0 | | libmcrypt | removed | | OpenLDAP | 2.4.45 | | PHP | 7.2.4 | | PostgreSQL | 10.3 | | Redis | removed | | rsync | 3.1.3 | # Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the [CFEngine Enterprise 3.12 Vagrant environment](https://docs.cfengine.com/docs/3.12/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.8 LTS and 3.10.4 LTS released!" url: /blog/2018/cfengine-3-7-8-lts-and-3-10-4-lts-released/index.md date: "2018-05-02" -------------------------------------------------------------------------------- We're happy to announce the maintenance releases 3.7.8 LTS _and_ 3.10.4 LTS today\! 3.7.8 LTS and 3.10.4 LTS, being maintenance (aka patch) releases, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, these releases primarily include bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see that - **3.7** LTS is maintained (and supported for Enterprise customers) until **August 31st 2018** - **3.10** LTS is maintained (and supported for Enterprise customers) until **December 27th 2019** Want to start contributing but unsure how? - Send documentation [updates](https://docs.cfengine.com/) as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy that are OPEN or TODO]() that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core). ## Improvements and fixes We have introduced a number of smaller fixes for these releases. Some of the issues we have fixed in CFEngine Community include a bug in 3.7.3 and 3.10.3 where some data that was not on the system showed up when running cf-agent -v. We called this a palimpsest bug, and it has now been fixed. We have added the correct ipv6\_ prefix to the IPv6 class. We keep caring about all platforms, and have fixed a bug specific to AIX where memory usage was not reported correctly. We have made a change where isExecutable() no longer returns an error when it follows a broken symlink, but only returns false, which is the same behavior as it will have for a file that is not there. CFEngine now loads augments at the end of context discovery. This means that classes defined as part of the context discovery - e.g. 'am_policy_hub' and 'policy_server' - can be used in the augments. In addition, we have improved the documentation in several places where it was not as clear or correct as it could be. In CFEngine Enterprise, we have fixed several minor issues. There was a bug where some widgets that should be movable in Mission Portal would snap back to where they were, that has now been fixed. You can now get command line info about all processes at once on Windows. We have also introduced a fix that significantly reduces the CPU usage during evaluation of the _packages_ promise on Windows. System V init now notifies when it skips managing postgres because of HA. We have also added a very good guide to the documentation, "How to securely bootstrap your environment". As usual, dependencies have been upgraded in order to get the latest security, performance and reliability improvements. You can see a list of the issues fixed in our public Jira for 3.7.8 [here](https://northerntech.atlassian.net/issues/?jql=statusCategory%20%3D%20done%20AND%20project%20%3D%2010100%20AND%20fixVersion%20%3D%2011301%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC), and for 3.10.4 [here](https://northerntech.atlassian.net/issues/?jql=statusCategory%20%3D%20done%20AND%20project%20%3D%2010100%20AND%20fixVersion%20%3D%2011301%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC). For more details please see the relevant ChangeLog entries: - CFEngine 3.7.8 LTS [community](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.7/guide-latest-release-policy-framework-updates.html) - CFEngine 3.10.4 LTS [community](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-upgrading.html) or [3.7](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact sales](mailto:sales@northern.tech?subject=Request%20quote%20for%20upgrading%20CFEngine) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [3.10](https://cfengine.com/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) or [3.7](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.7 LTS and 3.10.3 LTS released!" url: /blog/2018/cfengine-3-7-7-lts-and-3-10-3-lts-released/index.md date: "2018-02-06" -------------------------------------------------------------------------------- We're happy to announce the maintenance releases 3.7.7 LTS _and_ 3.10.3 LTS today\! 3.7.7 LTS and 3.10.3 LTS, being maintenance (aka patch) releases, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, these releases primarily include bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](/product/supported-versions/), we can see that - **3.7** LTS is maintained (and supported for Enterprise customers) until **July 17th 2018** - **3.10** LTS is maintained (and supported for Enterprise customers) until **December 27th 2019** If you are planning to contribute features or fixes to the next minor release 3.12.0 (thank you\!), please note that we would need the pull requests ready for merging by the end of February 2018. Want to start contributing but unsure how? - Send documentation [updates](https://docs.cfengine.com) as pull requests to [cfengine/documentation](https://github.com/cfengine/documentation). - Search for [issues labeled easy](https://northerntech.atlassian.net/browse/CFE-2754?jql=project%20%3D%20CFE%20AND%20labels%20%3D%20easy) that are good candidates for new contributors to [cfengine/core](https://github.com/cfengine/core). ## Improvements and fixes Automatic reload of def.json (aka. augments file) in cf-execd has now been fixed, so when you make changes here cf-execd will pick them up without a process restart. There are several improvements to package management in masterfiles. The systemd integration has received several fixes, for example to ensure that no processes created by CFEngine are affected by managing the CFEngine systemd units. In 3.10.3, inventory support for Xen hypervisors and Alpine Linux has been improved further. In CFEngine Enterprise, you can now easily graph and report upon the policy version distribution across your hosts, as the Policy Release ID is inventoried and thus available out-of-the-box in the Mission Portal Inventory report. Use of the Enterprise APIs is made easier, as malformed API requests are explicitly flagged in the responses. The maximum length for Mission Portal passwords has been increased to 100 characters, for those who prefer passphrases. On the scalability side, the database tables can now hold much more data as they now use a larger field for the ID column, which is helpful if you are merging databases from several hubs into one for reporting purposes. Environments with less reliable network can enjoy more reliable report collection with a default timeout for report collection (cf-hub) of 30 seconds, increased from the previous 10 second limit. As usual, dependencies have been upgraded in order to get the latest security, performance and reliability improvements. For more details please see the relevant ChangeLog entries: - CFEngine 3.7.7 LTS [community](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.7/guide-latest-release-policy-framework-updates.html) - CFEngine 3.10.3 LTS [community](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html) ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.10](/docs/3.10/guide-installation-and-configuration-upgrading.html) or [3.7](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine customer support](https://cfengine.zendesk.com/hc/en-us) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the CFEngine Enterprise Vagrant environment for [3.10](/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) or [3.7](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.6 released!" url: /blog/2017/cfengine-3-7-6-released/index.md date: "2017-09-13" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.7.6 is released\! With 3.7 being a stable LTS branch, 3.7.6 brings bug fixes and stability enhancements to the CFEngine customers and community. Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see: - **3.7** LTS is maintained (and supported for enterprise customers) until **July 17th 2018**. - **3.10** LTS is maintained (and supported for enterprise customers) until **December 27th 2019**. If you are planning to contribute a feature to the next feature release (thank you\!), please note that we would need the pull requests ready for merging by the end of September for incorporation into 3.12. If you are planning to contribute fixes to 3.10 or 3.7 LTS please note that we would need the pull requests ready for merging by the end of September for incorporation into 3.7.7 and 3.10.3. RPM packages now respect the `chkconfig` specified state when stopping a service. Now if the cfengine3 service is off for runlevel 2 the CFEngine services are stopped when you switch to runlevel 2. cf-monitord now correctly detects the usernames for processes on AIX. Classification when running under the Xen Hypervisor was also fixed. Masterfiles got fixes to the `apt_get` package module so that it works correctly when more than one source repository contains the package. Masterfiles also saw the addition of oslevel (on AIX), mailx (on Linux, Darwin, OpenBSD, NetBSD, and FreeBSD) to the paths bundle. The [`prunetree` bundle](https://docs.cfengine.com/docs/3.7/reference-standard-library-common.html#prunetree) was added to the standard library making it easier to recursively delete files and directories up to a specified depth older than a specified number of days. Enterprise gets bug fixes related to exporting reports and sharing host categorization views and reports. Additionally when hostnames are displayed in reports they now link to the individual host info page and usernames are now allowed to contain dots (`.`). Masterfiles now ensures the postrgres log file is rotated. The verbosity of some maintenance policy was increased and the policy to clear a build up of unreported data now includes `previous_state` and `untracked reports`. Some Enterprise dependencies were updated: - Update libcurl from 7.53.1 to 7.54.1 - Update PCRE from 8.40 to 8.41 - Upgrade openldap from 2.4.44 to 2.4.45 - Update PHP from 5.6.30 to 5.6.31 Thanks to everyone who contributed to this release\! For more details on the improvements in the releases, please see the relevant ChangeLogs: - 3.7.5 LTS ChangeLog: [core](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html), [enterprise](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html), and [masterfiles](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-masterfiles.html). Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.10/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine](mailto:contact@cfengine.com) customer support to receive a fixed-price quote for upgrading your CFEngine infrastructure. Get it\! CFEngine Enterprise packages can be [downloaded here](https://cfengine.com/product/free-download) or you can take a quick spin with the CFEngine Enterprise 3.7 [Vagrant environment](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/product/community/source-code/), [packages](https://cfengine.com/product/community/), and is available in both [apt and yum package repositories](https://cfengine.com/product/community/cfengine-linux-distros/) - to make installation as easy as possible\! We hope you enjoy the new release, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help%20cfengine)\! -------------------------------------------------------------------------------- title: "Security Through Automation" url: /blog/2017/security-through-automation/index.md date: "2017-08-16" -------------------------------------------------------------------------------- Today's approach to securing IT infrastructure is passé. In a dynamic world of unpredictable and often frequent infrastructure changes, the traditional approach to security falls short. It is no longer sufficient to just scan frequently for vulnerabilities and then try to interpret this data in real time without (human) error. Additionally, despite smart analytics, this approach to illuminating security issues and remediating them is extremely time consuming. How many organizations can really claim to have identified and fixed all vulnerabilities? None\! Automation has brought agility and consistency to infrastructure and other workflow services now. Security can and should expect to see similar gains. In this blog we explore some of the reasons that make organizations vulnerable and provide guidance on how they can better counter and secure their infrastructure and applications. ## You Are Never Stronger Than The Weakest Link ![](images/CFEngine-blog-security-automation-weak-link-800x533.png) Adversaries don't care about the latest in fashion\! Whether the open backdoor is offered by the latest shiny RHEL version, or one of its aging versions, attackers don't care. You are never stronger than the weakest link. This implies all systems must at all times be under continuous scrutiny and protection. All running systems need active protection. It doesn't matter that the team which previously was responsible for a few boxes in the far corner of your data center no longer works for the company, or that the couple of virtual instances someone in the outsourcing team launched a few years ago no longer seem to be in active use. Any active system must at all times be secured because the main approach of adversaries is to identify and try to exploit the weakest links. ## Defy Unintentional Weak Links Like adversaries don't care about fashion, so should the security industry. This will forever be a cat-and-mouse game. Therefore IT-operations benefits from a layered approach. The more layered the security provided the more likely the weakest link becomes less weak, a bit more hidden, a bit more obscure. However, just as equally futile as spending time writing management reports no one reads are security reports that are not acted upon. Who cares if one was aware of the vulnerability that was exploited? Unfortunately, the amount of work that needs to be done to stay secure always seems to exceed available resources. With AI still in its infancy and the "smartness" offered by security analytics tools not appearing to extend beyond some simple heuristic algorithms, using automation to ensure active enforcement of a security baseline can go a long way. With automation, asserting all machines to be compliant with PCI-DSS, (or any other compliance schema) becomes plausible. The extra work needed to improve security is minimal and only really requires tighter cooperation between operations and security teams. Knowing that all running RHEL machines whether the latest shiny version, or the forgotten corner boxes or cloud running instances, continuously comply with security baselines greatly improves the level of security in ways no other scanning system can provide. ## Dynamic Organizations Need Security Automation The Most A startup with a small homogenous infrastructure is vastly different in its security and operations needs than large heterogeneous infrastructures found in bigger companies. Each additional layer of complexity whether in form of new operating systems, middleware, application teams, environments (dev, test, prod, etc.) or new lines of businesses multiplies the security challenge. The more dynamic the organization, the more the need for automated security. The traditional mode of reading vulnerability reports and then manually acting upon them rapidly becomes impractical. For large organizations, the only true constant is change, and each change poses a new potential vulnerability. The more the number of people, the more the desire for manual system intervention ("I just need to…"), and the more lines of businesses with disparate architecture and technology stacks the more the complexity and potential for serious security gaps - due to the weakest link syndrome. Large organizations have thousands of applications, and millions of configuration settings. Add to this that speed wins and security comes second, one can easily argue for the need for security through automation. Security issues don't always emanate from within organizations. Acquisitions and outsourcing come as two major sources of security challenges. Once external consultants or systems gain internal access, or an acquired organization's IT operations merge, whole new sets of security gaps emerge. Again, automation provides a solid solution to ensure hygiene and basic defense. ## Automated Security with CFEngine CFEngine's autonomous architecture and self-healing capabilities fits the bill perfectly to play the role of an automated security mechanism. By defining and dividing your IT-infrastructure as smaller autonomous units, CFEngine will, thanks to its lightweight and cross-platform capabilities, continuously ensure security compliance, whether a system runs on an old RHEL box in a faraway corner, or new shiny RHEL instance on AWS. As long as security baseline policies exist for different software versions, and new versions rejected for provisioning before policies exist, CFEngine will meticulously do all the work. No manual intervention, or humongous reports to worry about. By making it mandatory to have CFEngine installed and running on any connected system an invaluable extra layer of security can be established. -------------------------------------------------------------------------------- title: "CFEngine 3.10.2 LTS and 3.11.0 non-LTS released!" url: /blog/2017/cfengine-3-10-2-lts-and-3-11-0-non-lts-released/index.md date: "2017-08-11" -------------------------------------------------------------------------------- We're happy to announce the maintenance release 3.10.2 LTS _and_ the new 3.11.0 non-LTS release today\! 3.10.2 LTS, being a maintenance (aka patch) release, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, the release primarily includes bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. 3.11.0 non-LTS includes new features and is not covered under the CFEngine Enterprise support contract. Looking at the [CFEngine release schedule](/product/supported-versions/), we can see that - **3.10** LTS is maintained (and supported for Enterprise customers) until **December 27th 2019** - **3.11** non-LTS is maintained (but unsupported for Enterprise customers) until **December 2017** If you are planning to [contribute features](https://github.com/cfengine/core/blob/master/HACKING.md) to the next feature release (thank you\!), please note that we would need the pull requests ready for merging by **early September** in order to have time to incorporate them into 3.12. ## Improvements in 3.10.2 LTS zypper is supported as a new package module for the [package promise implementation introduced in 3.7](https://cfengine.com/learn/managing-packages/), making package management with Zypper much more reliable. If you are currently using the zypper _package_method_ (old package promise implementation), you might consider switching to this new package module instead. Several issues around CFEngine start (init/systemd) scripts on multiple platforms have been fixed. In CFEngine Enterprise Mission Portal, there was an issue exporting csv files in some environments, which is now resolved. The inventory report also correctly parses lists in all cases, not displaying any curly braces in the report itself. As always, dependencies have been upgraded. Most notably, the CFEngine Enterprise 3.10.2 hub is shipped with Apache 2.4.27, PostgreSQL 9.6.3, PHP 5.6.31 and Redis 3.2.9. For more details please see the CFEngine 3.10.2 LTS ChangeLog for [core](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html). ## New functionality in 3.11.0 non-LTS #### Simplifying policy Several features have been introduced that make it faster and easier to write CFEngine policy. Most notably, the `with` attribute helps eliminate intermediate variables when looping. Here is an example: ```cf3 bundle agent main { vars: "todo" slist => { "a 1", "b 2", "c 3" }; # Here, `with` is the canonified version of $(todo), letting us avoid an # intermediate canonification array. "$(with)" string => "$(todo)", with => canonify($(todo)); "complex" data => ' { "x": 200, "y": [ 1, 2, null, true, false ] } '; reports: "For iterable '$(todo)' we created variable '$(with)' and its value is '$(todo)'" with => canonify($(todo)); "We can print a data container compactly without creating a temporary variable: $(with)" with => format("%S", complex); "We can print a data container fully without creating a temporary variable: $(with)" with => storejson(complex); } ``` ```example R: For iterable 'a 1' we created variable 'a_1' and its value is 'a 1' R: For iterable 'b 2' we created variable 'b_2' and its value is 'b 2' R: For iterable 'c 3' we created variable 'c_3' and its value is 'c 3' R: We can print a data container compactly without creating a temporary variable: {"x":200,"y":[1,2,null,true,false]} R: We can print a data container fully without creating a temporary variable: { "x": 200, "y": [ 1, 2, null, true, false ] } ``` #### Expanded system discovery CFEngine now leverages the standardized file `/etc/os-release` to do operating system discovery. This enables much broader OS discovery, for example the Amazon Linux and CoreOS platforms are now detected as classes and variables. You can use the `cf-promises` options `--show-classes` and `--show-vars` to see the new classes and variables on your favorite platform\! Environment files can now be read in natively into CFEngine, with the support of the new [readenvfile() function](https://docs.cfengine.com/docs/3.11/reference-functions-readenvfile.html), as well as ENV support in the [readdata() function](https://docs.cfengine.com/docs/3.11/reference-functions-readdata.html). #### Other changes Also in 3.11.0, zypper is supported as a new package module for the [package promise implementation introduced in 3.7](https://cfengine.com/learn/managing-packages/), making package management with Zypper much more reliable. If you are currently using the zypper _package_method_ (old package promise implementation), you might consider switching to this new package module instead. Previously, heavily loaded hubs that were handling multiple concurrent connections would occasionally reach the system resource limit of max open files, and manual tweaking was needed ( nofile limit in /etc/security/limits.conf ). In 3.11.0 (and 3.10.1) cf-serverd auto-configures this limit according to the maxconnections policy setting, so manual tweaking of the system limit should never be needed again. CFEngine Enterprise now has a global search box, allowing for hosts to be directly located by hostname or IP address from anywhere in the UI. Mission Portal's application logs have been relocated from `/var/cfengine/httpd/htdocs/application/logs` to `/var/cfengine/httpd/logs/application`. As usual, dependencies have been upgraded in order to get the latest security, performance and reliability improvements. For more details please see the CFEngine 3.11.0 non-LTS ChangeLog for [core](https://docs.cfengine.com/docs/3.11/guide-latest-release-whatsnew-changelog-core.html), [masterfiles](https://docs.cfengine.com/docs/3.11/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [enterprise](https://docs.cfengine.com/docs/3.11/guide-latest-release-whatsnew-changelog-enterprise.html) ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [3.10 upgrade documentation](/docs/3.10/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine customer support](https://cfengine.zendesk.com/hc/en-us) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the [CFEngine Enterprise 3.10 Vagrant environment](/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Discovering SSL Certificates and Expirations" url: /blog/2017/discovering-ssl-certificates-and-expirations/index.md date: "2017-06-19" --------------------------------------------------------------------------------
With more and more services using SSL keeping track of the certificates in use across a global infrastructure can be challenging. The inventory reporting features in CFEngine Enterprise can be useful in identifying services using SSL as well as when their certificates will expire. `cf-monitord` provides lists of ports that are listening. We can use `openssl `to connect to each listening port and if successful we can extract the certificate information for inventory. We won't be able to find ALL certificates like this. This policy only covers up-front SSL/TLS. From [Serverfault](https://serverfault.com/a/339383): > For [up-front SSL/TLS](https://stackoverflow.com/a/3661416/372643), > you can check whether it will accept a TLS ClientHello (i.e. be a TLS > server from the start of the connection), but using `echo "" | openssl > s_client -connect hostname:port (echo "" |` is optional, it will just > stop openssl as soon as it has established the connection, as you > probably don't want to send anything specific). For ["upgraded" > SSL/TLS connections](https://stackoverflow.com/a/3661416/372643), done > after a command at the application protocol level (such as > `STARTTLS`), this can be trickier. You can do this for by adding > `-starttls the_name_of_the_protocol` to this `openssl s_client` > command. According to the OpenSSL documentation, *"Currently, the only > supported \[protocol names\] are "smtp", "pop3", "imap", and "ftp"".* > This won't help you for LDAP (if configured to use Start TLS and not > up-front TLS), MySQL, PostgreSQL, … For these, you may simply have to > look into their respective configuration files. Automating this > process would require a tool that can understand all these protocols, > which can be quite difficult. Additionally, it will not discover [SNI certificates](https://en.wikipedia.org/wiki/Server_Name_Indication) because the probe does not provide the server name to in the probe. While there is no silver bullet this policy can aid in discovering a significant number of ssl certificates in use along with their expirations and it can be expanded with additional detection mechanisms as well. Some things we would like to collect: - Unix formatted date when certificate will expire - Human readable and lexically sort-able formatted date (ISO 8601) - Ports using SSL - Days until port certificate expires Additionally it would be nice to have a class defined if the number of days until the certificate expiration is less than some threshold. The threshold should be configurable without modifying policy.
Listing 1: `services/ssl_port_inventory/main.cf` ````cf3 bundle agent ssl_port_inventory # @brief Discover and inventory the ports and certificate expiration date where # SSL seems to be in use. # # **Example Augments:** # # ``` # { # "classes": { # "autorun_ssl_port_inventory": [ "any" ], # "services_autorun": [ "any" ] # }, # "inputs": [ "services/ssl_port_inventory/main.cf" ], # "vars": { # "ssl_port_inventory": { # "thresholds" { # "warning": { # "443": "90", # "8443": "90", # } # "critical": { # "5308": "365" # } # }} # } # } # ``` { meta: autorun_ssl_port_inventory:: "tags" slist => { "autorun" }; vars: "ports" slist => { @(mon.listening_ports) }; "regextract" string => "notBefore=(?\w+)\s+(?\d+)\s+\S+\s+(?\d+)\s+.*notAfter=(?\w+)\s+(?\d+)\s+\S+\s+(?\d+)\s+.*"; "timeout" string => ifelse( fileexists("/usr/bin/timeout"), "/usr/bin/timeout", "/bin/timeout" ); "now" string => strftime("gmtime", "%s", now() ); classes: "found_monitord_listening_ports" expression => isvariable( "mon.listening_tcp4_ports" ); found_monitord_listening_ports:: "port_$(ports)_ssl_connected" expression => returnszero("$(timeout) 1 /usr/bin/openssl s_client -connect localhost:$(ports) 2>/dev/null | $(timeout) 1 /usr/bin/openssl x509 -noout 2>/dev/null", "useshell"); vars: "port_$(ports)_ssl_connected":: # Example string we are parsing: # notBefore=Jun 14 15:54:23 2017 GMT\nnotAfter=Jun 12 15:54:23 2027 GMT "date_$(ports)" data => data_regextract( $(regextract), execresult("$(timeout) 1 /usr/bin/openssl s_client -connect 127.0.0.1:$(ports) 2>/dev/null | /usr/bin/openssl x509 -noout -dates 2>/dev/null", useshell ) ), unless => isvariable( $(this.promiser) ); # @var ssl_$(ports) The ports that have been found using SSL. **Example:** `[443, 5308]` "ssl_$(ports)" string => "$(ports)", meta => { "inventory", "attribute_name=SSL Ports"}, if => isvariable( "data_$(ports)" ), unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_expiration_unix Date in unix format. **Example:** `1497715408` "inventory_$(ports)_ssl_expiration_unix" string => execresult("/bin/date -d'$(date_$(ports)[end_M]) $(date_$(ports)[end_D]) $(date_$(ports)[end_Y])' +%s", useshell), meta => { "inventory", "attribute_name=Port $(ports) SSL Expires UNIX" }, unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_expiration_iso_8601 Date in ISO 8601 format (YYYY-MM-DD). **Example:** `2017-06-17` "inventory_$(ports)_ssl_expiration_iso_8601" string => execresult("/bin/date -d'$(date_$(ports)[end_M]) $(date_$(ports)[end_D]) $(date_$(ports)[end_Y])' +%Y-%m-%d", useshell), meta => { "inventory", "attribute_name=Port $(ports) SSL Expires iso 8601" }, unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_days_until_expiration the number of days until the certificate on `port` expires. **Example:** `05` "inventory_$(ports)_ssl_days_until_expiration" string => format( "%d", eval("($(inventory_$(ports)_ssl_expiration_unix)-$(now))/86400", math, infix)), meta => { "inventory", "attribute_name=Days until $(ports) SSL Expires" }, unless => isvariable( $(this.promiser) ); # { "vars": { "ssl_port_inventory": { "thresholds": "warning": { "443": "30" }}}} # $(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)]) "thresholds" slist => getindices( "def.$(this.bundle)[thresholds]" ); classes: # @class ssl_port_inventory_port_$(ports)_threshold_$(thresholds) is defined if inventory_$(ports)_ssl_days_until_expiration < def.ssl_port_inventory[$(thresholds)][$(ports)] "$(this.bundle)_port_$(ports)_threshold_$(thresholds)" meta => { "$(this.bundle)_threshold", "report" }, expression => islessthan( "$(inventory_$(ports)_ssl_days_until_expiration)", "$(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)])" ); vars: "_threshold_c" slist => classesmatching( ".*", "$(this.bundle)_threshold" ); reports: !found_monitord_listening_ports:: "ERROR: Unable to find open ports $(this.bundle) requires cf-monitord to be running."; DEBUG|DEBUG_ssl_port_inventory:: "Found threshold definition: Port $(ports) $(thresholds) if there are less than $(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)]) days until the certificate expires"; "Detected $(ports) is using SSL because of successful connection using openssl s_client" if => "port_$(ports)_ssl_connected"; "Certificate on port $(ports) expires on $(inventory_$(ports)_ssl_expiration_iso_8601) in $(inventory_$(ports)_ssl_days_until_expiration) days"; "Threshold Class: $(_threshold_c)"; } ````
Going through the specifics of the policy: The policy is tagged for `autorun` if `autorun_ssl_port_inventory` is defined.
Listing 2: Autorun capable policy ```cf3 meta: autorun_ssl_port_inventory:: "tags" slist => { "autorun" }; ```
Next we define some variables for re-use. Notably we get the list of ports we should be monitoring from `mon.listening_ports` which is provided by `cf-monitord`. If `cf-monitord` isn't running then the agent will not have any ports to monitor. We also define `regextract` as our pattern to parse the result of our probe with `openssl` later, `timeout` as the path to the gnu `timeout `utility to prevent commands from getting stuck and the current time in unix format for use in later comparisons.
Listing 3: Define variables for re-use ```cf3 vars: "ports" slist => { @(mon.listening_ports) }; "regextract" string => "notBefore=(?\w+)\s+(?\d+)\s+\S+\s+(?\d+)\s+.*notAfter=(?\w+)\s+(?\d+)\s+\S+\s+(?\d+)\s+.*"; "timeout" string => ifelse( fileexists("/usr/bin/timeout"), "/usr/bin/timeout", "/bin/timeout" ); "now" string => strftime("gmtime", "%s", now() ); ```
Here we define a class if we have found the ports listening provided by `cf-monitord` and if we have ports we probe them with `openssl s_client -connect` and define classes if they appear to be speaking SSL.
Listing 4: Determine which ports are SSL enabled ```cf3 classes: "found_monitord_listening_ports" expression => isvariable( "mon.listening_tcp4_ports" ); found_monitord_listening_ports:: "port_$(ports)_ssl_connected" expression => returnszero("$(timeout) 1 /usr/bin/openssl s_client -connect localhost:$(ports) 2>/dev/null | $(timeout) 1 /usr/bin/openssl x509 -noout 2>/dev/null", "useshell"); ```
When we know which ports are speaking ssl we begin to extract the data provided by our probe.
Listing 5: Parse certificate information for SSL ports ```cf3 vars: "port_$(ports)_ssl_connected":: # Example string we are parsing: # notBefore=Jun 14 15:54:23 2017 GMT\nnotAfter=Jun 12 15:54:23 2027 GMT "date_$(ports)" data => data_regextract( $(regextract), execresult("$(timeout) 1 /usr/bin/openssl s_client -connect 127.0.0.1:$(ports) 2>/dev/null | /usr/bin/openssl x509 -noout -dates 2>/dev/null", useshell ) ), unless => isvariable( $(this.promiser) ); ```
We inventory each port that speaks SSL and then we inventory the expiration dates for each port in our desired formats.
Listing 6: Inventory SSL Ports and expiration dates in multiple formats ```cf3 # @var ssl_$(ports) The ports that have been found using SSL. **Example:** `[443, 5308]` "ssl_$(ports)" string => "$(ports)", meta => { "inventory", "attribute_name=SSL Ports"}, if => isvariable( "data_$(ports)" ), unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_expiration_unix Date in unix format. **Example:** `1497715408` "inventory_$(ports)_ssl_expiration_unix" string => execresult("/bin/date -d'$(date_$(ports)[end_M]) $(date_$(ports)[end_D]) $(date_$(ports)[end_Y])' +%s", useshell), meta => { "inventory", "attribute_name=Port $(ports) SSL Expires UNIX" }, unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_expiration_iso_8601 Date in ISO 8601 format (YYYY-MM-DD). **Example:** `2017-06-17` "inventory_$(ports)_ssl_expiration_iso_8601" string => execresult("/bin/date -d'$(date_$(ports)[end_M]) $(date_$(ports)[end_D]) $(date_$(ports)[end_Y])' +%Y-%m-%d", useshell), meta => { "inventory", "attribute_name=Port $(ports) SSL Expires iso 8601" }, unless => isvariable( $(this.promiser) ); # @var inventory_$(ports)_ssl_days_until_expiration the number of days until the certificate on `port` expires. **Example:** `05` "inventory_$(ports)_ssl_days_until_expiration" string => format( "%d", eval("($(inventory_$(ports)_ssl_expiration_unix)-$(now))/86400", math, infix)), meta => { "inventory", "attribute_name=Days until $(ports) SSL Expires" }, unless => isvariable( $(this.promiser) ); ```
Next we define classes for any tripped thresholds as defined in `def.ssl_port_inventory` data structure in the `Example Augments` section of the file.
Listing 7: Define threshold based classes ```cf3 # { "vars": { "ssl_port_inventory": { "thresholds": "warning": { "443": "30" }}}} # $(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)]) "thresholds" slist => getindices( "def.$(this.bundle)[thresholds]" ); classes: # @class ssl_port_inventory_port_$(ports)_threshold_$(thresholds) is defined if inventory_$(ports)_ssl_days_until_expiration < def.ssl_port_inventory[$(thresholds)][$(ports)] "$(this.bundle)_port_$(ports)_threshold_$(thresholds)" meta => { "$(this.bundle)_threshold", "report" }, expression => islessthan( "$(inventory_$(ports)_ssl_days_until_expiration)", "$(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)])" ); ```
And finally we collect information about any threshold classes defined and report information if relevant.
Listing 8: Report the results ```cf3 vars: "_threshold_c" slist => classesmatching( ".*", "$(this.bundle)_threshold" ); reports: !found_monitord_listening_ports:: "ERROR: Unable to find open ports $(this.bundle) requires cf-monitord to be running."; DEBUG|DEBUG_ssl_port_inventory:: "Found threshold definition: Port $(ports) $(thresholds) if there are less than $(def.ssl_port_inventory[thresholds][$(thresholds)][$(ports)]) days until the certificate expires"; "Detected $(ports) is using SSL because of successful connection using openssl s_client" if => "port_$(ports)_ssl_connected"; "Certificate on port $(ports) expires on $(inventory_$(ports)_ssl_expiration_iso_8601) in $(inventory_$(ports)_ssl_days_until_expiration) days"; "Threshold Class: $(_threshold_c)"; ```
We can then report on the SSL information inventoried and build dashboards with alerts based on the thresholds. [![](images/Mission-Portal-Dashboard-2-800x500.png)](images/Mission-Portal-Dashboard-2.png) [![](images/Mission-Portal-Dashboard-3-800x428.png)](images/Mission-Portal-Dashboard-3.png) [![](images/Mission-Portal-Dashboard-1-800x432.png)](images/Mission-Portal-Dashboard-1.png) [![](images/Mission-Portal-SQL-Reports-1-800x607.png)](images/Mission-Portal-SQL-Reports-1.png)
-------------------------------------------------------------------------------- title: "3.11.0 Beta ready for testing!" url: /blog/2017/3-11-0-beta-ready-for-testing/index.md date: "2017-06-14" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.11.0 Beta (non-LTS) is now ready. Thanks to everyone for all of the contributions\! Please test extensively and submit bug reports. 3.11.x introduces some new features and deprecates some underutilized functionality. Please note that 3.11.0 will be a non-LTS release, which means that it will be maintained only for 6 months from the release date and not supported for CFEngine Enterprise customers (but Enterprise packages will be available). Looking at the [CFEngine release schedule](https://cfengine.com/product/supported-versions/), we can see: - **3.7** LTS is maintained (and supported for Enterprise customers) until **July 17th 2018** - **3.10** LTS is maintained (and supported for Enterprise customers) until **December 27th 2019** - **3.11** non-LTS will be maintained for 6 months after the release of 3.11.0. The [`guest_environments` promise type](https://docs.cfengine.com/docs/3.11/reference-promise-types-guest_environments.html) has been removed and the associated `libvirt` dependency has been dropped. Support for postgres in [`database` type promises](https://docs.cfengine.com/docs/3.11/reference-promise-types-databases.html) has been removed and the associated agent dependency has also been removed. Additionally please note that upgrades are unsupported for the BETA release. The new `with` attribute that allows the use of function calls on a promiser ([CFE-1092](https://northerntech.atlassian.net/browse/CFE-1092)). This feature can help in simplifying policy by removing the need for intermediary variables.
Listing 1: Example usage of `with` attribute. ```cf3 bundle agent main { vars: "todo" slist => { "a 1", "b 2", "c 3" }; # Here, `with` is the canonified version of $(todo), letting us avoid an # intermediate canonification array. "$(with)" string => "$(todo)", with => canonify($(todo)); "complex" data => ' { "x": 200, "y": [ 1, 2, null, true, false ] } '; reports: "For iterable '$(todo)' we created variable '$(with)' and its value is '$(todo)'" with => canonify($(todo)); "We can print a data container compactly without creating a temporary variable: $(with)" with => format("%S", complex); "We can print a data container fully without creating a temporary variable: $(with)" with => storejson(complex); } ```
``` example R: For iterable 'a 1' we created variable 'a_1' and its value is 'a 1' R: For iterable 'b 2' we created variable 'b_2' and its value is 'b 2' R: For iterable 'c 3' we created variable 'c_3' and its value is 'c 3' R: We can print a data container compactly without creating a temporary variable: {"x":200,"y":[1,2,null,true,false]} R: We can print a data container fully without creating a temporary variable: { "x": 200, "y": [ 1, 2, null, true, false ] } ``` System discovery now leverages `/etc/os-release` and populates `sys.os_release `with the parsed data. Amazon Linux and CoreOS are now automatically detected during system discovery. Now, when running on Amazon Linux the `AmazonLinux `class will be defined and `sys.flavour` will be set to `AmazonLinux` and when running on CoreOS the `coreos` class is defined and `sys.flavor` is set to `coreos_VERSION` where VERSION is derived from `/etc/os-release`. For example `coreos_1185.3.0` ([ENT-3046](https://northerntech.atlassian.net/browse/ENT-3046), [CFE-1881](https://northerntech.atlassian.net/browse/CFE-1881)). The new functions `sysctlvalue()` and `data_sysctlvalues()` for extracting data from `/proc/sys` are available on `Linux` systems ([CFE-2513](https://northerntech.atlassian.net/browse/CFE-2513)). The new `readenvfile()` function parses environment files ( `KEY = VALUE`, or `KEY = "VALUE"` ). The `readdata()` function can now parse `ENV` files now detects files ending in `.yml` as `YAML` and files ending in `.env` as `ENV `([CFE-1881](https://northerntech.atlassian.net/browse/CFE-1881)). `cf-agent` gained the `--show-evaluated-vars` and `--show-evaluated-classes `options which works similarly to `cf-promises --show-vars` and `cf-promises --show-classes`. The difference being that it is displayed at the end of the agent run and thus has access to classes and variables that were defined during full policy evaluation instead of only those available after system discovery and pre-evaluation. Additionally `cf-promises` -`-show-vars` and `show-classes `options can now take an optional regular expression used to filter the output. A new tool `cf-net` has been added. It is a CLI tool targeted at testing and debugging the network protocol ([CFE-2493](https://northerntech.atlassian.net/browse/CFE-2493)). In the MPF the default update policy has been made more resilient against partial or corrupt updates. You can now [enable additional verification on files transferred during policy updates](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#verify-update-transfers) by defining the `cfengine_internal_verify_update_transfers` class. Additionally if there is a problem transferring any policy file the `cf_promises_validated` marker is cleared in order to ensure a full update scan on the next agent execution ([CFE-2587](https://northerntech.atlassian.net/browse/CFE-2587)). The self upgrade policy has been re-factored into a standalone policy entry (`standalone_self_upgrade.cf`). Agents run this standalone policy as a sub agent. On `systemd` managed hosts the run is executed via `systemd-run` in order to allow for the package upgrade to not be terminated early when the service unit shuts down ([ENT-3155](https://northerntech.atlassian.net/browse/ENT-3155)). The `files_single_copy` agent control can now be specified via augments ([CFE-2458](https://northerntech.atlassian.net/browse/CFE-2458)). The [`bundlesequence` for the main policy (`promises.cf`) can now be augmented by defining the variable `def.control_common_bundlesequence_end`](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#append-to-the-main-bundlesequence). The `apt_get` `package_module` now works with both python 2 and 3 ([CFE-2445](https://northerntech.atlassian.net/browse/CFE-2445)) and is now version aware. For `apt-get` versions prior to 1.1 `--force-yes` is used and in versions 1.1 and later is replaced by `--allow-downgrades`, `--allow=remove-essential`, and `--allow-change-held-packages` ([CFE-2360](https://northerntech.atlassian.net/browse/CFE-2360)). Additionally a bug preventing package updates when multiple sources are available ([CFE-2605](https://northerntech.atlassian.net/browse/CFE-2605)). Fix `apt_get` `package_module` incorrectly using interactive mode when a configuration file is created before a package is installed by always using the `--force-confold` and `--force-confdef` options Fix bug which prevented updates from being picked up if there was more than one source listed in the `apt upgrade` output, without a comma in between ([CFE-2605](https://northerntech.atlassian.net/browse/CFE-2605)). `oslevel` inventory has been added for AIX hosts ([ENT-3117](https://northerntech.atlassian.net/browse/ENT-3117)) and `package_module `based package inventory is disabled on old distributions with unsupported python versions (e.g. `redhat_4`) ([CFE-2602](https://northerntech.atlassian.net/browse/CFE-2602)). A new class [`client_initated_reporting_enable`](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#enable-client-initiated-reporting) ([CFE-2623](https://northerntech.atlassian.net/browse/CFE-2623)) introduced along with the new variable [`def.mpf_access_rules_collect_calls_admit_ips`](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#configure-networks-allowed-to-make-collect_calls-client-initiated-reporting) ([CFE-2624](https://northerntech.atlassian.net/browse/CFE-2624)), and `def.control_hub_exclude_hosts` controlling the [networks for which the hub should not initiate report collection](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#exclude-hosts-from-hub-initiated-report-collection) ([CFE-2622](https://northerntech.atlassian.net/browse/CFE-2622)) in order to make it easier to enable and configure client initiated reporting in CFEngine Enterprise without modifying policy. New variables `def.default_data_select_host_monitoring_include` and `def.default_data_select_policy_hub_monitoring_include` ([CFE-2505](https://northerntech.atlassian.net/browse/CFE-2505)) were introduced to make it easier to [specify which measurements taken by `cf-monitord` should be collected for central reporting](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#configure-enterprise-measurement-monitoring-collection). [The `default_repository` can now be configured](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#configure-default-repository-for-file-backups) by defining the variable `def.control_agent_default_repository` when the `mpf_control_agent_default_repository` class is defined. The new variable `def.max_client_history_size` was introduced to allow [controlling the amount of client side reports waiting to be collected](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework.html#adjust-the-maximum-amount-of-client-side-report-data-to-retain-cfengine-enterprise) into the CFEngine Enterprise hub that are allowed to build up before being purged. Additionally `previous_state` and `untracked_reports` are now included in the purge when `def.max_client_history_size` threshold is reached ([ENT-3161](https://northerntech.atlassian.net/browse/ENT-3161)). Promises with handles matching the regular expression `noreport_.*` are no longer collected for central reporting in CFEngine Enterprise by default ([ENT-3061](https://northerntech.atlassian.net/browse/ENT-3061)). The [`prunetree` agent bundle](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework-lib-bundles.html#prunetree) has been added to the standard library ([CFE-2608](https://northerntech.atlassian.net/browse/CFE-2608)) complementing the existing [`prunedir` bundle](https://docs.cfengine.com/docs/3.11/reference-masterfiles-policy-framework-lib-bundles.html#prunedir). The ability to select an INI region containing meta characters has been fixed ([CFE-2519](https://northerntech.atlassian.net/browse/CFE-2519)). For example this fixes the ability to manage hidden smb shares like `[hidden$]`. `select_region INI_section` has also been fixed so that the end of a file is sufficient to match the end of the section ([CFE-2519](https://northerntech.atlassian.net/browse/CFE-2519)). A new **shortcut** for `templates` was added ([CFE-2582](https://northerntech.atlassian.net/browse/CFE-2582)). The `set_config_values()` and `set_line_based()` `edit_line` bundles behavior changed. Now when inserting a configuration option with an empty value a trailing space is no longer inserted ([CFE-2466](https://northerntech.atlassian.net/browse/CFE-2466)). `systemd` services are now only "re-started" if the service is already running ([CFE-2541](https://northerntech.atlassian.net/browse/CFE-2541)). If the service is NOT running, it will simply be started. The `paths` bundle in the standard library now uses the path to POSIX compliant tools by default. The behavior can be disabled by defining the class `disable_mpf_stdlib_use_posix_utils` ([CFE-2616](https://northerntech.atlassian.net/browse/CFE-2616)). Directories for `ubuntu_16` and `centos_7` are now automatically created on Enterprise hubs ([ENT-3136](https://northerntech.atlassian.net/browse/ENT-3136)). The default files for which detailed change monitoring is tracked are now only monitored if the files actually exist ([ENT-3040](https://northerntech.atlassian.net/browse/ENT-3040)). `cf-serverd` now listens on both IPv4 and IPv6 interfaces by default ([CFE-528](https://northerntech.atlassian.net/browse/CFE-528)). A new `package_module` for `zypper` was introduced ([CFE-2533](https://northerntech.atlassian.net/browse/CFE-2533)). In addition to the improvements in core CFEngine Enterprise has seen the addition of a global search box allowing for hosts to be easily located by hostname or IP address ([ENT-3059](https://northerntech.atlassian.net/browse/ENT-3059)). Host info pages now include a button to copy the url for a direct link to the currently viewed host ([ENT-3096](https://northerntech.atlassian.net/browse/ENT-3096)) and hostnames shown the host tree summary reports now link to the host specific info page ([ENT-3094](https://northerntech.atlassian.net/browse/ENT-3094)). The Changes report now defaults to "todays" changes instead of changes from the last 7 days ([ENT-2840](https://northerntech.atlassian.net/browse/ENT-2840)). Exporting of CSV reports and access to reports shared with users having multiple roles has been fixed ([ENT-3099](https://northerntech.atlassian.net/browse/ENT-3099)). The query API has been optimized ([ENT-2829](https://northerntech.atlassian.net/browse/ENT-2829)) and the sorting of columns from an exported inventory report are now retained ([ENT-614](https://northerntech.atlassian.net/browse/ENT-614)). A new binary tarball package was introduced making it easier to test and use cfengine on platforms without traditional native package managers. Mission Portal's application logs have been relocated from `/var/cfengine/httpd/htdocs/application/logs` to `/var/cfengine/httpd/logs/application` ([ENT-2758](https://northerntech.atlassian.net/browse/ENT-2758)). The following dependencies have been upgraded: - Update redis from 3.2.6 to 3.2.8 - Update git from 2.10.2 to 2.13.0 - Update autoconf from 2.60 to 2.69 - Update postgresql for hub from 9.6.1 to 9.6.2 - Update curl from 7.50.3 to 7.53.1 - Update libiconv from 1.14 to 1.15 - Update pcre from 8.39 to 8.40 - Upgrade to LMDB 0.9.19 - Update postgresql from 9.0.22 to 9.0.23 - Update libcurl to 7.54.0 - Update php from 5.6.29 to 5.6.30 - Update lcov from 1.10 to 1.13 -------------------------------------------------------------------------------- title: "CFEngine 3.7.5 LTS and 3.10.1 LTS released!" url: /blog/2017/cfengine-3-7-5-lts-and-3-10-1-lts-released/index.md date: "2017-03-30" -------------------------------------------------------------------------------- We're happy to announce maintenance releases for both supported CFEngine release branches today\! Being maintenance (aka patch) releases, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, the releases primarily include bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](/product/supported-versions/), we can see that - **3.7** LTS is maintained (and supported for Enterprise customers) until **July 17th 2018** - **3.9** non-LTS is **no longer maintained** - **3.10** LTS is maintained (and supported for Enterprise customers) until **December 27th 2019** If you are planning to [contribute features](https://github.com/cfengine/core/blob/master/HACKING.md) to the next feature release (thank you\!), please note that we would need the pull requests ready for merging by **early-April** in order to have time to incorporate them into 3.11. If you are planning to contribute fixes to 3.10 LTS please note that we would need the pull requests ready for merging by **early-May** in order to have time to incorporate them into 3.10.2 LTS. ## Changes and improvements In 3.10.1 LTS detection of Amazon Linux and CoreOS has been added and cf-serverd now automatically adjusts the number of open files based on maxconnections. 3.7.5 LTS received a fix to cf-agent's connection caching. This should greatly reduce the hub's load, but all clients should be upgraded to the new version for the full benefit to take effect. cf-serverd has also been fixed so that it re-reads the augments file (def.json) when automatically re-loading policy. In masterfiles the resilience of the default update policy has been improved. Previously an agent would not re-scan for updates if a policy update was not fully completed, but did not result in invalid policy. Hosts would wait until the next policy update before scanning for changes. Now the cf_promises_validated flag is cleared if there is any error while updating inputs. In order to align the systemd and systemv init behavior in the standard_services bundle services are started if they were not already running when policy restart is requested. The apt_get package module now automatically detects the correct python version (so it works out of the box on newer Ubuntu hosts) and is now version aware and now only uses --force-yes for apt-get versions 1.1 and earlier so that all versions of debian or Ubuntu work, from as far back as debian 4 up to recent releases. #### Enterprise edition specific changes - Solaris 10 x86 has been added as a supported platform. - Custom measurement promises no longer cause errors to be logged from cf-monitord and parallel plan execution is now enabled by default in the postgres config. #### Dependency upgrades The bundled dependencies have been upgraded to bring in the latest security, performance and reliability improvements. Notably both 3.7.5 LTS and 3.10.1 LTS are bundled with PCRE 8.40 and OpenSSL 1.0.2k. On the Enterprise server, both releases come with PHP 5.6.30, while the Apache web server is at version 2.2.32 in CFEngine 3.7.5 LTS and 2.4.25 in CFEngine 3.10.1 LTS. For more details on the improvements in the releases, please see the relevant ChangeLogs: - 3.7.5 LTS ChangeLog: [core](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html) and [enterprise](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html) - 3.10.1 LTS ChangeLog: [core](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html), [masterfiles](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-masterfiles-policy-framework.html), and [enterprise](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html) ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](/docs/3.10/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine customer support](https://cfengine.zendesk.com/hc/en-us) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can take a quick spin with the [CFEngine Enterprise 3.7 Vagrant environment](/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) or [CFEngine Enterprise 3.10 Vagrant environment](/docs/3.10/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Config Management Camp Gent 2017: Recap" url: /blog/2017/config-management-camp-gent-2017-recap/index.md date: "2017-02-17" -------------------------------------------------------------------------------- Last week we attended [Config Management Camp](https://cfgmgmtcamp.eu/) in Gent Belgium. ![](images/IMG_20170207_110501-2.jpg) How did automation start? In the beginning Mark split light from dark... There were 7 talks in the CFEngine devroom. Christian Linden kicked things off with his presentation "[Get set for getting work done by CFEngine](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/cfengine-kick-off-christian-linden)". He covered the basics of what you need to know when getting started with CFEngine as well as some tips to help in debugging policies. After Christians Kickoff I talked about "[Testing CFEngine Policy](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/testing-cfengine-policy-nick-anderson)". Why you should test, currently available testing frameworks, how to get started writing tests that produce TAP and or JUnit output and finally how to expand CFEngines test coverage while contributing to examples in the documentation. Martin Simons wrapped up the first day with his talk "[CFEngine Hero of IoT](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/cfengine-hero-of-iot-martin-simons)". Martin brought a Raspberry Pi running Raspbian told us how he is using it to implement a motion-activated security system built with commodity hardware at home and then demonstrated how CFEngine can easily manage it and deployed a tomcat application. The main topic of conversation on Tuesday was data. How to separate it from policy, how to make use of external data, and how it can make your life easier in [SURFsara](https://en.wikipedia.org/wiki/SURFsara). Bas van der Vlies started us off with his talk "[How to use the augments file (def.json)](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/augments-def-json-bas-van-der-vlies)". He showed how he defined a host specific data file that let him easily override default values for a specific service/bundle. ![](images/bas_cfengine_room-IMG_1224-150x150.jpg)He discussed how this has had great benefits for non CFEngine users in his environment, allowing them to easily identify which service/bundles are activated for a host and exceptions that each host is overriding. Next Jurica Borozan talked about "[Merging Technologies, ideas on using CFEngine with cloud and container technologies](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/merging-tech-cfengine-with-cloud-and-containers-jurica-borozan)". He discussed how his tag based classification system enabled host specific policies to be delivered and the varying ways that this common model could be applied to physical systems, docker containers, and cloud hosts leveraging the features available in each technology stack. ![](images/jurica_cfengine_room-IMG_1230-150x150.jpg)He finished up by demonstrating his policy that managed maintaining a specific number of AWS instances for a given image and how a simple change in his data source (a json file) resulted in nodes being provisioned or destroyed within 5 minutes. Neil Watson gave a short remote talk "[CFEngine Simplified with EFL](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/cfengine-simplified-with-efl-neil-watson)". He discussed the design philosophy of EFL and how you can leverage the framework to get things accomplished with CFEngine without having to know any policy. I wrapped up talks with a short presentation on "[The Masterfiles policy Framework: A short history and future direction](https://cfgmgmtcamp.org/ghent2017/schedule/cfengine/masterfiles-policy-framework-history-and-future-nick-anderson)". I presented the lineage of the current policy framework, how we got to where we are today and how data (the augments file def.json) is very much at the core of how we expect to improve usability and ease future policy framework upgrades without having to edit policy. ![](images/cfengine_table-IMG_1236-800x600.jpg) Special thanks to Christian, Martin, Bas, and Neil for sharing your thoughts and presentations. We look forward to seeing everyone again next year\! -------------------------------------------------------------------------------- title: "CFEngine 3.10.0 LTS released!" url: /blog/2016/cfengine-3-10-lts-released/index.md date: "2016-12-28" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.10.0 LTS now is released\! One of the key advantages of CFEngine is its unparalleled performance, and this has received further attention in 3.10.0\! A big thanks to everyone testing the [3.10.0 beta release](https://cfengine.com/company/blog-detail/cfengine-3-10-0-lts-beta-ready-for-testing)\! Being an **LTS** release, 3.10 will be supported until December 2019. If you are using the previous LTS, 3.7, you will also benefit from all the new features, improvements and testing of all 3.8 and 3.9 releases, which you can read more about in the CFEngine [3.8.0](https://cfengine.com/company/blog-detail/cfengine-3-8-0-non-lts-released/) and [3.9.0](https://cfengine.com/company/blog-detail/cfengine-3-9-0-non-lts-released/) release posts. ## Double query speed for Enterprise API Enterprise hubs running at scale in the thousands of clients sometimes took quite a few seconds to respond to complex API queries. One of the reasons for this taking a long time is the need for finding the number of results returned along with the actual results themselves. This number is used by the Mission Portal and API clients in order to split the result up into pages so that not the whole data set needs to be returned at the same time. In previous versions of CFEngine Enterprise, this was done by first issuing a SQL COUNT query before the actual data query. However, utilizing some smart logic in PostrgreSQL, CFEngine 3.10 has reduced these two queries into one. This means that you can expect all queries, including Mission Portal reports, alerts and the Inventory reporting to return **twice as fast** as previous versions of CFEngine\! ## New variable expansion engine A high-demand improvement included in 3.10.0 relates to [speeding up variable expansion over large data structures](https://northerntech.atlassian.net/browse/CFE-2162). This means that working with large JSON-files or nested "classic" arrays is now much more efficient. We can let the numbers speak for themselves. In this simple test, we are using a JSON file with about 2 KB of size ([locations.json](images/locations.json_.gz) - placed in `/tmp`), together with a policy that deeply iterates over it in order to define classes based on subnets ([ipranges.cf](images/ipranges.cf_.gz)). To test the performance we wrap cf-agent in time: `/usr/bin/time -f "Elapsed (sec) %e\nMaxRSS (KB) %M" cf-agent -K ipranges.cf` The run with CFEngine 3.7.4 yields the following: R: CFEngine Version 3.7.4 Elapsed (sec) **59.34** MaxRSS (KB) 32960 While CFEngine 3.10.0 gives: R: CFEngine Version 3.10.0b1 Elapsed (sec) **0.08** MaxRSS (KB) 32224 The difference is almost hard to believe; in this case CFEngine 3.10 is **750 times faster** than 3.7, while the consumed memory is about the same. What is more, the performance improvement in 3.10 will be even larger as the size of the data grows\! Please note that in order to make this vast improvement possible, it was necessary to replace the variable expansion engine in CFEngine. We have been very careful to ensure that policy is backward-compatible in 3.10, including running our some 1500 policy-based acceptance tests (which all pass), and [Neil Watson](https://ca.linkedin.com/in/neilhwatson) tells us the [EFL](https://github.com/neilhwatson/evolve_cfengine_freelib) acceptance test (which all pass). The 3.10.0 beta did not expose any serious incompatibility issues, but other minor issues have been addressed in 3.10.0. One notable change in order of operations is that an additional pass resolving vars was added in pre-evaluation. Now pre-evaluation does vars, classes, vars; previously it was only classes, vars. ## Expanded Ubuntu and Solaris support We are happy to announce that 3.10 has official support for the latest LTS releases from Canonical: **Ubuntu 14.04 LTS** and **Ubuntu 16.04 LTS**. This includes both host and Enterprise hub support on these platforms\! In addition, CFEngine Enterprise customers can expect official support for **Solaris 10 x86** in 3.10. There is no package for Solaris 10 x86 yet, but we will make a package available as soon as it has passed internal testing. You can find the list of currently supported platforms [here](https://docs.cfengine.com/docs/3.10/guide-latest-release-supported-platforms.html). ## systemd unit improvements The systemd support has received a major overhaul in CFEngine 3.10. All CFEngine services, like cf-serverd, cf-hub, cf-execd, now have their own independent systemd unit. This means that systemd will manage them independently, including monitor and restart them immediately if they stop for any reason. Please note _currently_ systemctl stop cfengine3 will stop all other CFEngine services. It returns immediately and systemd takes care of bringing down the other services. ## Bootstrap to hostname and alternative port 3.10 also introduces the ability to bootstrap to a hostname and specify the port used by cf-agent to fetch policy (e.g. `cf-agent --bootstrap myhub.cfengine.com:5309)`. This information is written to `$(sys.workdir)/policy_server.dat`. The new variable `sys.policy_hub_port` will contain the port to use. Prior to 3.10 the IP address of the host you bootstrapped to was recorded and used as the source for `$(sys.policy_hub)`. Now the hostname itself is stored so that agents can seamlessly follow a hub when its IP changes. This integrates seamlessly with any existing policy because the `sys.policy_hub` variable is still an IP address, but now resolved via DNS at the start of the agent run instead of being resolved at bootstrap time. ## Other improvements For a full list of the improvements in 3.10.0, please see the [Community ChangeLog](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise ChangeLog](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html). You will also note that all the [Known issues in 3.10.0 beta](https://cfengine.com/company/blog-detail/cfengine-3-10-0-lts-beta-ready-for-testing/#known-issues) are fixed. ## Get it\! You can download the CFEngine [community 3.10.0 packages and source code](https://cfengine.com/product/community/). If you are an Enterprise customer, the Enterprise packages [can be downloaded here](https://cfengine.com/download/). We hope you enjoy 3.10.0, and we look forward to hearing about your experience in the [CFEngine Mailing List](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Community Advisory Board Q4 2016" url: /blog/2016/cab-q4-2016/index.md date: "2016-11-17" -------------------------------------------------------------------------------- The notes from the Q4 2016 meeting can be found in the [github repository](https://github.com/cfengine/cfengine-community-advisory-board/blob/master/minutes/2016-11-17.org). This meeting marks 2 years for the Community Advisory Board. As a reminder the current board members include: - [Bas van der Vlies](https://www.linkedin.com/in/basvandervlies) - [Jonathan Clarke](https://www.linkedin.com/in/clarkejonathan) - [Mike Svoboda](https://www.linkedin.com/in/mikesvoboda) - [Ted Zlatanov](https://www.linkedin.com/in/teodorzlatanov) - [Neil Watson](https://ca.linkedin.com/in/neilhwatson) - [Eystein Stenberg](https://www.linkedin.com/in/eysteinstenberg) - [Nick Anderson](https://www.linkedin.com/in/hithisisnick) The general consensus was that the meetings have been valuable and we should continue the efforts. We have decided to pilot having the next advisory board meeting in `#cfengine` instead of `#cfengine-cab` in order to increase community awareness and get additional input from other users with different perspectives. As always if you have input or would like the Community Advisory Board to consider a topic please consider opening a pull request to the [topics](https://github.com/cfengine/cfengine-community-advisory-board/blob/master/topics.org), emailing or by reaching out directly to one of the board members. -------------------------------------------------------------------------------- title: "CFEngine 3.10.0 LTS beta ready for testing!" url: /blog/2016/cfengine-3-10-0-lts-beta-ready-for-testing/index.md date: "2016-11-08" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.10.0 LTS beta is now ready for testing\! The established [CFEngine release schedule](/product/supported-versions/) shows that the 3.10.0 final version is due in December 2016, so it is time to test and fix any remaining issues\! Being an **LTS** release, 3.10 will be supported until December 2019. ## New variable expansion engine A high-demand improvement included in the 3.10 beta relates to [speeding up variable expansion over large data structures](https://northerntech.atlassian.net/browse/CFE-2162). This means that working with large JSON-files or nested "classic" arrays is now much more efficient. We can let the numbers speak for themselves. In this simple test, we are using a JSON file with about 2 KB of size ([locations.json](images/locations.json_.gz) - placed in `/tmp`), together with a policy that deeply iterates over it in order to define classes based on subnets ([ipranges.cf](images/ipranges.cf_.gz)). To test the performance we wrap cf-agent in time: `/usr/bin/time -f "Elapsed (sec) %e\nMaxRSS (KB) %M" cf-agent -K ipranges.cf` The run with CFEngine 3.7.4 yields the following: R: CFEngine Version 3.7.4 Elapsed (sec) **59.34** MaxRSS (KB) 32960 While CFEngine 3.10.0 beta gives: R: CFEngine Version 3.10.0b1 Elapsed (sec) **0.08** MaxRSS (KB) 32224 The difference is almost hard to believe; in this case CFEngine 3.10 is **750 times faster** than 3.7, while the consumed memory is about the same. What is more, the performance improvement in 3.10 will be even larger as the size of the data grows\! Please note that in order to make this vast improvement possible, it was necessary to replace the variable expansion engine in CFEngine. We have been very careful to ensure that policy is backward-compatible in 3.10, including running our some 1500 policy-based acceptance tests (which all pass). However, only you can test with your policy, so **please make sure that your policy still works to expectations in 3.10 beta**, emphasising on areas where you might depend on the ordering or nested expansion. One notable change in order of operations is that an additional pass resolving classes was added in pre-evaluation. Now pre-evaluation does vars, classes, vars; previously it was only classes, vars. ## Expanded Ubuntu and Solaris support We are happy to announce that 3.10 will have official support for the latest LTS releases from Canonical: **Ubuntu 14.04 LTS** and **Ubuntu 16.04 LTS**. This includes both host and Enterprise hub support on these platforms\! If you are interested in using CFEngine on Ubuntu, now is a great time to make sure it works according to your expectations\! In addition, CFEngine Enterprise customers can expect official support for **Solaris 10 x86** in 3.10. There is no beta package for Solaris 10 x86, but we will make a package available as soon as it has passed internal testing. You can find the list of currently supported platforms [here](https://docs.cfengine.com/docs/3.10/guide-latest-release-supported-platforms.html). ## systemd unit improvements The systemd support has received a major overhaul in CFEngine 3.10. All CFEngine services, like cf-serverd, cf-hub, cf-execd, now have their own independent systemd unit. This means that systemd will manage them independently, including monitor and restart them immediately if they stop for any reason. Please note _currently_ systemctl stop cfengine3 will stop all other CFEngine services. It returns immediately and systemd takes care of bringing down the other services. ## Bootstrap to hostname and alternative port 3.10 also introduces the **experimental** ability to bootstrap to a hostname and specify the port used by cf-agent to fetch policy (e.g. `cf-agent --bootstrap myhub.cfengine.com:5309)`. This information is written to `$(sys.workdir)/policy_server.dat`. `sys.policy_hub` will contain the hostname and the new variable `sys.policy_hub_port` will contain the port to use. Be-ware the Masterfiles Policy Framework does not account for the new bootstrap semantics out of the box. You must ensure that your ACL is configured appropriately as the default acl will not work with hostnames and the port used for bootstrap is not currently used to configure cf-serverd. ## Other improvements These were some of the highlights in 3.10 that should be tested to ensure the final production release meets your expectations. However, as always, there are a vast number of improvements in this new CFEngine release. Thank you to the community contributors who keep making CFEngine better for everyone\! For a full list of the improvements in 3.10.0 beta, please see the [Community ChangeLog](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise ChangeLog](https://docs.cfengine.com/docs/3.10/guide-latest-release-whatsnew-changelog-enterprise.html). ## Provide feedback The main goal of releasing the beta is to make sure that 3.10.0 works to your expectations in your environment. So if you get the chance to test out some of the features or perhaps the compatibility with your existing policy, please let us know how it went\! The best way to provide feedback is to submit issues relating to the beta in [the issue tracker.](https://northerntech.atlassian.net/projects/CFE) Please make sure to set the "Affects Version/s" field to "3.10.0 beta". For general questions and discussions, you can also use the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine) or the [\#cfengine](https://web.libera.chat/?channel=#cfengine) and [\#cfengine-dev](https://web.libera.chat/?channel=#cfengine-dev) chat rooms on libera.chat. ## Get it\! You can download the CFEngine [community 3.10.0 beta packages and source code](https://cfengine.com/product/community/). If you are an Enterprise customer, the Enterprise beta packages [can be downloaded here](https://cfengine.com/download/). Please note that this is a **beta release**, so we do not recommend installing it in a production environment. We hope you enjoy the new features, and we look forward to hearing about your experience\! ## Known issues The following issues have been found during testing and not fixed in the beta release as they are not considered impeding to testing the beta. They will all be resolved in the final production release of 3.10.0. - [CFE-2466](https://northerntech.atlassian.net/browse/CFE-2466) Can't insert_lines if referencing a variable that contains an empty string - [ENT-2841](https://northerntech.atlassian.net/browse/ENT-2841) "systemctl stop cfengine" doesn't stop all services immediately - [ENT-2846](https://northerntech.atlassian.net/browse/ENT-2846) Upgrading CFEngine Enterprise from 3.7.4 to 3.10.0beta does not allow logging into Mission Portal - [ENT-2845](https://northerntech.atlassian.net/browse/ENT-2845) CFEngine 3.10.0beta on Ubuntu 12 is attempting to use systemd when it should not. - [CFE-2495](https://northerntech.atlassian.net/browse/CFE-2495) cf-serverd ACLs should work with hostname or hostname/xx entries - AIX does not support alphabetic versions, so 3.10.0 Enterprise beta has version "3.10.99X" (where X is beta build number) on AIX. Final AIX release will have version "3.10.0" like all other platforms. -------------------------------------------------------------------------------- title: "Extending the CFEngine Policy Update Procedure" url: /blog/2016/extending-the-cfengine-policy-update-procedure/index.md date: "2016-10-28" -------------------------------------------------------------------------------- ## Introduction The policy update procedure in the [masterfiles policy framework](https://github.com/cfengine/masterfiles) is fairly straightforward - copy all files matching [$(update\_def.input\_name\_patterns)](https://github.com/cfengine/masterfiles/search?utf8=%E2%9C%93&q=input_name_patterns) from hub's $(sys.masterdir) directory to clients $(sys.inputdir) directory. Copying everything everywhere and then use some selection (or "classes") to determine what to do is fine at beginning. My first solution to manage computers in some computer system or landscape was to use "IPv4\_\*" (or hostname) classes, that pretty soon became hard to maintain and very hard to change. So, I needed a new model that improves maintainability and deployment flexibility of policy. This extension of update procedure enables each computer in some landscape or system to receive and execute only those policies that are specific to it. This improves maintainability and makes systems less cluttered with unneeded information/files. This model extends the existing update procedure to fetch client specific policy files from a separate repository directory on hub (outside $(sys.masterdir)). It copies policies to "$(sys.inputdir)/services/autorun" directory on client. Thanks the ["autorun" and "augments"](https://cfengine.com/company/blog-detail/modular-policies-with-autorun-and-the-augments_file/) features this approach is easy to implement. ## Tags The first thing to consider is tagging the CFEngine clients. Tags are very practical and flexible way to uniquely identify (i.e. server1, server2, etc.) and/or group (i.e. redhat, database, etc.) computing resources. In order to assign unique set of tags to each CFEngine client, I recommend putting them into a file outside $(sys.masterdir) and $(sys.inputdir) directories or when inside to use file with extension not matching $(update\_def.input name patterns). I.e. my implementation uses "$(sys.workdir)/node.tags" file and I do not use masterfile policy update procedure to manage it - more on implementations of tags file is explained in "how to tag" topic. Beside those from the tags file, my implementation uses IP address and hostname as tags too. ## How it works To implement this concept I wrote jb\_update\_policy3.cf (it is my 3rd version). It uses CFEngine idea of validation timestamp ([cf\_promises\_validated](https://www.cmdln.org/2012/10/24/cfengine-3-policy-update-or-how-cf_promises_validated-works/)) to determine if files are to be copied from hub and thus avoid heavy traffic on server and in network. On client side, this policy copies files from multiple source subdirectories and it has to take care of multiple timestamp validation files. On the hub, it has to check for changes on the policies inside repository subdirectories (which are named after tags, IPs and/or hostnames) and update validation timestamps in them. There is also a requirement to build a directory structure (named after tags, IPs and/or hostnames) in the other repository on the hub containing policy files to be distributed to remote agents. When, for example, required tag named subdirectory in that repository is not available on the hub, nothing gets copied (and promise not kept is indicated in agent verbose output). Some meaningful naming convention for policy files is required in order to avoid overwriting because they all get copied to same "service/autorun" subdirectory. NOTE "Autorun" policies are not required to be in the "service/autorun" subdirectory. Any bundle found in inputs with the autorun tag will be run automatically. ## Schema This diagram shows where a client copies files from the secondary policy repository based on its tags. ![schema](images/schema.jpg) In this schema, each of the two CFEngine clients has two tags and uses IP address to fetch policy files from correspondingly named subdirectories in repository directory. Note that "tag3" is common to both clients. ## Implementation The implementation uses the [augments file](https://cfengine.com/company/blog-detail/modular-policies-with-autorun-and-the-augments_file/) (aka def.json) to define needed variables.
```json { "classes" : { "services_autorun" : [ "any" ] }, "vars" : { "jb_repository_dir" : "$(sys.workdir)/repository", "jb_repository_perms" : "0600", "node_tags_file" : "$(sys.workdir)/node.tags" } } ```
ACLs for serving files from the secondary repository are defined in "jb\_repository\_acl\_rules.cf" which is included via "promises.cf".
```cf3 bundle server jb_repository_acl_rules { access: any:: "$(def.jb_repository_dir)" handle => "server_access_grant_access_jb_repository_dir", comment => "Grant access to the JB repository", admit => { @(def.acl) }; } ```
The update.cf policy that ships in the stock Masterfiles Policy Framework must be extended to include "jb\_update\_policy3" in the bundlesequence.
```cf3 body common control { bundlesequence => { "update_def", "u_cfengine_enterprise", @(u_cfengine_enterprise.def), "cfe_internal_dc_workflow", "cfe_internal_bins", "cfe_internal_update_policy", "cfe_internal_update_bins", "cfe_internal_update_processes", "jb_update_policy3" }; version => "update.cf $(update_def.current_version)"; inputs => { @(cfengine_update_controls.update_def_inputs), "cfe_internal/update/update_bins.cf", "cfe_internal/update/cfe_internal_dc_workflow.cf", "cfe_internal/update/cfe_internal_local_git_remote.cf", "cfe_internal/update/cfe_internal_update_from_repository.cf", "cfe_internal/update/update_policy.cf", "cfe_internal/update/update_processes.cf", "jb_update_policy3.cf" }; # Uncomment to connect to the hub using latest protocol. #protocol_version => "latest"; } ```
Since I am using the "autorun" feature of the Masterfiles Policy Framework each of the agent bundles in my secondary repository is tagged.
```cf3 meta: "tags" slist => { "autorun" }; ```
## How to classify hosts ("tagging")? There are multiple ways to set up or prepare the tags file content depending on the specific infrastructure (physical computer, VM, container or cloud). Here are some of my ideas on this. In my opinion, manually adding or managing content of the tags file is an option only for testing or lab environment but not for production (or even some relatively small number of CFEngine clients). A better way would be to let CFEngine to manage tags somehow. I.e. IP, hostname or MAC could be considered as tags and used directly. Another idea on managing tags is to make it 2-phased job. First phase would be to (re)distribute tags to the CFEngine clients using certain criteria and let some policy (re)write content of tags file which will be followed by second phase - fetch and run "autorun" policies according to tags. Cloud appliances, like AWS, could use user data file. Since content of this file could be a shell or "cloudinit" script, it is possible to use it to populate content of tags file once instance is started. Consider following simple AWS example:
```bash #!/bin/sh echo "tag1;tag2" > /var/cfengine/node.tags ```
In AWS, it is possible to set tag values as user data, fetch them at boot time (i.e. curl http://169.254.169.254/latest/user-data/) and write down to the tags file or it to use them directly in some update policy script - BTW from version 3.8.0 there is [url\_get()](https://docs.cfengine.com/docs/3.8/reference-functions-url_get.html) function in CFEngine. A container appliance like Docker has a feature for mounting single files, meaning a tag file could be prepared outside the container and then mounted and used by CFEngine client running inside container. Something like this:
```cf3 files: "$(tags_file_dir)/container1.tags" comment => "Creates tags file for container1", create => "true", edit_defaults => empty, edit_line => insert_lines("tag1;tag2"); commands: "/usr/bin/docker run --volume $(tags_file_dir)/container1.tags:/var/cfengine/node.tags ..." comment => "Running container container1"; ```
## Hub side of the story This policy update procedure needs prepared and maintained directory structures and policies in repository on the CFEngine hub where clients can look for and copy those policy files. Manual maintenance is of course out of question except for maybe some small test(s) so what I would recommend is to use CFEngine for this purpose. I developed another CFEngine policy that does exactly that task with following in mind: - all policies have unique names and they are located in multiple source directories somewhere on the hub but outside masterfiles or secondary repository - in tag named subdirectories (i.e. "tag1", "tag2", \, etc.) inside secondary repository, hard links will be created to policies in source directories - separation of processing code and data is recommended - into policy file (i.e. "policy1.cf") and JSON or YAML data file named like policy file, (i.e. "policy1.json") - in tag named subdirectories (i.e. "tag1", "tag2", \, etc.) hard links will be created to data files found in source directories too - if any file (data or policy) found in source directories is prefixed with tag name (i.e. "tag1\_policy1.json") that file will be hard linked in that subdirectory - which enables existence of tag specific files (i.e. Ubuntu vim packages are different from RedHat so I can have "ubuntu\_vim\_install.json" and "redhat\_vim\_install.json" data files and one "vim\_install.cf" policy file) - hard links inside subdirectories lose prefix - they are already located in tag named subdirectory (i.e. ubuntu\_vim\_install.json links to vim\_install.json) - in each subdirectory there will be validation timestamp file created, that indicates changes on files - all files and links that do not belong in subdirectory must be removed This diagram shows how the file sources are transformed when copied to the destination. ![policy\_linking](images/policy_linking.jpg) Just to emphasize: "policy3.cf" is used (link created inside directories) both with computing nodes tagged with "tag1" and "tag2" but they use different data files (prefixed with tag name in subsubdirectory named "sourcedir2"). ## Demo Here are some screenshots from my "lab". I produced them from multiple computing nodes: CFE hub (jb02-centos7), docker container (jbcontainer2) on the hub host and another VM (jb01-centos6). On the hub, my repository directories "/var/cfengine/repository/cfengine" looks like this: ![demo1](images/demo1.jpg) There are 4 IP tags "172.16.98.(130|135|149|214)" and 4 "named" tags "jbcontainer(1|2|3-1|3-2)" - resulting in 8 sub directories. Inside policies source directory it looks like this (example of "/var/cfengine/repository/policies/database" directory): ![demo2](images/demo2.jpg) Linked policy or data files are in blue and there are some tag specific data files for IP named tags (JSON files prefixed with IP). Inside some tags subdirectories, it looks like following: ![demo3](images/demo3.png) Linked files are in blue and there is validation file named "jb\_promises\_validated". In one of docker container, directory "/var/cfengine/inputs/services/autorun" contains: ![demo4](images/demo4.png) In another VM computing node, the same directory contains: ![demo5](images/demo5.png) As it can be seen validation file named "jb\_promises\_validated" is suffixed with tag name, for the case when policy files get acquired from multiple sub directories (tags) on the hub. I have tested this concept so far with CentOS, Ubuntu, pfSense/FreeBSD and inside Docker containers. Source code of those policies can be found on my GitHub [here](https://github.com/jborozan/cfengine-repository/blob/master/cfengine/jb_update_policy3.cf), [here](https://github.com/jborozan/cfengine-repository/blob/master/cfengine/jb_setup_client_dirs.cf) and [here](https://github.com/jborozan/cfengine-repository/blob/master/cfengine/jb_setup_client_dirs.json) and detailed explanation how it works on my blog [here](https://juricaborozan.blogspot.com/2016/01/cfengine-and-updatecf-extension.html) and [here](https://juricaborozan.blogspot.com/2016/02/cfengine-and-updatecf-extension-2.html). ## Conclusion This model enables relatively simple and "clean" deployment of policy files on CFEngine clients. Notably it follows conventions from CFEngine Masterfiles Policy Framework, implying minimal modification to default policy. With separation of policy code and data, it starts resembling Design Center. Further improvements in field of hub policy repository organization and maintenance are possible for sure, which I leave to users. I think that this concept takes us to step closer to simplified "droplets" idea - policy files that can be dropped to hub directory and deployed on clients. -------------------------------------------------------------------------------- title: "Interview with CFEngine user Natxo Asenjo" url: /blog/2016/interview-with-cfengine-user-natxo-asenjo/index.md date: "2016-09-29" -------------------------------------------------------------------------------- This interview was conducted by Aleksey Tsalolikhin / 3 Aug 2016 Q: What can you tell our readers about yourself? How did you start using CFEngine? A: I am a Spanish sysadmin living and working in The Netherlands (Dutch spouse). All round (multidisciplinary) sysadmin, with a focus on automation, bootstrapping infrastructures, high availability, disaster recovery, storage. I started using CFEngine about 8/9 years ago (or was it 10?). Back in the day I was the only admin doing Linux at our company, and the Linux infrastructure was growing every week and it was getting impossible to manage everything by hand. So CFEngine to the rescue. I found the tutorials on the [debian-administration.org](https://debian-administration.org/tag/CFEngine) site, looking at the publication dates, it must have been 10 years ago, time flies) and it seemed quite doable. Later I came across the Campi/Bauer book: [Automating Linux and Unix System Administration](https://www.amazon.com/Automating-Linux-System-Administration-Experts/dp/1430210591) and re-implemented quite a bit of stuff with their tips, not only CFEngine but also FAI/Kickstart. ![natxo](images/natxo-1-600x800.jpg) Q: Could you give us an example of something you've automated with CFEngine that stands out as having made your life easier, helps you to sleep better at night, or has really made a difference for the business? A: Stuff I've used CFEngine for: - disabling/enabling services; - configuration mail routing; - seLinux; - iptables; - yum/apt repositories; - installation software; - configuration syslogd/journald/logstash forwarders for ELK; - distribution of sysadmin scripts/configuration files/motd's; - distribution known\_hosts files for ssh clients; - distribution and configuration of monitoring software (nagios: nrpe/snmpd); - configuration ntpd/chronyd; Anything that needs doing basically. We have a number of host types (apache2, tftpd, mysqld, java application servers, etc). Our Linux infra is coupled to a freeipa environment with the automember plugin, so if the host is called tftp\\\* (where \\\* is 'whatever' ) then that host will be member of an LDAP hostgroup, which in FreeIPA gets populated as a netgroup as well. We can use those netgroups to automatically define classes in CFEngine and those classes are coupled to actions (install apache2, configure it according to x, y, z, etc.). My colleagues install the OS using PXE and the only thing they need to do once the installation is done is give the host a name (part of joining the FreeIPA domain). The rest gets done automatically. So this is about it in a nutshell. Without CFEngine this company would not have the Linux infrastructure it now has and I would not have had time to accomplish many other projects (I became much more efficient, so I got assigned to many more projects). Q: You said you have about 150 Linux nodes in your network. How many are still on cf2 and how many are on cf3? A: cf2: 70%, cf3 the rest (growing). Q: When we corresponded earlier, you said you "left the old cf2 environment alone (it basically never dies and does not need new modifications)". What led you to start using cf3? A: Lack of support for cf2. We use CentOS and with CentOS 7 the cf2 binaries were a bit long in the tooth. Q: What's the biggest advantage of cf3 over cf2 for you? A: cf3 is more consistent in its syntax. Plus the standard library has a lot of scaffolding already set up for us (services, for instance). The packages method is much nicer as well. Q: What are you most proud of about your setup? A: Difficult to say, in the end this is just a tool. It's nice to see things work the way they are supposed to, but that is our work. Q: What do you enjoy most about the CFEngine community? A: Very fast (good) answers and advice for alternative solutions to the issues one posts to the mailing list. Nice, polite, civil, discussions. All communities should be like this. Q: Any advice for the CFEngine 2 users out there? a: Even though cf2 is an excellent tool, the fact that it is no longer supported means that you will be on your own if something happens to your setup. This could be an issue. if you have a very stable and static environment where cf2 just works, leave that alone and set a new cf3 environment next to it. Sooner or later you will get a system where cf2 will not run without much effort, and at that time you will have a problem. Q: Anything else you'd like to share with the CFEngine community? A: I learnt a lot using [your tutorial](https://www.softcover.io/read/72be7d4f/cf3/) and [Brian Bennett's](https://digitalelf.net) ([@bahamat](https://twitter.com/bahamat)) posts, specially the [cf-primer: from zero to hero](https://digitalelf.net/cf-primer/). Those are the hands on tutorials that the community needs. CFEngine has a reputation for being hard, but it's not really true. Aleksey Tsalolikhin is a CFEngine consultant and trainer at [Vertical Sysadmin](https://www.verticalsysadmin.com/). -------------------------------------------------------------------------------- title: "Executing CFEngine SRC blocks in Emacs/Spacemacs Orgmode" url: /blog/2016/executing-cfengine-src-blocks-in-emacsspacemacs-orgmode/index.md date: "2016-09-12" -------------------------------------------------------------------------------- A few months ago [I posted a link](https://groups.google.com/forum/#!topic/help-cfengine/CCsUaRzWDZk) on the help list to the [CFEngine layer](https://github.com/syl20bnr/spacemacs/tree/develop/layers/%2Btools/cfengine) for [spacemacs](https://spacemacs.org/). Since then I have learned there are a few other org-mode users so I wanted to share how I got `cfengine3` src blocks execution working. I added the following to my `dotspacemacs/user-init`.
``` src src-emacs-lisp (defcustom org-babel-cfengine3-command "/var/cfengine/bin/cf-agent" "Name of command to use for executing CFEngine policy.") (defvar org-babel-cfengine3-command-options "--no-lock" "Option string that should be passed to the agent. Note that --file will be appended to the options.") (defvar org-babel-cfengine3-file-control-stdlib "body file control{ inputs => { '$(sys.libdir)/stdlib.cf' };}\n" "File control body to include the standard library from $(sys.libdir). It is useful to inject into an example source block before execution so that bundles and bodies from the standard library are automatically available.") (defun org-babel-execute:cfengine3 (body params) "Actuate a block of CFEngine 3 policy. This function is called by `org-babel-execute-src-block'. A temporary file is constructed containing `org-babel-cfengine3-file-control-stdlib and the body of the src block. `org-babel-cfengine3-command' is used to execute the temporary file." (let* ((temporary-file-directory ".") (tempfile (make-temp-file "cfengine3-"))) (with-temp-file tempfile ;; TODO Consider making automatic stdlib inclusion optional (insert org-babel-cfengine3-file-control-stdlib) (insert body)) (unwind-protect (shell-command-to-string (concat org-babel-cfengine3-command " " ;; TODO Consider adding a header option to specify bundlesequence org-babel-cfengine3-command-options " " (format " --file %s" tempfile))) (delete-file tempfile)))) ```
Now any time I have a `cfengine3` SRC block in org-mode I can simply run `org-babel-execute-src-block` ( `CTRL-c CTRL-c` in emacs/spacemacs documentation this is written as `C-c C-c` ) to have the block written to a temporary file, executed and the temporary file deleted. For example with my insertion placed inside the following SRC block. ``` example #+Name: example policy #+BEGIN_SRC cfengine3 :results raw :wrap EXAMPLE :exports both bundle agent main { reports: "$(sys.date)"; "CFEngine $(sys.cf_version)"; "Automatic inclusion of '$(sys.libdir)/stdlib.cf' when executed via org-babel-execute-src-block" classes => results("bundle", "example"); vars: "c" slist => classesmatching("example_.*"); reports: "class: '$(c)'"; } #+END_SRC ``` I type `C-c C-c` and my RESULTS are inserted.
``` example #+RESULTS: example policy #+BEGIN_EXAMPLE R: Mon Sep 12 08:26:49 2016 R: CFEngine 3.9.0 R: Automatic inclusion of '/home/nickanderson/.cfagent/inputs/lib/stdlib.cf' when executed via org-babel-execute-src-block R: class: 'example_kept' R: class: 'example_reached' #+END_EXAMPLE ```
Note that the standard library is automatically included by injecting a file control body into the temporary file before execution. For a block to be executed correctly it must contain either a `body common control` with the `bundlesequence` specified or include a bundle named `main`. Here you can see the exported results, now when I export my document I get a nicely formatted `cfengine3` policy block along with the accompanying execution output.
```cf3 bundle agent main { reports: "$(sys.date)"; "CFEngine $(sys.cf_version)"; "Automatic inclusion of '$(sys.libdir)/stdlib.cf' when executed via org-babel-execute-src-block" classes => results("bundle", "example"); vars: "c" slist => classesmatching("example_.*"); reports: "class: '$(c)'"; } ```
``` example R: Mon Sep 12 08:28:48 2016 R: CFEngine 3.9.0 R: Automatic inclusion of '/home/nickanderson/.cfagent/inputs/lib/stdlib.cf' when executed via org-babel-execute-src-block R: class: 'example_kept' R: class: 'example_reached' ``` -------------------------------------------------------------------------------- title: "Infrastructure management at scale: An overview of CFEngine Enterprise" url: /blog/2016/infrastructure-management-at-scale-an-overview-of-cfengine-enterprise/index.md date: "2016-08-26" -------------------------------------------------------------------------------- This post was authored by [Aleksey Tsalolikhin](https://verticalsysadmin.com/blog/infrastructure-management-at-scale/) and has been [re-published](https://verticalsysadmin.com/blog/infrastructure-management-at-scale/) with his consent. I recently spoke at [Digital Media Educators Conference](https://web.archive.org/web/20170925004123/http://ict-dm.net/faculty-pd/dmec) (DMEC) on Infrastructure Management at Scale and the skills educators need to impart to up and coming system administrators. This conference serves the California community college system, which is dear to my heart. My mother worked at [West Los Angeles College](https://www.wlac.edu/) library her entire professional life in America, since we arrived in 1988. I used to volunteer and help her out with shelving in the summer. I was a very poor helper since I kept getting distracted by all the delicious books and did more reading than shelving. While in high school I took computer programming, math and English at West Los Angeles College and at Santa Monica Community College, at first during summer break and then concurrent with eleventh grade, which allowed me to go to University instead of going to 12th grade. So I have a personal connection to the California community college system and I jumped at the chance to contribute a talk: ![DMEC Listing](images/0-cover-1.png) Because my presentation was in the [Data Representation](https://ict-dm.net/index.php?option=com_zoo&view=category&layout=category&Itemid=820) track, I focused on Inventory and Compliance Reporting so I could show off CFEngine's slick UI. I started by laying out CFEngine's philosophic groundwork: - [Promise Theory](https://en.wikipedia.org/wiki/Promise_theory) and the advantages of voluntary cooperation and distributed work over the limitations of imposed direct control. - The advantages of pull over push (see "Push versus pull" in [Deconstructing the \`CAP theorem' for CM and DevOps](https://markburgess.org/blog_cap2.html) by the author of CFEngine for more on this), and - The [Dunbar numbers](https://en.wikipedia.org/wiki/Dunbar%27s_number) which constrain the quality and quantity of relationships sysadmins are able to have with their infrastructures. The rest of the talk demonstrated how the design of CFEngine uses Dunbar numbers to focus the information it presents. ![Dunbar Numbers](images/1b-dunbar-iomedica-org.png) We also talked about [what computer system administration IS](https://lopsa.org/System-Admin-Careers), and what the challenges are and how we handle them. Then I introduced the CFEngine dashboard: ![CFEngine Enterprise Dashboard](images/dashboard_1.png) I pointed out the header which holds the host count (2, including the hub itself) and the health indicator (OK); the graph of Changes made by CFEngine, the fact that both of our hosts have Software Updates available (1 alert triggered on 2 hosts), and that we have 100% compliance on promise compliance and system health (green check-marks). The next slide, adding a third host (notice the hosts indicator up top), shows how the Alert for Software Updates changes to a 2/3 arc, as, right after adding the host, as at this point the hub knows 2 out of 3 hosts are missing software updates. Once the agent runs on the third host and the hub collects the report, the Alert will change back to a full circle with 3 out of 3 hosts are missing software updates. ![CFEngine Dashboard](images/dashboard_2_out_of_3_hosts_have_updates_available.png) The next slide illustrates how CFEngine communicates the **severity** of the alert: critical issues are indicated in red, less severe in orange (amber for you Aussies), and mildest level is yellow. I induced a policy non-compliance situation on one of the three hosts (e.g., promised a file edit but prevented CFEngine from accessing the file by filling up the disk), so the Promise Compliance alert spans 1/3 of the circle (1 out of 3 hosts). ![CFEngine Dashboard](images/dashboard_3_policy_compliance_not_kept.png) Notice also that if CFEngine is unable to collect reports from a host or if an agent stops running on a host, the health indicator at the top of the screen changes from OK to a red number indicating the number of issues: ![CFEngine Dashboard](images/dashboard_6_health_0.png) You can see the number and type of issues: ![CFEngine Dashboard](images/dashboard_7_health_1.png) Notice that the Dunbar numbers are in play here: CFEngine tells you there are issues, and if you want more data, then you can have it. But it doesn't throw all the detail at you at once, that would be too much. You can get more detail on which hosts are not reporting by selecting "Hosts not reporting" from the health indicator menu: ![CFEngine Dashboard](images/dashboard_7_health_1.png) You can then select a host in the list of hosts not reporting to see the info for that host (host detail). ![CFEngine Enterprise Dashboard](images/dashboard_8_health.png) ![CFEngine Enterprise Hosts](images/hosts_tree.png) That actually takes us to the "Hosts" tab. The "Hosts" tab starts in the "all hosts" view, where you see the promise compliance summary for your infrastructure: ![CFEngine Enterprise Policy Compliance](images/promise-compliance-1.png) You can list the hosts that have less than 100% compliance: ![CFEngine Enterprise Policy Compliance](images/promise-compliance-2.png) You can see which promises were not kept on each host: ![CFEngine Enterprise Policy Compliance](images/promise-compliance-3.png) And that takes us to the "Reports" tab. There are many reports available but let's take a look at the Inventory Report. It starts out with four basic columns but you can add more: ![CFEngine Enterprise Inventory](images/timezone-0.png) You can extend inventory collection by writing CFEngine promises, for example, here I've added inventory of the host's timezone: ![CFEngine Enterprise Inventory](images/timezone-1.png) Let's say our company policy says all hosts must be in the UTC timezone. But in reality we have this: ![CFEngine Enterprise Inventory](images/timezone-2.png) You can sort the column contents by selecting the column heading, this groups the outliers and brings them into view: ![CFEngine Enterprise Inventory Sorting](images/timezone-3-sorted.png) You can graphically summarize column contents by selecting "Chart Data": ![CFEngine Enterprise Inventory Charts](images/timezone-4-chart-data-dialog.png) Voila\! ![CFEngine Enterprise Inventory Pie Chart](images/timezone-5-pie-chart.png) Hover over a slice to get more detail: ![CFEngine Enterprise Inventory Pie Chart Details](images/timezone-6-percentages.png) Or switch to column view: ![CFEngine Enterprise Inventory Bar Chart](images/timezone-7-bar-graph.png) Here is another example: ![CFEngine Enterprise Inventory Active Directory Status](images/timezone-8-AD-status.png) The charts can be exported and embedded in reports to management, auditors, etc. Want to give CFEngine Enterprise a try? It's very easy to [download](https://cfengine.com/download/) and install the hub package. Feel free to [email me](mailto:aleksey@verticalsysadmin.com) if you have any questions\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.4 and 3.9.1 released!" url: /blog/2016/cfengine-3-7-4-and-3-9-1-released/index.md date: "2016-08-03" -------------------------------------------------------------------------------- We're happy to announce maintenance releases for both supported CFEngine release branches today\! Being maintenance (aka patch) releases, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, the releases primarily include bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](/product/supported-versions/), we can see that - **3.6** and **3.8** are **no longer maintained**. If you are still on any of these versions, please consider upgrading. - **3.7** LTS is maintained (and supported for Enterprise customers) until **July 17th 2018** - **3.9** non-LTS is maintained until **December 2016** For CFEngine Enterprise customers, the only supported release today is 3.7. The next supported release will be 3.10 LTS due end of 2016, which will be supported through the end of 2019. If you are planning to [contribute improvements](https://github.com/cfengine/core/blob/master/HACKING.md) to 3.10 (thank you\!), please note that we would need the pull requests ready for merging by **mid-September** in order to have time to incorporate them into 3.10. ## Changes and improvements The systemd units and System V init scripts to manage CFEngine's daemons have received several enhancements. The systemd units now require network before starting CFEngine (because cf-serverd and cf-agent might use the network), have received more appropriate file permissions and are managed through CFEngine policy if systemd units should fail. The System V scripts will now exit with a failure and output an error if you try to "start" CFEngine when there is no promises.cf present. There is also [ongoing work](https://northerntech.atlassian.net/browse/CFE-2278) to make the systemd units more modular for CFEngine 3.10. The memory management of core components has been improved in several areas, fixing some memory leaks and ensuring two-dimensional "classic arrays" used by building bracketed strings and [getindices()](https://docs.cfengine.com/lts/reference-functions-getindices.html) are properly expanded in 3.9.1. Package management on SUSE Linux has also gotten some enhancements, reducing the network traffic and making CFEngine upgrades more reliable. #### Enterprise edition specific changes The Enterprise edition has gotten a new out-of-the-box inventory attribute _System product name._ Its value corresponds to the output from _dmidecode -s system-product-name_, and it is a good complement to the existing _System manufacturer_ inventory attribute. Also note that CFEngine Enterprise allows you to report any [custom inventory from your policy](https://cfengine.com/learn/report-custom-inventory-string/). Several security hardening changes for the CFEngine Enterprise server has been implemented, such as disabling exposure of the running php version and using more restrictive permissions for backend services. Finally, the hub package dependency on _libltdl_ has been removed; so you will no longer need to have that package installed on your CFEngine Enterprise servers. #### Dependency upgrades The bundled dependencies have been upgraded to bring in the latest security, performance and reliability improvements. Both 3.7.4 and 3.9.1 are bundled with PCRE 8.39 and OpenSSL 1.0.2h. On the Enterprise server, both releases come with PHP 5.6.24, while the Apache web server is at version 2.2.31 in CFEngine 3.7.4 and 2.4.23 in CFEngine 3.9.1. For more details on the improvements in the releases, please see the relevant ChangeLogs: - 3.7.4 ChangeLog: [core](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html) and [enterprise](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html) - 3.9.1 ChangeLog: [core](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-core.html) and [enterprise](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-enterprise.html) ## Test the new and faster variable iteration engine An issue that has brought a lot of attention is the [slow iteration through large variable sets](https://northerntech.atlassian.net/browse/CFE-2162). This is by no means a new issue, it has probably existed for around 10 years, but the introduction of enhanced external data support with functions like [readjson()](https://docs.cfengine.com/lts/reference-functions-readjson.html) and [readyaml()](https://docs.cfengine.com/lts/reference-functions-readyaml.html) as well as enhanced data manipulation features makes this issue surface more frequently. We have recently produced a new experimental build that passes all acceptance tests, and [packages are available for testing](https://northerntech.atlassian.net/browse/CFE-2162?focusedCommentId=88124). Please help ensure that this improvement can be merged by testing your policy with these packages and let us know if everything works as expected or not by leaving an issue comment. Backward compatibility is very important, so we can not move forward without sufficient testing in real environments. Thank you for your assistance in improving CFEngine\! ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine customer support](https://cfengine.zendesk.com/hc/en-us) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can give 3.7.4 a quick spin with the [CFEngine Enterprise Vagrant environment](/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.9.0 non-LTS released!" url: /blog/2016/cfengine-3-9-0-non-lts-released/index.md date: "2016-06-15" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.9.0 non-LTS now is released\! A big thanks to everyone testing the [3.9.0 beta release](https://cfengine.com/company/blog-detail/cfengine-3-9-0-non-lts-beta-ready-for-testing)\! Please note that this is a non-LTS release, which means that it is maintained for 6 months from the release date and not supported for CFEngine Enterprise customers (but packages are available for testing). The established [CFEngine release schedule](/product/supported-versions/) gives an overview over the release timelines for all releases. ## Running remote bundles CFEngine 3.9 introduces a simple way to trigger bundles to run remotely by using _cf-runagent --remote-bundles_ to ask a remote cf-serverd to run a given set of bundles. For example: cf-runagent --hail web01.example.com,web02.example.com \ --remote-bundles fetch_latest_webapp,restart_tomcat Security is taken into account by requiring that all these conditions are satisfied: 1. the bundle(s) already exist in the remote system's policy 2. the cf-runagent client is allowed to connect with [allowallconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowallconnects) or [allowconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowconnects) 3. the cf-runagent client and bundle(s) are explicitly given access to with [resource_type bundle](https://docs.cfengine.com/docs/3.9/reference-promise-types-access.html#resource_type) regular expressions 4. bundle parameters are not allowed; only bundles without parameters can be run This feature can be very useful if you have one-off tasks that you need to run, like updating or restarting some applications or systems. ## Email filtering from CFEngine clients **![](images/Mail.ico)**We planned to improve the logging capabilities of CFEngine as part of the [3.9 theme](/company/blog-detail/release-theme-for-cfengine-3-9-logging/). The most requested feature in this theme was a way to filter what gets emailed from CFEngine clients, as this can lead to a high volume of emails if left uncontrolled. The CFEngine clients always filter out email content that is the same as the previous message, however more fine-grained line-filtering capabilities have been implemented in 3.9. In 3.9 the CFEngine executor control attributes [mailfilter_exclude](https://docs.cfengine.com/docs/3.9/reference-components-cf-execd.html#mailfilter_exclude) and [mailfilter_include](https://docs.cfengine.com/docs/3.9/reference-components-cf-execd.html#mailfilter_include) are introduced, making it possible to filter out certain output lines that you do not want emailed. For example, the following cf-execd configuration will ensure no lines matching "Permission denied" are ever emailed. ```cf3 body executor control { mailfilter_exclude => { ".*Permission denied.*" }; } ``` ## More inventory out-of-the-box In 3.9, a mapping from IP address to network interface is provided with the new special variable _sys.ip2iface_. For Linux platforms with /proc/net/, several new network-related sys variables and functions have been introduced. These include [_sys.inet_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html), [_sys.inet6_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html) and [_sys.interfaces_data_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html), as well as the function [network_connections()](https://docs.cfengine.com/docs/3.9/reference-functions-network_connections.html) which returns all current network connections. Process inventory has also been improved with the brand new functions [findprocesses()](https://docs.cfengine.com/docs/3.9/reference-functions-findprocesses.html) and [processexists()](https://docs.cfengine.com/docs/3.9/reference-functions-processexists.html) which, as their names indicate, return a list of processes and checks for existence of processes based on a regular expression. ## Powerful data manipulation CFEngine 3.9 makes working with function parameters much more convenient by allowing complex parameters like lists and JSON to be passed directly without storing them in a temporary variable, e.g. `mergedata('[ "thing", { "mykey": "myvalue" } ]')`. See the [documentation on Collecting functions](https://docs.cfengine.com/docs/3.9/reference-functions.html#collecting-functions) for more information. When working with data it can often be useful to do string replacements before using the data in configuration files. This can now easily be achieved with the new [regex_replace()](https://docs.cfengine.com/docs/3.9/reference-functions-regex_replace.html) function. If you need more advanced data manipulation capabilities, it is worth looking into the new _json_pipe_ argument to [mapdata()](https://docs.cfengine.com/docs/3.9/reference-functions-mapdata.html). This should meet all your JSON data manipulation needs by leveraging external tools such as [jq](https://stedolan.github.io/jq/) (also see the new _def.jq_ variable). ## Hardening of the CFEngine Server cf-serverd now only accepts TLS connections by default, while 3.6, 3.7 and 3.8 cf-serverd accepted both "classic" and TLS connections by default. If your clients are all on 3.7 or later, there are no compatibility issues as 3.7 clients (e.g. cf-agent) use TLS by default. However, if you have clients on 3.6 or earlier you can enable "classic" protocol on the server side by adjusting [allowlegacyconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowlegacyconnects) accordingly. Support for plain http for the Mission Portal and API has now been disabled by default, so only https-connections are accepted. This is implemented by redirecting plain http requests to https. If you need plain http, this can be enabled again by [defining the class _cfe_enterprise_enable_plain_http_](https://docs.cfengine.com/docs/3.9/enterprise-cfengine-guide-hub_administration-enable-plain-http.html), e.g. in _def.json._ The default RSA key-size for Apache has been increased from 2048 to 4096 bits. However note that the [default self-signed certificate can easily be replaced](https://docs.cfengine.com/docs/3.9/enterprise-cfengine-guide-hub_administration-custom-https-certificate.html). \*\* To further strengthen security, the admin user is now required to change password upon first login to the Mission Portal in order to avoid default passwords, as shown below. ![change-password](images/change-password.png) Finally, the files comprising the Mission Portal web application have received more restrictive file permissions to reduce the risk of attacks against the web server. ## Other improvements ![freebsd-logo](images/freebsd-logo.png)Several new [CFEngine package modules](https://docs.cfengine.com/docs/3.9/reference-standard-library-package_modules.html) for FreeBSD and AIX have been contributed by the CFEngine community and released in 3.9; namely _freebsd_ports_, _pkg_ and _pkgsrc_ for FreeBSD and _nimclient_ for AIX. This enables FreeBSD and AIX users to leverage the [improved packages: promise introduced in 3.7](https://docs.cfengine.com/lts/reference-promise-types-packages.html) which is more reliable. Internally, the process parsing implementation in CFEngine has been rewritten to make it more reliable. There have been several issues with getting the process table and parsing it reliably on platforms like Solaris where the available ps utilities sometimes mangle their output columns. This means that promises and functions that rely on process information will work more reliably on all platforms, but please note that **process: promises no longer match against the entire ps output, just the command and arguments**. This is normally what is interesting to match against, so most likely your policy will be compatible, but it is best to do a quick check to make sure. More information is available in the [3.9 processes: documentation](https://docs.cfengine.com/docs/3.9/reference-promise-types-processes.html). In 3.9 variable definitions are included in debug mode, in case you need to see where a particular variable gets defined and which value it takes. This can help tremendously in diagnosing a large CFEngine policy. All packaged dependencies have been upgraded to their latest stable version, in order to receive reliability and security enhancements. Search for "CFEngine dependencies" in the ChangeLogs below to see exact versions. Another high-demand fix aimed for CFEngine 3.9 relates to [speeding up variable expansion over large data structures](https://northerntech.atlassian.net/browse/CFE-2162). We have worked really hard to address this complex issue, but unfortunately it did not make it in time for the 3.9 release. However, good progress has been made lately, and [packages are available for testing](https://northerntech.atlassian.net/browse/CFE-2162) (see the latest comments). The final fix will be available in CFEngine 3.10.0 LTS due for December 2016 and you can get it soon by following the master branch. Please note the [known issue on upgrading an existing CFEngine Enterprise installations](https://docs.cfengine.com/docs/3.9/guide-latest-release-known-issues.html#enterprise-hub---mission-portal-database-not-properly-preserved). If you are upgrading for testing purposes check out the [upgrade documentation](https://docs.cfengine.com/docs/3.9/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. For a full list of the improvements in 3.9.0, please see the [Community ChangeLog](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise ChangeLog](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-enterprise.html). ## Get it\! If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), and [packages](https://cfengine.com/community/download/) for 3.9.0\! If you are an Enterprise customer, please note that [3.9 is a non-LTS release and thus not covered under the support contract](/product/supported-versions/). If you would like to test 3.9.0, the Enterprise packages [can be downloaded here](https://cfengine.com/download/). We hope you enjoy 3.9.0, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "How efficient is your IT infrastructure and what can you do?" url: /blog/2016/how-efficient-is-your-it-infrastructure-and-what-can-you-do/index.md date: "2016-06-06" -------------------------------------------------------------------------------- As someone responsible for IT operations, you have probably spent most of your time thinking of ways to improve your team's effectiveness and efficiency to make it more competitive. At CFEngine, we have witnessed the management of IT infrastructure and organizations operating across the spectrum of the performance scale. Where are you on that scale and how can you improve? In the world of IT operations, a negatively skewed distribution seems to exist. The majority of companies, despite their admirable DevOps initiatives, struggle. The great minority who are masters of execution continuously increase their relative competitiveness leaving the others behind. In my previous post, "The painful road to the digital economy," I pointed out some ideas as to why becoming more competitive seems so hard for Fortune 2000 companies and suggested ways to improve. It might be worth a read if you find yourself too far on the left side in the illustration below. The characteristics of a poor and reactive versus proactive and high-performing IT organization can be many. Below you will find some of the most common observed with our own users. Have a look and compare them with your own state of operations. [![](images/CFEngine_from_reactive_to_proactive.png)](/blog/2016/images/CFEngine_from_reactive_to_proactive.png) ## From reactive to proactive The more proactive your organization is, the more high-performing it is. The correlation would be close to perfect. Excelling organizations, to the surprise of many, appear to be the ones who spend the most time "thinking" instead of "doing". Thinkers leave the doing to the machines, while doers, well they are doers and never have the time to think. The first step in improving entails finding time to think. At Facebook, they have a wonderful mantra that an engineer shall never wait for the machine. The machine shall always be ready for the human. Whatever they do, they seek to ensure there is no computational waiting time. To get there you need to think, be thoughtful about every change you do and continuously seek to improve. Thoughtless execution won't get you far. More prudence in time allocation will, for most, lead to more proactivity. Be self-aware of time. It is a good start. However and unfortunately, reactive organizations often never find the time to think, because there is no time available. In our experience, almost all leading IT organizations sooner or later gravitate towards having a cross-functional automation team of thinkers. As someone responsible for a large infrastructure, you might be able to jump ahead and hire one or two personnel in a new automation team whose only responsibility is to free up the doers to do more thinking. That action alone might bear great fruits. **Tip: Create a small (1 - 2 person) automation team** ## Apply Zorro circles No one becomes a champion in a day. Talent, hard work, and long-term planning is required in today's highly competitive environment. Addressing all the characteristics in the above illustration will be overwhelming. For most IT leaders, their moonshot project should not be to "fix" IT operations. Applying the concept of gradual improvement or Zorro circles1 seems more appropriate. Focus on a few items and do not move on until these are fully mastered. Highly reactive IT operations often suffer from external locus of control which depletes progress. As someone responsible, this highly undesirable attitude can be addressed by redirecting focus to small and achievable plans. ## The first circle: change So what should be your first circle? Enforce version control everywhere, coerce SSH into individual machines, weekly deployments, master a new shiny DevOps tool, decrease mean time to repair by something measurable, buy beer for everyone? While all of these are great ideas (especially the last one), I strongly advocate that the first circle to master is the circle of change. Change is everything. Not mastering change leads nowhere. The single biggest obstacle for improvements in Fortune 2000 companies, I would claim, is the resistance for change, a resistance melted into the walls and structure of these organizations. What can be done? Besides trying to break down walls and rigid structures, which most will fail at, just making smaller changes for the sake of change can be a good start. Minimal and perceived harmless changes lead to motion, which is important in defying rigidity. Once the wheels are in motion, change hopefully becomes a habit, and something that actually leads to new things/processes/results. Being able to master change is the prerequisite for addressing the characteristics mentioned above. When looking at high-performing IT organizations they change frequently. Change is their only constant. As someone responsible, mastering the process of change (based on voluntary cooperation and not management by fiat), in which the organization embraces change should be your first circle. With this circle mastered, the next circle will be much easier to move forward with, and you can gradually try to address the characteristics of proactive IT organizations. After all, what high-performing IT organizations are doing and how they operate is quite well known. The challenge revolves around getting your people to follow the recipe. 1)As much as our masked friend wanted to become the best, it was only after he mastered and felt in control of one small circle could he expand his circle to include more capabilities, before he eventually became the legend he is now known to be. -------------------------------------------------------------------------------- title: "CFEngine 3.9.0 non-LTS beta ready for testing!" url: /blog/2016/cfengine-3-9-0-non-lts-beta-ready-for-testing/index.md date: "2016-05-11" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.9.0 non-LTS beta is now ready for testing\! The established [CFEngine release schedule](/product/supported-versions/) shows that the 3.9.0 final version is due in June 2016, so it's time to test and fix any remaining issues. Please note that this is a non-LTS release, which means that it is maintained for 6 months from the release date and not supported for CFEngine Enterprise customers, but packages are available for testing. ## Running remote bundles CFEngine 3.9 introduces a simple way to trigger bundles to run remotely by using _cf-runagent --remote-bundles_ to ask a remote cf-serverd to run a given set of bundles. For example: cf-runagent --hail web01.example.com,web02.example.com \ --remote-bundles fetch_latest_webapp,restart_tomcat Security is taken into account by requiring that all these conditions are satisfied: 1. the bundle(s) already exist in the remote system's policy 2. the cf-runagent client is allowed to connect with [allowallconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowallconnects) or [allowconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowconnects) 3. the cf-runagent client and bundle(s) are explicitly given access to with [resource_type bundle](https://docs.cfengine.com/docs/3.9/reference-promise-types-access.html#resource_type) regular expressions 4. bundle parameters are not allowed; only bundles without parameters can be run This feature can be very useful if you have one-off tasks that you need to run, like updating or restarting some applications or systems. ## Email filtering from CFEngine clients **![](images/Mail.ico)**We planned to improve the logging capabilities of CFEngine as part of the [3.9 theme](/company/blog-detail/release-theme-for-cfengine-3-9-logging/). The most requested feature in this theme was a way to filter what gets emailed from CFEngine clients, as this can lead to a high volume of emails if left uncontrolled. The CFEngine clients always filter out email content that is the same as the previous message, however more fine-grained line-filtering capabilities have been implemented in 3.9. In 3.9 the CFEngine executor control attributes [mailfilter_exclude](https://docs.cfengine.com/docs/3.9/reference-components-cf-execd.html#mailfilter_exclude) and [mailfilter_include](https://docs.cfengine.com/docs/3.9/reference-components-cf-execd.html#mailfilter_include) are introduced, making it possible to filter out certain output lines that you do not want emailed. For example, the following cf-execd configuration will ensure no lines matching "Permission denied" are ever emailed. ```cf3 body executor control { mailfilter_exclude => { ".*Permission denied.*" }; } ``` ## More inventory out-of-the-box In 3.9, a mapping from IP address to network interface is provided with the new special variable _sys.ip2iface_. For Linux platforms with /proc/net/, several new network-related sys variables and functions have been introduced. These include [_sys.inet_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html), [_sys.inet6_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html) and [_sys.interfaces_data_](https://docs.cfengine.com/docs/3.9/reference-special-variables-sys.html), as well as the function [network_connections()](https://docs.cfengine.com/docs/3.9/reference-functions-network_connections.html) which returns all current network connections. Process inventory has also been improved with the brand new functions [findprocesses()](https://docs.cfengine.com/docs/3.9/reference-functions-findprocesses.html) and [processexists()](https://docs.cfengine.com/docs/3.9/reference-functions-processexists.html) which, as their names indicate, return a list of processes and checks for existence of processes based on a regular expression. ## Powerful data manipulation CFEngine 3.9 makes working with function parameters much more convenient by allowing complex parameters like lists and JSON to be passed directly without storing them in a temporary variable, e.g. `mergedata('[ "thing", { "mykey": "myvalue" } ]')`. See the [documentation on Collecting functions](https://docs.cfengine.com/docs/3.9/reference-functions.html#collecting-functions) for more information. When working with data it can often be useful to do string replacements before using the data in configuration files. This can now easily be achieved with the new [regex_replace()](https://docs.cfengine.com/docs/3.9/reference-functions-regex_replace.html) function. If you need more advanced data manipulation capabilities, it is worth looking into the new _json_pipe_ argument to [mapdata()](https://docs.cfengine.com/docs/3.9/reference-functions-mapdata.html). This should meet all your JSON data manipulation needs by leveraging external tools such as _jq_ (also see the new _def.jq_ variable). ## Hardening of the CFEngine Server cf-serverd now only accepts TLS connections by default, while 3.6, 3.7 and 3.8 cf-serverd accepted both "classic" and TLS connections by default. If your clients are all on 3.7 or later, there are no compatibility issues as 3.7 clients (e.g. cf-agent) use TLS by default. However, if you have clients on 3.6 or earlier you can enable "classic" protocol on the server side by adjusting [allowlegacyconnects](https://docs.cfengine.com/docs/3.9/reference-components-cf-serverd.html#allowlegacyconnects) accordingly. Support for plain http for the Mission Portal and API has now been disabled by default, so only https-connections are accepted. This is implemented by redirecting plain http requests to https. If you need plain http, this can be enabled again by [defining the class _cfe_enterprise_enable_plain_http_](https://docs.cfengine.com/docs/3.9/enterprise-cfengine-guide-hub_administration-enable-plain-http.html), e.g. in _def.json._ The default RSA key-size for Apache has been increased from 2048 to 4096 bits. However note that the [default self-signed certificate can easily be replaced](https://docs.cfengine.com/docs/3.9/enterprise-cfengine-guide-hub_administration-custom-https-certificate.html). \*\* To further strengthen security, the admin user is now required to change password upon first login to the Mission Portal in order to avoid default passwords, as shown below. ![change-password](images/change-password.png) Finally, the files comprising the Mission Portal web application have received more restrictive file permissions to reduce the risk of attacks against the web server. ## Other improvements ![freebsd-logo](images/freebsd-logo.png)Several new [CFEngine package modules](https://docs.cfengine.com/docs/3.9/reference-standard-library-package_modules.html) for FreeBSD and AIX have been contributed by the CFEngine community and released in 3.9; namely _freebsd_ports_, _pkg_ and _pkgsrc_ for FreeBSD and _nimclient_ for AIX. This enables FreeBSD and AIX users to leverage the [improved packages: promise introduced in 3.7](https://docs.cfengine.com/lts/reference-promise-types-packages.html) which is more reliable. Internally, the process parsing implementation in CFEngine has been rewritten to make it more reliable. There have been several issues with getting the process table and parsing it reliably on platforms like Solaris where the available ps utilities sometimes mangle their output columns. This means that promises and functions that rely on process information will work more reliably on all platforms, but please note that **process: promises no longer match against the entire ps output, just the command and arguments**. This is normally what is interesting to match against, so most likely your policy will be compatible, but it is best to do a quick check to make sure. More information is available in the [3.9 processes: documentation](https://docs.cfengine.com/docs/3.9/reference-promise-types-processes.html). Another high-demand fix aimed for CFEngine 3.9 relates to [speeding up variable expansion over large data structures](https://northerntech.atlassian.net/browse/CFE-2162). We have worked really hard to address this complex issue, but unfortunately it did not make it in time for the 3.9 beta release. However, the fix will be found in master within weeks and packages including it will be available as part of the [CFEngine Community nightly builds](https://builds.cfengine.com/pub/nightly/) then. If this fix receives sufficient community testing once in master, we will consider releasing it into a stable release, otherwise it will be available in CFEngine 3.10.0 LTS due for December 2016 at the latest. In 3.9 variable definitions are included in debug mode, in case you need to see where a particular variable gets defined and which value it takes. This can help tremendously in diagnosing a large CFEngine policy. All packaged dependencies have been upgraded to their latest stable version, in order to receive reliability and security enhancements. Search for "CFEngine dependencies" in the ChangeLogs below to see exact versions. For a full list of the improvements in 3.9.0 beta, please see the [Community ChangeLog](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise ChangeLog](https://docs.cfengine.com/docs/3.9/guide-latest-release-whatsnew-changelog-enterprise.html). ## Provide feedback The main goal of releasing the beta is to make sure that 3.9.0 works to your expectations in your environment. So if you get the chance to test out some of the features or perhaps the compatibility with your existing policy, please let us know how it went\! The best way to provide feedback is to submit issues relating to the beta in [the issue tracker.](https://northerntech.atlassian.net/projects/CFE) Please make sure to set the "Affects Version/s" field to "3.9.0 beta". For general questions and discussions, you can also use the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine) or the [\#cfengine](https://web.libera.chat/?channel=#cfengine) and [\#cfengine-dev](https://web.libera.chat/?channel=#cfengine-dev) chat rooms on libera.chat. ## Get it\! You can download the CFEngine [community 3.9.0 beta packages and source code](https://cfengine.com/product/community/). If you are an Enterprise customer, please note that [3.9 is a non-LTS release and thus not covered under the support contract](/product/supported-versions/). If you would like to test 3.9 beta, the Enterprise beta packages [can be downloaded here](https://cfengine.com/download/). Please note that this is a **beta release**, so we do not recommend installing it in a production environment. We hope you enjoy the new features, and we look forward to hearing about your experience\! ## Known issues - AIX does not support alphabetic versions, so 3.9.0 Enterprise beta has version "3.9.99X" (where X is beta build number) on AIX. Final AIX release will have version "3.9.0" like all other platforms. -------------------------------------------------------------------------------- title: "Badlock Reporting and Remediation" url: /blog/2016/badlock-reporting-and-remediation/index.md date: "2016-04-19" -------------------------------------------------------------------------------- By now you have probably heard about the [Badlock vulnerability](https://access.redhat.com/security/vulnerabilities/badlock) ([CVE-2016-2118](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2118)) in DCE/RPC-based SAMR and LSA protocols used in the Microsoft Windows ActiveDirectory infrastructure as well as other [critical security flows in Samba](https://access.redhat.com/articles/2243351). With CFEngine Enterprise you can simply tag any variable or class and MissionPortals Inventory reporting interface will be automatically extended with the new attributes. This makes it easy to identify vulnerable hosts. ![inventory\_report\_vulnerable\_cves](images/inventory_report_vulnerable_cves-800x382.png) ![inventory\_report\_vulnerable\_cves\_chart](images/inventory_report_vulnerable_cves_chart-800x382.png) Dashboard alerts can be created to alert on vulnerable hosts for specific subsets of infrastructure. ![define\_alert](images/define_alert-793x800.png) ![alert\_status\_vulnerable\_hosts](images/alert_status_vulnerable_hosts-800x382.png) Dashboard alerts can be [integrated with other systems](https://cfengine.com/learn/integrate-with-other-tools). For example you could [automatically open an issue in Jira](https://cfengine.com/learn/ticketing-system-integration-jira/) when vulnerable hosts are found. If you would like to use CFEngine to detect, repair and report on Badlock in your infrastructure, we have prepared some policies you can use: - [Badlock reporting and remediation policy](https://github.com/nickanderson/cfengine-CVE-2016-2118) - [Implementation Tutorial](https://github.com/nickanderson/cfengine-CVE-2016-2118/blob/master/docs/implementation_tutorial.org) -------------------------------------------------------------------------------- title: "CFEngine 3.7.3, 3.6.7 and 3.8.2 released!" url: /blog/2016/cfengine-3-7-3-3-6-7-and-3-8-2-released/index.md date: "2016-04-01" -------------------------------------------------------------------------------- We're happy to announce maintenance releases for all supported CFEngine release branches today\! Being maintenance (aka patch) releases, the goal is to increase stability and reliability for CFEngine users and enable a safe upgrade-path. As such, the releases primarily include bugfixes and low-risk changes that do not impact the compatibility between previous patch releases. Looking at the [CFEngine release schedule](/product/supported-versions/), we can see that - **3.7** LTS is maintained until **July 17th 2018** - **3.6** is maintained until **July 17th 2016** (released before CFEngine incorporated the LTS model) - **3.8** (being a non-LTS release) is maintained until **3.9.0 is released** For CFEngine Enterprise customers, the supported releases as of today are 3.7 and 3.6. If you are on 3.6, it is a good time to prepare for an upgrade to 3.7, as the next supported minor release will be 3.10 LTS due end of 2016. If you are planning to contribute improvements to 3.9 (thank you\!), please note that we would need the pull requests by beginning of April in order to have time to incorporate them into 3.9. ## Changes and improvements The [def.json feature for overriding CFEngine default configuration](/company/blog-detail/modular-policies-with-autorun-and-the-augments_file/) was introduced in 3.7 and has since seen significant adoption. The original goal of this feature is to make upgrades easier, as it makes the difference between vanilla masterfiles and user masterfiles much easier to manage by separating CFEngine settings like mail configuration out of masterfiles. However, it is also being used to load custom data into CFEngine, like the contents of a CMDB. A feature to parse def.json natively in CFEngine (in C code) to support early data loading was [introduced in 3.8.1](/company/blog-detail/cfengine-3-8-1-non-lts-released/). This improvement is included in 3.7.3. Furthermore, several reported stability and usability improvements of def.json have been introduced in 3.7.3 and 3.8.2 (3.6 does not have this feature). The evaluator component of CFEngine, responsible for defining classes, variables and deciding on which parts of the policy should be executed, has received several important stability enhancements in all branches (3.7, 3.6 and 3.8). Please note that these improvements also lead to some functions (most notably [classesmatching()](https://docs.cfengine.com/lts/reference-functions-classesmatching.html)) returning classes and variables in different order than in past releases. Such functions do not define the order of which the data is returned, so it may change in any release. If you depend on ordering, please use data manipulation functions like [sort()](https://docs.cfengine.com/lts/reference-functions-sort.html). The [new packages promise in 3.7](https://docs.cfengine.com/lts/reference-promise-types-packages.html) and later versions have received several usability enhancements in 3.7.3 and 3.8.2. Most notably, the verbosity of the promise type during normal operation has been lowered, so that users no longer get "spammed" when there is no error in execution. 3.6.7 includes backports of relevant issue-fixes that have been incorporated into 3.7 and 3.8 since the [release of 3.6.6 in July 2015](/company/blog-detail/cfengine-3-6-6-released-backend-performance-and-reliability/). For more details, please refer to the 3.6 ChangeLog for [Core](/docs/3.6/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise](/docs/3.6/guide-latest-release-whatsnew-changelog-enterprise.html). The bundled dependencies of all three CFEngine releases have been upgraded to bring in the latest security, performance and reliability improvements. For more details on the exact dependencies that are bundled with CFEngine, please refer to the Core ChangeLog for [3.7](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html), [3.6](https://docs.cfengine.com/docs/3.6/guide-latest-release-whatsnew-changelog-core.html) and [3.8](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-core.html) (search for dependencies). These ChangeLogs also include details of other improvements that have been made. ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the upgrade documentation for [3.7](/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) or [3.6](/docs/3.6/guide-installation-and-configuration-upgrading-to-36.html) for guidelines to make the process as smooth as possible. We are happy to assist CFEngine Enterprise customers with upgrading\! Please [contact CFEngine customer support](https://cfengine.zendesk.com/hc/en-us) to receive a fixed-price quote for upgrading your CFEngine infrastructure. ## Get it\! CFEngine Enterprise packages [can be downloaded here](https://cfengine.com/download/) or you can give 3.7.3 a quick spin with the [CFEngine Enterprise Vagrant environment](/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). Community Edition is released as [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/) and [Linux package repositories](/product/community/cfengine-linux-distros/) -- to make installation as easy as possible\! We hope you enjoy the new releases, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Q&A with eZuce's Michael Picher" url: /blog/2016/qa-with-ezuces-michael-picher/index.md date: "2016-03-30" -------------------------------------------------------------------------------- CFEngine is one of the most widely used infrastructure automation solutions in the world, being leveraged by some of the biggest organizations in the world for infrastructure management. **eZuce** is an Enterprise provider of Unified Communications, Voice and Video Conferencing solutions and is being used by many of their customers to improve business processes. eZuce has been shipping CFEngine inside of UniteMe since 2012 and have benefited from the following by using the popular configuration management solution: - An easy to understand scripting language to modify the large amount of configuration files - Easy and secure communications between servers - Leveraging another open source project within eZuce's open source project. We did a brief Q\&A with Michael Picher, VP of Product Innovation at eZuce, on their usage of CFEngine: *CFE: Why did your team decide to use CFEngine as opposed to other infrastructure automation tools?* MP: CFEngine was chosen for its lightweight footprint, and because it allows developers and end customers to easily modify existing policy and create their own recipes. *CFE: What is the most obvious value (product delivery/product-to-market) achieved with CFEngine?* MP: Implementing CFEngine facilitated reliable scaling for UniteMe clustering. After implementing cluster management with CFEngine cluster sizes were able to grow to more than double previous limits while remaining more stable due to lower system overhead. Addition of new nodes and scaling systems into the cloud was also greatly simplified. CFEngine was also able to provide automated management for services that previously had to be manually configured by customers. Now we are able to expose systems configuration in a friendly way via our user interface. *CFE: What would you recommend to potential users of CFEngine in getting started?* MP: Definitely check out the Getting Started section on CFEngine's web site. There are tutorials, examples and some excellent blog articles to help give you a leg-up on your project. *Find out more about eZuce at [ezuce.com](https://web.archive.org/web/20190905055205/https://ezuce.com/).* -------------------------------------------------------------------------------- title: "Migration of our issue tracker to JIRA" url: /blog/2016/moving-to-a-new-ticket-system-jira/index.md date: "2016-03-16" -------------------------------------------------------------------------------- **Update Mar 23, 2016:** We are happy to announce that the transition to JIRA has been successful. The new URL is . **All existing issues from Redmine have been transferred to JIRA. Redirects are in place for the old URLs. All users have been migrated, however please be aware you must request a new password the first time you login.** As you can read from the URL, CFEngine is now a project under Mender Software AS. Mender Software is the name of the Norwegian holding company that is the owner of CFEngine. For several years we had been using Redmine to track CFEngine issues and bugs. Redmine served us well, but after thorough evaluation we found that time has come to move on to JIRA. The main benefits for this shift are: - Community can vote on issues - Improved integration with our internal development process - Improved reporting capabilities We hope the move to JIRA will increase community participation and improve communication between internal and external CFEngine contributors. With this we also expect a more transparent planning and tracking of issues, which in return will lead to more efficient and predictable decisions and prioritizations. Overall we believe this move will make our engineering and development process more efficient. We look forward to seeing you all on our new ticketing system. Follow also our [Twitter account](http://Twitter%20account) for updates. -------------------------------------------------------------------------------- title: "CFEngine at Configuration Management Camp 2016" url: /blog/2016/cfengine-was-in-configuration-management-camp-2016/index.md date: "2016-02-16" -------------------------------------------------------------------------------- ![cfgmgmtcamp2016-logo](images/cfgmgmtcamp2016-logo.png)CFEngine AS was present in the [Configuration Management Camp](https://cfgmgmtcamp.eu) that took place in Gent, Belgium the 1st and 2nd February of 2016. This is **the** event on open source configuration management tools, scheduled immediately following [FOSDEM](https://fosdem.org/) and is located close to Brussels, so it usually has thousands of attendants. This year's [main track topics](https://cfgmgmtcamp.org/ghent2016/schedule/) mostly revolved around **security**, **orchestration**, and **application containers**. We had a separate room for [CFEngine](https://cfengine.com/) and [Rudder](https://rudder-project.org/) related topics with an exciting schedule: | | | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Monday, February 1 | | | 14:00 | [CFEngine Champions Panel](https://cfgmgmtcamp.org/ghent2016/schedule/cfengine-rudder/pannel) | | 14:40 | [Security Practices with CFEngine](https://cfengine.com/download-file/8635/) | | 15:20 | Break | | 15:40 | [Testing Policy and the Core test framework](https://github.com/nickanderson/presentation-testing-cfengine-policy/tree/af051f65d16910e95d935e1cb881f11e40cd4276) | | 16:20 | [Messaging CFEngine and Data](https://www.slideshare.net/webhuis/config-managementcampgent-2016) | | Tuesday, February 2 | | | 14:00 | [The CFEngine Roadshow BYOVM](https://cfgmgmtcamp.org/ghent2016/schedule/cfengine-rudder/roadshow) | | 14:40 | [Using ncf building blocks to help writing CFEngine policies](https://www.slideshare.net/normation/using-ncf-building-blocks-to-help-writing-cfengine-policies) | | 15:20 | Break | | 15:40 | [Integrating Rudder and CFEngine Mission Portal](https://www.slideshare.net/normation/integrating-rudder-and-cfengine-mission-portal) | Effectively leveraging data from external sources and best practices with regard to policy style were popular topics in the "_CFEngine Champions Panel"_. _"Security Practices with CFEngine"_ discussed at a high level various ways of distributing policy in a complex infrastructure and ways to minimize the impact of a security breach. The session concluded by demonstrating CFEngine policy both score and automatically re-mediate industry standard Center for Internet Security [CIS benchmarks](https://benchmarks.cisecurity.org/downloads/benchmarks/). https://www.slideshare.net/slideshow/embed\_code/58317897 _"Testing Policy and the Core test framework_" discussed what people are currently doing to test their CFEngine policy. It covered at a high level how the [CFEngine Core acceptance test framework](https://github.com/cfengine/core/tree/master/tests/acceptance) works and how contributions to core examples can both improve the documentation as well as add additional test coverage. https://www.slideshare.net/slideshow/embed\_code/58318232 During _"Messaging CFEngine and Data"_ Martin Simons demonstrated integrating CFEngine agents with a _ZeroMQ_ message bus. Clients reported their discovered information and hosts using the nagios role used the reported data to automatically configure nagios checks and alerts for the services in use. _"The CFEngine Roadshow_" presentation given by Martin Simons was a very neat interactive demo. Participants were able to spin up fresh virtual machines on their own laptops and after classifying the host by setting its hostname the host was automatically configured with users, ssh keys, and services as defined by the roadshow policy. Alexis Mousset and Benoit Peccatte presented on _ncf_ which is a framework from [Normation](https://www.normation.com) for CFEngine policy that favours readability over all else. They demoed the _ncf builder_, a web interface that can use the create and edit policies composed of generic methods provided with ncf. Nicolas Charles demonstrated "_Integrating Rudder and CFEngine Mission Portal"_ bringing the codeless automation and detailed compliance reporting from under the same roof, by combining the powers of both [Rudder](https://www.rudder-project.org) and [CFEngine Enterprise](https://cfengine.com/product/). -------------------------------------------------------------------------------- title: "CFEngine 3.8.1 non-LTS released" url: /blog/2016/cfengine-3-8-1-non-lts-released/index.md date: "2016-02-01" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.8.1 non-LTS now is released\! Please note that this is a non-LTS release, which means that it is maintained for 6 months from the minor version's (3.8.0 in this case) release date and not supported for CFEngine Enterprise customers (but packages are available for testing). The established [CFEngine release schedule](/product/supported-versions/) gives an overview over the release timelines for all releases. ## Changes and improvements The [def.json feature for overriding CFEngine default configuration](/company/blog-detail/modular-policies-with-autorun-and-the-augments_file/) was introduced in 3.7 and has since seen significant adoption. The main goal of this feature is to make upgrades easier, as it makes the difference between vanilla masterfiles and user masterfiles much easier to manage. However, several users have reported unexpected behaviour with the feature, because some of the variables in def.json were not parsed early enough to be taken into account by CFEngine. This triggered several discussions, also by the [CFEngine Community Advisory Board](https://github.com/cfengine/cfengine-community-advisory-board/blob/master/minutes/2015-10-14.org#defjsonaugments-file-from-c). A new design has been introduced to parse def.json natively in C, which should resolve this issue. If you had this problem, please let us know if it is resolved in 3.8.1\! All bundled dependencies have been upgraded to their latest version to bring in the latest security, performance and reliability improvements. All 3.8.1 CFEngine installations include these key dependencies, among others: OpenSSL 1.0.2e, PCRE 8.38, libxml2 2.9.3, OpenLDAP 2.4.43, libcurl 7.46.0, LMDB 0.9.17. CFEngine Enterprise Policy Servers also have upgraded dependencies and include PHP 5.6.17, Redis 3.0.6, git 2.6.5. [OpenSSL recently published a security advisory](https://openssl.org/news/secadv/20160128.txt), but CFEngine is not affected because neither DH nor DHE key exchange can be used by CFEngine. It is probably a good idea to check other products you use, though. Please see the [Community Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-enterprise.html) for a detailed list of new features and changes. We highly appreciate community contributors for pull requests that made it in time for 3.8.1\! ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.8/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. ## Get it\! If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), and [packages](https://cfengine.com/community/download/) for 3.8.1\! If you are an Enterprise customer, please note that [3.8 is a non-LTS release and thus not covered under the support contract](/product/supported-versions/). If you would like to test 3.8.1, the Enterprise packages [can be downloaded here](https://cfengine.com/download/). We hope you enjoy 3.8.1, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Release theme for CFEngine 3.9: Logging" url: /blog/2016/release-theme-for-cfengine-3-9-logging/index.md date: "2016-01-16" -------------------------------------------------------------------------------- **UPDATE, 2016-02-22**: After [feedback from CFEngine users](https://groups.google.com/forum/#!topic/help-cfengine/rgn1cQf6f6Q) and several discussions around a [variable expansion performance issue](https://dev.cfengine.com/issues/7631), we decided to put this expansion issue higher on the priority list for 3.9. We still plan to address some high-impact logging issues for 3.9, most notably adding [a line-level email-filtering feature](https://dev.cfengine.com/issues/7938). With [CFEngine 3.8 released on target in December](/company/blog-detail/cfengine-3-8-0-non-lts-released/), and we've entered 2016 it's time to look ahead to CFEngine 3.9\! According to the [CFEngine release schedule](/product/supported-versions/), 3.9 will be a non-LTS release and is due for June 2016. If you're considering to *contribute a major change or feature to CFEngine* (thank you\!) and would like to see it in 3.9, please note that we have a beta period for one month, so your pull request should be ready (i.e. reviewed and any adjustments made) in *early April* in order to make it. We'd also like to share that *logging* will be the release theme for 3.9\! ## Provide your feedback on logging\! As a theme is more abstract than features and [CFEngine releases are now time-driven](/company/blog-detail/releases-after-3-7-0/), there is no exhaustive list of logging features or improvements that will be included in 3.9. Rather, the goal is to find the most valuable features and ideas to work on based on input from CFEngine users, so please make sure to make your voice heard if you have ideas (the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine) is a good place to start a discussion)\! There is [an issue in the bugtracker](https://dev.cfengine.com/issues/5430) discussing some ideas for client-side log filtering and destinations (console, syslog, files). If you think CFEngine is too verbose or not verbose enough in certain circumstances, this is a good issue to watch and participate in. It also has several related issues with ideas relevant to logging. ## What this means for you The obvious answer is that you will see improvements in logging for 3.9. However, if you are interested in [contributing to CFEngine](https://github.com/cfengine/core/blob/master/HACKING.md), we would like to encourage to also think about any logging-related contributions you might be interested in. Also please note the *early April 2016* cutoff for being able to process pull requests for 3.9. Of course, we will still maintain and improve the stability of CFEngine, so any bugs will continue to be addressed. The logging theme is driven by user input, for example there have been several logging-related discussions in the community lately. We therefore hope you feel this is relevant to you and look forward to incorporating your input in the issues above, or in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Congratulations 2015 CFEngine Champions" url: /blog/2016/congratulations-2015-cfengine-champions/index.md date: "2016-01-13" -------------------------------------------------------------------------------- Each year we like to take a moment to recognize outstanding community members for their contributions. Contributions come in the form of code contributions to core, organizing community meet-ups, giving talks about CFEngine at conferences, sharing policy, to helping other users on the mailing list and on IRC. This year the [Community Advisory Board](https://cfengine.com/company/blog-detail/announcing-the-cfengine-community-advisory-board/) was responsible for selecting champions from the nominations, and it is my honor to announce the 2015 [CFEngine Champion Hall of Fame](https://cfengine.com/engage/cfengine-champions/) inductees. Congratulations, and thanks for all of your efforts\! ## Jon Henrik Bjørnstad ![](images/jh_bio_200x200.jpg) [Jon Henrik](https://www.linkedin.com/in/jonhenrikbjornstad) co-founded CFEngine in 2008 and worked as Infrastructure Engineer and Product Manager until he left the company to pursue other opportunities in 2013. He holds a MSc. in Network and System Administration and has 10 years of professional experience. Apart from his employment in CFEngine he has been working mostly as a System Administrator within Finance, Oil & Gas and manufacturing. He also co-founded [www.cfengineers.net](https://web.archive.org/web/20180411143513/http://cfengineers.net/) in 2013 which mainly contributes to the community by providing CFEngine packages and tools. He lives in Oslo, Norway with his girlfriend and 2 kids. ## William Orr ![](images/will_orr_200x200.jpg) [Will](https://www.linkedin.com/in/willorr) ([@worr](https://twitter.com/worr)) is a relatively new contributor to CFEngine's core, and a drive-by contributor to many other open source projects. He's known within CFEngine largely for taking care of many of the 4k problems, adding global keys to bodies and hanging out on the dev mailing list. Will works as a Senior Systems Engineer at LinkedIn, largely responsible for developing tooling and automation. At work, he evangelizes use of and contribution back to open source projects. Since starting to work on CFEngine, another LinkedIn employee has contributed to CFEngine and another is getting bootstrapped. Will graduated from Rochester Institute of Technology with a BS in Applied Networking and Systems Administration in 2013. In his spare time, he works on various FOSS projects, hacks on kernels and makes dumb computer jokes on Twitter. ## Benoit Peccatte ![](images/Benoit-200x200.jpg) [Benoit Peccatte](https://www.linkedin.com/in/benoit-peccatte-09aaa21) ([@peckpeck](https://twitter.com/peckpeck)) lives in Paris, he has worked as a sysadmin, developer, and security engineer for several years where he had the opportunity to work with CFEngine v2 and v3. He returned to his preferred field of system administration, where he has been almost exclusively using and hacking open source tools for the past 8 years. He is now employed by Normation to improve [Rudder](https://www.rudder-project.org) by combining these experiences. -------------------------------------------------------------------------------- title: "CFEngine 3.8.0 non-LTS released!" url: /blog/2015/cfengine-3-8-0-non-lts-released/index.md date: "2015-12-05" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.8.0 non-LTS now is released\! A big thanks to everyone testing the [3.8.0 beta release](/company/blog-detail/cfengine-3-8-0-non-lts-beta-is-ready-for-testing/)\! During the month it has been available, the 3.8.0 beta has seen hundreds of downloads widely distributed across platforms, both in Community and Enterprise editions. This testing helps ensure high quality of the final release. Of course, bugfixes in [recently released 3.7.2](/company/blog-detail/cfengine-3-7-2-released-policy-distribution-reliability/) are also incorporated into 3.8.0. Please note that this is a non-LTS release, which means that it is maintained for 6 months from the release date and not supported for CFEngine Enterprise customers (but packages are available for testing). The established [CFEngine release schedule](/product/supported-versions/) gives an overview over the release timelines for all releases. ## Changes and improvements 3.8 brings numerous new features for the policy language, most notably http support with the [url_get() function](https://docs.cfengine.com/docs/3.8/reference-functions-url_get.html), [body inheritance](https://docs.cfengine.com/docs/3.8/reference-promise-types.html#inherit_from), and [support for persistent classes through the module protocol](https://docs.cfengine.com/docs/3.8/reference-promise-types-commands.html#module). The locations of log files have been cleaned up and standardized under /var/cfengine/log. Furthermore, some unnecessary log files like cf3.$(hostname -s).runlog have been removed. This leads to a significant performance improvement for storage access and makes it easier to work with the remaining log files, e.g. to mount them to separate partitions. All packaged dependencies have been upgraded to their latest version to bring in the latest security, performance and reliability improvements. In order to support the [url_get() function](https://docs.cfengine.com/docs/3.8/reference-functions-url_get.html), libcurl is added as a library packaged with CFEngine 3.8 and future versions, which also enables http support for other interesting features in the future\! Please see the [Community Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-enterprise.html) for a detailed list of new features and changes. We highly appreciate community contributors for pull requests that made it in time for 3.8, and we're looking forward to incorporating the pending ones into the 3.9 release\! ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.8/guide-installation-and-configuration-upgrading.html) for guidelines to make the process as smooth as possible. ## Get it\! If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), and [packages](https://cfengine.com/community/download/) for 3.8.0\! If you are an Enterprise customer, please note that [3.8 is a non-LTS release and thus not covered under the support contract](/product/supported-versions/). If you would like to test 3.8.0, the Enterprise packages [can be downloaded here](https://cfengine.com/download/). We hope you enjoy 3.8.0, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.2 released: policy distribution reliability" url: /blog/2015/cfengine-3-7-2-released-policy-distribution-reliability/index.md date: "2015-12-02" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.7.2 is released\! With 3.7 being a [stable LTS branch](/product/supported-versions/), 3.7.2 brings numerous stability enhancements and bugfixes to the CFEngine customers and community. The main focus area for 3.7.2 is to improve fault-tolerance and performance of policy distribution in unreliable networks. ## Enhanced reliability of policy distribution The ability to distribute policy from the Policy Server to clients is a critical function of CFEngine, which also must work well in unreliable networks. CFEngine already has protection against corruption of the main policy (promises.cf) with the separate update policy (update.cf). In turn, if the update policy fails validation, failsafe.cf is run. In 3.7.2 failsafe.cf is also re-generated if it does not exist (but not overwritten since users may have a custom failsafe.cf in some cases) -- basically simulating a fresh bootstrap. So this means that CFEngine will now be able to recover from corruption of any policy and wipe of the entire inputs directory\! In addition, we have been chasing [an intermittent issue](https://dev.cfengine.com/issues/6027) that in some extremely rare cases results in a directory being turned into a file for some time, and we now have evidence that this is completely fixed in 3.7.2\! Performance of policy distribution with cf-serverd at scale (4000+ clients) has been significantly improved by reducing [lock contention of malloc()](https://dev.cfengine.com/issues/7624) as well as [lock contention of getpwnam()](https://dev.cfengine.com/issues/7643). Finally, for CFEngine Enterprise installations using call collect (client-initiated reporting), the default [collect_window](https://docs.cfengine.com/docs/3.7/reference-components-cf-serverd.html#collect_window) has been increased from 10 to 30 seconds. This ensures call-collect works reliably in scaled environments (thousands of clients) with default configuration. In sum these changes will lead to much more reliable policy distribution -- and hopefully provide users with some additional peace of mind\! ## Other improvements File management features have gotten numerous enhancements. File-editing now allows [editing the same value in multiple regions](https://dev.cfengine.com/issues/7460). The log level of several important messages has been changed to error, including [readfile() on files not found](https://dev.cfengine.com/issues/7702) as well as errors saving mustache templates. Also, [unnecessary error messages](https://dev.cfengine.com/issues/6996) when using file_select.file_types =\> "symlink" have been removed. CFEngine Enterprise 3.7.2 has gotten several reporting-related fixes. Emailing of CSV-only reports has been fixed and the CSV-only reports are now saved in the same directory as the PDF reports (/var/cfengine/httpd/htdocs/tmp). Exporting CSV reports over https did not work in 3.7.0 and 3.7.1, but this has also been fixed. Finally, CFEngine Enterprise no longer produces an error message if software inventory is not available on a platform, which commonly occurs on Solaris and AIX. CFEngine packages have been made more robust. To ensure reliable operation, CFEngine packages include all library dependencies. In cases where there are system libraries installed as well, 3.7.2 has improvements to make sure that only the bundled libraries are loaded by CFEngine - not the system libraries. Solaris installations now use significantly less space because [library symlinks are used instead of copies](https://dev.cfengine.com/issues/7591) and a (harmless) installation error message has been fixed on AIX. Library dependencies have been upgraded in the agent and hub packages in order to improve security and reliability. On the Policy Server side CFEngine Enterprise 3.7.2 comes with Apache 2.2.29, PHP 5.6.14, PostgreSQL 9.3.9 and Redis 2.8.23, while community and agents have OpenSSL 1.0.2d and LMDB 0.9.14 as key library versions. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading an existing CFEngine Enterprise installation, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. ## Get it\! As always, you can [download CFEngine Enterprise 3.7.2 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.7/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.7.2 vagrant environment](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.7.2, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.8.0 non-LTS beta is ready for testing" url: /blog/2015/cfengine-3-8-0-non-lts-beta-is-ready-for-testing/index.md date: "2015-11-11" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.8.0 non-LTS beta is now ready for testing\! The established [CFEngine release schedule](/product/supported-versions/) shows that the 3.8.0 final version is due before January 2016, so it's time to test and fix any remaining issues. Please note that this is a non-LTS release, which means that it is maintained for 6 months from the release date and not supported for CFEngine Enterprise customers, but packages are available for testing. ## Changes and improvements 3.8 brings numerous new features for the policy language, most notably http support with the [url_get() function](https://docs.cfengine.com/docs/3.8/reference-functions-url_get.html), [body inheritance](https://docs.cfengine.com/docs/3.8/reference-promise-types.html#inherit_from), and [support for persistent classes through the module protocol](https://docs.cfengine.com/docs/3.8/reference-promise-types-commands.html#module). The locations of log files have been cleaned up and standardized under /var/cfengine/log. Furthermore, some unnecessary log files like cf3.$(hostname -s).runlog have been removed. This leads to a significant performance improvement for storage access and makes it easier to work with the remaining log files, e.g. to mount them to separate partitions. All packaged dependencies have been upgraded to their latest version to bring in the latest security, performance and reliability improvements. Please see the [Community Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.8/guide-latest-release-whatsnew-changelog-enterprise.html) for a detailed list of new features and changes. ## How to provide feedback The main goal of releasing the beta is to make sure that 3.8.0 works to your expectations in your environment. So if you get the chance to test out some of the features or perhaps the compatibility with your existing policy, please let us know how it went\! The best way to provide feedback is to submit issues relating to the beta in [the issue tracker.](https://dev.cfengine.com) Please make sure to set the "Found in version" field to "3.8.0 beta". For general questions and discussions, you can also use the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine) or the [\#cfengine](https://web.libera.chat/?channel=#cfengine) and [\#cfengine-dev](https://web.libera.chat/?channel=#cfengine-dev) chat rooms on libera.chat. ## Get it\! You can download the CFEngine [community 3.8.0 beta packages and source code](https://cfengine.com/product/community/). If you are an Enterprise customer, please note that [3.8 is a non-LTS release and thus not covered under the support contract](/product/supported-versions/). If you would like to test 3.8, the Enterprise beta packages [can be downloaded here](https://cfengine.com/download/). Please note that this is a **beta release**, so we do not recommend installing it in a production environment. We hope you enjoy the new features, and we look forward to hearing about your experience\! ## Known issues - Inability to log into Mission Portal over https - Enterprise Windows package is broken - Call collect is broken - AIX does not support alphabetic versions, so 3.8.0 Enterprise beta has version "3.8.99X" (where X is beta build number) on AIX. Final AIX release will have version "3.8.0" like all other platforms. -------------------------------------------------------------------------------- title: "Top 15 most valuable reports in CFEngine Enterprise" url: /blog/2015/top-15-most-valuable-reports-in-cfengine-enterprise/index.md date: "2015-09-18" -------------------------------------------------------------------------------- There is a lot of value to be gained from CFEngine's reporting features, so we've put together a slideshow to showcase 15 of the top reports that give you the best information and control over your infrastructure. If you like what you see, please contact us for any assistance in creating these reports yourself. https://www.slideshare.net/slideshow/embed\_code/key/NEiaPxUfGWdsdw
**[15 most valuable reports with CFEngine](//www.slideshare.net/cfengine/15-nice-reports-with-cf-engine "15 most valuable reports with CFEngine")** from **[CFEngine](//www.slideshare.net/cfengine)**
-------------------------------------------------------------------------------- title: "CFEngine 3.7.1 released: stability enhancements" url: /blog/2015/cfengine-3-7-1-released-stability/index.md date: "2015-09-09" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.7.1 is now ready\! Given that this is the first maintenance release in the 3.7 branch, the focus is primarily on stability and reliability of new features added in 3.7.0. For improved resiliency of Enterprise High Availability in environments that need it, we also added support for an offsite replication node. ## Offsite replication support for Enterprise High Availability The High Availability cluster has until now consisted of 2 nodes; an active and passive. If the active goes down, the clients would fail over to the passive. However, some environments require the CFEngine Server to continue to operate even during a complete datacenter failure. In 3.7.1, support for a third HA node has been added: an offsite replication node. This node will replicate data from the active node, and can be manually promoted to an active node in case both the two other nodes go down. You can read more in the [new section of the High Availability installation documentation](https://docs.cfengine.com/latest/enterprise-cfengine-guide-high-availability-installation-guide.html#configuring-3rd-node-as-disaster-recovery-or-database-backup-optional). ## Stability enhancements The new [3.7 packages promise](https://docs.cfengine.com/docs/3.7/reference-promise-types-packages.html) has gotten a few minor behavioural changes, based on user feedback. Most notably, if you use policy =\> "absent", it will in 3.7.1 remove dependencies required to complete a removal as well. In 3.7.0, it would stop if there were any dependencies that needed to be removed. The init scripts have gotten extra checks to ensure all CFEngine processes are stopped if the "stop" state is issued. This will alleviate certain situations when upgrading from 3.7.1 to future versions, which could leave several CFEngine processes of the same type running. Dependencies have been upgraded in the pre-compiled agent and hub packages in order to improve security and reliability. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading from a previous release, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. ## Get it\! As always, you can [download CFEngine Enterprise 3.7.1 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.7/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.7.1 vagrant environment](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.7.1, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Release theme for CFEngine 3.8: performance" url: /blog/2015/release-theme-for-cfengine-3-8-performance/index.md date: "2015-08-06" -------------------------------------------------------------------------------- Now that [CFEngine 3.7.0 is released](https://cfengine.com/company/blog-detail/cfengine-3-7-0-released-new-package-promise-and-change-reporting/), introducing simplified package management, change management, dashboard sharing and enhanced network security, it's time to look forward. The CFEngine 3.7 and 3.6 releases will mainly include stability and security enhancements as they are now stable branches. [A blog-post on the release schedule going forward](https://cfengine.com/company/blog-detail/releases-after-3-7-0/) explains this in more detail. The next feature-release up is 3.8 (non-LTS), due for December 2015, and we'd like to share that *performance* will be the main release theme for 3.8\! Secondly, we also plan to include support for *phased rollout* in 3.8. ## Key performance metrics CFEngine is typically deployed in a client-server fashion, where the Policy Server is a dedicated machine that is responsible for delivering the policy to the clients. So conceptually there are two types of CFEngine roles we could look at optimizing; the Policy Server or the client. However, as CFEngine clients are installed on nodes that do the valuable work of running application services, most users care the most about the *performance of the client* -- so this is what we'll focus on for 3.8. There are several metrics that we will measure and improve for 3.8, in prioritized order: - **Disk writes**. This is very important in situations where CFEngine clients use shared storage; such as in OpenStack and other cloud environments. It is also important for lifetime of flash storage. - **CPU usage**. CFEngine should let applications use as much of the CPU as possible. This is especially important in container environments. - **Memory usage**. Likewise; as much memory as possible should be available to applications. - **Disk usage**. Installation and run-time storage usage of CFEngine should be as low as possible; both for binaries and state data. The first step is to measure status quo for these metrics with 3.7.0. We then plan to profile and optimize the areas that would benefit the most. We will know more hard numbers on what to expect in 3.8 once the measurements have been done; so [watch for updates and contribute to improving client performance](https://dev.cfengine.com/issues/7434)\! ## Phased rollout Most Enterprise installations use some concept of environments in order to test changes before they're released to production; normally dev, test and prod environments are used. As CFEngine is one of the key tools used to change Enterprise systems, we would like to better incorporate such workflows into CFEngine. There are a number of considerations to take into account; both with respect to the size and definition of the phases (do you want to release changes to all of production at once?), as well as the workflow (integration with version control systems) and isolation (is anything shared at all or is the policy and infrastructure completely isolated?). In general, choosing different aspects of these considerations generates tradeoffs with respect to ease of management and security/isolation. For example, if everything is shared and released everywhere at once then it is easy to manage but the isolation properties are poor. Which form this will take in CFEngine is something we will discover from your input; so if this is interesting make sure to [watch for updates and contribute to phased rollout](https://dev.cfengine.com/issues/7433)\! ## What this means for you The obvious answer is that CFEngine users will see improvements in client performance and better management of rollouts for 3.8. However, if you are interested in [contributing to CFEngine](https://github.com/cfengine/core/blob/master/HACKING.md), we wanted to share the plans in order to improve collaboration with the CFEngine community. For example, it is more likely we can prioritize [an issue report](https://dev.cfengine.com) or pull request (be it in CFEngine [core](https://github.com/cfengine/core) or [masterfiles](https://github.com/cfengine/masterfiles/)) that improves CPU utilization over one that adds support for a new promise type. Of course, we will still maintain and improve the stability of CFEngine, so any bugs will continue to be addressed. We hope you feel this is relevant and are happy to see your input in either of the two issues above, or in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Modular policies with autorun and the augments_file" url: /blog/2015/modular-policies-with-autorun-and-the-augments_file/index.md date: "2015-07-26" -------------------------------------------------------------------------------- 1, 2, 3.7 GO\! CFEngine [3.7 was released](https://cfengine.com/company/blog-detail/cfengine-3-7-0-released-new-package-promise-and-change-reporting/) just over a week ago and one of the neat things with 3.7 is the new [augments\_file](https://docs.cfengine.com/docs/3.7/guide-writing-and-serving-policy-policy-framework.html#augments_file-variable) also known as [def.json](https://github.com/cfengine/masterfiles/blob/3.7.0/example_def.json) or [overrides](https://github.com/cfengine/masterfiles/blob/3.7.0/controls/3.7/def.cf#L38). What's so neat about it? It's going to make your future policy upgrades easier\! I will be using the [CFEngine Enterprise Vagrant Environment](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html) because it's a really quick and easy way to stand up a test environment. Here is my fresh 3.7.1 environment. [![](images/2015-07-26_Selection_006-800x206.png)](images/2015-07-26_Selection_006.png) Looks like my instances are ready to go [![](images/2015-11-09_Selection_001-800x122.png)](images/2015-11-09_Selection_001.png) Show CFEngine Version First I log into the hub and wheel up to root. vagrant ssh hub sudo -i I need something to augment inputs with. I will integrate some policy that does inventory for [SSH Host Key Fingerprints](https://github.com/nickanderson/cfengine-ssh#inventory_ssh_host_key_fingerprints). # Add some additional Policy that needs included into inputs export COMMIT=e1b7b4646dbcecfe35cb3c7471401850a29909ac wget https://github.com/nickanderson/cfengine-ssh/archive/$COMMIT.tar.gz -O /tmp/$COMMIT.tar.gz tar -C /tmp -zxf /tmp/$COMMIT.tar.gz rm -f /tmp/$COMMIT.tar.gz mv /tmp/cfengine-ssh-$COMMIT /var/cfengine/masterfiles/services/ssh Next I create my augments\_file (/var/cfengine/masterfiles/def.json) with the following content. { "classes": { "services_autorun": [ "any" ] }, "inputs": [ "services/ssh/def.cf" ] } If you're looking for a more full example see [example\_def.json](https://github.com/cfengine/masterfiles/blob/3.7.0/example_def.json) from the masterfiles repository. I just enabled the [autorun feature](https://docs.cfengine.com/docs/3.7/guide-writing-and-serving-policy-policy-framework.html#services_autorun-bundle) **and** added the entry to my ssh policy to inputs without touching any files shipped in the masterfiles policy framework (minus that patch that is already upstream), now I just need to make sure my bundles get activated. I am going to do that by creating /var/cfengine/masterfiles/services/autorun/ssh.cf with the following content. bundle agent ssh_autorun { meta: "tags" slist => { "autorun" }; methods: linux:: "SSH Host Key Fingerprint Inventory" usebundle => inventory_ssh_host_key_fingerprints; am_policy_hub.enterprise_edition:: "Inventory Duplicate SSH Host Key Fingerprints" usebundle => inventory_duplicate_ssh_host_key_fingerprints; } Further Reading: [Brian Bennett](https://www.linkedin.com/in/brianbennettsd) has a [nice write up](https://digitalelf.net/2014/07/a-primer-on-cfengine-3-dot-6-autorun/) on how he uses autorun, Bernt Jernberg also contributed a nice piece on how to [use autorun with meta tags and hard classes](https://cfengine.com/company/blog-detail/dynamic-bundlesequence-with-autorun-meta-tags-and-hard-classes/). I double check that my policy validates after the changes. cf-promises -cf /var/cfengine/masterfiles/promises.cf After a few minutes I can see the new inventory attributes in Mission Portal. [![](images/2015-07-26_Selection_003-800x356.png)](images/2015-07-26_Selection_003.png) It inventories each found host fingerprint as well as all of the known host key types. [![Inventory SSH Host Fingerprint (DSA)](images/2015-07-26_Selection_005-800x356.png)](images/2015-07-26_Selection_005.png) -------------------------------------------------------------------------------- title: "CFEngine 3.6.6 released: backend performance and reliability" url: /blog/2015/cfengine-3-6-6-released-backend-performance-and-reliability/index.md date: "2015-07-20" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.6.6 is now ready\! The new release has improved performance and reliability, especially on the CFEngine Enterprise backend, as well as a stability and performance improvements across Unix and Windows agent platforms\! Given that this is the sixth maintenance release in the 3.6 branch, the focus is primarily on stability and performance enhancements. ## Enterprise report collection enhancements CFEngine 3.6.6 significantly improves the performance and reliability of the Enterprise reporting backend in several ways. A [known issue](https://docs.cfengine.com/docs/3.6/guide-latest-release-known-issues.html#enterprise-mission-portal-is-slow-and-or-var-cfengine-state-pg-consumes-a-lot-of-space-iops-or-cpu-utilization-is-high) causing higher CPU usage of PostgreSQL has been resolved. Secondly, the PostgreSQL maintenance settings (known as "vacuuming") have been optimized to reduce the disk fragmentation and thus supporting policies with frequently changing promises much better. For example, if there are frequent uses of if_elapsed in the policy, this optimization will make a significant difference. In some instances, the Enterprise hub would log errors about "status_pkey" to syslog, due to duplicate reports coming from clients. This has also been resolved in 3.6.6. Finally, environments where client-initiated reporting is enabled (known as "call collect") will see big scalability enhancements. Call collect has been scale tested to several thousand clients over a long period of time. ## Other changes Windows process matching has been optimized to use internal WMI calls instead of relying on the wmic binary. This not only makes process listing more reliable, but also much more performant as CFEngine will no longer need to create additional processes every time a process table is needed. Dependencies like OpenSSL, Apache and PHP have been upgraded to their latest maintenance release to take advantage of new stability and security fixes. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/docs/3.6/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.6/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading from a previous release, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-upgrading-to-36.html) for guidelines to make the process as smooth as possible. ## Get it\! As always, you can [download CFEngine Enterprise 3.6.6 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.6/guide-latest-release-supported-platforms.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/) and [Linux packages](https://cfengine.com/community/download/)\! We hope you enjoy 3.6.6, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.0 released: New package promise and change reporting" url: /blog/2015/cfengine-3-7-0-released-new-package-promise-and-change-reporting/index.md date: "2015-07-17" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.7.0 is now ready\! The 3.7.0 release contains a brand new packages promise, expanded platform support, enhanced network security, improved Enterprise reporting capabilities and much more\! ## New packages promise A new packages promise has been developed in collaboration with the CFEngine community and users. It is designed to be reliable, simple and easy to use. We hope you enjoy the experience\! It reuses the same promise type as the previous packages promise (packages:), but CFEngine will determine which one to use based on the attributes that are used in the promise type. The packages: promise type is fully backward-compatible, so any packages promises that you have from 3.6 or earlier versions should still work with 3.7. Currently supported platforms for the new packages promise include those based on yum/rpm (using package_module =\> yum) and apt/deb (using package_module =\> apt_get), but it can easily be extended by adding new package modules. Package modules are essentially wrappers for the package managers that implement the CFEngine package module protocol. As an example, you can use the following promise in 3.7 to track the latest package of apache on Red Hat systems: ` packages: "httpd" policy => "present", version => "latest", package_module => yum;` You can read more about the new packages promise type in the [packages reference documentation](https://docs.cfengine.com/docs/3.7/reference-promise-types-packages.html). ## Expanded platform support CFEngine 3.7 improves the support for RHEL 7 and CentOS 7. These platforms were already supported as agent platforms, but the rpm packages have now been unified so that there is a single agent RPM that works across all of these platform versions. These platforms can now run CFEngine Enterprise hubs as well. In addition, both Debian 7 and 8 have been added to the supported platform list; both as Enterprise hubs and agents. In general, systemd-enabled distributions are now fully supported. You can see the full list of supported platforms for 3.7 in the [Supported Platforms and Versions documentation page](https://docs.cfengine.com/docs/3.7/guide-latest-release-supported-platforms.html). ## Enterprise multi-dashboard support The Enterprise edition has new features for creating and managing multiple dashboards\! In order to improve collaboration with your team, you can also share dashboards with your colleagues and see dashboards that your colleagues have shared. [![](images/dashboard-multiple-flow-3.7.0.png)](/blog/2015/images/dashboard-multiple-flow-3.7.0.png) ## Enterprise change reporting CFEngine Enterprise users will also see the new Changes reporting feature; both the dashboard widget and reporting interface for displaying and filtering which changes CFEngine has made to the infrastructure. [![](images/changes-reporting-flow-3.7.0.png)](/blog/2015/images/changes-reporting-flow-3.7.0.png) The change reporting features also include a fully functional REST API, in addition to the user interface. ## Network and security enhancements In untrusted networks it is now easier to bootstrap securely, by pre-establishing trust with cf-key --trust-key and then run cf-agent --bootstrap \ --trust-server=no. The CFEngine network protocol now uses TLS by default to secure the connections. Note that this has the implication that 3.7 agents cannot bootstrap to servers with version 3.5 or earlier (3.6 supports both TLS and the legacy security protocol). Which security protocol that is allowed on the server side can be controlled with allowlegacyconnects in body server control. On the client side you may use the protocol_version attribute found in body common control and body copy_from. To control which TLS version and ciphers used, new attributes have been added. On the client side, body common control has the attributes tls_min_version and tls_ciphers. The siblings for the server side are found in body server control; allowtlsversion and allowciphers. ## Other changes Dependencies have been upgraded in the pre-compiled agent and hub packages; most notably OpenSSL 1.0.2d, PHP 5.6.9, LMDB 0.9.14, PCRE 8.37, PostgreSQL 9.3.7, Redis 2.8.20 and Codeigniter 2.2.2. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/docs/3.7/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading from a previous release, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-upgrading-to-37.html) for guidelines to make the process as smooth as possible. ## Get it\! As always, you can [download CFEngine Enterprise 3.7.0 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.7/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.7.0 vagrant environment](https://docs.cfengine.com/docs/3.7/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.7.0, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Press release: CFEngine 3.7 now available" url: /blog/2015/announcing-cfengine-3-7/index.md date: "2015-07-17" -------------------------------------------------------------------------------- California, CA (Jul 17, 2015) - CFEngine, Inc., IT Automation pioneer and leader in infrastructure management software, today announced the general availability of its flagship product CFEngine Enterprise and CFEngine Community, version 3.7 ## Easier application deployment The 3.7.0 release contains a brand new technology for application deployments. According to Marcin Pasinski, a new packages deployment mechanism has been developed in collaboration with the CFEngine community and users. It is designed to be more reliable, simple and easy to use. The new technology co-exists and is compatible with the previous version of the packages promise type in CFEngine. This means it will be fully backward-compatible, so any packages promises that you have from 3.6 or earlier versions should still work with 3.7. Currently supported platforms for the new packages promise include those based on yum/rpm and apt/deb, but it can easily be extended by adding new package modules. Package modules are essentially wrappers for the package managers that implement the CFEngine package module protocol, Marcin Pasinski continues. This allows for a very modular and flexible approach, in line with the design principles of CFEngine. ## Increased platform support CFEngine 3.7 improves the support for RHEL 7 and CentOS 7. These platforms were already supported as agent platforms, but the rpm packages have now been unified so that there is a single agent RPM that works across all of these platform versions. These platforms can now run CFEngine Enterprise hubs as well. In addition, both Debian 7 and 8 have been added to the supported platform list; both as Enterprise hubs and agents. In general, systemd-enabled distributions are now fully supported. ## Easier collaboration and knowledge sharing A very popular feature of CFEngine Enterprise is the status dashboard. The dashboard allows its users to customize a single page tailored to his specific needs. Whether it be alerts of non-compliant behavior, inventory thresholds or the state of a roll-out, the dashboard gives you exactly the information needed. The dashboard is as customizable as the CFEngine language itself, allowing for indefinite setups. With 3.7 we have made it easy for users to share their dashboard with others, according to Eystein Stenberg. This means teams can now quickly create theme-specific dashboards and share them. We think Security and Management dashboards are going to be two killers, he adds. ## Any change done by CFEngine is now being tracked At large scale systems, where there are thousands of servers, CFEngine typically conducts billions of compliance checks per day. This also means that in cases of contradictions, where CFEngine will repair a state every 5 minutes it runs, there will be thousands of similar changes made to the system. Due to compliance reasons, our customers wanted CFEngine to store every change done, regardless of the size. We have listened and have introduced Change Reporting in 3.7. A fully customizable reporting mechanism (for the ones afraid of full disks) is now available to Enterprise customers through the dashboard widget and reporting interface. The change reporting features also include a fully functional REST API, in addition to the user interface. ## Network and security enhancements In untrusted networks it is now easier to bootstrap securely with CFEngine. The CFEngine network protocol uses TLS by default to secure the connections. Which security protocol that is allowed on the server side can be defined by the user. To control which TLS version and ciphers used, new attributes have been added to the language. These addition allows for a more secure approach to configuration management in untrusted network. For more information, please see: https://cfengine.com/company/blog-detail/cfengine-3-7-0-released-new-package-promise-and-change-reporting ## ABOUT CFEngine CFEngine, Inc., a pioneer in IT Automation at Web-scale, enables organizations to become more agile and thrive in the Cloud Era by radically simplifying, automating and transforming the way they build, deliver and consume IT infrastructure and applications. With CFEngine, some of the largest IT organizations provision resources and deploy new applications orders of magnitude faster, while ensuring continuous availability, security and compliance in large-scale, very dynamic and highly complex environments. CFEngine has users in more than 100 countries, including many of the world's largest financial organizations such as JPMC, as well as companies like LinkedIn, Chevron, DirecTV, Locaweb, and Deutsche Telekom. -------------------------------------------------------------------------------- title: "Releases after 3.7.0" url: /blog/2015/releases-after-3-7-0/index.md date: "2015-07-15" -------------------------------------------------------------------------------- There has been a lot of discussions lately about changing how the CFEngine releases work, and what we want is to have something that is more systematic and predictable. There are many points that need to be considered, for instance: How often should we release a new feature release? - How often will people actually upgrade? - How often would contributors like to see new releases (with their fixes in)? How long should feature releases be supported? How many releases can we make a year from a workload perspective? When should the releases be? These are some of the points that have fueled the discussions, and some of them are in direct conflict with each other, so it's hard to find the perfect solution. What we have come up with is what we believe is the best compromise of the above considerations, and should hopefully make most people happy. The first thing we will do is to introduce the concept of Long Term Support releases (LTS for short). A good example of another company that uses LTS releases is [Canonical](https://www.canonical.com/), which releases their [Ubuntu Operating System](https://www.ubuntu.com/) every half year, but [LTS releases](https://wiki.ubuntu.com/LTS) only every second year. What makes the difference is that the non-LTS releases have a shorter support span, whereas the LTS releases have a support period of several years. The second thing we will do is to switch to time based releases, which means that we will no longer delay a release in order to wait for features that are in development, but will release at regular intervals with the features that are available at that time. The advantage of this is that releases are predictable and users will know when the next release containing their fix will be. It is also a great way to bring features out faster, both from us and from contributors. Here are the differences between a non-LTS and an LTS release: | Feature | non-LTS | LTS | | --------------------------------- | -------------- | --------------- | | Release interval | Every 6 months | Every 18 months | | Enterprise and Community packages | Yes | Yes | | Period of maintenance updates | 6 months | 3 years | | Enterprise professional support | No | 3 years | From a stability perspective, here is what can be expected from each type of release: | Feature | non-LTS | LTS | | ------------------------------------ | ------- | --- | | Based on a stable branch1 | Yes | Yes | | Subject to full automatic test suite | Yes | Yes | | Subject to scale testing | No | Yes | | Subject to manual user testing | No | Yes | 1 "Stable branch" means that it is subject to a strict policy with regards to feature integrations, code refactorings and bug fixes. Only critical bug fixes are allowed in such branches, which keeps them stable and avoids the inherent stability risk of new features. New features will only be made available in new feature releases. The feature releases are aimed to be released in June and December each year, but be aware that we cannot guarantee these release times, and we may occasionally have to delay a release (however, this will not affect the time of the next release). This implies that LTS releases will be released on a schedule that alternates between summer and winter, with 18 months in between. Maintenance releases will be released as long as the release is supported, based on need. What all this boils down to is that 3.7.0 is our first LTS release, which means that 3.8 will be a non-LTS release. Here is a figure showing the release and support schedule from 3.7 onwards: [![Graphical time schedule of CFEngine releases](images/release-schedule-3.7-onwards-fixed.svg)](images/release-schedule-3.7-onwards-fixed.svg) Release schedule **Update (29 Dec 2015):** Release diagram has been changed slightly to better reflect that intended target is start of June/December, not start of July/January. Our hope is that this release schedule will give the most benefit to all parties involved. Over time, we may make additional adjustments to the plan if we find that certain aspects are not working as intended, but this is the current and effective schedule. Enjoy\! -------------------------------------------------------------------------------- title: "CFEngine 3.7.0 beta is ready for testing" url: /blog/2015/cfengine-3-7-0-beta/index.md date: "2015-06-17" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.7.0 beta is now ready for testing\! The 3.7.0 beta contains a brand new packages promise, enhanced network security, improved Enterprise reporting capabilities and much more\! ## New packages promise A new packages promise has been developed; it reuses the same promise type as the previous packages promise (packages:), but CFEngine will determine which one to use based on the attributes that are used in the promise type. The packages: promise type is fully backward-compatible, so any packages promises that you have from 3.6 or earlier versions should still work with 3.7. Currently supported platforms for the new packages promise include those based on yum/rpm (using package_module =\> yum) and apt/deb (using package_module =\> apt_get), but it can easily be extended by adding a new package_module. As an example, you can use the following promise to track the latest package of apache on Red Hat systems: ` packages: "httpd" policy => "present", version => "latest", package_module => yum;` The [development ticket for the new package promise](https://dev.cfengine.com/issues/6404) contains more details about the promise and how to extend it. ## Network security enhancements In untrusted networks it is now easier to bootstrap securely, by pre-establishing trust with cf-key --trust-key and then run cf-agent --bootstrap \ --trust-server=no. The default security protocol for outgoing connections in 3.7 is changed to TLS. Note that this has the implication that 3.7 agents cannot bootstrap to servers with version 3.5 or earlier (3.6 supports both TLS and the legacy security protocol). Which security protocol that is allowed can be controlled on the server side with allowlegacyconnects in body server control. On the client side you may use the protocol_version attribute found in body common control and body copy_from. To control which TLS version and ciphers used, new attributes have been added. On the client side, body common control has the attributes tls_min_version and tls_ciphers. The siblings for the server side are found in body server control; allowtlsversion and allowciphers. ## Enterprise reporting improvements The Enterprise edition has new features for creating and managing multiple dashboards and sharing them with your colleagues. [![](images/dashboard-multiple-flow-3.7.0.png)](/blog/2015/images/dashboard-multiple-flow-3.7.0.png) You will also see the new Changes reporting feature; both the widget and reporting interface for displaying which changes CFEngine has made to the infrastructure. [![](images/changes-reporting-flow-3.7.0.png)](/blog/2015/images/changes-reporting-flow-3.7.0.png) ## Other changes Dependencies have been upgraded in the pre-compiled agent and hub packages; most notably OpenSSL 1.0.2c, PHP 5.6.9, LMDB 0.9.14, PCRE 8.37, PostgreSQL 9.3.7, Redis 2.8.20 and Codeigniter 2.2.2. Please see the [Community Edition Changelog](https://github.com/cfengine/core/blob/3.7.0b1-build4/ChangeLog) for a more detailed list of new features and changes. ## How to provide feedback The main goal of releasing the beta is to make sure that 3.7.0 works to your expectations in your environment. So if you get the chance to test out some of the features or perhaps the compatibility with your existing policy, please let us know how it went\! Enterprise customers can submit support tickets with a subject that includes 3.7 beta. The most effective way to provide feedback for community edition users is through the [3.7.0 community beta survey](https://www.surveymonkey.com/r/L38JVCQ), as it also contains hints on what to test and how to test the beta. If you submit issues relating to the beta in [the issue tracker](https://dev.cfengine.com), please make sure to set the "Found in version" field to "3.7.0 beta". For general questions and discussions, you can also use the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine). ## Get it\! You can download the CFEngine [community 3.7.0 beta packages and source code](https://cfengine.com/product/community/). If you are an Enterprise customer, you should have received an invitation to participate in the beta program some time ago. If you would like to participate in the beta program access now, please open a support request. The Enterprise beta packages [can be downloaded here](https://cfengine.com/product/enterprise/cfengine-enterprise-free-25-previous/). Please note that this is a **beta release**, so we do not recommend installing it in a production environment. We hope you enjoy the new features, and we look forward to hearing about your experience\! ## Known issues The following notable issues have been found during testing of the beta; so please note that there is no need to resubmit them if you come across them. - On Ubuntu 10 and Debian 7 Enterprise hubs, libltdl7 needs to be installed prior to installing the hub package. - AIX does not support alphabetic versions, so 3.7.0 Enterprise beta has version "3.7.99X" (where X is beta build number) on AIX. Final AIX release will have version "3.7.0" like all other platforms. - Upgrading to Enterprise 3.7 beta will result in old alert widgets being made ‘dashboardless' until migration is fixed - as a result they won't be available through Mission Portal. - New package promise must be enabled explicitly, both for inventory and promises (uncomment in promises.cf and packages.cf). - New package promise needs [improved diagnostics logging on errors](https://dev.cfengine.com/issues/7278). -------------------------------------------------------------------------------- title: "Dynamic bundlesequence with autorun, meta tags and hard classes" url: /blog/2015/dynamic-bundlesequence-with-autorun-meta-tags-and-hard-classes/index.md date: "2015-06-11" -------------------------------------------------------------------------------- Thanks to _Nick Anderson_ and _Aleksey Tsalolikhin_ for feedback and valuable insight. ## Purpose In this document I will show you how autorun and meta tags will simplify your daily work with CFEngine. There will be no more hard coding of bundles in bundlesequence and you may still run bundles in order by name. ## Prerequisite This document assumes that you have installed a binary package from CFEngine's official site cfengine.com. The code in this document is tested with CFEngine community version 3.6.5. All paths are relative to `/var/cfengine/inputs` unless stated otherwise. For an introduction to CFEngine please see [here](https://docs.cfengine.com/latest/guide-introduction.html). All files created in this post shall be put in `services/autorun`. ## Autorun Autorun enables you to create a dynamic bundlesequence without hard coding the names of bundles. The meta promise type lets you tag bundles for easy look up. Autorun is disabled by default. In def.cf, change the line: ```cfengine3 "services_autorun" expression => "!any"; ``` to: ```cfengine3 "services_autorun" expression => "any"; ``` This will run all bundles in services/autorun with tag autorun: ```cfengine3 meta: "tags" slist => { "autorun" }; ``` For this blog post we only want one bundle with this tag. Create my_autorun.cf with the following contents: ```cfengine3 {file="my_autorun.cf"} body file control { inputs => { @(my_autorun.inputs) }; } bundle agent my_autorun { meta: "tags" slist => { "autorun" }; vars: any:: "role" string => "none"; "osrole" string => ifelse("sunos_5_10.db", "sunos_5.10_db", "redhat_6.app", "redhat_6_app", "aix_6_1.web", "aix_6.1_web", "none"); web:: "role" string => "web"; app:: "role" string => "app"; db:: "role" string => "db"; any:: "bundle_input" slist => bundlesmatching(".*", "^(any|$(sys.class)|$(sys.flavour)|$(sys.uqhost)|$(role)|$(osrole))$"); "inputs" slist => findfiles("$(this.promise_dirname)/*/*\.cf"); list_populated:: "bundle" slist => sort("bundle_input","lex"); classes: # Stop spam from sort function. "list_populated" expression => isvariable("bundle_input"); # Create role class. "web" or => { classmatch("www.*") }; "app" or => { classmatch("app.*") }; "db" or => { classmatch("db.*") }; methods: "$(bundle)" usebundle => "$(bundle)", comment => "Run all discovered bundles"; reports: inform_mode:: "autorun is executing"; "$(this.bundle): found bundle $(bundle)"; } ``` The function findfiles() will look for all files with the .cf suffix in all sub directories under services/autorun. The bundlesmatching() will search for tags in bundles in files found. `any` - Run on all hosts. Even though this is a class, here it is just a string. `$(sys.class)` - linux, solaris, aix etc. `$(sys.flavour)` - redhat_6, oracle_6, sunos_5.10, aix_6.1 `$(sys.uqhost)` - Unqualified hostname, ex: db01. Dashes allowed. No need to canonify. `$(role)` - The role of a host. Can be anything. Ex web, app, db etc. `$(osrole)` - OS and role combined like an and statement. It is important that all variables are defined otherwise the "bundles_input" list will be empty and no bundles will be found. It will match the whole string with (^....$) so on a Solaris 10 node the tag "sunos_5" will not match $(sys.flavour). Role is just an example of how you may classify your nodes. We check to see if list "bundle_input" is populated so the sort function has something to consume. The sorted list is looped over under methods. Bundles that need to be run in a specific order may be prefixed with a string that does not exist anywhere else. For example a three letter string followed by a digit. Names with lowercase a are run first. aaa1_first aaa2_second bundles beginning with zzz will run last: zzz1_second_to_last zzz2_last For global variables etc. accessible everywhere we create a file called global/global.cf: ```cfengine3 {file="global/global.cf"} bundle common G { meta: "tags" slist => { "any" }; vars: "cache_base" string => "/var/cache"; "cache" string => "$(G.cache_base)/cfengine"; reports: "$(this.bundle)"; } ``` To make sure globals are run first we use an uppercase G. All other bundles must begin with lowercase letters and not a digit. AAA1 and 1aaa will run before G. We keep the paths as short as possible and G still tells us what it is all about. Bundles that are to be run everywhere regardless of OS or role go into the file os/any.cf with tag any: ```cfengine3 {file="os/any.cf"} bundle agent aaa1_create_agent_cache { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } bundle agent aaa2_download_files { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } ``` Bundles for Solaris go into os/solaris.cf: ```cfengine3 {file="os/solaris.cf"} bundle agent my_solaris { meta: "tags" slist => { "solaris" }; reports: "$(this.bundle)"; } ``` Files for specific roles role/web.cf: ```cfengine3 {file="role/web.cf"} bundle agent my_web { meta: "tags" slist => { "web" }; reports: "$(this.bundle)"; } ``` role/app.cf: ```cfengine3 {file="role/app.cf"} bundle agent my_app { meta: "tags" slist => { "app" }; reports: "$(this.bundle)"; } ``` role/db.cf: ```cfengine3 {file="role/db.cf"} bundle agent my_db { meta: "tags" slist => { "db" }; reports: "$(this.bundle)"; } ``` Bundles that shall run last go into last/last.cf: ```cfengine3 {file="last/last.cf"} bundle agent zzz1_second_to_last { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } bundle agent zzz2_last { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } ``` Usually order between bundles does not matter but if it does you can use a prefix to group them together. We append the following to `os/any.cf`: ```cfengine3 bundle agent nnn1_do_something_in_order { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } bundle agent nnn2_do_something_in_order { meta: "tags" slist => { "any" }; reports: "$(this.bundle)"; } ``` The structure in services/autorun will look like this: ```example my_autorun.cf global/global.cf os/any.cf os/solaris.cf role/app.cf role/db.cf role/web.cf last/last.cf ``` With this extra sub directory layer bundle services_autorun will only find and parse my_autorun.cf. On a Solaris node called db01 we get: ```command /var/cfengine/bin/cf-agent -K ``` ```output R: G R: aaa1_create_agent_cache R: aaa2_download_files R: my_db R: my_solaris R: nnn1_do_something_in_order R: nnn2_do_something_in_order R: zzz1_second_to_last R: zzz2_last ``` If you specify more than one tag you get an or: ```cfengine3 meta: "tags" slist => { "sunos_5.10","oracle_6" }; ``` Creating an and statement can be achieved by using the ifelse() function. Here we have the variable osrole which combines OS and role. ```cfengine3 "osrole" string => ifelse("sunos_5_10.db", "sunos_5.10_db", "redhat_6.app", "redhat_6_app", "aix_6_1.web", "aix_6.1_web", "none"); ``` You can of course create an and statement using classmatch(), it is up to you. You have to make sure that contents of services/autorun match your repository. Extra or old files can easily get you into duplicate definition of bundles. Purging contents periodically will minimize the risk. To enable purge, in update.cf and def.cf change: ```cfengine3 "cfengine_internal_purge_policies" expression => "!any"; ``` to ```cfengine3 "cfengine_internal_purge_policies" expression => "any"; ``` ## Known problem I discovered that cf-promises is unable to catch syntax errors in bundles defined in sub directories under services/autorun. It finds the .cf files and seems to parse them but syntax is not checked. It does not help running: ```command cf-promises -c --eval-functions ``` As a workaround you can use: ```command cf-agent -nK ``` For more information please look here: ## Conclusion You can have several bundles per "type" of tag and you don't have to canonify hostnames. Autorun and meta tags will help you add policies faster than a traditional bundlesequence. ## Next step This blog post started out as a thread in the CFEngine help forum: Please feel free to join or give feedback. -------------------------------------------------------------------------------- title: "Announcing the CFEngine Community Advisory Board" url: /blog/2015/announcing-the-cfengine-community-advisory-board/index.md date: "2015-04-29" -------------------------------------------------------------------------------- As CFEngine continues its evolution and adds to the large number of users with a stake in the future of the project, we have established a Community Advisory Board. The aim of the Community Advisory Board is to advise CFEngine AS and the CFEngine project core committers and team leadership on matters relating to supporting the long-term governance, structure, and roadmap of the CFEngine open source project. The Community Advisory Board is not intended to replace existing mechanisms for community input but instead augment it and provide a consolidated opinion from the broader CFEngine community. Feel free to discuss your hopes, dreams, and concerns with any board member. Any outside party may bring an issue before the CFEngine Community Advisory Board by emailing . The following candidates were selected based on past contributions: - Bas van der Vlies - Jonathan Clarke - Marco Marongiu - Martin Simmons - Mike Svoboda - Neil Watson - Ted Zlatanov Thank you to the following board members who were elected by the candidate pool, and will serve for 2 years as currently outlined in the [charter](https://github.com/cfengine/cfengine-community-advisory-board/blob/master/charter.md#meetings-and-memberships). We appreciate your contributions and the time you spend supporting the community at large. The members are: #### **Bas van der Vlies** Among other contributions Bas has worked with the Debian team to ensure that the latest versions of CFEngine are available in the distro repositories. #### **Jonathan Clarke** Jonathan has long been a CFEngine advocate. He has also shepherded [NCF](https://github.com/Normation/ncf), a CFEngine framework, and [Rudder](https://rudder-project.org/) which provides a front end and reporting interface built on top of CFEngine. #### **Mike Svoboda** Mike is a long time CFEngine advocate and can regularly be found on the [CFEngine help mailing list](https://groups.google.com/forum/#!forum/help-cfengine). #### **Neil Watson** Neil has been using and advocating CFEngine since version 2. He has contributed the [Evolve Thinking free library](https://github.com/evolvethinking/evolve_cfengine_freelib) as well as [Delta Reporting](https://github.com/evolvethinking/delta_reporting). He can frequently be found helping users in [\#cfengine on libera.chat](https://web.libera.chat/?channel=#cfengine) and the [CFEngine help mailing list](https://groups.google.com/forum/#!forum/help-cfengine). #### **Ted Zlatanov** Ted can frequently be found fixing bugs, and submitting pull requests for feature enhancements. We can thank Ted for many of the new useful functions available in 3.6 and the upcoming 3.7 release. We encourage you to reach out to the Community Advisory Board members and discuss how you would like to see CFEngine evolve. More information about the scope of CFEngine Community Advisory Board can be found [on github](https://github.com/cfengine/cfengine-community-advisory-board/blob/master/charter.md). Meeting minutes are also published [on github](https://github.com/cfengine/cfengine-community-advisory-board/tree/master/minutes). -------------------------------------------------------------------------------- title: "Securely deploying CFEngine on untrusted networks" url: /blog/2015/securely-deploying-cfengine-on-untrusted-networks/index.md date: "2015-04-22" -------------------------------------------------------------------------------- CFEngine's trust model is based on the secure exchange of keys. This exchange of keys between client and hub, can either happen manually or automatically. Usually this step is automated as a dead-simple "bootstrap" procedure: ``` example cf-agent --bootstrap $HUB_IP ``` It is presumed that during this first key exchange, *the network is trusted*, and no attacker will hijack the connection. After "bootstrapping" is complete, the node can be deployed in the open internet, and all connections are considered secure. However there are cases where initial CFEngine deployment is happening over an insecure network, for example the Internet. In such cases we already have a secure channel to the clients, usually ssh, and we use this channel to *manually establish trust* from the hub to the clients and vice-versa. ## Manual trust establishment This procedure concerns CFEngine version 3.6 or earlier. While this fully manual procedure should always work, from version 3.7 onwards there is a simpler semi-automatic procedure for establishing trust. ## On the policy hub We must change the policy we're distributing to fully locked-down settings. So after we have set-up our hub (using the standard procedure of `cf-agent --bootstrap $HUB_IP`) we take care of the following: - Cf-serverd must never accept a connection from a client presenting an untrusted key. So in `body server control` we set: ``` example trustkeysfrom => {}; ``` - Since we will be manually bootstrapping the clients, we need to distribute a proper `failsafe.cf` policy. (**NOTE:** `failsafe.cf` is a file auto-generated in the `inputs` directory when we run `cf-agent --bootstrap`). In order to do that, we copy hub's `failsafe.cf` to `masterfiles`: ``` example cp /var/cfengine/inputs/failsafe.cf /var/cfengine/masterfiles/ ``` We'll edit that copy in `masterfiles` in the next step. - All `copy_from` files promises must never connect to an untrusted server, which means that the following line should not be found anywhere: ``` example trustkey => "true" ``` All occurrences of `trustkey` in `masterfiles` directory should be changed to "false", or be removed (since it defaults to false anyway). It is certain that `failsafe.cf` that we copied in the previous step will contain such occurrences that should be changed. (Those occurrences are the reason that automatic bootstrapping requires a trusted network). - The previous changes in `masterfiles` need to be properly propagated to the `inputs` directory. The automated way to do that is to run the update.cf policy: ``` example cf-agent -f update.cf ``` - Get the hub's key fingerprint, we'll need it later: ``` example HUB_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub` ``` ## On each client we deploy We should **not** follow the automatic method, i.e. the `cf-agent --bootstrap` command. We will perform a *manual bootstrap*. - Generate a private/public key pair by running `cf-key`. - Get the client's key fingerprint, we'll need it later: ``` example CLIENT_KEY=`cf-key -p /var/cfengine/ppkeys/localhost.pub` ``` - Write the policy hub's IP address to `policy_server.dat`: ``` example echo $HUB_IP > /var/cfengine/policy_server.dat ``` - Manually copy the modified `failsafe.cf` from hub's `masterfiles` directory into client's `inputs` directory. You should do it in a secure manner, for example using `scp` with properly trusted fingerprint of the remote host. - **NOTE:** At this step, you can try running the failsafe policy. Because trust between the two hosts has not been established, *you will get a failure*, which is totally expected and means everything is correct and secure: ``` example # cf-agent -f failsafe.cf error: TRUST FAILED, server presented untrusted key: MD5=cc27570b8b831192d9f20b54d07dd80b error: No suitable server responded to hail error: TRUST FAILED, server presented untrusted key: MD5=cc27570b8b831192d9f20b54d07dd80b error: No suitable server responded to hail [ ... ] ``` - Put the hub's key into the client's trusted keys: ``` example scp $HUB_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${HUB_KEY}.pub ``` ## Final steps - Put the client's key into the hub's trusted keys. So on the hub, run: ``` example scp $CLIENT_IP:/var/cfengine/ppkeys/localhost.pub /var/cfengine/ppkeys/root-${CLIENT_KEY}.pub ``` - If you now run the failsafe policy on each and every client, it should succeed: ``` example cf-agent -f failsafe.cf ``` Congratulations, you have performed a fully manual bootstrap procedure for your clients\! P.S. In a future blog post, I will present how you can use the new (as of 3.6) `shortcut` and `admit_keys` attributes to easily and securely transfer files or policy, restricting access only to specific clients. -------------------------------------------------------------------------------- title: "Should I pay my senior System Engineer $150,000?" url: /blog/2015/should-i-pay-my-senior-system-engineer-150000/index.md date: "2015-04-08" -------------------------------------------------------------------------------- If you are working for a Fortune 2,000, and hold P\&L, or in other ways are responsible for the compensation levels of your SA team and have doubts answering this question, I will argue your operations are not automated, nor will it be competitive in tomorrow's IT-operations market. The industry unites around the belief that automation is the only way to stay on top of IT Operations. Automation has become a prerequisite for supporting the business in their growing demands. However, few are willing or capable of adjusting their outdated view when it comes to the competence mix needed and new cost allocations associated with the transition to automation. Most of the times we meet dreamers with Winnie the Pooh attitudes wanting both. Highly automated operations, **and** cheap labor. Well, the sad truth is you can't. You cannot end up with a highly automated IT infrastructure run by fewer, very competent engineers and keep paying traditional average SA salaries. ## Blame the corporate ladder? The attitude among automation laggards is an oxymoron, and a reason why many automation projects fail to live up to their promise. For people with successful experience under their belt this is obvious, but why is it so hard for the others to follow up? When we look at the organizational structure of highly efficient IT infrastructures and compare them with less efficient, we see clear patterns. I have previously argued how the view of the involved engineers needs to move from [System Administrators to System Engineer](https://cfengine.com/company/blog-detail/from-system-administrator-to-system-engineer/), how a key to success lies in the [culture](https://cfengine.com/company/blog-detail/painful-road-digital-economy/) , and how this can lead to [amazing results](https://cfengine.com/company/blog-detail/10000-servers-managed-per-system-administrator-is-that-possible/) . After experiencing VPs and Directors of IT Operations, a big blame as to why it is hard to follow this through, seems to lie with the corporate ladder. Who in their politically correct minds would pay subordinates more than themselves? Apparently very few. If you are a Director of IT Operations today your salary might be in the $130k - $200k range. If you are a Manager it will be lower. How can you then pay your SA senior team members $150k? The answer: You don't. So, here we are. An average Fortune 2,000 company will jump on the DevOps bandwagon, they will buy an Infrastructure as code solution and initiate a new automation project. Unless you change the culture, train your people to embrace automation, get rid of the ones who stick to "this is how we always have done it", your IT Operations will only improve slightly, if any at all. You will be able to install the new shiny automation software, but there it stops. Unfortunately. The promise of highly automated IT Operations with multiple daily deployments into production and a proactive way of working where the majority of changes relate to new features and not bug-fixes becomes a broken one, and the organization is left with the old stressful reactive mode of operations. ## Automation requires top management involvement Corporate ladder and established compensation structures seems to be a main prohibitor for why the necessary competence is not brought in / trained. This is a major reason for why new automation projects need support and mandate from senior management. Senior management has the power and mandate to call for the necessary changes in compensation structures and culture needed. Without this support, we will continue to see automation projects with mediocre results. As the world is running on software at an accelerated pace, Fortune 2,000 companies shouldn't be surprised when new younger organizations suddenly pass by at high speed. ....and yes, $150,000 is base salary. -------------------------------------------------------------------------------- title: "CFEngine 3.6.5 released: Red Hat 7 and performance enhancements" url: /blog/2015/cfengine-3-6-5-now-available-red-hat-7-agent-performance/index.md date: "2015-03-12" -------------------------------------------------------------------------------- We're happy to announce that CFEngine 3.6.5 is now ready\! The new release has improved support for Red Hat Enterprise Linux 7 and CentOS 7, as well as other distributions using systemd. In addition, performance of the agent in the Enterprise edition has been greatly improved. ## Red Hat Enterprise Linux 7 support As CFEngine 3.6.3 introduced [experimental support for Red Hat Enterprise Linux 7](/blog/cfengine-3-6-3-now-available-hp-ux-support-ui-performance/), the main outstanding issue was proper management of CFEngine processes with systemd. Systemd support is now integrated into the packages and a [systemd class is defined on systemd-enabled platforms](https://github.com/cfengine/core/pull/2119). Also note that [services promises](https://docs.cfengine.com/latest/reference-promise-types-services.html) already detects and supports systemd. For 3.6 there are new packages for Red Hat 7, so make sure you pick those for this OS family version (only). These packages will only work properly on Red Hat 7, CentOS 7 and similar, but not on previous versions of the operating systems. The Enterprise hub does not yet support Red Hat 7, but the Enterprise agent and the community edition does. We plan to unify these packages in 3.7, so that there is one package for each OS type. If you're interested in the details, you can find the [CFEngine buildscripts on GitHub](https://github.com/cfengine/buildscripts). Also see the [Known issues documentation page](https://docs.cfengine.com/docs/3.6/guide-latest-release-known-issues.html) for platform-specific sections and please report any other issues you find in the [issue tracker](https://dev.cfengine.com/). ## Enterprise agent storage performance enhancements CFEngine is known for its lightweight footprint and high performance. This is very important in environments with shared resources, like private clouds and HPC, as well as in resource-constrained environments like connected and embedded devices. The Enterprise edition stores reporting data locally before it gets transferred to the hub, which naturally results in reads and writes on persistent storage. The CFEngine Enterprise 3.6.5 agent brings enhancements here in two key areas: reducing read and write operations (IOPS) and lowering storage consumption. By measuring storage activity side-by side with [New Relic](https://newrelic.com/), you can clearly see the following optimization: 3.6.5 has **70% reduced I/O rate** and **60% lower IOPS** compared to 3.6.4. [![](images/cfengine-enterprise-agent-io-3.6.4-vs.3.6.5.png)](/blog/2015/images/cfengine-enterprise-agent-io-3.6.4-vs.3.6.5.png) You can also have a look at the [storage statistics for 3.6.4](images/cfe-enterprise-agent-io-3.6.4.png) and [3.6.5](images/cfe-enterprise-agent-io-3.6.5.png) individually for more details. Secondly, the 3.6.5 Enterprise agent installation consumes about 35 MB of space in /var/cfengine on Linux, a **reduction of about 20%** since last release. This is seen after running the agent over some time. There are still some opportunities for optimizing the agent storage performance, you can watch [this Redmine issue](https://dev.cfengine.com/issues/6956) if you are interested in the details and progress on further enhancements. ## Dashboard alert usability enhancements The usability of alerts has been further improved in 3.6.5. If you have no alerts defined for a certain priority, these are now in the background so that defined alerts more clearly stand out on the dashboard. [![](images/alert-widget-software-updates-3.6.5.png)](/blog/2015/images/alert-widget-software-updates-3.6.5.png) The flow has also been enhanced by only including the widget you clicked in the widget drill-down, while still allowing the option to show all alerts in one page. [![](images/alert-list-software-updates-3.6.5.png)](/blog/2015/images/alert-list-software-updates-3.6.5.png) ## Other changes Deleting hosts from the CFEngine Enterprise UI has been optimized significantly by adding select database indexes. If you delete several hosts from the Enterprise UI or API, you will easily notice this improvement. Package inventory discovery for the Enterprise edition is now more tightly controlled using ifelapsed in masterfiles, so that it will happen at most once a day per host. This should further enhance performance and lower network consumption. Exported reports generated in the Enterprise edition are stored in /var/cfengine/httpd/htdocs/tmp on the hub. They have now gotten an expiry time of 30 days in order to not fill up the disk. This setting can be adjusted with the variable purge_scheduled_reports_older_than_days in masterfiles/def.cf. To include the last stability and security enhancements, PHP was upgraded to version 5.4.38 on the Enterprise hub. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/latest/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/latest/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading from a previous release, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-upgrading-to-36.html) for guidelines to make the process as smooth as possible. ## Get it\! As always, you can [download CFEngine Enterprise 3.6.5 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.6/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.6.5 vagrant environment](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.6.5, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Roundup of Config Management Camp 2015" url: /blog/2015/roundup-config-management-camp-2015/index.md date: "2015-02-19" -------------------------------------------------------------------------------- ![cfgmgmtcamp-logo](images/cfgmgmtcamp-logo.png)We were pleased to once again be present at this year's [Config Management Camp](http://cfgmgmtcamp.eu "http://cfgmgmtcamp.eu") in Gent, Belgium. Our two representatives, Jimis and Kristian, spent a very informative and rewarding couple of days in the company of many of our community members. The first of our two talks provided some information about our release process at CFEngine. Our recent focus has been on stability and hardening of our 3.6 series. Lately we have adapted to a nice rhythm of time-based maintenance releases and the feedback from our community has been overwhelmingly positive on this point. [Our first presentation](images/CfgMgmtCamp-2015-Presentation.pdf) outlined our reasoning for this choice, as well as a sneak peek of some features we have planned for the next feature release - 3.7.0. You can read the presentation slides below. https://www.slideshare.net/slideshow/embed\_code/44821079 The second presentation was a technical talk titled "[*How to Deploy CFEngine in the Open Internet*](images/cfgcamp-2015.pdf)", which covered the bootstrap procedure, trust management and secure deployment. It also included two demos: one was a fully manual bootstrap procedure; the other showed how to do selective asset distribution using two new attributes in CFEngine 3.6 - "admit\_keys" and "shortcut". https://www.slideshare.net/slideshow/embed\_code/44821079 One other highlight of the camp was a demo from our friends at [Rudder](http://www.rudder-project.org "http://www.rudder-project.org") - it was an impressive new look at their latest work. Rudder combines the power of CFEngine's configuration management agent with a handful of other tools to provide a web-driven solution for IT infrastructure automation and compliance. All in all a very enjoyable and productive trip - we thank those that attended our talks, and we hope to see you there next year\! -------------------------------------------------------------------------------- title: "Collecting and reporting on custom inventory" url: /blog/2015/collecting-reporting-custom-inventory/index.md date: "2015-02-02" -------------------------------------------------------------------------------- Sometimes you need to collect some small amount of data from your hosts and aggregate it to a central location for reporting. For example, you could have a command that returns the status of a hardware check that you run regularly to detect hardware faults. Or you could want a report of all the hosts' DNS configuration. In either case, you can use the CFEngine Enterprise inventory collection and reporting mechanism. This is based on a variable or class from the CFEngine language, but with a specific `meta` tag attached to it. For example, to get the System UUID from your hosts you could integrate the following to your policy: `vars: "system_uuid" string => execresult("/usr/sbin/dmidecode/dmidecode -s system-uuid", "useshell"), meta => { "inventory", "attribute_name=System UUID" };` After 15 minutes you can see this as a new inventory attribute in the CFEngine Enterprise Mission Portal UI, under Reports -\> Inventory: [![](images/inventory-report-system-uuid.png)](/blog/2015/images/inventory-report-system-uuid.png) The inventory reporting interface is very easy to use, so anyone can be allowed to generate reports - saving the system administrator from requests to make "special reports". You can learn more about [creating custom inventory in the learn section](/learn/report-custom-inventory-string/). -------------------------------------------------------------------------------- title: "CFEngine 3.6.4 now available: hub self-protection and stability" url: /blog/2015/cfengine-3-6-4-now-available-hub-self-protection-stability/index.md date: "2015-01-27" -------------------------------------------------------------------------------- The CFEngine lab has been brewing on 3.6.4 over Christmas and is finally ready to release it to the world\! With 3.6.4 being a patch release, it has a focus on stability and reliability of both the server and agent side. This is also the first release where user input has been incorporated in form of [casting a vote](/voting-for-issues/) for what to be improved - so make your voice heard\! ## Enterprise hub self-protection A major stability enhancement for report collection in CFEngine Enterprise hubs has been added to 3.6.4. The Enterprise hub collects reports from clients every 5 minutes by default, but in cases where it can not collect a round of reports from a client it will try to get what is missed the next time from that client; after another 5 minutes. This could happen for several reasons, for example that the client is temporarily offline. The amount of data that needs to be collected once the report collection succeeds again is proportional to the amount of time has passed since the last success. However, if the hub has been unsuccessful at collecting reports from a large amount of clients over a longer period of time, this can cause a high load on the hub once it succeeds again; potentially making the hub unstable and unresponsive until the collection is done. CFEngine 3.6.4 addresses this potential stability problem by introducing a maximum threshold on how long history the hub will try to collect from clients that have been offline; through the body hub control attribute [client_history_timeout](https://docs.cfengine.com/latest/reference-components-cf-hub.html#client_history_timeout). By default the hub will discard the missed history (known as issuing a "rebase" collection query) from clients if more than 6 hours have passed since last successful collection in order to protect itself. If a client comes back after 6 hours, the load on the hub for discarding these last 6 hours versus collecting them are about the same -- thus the 6 hour default. However, you can adjust this threshold if you expect your clients to be offline for longer amounts of time during normal operations. Note that in either case, history that _already exists_ about the client in the hub's database is _not discarded_. ## Enterprise host report maximum disk usage The Enterprise clients have also gotten a self-protection mechanism in case the hub does not collect their accumulated reports over time. They will normally store the historic (diff) reports locally until the hub collects them, but with 3.6.4 a threshold on the maximum disk utilization on the client is introduced. If this threshold is hit, the client will merge the recent history so that only the current state is kept. The default maximum size for reports at the client side is 50MB, which should be more than enough for most environments, but it can be adjusted as part of the max_client_history_size variable located in masterfiles/def.cf if you have client agents running frequently while hub collection running very infrequently. ## HP-UX reliability improvements As [CFEngine 3.6.3 officially added support for HP-UX](/cfengine-3-6-3-now-available-hp-ux-support-ui-performance/), there were a few feature areas that needed some stabilization work to be 100% on par with the rest of the platforms. CFEngine 3.6.4 improves reliability of [process promises](https://docs.cfengine.com/docs/3.6/reference-promise-types-processes.html) and adds support for [users promises](https://docs.cfengine.com/docs/3.6/reference-promise-types-users.html) and [edit_xml bundles](https://docs.cfengine.com/docs/3.6/reference-promise-types-edit_xml.html) on HP-UX\! Network inventory such as interfaces, IP-addresses and Ports listening has also been introduced as agent variables and classes, as well as CFEngine Enterprise reports. We provide a [depot package for CFEngine Enterprise 3.6.4 that supports HP-UX 11.23](https://cfengine.com/download/) and later versions on the Itanium architecture. If you are using the Community Edition you can download the [source code](https://cfengine.com/source-code/) and compile it for HP-UX. ## Mission Portal UI enhancements The Mission Portal UI has gotten several usability improvements since 3.6.3\! While looking at a report, it will now let you refresh the browser page (e.g. using F5) while still keeping the filters and columns you have set. This is useful if you are waiting for some data to come back from the agents and want to check the latest status. The monitoring component of CFEngine Enterprise is disabled by default, but 3.6.4 adds a note about this and a description on how to turn it on, directly in the Mission Portal\! [![](images/monitoring-enable-notification.png)](/blog/2015/images/monitoring-enable-notification.png) [![](images/health-agents-not-running.png)](/blog/2015/images/health-agents-not-running.png) Finally, the Health indicator has gotten a clearer and more consistent text describing what is wrong in case some of your agents are not working as they should. ## Other enhancements The dependencies bundled with the packages provided by CFEngine have been upgraded to incorporate the latest security and stability enhancements. _OpenSSL_ is bundled with the host package on both Community and Enterprise editions and has been upgraded to version _0.9.8ze._ Enterprise hubs now have _Apache 2.2.29_ and _PHP 5.4.36_. As for client-side fixes, CFEngine 3.6.4 works better [in some Debian-based VMs](https://dev.cfengine.com/issues/3874), [with very large lists](https://dev.cfengine.com/issues/6672) and [with chkconfig in services promises](https://dev.cfengine.com/issues/6872). Networking inventory such as interfaces, IP-addresses and Ports listening has been added on _Solaris 9, 10, 11_ and _HP-UX_, both as agent variables and classes as well as the inventory reports of CFEngine Enterprise. As usual, you can find a more detailed list of changes in the [Community Change Log](https://docs.cfengine.com/latest/guide-latest-release-whatsnew-changelog-core.html) and [Enterprise Change Log](https://docs.cfengine.com/latest/guide-latest-release-whatsnew-changelog-enterprise.html). ## Upgrading? If you're upgrading from a previous release, check out the [upgrade documentation](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-upgrading-to-36.html) for guidelines to make the process is as smooth as possible. ## Get it while it's fresh\! The CFEngine Enterprise 3.6.4 packages can be found on the [CFEngine Enterprise download page](https://cfengine.com/download/). Alternatively, if you just want to quickly test 3.6.4, you can use the [CFEngine 3.6.4 vagrant environment](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you use the Community Edition we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - take your pick\! Please make sure to leave any feedback in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine). We, the CFEngine team, really hope you enjoy 3.6.4 and wish you the best with your automation projects\! -------------------------------------------------------------------------------- title: "The Painful Road to the Digital Economy" url: /blog/2015/painful-road-digital-economy/index.md date: "2015-01-18" -------------------------------------------------------------------------------- Young and fearless, Born Digital Organizations (BDOs) now seriously challenge incumbent businesses, business models and value chains across industries. These digital organizations set new standards when it comes to frequency of new product features, cross channel compatibility, 24/7-365 availability, customer customization, user-interface friendliness and price points. Attributes of Born Digital Products: - Frequent (daily) product updates - Work across all channels - Always on - Aware of its user(s) - Friendly user-interface - More affordable (often product-as-a-service) Clunky products with infrequent updates that are not fully compatible across various channels or always available for consumption face a dark future in a world of less loyal customers. For many incumbents, it is going to be a painful road to the digital economy. ## **Digitization is changing the global economy** Over the last period of time a number of industries have experienced structural upheavals. Industrial mammoths have become extinct, and others struggle desperately to survive. The main reason being successful Born Digital organizations **(BDOs)**. Spearheaded by **iTunes**, followed by **Spotify** and later others, digital music surpassed physical music sales a couple of years back. Music stores got turned into museums overnight. With transaction costs close to zero and worldwide distribution available at a click, music labels have been forced into new business models. **Blockbuster**, a famous video rental and dvd by email chain, with more than 60,000 employees and 9,000 stores at its peak, is no longer in business, while **Netflix** a competitor who successfully transitioned into online streaming services and the digital economy, today holds a market value of almost [**$20bn**](https://ycharts.com/companies/NFLX/market_cap). Netflix is not resting on its digital laurels, but continues to grab land in new markets, like the satellite broadcasters and the film studios. **Tesla**, elegantly riding the eco-friendly wave, has in no time threatened one of the most established and tightly coupled industries. A new innovative business model that excludes the retailers is not only shaking up the industry, but also "lobbied" politicians. Who would have thought that Tesla at the beginning of 2015 is valued at almost 50% of General Motors? **Amazon** revolutionized the way books are purchased. Borders, a leading book retailer once with almost [**20,000 employees**](https://en.wikipedia.org/wiki/Borders_Group), had to close down. Amazon is today the number one bookseller in the world. On its adventure Amazon, thanks to agility challenges industry after industry. Retailers, in almost any industry now find themselves in deep compete with the low-margin mostly automated business of Amazon. Soon, it is expected to see Amazon providing automated delivery service directly to its customers, which will reduce time to delivery and lower the cost. This will impose great challenges to the delivery services industry. Finally, being in the industry we are in, we have all seen how Amazon AWS revolutionized the IT-industry. The list of disrupted companies and industries continues to grow. Social media companies like **Twitter** and **Facebook** have changed the way news are consumed, seriously challenging established mass media players. Let's not forget the biggest one of them all, **Google**, with a market capitalization of more than **[$330bn](https://ycharts.com/companies/GOOG/market_cap)**. Google, a Born Digital Organization is eating up the world's advertising budget at rapid pace, and it sees no boundaries. Indisputable the world is quickly becoming digitized. The most successful BDOs are harvesting market shares across industries. How come the incumbents allow this? The reason can mostly be explained by the attributes of BDOs products (see above), which prove to be superior to what the incumbents are capable of producing. Incumbents struggle to set the speed and operate in the new lean way. **So, what can incumbent Fortune 2,000 organizations do?** At CFEngine we have had the pleasure of working with both sides of the camp, and the rest of this blog is an attempt to share some insight and advice the incumbents on how to less painfully transition to the Digital Economy, from an IT-operations point of view. ## **Incumbents: "Let's copy..."** Most Fortune 2,000 companies have accepted the dawn of a new digital economy. Management consulting companies like [**McKinsey**](https://www.mckinsey.com/) and **[Boston Consulting](https://www.bcg.com/)** Group have shared this insight with upper management, analyst companies like **[Gartner](https://www.gartner.com)** and **[Forrester](https://www.forrester.com)** keep publishing article after article to middle management. Representatives from the biggest corporations have gone on study tours to Facebook, Twitter, Netlifx and similar BDOs to learn. The main conclusions from these meetings, reports and insights seem to carry the same story, and, generally, it goes like this: Executive at Fortune 2,000: *"What should we do?"* Advisors of executive management: *"You must copy what the best in class do"* Executive at Fortune 2,000: *"What do they do?"* Advisors of executive management: *"They use the cloud"* Executive at Fortune 2,000: *"My business is unique. Won't work."* Advisors of executive management: *"Well, then build your own cloud"* Executive at Fortune 2,000: *"Great\! Let's do that\!"* Today we rarely see any Fortune 2,000 company that is not either planning or in the testing-phase building out their own private cloud. Whether they use **[Open Stack](https://www.openstack.org/)**, **[Cloud Stack](https://cloudstack.apache.org/)** or any other "stack", very few private clouds have hit the production, but they all try hard. Most of the advice though seemed to have missed out on some very important issues, and in particular the most important of them all - the people aspect. ## **Why the road is going to be Painful** BDOs move very fast, and they work diligently to ensure the quality of service is good enough for its customers. Incumbents look to BDOs and want to copy tooling and processes. The path to lean IT turns out very painful for incumbents as it calls for change in organizational structures and trust models. Secondly, rigid, non-interoperable legacy systems still run a significant part of the incumbent business and must remain part of the equation. **Lack of trust** In order to move fast one needs to remove time wastes. People and processes are big time wasters. Seek to replace these with automation to unleash the flow. At BDOs we find very efficient change processes where few people, sometimes only one, single handedly can respond to customer requests and push changes into production. To incumbents this might seem crazy, but given that all changes are, as code, rolled out in phases and have full audit trail, this might not be as risky as one thinks it is. Among today's incumbents we don't see much empowerment. We rather see fear of change because precautions of doing wrong can be devastating for one's career. While the trust model of BDOs is built upon empowerment, the case often seems inverse with the incumbents. **Legacy systems still drive revenue** Among BDOs, API-driven systems generate all the revenue. Making incremental changes (micro-services), and modify / add / delete related services is easy and quick. Their systems are built with agility in mind. For the incumbents the story differs. Legacy systems with, if any, rare interfaces still generate significant revenue, and cannot be turned off. Making new services work across all channels prove really hard and time consuming. Lack of interoperability among systems is prohibiting lean IT and meeting customer experience of integration across any channel. How many have not painfully experienced IVR systems embraced by most Fortune 2000 companies, where regardless of what you enter, you have to restate your name and number when a representative finally takes your call? As much as the incumbents want to copy and become agile like the BDOs, we see how their current trust-model and legacy systems work against the goal. It doesn't matter what technology or cloud "stack" one decide to take on unless these more fundamental issues are being addressed. So, what can you do? ## **What to do?** The majority of today's automation, devops, private cloud and "Next Gen" initiatives to transit to the Digital Economy among incumbents seem to focus on technology, not people. Perhaps, due to the law of least resistance? (Changing technology is easy while changing people and processes hard). The results of the initiatives one can say end up in either one of two camps: Hopping Along or Green Field. **Hopping Along** Without upper management support and involvement these projects will fail. Why? Because, looking at BDOs, we find a different trust-model, and changing trust-model among an established organization is insurmountable, and impossible without upper management involvement. In addition to upper management involvement, change agents within the organization, technical engineers and team leads, that are willing to stick their neck out and carry the flag, the flag of automation and empowerment, are needed. Finally, a successful "Next Gen" project will lead to labor restructuring, and change agents under stress. For support, dedicated HR people who understand where things are going from early on are called for. Very few projects have all of these three professionals lined up. The inevitable result is no change in trust-model and thereby no success at copying the BDOs. Most organizations end up hopping along in increments, failing at making the big leap necessary to compete in the Digital Economy. A typical scenario is projects investigated by architects and led by Director, or at best a VP-level person. They end up with a new set of tools and tool chain, but the people and trust-model remain the same. The results are marginal. The path to digitization goes through lean IT, and is for the most part about people, not technology. Without changing trust-models, incumbents are likely to continue to hop along for a long period of time. **Green Field** Not all incumbents "Next Gen" project end up "Hopping Along". Some projects get upper management support with real mandate to change. Due to rigid legacy systems and established bureaucracies, a normal conclusion then becomes that it is too hard to digitize the organization from within. Such a conclusion leads to spin-offs, or totally new organizations. New organizations that are designed like BDOs from the very start. One hires new young people that bring with them the latest and greatest of technology and mindset. There is support by upper management and mandate to do whatever is needed to become best-in-class, including creating a modern auditable trust-model. The most famous example of this is probably **[GE Software](https://www.gesoftware.com/)**, a billion dollar project from GE to prepare the company for the Digital Economy. GE Software started out in California last year, and is seeking to employ more than 1,000 software engineers. The project is embraced by their ceo **[Jeff Immelt](https://web.archive.org/web/20161115184335/http://www.ge.com/digital/blog/ge-ceo-jeff-immelt-talks-industrial-internet-all-things-d-conference)** and probably set out to be able to compete with BDOs. GE will, like other incumbents, struggle with their legacy systems, but they might have addressed the more critical trust-model. ## **How to ease the transition to the Digital Economy** The role of rigid legacy systems and consolidation of services will remain a hard nut for most incumbents. This is an inevitable challenge that even today's API-driven technologies one day probably will face. However, to best improve, one needs to adapt lean IT. Looking at the BDOs, trying to align more to their trust-model is probably where incumbents have the most to win. Incumbents that succeed adapting modern trust-models will have an easier transition to the Digital Economy. This blog will end with 4 concrete pieces of advice to the incumbents wanting to ease the transition to the Digital Economy, from an IT-operations perspective. Three about people, one about the technology, the ratio at which people and technology should be thought of for IT-projects. **Advise \#1: Think Automation and hire only the best** Erik Brynjolfsson and Andrew McAfee argue in their books [The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies](https://www.amazon.com/The-Second-Machine-Age-Technologies/dp/0393239357) and [Race Against the Machine](https://www.amazon.com/gp/product/0984725113/ref=pd_lpo_sbs_dp_ss_2?pf_rd_p=1944579862&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i=0393239357&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=1G05RE62MA3NF4BEQ1RT), and Irreversibly Transforming Employment and the Economy that we are moving towards a society of Stars and Superstars, where jobs are automated. We are starting to see this in the IT-organizations of the BDOs, where only a hand-few of people is now doing the job of hundreds. Budget, and be willing to pay your System Engineer $150,000, not $70,000. Hire only the System Administrator who is worth $150,000. **Advise\#2: Create a trust-based culture** Unless lead-times are significantly cut, and people empowered, creating a lean organization that is able to compete with BDO, is very hard. Ensure upper management (C-level) supports the next "Next gen" project, and has granted necessary mandates for change. Work in concert with HR from day one to help them see the bigger picture of the desired end-states. **Advise\#3: Identify and empower the change agents** Transforming an incumbent organization requires courageous and competent change agents. The drastic changes implied by operating more like BDOs, will lead to conflicts and internal opposition. Ensure that these engineer and leaders understand the importance becoming change agents who carry a flag - that they are essential to show the way to the rest. **Advise\#4: Select a flexible automation tool that works across all platforms** Legacy systems will forever be revenue generating. One day, even today's systems will become legacy. BDO operations are highly automated. Ensure your automation tool and frameworks work across all channels and are well positioned for future technologies. -------------------------------------------------------------------------------- title: "Configuration Management Camp is coming up" url: /blog/2015/configuration-management-camp-coming/index.md date: "2015-01-16" -------------------------------------------------------------------------------- [![cfgmgmtcamp-logo](images/cfgmgmtcamp-logo.png)Configuration Management Camp](https://cfgmgmtcamp.eu/) in Ghent, Belgium is coming up, and CFEngine is of course going there\! This year we'll be sending two of CFEngine's core developers, Dimitrios Apostolou and myself, Kristian Amlie. Our currently planned talks for the event are: - How to securely deploy CFEngine in the open Internet: manage trust, provide selective access to policies, securely bootstrap new hosts and revoke old ones. - CFEngine releases, how do they happen? Learn how bug fixes, optimizations and features are scheduled for releases. Also a sneak peek at upcoming developments for CFEngine 3.7, and some time for discussion. - Finding the problem: Learn how to triage bugs in CFEngine. Also learn how to write an acceptance test. There will of course be plenty of other talks as well and a good chance to meet the guys that make CFEngine happen\! And of course you'll be able to catch talks from other Configuration Management providers as well. -------------------------------------------------------------------------------- title: "Voting for issues" url: /blog/2014/voting-for-issues/index.md date: "2014-12-19" -------------------------------------------------------------------------------- [![](images/INBOX104244.jpg)](/blog/2014/images/INBOX104244.jpg)It's sometimes difficult to maintain an overview over what issues matter most to the CFEngine community. Unfortunately our bug tracker, Redmine, has very poor support for letting the developers know what issues are important to the users, and prioritize according to this. So lately we have been looking into using the watcher count as a way to measure interest in issues. For clarity the watcher count is the number of people who have pressed the "Watch" button on a Redmine issue. Redmine doesn't have built in support for filtering according to watcher count, but we have built our own custom query tool to get this information. So we hereby encourage the community to press "Watch" on all the issues that are important to them\! We can of course not guarantee that all issues with many watchers will be resolved, but it certainly increases the odds that the particular issue makes it towards the top of the list. For technical reasons we can't publish our query, but below is a sneak peek at the current list: The top 25 issues as of right now, according to watcher count. Merry Christmas\! | | | | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | [1492](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1492) | Constant variables (policy =\> "constant") get redefined | | [3874](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3874) | 3.5.3 init.d/cfengine3 breaks CFEngine in Debian VMs | | [2208](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2208) | cf-serverd not handeling concurrency as well after upgrading to 3.4.2 | | [2419](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2419) | Default bindtointerface in server control body excluded ipv6 | | [3977](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3977) | Please add "gdb" like debugging for setting breakpoints and stepping through policy execution | | [1584](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1584) | Occasional non-repeatable "Naked variable expected" error message | | [2792](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2792) | sys.ip\_addresses doesn't return ipv6 addresses | | [997](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F997) | Strange behaviour when using templates | | [2243](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2243) | Identical lines are not inserted in edit\_line | | [2455](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2455) | cf-runagent does not understand ipv6 addresses | | [4734](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F4734) | cf\_promises\_validated: add the checksum of all the files under sys.masterdir | | [3037](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3037) | skipidentify causes bootstrap failure on ipv6 only agent. | | [3824](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3824) | Improve the situation for string type (maximum length, documentation) | | [6873](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F6873) | datastate() will run out of memory if used multiple times | | [1397](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1397) | Invalid expansion with slist in templates | | [1617](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1617) | Remote copy directory structure with symbolic links in it. | | [1732](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1732) | process\_count and restart\_class in the same process receive error (but still works) | | [1811](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F1811) | segfault on 3.4.0 | | [2276](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2276) | Unwanted package cache updates | | [2560](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2560) | cf\_lock database grows too fast becase of changing "reports" promises | | [2954](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F2954) | Bootstrap to hostname does not use the hostname for policy server communication | | [3015](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3015) | 3.5.0 error: Failed to chdir into flat file. | | [3017](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3017) | Rotation for promise\_summary.log | | [3039](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3039) | improve splaycalss | | [3041](//www.google.com/url?q=https%3A%2F%2Fdev.cfengine.com%2Fissues%2F3041) | Syslog facility on a per-promsie basis | -------------------------------------------------------------------------------- title: "CFEngine 3.6.3 now available: HP-UX support and UI performance" url: /blog/2014/cfengine-3-6-3-now-available-hp-ux-support-ui-performance/index.md date: "2014-12-06" -------------------------------------------------------------------------------- CFEngine 3.6.3 is released\! The new version brings broader platform support, UI performance and usability enhancements as well as bugfixes. It has again been about 8 weeks since the last release, and we are planning to further shorten the release intervals going forward to bring you enhancements faster. What you will also notice is that the focus is on stability and performance for 3.6.x releases, in order to make upgrades as safe as possible. Features and larger changes will be provided in the 3.7 branch. ## Broader platform support CFEngine is written in C and is therefore probably the most portable configuration management tool out there. However, there are tests that need to be run and usually minor changes needed in order to add a platform to the supported list. This has now been done for one more platform as CFEngine 3.6.3 adds **support for HP-UX 11.23 and later**, on the Itanium architecture\! This means that if you have HP-UX in your infrastructure, you can finally have one tool to manage everything\! For CFEngine Enterprise, a [binary depot package is provided](https://cfengine.com/download/). CFEngine community users can [compile from source](https://cfengine.com/source-code/), but now rest assured that HP-UX is a first-class citizen to CFEngine with respect to release testing. **Experimental support for Red Hat Enterprise Linux 7** is provided in 3.6.3. This means that the package and agent have gone through manual testing, but testing is not yet as comprehensive as the supported platforms, so there might be some minor issues discovered along the way. Furthermore, there is no Enterprise hub package for RHEL 7 yet. *The rpm-based host package can be used on RHEL 7.* Also see the [Known issues documentation page](https://docs.cfengine.com/docs/3.6/guide-latest-release-known-issues.html) for platform-specific sections and please report any other issues you find in the [issue tracker](https://dev.cfengine.com/). ## UI performance improvements The CFEngine Mission Portal UI Dashboard has gotten a big performance improvement; the **out-of-the-box alerts will now load 5x faster**\! This is due to optimization of Software updates type of alerts. If you have several Software updates alerts, the difference from 3.6.2 is going to be even more substantial. [![](images/alert-widget-system-health.png)](/blog/2014/images/alert-widget-system-health.png) The inventory reporting interface has also gotten a similar performance enhancement, with **load time being 3x as fast** as 3.6.2\! [![](images/inventory-os.png)](/blog/2014/images/inventory-os.png) Both these performance enhancements were obtained by adjusting the SQL query that is run against the PostgreSQL backend. Of course, they are fully backward-compatible, so existing alerts and reports will continue to function seamlessly. ## Other changes As usual, the Mission Portal UI has gotten several usability improvements. This time the LDAP settings search filter help text and validation is the most visible one. A more comprehensive list of changes for CFEngine Community can be found in the [change log](https://github.com/cfengine/core/blob/3.6.x/ChangeLog). ## Get it\! As always, you can [download CFEngine Enterprise 3.6.3 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.6/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.6.3 vagrant environment](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.6.3, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Learning CFEngine - An Automation Story" url: /blog/2014/learning-cfengine-automation-story/index.md date: "2014-11-26" -------------------------------------------------------------------------------- ### Hearing a user speak is worth more than self-glorification\! As we come up to year-end, it is time to start thinking about turkeys (well if you celebrate Thanksgiving like they do out here), and of course Christmas and the festive season it heralds. In this blog post I would like to thank a particular automation tool (no points for guessing which one), but really do so from the vantage point of a user and the progress they made with the tool. "[Today I started learning CFEngine 3](https://blog.remibergsma.com/2013/06/13/today-i-started-learning-cfengine-3/)". This was the title of an innocuous looking post from Remi on June 13'th 2013. In here he detailed his early experience attending a training session on CFEngine delivered by Diego Zamboni the author of [Learning CFEngine 3](https://www.amazon.com/books/dp/1449312209). What stood out even back then and at first glance was that CFEngine is a pretty darn good monitoring tool. It is capable of fixing issues or reporting to the user. ### Easy-Peasy? Maybe not. Logical and rational - Definitely Yes\! By August 2013 (less than 2 months after attending that first introduction to CFEngine class), Remi and team were well on their way. [They had automated hundreds of Linux servers](https://blog.remibergsma.com/2013/08/31/managing-hundreds-of-servers-with-cfengine-and-git-full-control-over-configuration/) and setup a detailed and thorough approach to leveraging CFEngine and Git to ensure their server farm was automated. But wait CFEngine is supposed to be **hard to use\!** It is supposed to get you all tangled up\! You have to "think" CFEngine as Remi puts it. A ‘class' in CFEngine is not what traditional OOP veterans may think it to be. And of course who doesn't miss the if/then/else or for/while constructs. If you ask me, every programmer most likely believes that she can write an if/then/else statement to cover for all contingencies. Especially when automating systems. Yeah right\! The point being, if the automation system delivers and does its job, it should not really matter whether the traditional programming constructs are present or not. ### So is your automation tool automated enough? The best analogy of CFEngine at work for me has always been the way [Mark Burgess (author of Promise Theory and CFEngine) explained it](https://web.archive.org/web/20160909015440/http://vimeo.com/album/2573383/video/77207216): *CFEngine and its approach to automation as he put it is analogous to the storm drain. It is there yet not something we actively worry about every waking minute. It is connected yet decentralized and autonomous. And most importantly, it is self-sufficient and knows exactly what to do when without impetus or prodding from a central master. The result - we don't have to recourse to power tools and extensive cleanups and post mortems. The drain or the agent humbly ensures promises are kept.* For the more technical amongst you Diego Zamboni in his book on CFEngine explains it thus: *‘CFEngine works on top of a theoretical model called ‘Promise Theory‘ by Mark Burgess (author of CFEngine). This theory models the behavior of autonomous agents in an environment without central authority, based on only promises of behavior made by each agent, and shows that even without central control , the system can converge to a stable state.'* [![convergence](images/convergence-800x543.png)](images/convergence.png) **"Where should I go?" -Alice. "That depends on where you want to end up." - The Cheshire Cat."** It is surely about perspective when you think about automation. Should it be directed or should it be autonomic. Should there be a central dictator or mutual observation and interaction among the constituents. Should the administrator believe they are in charge of telling the tool what to do under different situations? Or should the automation approach be one based on a system working within provided constraints to establish or strive to reach ideal state. One that is self aware to take action depending on state and the dynamics of the situation. I argue that this approach which is steeped in a perspective of being self aware is far superior to any of the tools-based approaches available. **Whenever you find yourself on the side of the majority, it is time to pause and reflect - Mark Twain** Going back to the protagonist in our story, Remi and team automated hundreds of Linux Servers in a very short time. So impressed were they with the results that his presentation ‘[Configuration Management in an Enterprise Linux Team](https://archive.fosdem.org/2014/schedule/event/use_case_configuration_management_enterprise_linux_environment/)' was subtitled ‘'How I automated myself out of a job". You can watch a [video recording of this talk at FosDem 2014](https://blog.remibergsma.com/2014/02/07/video-recording-of-my-talk-at-fosdem14-in-brussels/). CFEngine has been around for a while. We all are familiar with other tools that have since appeared on the horizon. However not only is the CFEngine approach unique compared to the others, it also is extremely powerful, secure and scalable. All while being truly autonomous and self-healing (remember systems converge to desired state) so you the admin can focus on other important things. ### Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So where does that leave us? As you contemplate 2014 and look forward to 2015 you have a choice; at least when it comes to configuration management and systems automation technologies. Maybe you did not. Maybe your organization had already standardized on one of many tools. You can be the change agent. [Try CFEngine Enterprise free](https://cfengine.com/download/). Use it on your sandbox with all of its features. And just to quell the last of the doubters, the good folks at [CFEngine even have easy to use tutorials](https://cfengine.com/learn/) for the most common things you would want to accomplish with CFEngine. As we wind down for the year, I throw this challenge your way. Pick a few machines, download and install CFEngine, learn to use it and I guarantee you will never look back\! And as for Remi who went from zero to hero in automating their entire linux server footprint in a short time, well he did not quite automate himself out of a job. Last I checked he was having fun building and managing his employer's CloudStack cloud\! If in 2015 you have a similar CFEngine story, we are all ears\! Happy holidays ahead\! -------------------------------------------------------------------------------- title: "2014 CFEngine Champions Nomination" url: /blog/2014/2014-cfengine-champions-nomination/index.md date: "2014-10-31" -------------------------------------------------------------------------------- ## Please nominate your favorite\! The CFEngine Champion program rewards the voluntary efforts of individuals who have significantly enhanced the CFEngine Community by promoting CFEngine and its use. The contributions of the CFEngine Community are a vital part of our company's ecosystem. [View previous champions](https://cfengine.com/community/cfengine-champions/meet-the-cfengine-champions/). [Please nominate your 2014 candidate here](https://docs.google.com/a/cmdln.org/forms/d/1dFnIWpRd6wEIzc9Ieun1HfPP7TcBzJGAvR_I0lG-xoc/viewform?edit_requested=true) ![CFEngine\_champion](images/champion.png) -------------------------------------------------------------------------------- title: "Introducing CFEngine Office Hour" url: /blog/2014/introducing-cfengine-office-hour/index.md date: "2014-10-30" -------------------------------------------------------------------------------- Thanks to Mike Svoboda at Linkedin and a league of experienced CFEngine users, we are happy to announce the **"CFEngine Office Hour".** Meet with CFEngine folks, bring your questions\! Here is what to expect: "Instead of lecturing about how we've used CFEngine, the focus of this office hour is dedicated to helping you\!" "Have you ever had a question that you wanted to ask, but didn't want to blast it out on the mailing list because it's too public? Would you like for someone to take a look at one of your policies and maybe suggest improvements? Have a question about how to approach an automation problem?" "The idea behind the office hour is that we want to help other folks in the community bootstrap their environment." "Getting over that initial learning curve can be quite a challenge. Having a video conference with a person whom you can ask questions of, and can interact with directly can make this process a lot easier." "Even if you've been using CFEngine for a few years, feel free to drop in. Maybe you can learn a thing or two by looking at policy examples." If you haven't joined the \#cfengine channel, we're on libera.chat. Feel free to drop by and ask questions there as well, there are typically a few of us around. We will post the times of Open Office Hours on our [Events page](https://cfengine.com/company/events/) We hope to see you\! -------------------------------------------------------------------------------- title: "CFEngine and JIRA - Integrating Configuration Management and Issue Tracking" url: /blog/2014/creating-jira-issue-noncompliance/index.md date: "2014-10-23" -------------------------------------------------------------------------------- In this blog post I would like to show how one of the [best configuration management solutions](https://www.cfengine.com) integrates with an equally well known ticketing system - Jira. When a specific policy becomes out of compliance, there is a common need to integrate this with a ticketing system. For example, you have an important web application configured and ensured to be running using CFEngine. If any aspect of that fails, you want to be notified immediately. But since you already get enough email, and you already use a ticketing system for all other tasks, you want to open an issue in the [JIRA issue tracking system](https://www.atlassian.com/software/jira) on such an event. [CFEngine 3.6.2](https://cfengine.com/company/blog-detail/3-6-2-release-high-availability-and-custom-actions/ "CFEngine 3.6.2 now availabile: Focus on High Availability and Custom actions") introduces [Custom actions](https://docs.cfengine.com/docs/3.6/enterprise-cfengine-guide-alerts-custom-actions.html) as a notification method for alerts, which virtually enables any notification method for any event happening in your infrastructure. In our new How To, we show how to [integrate CFEngine with JIRA using Custom actions](https://cfengine.com/learn/ticketing-system-integration-jira/). Let CFEngine open a ticket for you whenever something important happens with your infrastructure, and spend your time planning instead of monitoring\! -------------------------------------------------------------------------------- title: "Announcing new mailing list for developers: dev-cfengine" url: /blog/2014/announcing-new-mailing-list-developers-dev-cfengine/index.md date: "2014-10-20" -------------------------------------------------------------------------------- Dear CFEngine Community, we are proud to announce our new mailing list: [dev-cfengine](https://groups.google.com/forum/#!forum/dev-cfengine). Given that the contributions in both [Core](https://github.com/cfengine/core) and [Masterfiles](https://github.com/cfengine/masterfiles) repositories have been steadily increasing, the need for such a list became apparent. While patch submissions and code reviews will still be taking place using GitHub's pull requests, this list serves the purpose of facilitating any other discussion on the development of CFEngine. We are looking forward to seeing the community being active on that list. In addition, we, the CFEngine developers, are planning to participate with all our discussions that do not touch on CFEngine Enterprise. Regards, CFEngine AS -------------------------------------------------------------------------------- title: "POODLE, SSLv3 and CFEngine" url: /blog/2014/poodle-sslv3-cfengine/index.md date: "2014-10-20" -------------------------------------------------------------------------------- This post clarifies whether CFEngine is affected by the newly published vulnerability in the SSL protocol, [POODLE](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-3566 "POODLE"). CFEngine core functionality, i.e. agent-to-hub communication is **not** affected in any way by the POODLE vulnerability. If the [protocol version](https://docs.cfengine.com/docs/3.6/reference-components.html#protocol_version "Protocol version") is set to "classic" or "1", or is just left to be the default, then all communication happens using the legacy protocol which has nothing to do with SSL. If it is set to "latest" or "2", then TLS version 1.0 is used, which does \*not\* suffer from the specific flaw in SSL v3.0 that enables POODLE. So the vulnerability is not applicable in any case. CFEngine Enterprise provides the Mission Portal web interface, served via the Apache web server at port 443. Unfortunately the default package installation uses default Apache settings, and httpd currently accepts connections using SSL v3.0. To remedy the problem, the following line should be edited in /var/cfengine/httpd/conf/original/extra/httpd-ssl.conf from SSLProtocol all -SSLv2 to SSLProtocol all -SSLv2 -SSLv3 This is regarded a low risk vulnerability, hard to exploit, especially inside private subnets. In order for an attacker to steal a cookie, he must maliciously steal the victim's browsing session (e.g. by luring him into visiting a malicious website) and generate thousands of connections to the Mission Portal, while simultaneously intercepting and manipulating all of the victim's network data. SSL v3.0 will be disabled in the next version of CFEngine Enterprise Hub. -------------------------------------------------------------------------------- title: "A Primer on CFEngine 3.6 Autorun" url: /blog/2014/a-primer-on-cfengine-autorun/index.md date: "2014-10-07" -------------------------------------------------------------------------------- CFEngine recently released version 3.6, which makes deploying and using cfengine easier than ever before. The greatest improvement in 3.6, in my opinion, is by far the autorun feature. I'm going to demonstrate how to get a policy server set up with autorun properly configured. ## Installing CFEngine 3.6.2 The first step is to install the CFEngine package, which I'm not going to cover. But I will say that I recommend using an existing repository. Instructions on how to set this up are [here](https://cfengine.com/cfengine-linux-distros/). Or you can get binary packages [here](https://cfengine.com/community/download/). If you're not using Linux (like myself) you can get binary packages from [cfengineers.net](https://web.archive.org/web/20170523032021/http://www.cfengineers.net/downloads/cfengine-community-packages/).If you're inclined to build from source I expect that you don't need my help with that. Having installed the CFEngine package, the first thing to do is to generate keys. The keys may have already been generated for you, but running the command again won't harm anything. /var/cfengine/bin/cf-key ## Setting up Masterfiles and Enabling Autorun Next you'll need a copy of `masterfiles`. If you downloaded a binary community package from [cfengine.com](https://cfengine.com/community/download/) you'll find a copy in `/var/cfengine/share/CoreBase/masterfiles`. As of 3.6 the policy files have been decoupled from the core source code distribution so if you're getting CFEngine from somewhere else it may not come with `CoreBase`. In this case you'll want to get a copy of the [masterfiles](https://github.com/cfengine/masterfiles) repository at the tip of the branch for your version of CFEngine (in this case, 3.6.2), not from the master branch where the main development happens. There's already development going on for 3.7 in master so for consistency and repeatability grab an archive of 3.6.2. Going this route you also need a copy of the CFEngine `core` source code (although you do not need to build it). curl -LC - -o masterfiles-3.6.2.tar.gz https://github.com/cfengine/masterfiles/archive/3.6.2.tar.gz curl -LC - -o core-3.6.2.tar.gz https://github.com/cfengine/core/archive/3.6.2.tar.gz tar zxf masterfiles-3.6.2.tar.gz tar zxf core-3.6.2.tar.gz You'll now have the main `masterfiles` distribution unpacked. This isn't something that you can just copy into place, you need to run `make` to install it. cd masterfiles-3.6.2 ./autogen.sh --with-core=../core-3.6.2 make install INSTALL=/opt/local/bin/install datadir=/var/cfengine/masterfiles Note: Here I've included the path to `install`. This is required for SmartOS. For other systems you can probably just run `make install`. At this point it's time to bootstrap the server to itself. /var/cfengine/bin/cf-agent -B You should get a message here saying that the host has been successfully bootstrapped and a report stating ‘I'm a policy hub.' To enable autorun simply make the following change in `def.cf`. - "services_autorun" expression => "!any"; + "services_autorun" expression => "any"; **Note:** There's a bug in masterfiles-3.6.0, so make sure to use at least 3.6.2. ## Using Autorun With the default configuration autorun will search for any files in `services/autorun/` with the tag `autorun` and execute it. At this point you can see autorun working for yourself. /var/cfengine/bin/cf-agent -K -f update.cf /var/cfengine/bin/cf-agent -Kv Here I've enabled verbose mode. You can see in the verbose output that autorun is working. Now, like Han Solo, I've made a couple of _special_ modifications myself. I also like to leave the default files in pristine condition, as much as possible. This helps when upgrading. This is why I've only made very few changes to the default policies. It also means that instead of using `services/autorun.cf` I'll create a new autorun entry point. This entry point is the only bundle executed by the default autorun. I've saved this to `services/autorun/digitalelf.cf` ```cf3 body file control { agent:: inputs => { @(digitalelf_autorun.inputs) }; } bundle agent digitalelf_autorun { meta: "tags" slist => { "autorun" }; vars: "inputs" slist => findfiles("$(sys.masterdir)/services/autorun/*.cf"); "bundle" slist => bundlesmatching(".*", "digitalelf"); methods: "$(bundle)" usebundle => "$(bundle)", ifvarclass => "$(bundle)"; reports: inform_mode:: "digitalelf autorun is executing"; "$(this.bundle): found bundle $(bundle) with tag 'digitalelf'"; } ``` This works exactly the same as `autorun.cf`, except that it looks for bundles matching `digitalelf` and only runs them if the bundle name matches a defined class. Also note that enabling `inform_mode` (i.e., `cf-agent -I`) will report which bundles have been discovered for automatic execution. For example I have the following `services/autorun/any.cf`. ```cf3 bundle agent any { meta: # You must uncomment this line to enable autorun. "tags" slist => { "digitalelf" }; vars: linux:: "local_bin_dir" string => "/usr/local/bin/"; smartos:: "local_bin_dir" string => "/opt/local/bin/"; files: "/etc/motd" edit_line => insert_lines("Note: This host is managed by CFEngine."), handle => "declare_cfengine_in_motd", comment => "Make sure people know this host is managed by cfengine"; } ``` Since the tag is `digitalelf` it will be picked up by `services/autorun/digitalelf.cf` and because bundle name is `any`, it will match the class `any` in the methods promise, and therefore run. You can drop in bundles that match any existing hard class and it will automatically run. Want all `linux` or all `debian` hosts to have a particular configuration? There's a bundle for that. ## Extending Autorun You may already be familiar with my [cfengine layout](https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/) for dynamic bundlesequence and bundle layering. My existing dynamic bundlesequence is largely obsolete with autorun, but I still extensively use bundle stack layering. I've incorporated the classifications from [`bundle common classify`](https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/#dynamic-bundlesequence) directly into the `classes:` promises of `services/autorun/digitalelf.cf`. I can trigger bundles by discovered hard classes or with any user defined class created in `bundle agent digitalelf_autorun`. By using autorun bundles based on defined classes you can define classes from any source. Hostname (like I do), LDAP, DNS, from the filesystem, network API calls, etc. This article was submitted by [**Brian Bennett**](https://digitalelf.net/), a [**Vertical Sysadmin**](https://verticalsysadmin.com) consultant and trainer --- -------------------------------------------------------------------------------- title: "CFEngine 3.6.2 now availabile: Focus on High Availability and Custom actions" url: /blog/2014/3-6-2-release-high-availability-and-custom-actions/index.md date: "2014-10-02" -------------------------------------------------------------------------------- CFEngine 3.6.2 is now available - in both [Community](https://cfengine.com/community/download/) and [Enterprise](https://cfengine.com/download/) editions\! There are major new features in the Enterprise hub; High Availability and Custom actions. In addition, we have resolved numerous issues to provide you with a very stable release. It has been about 8 weeks since the 3.6.1 release, and we plan to continue on a 6-8 week schedule for maintenance releases going forward. ## High availability for the hub A common requirement for most enterprises is that key processes and mission critical applications are highly available - in essence to ensure there is *no single point of failure*. Although CFEngine is a distributed system, with decisions made by autonomous agents running on each node, the hub can be viewed as a single point of failure. Essentially, the hub has two responsibilities: - Serving latest policy to agents (control) - Collecting, aggregating and visualizing reports from the agents (visibility) The responsibility of serving the latest policy is usually the most important, because losing this ability would mean that the infrastructure can no longer be changed. However, given the autonomous nature of CFEngine, this is quite easy to make fault tolerant by creating a separate failover hub and adding it to the copy\_from list in the failsafe policy. For fault tolerance of the visibility aspect, the reporting features of the Enterprise edition need to get the same type of failover. This means setting up redundancy for all components, most importantly the PostgreSQL database. To solve this problem, *CFEngine Enterprise 3.6.2 introduces support for an active-passive High Availability configuration*, as shown in the diagram below. [![](images/ha.png)](/blog/2014/images/ha.png) One hub will be the active hub, while the other serves the role as a passive hub and is *a fully redundant instance of the active hub*. If the passive determines the active is down, it will become active and start serving the Mission Portal, collect reports and serve policy. Mission Portal in 3.6.2 has gotten a new indicator with the status of the High Availability configuration. [![](images/ha.jpg)](/blog/2014/images/ha.jpg) #### Installation and setup Existing CFEngine Enterprise installations can upgrade their single-node hub to a High Availability system in version 3.6.2. However, as with most High Availability systems, setting it up requires carefully following a series of steps with dependencies on network components. The setup can therefore be error-prone, so if you are a CFEngine Enterprise customer we recommend that you contact support for assistance if you do not feel 100% comfortable of doing this on your own. To learn more, you could have a look at the [Overview of High Availability](https://docs.cfengine.com/docs/3.6/enterprise-cfengine-guide-high-availability.html) as well as the [High Availability installation Guide](https://docs.cfengine.com/docs/3.6/enterprise-cfengine-guide-high-availability-installation-guide.html) in the documentation. Users of the CFEngine Community Edition should create their own High Availability setup using a redundant policy server, included in copy\_from in the update policy, and synchronize the policy servers' masterfiles with version control. The reason is that the Community Edition does not have the need for creating redundancy of the reporting, so this is an easier problem. ## Custom actions CFEngine Enterprise 3.6.0 debuted the new concept of 'alerts', which enables UI- and email-based notifications on conditions, like policy failures, insecure configurations, and software available for update. In CFEngine Enterprise 3.6.2 a new way to notify is introduced: *Custom actions*. A Custom action is a script that gets called with a reference to a file containing parameters about the triggered alert. Any scripting language is supported; the only limitation is that the hub needs an interpreter for the language. One typical use for a Custom action script is to integrate notification about promises not kept into a monitoring or ticketing system, like Nagios or JIRA. Using Custom actions, you can open a ticket if a policy bundle becomes not kept, for example. As an example, a bash-script that logs events about policy alerts to syslog on the hub is shown below. ` #!/bin/bash source $1 logger -i "Policy alert '$ALERT_NAME' $ALERT_STATUS. Now triggered on $ALERT_FAILED_HOST hosts. Defined with $ALERT_POLICY_CONDITION_FILTERBY='$ALERT_POLICY_CONDITION_FILTERITEMNAME', promise handle '$ALERT_POLICY_CONDITION_PROMISEHANDLE' and outcome $ALERT_POLICY_CONDITION_PROMISEOUTCOME" exit $? ` The resulting log message of a triggered alert may then look like the following. ` Sep 26 02:00:53 localhost user[18823]: Policy alert 'Web service' fail. Now triggered on 11 hosts. Defined with bundlename='web_service', promise handle '' and outcome NOTKEPT ` The script can be uploaded in the Mission Portal Settings by members of the *admin* role. [![](images/mp-settings-custom-notification-add.png)](/blog/2014/images/mp-settings-custom-notification-add.png) After this, the script may be associated with any alert. [![](images/create-alert-custom-action-syslog.png)](/blog/2014/images/create-alert-custom-action-syslog.png) At this point the script will be called whenever the alert changes state from OK to triggered, or the other way around. Now you can get rid of those email notifications and integrate notifications into your existing notification infrastructure -- what ever it may be\! To get started, you may have a look at the [Custom action documentation](https://docs.cfengine.com/docs/3.6/enterprise-cfengine-guide-alerts-custom-actions.html). ## Mission Portal dashboard The first thing you will notice when accessing the Mission Portal in 3.6.2 is that the dashboard has a new out-of-the-box alert widget called *System health*. [![](images/alert-widget-system-health1.png)](/blog/2014/images/alert-widget-system-health1.png) It contains one alert, *Low disk space,* which is triggered if a primary disk partition of any managed node runs below 15% free space. [![](images/alert-status-lowdisk.png)](/blog/2014/images/alert-status-lowdisk.png) For critical issues like these, you might also want to add an email notification or perhaps a Custom action to make sure you capture it as soon as it happens. As you can see in the above screenshot, the alert status pages have gotten a new, more intuitive and space-efficient design since 3.6.1. This is based on usage patterns we have seen developing since the initial release. ## LSB-based OS inventory If you upgrade the masterfiles framework (recommended), you will get access to a new inventory variable named *inventory\_os.description*. This variable will use *Linux Standard Base* to create a more human-friendly OS name, and fall back to CFEngine discovery (*sys.flavor*) if not available. The variable is also available as the *OS* inventory attribute in the Mission Portal. [![](images/inventory-os.png)](/blog/2014/images/inventory-os.png) ## Other changes The [*admit\_keys*](https://docs.cfengine.com/latest/reference-promise-types-access.html#admit_keys), [*admit\_ips*](https://docs.cfengine.com/latest/reference-promise-types-access.html#admit_ips) and [*admit\_hostnames*](https://docs.cfengine.com/latest/reference-promise-types-access.html#admit_hostnames) access promises were introduced in 3.6.0. When they are coupled with the dynamic variables *connection.ip*, *connection.hostname* and *connection.key,* they make it easy to dynamically grant access to resources based on the connecting client's identity. The *admit\_\** promises were initially enabled only when using the TLS protocol, but have been introduced to the legacy protocol in 3.6.2. On the Enterprise packaging side, we have upgraded from PHP 5.4.20 to 5.4.32 and the Windows package has now gotten the LibXML library bundled so that it can edit XML files with [edit\_xml promises](https://docs.cfengine.com/docs/3.6/reference-promise-types-edit_xml.html "CFEngine edit_xml promise type"). A more comprehensive list of changes for CFEngine Community can be found in the [change log](https://github.com/cfengine/core/blob/3.6.x/ChangeLog). ## Get it\! As always, you can [download CFEngine Enterprise 3.6.2 packages](https://cfengine.com/download/) for the [supported platforms](https://docs.cfengine.com/docs/3.6/guide-latest-release-supported-platforms.html), or give it a quick spin with the [CFEngine 3.6.2 vagrant environment](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-general-installation-installation-enterprise-vagrant.html). If you are using the Community Edition, we provide you with [source code](https://cfengine.com/source-code/), [packages](https://cfengine.com/community/download/), and [package repositories](https://cfengine.com/cfengine-linux-distros/) - to make sure we cover the distribution channel of your choice\! We hope you enjoy 3.6.2, and we look forward to hearing about your experience in the [CFEngine Google Group](https://groups.google.com/forum/#!forum/help-cfengine)\! -------------------------------------------------------------------------------- title: "Reporting and Remediation of Security Vulnerabilities" url: /blog/2014/reporting-remediation-security-vulnerabilities/index.md date: "2014-09-30" -------------------------------------------------------------------------------- [![](images/320px-Shellshock_logo.svg_.png)](/blog/2014/images/320px-Shellshock_logo.svg_.png)With the slew of recent security issues like [Supermarket Point of Sale Compromises](https://consumerist.com/2014/08/15/albertsons-jewel-osco-acme-shaws-supermarkets-hit-by-credit-card-data-breach/) not once [but twice](https://consumerist.com/2014/09/29/this-sounds-familiar-albertsons-jewel-osco-acme-shaws-hit-by-second-credit-card-data-breach/), other [large retailer card breaches](https://www.schneier.com/blog/archives/2014/04/heartbleed.html), the famed [Heartbleed](https://www.zdnet.com/shellshock-makes-heartbleed-look-insignificant-7000034143/) vulnerability and others in the news. We want to share an example of how CFEngine can be used to quickly identify and remediate affected systems. In our documentation please find the "[Reporting and Remediation of Security Vulnerabilities](https://docs.cfengine.com/docs/3.6/examples-tutorials-report_inventory_remediate_sec_vulnerabilities.html)" tutorial. The tutorial walks through policy to both identify and remediate the recent [\#shellshock](http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-7169) exploit. For those using CFEngine Enterprise there is guidance on creating dashboard alerts and inventory reports included. -------------------------------------------------------------------------------- title: "Guest Blog - Monitoring CFEngine with Zabbix" url: /blog/2014/guest-blog-monitoring-cfengine-zabbix/index.md date: "2014-08-26" -------------------------------------------------------------------------------- In this installment we turn to Danilo Fernando Chilene who recently wrote about **monitoring CFEngine with Zabbix. **The original blog can be found at - . In this particular piece learn about how Zabbix can be leveraged to monitor processes, memory use and the promise summary log in the context of CFEngine. If you have other such stories of CFEngine use, we would love to hear back from you. Thanks Danilo for a great post\! # Monitoring CFEngine With Zabbix I created a template to monitor `CFEngine` with Zabbix. This allows the monitoring of processes, memory use and the promise summary log.
The below shows what is included on the template:
1
2
3
4
5
6
7
8
9
Process CF-serverd
Process CF-monitord
Process CF-execd
Memory use process CF-serverd
Memory use process CF-monitord
Memory use process CF-execd
Log cf-engine - Log view of /var/cfengine/promise_summary.log
CFEngine listen port 5308
and generate alerts if (CF-serverd, CF-monitord, CF-execd) is down
The template can be downloaded here: [bit.ly/1qxKd5G](https://bit.ly/1qxKd5G) The steps are pretty simple, first you must import the template on Zabbix and then apply to host. ## Importing the Template and using it On Zabbix go to Configuration -\> Templates and click Import, Choose the file (as shown in the image below) and click Import. [![](images/Import.png)](images/Import.png)
Looks like the import was fine. [![](images/ImportSucess.png)](images/ImportSucess.png)
## Applying the template to monitored host Now you need to apply the template to monitored host, in our case just link the template to your CFEngine server. [![](images/Template_CFEngine.png)](images/Template_CFEngine.png)
Below are some screenshots of latest data, graph of memory usage and history of promise log: [![](images/LatestData_CFEngine.png)](images/LatestData_CFEngine.png)
[![](images/MemoryUseCFEngine.png)](images/MemoryUseCFEngine.png)
[![](images/Promiselog.png)](images/Promiselog.png) Thanks to the [CFEngine](https://cfengine.com/) team for this great configuration management solution. -------------------------------------------------------------------------------- title: "Take the Plunge! Upgrading to CFEngine 3.6.1" url: /blog/2014/upgrade-to-cfengine-3-6-1/index.md date: "2014-08-06" -------------------------------------------------------------------------------- We [recently announced the general availability](https://cfengine.com/company/blog-detail/cfengine-enterprise-3-6-1-maintenance-release-now-generally-available/) of CFEngine Enterprise 3.6.1. One of the key capabilities added to this maintenance release is a supported upgrade process. In today's post, I'll walk you through an outline of the upgrade procedure, which will hopefully provide a good starting point for you to map out the entire process for your CFEngine deployment. Note that the examples here assume starting the upgrade from a CFEngine 3.5.x install, but the same steps are applicable to version 3.0 as well. Don't forget to refer to our [online documentation](https://docs.cfengine.com/latest/guide-installation-and-configuration-upgrading-to-36.html) for the complete set of steps. ## How should you begin? Start with the Policy Server. Keep in mind that the agents are autonomous, so even as the Policy Server is taken down or disconnected from the hosts that it is managing, it will still be "business as usual" for the hosts, as they will continue to be governed by their existing set of policies. ## Back up your Policy Server Prior to performing any upgrade, you should back everything up in your current install (e.g. `/var/cfengine`). As an example, tar everything up in a CFEngine 3.5 deployment to a local directory like `/root/3.5`: $ mkdir -p /root/3.5 $ tar cvzf /root/3.5/cfengine.tar.gz /var/cfengine You may also wish to make a copy of your ppkeys as well as a list of the hosts connected to this Policy Server so you can easily look them up should you need them later. $ cp -r /var/cfengine/ppkeys /root/3.5/ppkeys $ cf-key -s > /root/3.5/hosts ## Prep your Policy Server for the upgrade There are a few things that need to be done prior to stopping the CFEngine processes on the Policy Server. ###### Merge your existing policies and masterfiles into CFEngine 3.6 This is going to be the most time consuming step as you will be merging all the policy bundles you have in your existing install into the CFEngine 3.6 version of the masterfiles. I recommend finding a separate machine on which you can perform a vanilla install of CFEngine 3.6.1. This way, you will be able to obtain pristine copies of the CFEngine 3.6 masterfiles from which you can start the merge process. Since you don't want to corrupt your existing CFEngine 3.5 environment, save the merged masterfiles into a directory location that is readily available, such as `/root/3.6/masterfiles`. You will be copying these files over to your upgraded Policy Server later. Since you have likely made a number of edits while merging the masterfiles, it's best to run them through `cf-promises` to ensure that they are well formed and will execute successfully in your existing environment, as well as in the upgraded CFEngine 3.6 environment. $ cf-promises -cf /var/cfengine/masterfiles/update.cf $ cf-promises -cf /var/cfengine/masterfiles/promises.cf If there is no resulting output from `cf-promises`, then you are all set to continue with the upgrade. Otherwise, you should fix any reported issues in your policies. ###### Set update\_policy.cf Open up the 3.6 version of `update_policy.cf` and set `trustkeysfrom` to trust all hosts. Without this setting, the hosts will not trust the IP address of the policy server once it has been upgraded to CFEngine 3.6. ## Export data from MongoDB (applicable to enterprise version installs only) If you are an enterprise customer, we've provided a utility called `cfmigrate` to help you export data from the previous MongoDB datastore. The data exported include users and role settings, as well as file-changes data. You can download a copy of `cfmigrate` from [here](https://s3.amazonaws.com/cfengine-package-repos/tools/cfmigrate). Once you have downloaded this handy utility onto the Policy Server, give it executable permissions and perform an export. $ chmod +x cfmigrate $ ./cfmigrate export By default, `cfmigrate` writes out data files extracted from MongoDB to the local directory. Towards the end of the Policy Server upgrade procedure, you will be executing `cfmigrate` again to import these data files into PostgreSQL. To see additional options for executing this utility, please refer to the help by issuing the following command: $ ./cfmigrate export -h ## Upgrade the Policy Server You are now ready to perform the actual upgrade of the Policy Server. This involves shutting down the existing CFEngine services, removing the existing instance, installing the new packages, and copying over the merged masterfiles. The documentation provides all of these steps, but I will repeat them here for your convenience. 1. Stop CFEngine Services $ service cfengine3 stop 2. Uninstall the previous CFEngine Policy Server package $ rpm -e cfengine-nova-hub # Red Hat based distribution $ dpkg -p cfengine-nova-hub # Debian based distribution $ rm -rf /var/cfengine 3. Install the new CFEngine Policy Server package. Click [here](https://cfengine.com/download/) to download the latest versions. $ rpm -i cfengine-nova-hub-3.6.1-1.x86_64.rpm # Red Hat based distribution $ dpkg --install cfengine-nova-hub_3.6.1-1_amd64.deb # Debian based distribution 4. Restore ppkeys from your backup $ cp /root/3.5/ppkeys/* /var/cfengine/ppkeys/ 5. Restore your Enterprise license file into `/var/cfengine/masterfiles/license.dat` 6. Remove the out-of-the-box masterfiles and replace them with your copy of the merged masterfiles $ rm -rf /var/cfengine/masterfiles/* $ cp /root/3.6/masterfiles/* /var/cfengine/masterfiles/ 7. Bootstrap the Policy Server to itself $ /var/cfengine/bin/cf-agent -B [Policy Server IP Address] 8. Import your enterprise data from MongoDB to PostgreSQL (applicable to enterprise version installs only) $ ./cfmigrate import To see additional options for running `cfmigrate`, please refer to `./cfmigrate import -h` 9. Verify that users can log in to the Mission Portal 10. Take the Policy Server online and verify that some clients have received the updated policies and can execute them without error ## Upgrade the Clients Now that you have successfully upgraded the Policy Server, you can now shift your focus to the clients. This one should be pretty straightforward. [Download the relevant client packages](https://cfengine.com/download/) onto the Policy Server, and then place them in `/var/cfengine/master_software_updates/` under the appropriate subdirectory for the operating systems of your hosts. When the client *"phones home"* to the Policy Server, they will find these updated packages to perform a *"self-upgrade"*. Note that you might also wish to take an iterative approach and limit the upgrade to a small subset of clients at a time. You can do this by setting the `trigger_update` class in `update.cf` to a subset of host IP addresses. Since you had previously set `trustkeysfrom` at the start of this entire process, you can now revert this back to its previous configuration in `update.cf` after the clients have been upgraded and are connected to the Policy Server. As the clients get upgraded, make sure that they appear in the Mission Portal if you are a user of the enterprise version. **And there you have it\! You have now successfully upgraded and are running our latest release, [CFEngine 3.6.1](https://cfengine.com/product/enterprise-3-6-0/)\!** -------------------------------------------------------------------------------- title: "Self-Repairing Deployment Pipelines" url: /blog/2014/self-repairing-deployment-pipelines/index.md date: "2014-08-05" -------------------------------------------------------------------------------- # *Or what we should mean by Distributed Orchestration* [![MakeVelocity.001.alt](images/MakeVelocity.001.alt_-300x165.png)](images/MakeVelocity.001.alt_.png) Orchestrating complicated distributed processes is an unfamiliar aspect of computing that leads to all kinds of confusions. We are not taught how to do it in college, so we end up trying to apply any methods we are taught, often in inappropriate ways. Promise theory paints a very simple picture of distributed orchestration. Rather than imagining that a central conductor (controller) somehow plays every instrument by remote magic wand, in an algorithmic fashion, promise theory says: let every player in an ensemble know their part, and leave them all to get on with it. The result is an emergent phenomenon. The natural dependences on one another will make them all play together. Over-thinking the storyline in a distributed process is the easiest way to get into a pickle. This is a key point: how we tell the story of a process and how it gets executed are two different things. Modern programming languages sometimes pretend they are the same, and sometimes separate the two entirely. [![MakeVelocity.002](images/MakeVelocity.002-300x170.jpg)](images/MakeVelocity.002.jpg) Scroll back time to 1992, and the world was having all the same problems as today, in different wrapping. Then there was `cron`, pumping out scripts on hourly, daily and weekly schedules. This was used not only for running jobs on the system but for basic configuration health checks. Cron scripts were like a wild horde of cats, each with their own lives, hard to bring into some sense of order. In the early 90s the acme of orchestration was to sort out all your cron jobs to do the right thing at the right time. The scripts had to be different on the machines too because the flavours of Unix were quite different -- and thus there was distributed complexity. Before CFEngine people would devise devious ways of creating one cronfile for each host and then pushing them out. This was considered to be orchestration in 1992. [![MakeVelocity.003](images/MakeVelocity.003-300x170.jpg)](images/MakeVelocity.003.jpg) One of the first use cases for CFEngine was to replace all of this with a single uniform model oriented language/interface. CFEngine was target oriented, because it had to be repeatable. Convergence . In this article I explain why virtual environments and containers are basically this issue all over again. [![MakeVelocity.004](images/MakeVelocity.004-300x170.jpg)](images/MakeVelocity.004.jpg) Another tool of this epoch is the `make` for building software from dependencies. In 1994, Richard Stallman pointed out to me that CFEngine was very like make. Indeed, this ended up influencing the syntax of the language. [![MakeVelocity.005](images/MakeVelocity.005-300x170.jpg)](images/MakeVelocity.005.jpg) The Makefile was different, it was the opposite of a script. Instead of starting in a known state and pushing out a sequence of transitions from there, it focused on the end state and asked how can I get to that desired end state? In math parlance, it was a change of boundary condition. This was an incredibly important idea, because it meant that -- no matter what kind of a mess you were in -- you would end up with the right outcome. This is far more important than knowing where you started from. [![MakeVelocity.006](images/MakeVelocity.006-300x170.jpg)](images/MakeVelocity.006.jpg) Makefiles did not offer much in the way of abstraction; you could substitute variables and make simple patterns, but this was sufficient for most tasks, because patterns are one of the most important mechanisms for dealing with complexity. Similarly, make was a serial processor running on a single machine, not really suitable for today's distributed execution requirements. The main concession to parallelism was the addition of "-j" to parallelize building of dependencies. [![MakeVelocity.008](images/MakeVelocity.008-300x170.jpg)](images/MakeVelocity.008.jpg) What was really needed was a model based approach where we could provide answers to the following questions: what, when, where, how and why. [![MakeVelocity.009](images/MakeVelocity.009-300x170.jpg)](images/MakeVelocity.009.jpg) So now we come to the world of today where software is no longer shackled to a workstation or a server, but potentially a small cog in a large system. And more than that - it is a platform for commerce in the modern world. It's not just developers and IT folks who care about having stuff built - it's everyone who uses a service. [![MakeVelocity.010](images/MakeVelocity.010-300x170.jpg)](images/MakeVelocity.010.jpg) Many of the problems we are looking to solve can be couched in the model of a deployment of some kind. Whether it is in-house software ("devops"), purchased off-the-shelf software (say "desktop") or even batch jobs in HPC clusters, all of these typically pass through a test phase before being deployed onto some infrastructure container, such as a server, process group, or even embedded device. [![MakeVelocity.011](images/MakeVelocity.011-300x170.jpg)](images/MakeVelocity.011.jpg) Alas the technologies we've invented are still very primitive. If we look back to the history of logic, it grew out of the need to hit objects with projectiles in warfare. Ballistics was the cultural origin of mathematics and logic in the days of Newton and Boole. Even today, we basically still try to catapult data and instructions into remote hosts using remote copies and shells. [![MakeVelocity.012](images/MakeVelocity.012-300x170.jpg)](images/MakeVelocity.012.jpg) So if a script is like a catapult, that takes us from one decision to the next in a scripted logic. Another name for this triggered branching process is a chain reaction (an explosion). A Makefile is the opposite: a convergent process like something sliding easily down a drain. The branching logic in a script leads to multitudes of parallel alternative worlds. When we branch in git or version control systems we add to this complexity. In a convergent process we are integrating possible worlds into a consistent outcome. This is the enabler for [continuous delivery](https://markburgess.org/blog_cd.html). [![MakeVelocity.013](images/MakeVelocity.013-300x170.jpg)](images/MakeVelocity.013.jpg) So developers might feel as though they have their triggered deployments under control, but are they really. No matter, we can go from this... [![MakeVelocity.014](images/MakeVelocity.014-300x170.jpg)](images/MakeVelocity.014.jpg) To this ... This picture illustrates for me the meaning of true automation. No one has to push a button to get a response. The response is proactive and distributed into the very fabric of the design -- not like an add-on. The picture contrasts how we go from manual labour to assisted manual labour, to a proper redesign of process. Automation that still needs humans to operate it is not automation, it is a crane or a power-suit. [![MakeVelocity.015](images/MakeVelocity.015-300x170.jpg)](images/MakeVelocity.015.jpg) CFEngine's model of promises is able to answer all of the questions what, when, where, how and why, at a basic level and has been carefully designed to have the kind of desired-end-state self-healing properties of a drain. Every CFEngine promise is a controlled implosion that leaves a desired end-state. [![MakeVelocity.016](images/MakeVelocity.016-300x170.jpg)](images/MakeVelocity.016.jpg) Today, configuration promises have to be supported across many different scales, from the smallest containers like a user identity, to processes, process groups, virtual and physical machines, local networks, organizational namespaces and even globally spanning administrative domains. How do we do that? The simple answer is that we always do it "from within" -- through autonomous agents that collaborate and take responsibility for keeping desired-end-state promises at all levels. [![MakeVelocity.017](images/MakeVelocity.017-300x170.jpg)](images/MakeVelocity.017.jpg) Traditionally, we think of management of boxes: server boxes, rack boxes, routing boxes, etc. We can certainly put an agent inside every one of those processing entities... [![MakeVelocity.018](images/MakeVelocity.018-300x170.jpg)](images/MakeVelocity.018.jpg) But we also need to be able to address abstract containers, labelled by the properties we use in our models of intent -- business purpose. These are things like: linux, smartos, webservers, storage devices, and so on. They describe the functional roles in a story about the business purpose of our system. [![MakeVelocity.019](images/MakeVelocity.019-300x170.jpg)](images/MakeVelocity.019.jpg) This brings up an important issue: how we tell stories. Despite what we are taught in software engineering, there is not only one version of reality when it comes to computer systems. There is the story: - Told by the designer - Told by the programmer - Told by the debugger - Told by the user... - etc When building software, it's all "if this then that, else while do..." That highly serialized (step by step) story only lives in the programmer's mind. It is an artificial view of reality designed to perpetuate a view of determinism. It is not what the system does. Often when something goes wrong with a system, we start telling simplified stories that focus on "root causes", which is again trying to condense the indeterminism of a system down to a simple deterministic fiction -- a story we can live with. So there is a serial story of transitions we go through when building systems that is represented by commands and hands-on interaction. Then there is a story about how tasks are delegated to different parts of the system. What is important to understand is that we should not try to tie the story about what a system does too closely to how it does it. This leads to inefficiencies and even problems. The goal of an orchestration tool is to try to paint a suitable story around *purpose* that makes a good compromise on these issues. Then we can leave the execution of the story to the individual characters in the play. [![MakeVelocity.020](images/MakeVelocity.020-300x170.jpg)](images/MakeVelocity.020.jpg) What CFEngine introduced was a kind of "make" for desired system state, which had types of target promises - not just files to be built or installed, but processes to run, services to maintain, and so on. It was a make not only for the binary data, but also for the runtime state. Moreover, it could span any number of machines and use the container abstractions to assign each part of the music to its player. [![MakeVelocity.021](images/MakeVelocity.021-300x170.jpg)](images/MakeVelocity.021.jpg) Embedded in this orchestration framework is still good old "make" and its rule based evaluation. [![MakeVelocity.022](images/MakeVelocity.022-300x170.jpg)](images/MakeVelocity.022.jpg) But it is a make that has storytelling attributes so that we can understand the purpose of the system independently of the algorithm used to build and maintain it. [![MakeVelocity.023](images/MakeVelocity.023-300x170.jpg)](images/MakeVelocity.023.jpg) In addition it has built-in tools that are tailored to the special transformations that are needed to build and maintain the full software stack. These tools had to be borrowed from other sources in make. [![MakeVelocity.024](images/MakeVelocity.024-300x170.jpg)](images/MakeVelocity.024.jpg) Now that we have a more powerful make that can handle static and runtime state across any kind of physical or abstract region, we turn it into a distributed service by making the agent on each separate box aware of all the context it needs to figure out its role in the whole system. Then, as long as we have defined desired outcome into simple target "buckets", convergence and orchestration of that state can happen entirely autonomously: without any human intervention. [![MakeVelocity.025](images/MakeVelocity.025-300x170.jpg)](images/MakeVelocity.025.jpg) Host containers can even spawn guest containers to keep specialized environments separate. This doesn't necessarily help execution, but it simplifies the telling of the story for human shepherds. So: how do we orchestrate a distributed process of assembling some software in an environment like this? It's simple: give everyone their job description and let them get on with it. [![MakeVelocity.026](images/MakeVelocity.026-300x170.jpg)](images/MakeVelocity.026.jpg) The trick is to use policy as the musical score for the orchestration. Policy assigns roles and appropriate guidance to each potential player in an ensemble. It can even assign roles to abstract groups that don't exist yet\! The idea is very flexible. In this example, we coordinate some docker containers (children) inside a parent host and enable them to work together, sharing necessary information, but not trying to push them into a sequence of steps like a programmer. A schematic of the arrangement: **Need to Insert Schematic here** [![schematic](images/schematic-300x160.png)](images/schematic.png) [![MakeVelocity.027](images/MakeVelocity.027-300x170.jpg)](images/MakeVelocity.027.jpg) The parent container promises to make specialized children each with a particular task. It makes its data available to them in their "in-boxes", and agrees to collect any results from them from their "out-boxes". Using make semantics, it can check to see whether any changes have been made that need to be picked up and integrated into a final result. [![MakeVelocity.028](images/MakeVelocity.028-300x170.jpg)](images/MakeVelocity.028.jpg) There is a CFEngine agent in each of these containers. Each of the agents knows the part it has to play, and the result will now emerge from the design, without having to drive the story step by step from some central controller. In the example, I used docker containers. For interested docker fans, this is the entire docker file. It is the analogue of the cron file I showed above. The principle is the same: replace all of the burden of handling variations on a simple model-based automation, designed to handle abstractions. All of the modelling is in CFEngine policy so that the story is told at an appropriate level of abstraction, rather than confusing a bricklayer version with an architect version. [![MakeVelocity.029](images/MakeVelocity.029-300x170.jpg)](images/MakeVelocity.029.jpg) The advantage of using this approach is that it can work even for the partially connected devices like mobile phones and pads that are increasingly a vital part of the software chain. Because there is no need for a micro-managed sequential bricklaying approach to ensuring process completion, the emergent effect can happen independently inside every container without different containers preventing one another from making progress by having to wait. [![MakeVelocity.032](images/MakeVelocity.032-300x170.jpg)](images/MakeVelocity.032.jpg) The ability to run on networking devices and incorporate them into the story is now a critical part of the story-telling. The abstractions that we are building are pushing all of the complexity down onto networking engineers. They have not yet benefitted from the abstractions that Open Source innovation used to transform server management. Now that is possible, and they become part of the single orchestrated storyline. [![MakeVelocity.033](images/MakeVelocity.033-300x170.jpg)](images/MakeVelocity.033.jpg) The goal of distributed management as story-telling is to make comprehensible system documentation directly executable by distributed agents so that there is only one version of events: the one that describes our intent rather than an explosion of changes triggered by a push button trigger. [![MakeVelocity.034](images/MakeVelocity.034-300x170.jpg)](images/MakeVelocity.034.jpg) The idea of story telling as a crucial part of human-machine design is part of a larger narrative. It starts with humans solving problems by trial and error. Then we learn better approaches, practice and rehearse. Once we have it down to a mechanical repetition, we are basically acting like machines, so why not get a machine to do it? Our first attempts at mechanizing are to make machines that imitate how humans work. Finally we separate the human activity from a true form of automation, just like the storm drain design in the illustration above. [![MakeVelocity.035](images/MakeVelocity.035-300x170.jpg)](images/MakeVelocity.035.jpg) I wanted to show how to make an orchestrated outcome *emerge* from a description of the different roles in a process. This is an unfamiliar idea to many, because we are not taught emergent methods in college. But we need to be less afraid of emergence, because all of the things we rely most on in our lives actually work in this way (see my blog [The Brain Horizon](https://markburgess.org/blog_brain.html)). When you don't have to worry about how something works, you can focus on what it does. This is how we need to be thinking about complex, multi-scale processes. Read more about these issues in my book: [In Search of Certainty](https://markburgess.org/certainty.html) [Watch the CFEngine for Docker automation demo](https://www.youtube.com/watch?v=XnwOSJCVogc&list=UUAiKge7NwhuWi-KNKt3U4HA "CFEngine and Docker - Container Automation") The CFEngine example policy used is shown below. It can also be downloaded [here](images/promises.cf) \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \# \# Orchestration of containers \# \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle common my { vars: "parent\_address" string =\> "172.17.42.1"; "name" string =\> "$(sys.uqhost)"; "jobs" slist =\> { "container\_1", "container\_2", "container\_3" }; classes: "containers" or =\> { @(jobs) }; } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent main { methods: parent:: "prepare continuous delivery source" comment =\> "The files where people make changes"; "collect build products" comment =\> "Reap the harvest"; "package for delivery" comment =\> "Tie it with a bow"; containers:: "fetch from the parent" comment =\> "pick up files and build something"; services: parent:: "children" comment =\> "Spawn some fledgings"; } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \# Methods \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent method\_prepare\_continuous\_delivery\_source { files: \# Make one task for each bud "/tmp/feeds/$(my.jobs)\_feed.txt" create =\> "true", edit\_line =\> append\_if\_no\_line("This is for $(my.jobs) from $(sys.uqhost)"); } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent method\_collect\_build\_products { files: "/tmp/results/." create =\> "true"; "/tmp/results/reply\_from\_$(my.jobs)" comment =\> "Pick up the latest on the pipeline", copy\_from =\> secure\_cp("/tmp/result\_for\_pickup.txt","$(sys.docker\_guest\_ip\[$(my.jobs)\])"); } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent method\_package\_for\_delivery { vars: "all\_files" slist =\> { "/tmp/results/reply\_from\_container\_1", "/tmp/results/reply\_from\_container\_2" }; commands: "/bin/tar zcf /tmp/package.tgz /tmp/results" if =\> makerule("/tmp/package.tgz", "@(all\_files)"); } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent service\_children(name,state) { guest\_environments: parent.\!cleanup:: "$(my.jobs)" guest\_details =\> stem\_cell, guest\_state =\> "create"; cleanup:: "$(my.jobs)" guest\_details =\> stem\_cell, guest\_state =\> "delete"; reports: \!cleanup:: "CONTAINERS running"; cleanup:: "CONTAINERS stopped"; } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# body guest\_details stem\_cell { guest\_type =\> "docker"; guest\_image\_name =\> "cf-stem-cell"; } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \# Containers \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# bundle agent method\_fetch\_from\_the\_parent { services: "ssh" comment =\> "So we can log in and inspect the containers"; "cfengine" comment =\> "Need the server running"; files: "/tmp/job\_description.txt" comment =\> "Pick up the latest on the pipeline", copy\_from =\> secure\_cp("/tmp/feeds/$(my.name)\_feed.txt","$(my.parent\_address)"); "/tmp/result\_for\_pickup.txt" create =\> "true", comment =\> "Do my part and deliver the result for pickup", edit\_line =\> signature("/tmp/job\_description.txt", "A postcard from sunny $(sys.ipv4) - Thanks, with love $(my.name)"); } \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \# Misc - try to keep this all in one file \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# body common control { bundlesequence =\> {"my", "main"}; } \#\# bundle server access\_control { access: "/tmp" admit =\> { "172.17.0.0/16" }; } \#\# body server control { allowconnects =\> { "172.17.0.0/16" }; allowallconnects =\> { "172.17.0.0/16" }; trustkeysfrom =\> { "172.17.0.0/16" }; } \#\# bundle agent service\_ssh(a,b) { processes: "/usr/sbin/sshd" restart\_class =\> "restart\_ssh"; commands: "/usr/sbin/sshd" ifvarclass =\> "restart\_ssh"; } \#\# bundle agent service\_cfengine(a,b) { processes: "cf-serverd" restart\_class =\> "restart\_cfengine"; commands: "/var/cfengine/bin/cf-serverd" ifvarclass =\> "restart\_cfengine"; } \#\# body copy\_from secure\_cp(from,server) { source =\> "$(from)"; servers =\> { "$(server)" }; compare =\> "digest"; encrypt =\> "true"; verify =\> "true"; trustkey =\> "true"; } \#\# bundle edit\_line append\_if\_no\_line(str) { insert\_lines: "$(str)" comment =\> "Append a line to the file if it doesn't already exist"; } \#\# bundle edit\_line signature(file,str) { insert\_lines: "$(file)" insert\_type =\> "file"; "$(str)"; } \#\# body contain in\_dir(s) { chdir =\> "$(s)"; } -------------------------------------------------------------------------------- title: "Achieving visibility into Arista network inventory (part 1)" url: /blog/2014/achieving-visibility-into-arista-network-inventory-part-1/index.md date: "2014-07-30" -------------------------------------------------------------------------------- ***This blog is the result of joint technology collaboration between Arista Networks and CFEngine Inc.*** ## Introduction It is an accepted fact that the disciplines of system administration and network administration are traditionally very disjoint: different teams, different knowledge, different tools and different goals. For example, does the networking team know which nodes and applications are affected by a change on a specific switch? Is the network topology actually the way it says it is in the design diagrams? Unfortunately, network design diagrams and server CMDBs quickly grow out of sync with reality in today's fast-changing world. This makes questions like the above virtually impossible to answer, thus leaving luck to play an important role in what changes lead to downtime, loss of revenue and late-hour emergency fixes. However, times are changing and one of the promises of the DevOps movement is enabling better collaboration across these teams. The first step towards collaboration is to enable holistic visibility of the current configuration of the environment, which includes applications, servers and the network connecting them. A popular use case for CFEngine today is to update server CMDBs with the current information. CFEngine can do this as it is running on all the nodes and provides a server-side API for the inventory - as it was seen 5 minutes ago. The missing piece is to also extend this and enable this in the network layer. In this blog series, **a joint-effort between Arista and CFEngine** **focused on network automation**, we will look at how to discover and report complete inventory information with CFEngine running on [Arista](https://www.arista.com) switches (this part), as well as how to easily extend the information using the new Arista APIs (part two). ## Setting up Arista vEOS Arista provides a virtual appliance of their software stack, so that it can be run on most hypervisors. This is what we will use for this tutorial. We can get the vEOS and aboot from , specifically . In this example, we used vEOS-4.13.7M.vmdk and Aboot-veos-2.0.8.iso with VirtualBox 4.1.12. To configure the Arista VM, set Fedora 32 bit as VM type, memory to 2GB, host only network, the disk to the vmdk file and boot from the aboot iso as a CD/DVD drive. It is important that the iso and vmdk file are on the same IDE controller, as shown in the screenshot below. [![vbox-eos-settings](images/vbox-eos-settings.png)](images/vbox-eos-settings.png) Now you can boot the VM, and you should be taken to a login screen. ## Setting up CFEngine #### The Policy Server We create a new VM with CentOS 6 that will serve as the CFEngine Enterprise Policy Server. For the network, we will use the same host only adapter as we did when creating the vEOS VM, and after creation set the IP address with `ifconfig eth0 192.168.33.100` (check the subnet on your hosts' vboxnet0 interface). We are now able to scp in and install the CFEngine Enterprise Red Hat 6 hub package found at the [CFEngine download page](https://cfengine.com/product/free-download) and bootstrap the hub to itself with `/var/cfengine/bin/cf-agent -B 192.168.33.100`. Make sure that the Policy Server does not have a firewall on port 5308, e.g. by disabling it with `iptables -I INPUT -j ACCEPT`. #### Arista vEOS switches as clients For the client, we log in to the Arista vEOS VM with the admin user, using a blank password. By typing `enable`, `bash` and `sudo -s`, we get into a bash shell as root, and set the IP address with `ifconfig ma1 192.168.33.101`. To install CFEngine Enterprise, we can use the "32-bit RPM-based distribution" Host package found on the CFEngine download page above. Finally, we bootstrap the Arista switch to the hub with `/var/cfengine/bin/cf-agent -B 192.168.33.100`. You might also need to configure the firewall to allow incoming connections from the policy server on port 5308, like for the Policy Server. You can use the same procedure to bootstrap more vEOS switches to CFEngine. CFEngine Enterprise will now start to collect reports from the Arista switch. You should see it in the Mission Portal web interface within 15 minutes (log in to http://192.168.33.100, admin/admin). ## Inventory reports out of the box By simply adding the Arista switches to CFEngine's management, we get inventory reports out of the box. Simply log in to the Mission Portal, go to Reports -\> Inventory, and you will be presented with the out of the box inventory attributes collected by CFEngine. The example below shows some general attributes, as well as the ports that are listening. [![inventory-ootb](images/inventory-ootb-1024x682.png)](images/inventory-ootb.png) We can also easily create charts of the inventory, e.g. the OS, as shown below. [![inventory-ootb-chart](images/inventory-ootb-chart.png)](images/inventory-ootb-chart.png) ## Summary This tutorial showed how we can easily add Arista switches under CFEngine management to gain visibility of them and their inventory. We also looked at how to effectively create inventory reports using the CFEngine Enterprise 3.6 inventory reporting interface. In the next part to this series, we will look at how to extend this information - in particular how to detect network topology relevant for making changes to the infrastructure. Please drop us a note if you have any questions or comments about Arista and network management with CFEngine Enterprise\! *Note: CFEngine is taking big strides in engaging with our network ecosystem of partners. In a recent blog post we detailed how to make IP (Layer3) a first-class netizen in the configuration world. You can read that piece [here](https://cfengine.com/company/blog-detail/ip-layer-3-configuration/)* -------------------------------------------------------------------------------- title: "CFEngine Enterprise 3.6.1 Maintenance Release Now Generally Available" url: /blog/2014/cfengine-enterprise-3-6-1-maintenance-release-now-generally-available/index.md date: "2014-07-28" -------------------------------------------------------------------------------- If you've seen what we have delivered with CFEngine Enterprise 3.6.0 earlier this summer, I'm sure you realized what a feature-rich release this turned out to be. But we didn't stop there. The CFEngine team has spent the past few weeks validating additional platforms, building a simplified upgrade process, and in the process, even added a few small enhancements to the mix\! Today, we're announcing CFEngine Enterprise 3.6.1 - a maintenance release with a number of goodies that I'm sure you will appreciate. **Expanded Platform Coverage** CFEngine Enterprise 3.6.1 now supports the following host platforms to provide you with comprehensive coverage within your IT infrastructure. | | | | -------------------------------------------------------------------------- | ----------------------------------------------------- | | **Operating System** | **Architecture** | | **IBM AIX 5.3** | PowerPC | | **IBM AIX 6.1** | PowerPC | | **IBM AIX 7.1** | PowerPC | | **Red Hat Enterprise Linux 4.x** | i386 | | **Red Hat Enterprise Linux 4.x** | x86-64 | | **SUSE Linux Enterprise Server 10.x** | i386 | | **SUSE Linux Enterprise Server 10.x** | x86-64 | | **SUSE Linux Enterprise Server 11.x** | i386 | | **SUSE Linux Enterprise Server 11.x** | x86-64 | | **Oracle Solaris 9.x** | SPARC | | **Oracle Solaris 10.x** | UltraSPARC | | **Oracle Solaris 10.x** | x86-64 | | **Windows Server 2008** | IA-32 and x86-64 | | **Windows Server 2008 R2** | x86-64 | | **Windows Server 2012** | x86-64 | **Streamlined Upgrade to CFEngine Enterprise 3.6.1** A lot of you are eager to get your environments up to version 3.6.x, and we want to make that upgrade process as seamless as possible for you. CFEngine Enterprise 3.6.1 supports automated upgrade from Enterprise versions 3.0 and 3.5.x. Upon upgrade of the policy hub, data from MongoDB will be extracted and imported in to the corresponding 3.6.1 schemas in PostgreSQL, as well as Mission Portal settings, user preferences, users and role definitions. Upgrading of agent hosts will be automated via the cf-upgrade binary. Additional information is available in the [documentation](https://docs.cfengine.com/docs/3.6/guide-installation-and-configuration-upgrading-to-36.html). **Minor Enhancements to the Mission Portal** We've made some minor tweaks to the Mission Portal user interface that will improve the overall experience of the inventory module. For starters we've added an out-of-the-box "Disk Free (%)" inventory attribute that gives you an indication of how much disk space is available on the server. You can therefore be alerted during situations when the disk space is running low[![](images/diskfree1.png)](/blog/2014/images/diskfree1.png) [![diskfree2](images/diskfree2-1024x561.png)](images/diskfree2.png) We've also made improvements to the inventory reporting column selector. It is now much more interactive, and simpler to add/delete columns on your reports on the fly: [![columnselector](images/columnselector-1024x445.jpg)](images/columnselector.jpg) In addition, if your Mission Portal implementation leverages a 3rd-party authentication mechanism such as LDAP or Active Directory, we've made changes to drastically improve Mission Portal performance. **Bug fixes** As always, maintenance releases include a number of bug fixes, and this release is no different. Visit our [changelog](https://github.com/cfengine/core/blob/3.6.x/ChangeLog) to view the bug fixes that have been merged into CFEngine Enterprise 3.6.1. So, go [download](https://cfengine.com/download/) CFEngine Enterprise 3.6.1 and experience it for yourself. Give it a whirl, and let us know what you think\! -------------------------------------------------------------------------------- title: "Recording Now Available: CFEngine Webinar for Financial Services Organizations" url: /blog/2014/watch-cfengine-webinar-financial-services-organizations/index.md date: "2014-07-18" -------------------------------------------------------------------------------- Thanks to all of you who tuned in for our special webinar this week 'Drive Financial Services Innovation Through IT Automation'. For those of you who could not attend, the recording is now available below - please be sure to watch and learn how CFEngine Enterprise 3.6.0 can enable your IT organization to automate applications and infrastructure specifically for challenges facing financial services firms: {{< youtube ctNaljGgorY >}} You can also check out the slides [here](http://www.slideshare.net/cfengine/webinar-drive-financial-services-innovation-through-it-automation "CFEngine Webinar Slides"). Happy viewing\! -------------------------------------------------------------------------------- title: "Join our Webinar for Financial Services Organizations This Thursday" url: /blog/2014/join-webinar-financial-services-organizations-thursday/index.md date: "2014-07-14" -------------------------------------------------------------------------------- Today's financial services firms are operating in an IT environment of ever-increasing complexity, unprecedented regulatory requirements, and a need to move faster than ever before. In order to effectively compete in this new reality, firms are turning to IT automation as a way to scale their systems without sacrificing security, uptime, or compliance. Join us for a special webinar this Thursday July 17th to learn about the new **CFEngine Enterprise 3.6.0** and how it can enable your IT organization to automate applications and infrastructure specifically for the challenges facing financial services firms today. Attend this live event to learn how CFEngine Enterprise 3.6.0 can enable your financial services organization to:
- Simplify audits with continual monitoring of system compliance and automatic remediation - Dramatically improve the speed to update your environments - Sharpen your visibility into critical configuration items and system inventory - Ensure your system is secure and in compliance with defined policy **Thursday July 17th** **10:00am PT / 1:00pm EDT (US), 6:00pm London UK (BST)** **[REGISTER TODAY](https://cfengine.com/webinars)**
-------------------------------------------------------------------------------- title: "CFEngine and Mark Burgess at Silicon Valley Devops Days" url: /blog/2014/mark-burgess-silicon-valley-devops-days/index.md date: "2014-07-11" -------------------------------------------------------------------------------- As any of you know, CFEngine was a proud Gold Sponsor of the [Silicon Valley 2014 DevOps Days Conference](https://devopsdays.org/events/2014-siliconvalley/) last month in Mountain View, along with other industry-leading sponsors such as IBM, CA Technologies, Dell, and Red Hat. If you were unable to attend this sold-out event, we are pleased to announce that a recording of CFEngine CTO Mark Burgess's talk on **'The Future of Configuration Management'** is now available online\! You can watch Mark's talk [here](https://web.archive.org/web/20151115113625/http://www.ustream.tv/recorded/49677732). To learn more about DevOps Days and find a future event near you, please check out the [conference website](https://devopsdays.org/). We hope to see you at the next event\! -------------------------------------------------------------------------------- title: "Watch the Introduction to CFEngine Enterprise 3.6.0 Webinar Recording" url: /blog/2014/introduction-cfengine-enterprise-3-6-0-webinar-demand/index.md date: "2014-07-10" -------------------------------------------------------------------------------- Thanks to all of you who tuned in for our 'Introduction to CFEngine Enterprise 3.6.0' Webinars last week. For those of you who could not attend, the recording is now available below - please be sure to watch and learn about CFEngine Enterprise 3.6.0 can help you: - Proactively alert on policy drifts in your environment through a simple-to-use, configurable dashboard - Gain comprehensive visibility into your IT infrastructure with enhanced compliance and extensible inventory reporting - Leverage new language abstractions to improve administration productivity - Achieve painless integration with other IT systems using native JSON support {{< youtube w-E5qKjuvII >}} -------------------------------------------------------------------------------- title: "Getting Started with CFEngine Webinar This Thursday" url: /blog/2014/join-vertical-sysadmin/index.md date: "2014-07-07" -------------------------------------------------------------------------------- Join CFEngine partner [Vertical Sysadmin](https://verticalsysadmin.com/) for a free webinar this Thursday - *Getting Started with CFEngine: Basic Concepts and Q\&A*. This event will cover topics from high-level system architecture to the anatomy of writing a promise to help you get productive quickly with CFEngine. Specific items to be covered include: - Basic terms - Promises, Bundles, Bodies, and Classes - Language concepts - Common promise types, attributes, and syntax - Design Center sketches - GUI features of CFEngine Enterprise **Date: Thursday, July 10, 2014** **Time: 9:00 am to 10:00am London UK, 4:00pm to 5:00pm Singapore (SGT)** **[Register today\!](https://www.eventbrite.com/e/getting-started-with-cfengine-basic-concepts-and-qa-tickets-11977907243)** -------------------------------------------------------------------------------- title: "Making IP (Layer 3) a First-Class Netizen in the Configuration World" url: /blog/2014/ip-layer-3-configuration/index.md date: "2014-07-03" -------------------------------------------------------------------------------- *Networking has become a utility in our daily lives, but it is still evolving as a platform for modern commerce. CFEngine has now taken networking beyond simple automation, by bringing modern self-repairing, model-oriented approaches to WebScale management at the IP layer.* Network interface configuration is nothing new in the world of configuration management. CFEngine versions 1 and 2 were plugging the obvious hole in the out-of-the-box experience during the mid 1990s, before operating systems finally got their acts together. In 2007, CFEngine even explored Layer 3 WAN management (IP routing by any other name) as a member of the EMANICS network of excellence - but, back then, the market was not ready. The dominant networking vendors were still largely concerned with proprietary architecture, steeped in feature-based competition. However, the more recent arrival of Linux as a network operating system has changed that. The adoption and development of the Linux kernel by several companies has pulled down the vendor wall to let in Open Source. Significantly, for the industry, the growth fueled by Open Source has led to the birth of modern datacenter engineering, and DevOps. Recently, with the discussion of Software Defined Networking (SDN), and the arrival of DevOps in the networking space, several tool providers have waded into the world of Layer 2 networking - bonds, bridges and VLANs - which is the comfort zone of most system administrators. However, the historically complex Layer 3 routing services have thus far been neglected. In order to change this, CFEngine founder Mark Burgess has been working with Dinesh Dutt of [Cumulus Networks](http://cumulusnetworks.com/ "Cumulus Networks") to come up with a model of Layer 3 networking based on CFEngine's Promise Theory approach to scalable management. ## SDN, by any other name The promise of SDN was to make networking better support business value, i.e. applications, while bringing a more policy-based approach to management with version control. This was the world that CFEngine brought to IT Automation and Management - now often called \`Infrastructure as Code'. However, new challenges often beat a retreat to centralized management. The central controller models of OpenFlow and other virtualized implementations of SDN provide detailed programming models, but this is done at the cost of re-centralizing the control model through a complex API. Moreover, the SDN APIs push the burden of implementation onto a new breed of programmer, who is expected to understand the complex historical meanderings of network management - an expensive and optimistic proposition. Conversely, our research into Promise Theory at CFEngine has been to leverage patterns and create desired end-state technology with self-repairing capability, which is the so-called \`executable documentation' model. By declaring simple promises and relying on its configuration engine, one can go beyond simple automation to bring vertical and horizontal resilience to network structures. CTO Mark Burgess says, "We did not want to just make an API to pass on the complexity of these systems to developers. We wanted to fundamentally simplify L3 configuration so that anyone could understand it. Much of the work went into distilling the complexity down to a manageable level, without sacrificing all-important decentralized aspects." ## From LAN to WAN and back Layer 3 routing protocols are notoriously arcane inventions, evolved over many years against a changing backdrop of business needs, so it is not surprising that this has taken an innovative approach. But today, Wide Area Networking models are becoming the way to scale the fattening Local Area Networks in datacenters. For example, a simple 2x2 redundant routing block (see fig 1), equilibrated by OSPF might look as simple as this (with unnumbered interfaces): [![2x2](images/2x2-150x150.png)](images/2x2.png) *Figure 1. 2x2 Redundant Routing Block.* ```cf3 bundle agent main() { interfaces: "swp1" link_services => ospf_area("0"); "swp2" link_services => ospf_area("0"); } body routing_services control { ospf_redistribute => { "kernel", "static" }; # Map linux hostnames to a router_id hostname_1:: ospf_router_id => "1.1.1.1"; hostname_2:: ospf_router_id => "2.2.2.2"; hostname_3:: ospf_router_id => "3.3.3.3"; hostname_4:: ospf_router_id => "4.4.4.4"; } ``` The ability to know the intended state, and to monitor the actual state, while using its machine-learning to trace performance characteristics now drops out of the CFEngine framework for free. (Note that the bulk of the configuration lies in naming the boxes -- a generic theme when managing name-spaces, according to promise theory.) This pattern is supplemented by a basic template like this, which can be kept in a standard library for re-use: ``` body link_services ospf_area(area) { ospf_area => "$(area)"; ospf_authentication_digest => "ABCDEFGHIJK"; ospf_link_type => "point-to-point"; ospf_hello_interval => "5"; } ``` The purpose of this very generic code is clearly not to micromanage details through continuous tweaking, but rather to make the problem go away as a managed service. Currently, CFEngine has built its reference implementation as part of the development-friendly Cumulus Linux ecosystem. ## The house that Clos built Large datacentres today are using non-blocking Fat Tree architectures to scale traffic levels horizontally with commodity hardware. These structures were pioneered by Charles Clos in the 1950s. The highly regular [Clos networks](http://en.wikipedia.org/wiki/Clos_network "Clos network") are well suited to patterns of promises. Dynamical change for application management can be handled by protocols like iBGP, without bringing new virtualization into the picture. For example, repeating the approach for iBGP, one might imagine a similar data-driven pattern for a 2x5 tree (fig 2): [![2x5](images/2x5-300x101.png)](images/2x5.png) *Figure 2. 2x5 Tree Diagram.* bundle agent LeafSpine { vars: # Generate the interface lists used on the routers "spine" slist => expandrange("swp[1-5]", "1"); # point to 5 leafsw "leaves" slist => expandrange("swp[1-2]", "1"); # point to 2 spinesw "net_adverts[leaf1]" slist => { "10.10.10.1/24", "10.10.20.1/24" }; "net_adverts[leaf2]" slist => { "10.10.30.1/24", "2001:0DB9:0:f101::1/64" }; "net_adverts[leaf3]" slist => { "192.168.1.0/24" }; "net_adverts[leaf4]" slist => { "192.168.1.0/24" }; "net_adverts[leaf5]" slist => { "192.168.1.0/24" }; "router_id[spine1]" string => "2.0.0.1"; "router_id[spine2]" string => "2.0.0.2"; "router_id[leaf1]" string => "1.0.0.1"; "router_id[leaf2]" string => "1.0.0.2"; "router_id[leaf3]" string => "1.0.0.3"; "router_id[leaf4]" string => "1.0.0.4"; "router_id[leaf5]" string => "1.0.0.5"; interfaces: spine:: "$(spine)" link_services => ibgp_reflector("server"); leaves:: "$(leaves)" link_services => ibgp_reflector("client"); } With the data specified in an associative array, two generic promise patterns are sufficient to configure the 2x5 tree. ## Model-based for WebScale Model-based configuration is not just a "nice to have", it is a "must have" feature of a scalable architecture. The three challenges of today's software stack remain: scale, complexity and knowledge management. Brute force can handle the first of these, but going forward, using a model-based approach is the only plausible option. What CFEngine adds is its usual promise of self-healing convergent end-state management and a knowledge-oriented approach. A simplification of datacenter operations, by de-coupling fast and slow changes with a versioned set of promises, takes away several burdens that haunt network engineers. This is about enabling DevOps to work collaboratively across the application-network divide. And that kind of evolution in the management of network services is long overdue. -------------------------------------------------------------------------------- title: "Guest Blog - WebScale Computing at the Flick of a Wrist" url: /blog/2014/webscale-computing-flick-wrist/index.md date: "2014-07-02" -------------------------------------------------------------------------------- In this installment we turn to Jon Henrik Bjørnstad. In this particular piece read about configuring, and managing connected devices with CFEngine. If you have other such stories of CFEngine use, we would love to hear back from you. Thanks Jon Henrik for a great post\! *![WMe\_wristband\_3](images/WMe_wristband_3-300x148.jpg)The world of partially connected devices forms the basis of both mobile IT and the much discussed Internet of Things ("IoT"). Hand-held tablets and phones equipped with applications for interacting with various services are merely the beginning of the pervasive computing experience, and smart home projects by Xerox, HP and Google merely underestimate the possible role of embedded devices in society at large.* The Android operating system is based on a stripped down Linux kernel, and is highly portable. This open platform has spread to everything from television sets and refrigerators to cars and in-flight entertainment systems. This diaspora of partially-connected devices is the interaction point for users and contributes in no small part to creating WebScale workloads. Fortunately, the issue of configuration, security and maintenance in a partially connected environment is the natural domain of CFEngine due to its distributed architecture and lightweight autonomous agent. **Self-healing configuration on the move** CFEngine was designed with the world of embedded computing firmly in mind (see CFEngine founder Mark Burgess' novel [Slogans](https://www.amazon.com/Slogans-Mark-Burgess-ebook/dp/B00FAJB80M/ref=sr_1_4?ie=UTF8&qid=1404322375&sr=8-4&keywords=slogans) from 2005, if you want to see how Mark was thinking). Today, any device - whether an interactive Parisian billboard, car system, smartphone, or experimental robot at the National Institute of Standards - can benefit from CFEngine's configuration technology. Unlike MDM solutions, CFEngine works offline and is self-repairing. What kinds of issues are there for configuration in a distributed on-and-offline embedded world? There are plenty of issues, but to see how a simple policy works, here is an example of how we might configure certain application defaults to make a device (in this case idle phones) less vulnerable to casual interference. **Database configuration settings** Android's "system registry" is the SQLite key-value store, and database promises in CFEngine Enterprise 3.6.x may be used to handle SQLite with simple row manipulation. Many embedded systems use SQLite as the backend for storing system configuration as key-value pairs; among them are the feature rich and Open Source Storage operating system FreeNAS in addition to the omnipresent platform for mobile devices, Android. We will cover configuration within FreeNAS in a later post. Key-value pair tables are ideal for the type of simple row manipulation CFEngine offers. Rows are simply updated or added according to a condition (a WHERE in classical SQL) and the values that should converge. When used with associative arrays this becomes a flexible and handy way of expressing system configuration. This example is quite mundane, but imagine that you cared about the screen timeout setting on your Android device because you don't want a phone lying around to remain unlocked. You want it to be set to 15 seconds (15000ms). In CFEngine you would express this as configuration setting as follows: **bundle agent android_settings ** **{** **databases:** **"settings.db/system"** **database\_server =\> sqlite("/data/data/com.android.providers.settings/databases"),** **database\_rows =\> { "name='screen\_off\_timeout',value=\>'15000'" },** **}** Now, even barring upgrades, resets or manual tampering, this policy will be checked regularly and restored to the policy settings by CFEngine's agent. There is no need for a network connection. Everything is local. The database table you currently want to configure is expressed as the promiser, in this case the "system" table in the "settings.db" database. As SQLite databases are embedded, you would also need to provide the path to the database which happens through the SQLite database\_server body. The WHERE clause is expressed with a simple '=' sign while the values that should be set (or converge) are expressed with an '=\>' sign (read: converge to). CFEngine imposes no limit on the number of WHERE clauses you can use or values you could set, but it will of course need to match the underlying table schema. Picture 1 below shows state before policy is applied (screenshot of a Samsung S2 phone): ![](images/Android_sqlite_blog_post_img1.jpg) We run the policy and see the following: **\# cf-agent -I ** **2014-07-02T13:41:51+0200 info: /default/android\_settings/databases/'settings.db/system'\[0\]: Table 'system' is as promised** **2014-07-02T13:41:51+0200 info: /default/android\_settings/databases/'settings.db/system'\[0\]: Updating promised row 'update system set value='15000' where name='screen\_off\_timeout'' to database table 'settings.db/system'** Picture 2 below shows the resultant system state after running the policy: ![](images/Android_sqlite_blog_post_img2.jpg) Run it again and CFEngine reports that the system is convergent with regards to the setting you have defined, ie. the promise is kept. **\# cf-agent -I ** **2014-07-02T13:41:43+0200 info: /default/android\_settings/databases/'settings.db/system'\[0\]: Table 'system' is as promised** This example is fairly simple, but it can be extended to be even more flexible and easier. As settings in Android are usually key-value pairs, it is desirable to express the configuration as that too. In CFEngine this is possible through the use of associative arrays. Consider the following bundle: **bundle agent android\_vars {** **vars:** **"settings\[screen\_off\_timeout\]" string =\> "15000";** **}** Here we can define as many system settings as we'd like and hide away the complexity of how the settings are applied in a separate bundle as shown below: **bundle agent android\_settings {** **vars:** **"settings" slist =\> maparray("name='$(this.k)',value=\>'$(this.v)'", "android\_vars.settings");** **databases:** **"settings.db/system"** **database\_server =\> sqlite("/data/data/com.android.providers.settings/databases"),** **database\_rows =\> { @(settings) };** **}** These examples clearly show how CFEngine enables you to describe system policy on Android in a transparent and convenient manner, and at the same time have the entire end-to-end software stack policy enforced by a true distributed configuration management system - truly WebScale. -------------------------------------------------------------------------------- title: "CFEngine Los Angeles Meetup Next Week" url: /blog/2014/cfengine-los-angeles-meetup-next-week/index.md date: "2014-07-01" -------------------------------------------------------------------------------- Are you a CFEngine user in sunny Los Angeles interested in learning more about the new CFEngine Enterprise 3.6.0? If so, you should definitely look into attending a joint meetup being held by the LA CFEngine User Group and the League of Professional System Administrators (LOPSA) next **Tuesday, July 8th, at 7pm** in the Marina Del Ray area. CFEngine System Engineer Nick Anderson will be at the event to speak about some of the new and enhanced features of CFEngine Enterprise 3.6.0, such as enhanced inventory and reporting capabilities, improved templating, and new language abstractions. He will also be available to answer any CFEngine questions you might have. **You can learn more and register** [here](https://www.meetup.com/League-of-Professional-System-Administrators-Los-Angeles/events/192385292/). We hope to see you there next week\! -------------------------------------------------------------------------------- title: "Now Available: Forrester Research Webinar Recording" url: /blog/2014/now-available-forrester-research-webinar-recording/index.md date: "2014-07-01" -------------------------------------------------------------------------------- We had a fantastic webinar last month with CFEngine CTO Mark Burgess and guest speaker Forrester Research, Inc. Vice President and Research Director Glenn O'Donnell. During this wide-ranging discussion, Mark and Glenn discussed the concept of continuous operations, and how great service design combined with automation can empower organizations to survive and thrive in what is becoming known as "The Age of the Customer." Mark also talked about Promise Theory and gave a short look at CFEngine Enterprise 3.6.0. Be sure to check out the recording [here](https://www.youtube.com/watch?v=HMs5KQ2yRK0)\! -------------------------------------------------------------------------------- title: "Register for Intro to CFEngine Enterprise 3.6.0 Webinars Next Week" url: /blog/2014/reminder-intro-webinars/index.md date: "2014-06-27" -------------------------------------------------------------------------------- There is still time to register for one of our upcoming "Introduction to CFEngine Enterprise 3.6.0" webinars being held next Tuesday and Wednesday. By attending one of these live webinars, you will learn how CFEngine Enterprise 3.6.0 can help you:
- Proactively alert on policy drifts in your environment through a simple-to-use, configurable dashboard - Gain comprehensive visibility into your IT infrastructure with enhanced compliance and extensible inventory reporting - Leverage new language abstractions to improve administration productivity - Achieve painless integration with other IT systems using native JSON support
Please visit our registration pages to learn more and sign up today: [cfengine.com/webinars](https://cfengine.com/webinars) We hope you can join us next week\! -------------------------------------------------------------------------------- title: "Join CFEngine at the Global Online Event 'WebScale Wednesday' This Week" url: /blog/2014/join-cfengine-global-online-event-webscale-wednesday-week/index.md date: "2014-06-23" -------------------------------------------------------------------------------- Are you interested in learning more about WebScale IT and its increasing relevance to the enterprise datacenter? If so, you should definitely check out the global online event 'WebScale Wednesday' being hosted this week by Nutanix. Our CTO and Co-founder Mark Burgess will be participating in this event alongside pioneers and practitioners from companies such as Dell, Twitter, and Citrix for an engaging and interactive discussion around WebScale IT and how it is making its way into the mainstream datacenter. The event streams live from 9am to noon this Wednesday, June 25th, and there are various sessions throughout the morning you can choose to attend. You can learn more about the agenda and speakers at the registration page [here](https://go.nutanix.com/web-scale-wednesday.html). Sign up before the event and get a free t-shirt - we hope to see you there\! -------------------------------------------------------------------------------- title: "CFEngine will be at DevOpsDays Silicon Valley" url: /blog/2014/cfengine-will-devopsdays-silicon-valley/index.md date: "2014-06-20" -------------------------------------------------------------------------------- Continuing an extremely hectic month at CFEngine we are wrapping up the month by sponsoring DevOpsDays Silicon Valley which is being held at the Computer History Museum. With an impressive array of sponsors and many attendees this will hopefully be an event to remember. So what will CFEngine be doing at DevOpsDays? Well, after lots of hard work CFEngine 3.6.0 has finally been released so we'll be showcasing that and explaining how we can help you in your DevOps journey. Mark Burgess will be presenting at the event. It's my first time seeing Mark present at DevOpsDays as a CFEngine employee. I saw an awesome session by Mark at DevOpsDays London last year which certainly influenced my decision to join CFEngine. I hope Mark's session leaves a lasting impression on those of you attending too\! The real magic of DevOpsDays is the open space sessions, although for me the ignite sessions are also special because the content is so varied, although this time I have decided not to submit an ignite session, I need a break\! I'm sure the open spaces will be enlightening as usual and I'm looking forward to Mark proposing an open space on the future of configuration management. There were many unanswered questions from his presentation of the SVDevOps meetup as well as some lively discussion making it perfect for an open space\! Look forward to seeing you next week\! -------------------------------------------------------------------------------- title: "Announcing General Availability of CFEngine Enterprise 3.6.0" url: /blog/2014/announcing-general-availability-cfengine-enterprise-3-6-0/index.md date: "2014-06-18" -------------------------------------------------------------------------------- ### Big news today - I am happy to share the news that CFEngine Enterprise 3.6.0 has been released\! This release is a huge step forward for our company and our customers, with a number of both new and enhanced features designed for providing a secure, scalable, and reliable foundation for IT automation. Customers and users are crucial when it comes to feedback and continuous improvement. The fact that CFEngine 3.6.0 leapfrogs the competition across many facets is testament to an innovative development team working in tandem with customers experiencing and solving real world IT issues together. With 3.6.0 we have addressed some key needs you put forth: - Automation with additional and fine grained control - Out of the box alerts and reporting capability - Scale and speed improvements to address cloud use cases Let me share with you what we have done to address the above requirements: ##### We created a powerful dashboard with customizable widgets so you know at a glance the state of your environment CFEngine Enterprise 3.6.0 presents you with a powerful visual dashboard, where you can set up customizable widgets to define scenarios unique to your environment. You can use this intuitive dashboard to set policy, create alerts to system drifts and deviations from desired state, diagnose issues, and take action accordingly. ##### We let you build your custom alerts and inventory reports so you keep in compliance New reporting capabilities enable you to create filtered compliance and inventory reports on any system attributes across environments, production tiers, operating systems, and hosting platforms - whether physical, virtual, or in the cloud. These reporting capabilities enable you to perform more frequent audits and ensure compliance of your infrastructure and applications. ##### We reduced the amount of code you have to write Improved language abstractions and an improved file templating engine based on the popular Mustache template system has resulted in up to a 90% reduction in coding. ##### We added native JSON support so you can integrate more easily with 3rd party apps CFEngine Enterprise 3.6.0 includes native support for JSON, which has become a standard format for applications and services to communicate with each other. ##### We have enhanced the platform further to truly provide IT Automation at WebScale CFEngine has always benefited from a highly distributed, lean-footprint, autonomous architecture. While this aligns itself very well to scale and agility (speed), we have further improved the platform to increase and improve these attributes of our platform manifold. A new PostgreSQL backend supports thousands of managed systems per hub. We have vastly improved the data collection speeds thus making reporting and inventory a breeze. Lastly this release sees a drastic improvement in the performance of both the Mission Portal UI and APIs. I welcome you to try CFEngine 3.6.0. We also welcome your support and feedback as we continue to enhance CFEngine to tackle your IT Automation needs\! Let's continue to partner\! Let's continue to engage\! Interested in Learning More? Here are several resources to help you get started: Visit our [3.6.0 Product Page](/product/enterprise-3-6-0/ "Enterprise 3.6.0") to experience the product, view demos, and learn more about new features and benefits. Register for one of our upcoming "Welcome to CFEngine 3.6.0 webinars: - [July 1st](https://cfengine.com/webinars) (Americas Time Zone) - [July 2nd](https://cfengine.com/webinars) (Europe Time Zone) Contact us at with questions. -------------------------------------------------------------------------------- title: "Reminder - Webinar with Forrester Research, Inc. This Thursday" url: /blog/2014/reminder-register-upcoming-webinar-forrester-research-june-19th/index.md date: "2014-06-16" -------------------------------------------------------------------------------- There is still time to register for our [upcoming webinar](https://cfengine.com/company/blog-detail/upcoming-webinar-with-forrester-research/) with **Forrester Research** this Thursday\! Please join CFEngine Co-Founder and CTO Mark Burgess and guest speaker Forrester Research, Inc. Vice President and Research Director Glenn O'Donnell for a fascinating look into the principles of continuous operations and service management - and how great service design combined with automation can empower your organization to survive and thrive in what is becoming known as "The Age of the Customer."
**WHEN: Thursday June 19th at 10am PT / 1pm ET / 6pm GMT**
[**CLICK HERE TO REGISTER TODAY**](https://cfengine.com/webinars "Webinar Registration") **Speakers:** **Glenn O'Donnell, VP and Research Director Serving Infrastructure & Operations at Forrester Research** Glenn serves Infrastructure & Operations Professionals. He is widely regarded as a top thought leader in IT service management, IT operations, and the broader social implications of technology evolution. Glenn's specialties are in data center automation and operational excellence. He is the co-author of The CMDB Imperative, a book on CMDB best practices.**** **Mark Burgess, Founder and CTO at CFEngine** Mark Burgess is the creator and main technical architect of the popular CFEngine open source IT automation software. He was the first person internationally to hold the title of ‘Professor of Network and System Administration', at Oslo University College. He now leads the research and development of the CFEngine software full time. -------------------------------------------------------------------------------- title: "DockerCon: An event full of excitement, curiosity and cries for help!" url: /blog/2014/dockercon-event-full-excitement-curiosity-well-confusion/index.md date: "2014-06-13" -------------------------------------------------------------------------------- [![dockercontable](images/dockercontable-300x224.png)](images/dockercontable.png)Earlier this week CFEngine sponsored DockerCon. Let me start by congratulating Docker on an awesome show. The energy, and enthusiasm was contagious but more so sincere. The commitment to establishing open standards without locking in users to Docker is commendable. The CFEngine table had heavy traffic and the conversations were frequently in depth. Attendees were not only wanting to know what value CFEngine adds on top of Docker but also what guidance on how others are using Docker. People we talked to understood the need to have a solution that could manage not only Docker containers but also their other physical and virtual infrastructure. We had very positive feedback on how we integrated Docker configurations into our CFEngine policies. For those of you who may be unaware Docker finally reached 1.0 status, which means the warning about running in production no longer applies, not that being pre 1.0 stopped the Docker community running Docker in production\! Some key takeaways for me were - Docker is attracting attention from most industries. Its usage isn't limited to startups and technology companies - The Docker company commitment to building open standards is very impressive and I hope CFEngine and others leverage these libraries where appropriate - Managing Docker containers at scale without additional tooling will be challenging - Containers, physical and virtual machines will co-exist and complement each other in the future, Docker isn't a drop in replacement for all use cases If you want to find out more about how CFEngine and Docker can be used together you can get our CFEngine and Docker presentation from [slideshare](https://www.slideshare.net/cfengine/the-benefits-of-using-cfengine-with-docker). -------------------------------------------------------------------------------- title: "Visit CFEngine at DockerCon14" url: /blog/2014/visit-cfengine-dockercon14/index.md date: "2014-06-02" -------------------------------------------------------------------------------- CFEngine are sponsoring DockerCon14. We're excited to be able to show you how CFEngine and Docker can be used together to help manage your docker containers and make sure that you don't go back down the path of golden images\! If that isn't enough to get you excited our very own Diego Zamboni is flying in from Mexico for the event. Stop by and have a chat with Diego about CFEngine, security or anything else. Diego will also be signing copies of Learning CFEngine 3. We look forward to seeing you there. -------------------------------------------------------------------------------- title: "No idol worshipping at DevOpsDays Pittsburgh" url: /blog/2014/idol-worshipping-devopsdays-pittsburgh/index.md date: "2014-06-02" -------------------------------------------------------------------------------- DevOpsDays events in Pittsburgh and Austin have both been awesome. It's great to see the events maturing over time. While it was the first DevOpsDays event in Pittsburgh the organizers were familiar with DevOpsDays events and made sure that they built on top of other successes. The lack of "Idol worshipping" aka "we love Amazon, Etsy, Facebook, Netflix" was refreshing\! We were treated to excellent sessions covering culture, PaaS, docker and other great topics. As long as you are in tech I'm certain there was plenty to interest you. The ignite sessions again were fun as always. There were some fun moments and some awkward moments\! I presented my ignite session from DevOpsDays Austin on why DevOpsDays is failing horses and unicorns. Eventually I'll track down a video of it and link to it, but it was well received. The awkward moment was that I presented after Nathan Harvey of Chef. He preached the familiar unicorn cry of "if you aren't happy quit your job". My ignite session has a slide that made fun of unicorns telling horses to quit their jobs and go work with unicorns because it really isn't helpful at all. There was a great open space session on diversity in tech which could have gone on for hours. It was constructive and we all realize there is a problem. While DevOpsDays PGH wasn't incredibly diverse it was heartening to see a significant number of women attending and presenting. In particular I'm looking forward to seeing Bridget Kromhout (@bridgetkromhout) and Cornelia Davis (@cdavisafc) present again. Awareness of CFEngine is growing, and we are getting recognition in presentations once again. People really seem pleased that we are still around and doing some awesome things. I can't wait until DevOpsDays Silicon Valley when we can talk about some really cool stuff we have coming\! June will be a very busy month for CFEngine. We'll be at DockerCon, Velocity Santa Clara and DevOpsDays Silicon Valley. -------------------------------------------------------------------------------- title: "What is WebScale and how does CFEngine help you achieve it?" url: /blog/2014/webscale-cfengine-help-achieve/index.md date: "2014-06-02" -------------------------------------------------------------------------------- This is a term often used today to acknowledge the extraordinary growth of the major web companies over a decade (social media, retailing, games, cloud etc) from handfuls of machines to the largest installations on the planet. The major web players today have datacenters with 10,000, 100,000 and even 1,000,000 computers serving their operations. Of course, this kind of growth is not appropriate for everyone. WebScale often goes together with quite singular or focused applications, by contrast with very complex industries that have to support thousands of applications for different lines of business. There is also a link to ideas of cloud computing. WebScale operations do not necessarily involve virtualisation, but typically there is a correlation between ideas of cloud computing and web scale. Some of the issues at web scale include: - Elastic scaling to handle bursty loads and seasonal trends. - Service oriented architectures. - Heavy loads inside data centers (so called East-West traffic) - Content delivery networks and caches for scaling - Continuous delivery of updates - High rate of change - Software defined infrastructure Opinions differ on how to manage all of these, but there are common themes. The solutions adopted by the major showcases might not apply to all businesses, but the innovations made by these companies can be applied by almost anyone to good effect. These are some of the specific challenges associated with web scale: from programming and architecting solutions to carrying the traffic over wires in the datacenter. The problems are partly about building software stacks, partly about maintaining them, and partly about robustness under pressure. How does CFEngine help get you there? CFEngine was conceived long before the web mushroomed, but it is still a great tool to help scale computing in all scenarios. This is because of the core principles it adheres to. CFEngine comprises a set of decentralised, agent-based controllers, that may be coordinated and measured through shared policy and data. It is fast, lightweight and resilient under pressure. Some of its properties include: Decentralized means scaling is unlimited. Hands-free changes are fully automated as repeatable desired end-states - High availability - no need to reprovision servers with minor drifts or errors - Versioning and knowledge management allows to match intent with outcome - Model-oriented means you can use patterns to reduce complexity - Speed means dynamic adaptation and rapid provisioning - Efficiency means little wastage in valuable resources - Pro-active repair means avoiding bad states - A language reflecting a desired end-state model keeps you on task, not distracted by programming your way out of trouble. - In-built monitoring saves setting up multiple models that are uncorrelated for verifying outcomes. - Machine-learning allows you to see trends and variability of workloads - Policy federation into human centric organisational centres is a key to socialising the processes like DevOps, auditing, validation, compliance CFEngine has been instrumental in helping to scale several of well known web giants (Facebook, Amazon, LinkedIn). It clearly has the tools to configure the whole software stack - but it also lets you choose just as much as you want. It doesn´t force you into only one approach. CFEngine´s history and Promise Theory-friendly design brings great flexibility. On June 19th Glenn O'Donnell and I will be discussing Continuous Operations for a competitive advantage in a joint webinar. You can find out more by visiting [https://cfengine.com/company/blog-detail/upcoming-webinar-with-forrester-research/](https://cfengine.com/company/blog-detail/upcoming-webinar-with-forrester-research/ "https://cfengine.com/company/blog-detail/upcoming-webinar-with-forrester-research/"). We hope to see you there. -------------------------------------------------------------------------------- title: "Is DevOps killing the ?" url: /blog/2014/devops-killing/index.md date: "2014-05-29" -------------------------------------------------------------------------------- We've already had fear-mongering about DevOps potentially killing off operations positions (thanks NoOps) and now it's the turn of articles about DevOps killing off the developer. Terms like full stack developer are thrown around and other un-helpful terms that recruiters seem to love. Let me make one thing perfectly clear. I don't believe DevOps is about training people so that people can do each other's jobs equally well. That doesn't make sense. There always need to be specialists. The cultural changes encouraged by the DevOps movement should result in people in Dev and Ops having an appreciation of what the other groups do and the kind of decisions they have to make and why. All too often I have witnessed developers putting technology X into a product because it is new and hot and they want to use it, no matter what the consequences to QA, operations and so on. Dev and Ops should work together. Sure, that might mean that someone from operations spends some time in code working with developers getting an understanding of what's going on and why, especially when it affects infrastructure design. The same goes for developers, spending some time doing some basic administration and maintenance of systems that traditionally fall outside of the developer role is a good thing. That said, having spent most of my time on the development side of the house and working very closely with operations I believe that developers should be involved in the consequences of their decisions. In fact the best developers I have worked with have cared about this and were extremely happy to work with operations teams if given the chance. Of course there has to be balance. If as a developer you are spending less time developing and more time playing with databases, infrastructure or whatever else you blame DevOps for I would guess there might be a couple of things going on: 1. The development team put something together that is hard to maintain once it is out of the hands of development and you are now being held accountable, or 2. You have a management problem. Either way I would say it's not something DevOps should take the blame for. Perhaps this sounds like I am bashing developers\! What I have said applies to Ops too. I've witnessed Ops being so resistant to change that developers feel like they are working with one or even both arms tied behind their backs due to constraints imposed on them. So no, DevOps isn't killing off the developer, it's not killing off operations or anyone else. It should be helping you collaborate and work effectively which will ultimately let you spend more time doing what you do best. -------------------------------------------------------------------------------- title: "CFEngine will be at DevOpsDays Pittsburgh" url: /blog/2014/cfengine-will-devopsdays-pittsburgh/index.md date: "2014-05-27" -------------------------------------------------------------------------------- I'm really looking forward to DevOpsDays Pittsburgh this week. The first DevOpsDays event in a city is always special. I'm not sure what to expect, but I think it's going to be great and at the very least I'll get to have some great conversations about DevOps, configuration management and play some werewolf\! [![DevOpsDaysPGH](images/DevOpsDaysPGH-150x150.png)](images/DevOpsDaysPGH.png) I'm due to present an ignite session too, I have one lined up already but there's a good chance I might come up with something new based on conversations and presentations on the first day of the event. I'll be at the event with Adi Aloni, so stop by and talk to us about CFEngine and some of the exciting things we are working on. -------------------------------------------------------------------------------- title: "Versioning masterfiles (configuration policy) in CFEngine Enterprise 3.6.0" url: /blog/2014/versioning-masterfiles-configuration-policy-cfengine-enterprise-3-6-0/index.md date: "2014-05-20" -------------------------------------------------------------------------------- ## Introduction CFEngine users version their policies. It's a reasonable, easy thing to do: you just put `/var/cfengine/masterfiles` under version control and... you're done? What do you think? How do you version your own infrastructure? ## Problem statement It turns out everyone likes convenience and writing the versioning machinery is hard. So for CFEngine Enterprise 3.6.0 we set out to provide version control integration with Git out of the box, disabled by default. This allows users to use branches for separate hubs (which enables a policy release pipeline) and enables Design Center integration. ### Release pipeline A build and release pipeline is how software is typically delivered to production through testing stages. In the case of CFEngine, policies are the software. Users have at least two stages, development and production, but typically the sequence has more stages including various forms of testing/QA and pre-production. ### Design Center The CFEngine Design Center is a way to augment your policies (in a way that does not conflict or override your own policies) through a GUI, using modular testable policies called sketches. It's like a Perl CPAN for CFEngine but with a GUI and awesome sauce mixed in. ## How to enable it To enable masterfiles versioning, you have to plan a little bit. These are the steps: ### Choose your repository You have two options: use the default local Git repository which comes with CFEngine Enterprise, or use a remote Git repository accessible via the `git` or `https` protocol. The first option is good for getting started quickly, but we strongly recommend the second option: using a remote repository, populated with the contents of the `3.6.x` branch of our masterfiles repository at https://github.com/cfengine/masterfiles. #### Using the default local Git repository The default repository is a local directory on the hub and set up by the `cfengine-hub` package. It's the default in the Mission Portal VCS integration panel and resides in `/opt/cfengine/masterfiles.git`. **PLEASE NOTE: you must use user "cfapache" to interact with this repository safely on the hub.** You do not have to do anything to set up this repository - it's already preconfigured and prepopulated out of the box. You just need to enable VCS deployments as described below. To check out this default repository, run the following commands on your hub (everything needs to be run as user `cfapache` for the permissions to be set correctly. The first two commands setup some basic information needed by git to manipulate the repository): git config user.email "your@email.address" git config user.name "Your Name" su - cfapache git clone /opt/cfengine/masterfiles.git And then make all the changes in the checked-out `masterfiles` repository. #### Using a remote repository To use a remote repository, you must enter its address, login credentials and the branch you want to use in the Mission Portal VCS integration panel. To access it, click on "Settings" in the top-left menu of the Mission Portal screen, and then select "Version control repository". This screen by default contains the settings for using the built-in local repository. [![](images/settings-menu.png)](/blog/2014/images/settings-menu.png) [![settings-vcs](images/settings-vcs-1024x341.png)](images/settings-vcs.png) ### Make sure your current masterfiles are in the chosen repository This is critical. When you start auto-deploying policy, you **will** overwrite your current `/var/cfengine/masterfiles`. So take the current contents thereof and make sure they are in the Git repository you chose in the previous step. For example, if you create a new repository in GitHub by following the instructions from https://help.github.com/articles/create-a-repo, you can add the contents of `masterfiles` to it with the following commands (assuming you are already in your local repository checkout): cp -r /var/cfengine/masterfiles/* . git add * git commit -m 'Initial masterfiles check in' git push ### Enable VCS deployments in the versioned `update.cf` In the file `update.cf` in your versioned masterfiles, change #"cfengine_internal_masterfiles_update" expression => "enterprise.!(cfengine_3_4|cfengine_3_5)"; "cfengine_internal_masterfiles_update" expression => "!any"; to "cfengine_internal_masterfiles_update" expression => "enterprise.!(cfengine_3_4|cfengine_3_5)"; #"cfengine_internal_masterfiles_update" expression => "!any"; This is simply commenting out one line and uncommenting another. Remember that you need to commit and push these changes to the repository you chose in the previous step, so that they are picked up when you deploy from the git repository. In your checked out `masterfiles` git repository, these commands should normally do the trick: git add update.cf git commit -m 'Enabled auto-policy updates' git push Now you need to do the first-time deployment, whereupon this new `update.cf` and the rest of your versioned masterfiles will overwrite `/var/cfengine/masterfiles`. We made that easy too, using standard CFEngine tools. Exit the `cfapache` account and run the following command as `root` on your hub: cf-agent -Dcfengine_internal_masterfiles_update -f update.cf Easy, right? You're done, from now on every time `update.cf` is run (by default, every 5 minutes) it will check out the repository and branch you configured in the Mission Portal VCS integration panel. Please note all the work is done as user `cfapache` except the very last step of writing into `/var/cfengine/masterfiles`. ## How it works The code is fairly simple and can even be modified if you have special requirements (e.g. Subversion integration). But out of the box there are three important components. All the scripts below are stored under `/var/cfengine/httpd/htdocs/api/dc-scripts/` in your CFEngine Enterprise hub. ### `common.sh` The script `common.sh` is loaded by the deployment script and does two things. First, it redirects all output to `/var/cfengine/outputs/dc-scripts.log`. So if you have problems, check there first. Second, the script sources `/opt/cfengine/dc-scripts/params.sh` where the essential parameters like repository address and branch live. That file is written out by the Mission Portal VCS integration panel, so it's the connection between the Mission Portal GUI and the underlying scripts. ### `masterfiles-stage.sh` This script is called to deploy the masterfiles from VCS to `/var/cfengine/masterfiles`. It's fairly complicated and does not depend on CFEngine itself by design; for instance it uses `rsync` to deploy the policies. You may want to review and even modify it, for example choosing to reject deployments that are too different from the current version (which could indicate a catastrophic failure or misconfiguration). This script also validates the policies using `cf-promises -T`. That command looks in a directory and ensures that `promises.cf` in the directory is valid. If it's not, an error will go in the log file and the script exits. **NOTE** this means that clients will **never** get invalid policies according to the hub, although a 3.5 or older client could still receive policies that are only valid in 3.6. So make sure you test with 3.5 or older if you anticipate that problem during migration, but in a homogeneous client population this is a wonderful guarantee. ### `pre-fetch.sh` and `post-update.sh` These scripts are run by the Mission Portal whenever the user configures sketches. They enable the Mission Portal to check out the policies, make changes to them, and then commit and push them back. ## Design Center integration The Design Center integration Just Works when you follow the procedure above to enable the VCS integration. You can then go into the Mission Portal, configure any sketch, and voila, in minutes that sketch will be activated across your infrastructure. ## Manual policy changes If you want to make manual changes to your policies, simply make those changes in a checkout of your masterfiles repository, commit and push the changes. The next time `update.cf` runs, your changes will be checked out and in minutes distributed through your entire infrastructure. ## Benefits To conclude, let's summarize the benefits of versioning your masterfiles using the built-in facilities in CFEngine Enterprise 3.6.0 - easy to use compared to home-grown VCS integration - supports Git out of the box and, with some work, can support others like Subversion - tested, reliable, and built-in - Design Center integration - supports any repository and branch per hub - your policies are validated before deployment - integration happens through shell scripts and `update.cf`, not C code or special policies -------------------------------------------------------------------------------- title: "Does DevOps work in the enterprise?" url: /blog/2014/devops-work-enterprise/index.md date: "2014-05-16" -------------------------------------------------------------------------------- I had a thought-provoking [article by Rachel Shannon-Solomon forwarded](https://www.wsj.com/articles/BL-CIOB-4503) to me. Essentially the article states that while DevOps is great for startups the enterprise isn't ready for it yet. I've read/heard people disagreeing with this but I think the article raises some great points. At DevOpsDays Austin I presented a session on why it isn't realistic to expect enterprises to become just like startups when adopting DevOps. There's a lot of technological and cultural complexity present in legacy enterprises that just aren't an issue in startups. DevOps practitioners from startups don't seem to have an answer either, quit your job and move to a startup might be a solution for an individual but it certainly doesn't help with DevOps adoption in the enterprise. Setting aside the cultural difficulties of doing DevOps in the enterprise there's the not so small issue that a lot of the tools available are suitable for startups but unproven and dare I say often unsuitable for the enterprise. Some vendors have been extremely skilled at attaching themselves to DevOps and trying to morph DevOps into something that helps them sell products. Unfortunately this doesn't change the fact that software isn't going to solve your DevOps adoption problem, in fact I would say some of the solutions I have seen presented can make matters worse. I've heard multiple times that being a Director of DevOps or having a similar title isn't ideal because it implies that the person is responsible for the success or failure of DevOps in an organization. I agree with that in any type of organization. The problems DevOps is trying to address span multiple organizational units and it takes much more than one person with a fancy job title and a mandate to "do DevOps" to make changes in the enterprise. Those of us who believe that the DevOps movement can make a difference in the enterprise have a responsibility to recognize challenges faced. Let's stop pointing out how awesome DevOps works in startups and get together to bring DevOps to the enterprise. Perhaps it is the DevOps movement that isn't quite ready for the enterprise. The good news is that there are people who passionately believe in helping bring DevOps to the enterprise. I believe their work will help transform the DevOps movement into something more enterprise friendly while still keeping the spirit of DevOps. -------------------------------------------------------------------------------- title: "Join us for a webinar with Forrester Research, Inc. on Continuous Operations for Competitive Advantage" url: /blog/2014/upcoming-webinar-with-forrester-research/index.md date: "2014-05-15" -------------------------------------------------------------------------------- Companies of all sizes are under pressure to succeed in competitive environments requiring them to move faster with expectations of always-on service from their IT organization. However, these same IT groups are faced with significant challenges from manual processes, homegrown patchwork solutions, disparate systems, and other legacy problems. In order to overcome these issues, companies need to automate their infrastructure in an intelligent way to keep up with the pace of the business while at the same time providing a secure and stable environment to deploy complex multi-tier applications to local or cloud infrastructure. By implementing a solution that enables **continuous operations**, companies can gain complete visibility and alert capabilities that can automate compliance checks, audits, and remediation using a policy-based approach. Join us for a discussion on **Thursday June 19th at 10am PT / 1pm ET / 6pm GMT **featuring guest speaker, **Forrester Research, Inc. Vice President and Research Director Glenn O'Donnell** **** and **CFEngine Co-Founder and CTO Mark Burgess** as they talk about the principles of continuous operations and how complete visibility combined with phased deployment rollouts can minimize risk and maximize uptime. Most importantly, you'll also learn about how you can get started on implementing continuous operations within your organization. We hope to see you there\! [**CLICK HERE TO REGISTER TODAY**](https://cfengine.com/webinars "Webinar Registration") **Speakers:** **Glenn O'Donnell, VP and Research Director Serving Infrastructure & Operations at Forrester Research** Glenn serves Infrastructure & Operations Professionals. He is widely regarded as a top thought leader in IT service management, IT operations, and the broader social implications of technology evolution. Glenn's specialties are in data center automation and operational excellence. He is the co-author of The CMDB Imperative, a book on CMDB best practices.**** **Mark Burgess, Founder and CTO at CFEngine** Mark Burgess is the creator and main technical architect of the popular CFEngine open source IT automation software. He was the first person internationally to hold the title of 'Professor of Network and System Administration', at Oslo University College. He now leads the research and development of the CFEngine software full time. -------------------------------------------------------------------------------- title: "DevOpsDays, horses and unicorns!" url: /blog/2014/austin-devopsdays-horses-unicorns/index.md date: "2014-05-14" -------------------------------------------------------------------------------- Once again DevOpsDays Austin was a fantastic event. This was the DevOpsDays event that I have been to that had two tracks. One track for what I would call traditional DevOpsDays sessions that focused on culture and organizational change, the other with more practical advice on how to get things done. I took the opportunity to get something off my chest that has been bothering me a bit about DevOpsDays and other DevOps events in general. I presented an ignite session titled "Why DevOpsDays (and other DevOps events) is failing both horses and unicorns". Essentially I proposed that focusing on Amazon, Etsy, Facebook and Netflix so much wasn't really helping the horses (aka the enterprise). Essentially there was a proposal and a reminder in the ignite: 1. Let's stop using the regular DevOps idols as examples and replace them with stories from other companies. If there aren't any other success stories then are we really making a difference at DevOpsDays events? 2. For the unicorns presenting always think about how what you are presenting applies to people in more traditional companies. Don't just say "if you can't do awesome in your current company quit your job and come work with us". That's not particularly helpful I didn't really expect a positive reaction to the session but was pleased that it resonated with people. If the evolution of DevOpsDays Austin over the past few years is anything to go by I'm really looking forward to seeing how other DevOpsDays events will evolve over the course of the year. Next up is DevOpsDays Pittsburgh. CFEngine is sponsoring and I'll be there to spread the word about CFEngine\! Hope to see you in Pittsburgh\! -------------------------------------------------------------------------------- title: "The Internet of Things - Where are we? CFEngine at the IoT Expo in San Francisco" url: /blog/2014/internet-things-cfengine-iot-expo-san-francisco/index.md date: "2014-05-09" -------------------------------------------------------------------------------- Yesterday, I had an amazing opportunity to participate in the inaugural Internet of Things (IoT) Expo held at Fisherman's Wharf in San Francisco. The event was conceived by Jeremy Geelan who is also behind the highly successful Cloud Expo events. [![IoTExpo](images/IoTExpo.png)](https://web.archive.org/web/20140919015641/http://iotexpo2014.com/) The IoT Expo was well attended with many excellent keynotes and product demonstrations. Among others, industry leaders such as Tony Shakib (VP, IoT vertical at Cisco), Omri Lachman (Co-Founder and CEO Humavox), and Joe Speed (Director of IoT at the Linux Foundation) gave us very compelling insights into what IoT can achieve, its potential, as well as issues that still need to be solved for. Later in the day, I participated in a panel discussion with Peter Moskovits (Kaazing) and David Nielsen (Cloud Computing Evangelist and Consultant) around the following questions: #### 1\. What is it about the present that allows us to consider the IoT as a practical, and real thing? Does anyone remember Microsoft SPOT? This was the first real connected smart watch for the masses and it was available 10 years ago. While one can argue that there were a lot of things missing, MS did get some things right: https://gigaom.com/2013/09/03/what-microsoft-got-right-with-its-smartwatch-nearly-a-decade-ago-more-than-you-think/ The SPOT may have been too far ahead of its time to be successful. If you look at current trends, the cost of fabrication and components has gone down tremendously in the last decade along with a sharp decrease in form factor. Today, you can actually fit a camera or a computer into something that measures 1 cu. mm. Or, how about IPv6? With nearly 100 Internet addresses now available for each atom on the planet, we can connect all the devices we want and still have addresses to spare\! Lastly, communications such as Wi-Fi, Broadband, Zigbee, and RFID have all improved by leaps and bounds. #### 2\. So what are some of the things that can hold IoT back? Two things come to mind. First, connected devices need to be able to speak amongst themselves and not just to their command and control. In other words, we cannot have an IoT for Verizon, one for MS, one for Cisco etc. We need to ensure devices speak a common language or we will learn from the Tower of Babel\! The second area that can be worrisome is security. If Heartbleed scared us, you can imagine the panic and mayhem that could be caused if these devices are broken into and altered to take different (than intended) action. From a CFEngine perspective our highly-distributed architecture enables us to play a very important role when it comes to IoT. It is predicted that there will be over 50 billion connected devices by 2025\! Which means that IoT is all about scale - and CFEngine is built to handle scale. Across our deployments we routinely manage tens of thousands of machines. We also have a spotless track record when it comes to security vulnerabilities. Zero to be exact. Lastly, the most important reason CFEngine will be the preferred automation platform for connected devices is our lightweight and autonomous end-point agents that can be embedded into the smallest of environments. This autonomy is at the heart of CFEngine and gives our agents independent decision-making capabilities instead of the typical master-slave, top-down approach to IT management. This will be increasingly important as connected devices grow in number and become more pervasive. We closed the panel talking about some unique use cases across industries such as medical devices where CFEngine is playing a key role ensuring connected devices perform as needed. CFEngine is at the forefront of IoT and we look forward to more participation in these discussions going forward. If you have a unique IoT story or are currently leveraging CFEngine in a unique way, we would love to hear from you\! -------------------------------------------------------------------------------- title: "CFEngine will be at DevOpsDays Austin" url: /blog/2014/3660/index.md date: "2014-04-30" -------------------------------------------------------------------------------- In a few days I'll be flying to Austin to attend DevOpsDays Austin. It was an awesome event last year and CFEngine is proud to be sponsoring the event. I'm looking forward to participating in open space sessions and hearing about the latest challenges, tips and tricks people have to share. It will also be my first chance this year to talk to people in person to see if DevOps is making advances in the traditional enterprise (aka horses) instead of just hearing about Netflix, Amazon, Etsy, Facebook (aka unicorns). See you in Austin\! -------------------------------------------------------------------------------- title: "The Future of Configuration Management - A talk by Mark Burgess" url: /blog/2014/future-configuration-management-talk-mark-burgess/index.md date: "2014-04-30" -------------------------------------------------------------------------------- We are pleased to announce that CFEngine Founder Mark Burgess will be speaking at the next Silicon Valley DevOps meeting about the future of configuration management. At the meetup you will not only learn about the future of Configuration Management as Mark sees it but also have the chance to network with some of the best and brightest in Silicon Valley who are passionate about DevOps and Configuration Management. [![InSearchOf](images/InSearchOf-208x300.jpg)](images/InSearchOf.jpg) We will also be raffling two signed copies of In Search of Certainty by Mark Burgess and two passes to Promise2014, the first CFEngine user conference. **Where:** Silicon Valley Innovation Center, 3200 Coronaodo Way, Santa Clara, CA **When:** May 15th at 7:00pm Learn more and reserve your spot [here](https://www.meetup.com/SVDevOps/events/176569272/). -------------------------------------------------------------------------------- title: "Inventory management in 3.6, part 1 - Showing variables and classes" url: /blog/2014/inventory-managemnet-in-3-6-part-1-showing-variables-and-classes/index.md date: "2014-04-14" -------------------------------------------------------------------------------- ## Inventory management in 3.6, part 1 - Showing variables and classes CFEngine 3.6 introduces a set of features for inventory management, and we'll have a closer look at one of them today. This feature is part of both the Community and Enterprise editions. It essentially outputs the inventory in terms of classes and variables at a local node. Have a quick look at cf-promises -h of 3.6: # cf-promises -h Usage: cf-promises \[OPTION\]... \[FILE\] Options: --eval-functions, - value - Evaluate functions during syntax checking (may catch more run-time errors). Possible values: 'yes', 'no'. Default is 'yes' --show-classes, - - Show discovered classes, including those defined in common bundles in policy --show-vars , - - Show discovered variables, including those defined anywhere in policy --help , -h - Print the help message ... You might see the two new options --show-classes and --show-vars. Let's test them out. # cf-promises --show-classes Class name Meta tags 10\_0\_2\_15 inventory,attribute\_name=none,source=agent,hardclass 127\_0\_0\_1 inventory,attribute\_name=none,source=agent,hardclass 192\_168\_33\_3 inventory,attribute\_name=none,source=agent,hardclass 1\_cpu source=agent,derived-from=sys.cpus,hardclass 64\_bit source=agent,hardclass April time\_based,source=agent,hardclass .... As you probably would expect, this will output all your global classes, together with some meta data about them. A snippet from the corresponding variables option looks like the following. # cf-promises --show-vars Variable name Variable value Meta tags ... default:sys.arch x86\_64 inventory,source=agent,attribute\_name=Architecture default:sys.bindir /var/cfengine/bin source=agent default:sys.cdate Tue\_Apr\_\_8\_23\_49\_37\_2014 time\_based,source=agent default:sys.cf\_agent "/var/cfengine/bin/cf-agent" cfe\_internal,source=agent default:sys.cf\_execd "/var/cfengine/bin/cf-execd" cfe\_internal,source=agent default:sys.cf\_key "/var/cfengine/bin/cf-key" cfe\_internal,source=agent ... The output should be fairly self-explanatory, but note that the variable name here contains the name space before the colon. The main goal of this feature is to make it easier for you to see which inventory you can work with as you develop your policy. In our next blog post, we will look more in depth at the meta tags, as well as enhancements to inventory reporting in CFEngine Enterprise. -------------------------------------------------------------------------------- title: "Fixing Heartbleed with CFEngine 3.5.x and above" url: /blog/2014/fixing-heartbleed-with-cfengine-3-5-x-and-above/index.md date: "2014-04-11" -------------------------------------------------------------------------------- A couple of days ago we informed you of the [status of the CFEngine products and services with respect to Heartbleed](https://cfengine.com/blog/cfengine-and-heartbleed). Today we would like to share with you some instructions and policies that you can use to check your systems for vulnerable versions of OpenSSL, and if needed upgrade it to its latest version. If you already have CFEngine deployed, adding this policy and deploying it to your systems takes only a few minutes, after which CFEngine will take care of performing the necessary checks and updates on your systems, whether you have a few or tens of thousands. If you are not using CFEngine yet, you can still use these policies to check it out, and run the fix by hand on your systems. These policies work for Ubuntu and RedHat-based systems (including CentOS). See below for some hints about how to customize it for other operating systems. 1. Make sure you have CFEngine [installed and running](https://cfengine.com/docs/3.5/getting-started-installation.html) on the systems where you want to run the fix. This policy has been tested with CFEngine 3.5.3, our latest stable release. 2. [Download](images/heartbleed_fix.tar) and unpack the files. You should get two files: fix\_heartbleed.cf and fix\_heartbleed\_standalone.cf. The first file contains the code that performs the checks and fixes, and the second one is a standalone policy file that loads and executes the first, and which you can run from the command line if you don't want to integrate the checks into your regular policy run. 3. **Review fix\_heartbleed.cf**. The file works out of the box on Ubuntu 10.04/12.04/13.10 and RedHat/CentOS 6.4, but you should review it anyway. In particular, you need to update the list of services you want to restart after updating OpenSSL. By default only Apache is restarted if it was running already. 4. If you want to run the policy by hand, either for testing before you deploy it, or for running only once on a system, run the following command: cf-agent -f ./fix_heartbleed_standalone.cf -Dverbose The -Dverbose option makes the policy print some messages to let you know what is happening. For example, on an Ubuntu 12.04 installation in which Apache was running, you should see something like this: ``` notice: R: Found OpenSSL 1.0.1-4ubuntu5.5: VULNERABLE TO HEARTBLEED notice: R: Updated package 'openssl' to latest version notice: R: Updated package 'libssl1.0.0' to latest version ``` Please note that this policy will only update OpenSSL if it detects a vulnerable version. If you have an older version of OpenSSL (e.g. 0.9.8k which comes with Ubuntu 10.04), it will be detected as not vulnerable and no update will be done. 5. If you want to deploy the policy to your entire CFEngine infrastructure, you need to integrate it into your promises.cf file. On your CFEngine policy server, follow these steps: 1. Copy fix\_heartbleed.cf to /var/cfengine/masterfiles/ (you do not need the standalone file in this case). 2. In your promises.cf file, add the following lines to the bundlesequence and inputs attributes in body common control: ``` bundlesequence => { ... "fix_heartbleed", }; inputs => { ... "fix_heartbleed.cf", }; ``` 3. Validate the syntax of your promises.cf file using the cf-promises command: cf-promises -f /var/cfengine/masterfiles/promises.cf Fix any errors shown. 4. The updated files will be distributed to your existing clients. After a few minutes, you will see the policy files being distributed and any vulnerable clients will start to be fixed. **Please note:** this policy only helps with upgrading OpenSSL to a non-vulnerable version. To fully recover from the potential effects of the bug, you may need to take additional measures depending on your setup and systems, such as regenerating SSL certificates and keys, and changing (or asking your users to change) relevant passwords. For more information please visit . If you want to customize this policy for other operating systems, you can follow the same patterns already in the policy. Look for the OS-specific sections by searching for ubuntu and redhat in the policy code, and adding similar constructs with the corresponding information for the new operating systems. We hope you find these instructions useful, both as a guide to fixing Heartbleed and as an example of how you can use the power of CFEngine to maintain your infrastructure. Once you perform the steps above, your entire infrastructure will be updated\! Whether you have 5 or 50,000 machines under CFEngine control, all of them will be updated accordingly. We will continue to monitor any developments around Heartbleed or any other security issues and continue to provide you with appropriate information, instructions or guidance. In the meantime, if you have any questions or feedback, please email us at .
[heartbleed\_fix.tar (16.5 KiB)](images/heartbleed_fix.tar)
-------------------------------------------------------------------------------- title: "CFEngine Sponsoring the Red Hat Summit Next Week" url: /blog/2014/cfengine-sponsoring-the-red-hat-summit-next-week/index.md date: "2014-04-11" -------------------------------------------------------------------------------- CFEngine is a proud sponsor of the Red Hat Summit next week in downtown San Francisco\! If you are attending the event, be sure to visit us at **Booth \#116** to say hello and learn more about new features of CFEngine 3.6 as well as get special discounts for our upcoming user conference, [Promise2014](https://www.promise14.com/). You can also join us for some drinks and snacks on Tuesday night at a Meetup we are hosting at Jillian's next door to the conference. Be sure to visit our [Meetup page](https://www.meetup.com/SF-Cfengine-Users-Group/events/173784432/) for more information and to sign up. Looking forward to seeing you next week\! -------------------------------------------------------------------------------- title: "Heartbleed Security Update for CFEngine Users and Customers" url: /blog/2014/heartbleed-security-update-for-cfengine-users-and-customers/index.md date: "2014-04-10" -------------------------------------------------------------------------------- As you may know, a serious vulnerability was recently announced in OpenSSL, commonly referred to as [Heartbleed](https://heartbleed.com/) or more officially by its CVE ID [CVE-2014-0160](https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-0160). This vulnerability affects the OpenSSL heartbeat mechanism and allows unauthorized access to private data including encryption keys, encrypted traffic and more. At CFEngine we use OpenSSL both in our infrastructure and in our products. The security of our users and customers is one of our primary concerns, so we immediately began investigating the possible impact of this bug. Here are our findings: - **CFEngine Enterprise and CFEngine Community binary packages are not affected.** Versions 3.5.3 and older do not use the OpenSSL encryption protocols (OpenSSL is used internally for encryption, which is unaffected by the bug), and version 3.6 (currently in beta) ships with a non-vulnerable version of OpenSSL for the implementation of its new TLS protocol. - CFEngine Community 3.6beta1 or 3.6beta2 **compiled from source may be affected, as it uses the version of OpenSSL installed on your system.** Make sure you check and update OpenSSL if needed, and recompile CFEngine as necessary. - Our Redmine bugtracker ([https://cfengine.com/dev](https://www.google.com/url?q=https%3A%2F%2Fcfengine.com%2Fdev&sa=D&sntz=1&usg=AFQjCNEIRcdkd1wW1r6A5tYbLOH0rk87rA)) and our (now deprecated) OTRS bugtracker for Enterprise customers are **not affected**. - Our main website ([https://cfengine.com/](https://www.google.com/url?q=https%3A%2F%2Fcfengine.com%2F&sa=D&sntz=1&usg=AFQjCNEu-oHDPeERq0xiVyMB8BdPFrKV_w)) **was affected**. The web server has been updated to a non-vulnerable version of OpenSSL. If you have a [cfengine.com](https://cfengine.com/) account, we highly recommend you change your password at [https://cfengine.com/inside/editAccount](https://www.google.com/url?q=https%3A%2F%2Fcfengine.com%2Finside%2FeditAccount&sa=D&sntz=1&usg=AFQjCNFvUwcf7uUsVx0j5I9-4zY58cqjzA). If you have an account in our old community bugtracker ([https://cfengine.com/bugtracker/](https://www.google.com/url?q=https%3A%2F%2Fcfengine.com%2Fbugtracker%2F&sa=D&sntz=1&usg=AFQjCNGCARnl2B69cJkp7sXvV6E2eIBL6Q)) you may also want to change your password there (particularly if you reused your password in any other service). - If you are an Enterprise user, we also strongly advise you to change the password of your [Zendesk account](https://www.google.com/url?q=https%3A%2F%2Fcfengine.zendesk.com%2F&sa=D&sntz=1&usg=AFQjCNErXT_fj5iA2NiknhJuaL00SiMyqA) used for support tickets. We will continue to monitor this situation and communicate any developments through our blog, or through direct email to customers when appropriate. We are committed to keeping CFEngine as the most secure configuration management solution while keeping an honest and open communication with you. If you have any questions or comments, please contact us at , or through your regular support channels if you are an Enterprise customer. Thank you, Diego Zamboni Sr. Security Advisor, CFEngine -------------------------------------------------------------------------------- title: "Introducing the Host Info Report for CFEngine Community Edition" url: /blog/2014/introducing-the-host-info-report-for-cfengine-community-edition/index.md date: "2014-04-02" -------------------------------------------------------------------------------- The Host Info Report has been available in CFEngine Enterprise for some time, it will now be available to community users in CFEngine 3.6. The Host Info Report is a great way to get value immediately if you are using CFEngine for the first time. So what does it do? The Host Info Report contains detailed information on the host that CFEngine is running on. Read on to learn more about how to enable and use the report. The host\_info\_report bundle has been added to the [masterfiles policy framework](https://github.com/cfengine/masterfiles) that ships with CFEngine 3.6. This bundle leverages the new mustache template functionality (and serves as a useful example of both mustache templates and the datastate() function) to render a simple plain text summary of host info. The bundle is not activated by default, it can of course be activated automatically by adding host\_info\_report to the bundlesequence or by a methods type promise. It can be activated on demand by calling the bundle manually using the --bundlesequence or -b options to cf-agent. Example: console \[root@hub \~\]\# cf-agent -b host\_info\_report R: Host info report generated and available at '/var/cfengine/reports/host\_info\_report.txt' /var/cfengine/reports/host\_info\_report.txt: # Host Information Generated: Mon Mar 31 15:11:46 2014 ## Identity Fully Qualified Hostname: hub Host ID: SHA=0cc801bb2b8ed7feb48f698415867314430552c49a24763f65d58e12d446dbac ## CFEngine Version: CFEngine Enterprise 3.6.0b2.630288b Last Agent Run: 2014-03-31 15:11:36 UTC Policy Release ID: d34e3cfe0c1a94ab6ce9bbfbe9b2ad0c47edf6a3 Policy Last Updated: 2014-03-28 21:06:34 UTC ## OS Architecture: x86\_64 Os: linux Release: 2.6.32-358.el6.x86\_64 Flavor: centos\_6 Version: \#1 SMP Fri Feb 22 00:31:26 UTC 2013 Uptime: 4186 minutes ## Hardware No. CPUs: 1 ## Network ### Interfaces - eth0: IPv4 10.0.2.15 - eth0: up broadcast running multicast - eth1: IPv4 192.168.33.2 - eth1: up broadcast running multicast - lo: IPv4 127.0.0.1 - lo: up loopback running ### IPv4 TCP Ports listening - 443 - 80 - 27017 - 5308 - 25 - 5432 - 631 - 22 - 6379 This report gives you a high level overview of the individual host including CFEngine, host identity, OS, hardware, and network configuration information. Additional information may be given depending on various classes that are defined. For example currently installed software will be shown if the class show\_software is defined. To manually run the report and show the software currently installed simply define show\_software using the --define or -D option to cf-agent. Example: console \[root@hub \~\]\# cf-agent -b host\_info\_report --define show\_software R: Host info report generated and available at '/var/cfengine/reports/host\_info\_report.txt' /var/cfengine/reports/host\_info\_report.txt (snipped for brevity): … ### IPv4 TCP Ports listening - 443 - 80 - 27017 - 5308 - 25 - 5432 - 631 - 22 - 6379 ## Software \[rpm\] plymouth-core-libs: version 0.8.3-27.el6.centos, arch x86\_64 \[rpm\] atmel-firmware: version 1.3-7.el6, arch noarch \[rpm\] libgfortran: version 4.4.7-3.el6, arch x86\_64 \[rpm\] ivtv-firmware: version 20080701-20.2, arch noarch \[rpm\] gpgme: version 1.1.8-3.el6, arch x86\_64 \[rpm\] centos-indexhtml: version 6-1.el6.centos, arch noarch \[rpm\] man: version 1.6f-32.el6, arch x86\_64 \[rpm\] diffstat: version 1.51-2.el6, arch x86\_64 \[rpm\] xmlrpc-c-client: version 1.16.24-1209.1840.el6, arch x86\_64 \[rpm\] eject: version 2.1.5-17.el6, arch x86\_64 … If you're interested in looking at the policy that implements the report you can use [cf-locate](https://raw.githubusercontent.com/cfengine/core/master/contrib/cf-locate/cf-locate) from contrib to quickly find and even display a bundle from your policy. ``` [root@hub ~]# cf-locate --full host_info_report$ /var/cfengine/masterfiles -> body or bundle matching 'host_info_report$' found in /var/cfengine/masterfiles/reports/host_info_report.cf:1 bundle agent host_info_report # @brief Generates a short high level summary of the executing host information # # **Example:** # `cf-agent -b host_info_report` { vars: "host_info_report_template" handle => "host_info_report_vars_host_info_report_template", string => "$(this.promise_dirname)/../templates/$(this.bundle).mustache", comment => "Where the report template is found"; "host_info_report_output" handle => "host_info_report_vars_host_info_report_output", string => "$(sys.workdir)/reports/$(this.bundle).txt", comment => "Where the host info report should be generated"; methods: show_software:: "Software" handle => "host_info_report_methods_software", usebundle => host_info_report_software, comment => "Collect information about software installed"; any:: "CFEngine" handle => "host_info_report_methods_cfengine", usebundle => host_info_report_cfengine, comment => "Collect information about CFEngine"; "Render" usebundle => host_info_report_render_txt, comment => "Render the host info report in plain text"; } ``` We hope you find this addition to the masterfiles policy framework useful and look forward to your pull requests to add additional information and improve its usefulness. -------------------------------------------------------------------------------- title: "Introducing The CFEngine 3.6 Highlights Series" url: /blog/2014/introducing-the-cfengine-3-6-highlights-series/index.md date: "2014-04-02" -------------------------------------------------------------------------------- Over the next few months we'll be blogging about some of the great features, enhancements and bug fixes coming in CFEngine 3.6. All too often new software is released without anyone taking the time to tell you what's been done and why it matters. To start the series we have a blog post by Nick Anderson on the new [Host Info Report in the community edition](https://cfengine.com/blog/introducing-the-host-info-report-for-cfengine-community-edition). Check back each week to learn more about CFEngine 3.6. -------------------------------------------------------------------------------- title: "The Dawn of Meltdowns" url: /blog/2014/the-dawn-of-meltdowns/index.md date: "2014-02-26" -------------------------------------------------------------------------------- We are quickly becoming a more software defined world. The music, movie, automotive and telecoms are examples of industries going through disruptiveness due to innovative software defined products. In this new world, IT-operational efficiency has risen to be more critical to businesses than ever. Thanks to software and open source, the world is a more transparent place. Innovation is as much about speed as it is about coming up with that great next new idea. In efficient markets, ideas are quickly copied. The best way to stay ahead of competition is to continuously bring new products and services to market. The value of speed at which a company introduces new products and improves new versions based on market feedback has become paramount to success. The world accelerates in its need for new products and services. The connected- and smart-phone based economies emerge as the leaders of this new world-order, where new versions of products are introduced to the market on weekly, and even daily basis. IT-operations play a very important role to support and enable businesses on their journey towards the pursuit of continuously offering new enhanced products and services. Not only is IT-operations an enabler of faster change-cycles, but probably one of the most critical components in ensuring that these new software defined products continuously meet customer expectations in terms of response-times, up-time and quality of service. The market expects products to work as promised, and any deviation will have an immediate negative business impact. In worse case it can lead to brand- and revenue damage. "The importance of IT-operational effectiveness and efficiency grows with the share of revenue deriving from software defined products". The following scenario illustrates the growing value of IT-operations. At CFEngine we work with several of the world's largest financial organizations, and large web-scale companies. These two groups of companies operate very differently due to reasons like culture, complexity and regulations. In a large financial organization a change often requires the sign-off of multiple persons, and the frequency of changes is weekly (at best). Web-scale companies on the other hand have adopted an agile change-process built on trust, automation and audit-trails. These companies make customer impacting changes several times a day. We can safely assume the speed at which web-scale companies operate is **10x faster** than the financial organizations. Assume that Amazon, which is a web-scale company, decides to introduce a digital currency, which they allow their customers to use to buy products on Amazon.com. Think of Amazon gradually becoming a bank where they compete against other banks. At both Amazon and let's say Bank A, they come up with this great software based idea at the same time. Amazon is, as we said, able to deploy customer-facing changes 10x faster than the bank. The business impact of IT-operational and IT-development efficiency will be devastating to Bank A in their competition as Amazon will be able to: - offer the new product way ahead of Bank A - offer several improved versions of the product based on market feedback before Bank A even comes out with their first version - get more satisfied customers as they respond faster to customer feedback - get valuable feedback and data from the market which can be used to come up with new related products - learn more and thereby become more knowledgeable about their customers' needs The severe business impact of IT inefficiency has led to a change in the view of IT. It is no longer only the innovators that realize the dramatic impact IT has on business. As more companies wake up to the new world, not even the most sacred cow remains untouchable. We are starting to see the meltdowns of organizational silos, business processes, and rigid cultures. Within application delivery, DevOps has become the new best-of-breed standard, tearing down the walls between developers and operations people, and thereby allowing for a more efficient flow of code from development to production. I call this the vertical meltdown, where every level in the software stack is up for a new dawn. Every process and person involved, from the specifications of a new product or service to its customer facing delivery and maintenance will be reviewed. The most agile IT-companies, in their hunt for time-wastes and process improvements, have found hardware components to be bottlenecks. This explains why network automation has gotten much attention lately. Soon we should expect storage to enter the scene. This represents the horizontal meltdown. Due to the significant business impact of IT inefficiency, everything from departmental walls and well-established processes are up for radical changes. We have seen this on the application delivery side with DevOps, and the same thing will happen with networking and storage. Processes leading towards trust, automation and audit-trails will become the new standard. At CFEngine our most automated and thereby agile customers often mention that our software's ability to support organizational changes and new processes have been of substantial value. This might sound strange, but as the meltdowns start to happen, make sure your automation tool supports these new organizational structures and processes. At the end of the day, there are only two things an IT-organization should be measured on: (1) Speed to introduce new/enhanced products and services and (2) life-time quality of service. The biggest obstacle is not technology, but processes and people. Become a change-agent today and melt those walls down\! -------------------------------------------------------------------------------- title: "Engaging with the CFEngine community at Scale 12x" url: /blog/2014/engaging-with-the-cfengine-community-at-scale-12x/index.md date: "2014-02-25" -------------------------------------------------------------------------------- My name is Jonathan Thorpe and I recently joined CFEngine to work with the CFEngine community. I've spent the past 15 years or so working in several different roles ranging from development, build maintenance, management and finally onto community and customer facing roles. When I got the opportunity to work with the CFEngine community it was a very easy decision to join CFEngine. While I've only been on the job for a week I've had an amazing experience so far. Scale was my first show with CFEngine and I came back extremely energized and enthusiastic about the CFEngine community in SoCal. ![SCALE Booth](images/SCALE-12x.png) By the end of Scale I'd heard about many interesting use cases, been asked about doing meetups and other community events and came back armed with lots of suggestions for how to enhance CFEngine and make it more accessible for users. We're already working on meetups in the LA area and surrounding counties and I'll be reaching out to community members around the globe to bring meetups and other CFEngine related events to you. We'll be announcing upcoming events on cfengine.com, twitter. Please follow @CFEngine and @jonathan\_thorpe to get the latest updates on what we are doing in the CFEngine community. If you have community events or other suggestions please send them to or tweet your ideas to us. -------------------------------------------------------------------------------- title: "Digging Up Bundles and Bodies" url: /blog/2014/digging-up-bundles-and-bodies/index.md date: "2014-01-31" -------------------------------------------------------------------------------- Originally posted by Nick on [cmdln.org](https://www.cmdln.org/2013/10/29/digging-up-bundles-and-bodies/) You never know when the Zombie or Cloud Apocalypse is coming. It's good to be able to locate those buried bodies quickly and easily. OK, enough bad jokes, but haven't you ever looked at some CFEngine policy and wondered to yourself, exactly what does "delete =\> tidy" or some other body or bundle do? I have. I even wrote a little [perl script](https://groups.google.com/d/msg/help-cfengine/i_ZNFA_SaAs/T62p2lY__HgJ) to locate the files that contained a specific body or bundle and then print out the single body or bundle. This past week my old script [got some love](https://github.com/cfengine/core/commits/master/contrib/cf-locate). Ted Zlatanov and Bishwa Shrestha reworked it a bit so that it is no longer a hackjob, and it's now included in [contrib](https://github.com/cfengine/core/tree/master/contrib/cf-locate) of the CFEngine core repository. I know I have found the script useful, and it's nice to see that other people think it's useful enough to put in contrib. I want to know if you think it's useful enough to be installed in your $PATH (like the CFEngine binaries) when a package is installed. I started a thread on the [help-cfengine](https://groups.google.com/forum/?hl=en#%21topic/help-cfengine/89VENZTA5fA) mailing list/google group/forum. If you would find it useful speak up\! If you hate it speak up, or better yet submit a pull request to add the functionality you want :-) -------------------------------------------------------------------------------- title: "CFEngine 3.6 Beta 1 released to Community" url: /blog/2014/cfengine-3-6-beta-1-released-to-community/index.md date: "2014-01-31" -------------------------------------------------------------------------------- We're proud to release the first Beta package of CFEngine 3.6 to the Community for testing. The new version of CFEngine introduces a huge number of new features to the CFEngine language and a lot of improvements behind the scene. Some highlights from the [ChangeLog file](https://github.com/cfengine/core/blob/master/ChangeLog): - New promise type "users" for managing local user accounts. - TLS authentication and fully encrypted network protocol - New attributes in 'bundle server access\_rules' - New variable type 'data' for handling of structured data - Tagging of classes and variables with meta data - Many new built-in variables - Many new functions You can download the beta packages for Debian and RedHat based Linux distributions from [https://cfengine.com/inside/myspace](https://cfengine.com/inside/myspace). The documentation for CFEngine 3.6 lives at and is work in progress. We'd like to invite everybody to help us improve CFEngine 3.6 by giving this first beta a trial run. However, this version of CFEngine is not yet ready for production environments, so do not upgrade your existing installation to these packages. Due to the network protocol and authentication changes, compatibility between 3.6 and previous CFEngine versions is limited at this point, and will be improved over the coming weeks as we get closer to the release. To report bugs, use our bug tracker at . Contributions in the form of pull requests can be made on GitHub at . A big shout-out to all the contributors in the community that have supported our work with feedback, bug reports, input to design discussions and code contributions\! -------------------------------------------------------------------------------- title: "Announcing the 2013 CFEngine Champions" url: /blog/2014/announcing-the-2013-cfengine-champions/index.md date: "2014-01-23" -------------------------------------------------------------------------------- Another year has passed and we would like to take a moment to thank the community members for their contributions. From speaking at conferences, epic blog posts, community meet-ups, to support on the CFEngine help list and in the IRC channel, new and veteran users alike don't have to look far to find inspiration or a helping hand. Each year we like to reflect on our many community contributors and honor those that have significantly enhanced the CFEngine Community. This year it is my great honor to announce this year's CFEngine Champion Hall of Fame inductees. [Bas van der Vlies](https://www.linkedin.com/in/basvandervlies/) Bas is a long time CFEngine community member and has contributed excellent work on the CFEngine parser. He has also been a tremendous help within the Debian team to ensure that the Debian repository gets up to date CFEngine builds. Bas is also found helping on the mailing list. [Brian Bennett](https://www.linkedin.com/in/brianbennettsd/) Brian has been a long time user of CFEngine and can easily be found on the CFEngine mailing list. He is also very active within the Debian team. This year Brian gave us his [Case Study in CFEngine Layout](https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/) and his [mini zero-to-hero](https://github.com/bahamat/cf-primer) introduction that one of our previous champions Aleksey Tsalolikhin delivered as a training session at LOPSA East. [Loïc Pefferkorn](https://www.linkedin.com/in/loicpefferkorn/) Loic has been an invaluable community member. In the last year he has given us [cfe-profiler](https://github.com/lpefferkorn/cfe-profiler) and [blog posts](https://web.archive.org/web/20140609012807/http://www.loicp.eu/blog) that explore the scalability and limitations in CFEngine. In addition to his tooling and writing he can also be found lurking in \#cfengine on [libera.chat](https://libera.chat) and the cfengine-help mailing list helping other community members. [Ted Zlatanov](https://github.com/tzz) Ted is frequently found supporting community users on the help mailing list, and championing fixes for bugs and new features on the bug tracker. In the past year he has contributed many new functions including reverse, shuffle, sort, strftime, unique, upcase and downcase just to name a few. Emacs lovers can thank Ted for his continued maintenance of the cfengine3-mode. Ted has been contributing to CFEngine from within as a CFEngine employee since 2012. Thank you Bas, Brian, Loïc, and Ted. You have significantly enhanced the CFEngine community through your individual and collective efforts. Special thanks to our previous champions who continue to be nominated and who continue to greatly support the CFEngine community. And finally to the whole community, thank you. We value each of your contributions. Our community is one of our greatest assets and CFEngine would not be what it is without you. To learn more about our champions, go to the [Champions Hall of Fame](https://cfengine.com/cfengine-champions-hall-of-fame). -------------------------------------------------------------------------------- title: "My First CFEngine Meetup" url: /blog/2013/my-first-cfengine-meetup/index.md date: "2013-12-12" -------------------------------------------------------------------------------- Having joined CFEngine only a few months ago (and being new to the Open Source movement and culture), attending our Bay Area meetup allowed me to come face to face with our community and hear about CFEngine deployment from the people who use it as their main tool. And let me tell you, there are some serious CFEngine warriors out there\! In this blog post I will feature two of them. By pure chance both of them are called Bruce, but don't worry, it's not a prerequisite to be called that to join our community\! Bruce Carlton, who organized the meetup, is a systems administrator at TRUSTe. He started using CFEngine in 2008 and quickly became an enthusiast. He likes to come up with analogies that will help describe CFEngine to non-technical audiences and explain its declarative nature. One analogy that he is still perfecting is to traffic signs that are declarative and assume that the driver is able to decipher them and act accordingly, versus traffic lights that are imperative and impose a command on the driver. To build on the same road traffic analogy, the declarative nature of CFEngine allows it to find its own route towards the desired destination, as opposed to an imperative tool that will get nowhere if one of the roads in its predetermined route is blocked. The other Bruce, Bruce Rafnel, is now with Equifax, where he was introduced to CFEngine eight years ago and expanded its use dramatically, including an upgrade to CFEngine 3 when the company moved its infrastructure to the cloud. Bruce made an excellent case for the scalability of CFEngine - because CF Agents are pulling policies from the hub, as opposed to the policy server pushing policies onto agents, the entire system is utilizing resources more efficiently, if only because a server will not try to make changes where there are none to be made. We all felt that the evening was very successful and that a meetup is a great way to share our CFEngine experiences and stories, as well as learn from each other's deployment of CFEngine and its use cases. Personally, I am looking forward to the next meetup and calling on all of you to follow our website announcements and tweets for information on events to come. -------------------------------------------------------------------------------- title: "Function Caching in CFEngine 3.6" url: /blog/2013/function-caching-in-cfengine-3-6/index.md date: "2013-12-11" -------------------------------------------------------------------------------- Many users have been asking for ways to limit the amount of some function invocations in CFEngine, in particular functions such as execresult and returnszero. First, let me try to explain why functions were called so many times to begin with, and how we have approached this for version 3.6. Functions may be executed during checking with cf-promises, or during normal evaluation. When cf-agent executes a policy, it first runs it through checking with cf-promises. Many policy checks are static in nature (types, keywords, etc.), but since CFEngine is a fairly dynamic language, certain problems may only arise at run-time. Therefore, cf-promises has traditionally attempted to find these problems by executing all functions. For 3.6, we have turned off function evaluation for cf-promises, but retained the old behavior optionally using the flag --eval-functions. After checking a policy, the agent proceeds with evaluation. Here, naturally all functions called from active promises should be called at least once. But why should each call be evaluated more than once for each agent run? CFEngine will only decide to not execute a promise iteration once the evaluation context for it has been set up. Suppose we have the following promise, where 'a' is a list variable. bundle agent test { reports: "Hello $(a)" ifvarclass =\> canonify("$(a)"); } To set up the evaluation context for a promise iteration, $(a) is dereferenced both in the promiser and the function call. Once this is done, the 'report' promise handler decided whether to actuate the given context, which it does by hashing the evaluation context and match it against the promise lock database. The last part prevents promises from being executed too often. In general it is not possible to know whether a promise should not be executed before its whole evaluation context is set up, including calling associated functions. Nevertheless, we decided for version 3.6 to group function calls into two categories: those that can have their results cached, and those that have to execute each time. We label the former category 'system' function, and presently they are execresult, returnszero as well as various network related functions. All calls to system function will be cached by default, meaning they will be executed at most once per run. Other function calls may be evaluated many times. This should hopefully bring performance benefits in the normal case. However, we have added an option to keep the old behavior, simply set 'cache\_system\_functions =\> false' in body common control. -------------------------------------------------------------------------------- title: "CFEngine 3.5.3 released" url: /blog/2013/cfengine-3-5-3-released/index.md date: "2013-12-10" -------------------------------------------------------------------------------- CFEngine 3.5.3 is now available for [download](https://cfengine.com/downloads). This is a maintenance release of CFEngine 3.5, and introduces a number of fixes and improvements to both Community and Enterprise editions. ### Changes in the CFEngine Core: - Improved security checks of symlink ownership. A symlink created by a user pointing to resources owned by a different user will no longer be followed. - Changed the way package versions are compared in package promises. (Redmine \#3314) In previous versions the comparison was inconsistent. This has been fixed, but may also lead to behavior changes in certain cases. In CFEngine 3.5.3, the comparison works as follows: \ \ \ For instance: apache-2.2.31 "\>=" "2.2.0" will result in the package being installed. Bug fixes: - fix cf-monitord crash due to incorrect array initialization (Redmine \#3180) - fix cf-serverd stat()'ing the file tree every second (Redmine \#3479) - correctly populate sys.hardware\_addresses variable (Redmine \#2936) - add support for Debian's GNU/kfreebsd to build system (Redmine \#3500) - fix possible stack corruption in guest\_environments promises (Redmine \#3552) - work-around hostname truncation in HP-UX's uname (Redmine \#3517) - fix body copy purging of empty directories (Redmine \#3429) - make discovery and loading of avahi libraries more robust - compile and packaging fixes for HP-UX, AIX and Solaris - fix fatal error in lsdir() when directory doesn't exist (Redmine \#3273) - fix epoch calculation for stime inrange calculation (Redmine \#2921) ### Enterprise-specific changes: Bug fixes: - purge old data for promises with long promise handles (Redmine \#3438) - fix constraint violation in PromiseDefinitions table which resulted in error every time this table was loaded - enable update of promise definitions database from policy - fix cfengine3 init.d script to correctly detect debian systems with yum installed Mission Portal: - various layout and UI fixes - fix editing of event trackers - speed up listing of hosts for promises not kept - maintain host context (Redmine \#3474) - ability to manually add context filter in the SQL app (Redmine \#3466) - host identifier settings simplified (Redmine \#3101) Packaging fixes: - Correct php.ini path in the packaged httpd (Redmine \#3445) - Add missing mongodb tools in ubuntu/debian hub packages (Redmine \#3444) - Fix manpath error for SLES (Redmine \#3539) - Fix file permissions - some policy files had executable bit set (Redmine \#3521) The MD5 checksums for the various community packages are: - cfengine-3.5.3.tar.gz - c840eb0163924ca657ab180fe5a170b4 - cfengine-community\_3.5.3-1\_amd64.deb - 2295f51e2d9c5ef1d500a0bcda5423ff - cfengine-community\_3.5.3-1\_i386.deb - d428a655dc35b62f88ce3e256fdda02c - cfengine-community-3.5.3-1.x86\_64.rpm - e2bcd53204b4e549a6992e7b7c88344c - cfengine-community-3.5.3-1.i386.rpm - 355301ef5e379383ee51550856245da6 [Community repositories](https://cfengine.com/cfengine-linux-distros) will be updated shortly. -------------------------------------------------------------------------------- title: "CFEngine 3.6 to increase data loading flexibility - Introducing the new data type 'data'" url: /blog/2013/cfengine-3-6-to-increase-data-loading-flexibility-introducing-the-new-data-type-data/index.md date: "2013-11-27" -------------------------------------------------------------------------------- sigurd.teigen@cfengine.com R\&D, CFEngine CFEngine has previously had multiple ways of loading data from an external file into a policy. This has been useful for basing policy on tabular data. For example, importing a CSV file to create accounts. Once data has been imported, it may be used using the existing CFEngine scalar and list data types. In order to make this more flexible, we are introducing a new data type 'data' for version 3.6 of CFEngine. A data container variable is an arbitrarily nested data structure of CFEngine primitives. You may think of a data container as a JSON document, because initially we are only supporting importing from JSON. However, this may change in the future allowing for importing data from YAML, XML and other files. Let us examine the flexibility and ease of use of this data type by way of an example. bundle agent test { vars: "doc" data =\> parsejson('{ "a": \[ 1, 2 \], "b": true }'); reports: "Hello $(doc\[a\]) $(doc\[b\])"; } This will output the following. $ cf-agent/cf-agent test.cf 2013-11-25T15:18:32-0500 notice: /main: R: Hello 1 true 2013-11-25T15:18:32-0500 notice: /main: R: Hello 2 true In this example, we use the function parsejson to assign a new data container. A data container may either be iterated as a list or as a scalar, depending on how it's indexed. Above, we see that we index into a list 'a' in 'doc'. Meanwhile, 'b' in 'doc' refers to a scalar. A list may contain other container types (list or maps), but only sibling scalars will be iterated over. Indexing works as in JavaScript, for example $(doc\[a\]\[1\]) would refer to the second element in the list 'a' in 'doc'. Currently, there is no inline syntax for data containers, although this is being planned for. A function readjson is available to import a file, while function mergedata may be used to join existing data container variables into a new variable. We hope that data containers will be a practical way of working with structured data in policy, replacing current uses of arrays. Data containers are also used with the new Mustache template feature also in version 3.6. Learn more about CFEngine by [visiting our website](https://www.cfengine.com) [Evaluate CFEngine for free](https://cfengine.com/evaluate-enterprise) (up to 25 licenses, unlimited usage) -------------------------------------------------------------------------------- title: "CFEngine and Docker - Ensure application availability and container integrity" url: /blog/2013/cfengine-and-docker-ensure-application-availability-and-container-integrity/index.md date: "2013-11-22" -------------------------------------------------------------------------------- eystein.maloy.stenberg@cfengine.com Product Management, CFEngine **Introduction** The beauty and power of Docker comes from its ability to containerize any application into a portable self-sufficient entity, one that can run anywhere. This enables developers to deliver an application with all dependencies in a layered image structure. As if that was not enough, the images can be shared in an image repository (known as the [Docker index](https://index.docker.io/)) and deployed to and run on any Docker host. CFEngine, a pioneer in IT automation, enables organizations to become more agile by radically simplifying, automating and transforming the way they build, deliver and consume IT infrastructure and applications. With CFEngine, some of the largest IT organizations provision resources and deploy new applications orders of magnitude faster, while ensuring continuous availability, security and compliance in large-scale, very dynamic and highly complex environments. After applications are deployed, maintenance tasks such as process management and signaling, log management, and managing run-time configuration consistency and compliance become important. CFEngine is designed to be very lightweight, distributed and fault-tolerant. These attributes make it very well suited for managing containerized Docker applications. CFEngine is an extremely versatile technology and a good complement to Docker, which is very strong at getting code containerized, shipped and deployed. **Advancing Process Management in Docker** Docker monitors one process in each running container and the container lives or dies with that process. By introducing CFEngine inside Docker containers, we can alleviate a few of the issues that may arise: - It is possible to easily start multiple processes within a container, all of which will be managed automatically. - If a managed process dies or crashes, CFEngine will start it again within 1 minute. - The container itself will live as long as the CFEngine scheduling daemon (*cf-execd*) lives. With CFEngine, we are able to decouple the life of the container from the uptime of the service it provides. **How it works** CFEngine, together with the *cfe-docker* integration policies, are installed as part of the Dockerfile. This essentially builds CFEngine into our Docker image. The advantage is that the Dockerfile's ‘ENTRYPOINT' takes an arbitrary amount of commands (with any desired arguments) as parameters. When we run the Docker container, the parameters get written to CFEngine policies and CFEngine takes over to ensure that the desired processes are running in the container. CFEngine scans the process table for the basename of the commands given to the ENTRYPOINT and runs the command to start the process if the basename is not found. For example, if one starts the container with docker run "/path/to/my/application parameters" CFEngine will look for a process named "application" and run the command. Later, if an entry for ‘application' is not found in the process table, CFEngine will execute "/path/to/my/application parameters" to re-start the application once again. The check on the process table happens every minute. **Test it out\!** To demonstrate the benefits of using CFEngine to manage processes in Docker containers, we will create a simple example that: - Starts both apache2 and sshd in a single container - Ensures that both these processes are continuously running To start off, we construct a Dockerfile that includes CFEngine, the *cfe-docker* policies, as well as our apache2 and sshd packages (also available at ):
FROM ubuntu MAINTAINER Eystein Måløy Stenberg RUN apt-get -y install wget lsb-release unzip # install latest CFEngine RUN wget -qO- https://cfengine.com/pub/gpg.key | apt-key add - RUN echo "deb https://cfengine.com/pub/apt $(lsb_release -cs) main" > /etc/apt/sources.list.d/cfengine-community.list RUN apt-get update RUN apt-get install cfengine-community # install cfe-docker process management policy RUN wget --no-check-certificate https://github.com/estenberg/cfe-docker/archive/master.zip -P /tmp/ && unzip /tmp/master.zip -d /tmp/ RUN cp /tmp/cfe-docker-master/cfengine/bin/* /var/cfengine/bin/ RUN cp /tmp/cfe-docker-master/cfengine/inputs/* /var/cfengine/inputs/ RUN rm -rf /tmp/cfe-docker-master /tmp/master.zip # apache2 and openssh is just for testing purposes RUN apt-get -y install openssh-server apache2 RUN mkdir -p /var/run/sshd RUN echo "root:password" | chpasswd # need a password for ssh ENTRYPOINT ["/var/cfengine/bin/docker_processes_run.sh"]
Build the container:
docker build -t managed_image .
Start the container with apache2 and sshd running and managed, forwarding a port to our SSH instance:
docker run -p 127.0.0.1:222:22 -d managed_image "/usr/sbin/sshd" "/etc/init.d/apache2 start"
We can now log in to our new container and see that both apache2 and sshd are running. We have set the root password to "password" in the Dockerfile above and can use that to log in with ssh:
ssh -p222 root@127.0.0.1 # ps -ef UID PID PPID C STIME TTY TIME CMD root 1 0 0 07:48 ? 00:00:00 /bin/bash /var/cfengine/bin/docker_processes_run.sh /usr/sbin/sshd /etc/init.d/apache2 start root 18 1 0 07:48 ? 00:00:00 /var/cfengine/bin/cf-execd -F root 20 1 0 07:48 ? 00:00:00 /usr/sbin/sshd root 32 1 0 07:48 ? 00:00:00 /usr/sbin/apache2 -k start www-data 34 32 0 07:48 ? 00:00:00 /usr/sbin/apache2 -k start www-data 35 32 0 07:48 ? 00:00:00 /usr/sbin/apache2 -k start www-data 36 32 0 07:48 ? 00:00:00 /usr/sbin/apache2 -k start root 93 20 0 07:48 ? 00:00:00 sshd: root@pts/0 root 105 93 0 07:48 pts/0 00:00:00 -bash root 112 105 0 07:49 pts/0 00:00:00 ps -ef
If we stop apache2, it will be started again within a minute by CFEngine:
``` # service apache2 status Apache2 is running (pid 32). # service apache2 stop * Stopping web server apache2 /usr/sbin/apache2ctl: 87: ulimit: error setting limit (Operation not permitted) apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName ... waiting [ OK ] # service apache2 status Apache2 is NOT running. # ... wait up to 1 minute... # service apache2 status Apache2 is running (pid 173). ```
**Adapting to your applications** To make sure your applications get managed in the same manner, there are just two things you need to adjust from the above example: - In the Dockerfile used above, install your applications instead of apache2 and sshd - When you start the container with *docker run*, specify the command line arguments to your applications rather than apache2 and sshd **That's it\!** **Future extensibility** Since CFEngine is a very mature management tool, it can be leveraged to solve additional Docker use cases. For example: - Secrets such as passwords should not be built into the Docker images, but can be discovered during run-time by CFEngine - Service dependencies such as database addresses can be discovered and configured in run-time by CFEngine We hope this post is useful and you can start experimenting with your own applications, CFEngine and Docker. We would love to hear back from you. Please let us know if you have feedback or ideas for areas where you think CFEngine could make your containerized life easier\! For now **‘Dock On'** and rest assured CFEngine is right there helping you achieve your application goals\! Learn more about CFEngine by [visiting our website](https://www.cfengine.com) [Evaluate CFEngine for free](https://cfengine.com/evaluate-enterprise) (up to 25 licenses, unlimited usage) -------------------------------------------------------------------------------- title: "10,000 Servers Managed Per System Administrator. Is That Possible?" url: /blog/2013/10000-servers-managed-per-system-administrator-is-that-possible/index.md date: "2013-11-18" -------------------------------------------------------------------------------- William Gibson once said: "The future is here, but it is not evenly distributed". Within the space of IT-operations and automation we see clear evidence of this. Last week at LISA, System Engineer [Mike Svoboda from LinkedIn gave us a glimpse of insight](http://www.slideshare.net/MikeSvoboda/lisa-2013-sysopsapi-leveraging-inmemory-key-value-stores-for-large-scale-operations-with-redis-and-cfengine#%21) into the future of infrastructure operations. The lucky audience got to learn how LinkedIn automates IT-operations at one of the largest websites in the world. What LinkedIn has achieved with less than a handful of engineers the last couple of years is truly bleeding edge. With a clear mandate from management to automate all aspects of their IT-operations 4 years ago, LinkedIn, with Mike Svoboda in the front seat, embarked on a journey that currently is among the most impressive we have seen here at CFEngine. Today every infrastructure change at LinkedIn is version controlled and automated. **Amazing results** At any point in time, LinkedIn can look back in time to see who did what on any server, within seconds. Per Mike Svoboda LinkedIn has real time visibility into all of their systems. New changes are (in a phased manner) deployed into production on a daily basis. "We know the impact of a change before we push it." Hardware failure detections and replacements are fully automated, scaling of the systems is horizontal and software based (no physical load balancers or unnecessary additional tiering). LinkedIn has even taken the art of automation to the level where they can spin up and down logical data-centers anywhere in the world within minutes\! Thanks to CFEngine and the resultant automated infrastructure, LinkedIn infrastructure staff can focus their efforts on higher order initiatives. Once you have achieved the level of automation we have seen at LinkedIn, adding more servers is a piece of cake. This is why the same handful of engineers that started out on this journey manages their large scale deployments today. What is even cooler, is that there are no plans for adding more engineers as LinkedIn continues to successfully expand its business. It is all automated\! The future is not evenly distributed. **Significant business value** LinkedIn extracts significant business value from its highly automated IT-operations. With daily changes into production, LinkedIn is able to test out new products quickly and adjust to market feedback faster than its competition. If it takes an average company about 4 months to deploy, test and roll-out a new service, while LinkedIn in the same period of time has been able to adjust its new service to market feedback several times, there is no magic to figuring out who is the most competitive. With automation comes also significantly higher quality of service and higher customer satisfaction due to pleasant service experience in terms of latency and "it just works as expected". Improved quality of service also means that a larger portion of changes relate to new features, and not fixes. The sum of this can be translated into business agility. The lean IT-operations we see at LinkedIn saves the company millions of dollars in labor costs, compared to less automated operations, and it allows the company to move much faster than its competitors. **The future will hit you sooner than you think** In today's world where more and more products become software defined, the negative impact of not running a highly automated IT-operations increases. Soon, highly automated IT-operations will become a prerequisite for staying in the game. If your competitor moves faster, is leaner and provides higher quality of service then you will soon be out of business. More and more companies are realizing this, and we see a fast growing interest in programmable infrastructures across all industries. Highly automated IT-operations is not only for fast-moving west-coast companies. This next generation IT-operations spans all industries. A few weeks ago, CFEngine was inducted into the JPMorgan Chase hall of innovation (link:). Take this as a sign of what is soon going to affect businesses across all industries, including your own. If you are a system administrator, embrace the future. Your future is bright (link: ). Congratulations to LinkedIn. What you have achieved is world-class\! Learn more about CFEngine by [visiting our website](https://www.cfengine.com) [Evaluate CFEngine for free](https://cfengine.com/evaluate-enterprise) (up to 25 licenses, unlimited usage) -------------------------------------------------------------------------------- title: "MailOnline Uses CFEngine for Infrastructure Management" url: /blog/2013/mailonline-uses-cfengine-for-infrastructure-management/index.md date: "2013-11-12" -------------------------------------------------------------------------------- Mahesh Kumar, VP Marketing CFEngine mahesh.kumar@cfengine.com This month (literally tomorrow) at [Velocity 2013 London](https://web.archive.org/web/20150923045244/http://velocityconf.com/velocityeu2013/public/schedule/detail/31085) Khushil Dep of the MailOnline will address the conference and present how his firm leverages CFEngine for their infrastructure management needs. As Khushil puts it "At the MailOnline we use CFEngine as the core of our configuration management. A way in which we describe our intentions in clear, precise and workable logic patterns which the Machine can understand without ambiguity." It would be incorrect to just call this out as a Velocity Conf 'address'. You could look at it as a presentation or a tutorial or a product capabilities demonstration all rolled into one. What unfolds should be very interesting indeed. Khushil plans to build out infrastructure as part of his talk, explore the CFEngine language and fundamentals, showcase reusable patterns, and leave the audience with their appetite whetted enough to go try CFEngine on their own\! Expect Khushil to cover: - What is CFEngine - How does it work? - What are promises really? - Setting up a Linux machine and a SmartOS zone. They preferred deploying Joyent SmartOS over KVM or carving out VM's on Amazon. Find out why MailOnline believes that Amazon and KVM do not deliver on a 'cloud architecture' promise - Deploying software - Deploying configuration - Ideas for using cf-monitord metrics - A tour of the Enterprise Portal - A tour of how this all works at MailOnline There is also a meetup planned for later in the night with Khushil and many other CFEngine engineers and users. Click on the link to get details about this meetup to be hosted at the Nomad Lounge, London. We expect a good turnout and some robust discussions on the days proceedings as well as anything and everything CFEngine. Do let us know what you thought about the conference, Khushil's presentation and the meetup. Tweet @CFEngine with questions or comments\! Learn more about CFEngine by [visiting our website](https://www.cfengine.com) [Evaluate CFEngine for free](https://cfengine.com/evaluate-enterprise) (up to 25 licenses, unlimited usage) -------------------------------------------------------------------------------- title: "CFEngine at LISA 2013" url: /blog/2013/cfengine-at-lisa-2013/index.md date: "2013-11-11" -------------------------------------------------------------------------------- Mahesh Kumar, VP Marketing CFEngine mahesh.kumar@cfengine.com The CFEngine team was at LISA 2013 in Washington, D.C. this past week. It was the perfect setting, the weather playing nice, and having some of the sharpest minds talking about their accomplishments in the large installation system administration space. There were training sessions earlier on as the conference started on the 3'rd of November. Wednesday's proceedings were kicked off by Jason Hoffman, Founder, Joyent. Jason regaled the packed audience with his talk about ‘Modern Infrastructure - The convergence of network, compute and data'. It was very well delivered and informational. CFEngine participated across a variety of avenues at Lisa 2013. From the team, Mark Burgess, Diego Zamboni, Nick Anderson, Tom Buck and myself attended. Diego co-presented with Mike Svoboda of LinkedIn. Mike spoke about how LinkedIn uses in-memory caches to solve operations questions. Questions such as ‘where does the httpd process run', ‘who has a network connection to machine Y' etc. LinkedIn is using CFEngine to answer some of these questions at scale, efficiently and in real time\! You can [watch the actual presentation here](https://www.youtube.com/watch?v=RIaCtdyDoOc&feature=youtu.be) We also had some good press coverage that picked up on Mike's presentation. You can [read EnterpriseTech's Timothy Prickett Morgan's indepth coverage](https://www.aiwire.net/2013/11/07/linkedin-copes-server-explosion-revved-cfengine/) of this talk. Another talk I really enjoyed was on 'Surveillance, the NSA and Everything'. This was delivered to a standing room only audience by Bruce Schneier, *Fellow, Berkman Center for Internet and Society. *With the NSA being in the news recently this was timely and very informative. Bruce is both a master of this domain, as well as someone who can argue and present his point across in a very compelling fashion. This was on display given he was not even physically at the conference. He delivered his talk via skype and still got a standing ovation at the end of it. We had a steady stream of visitors to our booth. It was good to meet and speak with CFEngine users as well as the curious who wanted to know more about the technology and the problems it solves. We kicked off the evening with a ‘Birds of a feather session'. Mark Burgess started it off with a short ‘state of the CFEngine nation', followed by a loose and open, but very energetic discussion. Topics ranged from cloud, to software defined networking, to integration and support for OEM offerings (both storage and network). The packed room enjoyed s'mores and beer. Mark and Diego also signed their books on the show floor on Thursday. All in all a great show for LISA and for CFEngine. We have more happening in the next few weeks. Velocity London is fast coming up and so are Dev Ops days in the UK. Khushil Dep will be presenting. This is one presentation you don't want to miss\! There is also a meetup planned for that same week, as well as a keynote from Mark for Dev Ops days. November has been busy and we are just getting warmed up here at CFEngine\! Learn more about CFEngine by [visiting our website](https://www.cfengine.com) [Evaluate CFEngine for free](https://cfengine.com/evaluate-enterprise) (up to 25 licenses, unlimited usage) -------------------------------------------------------------------------------- title: "From The Marketing Corner" url: /blog/2013/from-the-marketing-corner/index.md date: "2013-10-21" -------------------------------------------------------------------------------- Mahesh Kumar, VP Marketing CFEngine @mahesh\_CFE mahesh.kumar@cfengine.com Hello Everyone\! It is not too often that one gets to join a company with solid fundamentals when it comes to technology and one that finds itself as the change leader in an area that is evolving real-time (DevOps and Continuous Delivery). Well, I am thrilled to report that I am onboard here at CFEngine as their VP of Marketing. In the days ahead I will be reporting back frequently on the cool things we are doing here at CFEngine. Such as recently winning an award from JPMC that you can read more about in [our blog](https://cfengine.com/blog/cfengine-inducted-into-the-jpmorgan-chase-hall-of-innovation-1). Of course we could speak about the ground breaking and innovative areas we are focused on here at CFEngine such as continuous delivery and continuous operations. However I am getting ahead of myself. I wanted to use this post to introduce myself as well as talk about the recent [Velocity](https://web.archive.org/web/20150923045244/http://velocityconf.com/velocityeu2013) and [DevOps](https://devopsdays.org/events/2013-newyork/) Days events in NYC. It was unusually warm and nice in NYC this time of the year. Velocity started out with some great talks by practitioners and thought leaders in this space. Jeremy Edberg of Netflix delivered one of my favorite sessions. As the reliability architect at Netflix his hands are full; the talk he gave on ‘How Netflix Operates Multi-Region Clouds for Maximum Freedom and Agility' was informative and gave the audience a peek into the Netflix philosophy around operations - including not having a QA department or Engineer\! There were many other notable presentations from the likes of [Google, Facebook, Etsy, DataDog](https://web.archive.org/web/20150923045244/http://velocityconf.com/velocityeu2013/public/schedule/proceedings). -------------------------------------------------------------------------------- title: "User promises are coming" url: /blog/2013/user-promises-are-coming/index.md date: "2013-10-11" -------------------------------------------------------------------------------- A while ago, work was started on a new promise type in CFEngine, the "users" promise type. What it does is make promises about which users are supposed to be present on each host, what login credentials they have, what groups they are part of, and so on. This has traditionally been something one would have to write in the CFEngine policy using system commands, but with this new feature it becomes a much more convenient and robust solution for users management. The work is not finished, but it's starting to take form nicely, and currently all use cases work on the Linux platform. Work is underway to bring support to the commercial Unixes, as well as Windows. Let's take a look at what the new promise type looks like. This is a typical promise: ```cf3 bundle agent mybundle { "john" policy => "present", password => johns_password, group_primary => "users", group_secondary => { "printers", "payroll" }, description => "John Smith", home_dir => "/nfs/home/john", home_bundle => new_home_dir("/nfs/home/john"); } body password johns_password { format => "hash"; data => "zvSwyQioR9DNI"; # "john" } ``` I think most attributes here should be self explanatory to most system administrators, but some are worth talking about. For instance, the `policy` attribute takes one of three possible values, and tells CFEngine whether you want this user to be "present", "absent" or "locked". The `password` body is also worth mentioning. It has two modes of operation: You can either promise that the user needs a certain plaintext password to log in, or you can (as displayed above) promise that a certain password hash is associated with a user. This saves you the potential security risk of having plaintext passwords listed in the policy. Note that there may be differences in what each platform supports here; the details are still being worked out. And at last, the `home_bundle`. It takes as a value the name of a bundle that should be evaluated when that user is created, and it is expected to fill the user's home directory with initial config files and setup. After that, the bundle is never executed again for that user, unless the account is deleted and recreated. As I mentioned, this is still work in progress, but for those of you comfortable with compiling your own version of CFEngine, you can try this feature by checking out the development branch. Clone the git repository and check out the branch by using these commands: ``` $ git clone https://github.com/kacfengine/core.git $ git checkout -b user-promises origin/user-promises ``` I assume that if you try this, you are familiar with how to build CFEngine from source code. Also, keep in mind that at the time of writing, only Linux has been tested. This is a much requested feature, and we hope that it will make configuration management even easier with CFEngine\! It will be available in CFEngine version 3.6.0. -------------------------------------------------------------------------------- title: "In Search of Certainty" - The origins of CFEngine" url: /blog/2013/in-search-of-certainty-the-origins-of-cfengine/index.md date: "2013-09-27" -------------------------------------------------------------------------------- Some software is more opinionated than others. If it ever made sense to anthropomorphize software, the CFEngine agent would be a good candidate. A CFEngine installation is a society of independent agents, each with their own goals, schedules and limited view of the world, individually doing their best under current weather. Mark Burgess' new book 'In Search of Certainty' may be viewed as a popular anthology for the design considerations that went into writing CFEngine. To that end, we are taken on a fast paced journey through topics such as classical and quantum mechanics, information theory, game theory and biology. It is clear that Mark had a lot of fun writing this book, drawing on and connecting dots of an eclectic array of examples from everyday life. Along the way, questions about how to design good tools are explored from a physicist's point of view. If you like popular science of the Roger Penrose variety, you will enjoy reading 'In Search of Certainty'. If you want a deeper understanding of the backdrop on which CFEngine evolved, the book is indispensable. Want a signed copy of the book? Mark Burgess will host a book signing in New York, the week of October-14. More details to come. For a sneak peak into the first chapter of the book [click here.](https://markburgess.org/certainty.html) -------------------------------------------------------------------------------- title: "Busy week in New York" url: /blog/2013/busy-week-in-new-york/index.md date: "2013-09-17" -------------------------------------------------------------------------------- The CFEngine team is excited to present - CFEngine Week in New York City, October 14-18\! During that week we will be out in force in the Big Apple. Come meet us in one of the many events we are organizing or sponsoring: ### [Velocity Conference, Oct 14-16:](https://web.archive.org/web/20150923045244/http://velocityconf.com/velocityeu2013/public/content/sponsors) Most companies with outward-facing dynamic websites face the same challenges: pages must load quickly, infrastructure must scale efficiently, and sites and services must be reliable, without burning out the team or breaking the budget. Velocity is the best place on the planet for web ops and performance professionals to learn from peers, exchange ideas with experts, and share best practices and lessons learned. CFEngine will demonstrate solutions to support such websites with continuous application deployment and dynamic infrastructure provisioning. And CFEngine founder and author, Mark Burgess, will be at the show to sign copies of his new book, [In Search of Certainty](https://www.amazon.com/Search-Certainty-Science-Information-Infrastructure/dp/1491923075/). **We will be at booth 7, come see us there\!** ### [1-Day CFEngine Training, Oct 15.](https://cfengine.com-newyork.eventbrite.com) Designed for System Administrators and Architects, this class will present an in-depth overview of how to use CFEngine to automate and manage large-scale IT infrastructure. Through a combination of theory, practical examples, and hands-on tutorial, our experienced trainer will demonstrate how CFEngine can be used for a wide range of common SysAdmin tasks. The training will take place at the Velocity Conference. ### [NYC CFEngine Meetup, Oct 15](https://www.meetup.com/ny-cfengine/events/138664192/). **We are thrilled to announce our first New York meetup - our own famous Mark Burgess will moderate and present two exciting speakers: Laurent Raufaste, of NY technology company Percolate and Mike Svoboda of LinkedIn, Inc.** At Percolate, Laurent Raufaste has built IT infrastructure that runs on Amazon Web Services and uses EC2 Spot Instances extensively. While very cost effective, this strategy requires him to be tolerant to frequent interruptions of server availability, and be able to provision infrastructure and applications very fast. Laurent will show how he makes this work at Percolate. LinkedIn is currently the 10th largest website in the US by traffic. Mike Svoboda's operations team relies on automation to tie major parts of infrastructure together, which has allowed LinkedIn the flexibility to scale horizontally indefinitely. Mike will present how they deploy their infrastructure and how they maintain real-time inventory information from tens or hundreds of thousands of machines. We start at [7pm on Tuesday October 15](https://www.meetup.com/ny-cfengine/events/138664192/). Pizza and drinks will be provided\! Although the Meetup will take place at the same facility as the Velocity conference, it is open to all who are interested, not just Velocity attendees (who are of course also welcome\!). ### [New York DevOpsDays, Oct 17-18.](https://devopsdays.org/events/2013-newyork/program/) The DevOpsDays events always attract the best minds from the DevOps world and this event in New York promises to be another of these inspiring events. On Thursday October 17 [Mark Burgess will present](https://devopsdays.org/events/2013-newyork/program/) "What science tells us about information infrastructure", a talk about what is weak about the way we currently run infrastructure, how to think in systems, how to deal with scale and complexity using real scientific methods (instead of guesswork). And finally, why "APIs will be our undoing" and why devops is crucial to the success of infrastructure in the future. We all look forward to meeting you in New York\! -------------------------------------------------------------------------------- title: "Managing hundreds of Linux servers with CFEngine and Git: full control over configuration" url: /blog/2013/managing-hundreds-of-linux-servers-with-cfengine-and-git-full-control-over-configuration/index.md date: "2013-09-11" -------------------------------------------------------------------------------- Authored by Remi Bergsma - [http://blog.remibergsma.com](http://blog.remibergsma.com/2013/08/31/managing-hundreds-of-servers-with-cfengine-and-git-full-control-over-configuration/ "Remi Bergsma") Back in June, just before I went off for holiday, I attended [a CFEngine training in Amsterdam](https://blog.remibergsma.com/2013/06/13/today-i-started-learning-cfengine-3/). When I returned from holiday a few weeks later, me and my team started making plans to implement CFEngine in our environment. After two months of hard work, I'm proud to say we manage about 350 out of our 400 Linux servers with CFEngine\! The ride has been fun, although not always easy. In this post I'll give a quick overview of our CFEngine implementation, where I found useful info, etc. ### **CFEngine is different** To start, let me tell you that one of the most difficult parts of learning CFEngine is to get used to the terminology and to ‘think' CFEngine. For example, a ‘class' in CFEngine is not what you think it is. It has nothing to do with object oriented programming. It's more like a ‘context' that you can use to make decisions. There's no ‘flow control' in CFEngine either: no IF/THEN/ELSE, no FOR/FOREACH/WHILE etcetera. In CFEngine classes are used for decision making, and, since CFEngine is smart, it does looping automatically. This results in clean and easy-to-read code. > *CFEngine works on top of a theoretical model called ‘Promise Theory‘ > by Mark Burgess (author of CFEngine). This theory models the behavior > of autonomous agents in an environment without central authority, > based on only promises of behavior made by each agent, and shows that > even without central control , the system can converge to a stable > state.* To get used to it, read ‘[Learning CFEngine 3](https://www.amazon.com/books/dp/1449312209)‘ by Diego Zamboni, as it will walk you through all of it with a lot of examples. The quote above is also from the book. The basic idea is that each agent makes promises only about its own behavior, since that is all it can control. In CFEngine 3, everything is a promise. ### Examples: - a file promises to have a certain content and to be executable - a service promises to be running - a user account promises to exist (or not to exist) and have certain properties When CFEngine finds a promise is not kept, it will do everything it knows about to make the promise true. If it cannot reach the promised state at first, it tries to the next best. Over time, the system converges to the desired (promised) state. Once you get it and get used to it, it actually makes sense and is pretty easy to implement. ### **With great power comes great responsibility** This one-liner says it all. When you have a configuration management system that manages a lot of servers, you better be careful what promises you have it keep. This is why you need to manage CFEngine promises like software. You need version control and it needs to be flexible as well. I've read a lot about this subject and I believe Git is the way to go. [This blog by Brian Bennett pretty much nails it](https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/). I got a lot of inspiration from it, thanks Brian\! I implemented these ‘branches' in Git: - development (aka master) - beta - pre-production - production This works perfectly: develop new promises in the ‘development' branch, then merge to ‘beta' branch to test on some of our own test servers. When everything works together and seems stable, we merge to the next branch ‘pre-production'. This is then tested on \~15 real production servers so it better be good. But when it isn't, the impact is still not too high and it should be fixed before it ever hits ‘production'. Production branch is everything that is stable and is used on all \~350 servers. Every time we merge to either ‘pre-production' or ‘production', we create a Git ‘tag' with a date, that allows for easy roll backs. Whenever we need to get back to a certain state, we can always just checkout a tag. This is also very useful for audit trails, by the way. Actually, we're using another branch called ‘hotfix'. Whenever there's an emergency to fix, we branch a ‘hotfix' from ‘production' and do the fix. This is for example when a promise misbehaves. This branch is then merged to production when ready, and also to ‘development'. Git handles this nicely: whenever the hotfix makes it all the way from ‘development' to ‘production', Git recognizes this commit was already processed earlier and ignores it.
![Git commits, branches and tags in CFEngine repo](https://remibergsma.files.wordpress.com/2013/08/cfengine_git_branches1.png?w=580&h=511) Git commits, branches and tags in CFEngine repo
This is a screenshot from ‘Gitk' that shows the commits, branches (green) and tags (yellow). As you can see, ‘production' and ‘pre-production' are at the same level now, so nothing new is tested in ‘pre-production' at the moment. Quite some work is tested in the ‘beta' branch and there are already some fixes committed in ‘master'. Recently there was a ‘hotfix' branch that has now been merged. It should give an idea of how it works. It provides a clear overview and we now know about every change on the configuration of our servers. Clicking on a commit show what changed, who did it, etc. ### **CFEngine Policy hubs** For each of the 4 branches we've created a CFEngine policy hub. The policy hub is a server running CFEngine software that serves the given branch to the agents (the Linux servers connected to it). Linux servers can even switch between them by ‘bootstrapping' to one of the 4 policy hubs. Although we only use that on our test servers. ### **Manage what's ‘in flight' with a CFEngine Trello board** [Trello](https://www.trello.com) provides an intuitive and modern web interface that allows you to manage ‘cards' on different ‘lists' on a ‘board'. To get an idea, see the example Trello board below (click on it to enlarge).
[![Trello CFEngine board](https://remibergsma.files.wordpress.com/2013/08/cfengine_trello_board.png?w=580&h=142)](https://remibergsma.files.wordpress.com/2013/08/cfengine_trello_board.png) Trello CFEngine board
New cards are usually created in ‘Feature requests' or ‘Bugs' and then transferred to ‘Working on it\!'. The number of cards in this stage should be limited, as you can only work on a number of things at the same time. This is actually *Kanban* style. Next, we've created a list for each Git ‘branch' we have and cards flow from ‘beta' to ‘pre-production' and finally ‘production'. Moving cards is just dragging & dropping. Each month, cards in ‘production' are archived. This creates an overview of what new work is to be done (‘Feature requests' and ‘Bugs'), what we're currently working on and what's in each of the branches. Trello has the overview, Git has the code and the details. Also, Trello is perfect for communication between team members. Notes, comments, documents, lists, etcetera can all be created with ease. ### **Testing promises** To be able to test the promises on our local laptops, we're using a tool called [Vagrant](https://www.vagrantup.com/). Vagrant sets up Virtual Machines (for example using Virtual Box) and allows you to ‘destroy' and ‘create' them within minutes. All team members have a local Git checkout, that is also available in the Vagrant boxes. This allows us to test any change before even committing. We have Vagrant boxes setup for all Linux distributions we support. It's so easy and so fast to test changes that everybody does. And even when an error slips through, other team members will soon notice and it's usually fixed within minutes, before it ever hits the ‘beta' branch. ### **Bugs** We encountered a strange bug when using SLES 11 and CFEngine 3.5: CFEngine (community edition) got running with the ‘SIGPIPE' signal blocked. When CFEngine restarts SSH, this too gets running with ‘SIGPIPE' blocked. This results in ‘sudo' no longer working. It would just return nothing at all. It took us quite some time to figure out it was the ‘SIGPIPE' signal that was blocked. The root cause probably lies in an old ‘Bash' version (3.51) that SLES uses, combined with something CFEngine triggers. We've now implemented an automated work-around (made a CFEngine promise) that fixes the problem. We did some nice team work on this one\! ### **Conclusion** CFEngine's learning curve might be steep, but the result is definitely rewarding. Combined with Git and Trello it allows for fine control and great overview of configuration changes. Our whole team is involved in changes, they are reviewed and result in high quality code. This eventually makes the Linux servers we manage more stable. Also, it's a great feeling to be in-control and know what's going on our servers. From this point on, we'll continue to both scale horizontally (add more servers) and vertically (add more promises). After two months of daily working with CFEngine, I've to say I really like it and I enjoy writing promises. I'll keep you posted, I promise ![;-)](https://s1.wp.com/wp-includes/images/smilies/icon_wink.gif?m=1129645325g) Remi Bergsma - [http://blog.remibergsma.com](http://blog.remibergsma.com/2013/08/31/managing-hundreds-of-servers-with-cfengine-and-git-full-control-over-configuration/ "Remi Bergsma") -------------------------------------------------------------------------------- title: "CFEngine 3.5.2 released" url: /blog/2013/cfengine-3-5-2-released/index.md date: "2013-08-30" -------------------------------------------------------------------------------- CFEngine 3.5.2 is now available for [download](https://cfengine.com/downloads). This is a maintenance release of CFEngine 3.5, and introduces a number of fixes and improvements to both Community and Enterprise editions. New functionality has been included to the SQL builder and the Design Center integration in Mission Portal. Changes in CFEngine Enterprise: - MongoDB has been upgraded to version 2.2.4 - monitoring data has moved into a separate database - see db-move-monitoring-to-cfmonitor.js script to migrate data Improvements to the Mission Portal: - Reports can be published and shared between users - Various UI improvements - Optimizations in the report engine Bug-fixes in the CFEngine Core: - fix delayed abortclasses checking (Redmine \#2316, \#3114, \#3003) - fix maplist arguments bug (Redmine \#3256) - fix segfaults in cf-promises (Redmine \#3173, 3194) - fix build on Solaris 10/SmartOS (Redmine \#3097) - sanitize characters from /etc/issue in sys.flavor for Debian (Redmine \#2988) - Fix segfault when dealing with files or data \> 4K (Redmine \#2912, 2698) - Don't truncate keys to 126 characters in getindices (Redmine \#2626) - files created via log\_\* actions now have mode 600 (Redmine \#1578) - fix wrong log message when a promise is ignored due to 'ifvarclass' not matching - fix lifetime of persistent classes (Redmine \#3259) - fix segfault when process\_select body had no process\_result attribute - Default to AND'ed expression of all specified attributes (Redmine \#3224) - include system message in output when acl promises fail - fix invocation of standard\_services bundle and corresponding promise compliance (Redmine \#2869) Bug Fixes in CFEngine Enterprise - Fix segfault of cf-serverd on HP-UX - Do not start a mongodb repair unnecessarily - cf-hub -H now supports multiple hosts The MD5 checksums for the various packages are: - cfengine-3.5.2.tar.gz - fa5a5270803fa24e9ab662aae6f73b4a - cfengine-community\_3.5.2-1\_amd64.deb - 8d495de2fb8220ea0ee7efe022322413 - cfengine-community\_3.5.2-1\_i386.deb - ce3b92dbcb8bf559e77b51f55b37def3 - cfengine-community-3.5.2-1.i386.rpm - 78c5b5cfe2578c787d442abfd45c75fb - cfengine-community-3.5.2-1.x86\_64.rpm - c954517e86ee30f57b82db07dad1011e Community repositories will be updated shortly. -------------------------------------------------------------------------------- title: "8 Reasons Why System Administrators Deserve Your Appreciation" url: /blog/2013/8-reasons-why-system-administrators-deserve-your-appreciation/index.md date: "2013-07-25" -------------------------------------------------------------------------------- Every year, on the last Friday of July, we pause for a moment and think about the impact of system administrators on our world. It is hard to imagine how different our life would be if it was not for the sysadmin. The goal of the [Annual SysAdmin Appreciation Day](https://sysadminday.com) is to make us all think about that and find ways to express our appreciation for them. At CFEngine we know many of them. In the past weeks we have interviewed hundreds of them and we want to share some fun facts we have discovered which show why they deserve our appreciation. [![](images/8-reasons-why-sysadmins-deserve-your-appreciation_51f1c019c7ac5.png)](/blog/2013/images/8-reasons-why-sysadmins-deserve-your-appreciation_51f1c019c7ac5.png) -------------------------------------------------------------------------------- title: "CFEngine 3.5.1 released" url: /blog/2013/cfengine-3-5-1-released/index.md date: "2013-07-10" -------------------------------------------------------------------------------- CFEngine 3.5.1 is now available for [download](https://cfengine.com/downloads). This is the first maintenance release of CFEngine 3.5, and introduces a number of fixes and improvements to both Community and Enterprise editions. New functionality has been included to the SQL builder and the Design Center integration in Mission Portal. Community repositories will be updated shortly. **Core Changes:** - the CFEngine Standard Library in masterfiles/libraries is now split into promise-type specific policy files, and lives in a version-specific directory. This should have no impact on current code, but allows more granular include of needed stdlib elements (Redmine \#3044) - file changes are logged with log level Notice, not Error **Bug fixes:** - fix recursive copying of files (Redmine \#2965) - respect classes in templates (Redmine \#\#2928) - fix timestamps on Windows (Redmine \#2933) - fix non-root cf-agent flooding syslog (Redmine \#2980) - fix email flood from cf-execd due to timestamps in agent output (Redmine \#3011) - Preserve security context when editing or copying local files (Redmine \#2728) - fix path for sys.crontab on redhat systems (Redmine \#2553) - prevent incorrect "insert\_lines promise uses the same select\_line\_matching anchor" warning (Redmine \#2778) - Fix regression of setting VIPADDRESS to 127.0.0.1 (Redmine \#3010) - Fix "changes" promise not receiving status when file is missing (Redmine \#2820) - Fix symlinks being destroyed when editing them (Redmine \#2363) - Fix missing "promise kept" status for the last line in a file (Redmine \#2943) **Enterprise Specific Fixes:** - don't generate ERR message during db maintenance if environments couldn't be queried, changed to INFO message - Fix usemodule on Windows - Fixed replica set detection (regression in 3.5) - Set correct precision format when storing db diagnostics to avoid null-values - Fix possible division-by-zero bug in compliance meters **Mission Portal:** - SQL queries can be shared between users - Fix timing issues for downloading large SQL reports - Purge sketch data when no longer used by active sketches - Uninstall sketches that have no activation - Support for boolean, menu option and optional parameters in Design Center UI - UI fixes to user and role management pages - Delete navigation tree definitions of deleted users - Fixes to password reset - General UI improvements - Fixes for IE8 compatibility **Enterprise Hub Changes:** - Perform a database repair from init script if unclean shutdown of mongod is detected - Improved database connection handling during report collection by cf-hub. - REST APIs support an optional disableCache flag; when set, the backend always hits the MongoDB -------------------------------------------------------------------------------- title: "Modular cfengine_stdlib.cf" url: /blog/2013/modular-cfengine_stdlib-cf/index.md date: "2013-07-05" -------------------------------------------------------------------------------- After work and discussion, the CFEngine developer team produced https://github.com/cfengine/core/pull/784 to make libraries/cfengine\_stdlib.cf compatible with 3.4.x and older again, to modularize it, and to improve the upgrade experience. These changes will be in the upcoming CFEngine 3.5.1 release: - libraries/cfengine\_stdlib.cf is now 3.4-compatible again, so upgrading will not break existing CFEngine installations. - lib/3.5 and lib/3.6 contain the standard library with changes that require 3.5.x and 3.6.x, respectively - lib/3.5 and lib/3.6 have multiple files instead of a single cfengine\_stdlib.cf. This is the same content as the cfengine\_stdlib.cf content, but split by promise type (e.g. lib/3.5/files.cf has file-related bodies and bundles). - common.cf has promise bodies common to all promises, e.g. action and classes - paths.cf has a utility common bundle called paths that lets you say, for instance, "$(paths.path\[rpm\])" instead of hard-coding the location of the RPM manager executable. - 3.5.1 and newer will have new variables to store the CFEngine major, minor, and patch version separately. It all works just like before 3.5.0, except you don't have to include a huge library. If you just want to set classes for a promise, include common.cf. If you just want process filters, include processes.cf. promises.cf has been modified in 3.5.1 to include these new libraries, so by default in 3.5.1 you will still have all the bodies you know so well. That way, upgrading will not suddenly break your policies. -------------------------------------------------------------------------------- title: "CFEngine Community 3.5 Released" url: /blog/2013/cfengine-community-3-5-released/index.md date: "2013-06-12" -------------------------------------------------------------------------------- CFEngine 3.5.0 Community Edition is now available for [download](https://cfengine.com/inside/myspace)\! At the same time we have updated the CFEngine documentation, which now lives in a new design and information architecture at . This latest release of CFEngine Community comes with a ton of improvements and new language features for policy writers. Many of these changes have been motivated by the great feedback and reports from our users, and a lot of changes were contributed by members of the growing CFEngine community. A big "Thank You\!" to you all\! Our [package repositories](https://cfengine.com/cfengine-linux-distros) will be updated shortly, and we will continue from here with monthly maintenance releases of CFEngine 3.5.x. A complete list of changes, both in this Community release and in the upcoming Enterprise Edition, is available online in the [New in CFEngine](https://cfengine.com/docs/3.5/whats-new.html) section of the documentation. We are currently in [Boston at the RedHat Summit](https://cfengine.com/press_releases/cfengine-to-exhibit-at-the-red-hat-summit-2013-showcasing-the-benefits-of-automation) to introduce CFEngine Enterprise 3.5 - meet us there if you want to learn more\! -------------------------------------------------------------------------------- title: "CFEngine 3.5.0 - Release Candidate for Community" url: /blog/2013/cfengine-3-5-0-release-candidate-for-community/index.md date: "2013-06-03" -------------------------------------------------------------------------------- The Release Candidate of CFEngine 3.5.0 Community Edition is now available for download at: [https://cfengine.com/inside/myspace](https://cfengine.com/inside/myspace) We're looking forward to your feedback to the changes and improvements\! ## Significantly improved parsing and evaluation of policies A large number of parser and evaluation problems have been fixed. Iterating over **arrays with nested lists** has caused many headaches and workarounds in previous versions of CFEngine. With 3.5, many of those can disappear. - Policy developers get **clear and consistent messages** from the **stricter parser**. **Note:** If your policy has syntactically incorrect code, then the new parser will mark those as errors. Fixing those errors should be straight forward, but we're interested in hearing about your experience with this so that we can make this as smooth as possible and document the typical issues for the final release\! - cf-promises allows partial check of policy - now you are not required to have \`body common control\` to check syntax. This will allow better integration with editors to perform automatic syntax validation. - \`action\_policy =\> warn\` now sets \`not\_kept\` classes, which allows you to see deeper than first order changes that might occur during \*\*dry-runs\*\*. - The **new scope attribute** in classes bodies allows policy writers to set non global classes based on result of a promise. This reduces managing class naming conflicts, especially when multiple activations of a bundle may happen. - Significant enhancements to bundle common paths, including the ability to test with automatically defined classes. Policy writers don't have to maintain their own common paths locations, or hard code paths into each bundle for paths that are well known. - CFEngine operators benefit from a range of improvements in **logging output**, both to the syslog and to the standard output. ## New built-in functions and capabilities A broad range of new built-in functions simplify the coding of policies significantly and offer completely new functionality. - set operation functions that test for set membership and find intersections/differences: - every - none - some - intersection - difference - list functions to look up a specific element, find the list length, extract a portion of the list, reduce it to unique element, sort it, and filter it - nth - length - sublist - unique - sort - shuffle - filter - function that returns a list of all the classes that match a regular expression - classesmatching - function that converts date/time values into a formatted string - stftime - function for building a string from data, using sprintf semantics - format - function for making logical decisions in a function call based on classes - ifelse - mapping function for arrays - maparray - function for getting detailed file information - filestat ## Improvements and simplifications for CFEngine Operators **avahi auto discovery** is available now in addition to specifying a static ip. Clearer messages during the bootstrapping process makes it easier to analyze why things might not have worked. Self-diagnostics of agent binaries via new command line option **--self-diagnosics**, checks the environment and state files on which CFEngine operates for consistency. For a comprehensive list of changes in CFEngine Community, see the ChangeLog file in the package, or in the [GitHub repository](https://github.com/cfengine/core/tree/3.5.x). Documentation for 3.5 is still work in progress in our [new documentation repository](https://github.com/cfengine/documentation). -------------------------------------------------------------------------------- title: "CFEngine Music To Our Ears" url: /blog/2013/cfengine-music-to-our-ears/index.md date: "2013-05-30" -------------------------------------------------------------------------------- Bruce Carleton is the organizer of the San Francisco CFEngine User Group. He has a deep understanding of system administration for distributed systems and how to make it easy (and fun\!) to use CFEngine for that. He sent a message to the user group email list, that we want to share with you: Re: \[SF-Cfengine-Users-Group\] Automation Adoption Challenges and Analogies During the last SF CFEngine users group, we had a good discussion about promoting adoption of CFEngine in our various environments. It can be a challenging thing to do, and I think it's worth sharing some of my thoughts on the matter. This is an edited version of an email I sent out on the help-cfengine mailing list, so I apologize in advance for the resend to those who have dual subscriptions. I think there are two key parts to the confusion new users of CFEngine experience. Rather than having the more familiar programming paradigms like imperative or object oriented programming, there is a high level domain specific language for managing files, processes and other things on distributed systems. Another part of the confusion comes from the problem domain itself. Rather than the familiar and consistent execution environment of disk, memory, registers and the processor(s) of a single system, there are many systems that are distributed, some that may have high latency or intermittent connectivity. Mutual authentication is another significant challenge. At the risk of being overly general I'll try to use music composition as an analogy for writing CFEngine code. Programmers from outside the large systems arena, might be a bit like a guitar player. They have become very familiar with their six strings and learned the neck of their instrument. When they compose music, that music may be written in the context of a single guitar player, playing in the known environment of their six strings. Improvisation is easy to do, because there is little coordination necessary. The guitar player can make up an arrangement as they go, and if they are skillful, it will still sound great. Now let's throw that guitar player into the role of composing music for a symphony orchestra. An orchestra contains sections of string, brass, woodwind, and percussion instruments. Now the composer has to provide arrangements for each of its players, a more complicated task. Part of this, is that the composer can not concern themselves with exactly how the players will play their varied instruments. The players already know how and must be trusted to do so. The composer has to provide the music score, the specific arrangements for each section of the orchestra. Now lets take that back to CFEngine. When you write a CFEngine policy, it needs to provide instructions to a varied number of systems that have been classified in various ways, possibly using their operating system, subnet, application and various other characteristics. One very important part of this is using the high level abilities of CFEngine, or said another way, just worry about the music. When you deploy your policy (the music) the systems subject to the policy will already know how to execute the policy using cf-agent. Another thing to consider, is that once the symphony has begun (production), it can be difficult to change the arrangement. Improvisation is difficult when using provisioning tools like Kickstart or the provisioning API's at cloud providers. You must work out ways to safely re-provision large numbers of hosts. You are also exposed to failures of the control system for provisioning, which may need to be very complicated and possibly be shared with a large number of uncooperating users. That control system can become a single point of failure from the users perspective. This is where CFEngine can shine. When skillfully deployed, it can provide a scalable and fault tolerant way to change the music, in the middle of the symphony. Improvisation becomes possible again, albeit at a slower pace. Now some might say that's exactly what's wrong with CFEngine and tools like it, though I don't feel that way. You will have to judge for yourself, given your environment. Something that also merits mentioning is the current state of cluster management and the elusive single system image. Creating a single system image is an admirable goal, but, treating individual systems as "autonomous" is better in the systems management context. It is easier to back-fit onto existing servers, something that is important when trying to manage large numbers of systems in an existing infrastructure. It also scales better because individual servers do more for themselves. It helps to eliminate single points of failure in an infrastructure and turns total service failures, into service degradations. Another thing is, systems should have some ability to fix themselves. Too much time is spent on monitoring and alerting human response, rather than creating automation that simply fixes things when they stop working, then records the event for later analysis. So anyway, I'll stop rambling now. If you have made it to the end of this post, thanks for following along. I hope you found it to be a valuable way of thinking about CFEngine. Comments are welcome. Best, \--Bruce -------------------------------------------------------------------------------- title: "Windows PowerShell support" url: /blog/2013/windows-powershell-support/index.md date: "2013-05-24" -------------------------------------------------------------------------------- Recently support for Windows [PowerShell](https://en.wikipedia.org/wiki/Powershell) was merged into the Enterprise Windows version of CFEngine. PowerShell is Microsoft's enhanced shell, intended for more advanced system administration and programming tasks. The change to CFEngine means you can seamlessly use PowerShell in your configuration policies, just like the normal "cmd" shell. It's very easy, for example, say you have the following bundle: ```cf3 bundle agent mybundle { vars: "mymessage" string => execresult("echo This is a message from cmd", "useshell"); reports: "$(mymessage)"; } ``` This uses the regular "cmd" shell. To switch to PowerShell, just replace it with the following: ```cf3 bundle agent mybundle { vars: "mymessage" string => execresult("Write-Output This is a message from PowerShell", "powershell"); reports: "$(mymessage)"; } ``` You can also use the powershell class to detect whether PowerShell is available: ```cf3 bundle agent mybundle { vars: powershell:: "mymessage" string => execresult("Write-Output This is a message from PowerShell", "powershell"); !powershell:: "mymessage" string => execresult("echo This is a message from cmd", "useshell"); reports: "$(mymessage)"; } ``` You can use PowerShell in the two functions ["execresult()"](https://cfengine.com/manuals/cf3-Reference#Function-execresult) and ["returnszero()"](https://cfengine.com/manuals/cf3-Reference#Function-returnszero), as well as in the ["commands"](https://cfengine.com/manuals/cf3-Reference#commands-in-agent-promises) promise type. To use it in the "commands" promise type, use the "useshell" attribute in the "contain" body of a "commands" promise: ```cf3 bundle agent mybundle { commands: "Write-Output Hello world" contain => mycontainbody; } body contain mycontainbody { useshell => "powershell"; } ``` In relation to PowerShell support, a number of changes were also made to the non-Windows shell handling, both in the Enterprise and Community version of CFEngine: - Commands executed using "useshell" no longer require the first word to be an absolute path. This was originally in place for security reasons, but in a complex shell expression this security measure does not work, and was therefore removed entirely. Absolute path names are still required for the "noshell" type, which is recommended whenever possible. - The "useshell" attribute inside the "contain" body used to take boolean values. Now it takes one of the following values: "noshell", "useshell" or "powershell". PowerShell support will be available in CFEngine version 3.5.0. -------------------------------------------------------------------------------- title: "Behind the scenes: How do we test CFEngine" url: /blog/2013/behind-the-scenes-how-do-we-test-cfengine/index.md date: "2013-05-16" -------------------------------------------------------------------------------- Over the last year we have changed the way we test our software from a manual process to a highly automated process. This new system is capable of taking a change from our source code repository and follow it all the way up to where we update our internal staging servers, thus giving us incredibly valuable information while keeping manual intervention to a minimum. ## Overview Our test system consists of the following pieces: - Two policy hubs that manage our build and deploy farm - Vagrant to deploy new machines on the fly - A Jenkins master + build slaves - Build scripts that automate most processes We follow the mantra of eating our own dog food and therefore we use CFEngine to manage our build farm. All of our build slaves (and even the build master) are running CFEngine. This allows us to focus on the important things knowing that CFEngine will keep our build slaves behaving correctly. This has also served a double purpose, we have found (and fixed) a few issues by using CFEngine this way. We have also a second policy hub that is used to launch and configure machines on demand. We use this policy hub to control the hypervisor that hosts our build slaves and the Jenkins master. This policy hub is separated from the other policy hub since we like to have both a physical and conceptual boundary. It also contains policies to do house keeping in our hypervisor. As an added bonus, this allows us to reprovision our hypervisor very quickly on a new machine if the current machine fails. We will describe our testing procedures later, but we can tell you now that we need to be able to quickly deploy machines and bring them down after they are done. And for that we have found that Vagrant + CFEngine integration makes our life very easy. As most software shops, we have a full size Continuous Integration system. We choose Jenkins because of the myriad of plugins that make our life very easy. Finally, we keep our Jenkins' recipes to a minimum. We keep all of our build instructions into a separate repository. This has the advantage of allowing us to build things even if the whole CI system is down. And of course, it is very easy to keep track of them since they are in a proper source code repository. ## **Building and initial testing** We start the test cycle by triggering a build when a commit reaches our source code repository. This is done by configuring Jenkins so it checks our repositories every 5 minutes. This gives us some time to push several commits at once. Once Jenkins detects a commit it then bootstraps the code. This is, it runs the autogen script with the NO\_CONFIGURE=1 flag so configure and friends are properly generated. We do this in a special machine to make sure that no strange versions of either autoconf or automake produce a weird configure script. We then use Jenkins to preserve the workspace and use it as a base for the other build processes. The next station is to have a flash compilation process. This is just a sanity check that only compiles the code without running any tests. We run this in two platforms: mingw (cross compilation from Linux) and Ubuntu 10.04. If this stage fails, then the process is stopped and an email is sent to the committers and to our internal development mailing list. The next step is to build our software and run our unit and acceptance tests. We do this in several platforms, since there might be slight differences in compiler default options and location of header files. We run this stage only on platforms that take less than one hour to compile and run the tests and that are currently supported by their respective vendor. We have a different stage for platforms that take too long, so we do not start building on those if there is an obvious problem in the other platforms. The last step in this stage is to produce packages. In case of problems, committers and our internal mailing list gets a notification. ## Deployment testing Once we have finished producing packages, we start performing deployment tests. We run first the deployment of hub packages. For that we deploy a new machine and install the fresh packages into it. This way we are sure that there are no transients in the machine. After package installation we run a few sanity checks, to make sure that everything is working as expected. Our next stop is the deployment and test of agent packages. For that we start with deploying a hub machine (also on a clean machine), and then start several agent machines on demand. We install the packages on those agent machines and then bootstrap those machines to our hub. We run some sanity checks on the agent machines to make sure everything works as expected. For our enterprise version we run UI tests using Selenium and php unit. The tests are run on a machine that is provisioned on demand and results are sent back to our Jenkins installation. All of these tests are performed across multiple platforms. We use Vagrant to provision these machines since they are short lived and Vagrant is really good at provisioning machines. The machines are configured using a mixture of Vagrant scripts and CFEngine policies. Once the configuration phase is done we start running tests and report results back to our Jenkins master. ## Updating our staging areas The final step in this process is the update of our internal staging areas. Internally we always keep staging areas, which are basically just hubs with some agents deployed to them so we can see how our software is working. We have one staging area per branch we are working on. These stage areas live as long as they are the latest working revision of that branch. After a round of successful testing, we deploy a new machine from scratch and we use CFEngine to configure it. We install the new CFEngine package into it and bring the machine online. This is specially useful during development because it gives us quick feedback on the progress that we have made. ## Useful tips As we say, we rely on Jenkins to help us manage our builds. We use Jenkins for two main reasons: it is extremely simple and powerful, and it has a myriad of plugins. Particularly we rely on the following plugins: - Build Pipeline: To provide a nice pipeline view that is very simple and informative. - Email-ext: To provide customized email messages. - FTP publisher: To collect the packages and copy them to our internal ftp server. - GIT plugin: To access our repositories. - Multiple SCM plugin: We need to monitor and check out things from several repositories and this plugin lets you do exactly that. - Clone Workspace SCM plugin: As explained we bootstrap the code in a particular machine and then we copy it to other build slaves. This plugin hides all the complexity and lets you do exactly that. - Regression report plugin: With this plugin Jenkins tells us which tests failed. ## Conclusions If you are in a situation like ours where you not only need to be able to compile and package your software but also to deploy it and test it, then the combination of CFEngine, Vagrant and Jenkins is a winning move. We are able to quickly deploy new build slaves, deployment machines and even our whole infrastructure within minutes with the knowledge that the configuration that we need will be maintained by CFEngine, thus avoiding any drifts. We can easily customize the build slaves or the deployment machines by simply modifying our policies or writing new policies. Finally, if you would like to replicate this setup for your own software project let us know and we will try to help you get started. -------------------------------------------------------------------------------- title: "format(): sprintf semantics in CFEngine" url: /blog/2013/format-sprintf-semantics-in-cfengine/index.md date: "2013-05-16" -------------------------------------------------------------------------------- If you are a programmer, you probably know and maybe even love the sprintf family of functions, which format output based on a format specification and on some input data. Here's a [description of the formats](http://www.gnu.org/software/libc/manual/html_node/Table-of-Output-Conversions.html#Table-of-Output-Conversions "libc sprintf format description") available in the GNU libc implementation. I wrote an implementation called **format()** which will be in the upcoming 3.5.0 release. The **format()** function offers string and number formatting. To reduce the chance of security compromises, you can't print pointer offsets or single characters for instance. The available specifiers are: - **%s**: print a string - **%d**: print an integer - **%o** and **%x**: same as **%d** but in octal and hexadecimal representations, respectively - **%f**: print a real number In addition you can use the standard width specifiers. Briefly (feel free to try other variations): ```cf3 body common control { bundlesequence => { run }; } bundle agent run { vars: "a" string => format("%10.10s", "x"), comment => "pad input to 10 spaces; print no more than 10 characters"; "b" string => format("%-10.10s", "x"), comment => "left-indent input and pad to 10 spaces; print no more than 10 characters"; "c" string => format("%04d", 1), comment => "format as integer, padded with zeroes on the left to 4 digits"; "d" string => format("%07.2f", 1), comment => "format as floating point with two digits of precision, padded with zeroes on the left to 7 characters total"; "e" string => format("hello %s, my IP is %s", $(sys.policy_hub), $(sys.ipv4)), comment => "format with two input variables"; reports: "%10.10s on 'x' => '$(a)'"; "%-10.10s on 'x' => '$(b)'"; "%04d on '1' => '$(c)'"; "%07.2f on '1' => '$(d)'"; "hello my IP is... => '$(e)'"; } ``` Which produces (this agent is not bootstrapped so the policy hub is undefined): ``` 2013-05-16T11:43:53-0400 notice: R: %10.10s on 'x' => ' x' 2013-05-16T11:43:53-0400 notice: R: %-10.10s on 'x' => 'x ' 2013-05-16T11:43:53-0400 notice: R: %04d on '1' => '0001' 2013-05-16T11:43:53-0400 notice: R: %07.2f on '1' => '0001.00' 2013-05-16T11:43:53-0400 notice: R: hello my IP is... => 'hello undefined, my IP is 127.0.0.1' ``` I hope you enjoy this new function\! -------------------------------------------------------------------------------- title: "New Output Format" url: /blog/2013/new-output-format/index.md date: "2013-05-13" -------------------------------------------------------------------------------- Last week, we revamped the way CFEngine formats output. You can continue to use the old output format by specifying the option --legacy-output (-l), but this is considered deprecated and may be removed in a future version. The key features of the new output format are as follows. - Consistent look for all messages, as formatting is taken care of by the logger. - Separate formatting for terminal, syslog, and (eventually) file. Attempts to follow conventions for formatting to separate sinks. - ISO 8601 timestamps to terminal. - Every statement is single line. - No banners and decorations within messages. Conventions for how to write messages. - Logging levels (e.g. specifying --verbose always implies --inform), aligning with syslog levels. - Logging system knows about evaluation state and may format messages conditioned on what the CFEngine is currently trying to do. This is perhaps best illustrated by an example (in this case, written to terminal). 2013-05-13T13:19:34+0200 verbose: Using bundlesequence => {'main'} 2013-05-13T13:19:34+0200 verbose: /main/reports/'hello world': Report: hello world. 2013-05-13T13:19:34+0200 notice: R: hello world The path in the second statement, /main/reports/'hello world', is an example of CFEngine formatting output depending on evaluation state. Here it attempts to evaluate a reports promise in bundle "main", with the promiser being "Hello world". Also note how the output of the reports promise is written to terminal in the third line. Because all output is consistently formatted, this results in the terminal output having a timestamp and a log level, rather than the bare "R: hello world" as in the previous version (or with --legacy-output). We still have a bit to improve in logging. Specifically we want to - Improve the quality of messages. - Reduce the number of messages, generally printing only relevant information. - Use evaluation state in the formatting of more messages, meaning that the context in which the message originated will be more clear. Please let me know what you think of this change, and how we can improve\! ----- Sigurd Teigen I'm a programmer working for CFEngine, currently mainly on the community code base. Let me know what new features you would like to see in CFEngine Community. Twitter: @sigurdteigen ![tl\_files/cfengine/img\_new/blog/sigurd.jpg](images/sigurd.jpg) -------------------------------------------------------------------------------- title: "IT-Organizations Face Disruption - Automation to the Rescue" url: /blog/2013/it-organizations-face-disruption-automation-to-the-rescue/index.md date: "2013-05-09" -------------------------------------------------------------------------------- IT-organizations around the world face constant pressure to do more, do it faster and with fewer resources. They are seen as cost centers and as such benchmarked against the performance of nimbler competitors and the public cloud providers. IT-organizations feel that pressure from multiple sources at the same time. The CFO compares his internal IT-budgets with the costs of public cloud offerings and demands comparable savings. The lines of business know they can get the latest applications and platforms within minutes in the public cloud and complain about the slowness of their internal IT department. And the end-users expect continuous availability of all IT-services. Add Moore's law to describe the growth in scale and complexity of the IT infrastructure, and it becomes understandable why CIOs have the shortest tenure of all C-level executives. Luckily automation can come to the rescue for most IT-organizations. The business value produced by highly automated IT-organizations is increasing exponentially. They deliver unbeatable metrics when it comes to change-frequency, IT-service quality and operational costs. Some companies are now so good at automation that they can beat the economics of public clouds, move their application workloads back to their on-premise infrastructure and gain more granular control of their data, their policies and their compliance. I have previously argued ([Why Automation Leads to Greatly Reduced IT Costs](https://cfengine.com/blog/Why-automation-leads-to-greatly-reduced-IT-costs)) that IT-organizations that don't become highly automated run a great risk of being outsourced. Modern organizations will be managed by a new breed of System Administrators, the System Engineers ([From System Administrator to System Engineer](https://cfengine.com/blog/from-system-administrator-to-system-engineer)). IT-organizations that are highly automated will become much more productive than the ones that don't, and the delta grows exponentially with infrastructure size over time.
![The Value Gap](images/Valuegap.png) The value gap between manual and automated IT-operations grows exponentially with the increase of changes and size of infrastructure
Automation requires upfront investments. For very small environments (less than a hundred servers/VMs) there is not that much value in automation, but as the number of changes and the size of infrastructure grows, the value of automation increases exponentially. We see it especially in the number of tickets and mean time to repair (MTTR). IT-operational metrics, like number of support tickets, mean time to repair, time to deploy new changes and the numbers of system engineers are all metrics that flatten in a highly automated environment. Once all changes are version controlled and managed by CFEngine through patterns and models, it doesn't matter whether you are managing 5,000 or 50,000 servers. In the manual scenario all of these metrics will grow above linear and quickly result in an unsustainable situation of explosive headcount increases and poor IT performance. The automation train is leaving the station and if you are responsible for IT-operations in your organization the time is now to embark on your automation journey. ## Automation is a journey Automating IT-operations is a journey both with regards to the level of automation, the IT-service and the number of IT-services. Ideally all changes in IT-operations should be managed by an automation solution and recorded in version control. Very few organizations enjoy this luxury, but we are starting to see more and more that do. CFEngine has use-cases where one System Engineer manages thousands of servers, changes are deployed into production on a daily basis and automatic remediation of drifts saves the need for manual actions and ensures a high quality of service. For most IT-organizations the automation nirvana seems unreachable. A first important step is to realize that automation is a process that requires buy-in from people. Start slowly, get buy-in and then accelerate as everyone begins to recognize the benefits.
![The Automation Journey](images/automationjourney.png) Automation is a journey and it takes time\!
Start by focusing on your customer and the IT-services you are providing. Automate the most mission-critical ones first. Once you see the results, you can move on to the less mission-critical applications. In the end all IT-services should be automated because an outage anywhere in the stack can have cascading effects that are impossible to predict the outcome of. -------------------------------------------------------------------------------- title: "Syntax warnings" url: /blog/2013/syntax-warnings/index.md date: "2013-04-26" -------------------------------------------------------------------------------- Over the past month we have removed a bit of functionality for CFEngine. These were features that were useful (cf-report) and good ideas (knowledge management), but the implementation was either in an unfinished state or the design had unclear semantics (outputs promise type). In order to get a smoother transition for existing policy, we have introduced GCC-style warnings for cf-promises, initially to warn users about removed or deprecated syntax. Deprecated syntax warnings signal that the feature will be removed in a future version, but is still functional with the current version. Removed syntax warnings signal that the feature has been removed. As with your standard C compiler, these warnings will not result in syntax error unless specified. For example, to enable all warnings use cf-promises -Wall promises.cf , or to treat removed syntax warnings as errors, use cf-promises -Werror=removed promises.cf We plan to extend cf-promises with more warnings going forward. We will update man pages and documentation shortly. Please try it out and let us know your feedback\! -------------------------------------------------------------------------------- title: "A Case Study in CFEngine Layout" url: /blog/2013/a-case-study-in-cfengine-layout/index.md date: "2013-04-15" -------------------------------------------------------------------------------- Authored by Brian Bennett of [Digital Elf](https://digitalelf.net/2013/04/a-case-study-in-cfengine-layout/) I've been working a lot with CFEngine newbies. CFEngine has been described as flour, eggs, milk and butter. All the ingredients needed to make a cake. Getting the new CFEngine user to recognize, then become excited about the possibilities that CFEngine provides they are now faced with the question of "What next?" Indeed, anybody can throw some flour, eggs, milk and butter into a bowl, mix and bake it. But will it taste good? This is an exposé of how I have managed my CFEngine repository for more than eight years. This design was used to manage over 1,000 host instances. This works best if you have an agile infrastructure. Use [SmartOS](https://smartos.org/), [OpenStack](https://openstack.org/), [Amazon EC2](https://aws.amazon.com/ec2) (Disclosure: I work for Amazon), [CloudStack](https://cloudstack.apache.org/) or similar. ## The repository, and version control Firstly, place your CFEngine repository in some revision control. I am highly partial to `git`. Get the [Pro Git](https://www.amazon.com/gp/product/1430218339/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1430218339&linkCode=as2&tag=digitalelfnet-20) book (or [download](https://www.git-scm.com/book) it). Read chapters 1, 2, 3. This will make you a git power user. After you're comfortable using git read chapters 6 and 7. When you're hungry for more, read the rest. I symlink `/var/cfengine/masterfiles` to `/cfengine/inputs`. This contains all of my policy files. I also create `/cfengine/files` for files that get copied to remote systems. This mostly contains my configuration files. `/cfengine/` is initialized as a git repository. Changes made to either `inputs` or `files` should be atomic. Adding something new for Apache? Any `inputs` and `files` involved should be checked in as a single commit. This makes reverting a change easier. ### Environments I use four environments. - Alpha - Beta - pre-Production - Production I also lied about initializing `cfengine` as a git repository. I use a central repository server that contains only a bare git repository. The central repository has four branches. - master - beta - preprod - prod Astute readers will notice there's no alpha branch. I'll get to that later. **beta** is a full integration environment. Everything in beta is expected to work, yet not to be relied upon. That is to say, nothing should move to beta that is known broken. Beta will break. But don't do that intentionally. If it's half finished keep it out of beta. **prod** is the full production environment. Breaking this means losing money. Don't break this. Prod is tagged daily. Rolling back is done by checking out the appropriate tag. **preprod** is for final quality assurance testing. Preprod should be identical to prod except for changes to be imminently released to prod. Preprod can also be used for offline testing of the production environment without affecting capacity or availability. Preprod should be in your production network fabric. **master** is the trunk. All code is initially merged here, then merged to the appropriate branches. No one should be allowed to merge directly to any branch other than master. The repository czar merges commits to other branches. ## A DevOps Workflow This is why there's no alpha branch. Let's assume that you're going to be making a change to the configuration of Tomcat. 1. Spawn a new CFEngine instance. 2. `git clone` the CFEngine master branch and bootstrap the server to itself. 3. Spawn as many instances as necessary for your application to work. This will be at least Tomcat instances, possibly including Apache and Postgres instances and bootstrap all of them to your new CFEngine server instance. 4. By editing only the CFEngine files out of the cloned repository make all of your updates. 5. Code review 6. When that feature is ready push a single commit to master and merge to beta. 7. Integration testing in beta. 8. Changes that need to be made are done in the private instance set. When they're ready proceed from step 5. 9. When that feature is ready merge from beta to preprod. 10. Final QA testing. 11. Changes that need to be made are done in the private instance set. When they're ready proceed from step 5) (Yes, that means it goes through integration again). 12. When that feature is ready merge from preprod to prod. ## Managing The CFEngine Repository: Layers I use a layered approach. Each layer is contained within a single bundle. - Meta - These are things that affect every host that runs CFEngine. Layers that are based on intrinsic characteristics: - Operating system families - windows, unix (anything Unix like) - Operating System - linux, solaris, bsd, darwin - Distribution - debian, redhat, solaris11, omnios, freebsd, openbsd - Distro Version sub-layer - debian\_6, redhat\_6, centos\_6 Layers that are based on the role - Application - apache, postgresql, mysql, bind, tomcat(These are often named after packages) - Application sub-layer - apache1\_3, apache2, tomcat6, tomcat7 - Role - external\_web, internal\_web, proxy, smarthost - Hostname - web\_f7f274, web\_4d06a8 ### BundleLayer Mapping I generally contain one bundle per file, per layer. The default policy files that come with CFEngine are in what I consider the meta layer. This is a subset of my policy files to give you an idea of the organization. - `unix.cf` - `bundle agent unix` - `linux.cf` - `bundle agent linux` - `debian.cf` - `bundle agent debian` - `redhat.cf` - `bundle agent redhat` - `solaris.cf` - `bundle agent solaris` - `apache2.cf` - `bundle agent apache2` - `bind9.cf` - `bundle agent bind9` - `web_ext.cf` - `bundle agent web_ext` (policy for public facing web servers) - `dpkg.cf` - `bundle agent dpkg` (Package management common to Debian) - `rpm.cf` - `bundle agent rpm` (Package management common to RedHat) - `ips.cf` - `bundle agent ips` (Package management common to the Image Package System, used by Solaris) - `digitalelf_stdlib.cf` - Private library of bundles and bodies. This is similar in nature to `cfengine_stdlib.cf`, but I never change `cfengine_stdlib.cf`. I put things into my private library. When they are well tested I open a pull request with [`cfengine/core`](https://github.com/cfengine/core) to contribute it. All promises are added to the lowest layer bundle (with global being the lowest and hostname being the highest). Thus, changes to `/etc/resolv.conf`, because all Unix like systems treat `/etc/resolv.conf` alike goes into the `unix` layer. The `sysctl` handling is different per operating system so they go into `linux` and `bsd` bundles at the OS layer. An external facing web server, by nature of being a web server must include apache as does an internal facing web server, so each automatically pulls in `apache2`. Likewise canonical DNS servers and caching DNS servers alike pull in `bind9`. ## Dynamic bundlesequence Because of the layered approach, which inputs and bundles need to be run are dynamically generated. Public web servers running on Debian Linux will be able to select the `ext\_web`, `apache2`, `debian`, and `linux` bundles automatically. I can have the same web content on Solaris 11 and it will instead choose `ext\_web`, `apache2`, and `solaris` bundles. I have a very large header to `promises.cf` to facilitate this. Here is an excerpt, along with additional commentary of my `promises.cf` to show how the `bundlesequence` is dynamically generated. ```cf3 bundle common classify { # This section classifies hosts instances into roles based on the hostname # I use a completely virtualized infrastructure with hostnames determined by # on a role specific prefix and a hex string separated by an underscore. # The hex string is the last 3 bytes of the MAC address of the lowest # numbered interface (e.g., eth0). Instances are created this way by my # provisioning system. classes: "dns_ns" or => { classmatch("ns[0-9]*") }; "dns_forwarder" or => { classmatch("dns_[0-9a-f]*") }; "db_server" or => { classmatch("db_[0-9a-f]*") }; "gitlab" or => { classmatch("gitlab_[0-9a-f]*") }; "web_ext" or => { classmatch("www_[0-9a-f]*") }; "web_int" or => { classmatch("web_[0-9a-f]*") }; "xwiki" or => { classmatch("xwiki_[0-9a-f]*") }; # Roles choose application bundles "apache" expression => "dpkg_repo|web_ext|web_int"; "bind" expression => "dns_ns|dns_forwarder"; "postgresql" expression => "db_server"; "tomcat" expression => "xwiki|jira"; "rails" expression => "gitlab" # Roles and/or applications can be grouped "app_server" expression => "rails|tomcat" # Applications may also depend on other applications "sql_client" expression => "app_server"; "ssl" expression => "apache|tomcat|rails"; "stunnel" expression => "mysql"; } bundle common g { # This section assigns bundles to application/role/grouping classes. # An array is created, named **bundles**. Each *key* is named after # a *bundle*. The *value* of each key is the input file where that # bundle can be found. vars: # These classes were defined by me in the classify bundle apache:: "bundles[apache]" string => "apache.cf"; bind:: "bundles[bind]" string => "bind.cf"; postgresql:: "bundles[postgresql]" string => "postgresql.cf"; ssl:: "bundles[ssl]" string => "ssl.cf"; stunnel:: "bundles[stunnel]" string => "stunnel.cf"; # Thse are hard classes determined by cfengine. I don't need to explicitly # classify them. debian:: "bundles[dpkg]" string => "dpkg.cf"; "bundles[debian]" string => "debian.cf"; centos:: "bundles[rpm]" string => "rpm.cf"; "bundles[centos]" string => "centos.cf" sunos_5_11:: "bundles[ips]" string => "ips.cf"; "bundles[solaris]" string => "solaris.cf"; xen_dom0:: "bundles[xen_dom0]", string => "xen_dom.cf0"; # Now the magic. # I create two slists. One named "sequence" and one named "inputs". # The "sequence" slist contains a list of bundle names. # The "inputs" slist contains a list of input files. any:: "sequence" slist => getindices("bundles"); "inputs" slist => getvalues("bundles"); } body common control { # The bundlesequence now includes those things which are common to all, plus # the contents of the slist "sequence" (which has ben dynamically generated), # plus the unqualified hostname. bundlesequence => { "global", "main", "@{g.sequence}", "${sys.uqhost}"}; # The inputs now includes common libraries and main.cf which will be run by # all systems, plus the contents of the slist "inputs" (which has been # dynamically) generated, plus an input based on the unqualified hostname. inputs => { "cfengine_stdlib.cf", "digitalelf_stdlib.cf", "main.cf", "@{g.inputs}", "${sys.uqhost}.cf" }; # Sometimes I need to have any specific configuration for a single host (e.g., # one of dns_ns will be the master and the rest will be slaves so the master # needs special configuration). The following options will allow CFEngine to # skip the hostname bundle/input if one does not exist (which it usually # doesn't). ignore_missing_bundles => "true"; ignore_missing_inputs => "true"; version => "Community Promises.cf 1.0.0"; } ``` Notice that instances are automatically classified by their hostname. So if I need a new external web server I provision a new instance with the name prefix `www_` (I can also choose the OS at provisioning time). My provisioning system automatically assigns them a unique ID, creates the instance, installs the OS, installs cfengine, bootstraps it to the CFEngine master server, runs cfengine to apply the final configuration and finally adds the instance's services to the appropriate load balancer entries. I have repository mirrors of all platforms I run so a newly provisioned host can be in production with a perfect configuration in as little as five minutes. \[1\]: [Pro Git book](https://www.amazon.com/gp/product/1430218339/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1430218339&linkCode=as2&tag=digitalelfnet-20) \[2\]: -------------------------------------------------------------------------------- title: "March 2013 snapshot of CFEngine - get it while it's hot!" url: /blog/2013/march-2013-snapshot-of-cfengine-get-it-while-its-hot/index.md date: "2013-03-25" -------------------------------------------------------------------------------- The March snapshot release of CFEngine Community and Enterprise has just hit our webservers. The releases are called 3.5 Beta1 and 3.1 Beta1, respectively, and just as [last month](https://cfengine.com/blog/february-2013-snapshot): **this is work in progress**. ## Community Some low level changes have been introduced. It is now possible to have empty lists in policies, there is no need to use "cf\_null" anymore. We have continued our architectural work and we have changed FatalError for other less problematic error conditions along the code. This allows us to have a better error handling than what we used to have. The parser and the evaluator are under major work, and in this release you will have the opportunity of testing the improvements. As well we have fixed a few crashes and other bugs. ## Enterprise We have continued working on Hub Diagnostics, adding more measurements and making sure that we get the right information to diagnose potential problems. For the mission portal we have modified the navigation tree, adding the possibility of creating custom trees and sharing them between users. In our quest to make a more user friendly product we have started to move functionality around so people need to use less executables. We have moved the query functionality from cf-runagent to cf-hub in order to keep all the querying functionality in one place. ## Packaging We heard your problems about too many packages and too many combinations and we decided to do something about it. We now have only 4 packages for clients: - 32 bits for .deb systems - 64 bits for .deb systems - 32 bits for .rpm systems - 64 bits for .rpm systems Our enterprise customers will also find that the packages have a new and simpler name scheme. Agent packages will be called *cfengine-enterprise-agent-\-\*. We are working on a similar solution for enterprise hub packages, although **that is still work in progress**. ## Get it while it's hot\! Customers find the packages in a subdirectory "alpha\_releases(unstable)" in their download area on [cfengine.com](https://cfengine.com/ "cfengine.com"). If you are not a customer, but want to evaluate this release, please sign up [here](https://cfengine.com/25free) to download a [free version of CFEngine Enterprise](https://cfengine.com/25free). Once registered, you will have access to the latest stable release as well as to this monthly snapshot release. These free releases can be used to manage up to 25 nodes. Community users can clone our [github repository](https://github.com/cfengine/core) - check out tag 3.5.0b1 - and there are also [packages available here](https://cfengine.com/inside/myspace). A big shout-out to all community contributors - your improvements to the code and to our policies and promise libraries are again much appreciated :) -------------------------------------------------------------------------------- title: "CFEngine is very easy to install - and now so is bootstrapping!" url: /blog/2013/cfengine-is-very-easy-to-install-and-now-so-is-bootstrapping/index.md date: "2013-03-22" -------------------------------------------------------------------------------- CFEngine is very easy to install -- just one package per operating system if you are using one of the pre-compiled packages we provide for the [Community](https://cfengine.com/inside/myspace) or for [Enterprise](https://cfengine.com/enterprise-download) editions. We wanted to bring an equally effortless experience to the task of starting CFEngine; bootstrapping a client to a policy server. Previously, this has been done by running /var/cfengine/bin/cf-agent --bootstrap --policy-server which looks simple enough. However, the problem is that bootstrap and policy server are tightly connected - it does not make much sense to run them independently (bootstrap was actually just a shorthand for -f failsafe.cf). Sure enough, this has confused some new users of CFEngine; some would try to run /var/cfengine/bin/cf-agent --bootstrap on its own, and not get a clear error message on what was wrong. And you could wonder what the result of just /var/cfengine/bin/cf-agent --policy-server would be. This has led to the need to simplify and make the bootstrapping process more consistent. We have now removed the --policy-server option, and we take the IP-address to bootstrap to as a required argument to --bootstrap instead. From now on, bootstrapping CFEngine will entail just running /var/cfengine/bin/cf-agent --bootstrap This will affect CFEngine Community 3.5.0 and CFEngine Enterprise 3.1.0 and later versions. We received feedback from the community that it was desirable with backward-compatibility for such a drastic change, and we tried to retain that. In a transition period, cf-agent will detect most of the old option combinations and rewrite the arguments (see the [pull request](https://github.com/cfengine/core/pull/446)). However, note that this is a heuristic, temporary workaround (we will not detect -BIs, for example) that will go away soon. Zeroconf is supported by passing "--bootstrap :avahi". Please update your auto-installers when the time comes to upgrade to Community 3.5.0 or Enterprise 3.1.0 (e.g. your post-install scripts), and inform your good friends & colleagues about this change\! -------------------------------------------------------------------------------- title: "February 2013 snapshot of CFEngine - get it while it's hot!" url: /blog/2013/february-2013-snapshot-of-cfengine-get-it-while-its-hot/index.md date: "2013-02-27" -------------------------------------------------------------------------------- The February snapshot release of CFEngine Community and Enterprise has just hit our webservers. The releases are called 3.5 Alpha2 and 3.1 Alpha2, respectively, and just as [last month](https://cfengine.com/blog/january-2013-release-now-available): **this is work in progress**. ## Community Those of you who keep track of the commits in our [github repository](https://github.com/cfengine/core/commits/master) or follow us on [twitter](https://twitter.com/cfengine) will have noticed that the code refactoring has continued this month, and that some good things have happened in the wake of this: - cf-promises can now validate partial policies (without bundlesequence) - cf-promises can generate a JSON version of policy, and all other binaries accept .json input, which allows for better testing and tooling integration - body classes have a scope attribute - cf-key has new options for trust management - reports promises are now allowed in any context
As a casualty of the refactoring work, cf-report has for the time being been taken out. To what extent report generation will be part of the 3.5.0 Community release is at this point unclear, your feedback and use-cases for cf-report are very welcome\!
The road ahead promises lots of good things that are enabled by the work so far: we are looking into building an infrastructure for pluggable promises, allowing the capabilities of CFEngine to be extended dynamically (promisers of type user, anyone?). We also want to build debugging mechanisms into the evaluator - starting with callback functions that allow us to hook into the code when classes are set or variable values are changed, we can then build a frontend for stepping through policy code, setting breakpoints and inspecting the current state of the evaluation process. With infrastructure being code and software defined systems gradually replacing hard-wired IT systems, giving infrastructure engineers the same rich tools that software developers have been enjoying for decades seems like a logical step. ## Enterprise The two big areas of activity in February were hub diagnostics, and building an open architecture for knowledge management into the Mission Portal. For better diagnostics of the Enterprise Hub, cf-hub now collects performance and connectivity data for the report collection and database maintenance processes. Together with the collection of this data from the MongoDB storage and through the APIs that provide access to it, users will be able to identify performance trends, make assumptions about scalability trends of the reporting hub. Next step is collecting relevant data from the cf-serverd on the policy hub as well, which then helps understand how the policy distribution scales, and discover whether there are hosts that do not manage to pull down the latest policy updates. In the Mission Portal we have now integrated the plugin architecture, which allows for infrastructure analytics integrations to be created on top of the basic infrastructure of user management and data access. Such plugins can publish and subscribe to callback functions, which allows independent applications to mesh-up and enrich each other's data with context. A plugin that knows about git could enrich policy compliance reports with historical information of the respective lines in the policy code, and an integration with a manually maintained inventory database could compare the information there with the data collected by CFEngine and highlight differences. Through this kind of contextualization, the raw data collected from the system is turned into actionable understanding. Documentation is not yet available, but we hope to be able to open at least some of this infrastructure up to you in a later release. ## Get it while it's hot\! Customers find the packages in a subdirectory "alpha\_releases(unstable)" in their download area on [cfengine.com](https://cfengine.com/ "cfengine.com"). If you are not a customer, but want to evaluate this release, please sign up [here](https://cfengine.com/25free) to download a [free version of CFEngine Enterprise](https://cfengine.com/25free). Once registered, you will have access to the latest stable release as well as to this monthly snapshot release. These free releases can be used to manage up to 25 nodes. Community users can clone our [github repository](https://github.com/cfengine/core) - check out tag 3.5.0a2 - and there are also [packages available here](https://cfengine.com/inside/myspace). A big shout-out to all community contributors - your improvements to the code and to our policies and promise libraries are again much appreciated :) ----- ![tl\_files/cfengine/img\_new/blog/VolkerGrayscale.jpg](images/VolkerGrayscale.jpg) Volker Hilsheimer - I'm a software engineer turned team lead and project manager. I believe that there are universal patterns that we can observe in all systems, be it software, hardware or human, and that the principles of a shared purpose, autonomy of the individual and the desire to achieve mastery give us the energy to make progress. Twitter: @vhilsheimer -------------------------------------------------------------------------------- title: "Using Avahi for automatic configuration of CFEngine clients." url: /blog/2013/using-avahi-for-automatic-configuration-of-cfengine-clients/index.md date: "2013-02-26" -------------------------------------------------------------------------------- Hi everyone, Recently we have been thinking about how to make initial steps with setting up CFEngine easier. One way of making CFEngine easier to set up is to remove the need to manually specify the IP address of the policy hub for bootstrapping a client. We decided to use Zeroconf service discovery implemented in Avahi library. This post will explain how this works. **Registering the Policy Server** First step after setting up a Policy Server is to register it as a remote service using Avahi. All the user has to do is simply to execute cf-serverd with -A/--generate-avahi-config option. This will generate config file *cfengine-hub.service* in */etc/avahi/services* and restart avahi-daemon to make the service visible in the network. **Bootstrapping client** This process is as easy as registering the Policy Server. Simply run cf-agent with -B/--bootstrap option. If no IP address is specified, cf-agent will automatically search for a Policy Server. Once the server registered in the network will be found, cf-agent will bootstrap to it automatically without any further intervention from the user. In case there will be more than one Policy Server registered, list of available servers will be displayed and the user will be asked to bootstrap manually.
**Limitations** Unless reflector mode is enabled in Avahi configuration, automatic bootstrap will be limited to local network. In the current state automatic bootstrap is meant to run only on Linux systems.
**Dynamic libraries** We thought that the user should decide if this functionality will be enable or not. That is why Avahi is not bundled into the CFEngine packages. Instead if Avahi libraries are present in the system they will be automatically loaded during runtime. Cf-agent will look for libraries in standard install locations, depending on Linux distribution and architecture. In case the libraries are installed in different location, the user just needs to create environmental variable AVAHI\_PATH and specify the location of libavahi-client. This variable will be used by cf-agent to load libraries. -------------------------------------------------------------------------------- title: "Libutils: our friendly toolbox" url: /blog/2013/libutils-our-friendly-toolbox/index.md date: "2013-02-15" -------------------------------------------------------------------------------- Hello everybody\! For those of you following our development, you might have noticed that after our code split there is a new friend in town: libutils. The aim of libutils is to be a toolbox of commonly used data structures, algorithms and other handy stuff. Historically we had several implementations of some data structures all over our code, so we decided it was time to start putting all together and make sure we always used the same structures. So far on the data structure side you can find a double linked list, a map, a sequence and a buffer implementation. Our buffer implementation is a glorified byte array that can grow if needed. This allows us to stop worrying about putting too much into a char array. It also contains a memory cap, so it will stop us if we are using too much memory. On top of that, or technically speaking underneath it, it uses implicit sharing and reference counting, so copying it is both extremely fast and economical. While developing this there was the suggestion of using bstring instead. I looked at bstring and I decided against it for two reasons: - The development stopped in 2010. - It does not have reference counting nor implicit sharing. The first point is the most severe. Many will argue that the library reached a maturity where new releases are rare, however in the case of finding issues with it we need something that will react sooner rather than later. The second point is less relevant, although in the long run (and with CFEngine that might mean thousands of hours of running time) it shows its value. Being able to get shallow copies of objects is essential to keep our memory requirements low. On the algorithm side I just finished a proper rfc-compliant IP address parser for both IPV4 and IPV6 addresses. It will show up in our code in the very near future. We will continue adding more stuff into libutils, next in line is a dedicated list for Buffer so don't hesitate to comment or request features. And remember, although it is important to have neat data structures and cool algorithms, these are just the tools that we use to write CFEngine so the real benefit of this can be seen in the better performance and quality of our product\! -------------------------------------------------------------------------------- title: "Validating partial and runnable policy" url: /blog/2013/validating-partial-and-runnable-policy/index.md date: "2013-02-14" -------------------------------------------------------------------------------- Over the past week in CFEngine core dev, some results of refactoring work have come to fruition. We want to make cf-promises a flexible tool for analyzing policy code. Previously, cf-promises required a runnable policy, i.e. a policy that includes a body common control. This made it impractical to use for other policy files than the main file (typically promises.cf), as well as for policy still in development. We want to make it easier for people to use cf-promises while they are developing policy, and to enable the community to integrate with the tool. This week we changed the semantics of cf-promises a bit to enable this. We have split static policy checks into two categories, partial checks and integrity checks. If you give cf-promises a partial policy, it will only apply the partial checks. These include things like syntax checking, duplicate definitions, and other checks. This basically corresponds to the compilation stage in a typical C program. If you give cf-promises a runnable policy, it will additionally apply integrity checks, e.g. checking that statically resolvable body references exist. This can be compared to the linking stage in a C program. Optionally, passing the --full-check flag to cf-promises will ensure that it attempts to apply policy integrity checks. All other agents program use this flag to make sure the policy is validated as runnable before attempting evaluation. Please try it out and let us know your feedback\! ----- Sigurd Teigen I'm a programmer working for CFEngine, currently mainly on the community code base. Let me know what new features you would like to see in CFEngine Community. Twitter: @sigurdteigen ![tl\_files/cfengine/img\_new/blog/sigurd.jpg](images/sigurd.jpg) -------------------------------------------------------------------------------- title: "Upgrading From CFEngine 2 To CFEngine 3 - Webinar" url: /blog/2013/upgrading-from-cfengine-2-to-cfengine-3-webinar/index.md date: "2013-02-13" -------------------------------------------------------------------------------- {{< youtube OSTtcg-OQxc >}} Upgrading from CFEngine 2 to CFEngine 3 is easier than you think! In this Webinar, we will go through the steps required to successfully upgrade your CFEngine 2 environment to CFEngine 3. Topics covered in this Webinar will include: * Promise Theory and Convergence - How they are utilized in CFEngine 3 * New features of the CFEngine 3 Language * Migration strategies * Policy translation - From Files to Bundles and beyond * Validation, Testing and Rollout methods * Q&A Session So, if you are currently running CFEngine 2 in your environment and are considering a move to CFEngine 3, please view this Webinar. (Recorded on Wednesday, February 13, 2013) -------------------------------------------------------------------------------- title: "Why automation leads to greatly reduced IT-costs" url: /blog/2013/why-automation-leads-to-greatly-reduced-it-costs/index.md date: "2013-02-07" -------------------------------------------------------------------------------- At CFEngine we frequently meet companies that save millions of dollars annually as a result of working with our tools. In addition to improved productivity and increased quality of service, automation will make your IT-operations more cost effective. Actually, in order to *stay* competitive, I would argue that highly automated IT-operations is the only way for IT-organizations to meet the agility and cost requirements of today. **Automate or die** IT-operations that are not highly automated will not be able to keep up with expected productivity, quality of service and costs. The industry average for the number of servers managed per system administrator lies between 50 and 100. For companies automating their IT operations with CFEngine that number is much higher. At one company that uses CFEngine, a single system administrator manages more than 10,000 servers and knows he can scale beyond that. Needless to say that everything at that company is in version control and automated - manual changes are strictly forbidden. We see that companies that use CFEngine achieve on average a ratio of 500 to 1,000 servers per system administrator. How can you compete if you need 10 times more people than your competitors, especially when you also know that manual work causes mis-configurations, bottlenecks and added costs? The answer is easy: You can't. **Automate to Reduce IT-Costs** IT-organizations are at risk of becoming obsolete as CMOs and other internal customers use SaaS and PaaS providers instead of internal resources. To get these internal customers back, the first thing needed is to become more productive and cost-effective. Here is an interesting benchmark: **Salary costs for normal IT-operations: $4.5 million: **Assume a company with 5,000 servers that does not use automation to manage its servers. This company needs between 50 and 100 system administrators, we'll use the number 75 for the calculation. Using an average salary for a system administrator of $60,000 gets us to total salary-costs of 75 x $60,000 = $4,500,000. **Salary costs for CFEngine automated IT-operations: $1.8 million **A company that uses CFEngine automation needs 5 to 10 system administrators for 5,000 servers. We'll be conservative and use 10 for this calculation. At CFEngine we call such automation experts [System Engineers instead of System Administrators](https://cfengine.com/blog/from-system-administrator-to-system-engineer "From System Administrator to System Engineer"). They are the "3x-10x-guys", as they often earn 3x the salary of other system administrators and manage 10x their number of servers. For this case the total salary costs to manage 5,000 servers is 10 x $180K = $1,800,000. The conclusions from this calculation are crystal clear. A highly automated organization would save $2.7 million annually in salaries alone compared to a non-automated IT-organization. **The promise of 60% reduction in costs** For companies that migrate from legacy systems, like Bladelogic from BMC or Opsware from HP, we promise the customer a reduction in costs (labor and licenses) of at least 60%. By moving to CFEngine and creating a highly automated environment, most tickets for level 1 and level 2 incidents will go away, and the IT-operations will become much more pro-active focusing on how to improve, rather than fire-fighting. [CFEngine will take care of the fire-fighting for you](https://cfengine.com/blog/continuous-configuration-management-6-billion-checks-a-day "Continuous Configuration Management: the value of 6 billion checks a day"). The best way to stay competitive, is to make operations highly automated. CFEngine enables that and is the only solution that offers continuous maintenance, running billions of checks daily, and immediately correcting deviations of policies regardless of the scale and complexity of the IT infrastructure. We are proud to have produced some of the most impressive metrics in the industry. If you are head of IT-operations or responsible for IT-operations in your organizations, we would love to hear from you, and share what we think you should expect from next generation IT-operation automation. -------------------------------------------------------------------------------- title: "Back from FOSDEM 2013 - Impressions" url: /blog/2013/back-from-fosdem-2013-impressions/index.md date: "2013-02-05" -------------------------------------------------------------------------------- We're just back from [FOSDEM](https://fosdem.org), the Free and Open source Software Developer's Europe Meeting in Brussels. For the second year in a row there was a [DevRoom for Configuration Systems Management](https://fosdem.org/2013/schedule/track/configuration_systems_management/), with room for about 80 people, and long queues outside the door during every talk. Five of us (Alexandra, Geir, Mark, Sigurd and I) went down to the continent to listen to inspiring talks, meet with the infrastructure engineering community and enjoy strong, Belgian beer. Big shout-out to the organizers of FOSDEM in general and the DevRoom in particular, and to the folks at Puppet Labs and Opscode for co-sponsoring the Saturday evening dinner together with us\! The reoccurring themes that I heard this Saturday, brought up more or less explicitly in every talk throughout the day, were not surprising: - infrastructure needs to be testable, self-explanatory code - provisioning tools like [vagrant](https://www.vagrantup.com) have arrived in the mainstream of infrastructure engineering - software-defined infrastructure is a reality What does this mean for CFEngine? If Software Defined Infrastructure becomes the new reality, then we should make the tools that we have enjoyed for years in software engineering also available to infrastructure engineers\! Debugging and [linting](https://en.wikipedia.org/wiki/Lint_%28software%29) configuration policies; automated testing of infrastructure code on a unit, system and acceptance level (Bryan Berry's talk on [Integration Testing](https://speakerdeck.com/bryanwb/dont-shrug-integration-testing-for-infrastructure-with-ruby-frameworks)); integration with [software configuration management](https://en.wikipedia.org/wiki/Software_configuration_management) and [continuous integration](https://martinfowler.com/articles/continuousIntegration.html) tools. And lastly, measuring everything we can, and analyzing the data in the context of the model of the system we want to build and the business goals we have. Our friends at [Normation](https://www.normation.com/) gave a great talk about how they use CFEngine 3 and [Rudder](https://www.rudder.io/) to [keep track of security-oriented rules](https://fosdem.org/2013/schedule/event/automating_security_policies/) not when the auditor is at the door, but every minute\! I was happy to see that this is well aligned with many of our [current activities](https://cfengine.com/blog/january-2013-release-now-available) in CFEngine development\! I'm looking forward to the upcoming DevOpsDays in Europe and to FOSDEM 2014 where we can hopefully show some of the things we are working on right now\! -------------------------------------------------------------------------------- title: "January 2013 Release now Available" url: /blog/2013/january-2013-release-now-available/index.md date: "2013-01-30" -------------------------------------------------------------------------------- Today we released our first monthly release\! The January edition of CFEngine Enterprise and Community is now available to those who want to see what we are currently working on. The releases are called 3.1 alpha1 and 3.5 alpha1, respectively - and as the name suggests, **this is work in progress**. We have built packages for Enterprise and tested a small set of platforms: CentOS 5/6, RHEL 5/6, Ubuntu 10.04. Customers find the packages in a subdirectory "alpha\_releases(unstable)" in their download area on [cfengine.com](https://cfengine.com "cfengine.com"). If you are not a customer, but want to evaluate this release, please sign up [here](https://cfengine.com/25free) to download a [free version of CFEngine Enterprise](https://cfengine.com/25free). Once registered, you will have access to the latest stable release (currently 3.0.0) as well as to this monthly snapshot release. These free releases can be used to manage up to 25 nodes. Community users can clone our [github repository](https://github.com/cfengine/core) - check out tag 3.5.0a1, we don't branch off for these releases. There are also [tar balls with sources](https://cfengine.com/source-code). At this point, a big Thank You\! to all of you who have **contributed with patches** to the Community code and to our policies and promise libraries in the last couple of months\! Thanks for caring about CFEngine and for getting involved. So, what has happened in January? Let's start with... ## Community Those of you who followed the [commit log on github](https://github.com/cfengine/core/commits/master) will have noticed that a lot of structural changes have been made to the CFEngine core code. We have introduced a few implementations of basic data structures, separated the code into agent-specific directories, libpromises and a few utility libraries, and worked on separating the parser from the evaluation process. From a technical perspective this will make the individual components of CFEngine more testable on a unit-level, which allows us to write better code, fix bugs with more confidence, and add shiny new features later on. However, we also see a lot of goodness for our users from this: The new data structures will allow us to get rid of numerous limitations regarding maximum sizes of input buffers and hash tables. Plenty of you have reported bugs and had come up with workaround to those limitations. The risk that through dynamic allocation cf-agent becomes a resource hog is mitigated by giving all these data structures a dynamically specified upper limit. With a separated parser we open up for numerous integration possibilities. Syntax trees can be exported and imported from other formats, allowing significantly improved tooling support. Validation of policies can happen on the fly and without side-effects, which allows us to simplify the policy deployment process and improve the Design Center tools. Implementing new language features, such as support for structured data, becomes easier and less error prone as well. A clearly defined parser process and clearer separation to the evaluator will also allow us to define a plugin interface for promise types in the future. Plugins that implement new promise types need to be able to validate the policy language, add their branches to the syntax tree, and then validate and modify (create, update, delete) the respective objects in the system. Building higher-level abstractions of promise types could become a very powerful concept both in terms of capabilities, and in terms of customization of CFEngine: on embedded or mobile devices we see very different configurables than on servers. A new feature we have implemented in the Core is zeroconf-based discovery of policy hubs, using Avahi. This allows for a simplified bootstrapping process, if Avahi is present on your system. cf-serverd can generate an Avahi configuration file, and cf-agent will use libavahi to detect the policy hub. We detect and load libavahi at runtime (using dlopen), so no additional configuration flags or runtime dependencies are added. This is a first step, we plan to investigate fallback solutions to this that don't have the limitations of Avahi at a later stage. ## Enterprise On **Enterprise**, we have focused on three major projects: Completing data access through SQL reports and Enterprise APIs, collecting diagnostic data about the policy hub, and investigating a modular architecture for the Mission Portal. Via the **SQL Reports** app you have now access to promise logs, software reports, file diffs and compliance data. We have integrated **PCRE support** into SQL, so you can now use regular expressions in your SQL statements. The reports in the Hosts app have been rewritten to use the SQL backend, and will at a later stage probably disappear. In this particular release, the policy context (all promises, user promises, system promises) will not be functional for these reports, it will however come back at a later stage when we have implemented support for this in the SQL Report backend. In addition to collecting data from the hosts, the **Enterprise Hub** now also collects **diagnostic** data from MongoDB into a history of data points, which allows us to identify trends in the database, and correlate changes in the system to what's happening with the hub infrastructure. A next step is to collect similar data for cf-serverd on the hub so that we get information about throughput of data, responsiveness of the server, connection logs and stability, which we can again correlate with changes in the system to learn about the scalability of the existing infrastructure. This is not only useful information for those of you who operate Enterprise hubs, it also allows us to identify bottlenecks in the existing design, and to validate our models when we make future additions, such as building a reporting infrastructure to access aggregate data from multiple hubs. Lastly, we have started to investigate a **modularized** architecture of the **Mission Portal** tool. This allows more flexible development of specialized applications, which can then provide specific knowledge about certain aspects of your system. These applications will be able to create a **mesh of information** through well-defined interfaces so that an app that knows how to interpret the git commit history of your policy files can enrich information provided by a policy browser or host explorer. Observations like "Host as89 is not compliant and has unusually high CPU load ever since user X made commit foo to promises.cf" become possible if apps that know about git, compliance and vital signs start to talk to each other. By making the architecture open to everybody, others could develop additional integrations. These releases are snapshot releases of work in progress. Most things are not finished, some things might not work. **They are not designed for production environments and are not covered by our service level agreements.** However, we hope that we made some of you curious enough about the contents of the packages to give them a go in a test environment, and tell us what you think\! The bug tracker is over [here](https://cfengine.com/dev/projects/core). As mentioned [in my last blog post](https://cfengine.com/blog/more-cfengine-coming-your-way), we plan to have a production release twice a year - the next one before summer. ----- ![tl\_files/cfengine/img\_new/blog/VolkerGrayscale.jpg](images/VolkerGrayscale.jpg)Volker Hilsheimer I'm a software engineer turned team lead and project manager. I believe that there are universal patterns that we can observe in all systems, be it software, hardware or human, and that the principles of a shared purpose, autonomy of the individual and the desire to achieve mastery give us the energy to make progress. Twitter: @vhilsheimer -------------------------------------------------------------------------------- title: "More CFEngine coming your way - every month!" url: /blog/2013/more-cfengine-coming-your-way-every-month/index.md date: "2013-01-21" -------------------------------------------------------------------------------- When we reflected in our team about what worked and what didn't work so well during 2012, one thing stood out: the changes from release to release are too big, and that makes it bloody hard to test that everything works once all the different moving pieces are put together. Core functionality, policy language, upgrade policies, mongodb and apache configurations for enterprise, package names and content - any part moving without the other parts being adjusted will quickly lead to a frustrating experience for everybody. ## Small, incremental changes that converge towards a desired state Sounds familiar? Well, there's a reason why cf-agent is designed to run every 5 minutes on your system. Systems drift, and without verifying the status and making adjustments frequently, that drift multiplies, amplifies and quickly infects the whole system until it reaches a point where change is nearly impossible and where fixing the unwanted side-effect of any change becomes as expensive as implementing the changes in the first place. The result is that releasing new stuff becomes the most dreaded and risky activity in an organization, and that innovation comes to a stand-still. We see that in IT and in software engineering alike - and it is an organizational as much as it is a technical problem: releasing software is the most important activity of any software company - every hour that went into writing a magnificent new feature is of no value until users can get their hands on it - and if that activity is painful, then it will only be fixed if everybody feels that pain constantly. So, given that in CFEngine we truly believe in the value of agile system management and continuous self-healing of systems (and we, the engineers, are part of that system), it wasn't particularly difficult to agree that we need to release our software system more frequently and with a smaller set of changes. Those changes will be fewer and more testable on a system level - we already have continuous integration, testing and packaging to a staging area. However, by executing the whole release pipeline frequently we can make sure that separate but inter-dependent parts of the system don't drift apart from each other without anyone noticing. A big part of this is not just about shipping code, but about capturing and updating knowledge into a form that makes it accessible to all stakeholders, in- and outside the company. ## Monthly release of CFEngine "Pah", you might say, "What's the big deal; I'm running [continuous delivery](https://continuousdelivery.com) and release new software into production every 15 seconds\!". Well, we don't believe that we would do our users much of a favor if we were to roll out a new version of CFEngine every day or every week - CFEngine plays a fundamental role in many IT systems, and no matter how much trust you have in our ability to do a good job in building rock-solid CFEngine releases, we don't think it's very attractive for anyone to look at CFEngine as a rapidly changing and moving piece of technology. Starting with January 30th, we will release a tested snapshot of [CFEngine Enterprise](https://cfengine.com/enterprise) and [Community](https://cfengine.com/community) every last Wednesday of the month. These monthly releases will be designed to be technology previews, not stable and fully supported releases for a production environment. Every month will be organized around - a week of bug fixing - two weeks of development; bug fixes will be released as maintenance release (ie 3.0.x) - a week of testing, documentation updates and release execution These snapshots will be made available to all of you (packaged for a subset of our supported platforms) so that you can play around with the latest and tested improvements of both Community and Enterprise in a test environment, without having to pull the latest code from the [github](https://github.com/cfengine) repository and building it yourself. We are not quite sure yet how we will label those releases - we will probably call them "technology preview" or "alpha" releases. Product-quality releases that are covered by our SLA and designed for release into production will then be launched in larger intervals. We think that we will have two such supported releases every year, which follows the rhythm that we had in 2012. These will be schedule-driven releases; work that is not completed when the release is built will be left out of the release - either not integrated in the first place, or disabled/hidden. Let us know in the comments below what you think about this\! In future blogs tagged with *Development* we plan to share some more information about what our own continuous integration and deployment pipeline looks like, and what role CFEngine plays in current and future versions of that infrastructure. ----- ### Volker Hilsheimer ![tl\_files/cfengine/img\_new/blog/VolkerGrayscale.jpg](images/VolkerGrayscale.jpg)I'm a software engineer turned team lead and project manager. I believe that there are universal patterns that we can observe in all systems, be it software, hardware or human, and that the principles of a shared purpose, autonomy of the individual and the desire to achieve mastery give us the energy to make progress. Twitter: @vhilsheimer -------------------------------------------------------------------------------- title: "Welcome to the CFEngine Development Blog" url: /blog/2013/welcome-to-the-cfengine-development-blog/index.md date: "2013-01-18" -------------------------------------------------------------------------------- Happy New Year 2013, and welcome to the CFEngine Developer Blog\! On posts tagged with ***Development***, the guys and gals working on the CFEngine code will share their thoughts about current challenges, ongoing improvements and completed features. The nature of this information will by definition be technical and at times more interesting for those with a software development background than for [system engineers](https://cfengine.com/blog/from-system-administrator-to-system-engineer). This will be news from the "bleeding edge" rather than announcements of production-ready features - more on that, and how we are changing our release operations in 2013, in a separate post\! This is going to be both about CFEngine Community and [CFEngine Enterprise](https://cfengine.com/enterprise). The Community edition is already developed in public in our [github repository](https://github.com/cfengine/core), and we are very happy to see an increasing number of [pull-requests](https://github.com/cfengine/core/pulls) submitted from CFEngine users and coders in the community\! The blog posts here will help connect the changes in the code to the general direction that we are taking with the development of CFEngine. The Enterprise-specific functionality in CFEngine is not developed under an Open Source model, and I hope that our customers and users of the [Free25 edition](https://cfengine.com/25free) will find the information we share here helpful to understand what we are working on - and why\! - to plan ahead and to share their thoughts. So, I'm looking forward to a continuous stream of information and good discussions, and I'd like to invite you all to participate in the design of the future of CFEngine. My name is Volker Hilsheimer, and I'm responsible for the development activities in CFEngine. Feel free to contact me on twitter via @vhilsheimer if you want to get in touch with me directly\! -------------------------------------------------------------------------------- title: "CFEngine and the future of monitoring" url: /blog/2012/cfengine-and-the-future-of-monitoring/index.md date: "2012-12-28" --------------------------------------------------------------------------------
![](images/monitoring-blog-ff26a6c2.jpg)
*Since writing my earlier post on ([Model based monitoring](https://cfengine.com/blog/model-based-monitoring-with-cfengine)), I have talked to many users who encouraged me to describe CFEngine's simple capabilities in more detail. Although CFEngine is not intended as a traditional monitoring platform, it offers a considerable amount of human-friendly information, with a model that could be a hint of the future. At CFEngine, we like to innovate, and this post offers some hints about how we are thinking.* \* At what point did IT-monitoring lose its way? For me, the answer is when it forgot to think about its audience. Present day monitoring systems are typically mere data streaming appliances. They do little in the way of interpretation or analysis. Even with an increasing number of smart graphing tools, all the work of understanding and interpretation is left up to the user. That makes it very hard to answer concrete questions about either infrastructure or application. In the monitoring of complex systems, there is more than one kind of audience -- not everyone should be looking at the same data. Forensic investigation of system behaviour and scientific analysis that looks for meaning are very different activities than needing to know your speed and direction, or whether you are floating or sinking. Most tools can't decouple these -- so they waste users' time with too much or too little data. Scientists look for data. Operations are looking for contextualized information. Some monitoring is informational (for human analysis) and some is actionable (either by human or machine). CFEngine ensures that information can be acted upon as quickly as possible and that human time is not wasted by exposing users to raw data that is not part of a model. What are the key day-to-day wishes and questions we should be asking about monitoring? Here are some examples: - Tell me about the status and current promise(s) the system should keep. - Tell me about security, performance, compliance, etc. - Where is the greatest/least activity? - How much spare capacity do I have? - Where are things changing fastest? - When do I need to think about X? - What do I need to know about X? - What are the most important things that happened to X? - Why is this object/rule here? - What can I do about X? - Who changed X last? - When can I expect X to happen? - Where can I find X? - What things affect X? - What promises have been made about X? - What stories lead to conclusion X? - What patterns can we see that might lead to efficiencies of scale? - And so on ... Most monitoring systems cannot begin to answer these questions without a lot of manual work from users, because they are not in any way related to configuration systems that define the purpose and desired state of the system. This is where CFEngine has been innovative. In CFEngine, we only need one model -- because each configuration item implies automated monitoring, In the rest of this essay, I sketch out some notes about the directions and dead ends I see in the story of IT monitoring, viewed from my perspective as a long-time researcher and practitioner. I have no doubt that this will provoke mixed reactions, so let's get down to it. ## Model based monitoring Present day data streaming tools know so little about the data they are streaming that the best they can do is to report thresholds. Unfortunately thresholds have little long-term meaning. What is acceptable today, might be quite unacceptable tomorrow. Context is key. We need a more extensive system measurement philosophy if we want to understand the phenomena at work. Most monitoring collects a raw snapshot of the current actual state. Some can also record history and calculate statistics. CFEngine belongs to the few systems that use machine learning to compute averages, deviations, trends that allow users to see what is now compared to what came before. Moreover, CFEngine is probably unique in automatically building model-based results from system design plans, giving the ability to predict/detect anomalies. In CFEngine, the model comes from policy itself. It can do this because all infrastructure design is boiled down to an \`assembly language' of atomic promises.
There are many areas for improvement.

Threshold monitoring, watermarks, and aggregate notifications around these. Look at correlations. How do we divide the world between infrastructure and applications in a meaningful way? Then we need to link the information to a description of the model of our understanding that bridges intent and feedback. This is part of CFEngine's Knowledge Manifesto.

## Data, information and knowledge We can't expect users to be scientists, and we should not encourage them to stare at data they have no context for understanding, hoping to glean some meaning from traces. That is a job for a specialist. An inexperienced user might watch a value wanting to see if it will change, without realizing that it can never change for the next hour. This is both stressful and time-wasting. Is "big data" the answer? Well, the question is silly. Is it the answer to *what question*? Collecting every clue might get you a spot on CSI, but it won't help the average engineer to do anything except waste time sifting for gold (assuming he/she even knows how to recognize gold). To analyze big data requires specialized skills. For day to day operations, you want to minimize the amount of searching and re-thinking, not increase it. So "big data" is a dubious strategy for anyone but a system architect. ## Metrics for business and pleasure If we take seriously the *purpose*, or intent of the IT system, then this has to be reflected in what we try to measure. It is fruitless watching the system, hoping to see relevant phenomena, without any particular expectations. This is where *human need* must inform the machine. It might still be useful to collect data over time so that a suitable \`data scientist' can analyze them with a view to building such an understanding, perhaps for forensic analysis. The key thing lacking in monitoring is context (i.e. a model) that allows us to understand what we are seeing. We need to be able to group measurements according to their semantics, i.e. what they mean for the intended purpose of the system. Currently, CFEngine's promises are mostly about configuration. In Enterprise, \`measurements' promises can also define the health of systems. This can be modelled both for business relevance (e.g. maximum acceptable response time for a sales query) or more elemental items like \`at least 10G available storage'. We have also begun to model services and high level business objectives (goals), from component services (e.g. web farm) to an end-to-end business service (web farm, db cluster, web services, etc...). Ideally, all of these metrics would be tied to automated responses, that respect the human in the loop. | | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | For example consider the difference between a business measurement and a system one. You might have 100 business transactions per second (length 10ms), but the system is throttled by 5 minute TCP\_FINWAIT, so measurement correlations are at least 5 mins long. A novice should not need to know such technicalities to derive meaning from a system portal. | The idea of monitoring everything \`just in case' is a sign of hedging, which suggests a lack of understanding of the environment. That is normal in the beginning, of course, but if your system is reasonably under control it is possible to form a model for it that leads to significant simplifications (cost savings of human hours). | | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | At a recent customer visit, an operations centre was paying 40 persons to watch \`big screens' around the clock, just in the hope of seeing something unusual. Not only are humans easily de-sensitized to micro-anomalies, they tend to forget what they are looking for because the task is boring. | We need to separate metrics into those where we have a model for their behaviour, and those whose behaviour we really don't understand. Measures can probably be further divided in the three main groups: business-result related, platform-performance related, environmental \`weather' checks for calibration. These are the major separations that we hope to achieve in running a system. If there is no clean separation between these areas, then the system may be called "complex". What we hope to achieve with such measurements includes: - Capacity planning. - Cause-effect inference. - Performance tuning. - Fault diagnosis. - Drift assessment. Finally, different consumers of the information will want to organize their metrics on customized dashboards in a simple way. We need to get beyond trying to watch everything at the same time, to having confidence in model-based summaries. ## Data collection and aggregation How we collect data is important. A distributed process allows us to minimize the impact on resources, but we also need to aggregate data in order to compare results on a calibrated scale for immediate action, observe correlations etc. To do this efficiently, it makes sense to filter out irrelevant noise at the source, rather than burdening future searches. Collection can be optimized by a pull process, to keep network traffic predictable. In push-based notification systems (like triggered SNMP) we can end up with notification storms. Every dynamical process has a natural timescale; so, of course, it makes no sense to sample it faster than this limit, and research shows that many values change more slowly than users expect, while others exhibit meaningless noise that one should not be concerned about. Left to themselves, users will generally oversample out of uncertainty. This adds only cost without value. In CFEngine, measurement promises allow users to sample data at an appropriate timescale, up to a maximum of once every 2.5 minutes. That captures most use-cases
What can we improve?

We still need to make it easier to identify what the appropriate sampling rate is. Automated autocorrelation analysis of time-series data to help optimize sampling overhead. We can make improvements to the management of the data process, not only the analysis. Also, integration of data-feeds from other tools and APIs would allow us to "Bridge the gap between business and IT".

## Machine learning -- trends and decision loops We can relieve a lot of human burden by using unsupervised learning to detect context, time-scales and to summarize the main features of a change process. CFEngine has already been a pioneer in this area in the 1990s, and RRDtool has added some features to support this too. More science can be applied here. Techniques like Principal Component Analysis can help us to see large numbers of hosts through different lenses. ## Representing resource pools With the rise in importance of the web as a front end to almost everything, web-scaling increasingly dominates resource models. We care less about individual hosts and more about resource pools or clusters. CFEngine naturally allows us to group hosts by their role in the system, and in the Enterprise portal we are making it easier to view such services as services, measuring Quality of Service (QoS) as has been common in network management for years. We can use that in a variety of ways to represent information at the level of the relevant virtual container for that function. Creating more virtual container concepts that allow us to handle clusters and resource pools more easily will bring monitoring forward in this area. ## Understanding growth Given that IT is currently dominated by growth, one of the important measurements is the number of machines we operate, and size of resources at a given moment. We would expect this to correlate with the operational signals reaching individual hosts in some way. Current tools do not permit this kind of holistic correlation: measuring the number of machines (in various states) as a function of time, so that we can correlate other changes with the installation or retirement of machines. The total number of machines can affect any individual machine through shared resources. ## Access to data, notification and escalation mechanisms We are currently in a transition period where we are exploring the composition of different kinds of system, and there is therefore a love affair with APIs. "Just give us a REST API and we will build the world anew\!" But, I believe this is a fleeting trend that has to pass. Imagine if we treated other infrastructure in the same way: "Today everyone is a carpenter, here are the tools -- now build more desks, we need to expand\!" Such an organization would be seen as immature and even irresponsible. So we want to get users beyond raw data feeds, and toolboxes, to commodity contextualized information, backed up by a local knowledge of its meaning. What happens when something out of the ordinary happens, i.e. an anomaly that requires special attention? Summary notifications help\! Users should not have to look at a big screen to spot important events in a sea of change -- nor should there be an air traffic or mission control room required for the normal functioning of a system. That was the best technology available in the 1960s. Facebook is an impressive tool for displaying notifications and news about large numbers of relationships. A \`facebook' (hostbook) for hosts and services would allow us to maintain a close relationship with key favourites. Intelligent notifications of events we care about is what Facebook does well, and we can learn from this.
Where is this going?

Monitoring is at the interaction between humans and machine. So we create a process around human continuous situational awareness, that facilitates investigation for deep analysis, but mostly automates the analysis for broad consumption.

It can also be the basis of an automated orchestration of responses for system regulation. Innovative companies like Amazon have been managing costs in this way, using their elastic scaling capabilities for years already. The rest of the world still has some way to catch up.

## Summary/Postscript Ask yourself this. What is the purpose of monitoring? Is it a vanity mirror for looking at beautiful systems? Is it an alarm for fire fighting, or a crystal ball for strategic planning? Probably, it needs to be all of the above. But, above all, what it needs to be is something that helps human involvement and doesn't waste our time. To make CFEngine into a credible part of a monitoring infrastructure (for simple and complex needs), we are starting with a basic core of classical approaches to representing monitoring data -- a handle for people to latch onto and build a more modern view on top of. Users need to be able to drill down from any starting point to find *useful reasoning* (not just more data) around the condition of the system. This will build confidence in the information as a source of intelligence. We hope that others in the industry will continue to follow our thinking in infrastructure engineering. It is time for the industry to move forward. #### Acknowledgement Over the past year, I have benefitted from discussions with (in no particular order) Jason Dixon, Sal Jamil, Gleico Moraes and Juliano Martinez, John Willis, Damon Edwards, Patrick Debois, Kris Buytaert, John Allspaw, Arjan Eriks, Dan Klein, Kent Skaar and Reynold Jabbour. None of these gentlemen should be assumed to endorse the content above. Apologies if I have forgotten to mention anyone. -------------------------------------------------------------------------------- title: "From System Administrator to System Engineer" url: /blog/2012/from-system-administrator-to-system-engineer/index.md date: "2012-12-10" --------------------------------------------------------------------------------
![](images/team-work-e3f33223.png)
Automation of IT-operations can lead to fantastic productivity gains, increased quality of service and reduced operational costs. But what about the people, and their jobs? System Administrators will not become obsolete, but the nature of their work often changes in highly automated environments. The ones who adapt typically enter into more proactive roles. The ones who willingly or unwillingly are left behind, end up fighting fires and home made scripts, until they will eventually be replaced. This evolution should not be viewed as a threat, but as an opportunity. **Cultural DevOps changes on the verge** The craftsmanship involved in IT-operations is changing. The emergence of DevOps leads operations people to embracing change. Monologues (people telling each other what to do) turn into dialogues and both physical and virtual walls between Developers and Operations are diminishing. The strong DevOps wave will result in changed behavior in thousands of IT-organizations around the world. **Powerful automation tools on the verge** In addition to the cultural change we now see, next generation automation tools, like CFEngine, empower operations people to achieve operational metrics surpassing past performance significantly. New solutions allow for more proactive ways of working. The division of labor between human and machine, where the humans make decisions and machines execute, shape the most successful IT-operation companies today. In these companies we see a ratio of one System Administrator per 500 servers or more, daily and even hourly deployment rates, combined with 99.999% uptime. These powerful automation tools are hitting the mass-market and thereby become a natural tool in the toolbox of IT-Operations. **The dawn of System Engineers** System Administrators administrate systems, much like the way managers were administrating factory-workers up until the 80s. Operational improvements in the manufacturing industry lead to drastic changes. Factory-workers were replaced by machines, and administrators became specialized engineers whose main focus became to optimize the machines who did the actual work. The same evolution happens in IT-operations these days. Manual tasks done by System Administrators are slowly becoming obsolete and replaced by powerful software automation. The new breed of System Administrators focuses on efficiency (doing the right things) through dialogues with their peers (DevOps), and effectiveness (doing the things right), by focusing on improving their already powerful automation solution. The nature of this job looks more like engineering than administration. In CFEngine we embrace the new role. We think that the job-title "System Engineer" better conveys the description of the next generation of IT-operations professionals. We see examples of companies where, due to CFEngine and self-service, the need for SysAdmins to deal with level 1 and level 2 support-tickets has been vastly reduced. The new role of the System Administrator has transitioned into how to improve the system (through dialogue and automaton) as opposed to how to fix it. CFEngine will fix the system for you. An example: At a large-scale installation (5,000 servers), CFEngine normally runs more than 5 billion checks a day to ensure the system is compliant and thereby avoids the majority of level 1 and level 2 tickets (see: [The Value of 6 Billion Checks a Day](https://cfengine.com/blog/continuous-configuration-management-6-billion-checks-a-day)). CFEngine fixes several thousands non-compliant and unauthorized changes daily, work that otherwise would have been done by System Administrators. **Great opportunities are awaiting all System Administrators** System Administrators face great opportunities. The ones who embrace a high degree of automation will become critical to the companies they work for and will create significant business value. What is extra cool, is that this life can be a pretty nice one. One of the engineers we know who is responsible for thousands of servers at Linkedin, moved to Hawaii because he loves surfing. At Locaweb, the biggest cloud provider in South America, who is highly automated with CFEngine, one of their System Engineers, has a blog called "[sysadm must be lazy](https://web.archive.org/web/20141217121121/http://martinez.io/)". I think these two examples are great evidence of the cool attitudes and the exciting possibilities waiting ahead for the System Administrators who want to embrace automation and become Engineers. No one should think of automation as a threat, but rather as the biggest opportunity in a long time to really make a difference and add value to one's company. Companies hire people because they will add business value. More than ever, System Engineers will bring significant value to the companies they work for. If you become one, you will get exciting challenges, experience more credibility and a higher salary. Great System Engineers we come across often earn 2-3x the average salary of a System Administrator, and they get way fewer wake-up calls in the middle of the night, or like our friend at Locaweb puts it: "My life before CFEngine was hell. I could get up to 9 calls during one night. This has all ended now". **Start today** If you would like to learn more, and even talk to some of the System Engineers that have already come a long way, please [let us know](https://cfengine.com/contact) -------------------------------------------------------------------------------- title: "Debunking 5 Myths About Implementing Configuration Management" url: /blog/2012/debunking-5-myths-about-implementing-configuration-management/index.md date: "2012-11-28" -------------------------------------------------------------------------------- *This post originally appeared on [Dyn.com](https://www.dyn.com), written by Neil Schelly [@neilschelly](https://twitter.com/neilschelly)* We've recently made some big strides at Dyn in implementing a more modern configuration management platform (CFEngine 3) to replace an internally developed system that wasn't meeting our needs anymore. Reading about the options out there inevitably leads you to see comparisons of many tools to fill this need. I found myself also learning about several myths regarding implementing configuration management. In our case, some of these were the difference between a successful deployment and an abandoned effort. ## Myth 1: Servers are disposable. Rebuild them if they break. Reading best practices of configuration management, I got the impression that this is used as a crutch for when your configuration adjustments break existing systems. We'd prefer not to have to fix broken machines by rebuilding them from blank installs and I don't think we're alone in that thinking. I suspect most system administrators would rather have the existing machine continue to work than deploy a new one. With particular system architectures and the modern capabilities of virtual machines, servers may be more disposable than ever, but that doesn't mean they have to be. Some of our machines are a little too reliant on the hardware performance they get from dedicated storage, CPU time or dedicated network interface cards. Redeploying hardware machines through our distributed infrastructure from scratch isn't a trivial problem to resolve. It's also a problem that rarely comes up on its own. The promise/action arrangement of CFEngine configurations works well for us in this regard. We can be sure that the promises we make with CFEngine should always come out true, as well as the atomic operations to correct them shouldn't fail on any machine because of how atomic the promises are in the first place. ## Myth 2: Configuration Management is an all or nothing endeavor. The way we've arranged our promises in CFEngine has allowed us to overlay centrally controlled configurations on top of existing systems in production usage. We were able to build the promises/actions for managing users, then the promises/actions for managing groups, etc. We proceeded through each of these "features" that we needed one at a time. This approach just means that as one feature in our old configuration management tool is re-engineered, we can remove it from the old tool and let CFEngine handle it from here on out. A feature can be simple, like distribution of internally-recognized SSL certificate files or adding a note to the MOTD seen upon login. A feature can also be complex, like the deployment of one of our application installations, requiring installation of system packages, files deployed from our codebase, users and groups created, etc. The important part is that as you build the features you want to manage into the new configuration management solution, you can gradually move from your existing configuration management tool to your new tool. Building your rules this way also ensures that you know they'll work when you deploy them on machines in different stages of convergence. ## Myth 3: Constant updating is necessary to keep all your systems in sync. Out of the box, most modern configuration management systems will aim to constantly update your system to reach a state of eventual convergence with your configuration. CFEngine isn't any different in this regard, but that's a troubling scenario when you're overlaying configuration management onto existing boxes. When you ask about dry-run or reporting-only modes, you'll likely find some friction with best practices for modern configuration management. You're supposed to just trust the configuration management tool to accomplish what you've set it to do. You inevitably have to ask the question about what happens when you need to make a change to a system that is having trouble communicating with the central configuration authority for one reason or another. We've found that with CFEngine, we can assuage our discomfort with this concept using its dry-run modes. We can make local changes to a system if the need arises, such as during a DDoS mitigation when every second in the response time counts and central configuration may not be feasible. If we do, our configuration management systems will now start screaming for attention by alerting us to unkept promises, but it won't undo our work to mitigate an attack automatically. Once the incident requiring those manual changes is over, we can worry about making sure systems are in sync with configuration management again. Most importantly, we won't forget to do that because the systems will make it clear that they need attention as soon as we can focus on them again. This may not apply to other infrastructures, but in our case, it's important to recognize that configuration of our product is adjusting itself all the time. System configurations outside of our application don't actually change that much. This kind of reporting-only approach to configuration management works very well for our operations. ## Myth 4: Pick a tool and just start using it. This is great advice for getting started and getting familiar with configuration management, but lousy advice to get started really using it. The principle is that the strengths and weaknesses of each tool will balance out and everything will work out once you get comfortable with the tool you've chosen. The better message is that using any configuration management tool is better than using none. I'd be hard-pressed to disagree with that. I prefer to think it's best to pick the problem that is the largest pain point for your existing configuration methodology and then try to implement that in all the configuration management solutions you're looking to take for a test drive. Try it in each of the operating systems and versions you're trying to manage. The aim here: be sure that things that work great for you on a Linux box don't run into quirky side-effects on an OpenBSD box. Don't focus on building a whole system; build a whole feature instead. In our case, that first feature was user and group management where Dyn's amount of staff has scaled much faster than our ability to manage it in our old home-grown configuration management solution. Make sure you can manage that feature in each of the systems you're considering migrating to or else you're going to quickly see an abandoned project. Why would you spend so much effort building a new configuration management solution that will run into the same problems you're already coping with now? The best part of this approach: as you begin to deploy your new configuration management solution, it's already solving pain points in the old system\! ## Myth 5: This isn't scary. Just do it. Actually, when you tell your new configuration management system to "Go" in your production environment for the first time, it is kind of scary. It doesn't matter how long you've been testing, how many scenarios for failure you've planned for or how good your reverting strategy is. Ignoring that bit, just do it anyway. It is worth it. **Neil Schelly is the Senior System Administrator for Dyn, the IaaS (Infrastructure-as-a-Service) leader that specializes in DNS and Email Delivery services. Follow at Twitter: [@neilschelly](https://twitter.com/neilschelly).** -------------------------------------------------------------------------------- title: "Continuous Configuration Management: the value of 6 billion checks a day" url: /blog/2012/continuous-configuration-management-the-value-of-6-billion-checks-a-day/index.md date: "2012-11-15" -------------------------------------------------------------------------------- Software developers know that quality of software projects tends to deteriorate over time unless strong measures are taken to prevent this. Software entropy accelerates once the beginning of "software rot" has been allowed to set in, so the trick is to keep the software as clean as possible at all times. This is referred to as "[The Broken Windows Theory](https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/)" because the pattern is similar to what police departments learned about maintaining order in inner cities: fix the small things all the time, and so keep out the big problems. If this pattern applies to law enforcement and software development, does it also apply to Configuration Management? At CFEngine we believe it does, and we design our software to support continuous verification of the state of IT infrastructure and immediate repair if systems drift from their desired state. To give an idea of the scale of this approach, let's assume that we want to verify the settings of SSH across Linux servers. There are some 65 configuration settings that need to be decided and maintained. If they are deployed to 1,000 servers, then there are 65,000 checks to be made, just to verify the state of the infrastructure once. The default for CFEngine is that we check once per 5 minutes, 288x per day. In this example that means we do close to 20 million SSH checks per day. There are many reasons why making checks this frequently makes sense (see Mark Burgess' "[Ten Reasons for 5-minute configuration update and repair](https://cfengine.com/blog/ten-reasons-for-5-minute-configuration-repair)"). I received a very compelling argument from one of our customers, who reports that they run over 6 billion configuration checks per day, catching and automatically repairing more than 100,000 unplanned changes during a workday\! Without these repairs an average of 5% of their servers would drift out of compliance every day. At this level of granularity the repairs are easy to make automatically, and their cost is extremely low. When left to accumulate, these repairs are harder to make, problem cause analysis becomes harder, and the impact of the drift becomes large: 70-90% of all IT outages are due to manual and unauthorized changes. If people are required to make these repairs, it is not only costly but also very error prone. Our customer reports very low rates of outages and the ability to apply changes and introduce new capabilities at high frequency. They are confident about their infrastructure as a business enabler. Most agile IT-organizations today have successfully implemented the concepts of continuous integration and continuous delivery to control quality and reduce the time to market. They have automated the build, test and delivery of new software. To close the loop, it is time for the operations side of the IT-organization to get into the mindset of automation and "continuousness". Bleeding edge IT-organizations like the one mentioned above have already proven the model. As we go through the technology adoption cycle, we should expect a rapid increase in the uptake of continuous configuration management through automation. -------------------------------------------------------------------------------- title: "The Guru is in Session at LISA 2012" url: /blog/2012/the-guru-is-in-session-at-lisa-2012/index.md date: "2012-11-14" -------------------------------------------------------------------------------- [![LISA'12](images/lisa12_button_125x125.png)](https://www.usenix.org/conference/lisa12) Do you want to learn more about CFEngine, and hear about the latest developments, tools, tricks and tips? Please join us for "The Guru is in: CFEngine" at [LISA'12](https://www.usenix.org/conference/lisa12/tech-schedule/technical-sessions), a session where you will hear directly from CFEngine developers, engineers and power users. Some of the topics that we will discuss include the latest features in recent CFEngine releases, the CFEngine Design Center, and using Vagrant with CFEngine. Bring your own questions and topic suggestions too\! Whether you are currently using CFEngine or just want to learn more about it, whether you are a newbie or an expert, we are positive you will find this session informative and interesting. The date is Friday, December 14th, from 9:00-10:30AM (subject to change, please check the latest LISA schedule for the definite information). -------------------------------------------------------------------------------- title: "QNAP and Raspberry Pi Support" url: /blog/2012/qnap-and-raspberry-pi-support/index.md date: "2012-11-09" -------------------------------------------------------------------------------- ![raspberry-logo](images/raspberry-logo.png) Earlier this year we made several proof of concepts to demonstrate that CFEngine can run on embedded and mobile devices. We compiled CFEngine for a devkit 8500 board, Android, QNAP and Raspberry Pi. Due to popular demand we have made the QNAP and Raspberry Pi packages publicly available, free to use for up to 25 nodes. This is an alpha version of Enterprise 2.2 compiled for these platforms, please be aware that we do not provide support for these packages. Official versions may be provided at a later point. Download them at [https://cfengine.com/download](https://cfengine.com/download "Free download") -------------------------------------------------------------------------------- title: "CFEngine, virtualization and VMware vSphere 5 (ESXi)" url: /blog/2012/cfengine-virtualization-and-vmware-vsphere-5-esxi/index.md date: "2012-08-29" --------------------------------------------------------------------------------
![](images/vsphere5-b9f5ede6.png)
In today's environment where words such as virtualization, cloud and agility constantly buzz in our ears, enterprises look for ways to manage their increasingly dynamic infrastructure. The [Cloud::Services sketch](https://github.com/cfengine/design-center/tree/master/sketches/cloud/cloud_services "CFEngine Design Center") from the CFEngine Design Center implements a policy to manage virtual machines (VMs) on VMware vSphere 5 (ESXi). With this sketch, CFEngine can clone, start/stop, delete and configure VMs seamlessly alongside your physical infrastructure. The [Cloud::Services sketch](https://github.com/cfengine/design-center/tree/master/sketches/cloud/cloud_services "CFEngine Design Center") uses two of VMware's command line APIs, vCLI and vifs, to interface with the ESXi hosts and its VMs. This allows us to create fully working resources from an uninitiated state in matters of seconds or minutes. We treat the VMs as services and make sure the correct packages are installed to fulfill the requested tasks (web server, database server, load manager, etc). CFEngine's convergent model assures that your infrastructure is always in the desired state. If a virtual machine hangs for some reason, CFEngine will bring it back online or, alternatively, fire up a backup VM if that is what you want it to do. Not only does this give redundancy, it also increases the security aspect of your infrastructure. A tripwire on a VM could for instance initiate a process to take it offline until you can diagnose the issue, while another VM boots up to take over the original task. If a VM is deleted by accident, CFEngine will recreate it and configure it to serve the same service. CFEngine in combination with virtualization platforms and cloud allows for great flexibility with little effort. Although this brief article treats integration with VMware, the [Cloud::Services sketch](https://github.com/cfengine/design-center/tree/master/sketches/cloud/cloud_services "CFEngine Design Center") has similar capabilities for Amazon EC2. This can be extended to any virtualization or cloud platform with the appropriate APIs. CFEngine and Vagrant is another example. The ability to run on almost any platform, the fact that our software is lightweight and uses less memory and CPU cycles compared to our competitors make us confident that CFEngine is among the best tools for managing virtual resources. As many others, we believe that the future lies in mobile devices linked to virtualized services; with CFEngine you could manage almost every aspect of this infrastructure... Exciting indeed\! [Watch the demo video »](https://cfengine.com/demos/cfengine-vmware "CFEngine Integration with VMware") -------------------------------------------------------------------------------- title: "Using Splunk to analyze CFEngine logs" url: /blog/2012/using-splunk-to-analyze-cfengine-logs/index.md date: "2012-07-18" --------------------------------------------------------------------------------
![](images/splunk-blog-post-7a62b798.jpg)
## 0\. Why? CFEngine Enterprise collects very detailed, real-time information about the configuration of your IT Infrastructure. Splunk is an excellent enterprise search engine, engineered for speed, robustness, and scalability. You will learn how analyzing CFEngine logs with Splunk lets you see CFEngine data in new ways. You will learn how to access CFEngine's logs, what information they capture and how Splunk's "rex" command and other neat tools make that information accessible. As you'll see from this guide, analyzing CFEngine logs with Splunk lets you see your data in new ways as well. You will learn about Splunk's "rex" command and many other neat tools. We'll use CFEngine Enterprise in these examples but you can easily adapt them to CFEngine Community yourself or look at [https://cfengine.com/manuals\_files/SpecialTopic\_Reporting.pdf](https://cfengine.com/manuals_files/SpecialTopic_Reporting.pdf "Special Topics: Reporting") for ideas and inspiration. ## 1\. Install You need the Splunk server installed on an indexer machine. For small installations this is also the search server. We'll call this machine "indexer01" in this guide. All the work in this section will be done on a forwarder machine called "this-host-name", which sends all the log data to indexer01 but doesn't do any searching. Typically you'll have the lightweight Splunk forwarder installed in /opt/splunkforwarder. You'll probably have /opt/splunkforwarder/etc/system/local/inputs.conf looking like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[default]
host = this-host-name
# typical, but not required for this example
[monitor:///var/log]
_TCP_ROUTING = *
index = main
# everything under /var/cfengine will be monitored
[monitor:///var/cfengine]
_TCP_ROUTING = *
index = main
This is pretty simple so far. The /opt/splunkforwarder/etc/system/local/outputs.conf file is not much worse:
1
2
3
4
5
6
7
8
9
10
11
12
13
[tcpout]
defaultGroup = lan_receivers
[tcpout:lan_receivers]
autoLB = true
compressed = true
server = indexer01:8997
Note that both inputs.conf and outputs.conf are very bare. You should consult with your local Splunk administrator if you suspect you may have the Splunk forwarder already installed. ## 2\. Verify The first search you will do checks that you're finding CFEngine logs on a specific host: host="this-host-name" source=/var/cfengine/\*/cf\_\*.log Limit it to the last 24 hours. You should see at least a couple of lines. If you don't, check that the forwarder is working. The file /opt/splunkforwarder/var/log/splunkd.conf may be useful for debugging. From now on we won't specify the host name, so you'll be searching across your whole environment. See the wildcards "\*"? Those ensure we'll see logs that are in different directories too. Later searches will filter further to ensure only matching data is found, but using the source search parameter will at least speed up the search by limiting it to a few files instead of everything in the index. You could also set up a separate index for CFEngine logs, but that's far beyond the scope of this guide. ## 3\. Splunk away We'll start building a Splunk query and gradually expand it. The first task is field extraction. Let's pull out the repair date. We'll ignore it from now on, since Splunk captures the event arrival date, but we could use it to tag events more precisely. That should be done at the forwarder, though, and requires modifying inputs.conf and a few other files, so let's skip that for now and just go with Splunk's idea of event date. Let's take a look at the cf\_notkept.log file, which will tell us what promises were not kept.
1
source=/var/cfengine/*/cf_notkept.log
We can extract the useful data similarly to how we did it for repaired promises.
1
2
3
4
5
6
7
8
9
10
11
source=/var/cfengine/*/cf_notkept.log |
rex field=_raw "(?\d+),(?.*),(?.*)" |
search reason="*"
1341403857,internal_promise,Unable to establish connection with 192.168.2.52
Here we grab the promise type and the reason the promise was broken. Can we find the top reasons or failing promise types by host? Sure\!
1
2
3
4
5
6
7
8
9
10
11
source=/var/cfengine/*/cf_notkept.log |
rex field=_raw "(?\d+),(?.*),(?.*)" |
top reason by host
source=/var/cfengine/*/cf_notkept.log |
rex field=_raw "(?\d+),(?.*),(?.*)" | top promisetype by host
## 4\. Details, details We'll look at command promises next. From cf\_repair.log we'll extract the command ("promiser") and the "repairstatus" which is usually "succeeded". We also grab the "promisetype" which tags the command. The base search is for "-\>" to cut out data we don't need.
1
2
3
4
5
6
7
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),(?.*),.+Finished command related to promiser \"(?.*)\" -- (?.+)" | search promiser="*"
1341094354,cf_hub_memory, -> Finished command related to promiser "/root/cfhub_total_mem.sh" -- succeeded
This search will tag the fields, which you can use for filtering. So for instance you could say "I only want promisetype=internal\_promise" which happens to be for internal CFEngine use. Or you could chart the number of repaired promises by host, over time. It's easy to explore from that point on. Also see how we do search promiser="\*" at the end? That ensures we only get results with the "promiser" field defined. But what really happened in that magic "rex" command? It tells Splunk to extract fields from each line found. That's actually pretty expensive, so you're better off doing the field extraction at the forwarder, but let's assume you're not ready for that sort of commitment. The regular expression is like PCRE or Perl, using named captures to tag each field. It's fairly ugly and hard-coded in order to extract useful data out of each text line. This is unfortunately due to the default format of cfengine's logs, which is meant for human consumption. Let's look for other types of repaired promises. File permissions:
1
2
3
4
5
6
7
8
9
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),(?.*),.+Object (?.*) had permission (?\d+), changed it to (?\d+)" |
search promiser="*"
1341330108,update_files_sys_workdir_lib, -> Object /var/cfengine/lib/libtokyocabinet.so.9.8.0 had permission 755, changed it to 644
Here we extract the file name ("promiser") and the old and new permissions. File deletions are captured similarly:
1
2
3
4
5
6
7
8
9
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),(?.*),.+Deleted file (?.*)" |
search promiser="*"
1341330426,garbage_collection_files_tidy_outputs, -> Deleted file /var/cfengine/outputs/cf_localhost_localdomain__1340725618_Tue_Jun_26_17_46_58_2012_0x40229940
And let's grab file copies too:
1
2
3
4
5
6
7
8
9
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),.+Updated (?.*) from source (?.*) on (?.*)" |
search promiser="*"
1341330108,update_files_usr_local_sbin, -> Updated /usr/local/sbin/cf-agent from source /var/cfengine/bin/cf-agent on localhost
Reports are easy:
1
2
3
4
5
6
7
8
9
source=/var/cfengine/*/cf_repair.log Report |
rex field=_raw "(?\d+),(?.*),Report:\s+(?.*)" |
search promiser="*"
1341416803,cfengine_web_server_reports_disable_selinux,Report: NB - Disabling SELinux to allow PHP to function
Processes require two searches, one for "signalled 'SIGNAL'" and the other for "Making a one-time restart promise":
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),.+Signalled '(?.*)'.+to process \d+.*?(?[^/]*)\)" |
search promiser="*"
1341246660,update_processes_kill_cf_monitord, -> Signalled 'term' (15) to process 19781 (root 19781 1 19781 0.0 1.5 62400 15 7672 1 Jun29 00:02:25 /var/cfengine/bin/cf-monitord)
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),.+Making a one-time restart promise for (?[^/]*)" |
search promiser="*"
1341246660,update_processes_kill_cf_monitord, -> Making a one-time restart promise for cf-monitord
## 5\. Aggregation You will probably want to take all these searches and make a report out of them. As I mentioned, these searches are awkward because the logs are intended for people. But if you need to, use the "append" command Splunk provides. Here's an example:
1
2
3
4
5
6
7
8
9
source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),.+Signalled '(?.*)'.+to process \d+.*?(?[^/]*)\)" |
append [ search source=/var/cfengine/*/cf_repair.log "->" |
rex field=_raw "(?\d+),.+Making a one-time restart promise for (?[^/]*)" ] |
search promiser="*"
It's not complicated, just tedious, and s-l-o-w (you have to wait for the two searches to complete in series, not in parallel). If you find yourself doing this kind of searching aggregation, you should look into parsing the fields out at the forwarder and other Splunk tricks -------------------------------------------------------------------------------- title: "LinuxCon 2012 Presentation by Mark Burgess" url: /blog/2012/linuxcon-2012-presentation-by-mark-burgess/index.md date: "2012-06-29" --------------------------------------------------------------------------------
![](images/LinuxConBanner-d3864654.png)
We will be holding some sessions at LinuxCon in San Diego this summer and will be very happy to see you there\! We just got a 15% off coupon to give away, but you need to act fast - it is good for the first 10 attendees and then it expires. The code is **LFSD12345** - use it when you register for [LinuxCon 2012 here](https://events.linuxfoundation.org/events/linuxcon "LinuxCon 2012 in San Diego"). We wanted to share the abstract of Mark's session: ## Scaling configuration management across data center and cloud (aka "How to deploy 10,000 machines and still be home for dinner") The days of large IT organizations maintaining systems manually are ending. Today's infrastructure is dynamic, personalized, and malleable. IT infrastructure has become incredibly flexible and complicated as it increasingly uses cloud computing infrastructure in combination with traditional data-centers. To manage such hybrid infrastructures, organizations need configuration management solutions that enable system administrators to deploy and move workloads with ease, without disruption and with consistent behavior across the physical and virtual platforms. This presentation is for infrastructure architects, system administrators and software developers. Mark will discuss the present and future challenges of managing IT infrastructure and will review scenarios and strategies to scale hybrid infrastructures from a few to thousands or even millions of nodes, while ensuring reliability and compliance for all nodes throughout their entire life cycle. -------------------------------------------------------------------------------- title: "LinuxCon 2012 Presentation by Diego" url: /blog/2012/linuxcon-2012-presentation-by-diego/index.md date: "2012-06-22" --------------------------------------------------------------------------------
![](images/LinuxConBanner-d3864654.png)
We will be holding some sessions at LinuxCon in San Diego this summer and will be very happy to see you there\! We just got a 15% off coupon to give away, but you need to act fast - it is good for the first 10 attendees and then it expires. The code is **LFSD12345** - use it when you register for [LinuxCon 2012 here](https://events.linuxfoundation.org/events/linuxcon "LinuxCon 2012 in San Diego"). We wanted to share the abstract of Diego's session: ## A collaborative approach to managing Linux configurations Organizations struggle to manage server sprawl, virtualization, migrations to private, public and hybrid clouds and the need to integrate mobile computing devices in the IT infrastructure. At the same time cycle times need to be shortened, security needs to be improved and access needs to be made universal. This cannot all be done economically unless sysadmins can re-use design-patterns and code from others. Configuration Management companies like CFEngine, Chef and Puppet have started deploying repositories for configuration examples. What is needed is tooling to allow sysadmins to browse, mix, match and deploy the examples from such repositories without recoding. In the presentation Diego will present a model for doing just that and show some of the work he has done to make that a reality. This presentation is for system administrators, who work in dynamic organizations with customer facing web applications and frequent infrastructure changes. This presentation will show them how they build and maintain large-scale, complex Linux infrastructures with fewer resources and less outages. -------------------------------------------------------------------------------- title: "Model-based monitoring with CFEngine" url: /blog/2012/model-based-monitoring-with-cfengine/index.md date: "2012-06-10" --------------------------------------------------------------------------------
![](images/monitoring-blog-f8efe323.png)
* * *"A model is a lie that helps you to see the truth." (Howard Skipper)* *"There is nothing more practical than a good theory." (Kurt Lewin)* The past year has seen a plethora, one might even say an entire movement, of talks and blog posts under the heading "[Monitoring Sucks](https://github.com/monitoringsucks/blog-posts)". Plenty of valid criticisms have been made about the state of the art in monitoring. Back in 1998, I was similarly dissatisfied with the state of the art, and began to ask some basic questions that resulted in CFEngine's present day tools for system monitoring. This article is a reminder of CFEngine's smart, and extremely lightweight tools for de-centralized monitoring. These tools were designed to be adaptable, hands-free and to scale to tens of thousands of hosts, while handling machine-learning pattern matching, and responding automatically to thresholds and anomalies with minimal latency, with or without human intervention. ## An anecdote Merely integrating configuration management with arbitrary monitoring software will not necessarily lead to a useful system. I recall an episode, not many years ago, where a couple of student interns at different companies were trying to monitor web statistics on a web server farm. They both did exactly the same thing: they installed a popular open source monitoring tool and started collecting graphs. Because they were inexperienced, and didn't know what to look for, they chose system load from one of the outputs. What they found was a series of sharp peaks in the system load at regular intervals. At first they presented this result without question, but quickly realized that this did not make sense, as it disagreed with web server logs. Finally after digging around, they realized that the sharp spikes were the monitoring system itself kicking in and setting off hundreds of shell commands to probe the system. The monitoring system had achieved a perfect state of staring itself in the mirror of its own activity. The data were not disputed, but the result *meant* nothing as far as the web service was concerned. Werner Heisenberg made the point during the last century that the act of observing a system requires us to disturb it. If the disturbance we introduce is greater than the phenomenon itself, then we simply cannot observe it, no matter how much we might like to. ## Theory - or what's in a model Over the years, I've seen a lot of suggestions about how to collect and analyze data streams. Because we have a desire to "know" answers, there is a natural instinct to collect data from trusted sources. The question is: what sources do we trust, and why do we believe their integrity? Years ago, I worked with a space science department who collected satellite data. They spent so much time accumulating data that they were never able to analyze everything they collected. The result was a large cost and no result. By the time they considered analyzing the data, they were already irrelevant. On a more mundane level, our desire to know about the stars of Hollywood might lead us to study gossip rags or studio press releases. Which will give us the insight we need? Our desire to understand computers in our organization might lead us to SNMP or vmstat. Which of these do we choose? What do we trust? Science has a simple answer to this. You can't trust any source more than any other, a priori, unless you have some kind of a model in your mind of what it is you are looking for. That is the first step to evaluating whether the data you receive make sense. | | | --------------------------------------------------------------------------- | | *A model is a set of assumptions and expectations, formalized in some way.* | We see almost no mention of models in system monitoring. We are lucky if the axes of graphs are labelled with some kind of scale and units that indicate what we are looking at. With the occasional exception of some curve-fitting algorithms, modelling seems to be the undiscovered country of measurement. That is because there is weak integration between data sources and data rendering tools. A model, or a theory, is an idea about what to expect when you observe the world. For some reason, the idea of theory has been seen as the opposite of practice in recent times (something to be derided), but this is an unfair judgement, usually promulgated by those with a chip on their shoulder. The fact is that no one can do without a theory when they interpret data. The real question is how consistent are we in doing so? ## Data versus meaning: admitting to uncertainty All knowledge is subject to uncertainty. A useful way to think about this, is the simple model (self-reference intended) known as the [The DIKW hierarchy](https://en.wikipedia.org/wiki/DIKW). When we look at the world, we collect Data, often with limited reliability, and limited fidelity. Data are just values that mean nothing in isolation -- nothing until you put it into some kind of context with a model. For example, you learn that the weights of three people are 65, 70 and 20. What does this tell us? Very little. There are some differences in the numbers, but we don't know the cause. If the units are in kilogrammes, this could be two adults and a child, or it could be two overweight kids and an adult suffering from malnutrition. Three children, two with heavy shopping? Unless we state our assumptions and expectations, there is no way to interpret data. Ideally, the context would be presented along with the data so that we have something to compare the new data to. In computer monitoring, it is common to assume everything and state nothing. This can be based on past experience, or simply on speculation -- but our eventual estimate of *uncertainty* depends on this. The chief lesson of science is that there is always uncertainty, and we have to manage this. At this point, context and model make data information. If you can assimilate this and re-use it, it becomes knowledge. Wisdom is something I personally see as being something we aspire to down-the-line: an asymptote to be approached by the paradox of Zeno. DIWK. ## Presenting information instead of data Monitoring tools today throw data at humans, and expect us to make sense of them. Humans dutifully stare at moving traces and waste expensive hours in one of a number of time wasting activities: - Staring at the data, as if it were reality TV (without aim). - Investigating things that look unexpected (with inconsistent aims). - Comparing data to past behaviour and relating it to causes (manual analysis). - Trying to analyze the data using other tools (searching for meaning). As I was developing techniques for monitoring in CFEngine to fulfil the idea of self-regulation, I asked the simple question: is it possible for the tools to assist the humans rather than the other way around? Now, CFEngine's goal was never to be a monitoring tool, per se. But it was clear from the research that monitoring and system configuration need one another. (Today, Open Source is both a blessing and a curse in this matter: it is a blessing because we can get almost anything to play with, but it is a curse because Open Source projects have become strong *brands* that people don't want to mix together. Merging monitoring and configuration is considered heretical by many due to brand loyalty.) I began by studying computers in action to see what they did (see [Measuring Host Normality (2000-2002)](https://web.archive.org/web/20060504213244/http://research.iu.hio.no/papers/burgess-ACM-TOCS.pdf)), like Sir David Attenborough filming animals in the wild. In computing, one generally begins with the naive assumption that computers do what we told them, but that is naive indeed: we know that they do all kinds of things that we never told them to also. Amongst the key observations were the following points: - Computers only behave predictably if they are busy. - Computers are busy when people use them. - Computer activity is driven by the patterns of the human working week. The last of these points was a significant result: performance statistics have a strong periodic pattern, that peaks once per day for each day of the week. The largest peaks are usually early in the week, and they fall off towards the weekend. Sound familiar? The figure shows a typical pattern for a busy web-server, measured over several months, but with each successive week overlaid on the previous ones, so that the time scale is measured from Monday at 00:00 and ends on Sunday at 24:00. The points are the average over all weeks, and the vertical bars (what scientists call error bars) show the standard deviation, which is an order of magnitude estimate of the "uncertainty" or spread of the values. ![](images/normal_graph.png) The daily peaks are evident. But notice also the "error bars" which represent how much variation there is in the signal. These are small compared to the pattern itself. If that were not the case, any pattern we managed to see would merely be fiction, like faces in clouds. These early studies led to a stochastic model of computers in an environment of users, based on well known methods from physics (see [The Kinematics of Distributed Computing](https://web.archive.org/web/20060506071720/http://research.iu.hio.no/papers/compdynam.pdf)) that gave a measure of understanding as to *why* the data looked like this, and eventually it led to a simple reformulation of that model into cylindrical time coordinates (see [Two dimensional time-series for anomaly detection and regulation in adaptive systems](https://web.archive.org/web/20060505155712/http://research.iu.hio.no/papers/dsom2002.pdf)), so that we could factor out the trend of the week. The importance of the result was to confirm that it makes sense to store what happened in the past and compare it to what is happening now, because there is a *repeated pattern*. If a system never behaved the same way twice, nothing we see in the data (beyond increasing and decreasing) can mean anything. On busy systems, this pattern is significant enough to have meaning. On desktops and occasionally used servers, the pattern is simply noise, and every new event is an anomaly, as you might expect of something that is completely irregular. This picture makes sense, and allows us to trust what we are seeing. So why not make tools that present measured data in the context of this established model? Then we would immediately have a notion of what is normal (i.e. expected) and that is potentially interesting. Moreover, this determination can be completely automated and used to decide on what actions as a matter of policy. This is what CFEngine did. ## Self-regulation strengthens measured patterns by linking desired and actual state In 1998, I introduced the idea of [Computer Immunology](https://web.archive.org/web/20130927124436/http://research.iu.hio.no/asysadmpapers.php), suggesting that in order to make systems self-healing and autonomically adaptive, it was necessary to tie model-based monitoring to configuration systems. The Computer Immunology paper laid out a manifesto for research, which ultimately turned into a series of technical papers, proofs and implementations in CFEngine versions 2 and 3. The fruits of that research led to some very simple and efficient ways of filtering meaningful information from useless noise. The situation is analogous to configuration by scripts of relentless shell commands, and model-based configuration where there is a policy that describes desired state. Indeed, to know about a self-regulating system, there are two parts
       Desired State  ->  Actual State
If these things do not match, what are you going to do about it? Well that's up to you. ## Big data and brute force Today, I see that we are tempted by the allure of technologies to store large amounts of data. We have Big Data Envy of sites like Google and Amazon that have genuine needs to store large data. Some are suggesting that we should store every possible bit of data and simply search it. This was the CMDB dream, put forward mainly by large database manufacturers. Of course, it is a ridiculously inefficient way to approach measurement of a system that has a known purpose. It increases cost, scales poorly and it is literally meaningless to do so. If we want data to be relevant information, then we should be looking for ways to reduce the amount by factoring out expected signals from noise, not collecting more just because we can. If we know there is nothing to find in more data, there is no sense in collecting it. Increasing the amount of data does not necessarily increase your certainty, or the ability to determine cause-effect relationships. My personal view is that even curve-fitting or regression analysis of data (the love child of Gaussian statistics and spline fitting) is also a waste of time, as it introduces new uncertainties and characterization of observed deviations is a more honest tool that can easily be inspected by human judgement, without taxing anyone's brain. Anyway, computer statistics are rarely Normally distributed. Separation of fast and slow-moving variables (sometimes called de-trending) is a more useful standard technique for finding the "tides" of system behaviour, and this allows us to see signal from noise. When dealing with cumulative measures, like disk usage, the patterns are in deltas or rate of change. A complete description of a dynamical system can be found by measuring values and rates of change (called the phase-space of the system) over time. ## CFEngine's cf-monitord and vital signs: certainty, repeatability and forgetting the past CFEngine's monitoring was originally an experimental idea, but it has proven very useful, and it still has a unique point of view. The key to making sense of our computers is to repeat measurements and look for patterns. Merely collecting reams of time series graphs going far back in time does not allow us to see such patterns clearly. Humans can only perceive small amounts of data at a time. The tools should help us to do this, and present humans with information which accelerates their decision making rather than wasting their time with manual analysis or even guesswork. ![](images/cf-vitalsigns.png) A CFEngine weekly trend graph, showing the current values, average over months and envelope of uncertainty. A key design feature was to make cf-monitord almost unmeasurable in its load to the system. It uses direct system probes through the POSIX library to access data, and matches using regular expressions to offer a very flexible interface to gather almost any kind of data and analyze it before handing it over as contextualized information. In this way, it does not waste system resources, or human resources. In a similar way, it does not store endless amounts of data. Due to a clever machine-learning trick, using geometric series, only one week's worth of data are needed to achieve - Latest measured value over the past week. - Weighted average for current 5 minute interval going back several months (tunable) that progressively demotes and forgets old data. - Rate of change. - Estimated uncertainty (reliability) in the data for any 5 minute interval in a week. All that adds up to just a few megabytes per machine for around 50-100 metrics. ``` # Example promise to measure failed logins by ftp every 60 mins bundle monitor custom_measures { measurements: "/var/log/messages" # read from any source, avoid pipes to stay light handle => "ftp_failed_anonymous_login", stream_type => "file", data_type => "counter", match_value => scan_log(".*ftpd\[.*", ".*ANONYMOUS FTP LOGIN REFUSED.*"), history_type => "log", action => sample_rate("60"); # .... } ``` Understanding how often data need to be measured is straightforward: Nyquist's theorem (Fourier analysis) tells us that we need to sample at about twice the rate of the fastest change to capture its behaviour. If a process is changing quickly, we need to sample often. If it is changing slowly, it is just wasting time and resources to sample it more often than that (see [Ten Reasons for 5-minute configuration update and repair](https://cfengine.com/blog/ten-reasons-for-5-minute-configuration-repair)). ## Anomalies and alerts The CFEngine way is to classify the actual state of your system and then work towards a desired state, based on information relative to a clear predictable model. Even after 10 years, this is surprisingly new in concept. In classifying promised measurements, CFEngine feeds back information that can be used to respond to what is going on. Each measurement type, labelled by its handle (in the example above) sets classes about how the current state matches the learned periodogram model. As well as checking for fixed thresholds, you can characterize the state of your system relative to what is normal. The possibilities here are very extensive, and there is no room to describe them here. ``` bundle agent followup { reports: www_in_high_dev2:: "Incoming web hits is 2 stddev above normal for this time: $(sys.date)"; # print more data / logs / IP addresses etc } ``` A complete list of these class types is described in the documentation (see references at the end of this article). ## Give it a try Running CFEngine's monitoring subsystem is virtually free in terms of system resources. If you use CFEngine Enterprise edition, you will have data presented to you in a maximally meaningful way, and will be able to respond to it manually or automatically in a minimum of time. Centralized measurement systems, like SNMP, add significant uncertainty to data, as we do not know exactly when data were collected due to latencies. It also chokes the network with unnecessary traffic (most of which is meaningless). CFEngine collects all data locally and extracts meaningful patterns before allowing them to be aggregated for cross-machine comparisons, so there is greater consistency and distributed processing to minimize load. CFEngine's monitoring is not centralized, but distributed. As much analysis as possible is done by machine learning agents on individual hosts, so that latencies and uncertainties are kept to a minimum. The goal of monitoring in CFEngine is whatever you want it to be. It can replace systems like Nagios, Munin or Ganglia etc, for many purposes, but it doesn't have to. Choosing tools is not always a rational matter, but if your system is massive and busy, then there is great scope in the tools provided. Check it what is probably the lightest weight monitoring scheme available, and tie it to real-time configuration management as you like. ## Read more [Special Topics Guide on Monitoring: log management and anomaly detection](https://cfengine.com/manuals/st-monitoring). -------------------------------------------------------------------------------- title: "The CFEngine Design Center: component-based IT infrastructure engineering" url: /blog/2012/the-cfengine-design-center-component-based-it-infrastructure-engineering/index.md date: "2012-06-06" --------------------------------------------------------------------------------
![](images/DC-blog-d0a7532a.png)
We are proud to announce the immediate availability of the [CFEngine Design Center](https://cfengine.com/cfengine-design-center/), a community-driven place to exchange CFEngine code, tools and information. The CFEngine Design Center opens the door to powerful knowledge-based IT infrastructure management without the need to immediately learn a new language. It enables system administrators to "stand on the shoulders of giants" by seamlessly reusing the knowledge and code of peers who have performed similar tasks before, and also to contribute their own knowledge back to the community. It makes it possible to set up a fully-operational CFEngine infrastructure without the need to touch a single line of CFEngine code. ## Background CFEngine has long been the standard by which all configuration management systems are evaluated. CFEngine 2 manages millions of hosts throughout the world. CFEngine 3, released only three years ago, has set a new standard in flexibility, extensibility and consistency. In CFEngine 3, the policy language acquired powerful encapsulation, abstraction and reusability features, including [parameterizable bodies and bundles](https://cfengine.com/manuals/cf3-reference#Containers), [implicit looping](https://cfengine.com/manuals/cf3-Reference#Loops-and-lists-in-CFEngine-3), and much more. With these improvements, it became easier than ever to write reusable, modular CFEngine 3 components. ## Enter the Design Center At its core, the Design Center is a community-driven repository through which the CFEngine community can share ready-to-use components called *sketches*, as well as CFEngine policy examples and tools. It is [hosted on GitHub](https://github.com/cfengine/design-center), and anyone can use its contents and contribute to it. By these features alone, the Design Center is already an extremely valuable resource to the CFEngine community. Never before has it been so easy to share your CFEngine policy components and libraries, or to find and use those written by others. However, we see the Design Center as the start of something much greater than simply a collection of bundles that you can use. Each sketch in the Design Center represents a piece of **encapsulated knowledge** about one particular part of IT Infrastructure Engineering: how to solve a particular problem or perform a certain task. When you are presented with the same task or problem, you don't have to reinvent the wheel--you can rely on the accumulated knowledge of those before you, and also augment it. Furthermore, each sketch is not simply a "module" that does its work in isolation. Rather it integrates seamlessly into the rest of the existing CFEngine policies, being able to both draw from and contribute to the pool of information that allows CFEngine, and you as the system administrator, to have the best knowledge and make the best decisions at any given point in time. With the Design Center, it becomes possible to build and manage reliable, scalable infrastructure using best-of-breed components, without touching a single line of CFEngine code. Whether it is configuring the operating system, managing web applications or managing a cloud infrastructure, the Design Center can help. And of course, using Design Center sketches does not prevent you from writing your own policies--you still have access to the full power of the CFEngine policy language. Because all sketches are open source, you can even look at them as learning aids for advanced CFEngine techniques that you can later use in your own hand-written policies. To help you get started, we have developed the [cf-sketch](https://github.com/cfengine/design-center/wiki/Getting-started-with-cf%E2%80%93sketch) tool to manage sketches on your systems. Whether you are just getting started with CFEngine or are a veteran writing CFEngine policies, the Design Center has something for you. We invite you to [learn more about it](https://github.com/cfengine/design-center/wiki), [start using it](https://github.com/cfengine/design-center/wiki/Getting-started-with-cf%E2%80%93sketch), [learn about the tools available](https://github.com/cfengine/design-center/wiki/cf%E2%80%93sketch-manual), and even [write](https://github.com/cfengine/design-center/wiki/How-to-write-a-new-sketch) and [contribute](https://github.com/cfengine/design-center/wiki/Contributing-to-the-cfengine-design-center) your own sketches. We look forward to your participation and feedback\! We are just at the beginning of this exciting new journey. If you have any comments, please let us know at . -------------------------------------------------------------------------------- title: "CFEngine 3 Enterprise 2.2.0 release notes" url: /blog/2012/cfengine-3-enterprise-2-2-0-release-notes/index.md date: "2012-06-04" --------------------------------------------------------------------------------
![](images/class-include-exclude-0ff03b21-8970a7b4.png)
## Highlights The CFEngine Enterprise version 2.2.0 release is here, and this release is bringing major customer requests. The Mission Portal graphical user-interface has gotten the following new features. - Dynamic host grouping - Access control for reports - REST API ## Dynamic host grouping The new edition of CFEngine Enterprise allows you to use any CFEngine classes to build arbitrary groups of hosts. Groups can be arbitrarily nested, yielding a tree-like structure. Below is an example of how a grouping by operating system classes may look. ![](images/os-grouping.png) As you can see from the sample grouping, selecting a group will display compliance and reports only for that group in the right pane. You can see that the context "Linux" is currently selected. Adding a sub-group can be done by simply clicking on the plus-sign next to the group name. As a second example, to see promises not kept on Ubuntu hosts, you simply select the Ubuntu group in the left pane, and select the Promise not kept report in the right pane, which yields the results shown below. ![CFEngine - Ubuntu](images/ubuntu-promisenotkept.png) At the top of the left pane, you can see that our current grouping layout is called "OS". You can create an arbitrary amount of grouping layouts, for example to group by service (web server, db server, etc.), location (NY, CA, London, etc.), operating system and so on. As the foundation of this system is based on CFEngine classes, it is extremely flexible, and you can use any class automatically defined or defined in the policy. This system allows you to structure your hosts in ways that make sense in your environment. Perhaps you are interested in the compliance of the web servers, but want to see which Solaris servers have had changes in their passwords. Creating multiple grouping layouts makes questions like these easy to answer. However, also note that you can override the host grouping when generating reports, to include and exclude arbitrary lists of classes. For example, if you want to see all promises not kept on Ubuntu hosts, but you are not interested in the Ubuntu 10.4 hosts, you can create the following filter. ![CFEngine - filter by class](images/class-include-exclude.png) ## Access control for reports As multiple people from different teams may be using the Mission Portal for generating reports about their environment, access control quickly becomes an issue. For example, the database administrators should perhaps only see reports coming from database servers. The Mission Portal already supports LDAP and Active Directory authentication, potentially allowing a large set of users to access it. In this Enterprise release, the Mission Portal allows creation of roles with access to reporting data from subsets of machines and promise bundles. Again, the flexible CFEngine classes play a key role. Say you have a class called "db\_server" defined on all your database servers in your policy. With the new Mission Portal, you can easily assign this class to a role "Database administrators", and then assign your database administrator users to that role. The net effect of this is that the database administrators would only be able to see reports coming from the database servers. You can also limit the ability for users to browse parts of the policy, as defined by regular expressions on promise bundles. As an example, we have the same environment as in the above illustration, but are logged in as a user that can only see the windows class through his group memberships. You can see that both the Linux and Solaris groups became empty, and so it is impossible to see their host counts, compliance and reports. ![CFEngine - group by OS](images/os-grouping-windows.png) ## REST API Customers have often asked us for ways to export the valuable data found in the Mission Portal to other systems. It is already possible to generate csv and pdf reports from the Mission Portal web interface, and also export data with a CFEngine command-line tool. However, to take it even further, we have introduced support for REST API from the Enterprise hub. The CFEngine Enterprise hub knows the state of all systems, so there is no need to contact each and every of the CFEngine clients individually through REST. It can all be found at the hub, and it can generate summaries for you for the various types of reports to make it very efficient and concise. For example, you can use this interface to import promises not kept into a ticketing system like Redmine. More details on the new REST API can be found in a [separate blog post](https://cfengine.com/blog/rest-api-for-nova). ## Host operation diagnostics Below the compliance graph, you can quickly see if the hub has any connection issues, or if CFEngine is not running correctly on any of the hosts. ![CFEngine - Host Operations](images/host-operations.png) ## Other improvements The software reports are now updated by default every five minutes at the hub, as opposed to 6 hours in previous releases. The installed software also comes with timestamps for when they were discovered. Better orchestration functionality is made possible with the new [hostswithclass function](https://cfengine.com/manuals/cf3-Reference#Function-hostswithclass). With it, you can for example dynamically generate configuration for monitoring software to monitor hosts with the web\_server class set. As usual, many performance improvements and bug fixes have been included. We are very proud of all the features included in this Enterprise release. Of course, all the new features and fixes introduced in the [Community 3.3.3 release](https://cfengine.com/blog/cfengine-330-release-notes) are available in the enterprise edition, including Unix service promises, and the new templating engine. All this is made possible by our many users and customers requesting new features and presenting their use-cases. We are always listening to requests, so please keep them coming\! Existing customers get this major upgrade, as well as all future ones, as part of the relationship with CFEngine. -------------------------------------------------------------------------------- title: "CFEngine 3 Enterprise is leaving dry-dock ..." url: /blog/2012/cfengine-3-enterprise-is-leaving-dry-dock/index.md date: "2012-05-29" --------------------------------------------------------------------------------
![](images/enterprise-6575f62f.png)
*Today, the CFEngine team is announcing CFEngine 3 Enterprise. With the major part of the CFEngine 3 technology being in an open source core, our exploratory commercial edition, was originally dubbed \`CFEngine Nova' -- the \`New star in configuration management'. Today, CFEngine 3 is no longer a newcomer, but a proven solution in datacentres around the world. With today's launch, CFEngine 3 Enterprise leaves orbit and begins its voyage to manage an ever expanding universe of IT.* #### Low cost scalability and high performance and availability At CFEngine, we strive constantly to maintain CFEngine 3's position as the undisputed technology-leader in IT automation. With the focus today on IPv6, operational agility, green IT consolidation, cloud and virtualization, that story begins with offering unprecedented performance. Low memory, CPU, and I/O footprints make CFEngine 3's performance surpass competition by between 10-100 times, meaning that users can manage more hosts at a faster rate, with less hardware and smaller cloud instances, leading to considerably lower costs in the long run. This is a key differentiator for CFEngine in High Performance and E-Commerce shops. In mission-critical environments, CFEngine 3 Enterprise also supports high availability of policy status, with login access to the Mission Portal available within seconds at all times. A CFEngine \`hub', gathers status reports and monitored information in a highly compressed format; moreover, repairs to desired state an average of 2.5 minutes away by default. Users choose their own scaling plan, depending on their available hardware; each hub will support several thousand hosts with our lightweight CFDB (CFEngine's smart CMDB/SKMS replacement). CFEngine's attention to the close dialogue between automation and knowledge management, makes it the most advanced [Third Wave](https://cfengine.com/blog/sysadmin-3.0-and-the-third-wave) IT management solution, looking ahead to the future needs of infrastructure engineers. Already today, CFEngine 3's architecture has been designed to offer flexible federation in organizational control, with hosting of hubs possible both in public cloud, private datacentre, and even in the smart environments of the workplace. #### Affording enterprise scale, complexity and insight The technological benefits of the [CFEngine 3 Community Edition](https://cfengine.com/blog/cfengine-330-release-notes) are well documented, and they help users to automate the basics of Build, Deployment and Maintenance -- but in an enterprise setting this is not enough -- one needs the scaffolding of a knowledge system around IT to build trust. The value of CFEngine 3 Enterprise lies not only in its high performance, allowing a low Total Cost of Ownership, but in its ability to model and document complex business requirements. Through this kind of modelling, we understand systems and learn to trust in predictable behaviour. Security scanning can easily be added for additional change management, and compliance monitoring is also a key requirement for complex enterprises. Condensing the enormous amounts of information down to a meaningful summary in a scalable, timely and affordable manner is a principal goal of CFEngine 3 Enterprise. Going forward, massively federated infrastructure will benefit from [Project Constellation](https://cfengine.com/blog/a-preview-of-project-constellation-the-network-of-stars-492)'s inclusion as a configurable part of the CFEngine 3 Enterprise unified toolset. Based on our user feedback, the initially proposed Nova, Constellation, and Galaxy editions will now be consolidated into this single project, allowing users the freedom to choose and customize the functionality they want. Reporting on the *actual state* of IT infrastructure, and how it measures up against our desired (policy) state, with [five minute accuracy](https://cfengine.com/blog/ten-reasons-for-5-minute-configuration-repair), is the bedrock of a trusted computing platform. CFEngine 3 Enterprise does not require you to sacrifice any flexibility or precision of engineering to achieve this goal. Federation of control is the modern way to avoid the blunt \`one size fits all' thinking of *centralized control systems*. Each federated hub, today, supports role-based access (RBAC) to its data, and each individual host supports role-based execution of special policy items. Thus complex organizations, with multi-cultural expertise can co-exist peacefully and with confidence. CFEngine 3 Enterprise offers the possibility of enhanced Chinese Wall separation between hosts and users that share access to a CFEngine Mission Portal. #### DevOps and agile operations Big enterprise customers are an area where CFEngine excels, but today the industry is being changed from below. The new wave of online web-based companies, has forced scale and agility to new heights. These businesses revolve around web services, where a simpler infrastructure often suffices, but continuous deployment of applications is a new source of complexity. Several new initiatives are underway to integrate libraries of standard methods with fast deployment of virtual hosts in a hybrid cloud setting -- in our Design Centre project. CFEngine 3 Enterprise adds superior knowledge management around this armoury of simple pluggable modules, and that integrated knowledge overview can be accessed through a [REST API](https://cfengine.com/blog/rest-api-for-nova) enabling integration of CFDB data with other strategic tools. As always, we respect the cultural preferences of infrastructure engineers and their ways of working. Users can move easily from command-line interfaces (CLI) to the web-based Mission Portals. Reports can be found in the CLI console, or clicked through on the web from any device. In the Mission Portal, a model-based approach has been developed to give users individually customizable spanning-trees to navigate reports for arbitrary groups of hosts, in an intuitive way. Moreover, the integration of *semantic web* auto-analysis of policy puts diagnostic capabilities in the hands of smart engineers. These are just two examples of an invigorated drive to approach user-interaction development in a Third Wave knowledge-driven framework. The DevOps manifesto touts CAMS as the principles of sound IT infrastructure management. - Culture - supporting federated knowledge and unlimited diversity. - Automation - fast, scalable, hands-free operations. - Measurement - continuous observation with feedback to policy. - Sharing - access to reasoned information and knowledge through multiple interfaces. We think CFEngine 3 Enterprise addresses all of these points, and lays out a long-term roadmap for the changes to come. #### Orchestrating complex environments with CFEngine 3 Enterprise CFEngine installations number amongst the largest and most complex on the planet. Handling this complexity is not something that can be done by platform-building alone -- orchestrating complex multi-faceted components on top of that base is an increasingly large part of the problem. Today, as we bridge the gap between physical and virtual infrastructure, complex multi-host orchestrations are still relatively difficult to achieve. This will almost certainly become easier as cloud-like infrastructures mature. In the meantime, CFEngine 3 Enterprise enables parallel and sequential task-orchestration across the environment using familiar class-based mechanisms of the core. But centralized datacentres are not the desired end-state for all computing. The world of embedded devices is also on the rise, with pads and smart surfaces increasingly being powered by Linux derivative operating systems. CFEngine 3 scales so efficiently that it will run on even the leanest devices, for realizing the vision of smart environments. CFEngine's basic design tolerance of being off-line makes it the way forward for managing the diverse flora of systems we are about to see blossom in the commerce of the coming decade. Our recent integration work with QNAP storage technologies, Nokia, Android, ARM, wireless routers and other embedded systems, as well as more far-reaching ideas about smart infrastructure, has positioned CFEngine 3 Enterprise to be a unifying strategy for the true desired end-state of the Third Wave technology. Now, it just remains to install it on yours\! -------------------------------------------------------------------------------- title: "Scale and scalability" url: /blog/2012/scale-and-scalability/index.md date: "2012-05-22" --------------------------------------------------------------------------------
![](images/HiRise-9f36f1ec.jpg)
*If someone asks you about the scalability of your operations, don't tell them about the number of machines you run; tell them rather about what it costs you to tend them each month. The total cost of that burden can be summed up from the cost of hardware, software, maintenance, people, lost revenue during downtime, time lost during maintenance, and time wasted from not managing knowledge well.* At one customer site, CFEngine replaced an old second wave push-package provisioning system from one of the major vendors, with a lightweight CFEngine automatic maintenance framework, exchanging 200 hosts and a team of sysadmins in the push-framework for 5 CFEngine servers run by a single man. The push framework had a response time of about an hour, while the CFEngine framework handled 5 minute time resolution, all of this easily supporting a modest 10,000 hosts with configuration and monitoring of state. What is the cost of scaling? On a second occasion, we were part of a test in a virtual cloud to compare the performance of CFEngine against ruby-based alternatives. The initial test had to be abandoned because we had provisioned the smallest memory (cheapest) virtual machines for CFEngine, but these were not even able to load the Ruby sub-system. In the cloud, more memory means higher cost. What is the cost of scaling? Data transfer is another thing we pay for in datacentres, real or virtual. If you are provisioning hosts across a wide area and you maintain a single point of control with chatty protocols, you can run up a large bill. CFEngine customers have abandoned software with expensive push-pull models due to greedy protocol bandwidth. The answer here is to federate and maximize the autonomy of agents, thus minimizing reliance on fragile shared resources. Agility is reduced as delays are added by network latency, and timeouts from failures. CFEngine managed hosts do not need to exchange much data, even to achieve full monitoring, thanks to core principles of lazy evaluation, compression by natural selection and distribution of control. If hosts manage themselves, there are large savings to be made in time and transfer costs. What is the cost of scaling? #### What does a big mess cost? One of the hidden costs of management is insufficient knowledge. Time spent figuring out the what, the how and the why of an installation can add up to both time wasted and manpower. As a system grows in size and complexity, our knowledge of the system's true nature has to scale too. Getting hold of the person who designed a system (and expecting him or her to remember the details) might not even be possible if that person has already left the firm. The personnel cost to scaling is the highest one we have. True automation (the hands-free, not the remote control kind) uses models and design patterns to reduce the total amount of information we need to look at in order to govern our hosts. That makes it easier to know a system, because there is less to know. In Claude Shannon's information theory, there is the concept of compression to the smallest amount of data needed to represent the true intent. If that information grows linearly with the size of your system, it is called noise (or maximal, uncompressible entropy\!). CFEngine was designed with a promise language that is composed of compressed Shannon intentions, in the form of convergent promises. In 2003 I proved that this makes CFEngine's language a minimum entropy model -- which can significantly reduce the information burden, using patterns, thus making a system easier to know. In addition, we can document relationships between the governed parts of a system, and perceive general goals through the details that are more than just noise, and make predictions that become increasingly costly for humans alone, as a system scales. By separating what we intend from everything else that happens to our system, we keep a clarity of purpose even as the system numbers grow. CFEngine learns, as it goes, so that we always have a reference of what is normal and hence ignorable about behaviour. Scalability is the ability to grow your datacenter without growing the amount of work, time or money needed to support it. I mentioned some of the main factors here as a teaser. I challenge readers to look at their own systems with a critical eye: what does it cost you as you add more IT? See also: [CFEngine Special Topics Guide on Scale and Scalability](https://cfengine.com/manuals/st-scale) -------------------------------------------------------------------------------- title: "The CFEngine Test Suites" url: /blog/2012/the-cfengine-test-suites/index.md date: "2012-04-24" -------------------------------------------------------------------------------- When developing a large and flexible software system like CFEngine, it is an unfortunate truth that developers do not always get everything right at the first attempt. The source code is also changing frequently due to new features, optimizations and bug-fixes. As you can see from the [Ohloh online statistics](https://web.archive.org/web/20121104060602/http://www.ohloh.net/p/cfengine), thousands of lines are added and removed every week. All changes have the risk of not working in every scenario, or breaking existing functionality. In addition, build errors may introduce unexpected behaviour that you want to detect before deploying to production. There is a clear need for a system to limit these risks. This is where the CFEngine test systems come in to play. CFEngine has three extensive sets of tests, each covering a different purpose to ensure correct and performant operation. ## Acceptance tests As you might know, an acceptance test is a test where you know a certain input should generate a specific output. This is also known as black-box testing, because you do not consider the internals (the "contents of the box") of the software, merely the resulting output. In the [CFEngine github source repository](https://github.com/cfengine/core), you can find the acceptance tests under [tests/acceptance](https://github.com/cfengine/core/tree/master/tests/acceptance). These are CFEngine policies that have (at least) two bundles: test and check. The test bundle is the actual policy that we are testing for correctness, while the check bundle compares the actual against the expected result after the policy has been run. For example, we may try to create an empty file in the test bundle, and check if the file exists in the check bundle. At the time of writing, there are 939 of these acceptance tests. They can be run by using the testall script: ``` ~/core/tests/acceptance$ ./testall ====================================================================== Testsuite started at 2012-04-24 11:13:38 ---------------------------------------------------------------------- Total tests: 939 ./00_basics/01_compiler/001.cf Pass ./00_basics/01_compiler/002.x.cf Pass .... .... ====================================================================== Testsuite finished at 2012-04-24 11:01:21 (172 seconds) Passed tests: 692 Failed tests: 0 Failed to crash tests: 0 Skipped tests: 247 ``` Have a look in the [README file](https://github.com/cfengine/core/blob/master/tests/acceptance/README) for a more extensive description on how to use the suite and create new tests. ## Load tests [Load tests](https://github.com/cfengine/core/tree/master/tests/load) are intended to measure the performance and correctness under a high load. For example, imagine you have a policy server with thousands of clients. The code that cf-serverd executes on that policy server will have to perform very well, but correctness during concurrent (threaded) operations is equally important. Code executed in scenarios like these is intended to be covered by the set of load tests. At the moment, only a test for embedded database operation is covered by the load tests, but this will (like the other tests) expand over time. This test was used extensively by developers when vastly improving the database performance in the [3.3.0 release](https://cfengine.com/blog/cfengine-330-release-notes). LinkedIn reported a [performance improvement of 14 to 3 seconds](https://cfengine.com/forum/read.php?3,25626,25626#msg-25626) during file transfers from their policy server. It all started with this simple load test. ## Unit tests Finally, [unit tests](https://github.com/cfengine/core/tree/master/tests/unit) are small programs that test very specific functionality within CFEngine. Manipulation of internal data-structures, such as the lists that hold all the policy variables, is a common test area for unit-tests. These are also known as white-box tests, because we care not only about the total input and output of CFEngine, but the internal workings. To run them after you have cloned the core repository, you first have to compile them individually. ~/core/tests/unit$ make alloc_test CC alloc_test.o CC cmockery.lo CC test.lo CC alloc.lo CCLD libtest.la CCLD alloc_test ~/core/tests/unit$ ./alloc_test test_xasprintf: Starting test test_xasprintf: Test completed successfully. test_xvasprintf: Starting test test_xvasprintf: Test completed successfully. All 2 tests passed These three sets of tests ensure the long-term correctness and performance of CFEngine. They are regularly run by the CFEngine build system, so that the developers will get feedback as soon as any do not pass anymore. This way, errors can be corrected promptly and before any users run the code. Please consider submitting your own tests if you find issues or see areas that are not covered. This can be done through a [github pull request](https://help.github.com/send-pull-requests/). -------------------------------------------------------------------------------- title: "CFEngine 3.3.0 Release Notes" url: /blog/2012/cfengine-3-3-0-release-notes/index.md date: "2012-04-11" -------------------------------------------------------------------------------- **Update \[April 12, 2012\]**: Added information about the new templating engine. ## Highlights The CFEngine Community 3.3.0 release is here, with the largest set of improvements since the introduction of version 3.0\! We have added Virtualization and SQL promises to the open source edition and introduced service-promises for Unix. A new templating engine is in place. Some potential "gotcha" issues have been fixed, to streamline and simplify the use of CFEngine. The embedded database code has been significantly refactored, optimized and made much more robust. A large number of useful variables, classes and functions have been introduced. And as usual, a set of bugs have been fixed. We will look more in-depth into the following: - Unix service promises - New templating engine - Virtualization promises - SQL-promises - User-experience enhancements - New classes, functions and variables - Improvements to the embedded-db code [Download now »](https://cfengine.com/downloads "Download CFEngine 3.3.0") ## Unix service promises ### One standard service bundle A very powerful and convenient promise type called services has now been implemented for Unix users. Have a look at the following CFEngine snippet. ```cf3 bundle agent service_catalogue { services: "syslog" service_policy => "start"; "www" service_policy => "stop"; } ``` This will ensure the syslog service is running and that the web service is not running. At this point you are probably a bit suspicious on how CFEngine can know how to manage the syslog and web service in your specific environment. The secret lies in CFEngine's powerful abstraction capabilities. To manage a Unix service, there are three things CFEngine needs to know: the process name, the start command and the stop command. If you do not give CFEngine any additional information about this for your service, it will look for a reserved bundle to describe this: standard\_services. For our example above, it looks like the following. ```cf3 bundle agent standard_services(service, state) { vars: debian:: "startcommand[www]" string => "/etc/init.d/apache2 start"; "stopcommand[www]" string => "/etc/init.d/apache2 stop"; "processname[www]" string => "apache2"; "startcommand[syslog]" string => "/etc/init.d/rsyslog start"; "stopcommand[syslog]" string => "/etc/init.d/rsyslog stop"; "processname[syslog]" string => "rsyslogd"; classes: "start" expression => strcmp("start","$(state)"); "stop" expression => strcmp("stop","$(state)"); processes: start:: ".*$(processname[$(service)]).*" comment => "Verify that the service appears in the process table", restart_class => "restart_$(service)"; stop:: ".*$(processname[$(service)]).*" comment => "Verify that the service does not appear in the process table", process_stop => "$(stopcommand[$(service)])", signals => { "term", "kill"}; commands: "$(startcommand[$(service)])" comment => "Execute command to restart the $(service) service", ifvarclass => "restart_$(service)"; } ``` Let us have a look at what is going on here. As you can see, the standard\_services bundle takes two parameters: the service name and state. If you look at our service\_catalogue bundle above, you can see that the promiser is the service name and the service\_policy attribute is passed as the state parameter. The standard\_services bundle defines the three required strings for each service for each platform: the start and stop commands, and the process name. If we are to support OpenSUSE for the www service, we would need to add three new strings since the process and executable is called "httpd" here. Thus it is very easy to extend the support for new platforms, as well as new services. The complete policy is available here for download [here](https://github.com/cfengine/core/blob/460255133160baa49a92dad9ed04b8e5b029c116/examples/service_catalogue.cf). You may also include other promises in the standard\_services bundle, such as promises for installing packages and service-specific configuration. However, in these more complex scenarios, you probably want to create a separate bundle for each service. We will consider this next. ### Separate bundle for each service Let us modify our policy slightly to the following. ```cf3 bundle agent service_catalogue_separate { services: "foo" service_policy => "start", service_method => service_bundle_separate; "bar" service_policy => "stop", service_method => service_bundle_separate; } body service_method service_bundle_separate { service_bundle => $(this.promiser)("$(this.service_policy)"); } bundle agent foo(service_policy) { reports: cfengine_3:: "we need to ensure $(service_policy) of foo"; } bundle agent bar(service_policy) { reports: cfengine_3:: "we need to ensure $(service_policy) of bar"; } ``` From the above, you can see the service\_method body that is the glue to what we are doing. In it, the service\_bundle attribute defines which bundle should be called. In our previous example, we did not define this, so the service attribute would default to standard\_services. Now, however, we are making the service\_bundle expand to the current promiser, which in our example is "foo" and "bar". These are now the bundles that are in charge of handling their respective services. Running the above policy would make the agent produce the following. $ cf-agent -Kf ./service_catalogue_separate.cf R: we need to ensure start of foo R: we need to ensure stop of bar Now we just need to tell CFEngine how to work with foo and bar. At this stage, I hope you agree that services: promises will simplify and clarify your policy significantly. As always, input and feedback is much appreciated. ## New templating engine Another very convenient new feature in CFEngine 3.3.0 is the existence of a new templating engine, with extensive new capabilities in addition to scalar expansion, which was available in previous versions. As an example, consider the following bundle: bundle agent test { vars: "list" slist => { "1", "2", "3"}; files: "/tmp/expander" create => "true", edit_template => "$(sys.workdir)/templates/input.txt"; } And the following template file: #This is a template file /templates/input.txt These lines apply to anyone [%CFEngine solaris.Monday:: %] Everything here applies only to solaris on Mondays until overridden... [%CFEngine linux:: %] Everything after here now applies now to linux only. [%CFEngine BEGIN %] This is a block of text That contains list variables: $(test.list) With text before and after. [%CFEngine END %] nameserver $(test.list) The lines that start with *\[%CFEngine* indicate directives for the template engine. There are two types of directives: *\[%CFEngine classexpression:: %\]* indicates that the following lines (until the end of the file, or the next class expression) will be included in the output only when the \`classexpression\` is true. *\[%CFEngine BEGIN/END %\]* denotes a block of lines that should be handled as a single unit, particularly for the purposes of list expansion. In the new template engine, list expansion works just like in the CFEngine policy language: if you reference a list variable as a scalar (as in $(test.list) in the example), the line or block of text in which it is included will be repeated for every value of the list. ## Virtualization promises Another new promise type, guest\_environments, is introduced in CFEngine Community 3.3.0. On systems with a hypervisor running, this allows you to make promises about virtual machine guests. Thanks to interfacing with the [libvirt library](http://libvirt.org/ "Libvirt homepage"), a range of hypervisors are supported, including KVM, Xen, VMware and more. CFEngine supports creating, starting, suspending, stopping, and deleting virtual guest machines. Currently, CFEngine's cf-agent must be running on the hypervisor of the guest machine to manage it. We will go through a couple of examples for managing the life-cycle of a virtual machine, using KVM/QEMU as hypervisor. A complete self-contained policy with all the examples below is [available here](https://github.com/cfengine/core/blob/460255133160baa49a92dad9ed04b8e5b029c116/examples/guest_environment_kvm.cf "CFEngine policy for KVM management"). ### Creating a virtual machine bundle agent kvm_create { guest_environments: "my_host_machine" environment_resources => mykvm("my_host_machine", "x86_64", "1", "1048576", "/var/lib/libvirt/images/ubuntu104-64-clone.img"), environment_type => "kvm", environment_state => "create", environment_host => "ubuntu"; } $ cf-agent -Kvf ./guest_environment_kvm.cf ... core> -> Selecting environment type "kvm" -> "qemu:///session" core> -> Found 0 running guest environments on this host (including enclosure) core> -> Found 6 dormant guest environments on this host core> ---> Found a suspended, domain environment called "centos62-64" core> ---> Found a suspended, domain environment called "rhel55-64" core> ---> Found a suspended, domain environment called "ubuntu104-64" core> ---> Found a suspended, domain environment called "win2003-32" core> ---> Found a suspended, domain environment called "win2008-64" core> ---> Found a suspended, domain environment called "ubuntu104-64-2" core> -> Created a virtual domain "my_host_machine" ... In this example, we are passing the machine name, architecture, cpu count, memory size and disk image to the resource-body. The resource body takes care of inputting these variables into a XML-structure that libvirt understands. Again, we are leveraging the benefits of CFEngine's unique ability to create arbitrary abstraction levels with promise bodies. Note that this example assumes a QEMU/KVM image is created with a basic OS installed. It is possible to set up a more complex scenario, where the disk image is built from a PXE-boot service, but this is out of the scope of these notes. Please leave a comment if you would like more advanced tutorials. Under the hood of the mykvm body, libvirt configuration is given as an XML-file to provide the necessary guest machine definition. There are more examples of the syntax in the libvirt documentation (https://libvirt.org/formatnode.html). Another interesting aspect of this promise is the environment\_host promise attribute: it specifies where the guest should be created. On hosts where this class is set, the promise will be checked, on all other hosts, CFEngine will ensure the machine is not running. ### Suspending a machine Continuing with our example above, all that needs to be done to suspend our machine is the following. bundle agent kvm_suspend { guest_environments: "my_host_machine" environment_type => "kvm", environment_state => "suspended", environment_host => "ubuntu"; } $ cf-agent -Kvf ./guest_environment_kvm.cf -b kvm_suspend ... core> -> Virtual domain "my_host_machine" running, suspending ... As usual, CFEngine has nice convergent behavior for virtual machines too. $ cf-agent -Kvf ./guest_environment_kvm.cf -b kvm_suspend ... core> -> Virtual domain "my_host_machine" is suspended - promise kept ... ### Deleting a machine Stopping and deleting the machine definition can easily be done as follows. bundle agent kvm_delete { guest_environments: "my_host_machine" environment_type => "kvm", environment_state => "delete", environment_host => "ubuntu"; } $ cf-agent -Kvf ./guest_environment_kvm.cf -b kvm_delete ... core> -> Deleted virtual domain "my_host_machine" ... This covers the basic use of guest\_environments promises in CFEngine. There are more advanced features to be found for creation by looking at the libvirt documentation. If you have a use-case for virtual machines that you do not see covered, please do not hesitate to let us know. ## SQL-promises The databases: promise type has been available in the commercial [CFEngine Nova](https://cfengine.com/nova "CFEngine Nova") edition for some time. We decided to move this to the Community edition, so that a broader set of users can benefit from it. This allows you to make promises about the structure of [MySQL](http://mysql.com/ "MySQL") and [PostgreSQL](http://www.postgresql.org/ "PostgreSQL") database tables. Please note that it is currently not possible to make promises about rows of SQL databases, we had to leave some features for future releases. Let us jump straight into an example. We will use MySQL as back-end in this example, but thanks to the generic CFEngine policy language, all that is needed to adjust to PostgreSQL is to change the database\_server attribute to local\_postgresql below. We would like the database cfengine\_db to contain the table users. The users-table should have three fields: username, password and email. The creation of this table (and database) can be done in CFEngine as follows. bundle agent databases { databases: "cfengine_db/users" database_operation => "create", database_type => "sql", database_columns => { "username,varchar,50", "password,varchar,80", "email,varchar,20", }, database_server => local_mysql("root", ""); } By running CFEngine, you see the following output. $ cf-agent -Kf sql.cf !! The database did not contain the promised table "cfengine_db.users" -> Database.table cfengine_db.users doesn't seem to exist, creating I: Made in version 'not specified' of '/home/a10003/.cfagent/inputs/sql.cf' near line 13 -> Trying to create table users After this run, you can verify from MySQL that CFEngine did indeed create the database and table. mysql> desc users; +----------+-------------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +----------+-------------+------+-----+---------+-------+ | username | varchar(50) | YES | | NULL | | | password | varchar(80) | YES | | NULL | | | email | varchar(20) | YES | | NULL | | +----------+-------------+------+-----+---------+-------+ 3 rows in set (0.00 sec) If the table is already there, but deviates from the CFEngine promise, CFEngine will only warn you about this. This is because it could be dangerous to automatically start dropping table columns. If you would nonetheless like to see support for altering tables, please add a comment or make a feature request at [the CFEngine bugtracker](http://bug.cfengine.com "The CFEngine bugtracker"). The complete policy for the above is [available for download](https://github.com/cfengine/core/blob/460255133160baa49a92dad9ed04b8e5b029c116/examples/sql_table_structure.cf "CFEngine policy for SQL management"). ## User-experience enhancements As I wrote in my [introductory blog post](https://cfengine.com/blog/meet-the-cfengine-team-eystein), there were two issues I wanted to see fixed. The first one relates to the automatic reloading of policy for the CFEngine scheduler, cf-execd. This process has the important role of running CFEngine at the schedule defined in policy, and will also email you the output from CFEngine if you configure it to. In earlier releases, you had to kill and restart cf-execd if you changed its policy, such as changing the run schedule or the mail address. Needless to say, this is a quite risky operation: if cf-execd does not come back up, cf-agent does not get run and your host is no longer maintained by CFEngine. I am therefore quite happy to announce that we have now implemented automatic reloading of the cf-execd policy, just like the other components\! The second issue is related to expansion of variables in the policy. Consider the following policy. bundle common def { vars: "global_list" slist => { "a", "b", "c" }; } bundle agent test { reports: cfengine_3:: "$(def.global_list)"; } This now correctly expands, without the need to copy the list to a local scope: $ cf-agent -Kf ./global_list_expansion.cf R: a R: b R: c You can test this yourself with the [policy found here](https://github.com/cfengine/core/blob/460255133160baa49a92dad9ed04b8e5b029c116/examples/global_list_expansion.cf "CFEngine policy for global list expansion"). The file cf\_promises\_validated that is being created in masterfiles (see the [3.1.2 change log](http://www.blogcompiler.com/2010/12/29/cfengine-3-1-2-extended-change-log/ "CFEngine 3.1.2 extended change log")) is now filled with a date-string when the policy changes instead of being touched. This is because touching the file as a means for detecting policy updates is more fragile if clocks are not synchronized across the policy server and clients. You will also see that we have changed from u\_dcp to u\_rcp in the failsafe.cf policy that we bundle with CFEngine, and I suggest that you also change your policy if you currently use the u\_dcp body. Furthermore, we changed the default failsafe.cf u\_rcp body to have trustkey set to false instead of true. This is because this policy is run after the client is bootstrapped, so there is no need for the client to accept new keys at this stage, tightening security further. You can find all these policy enhancements in /var/cfengine/masterfiles when you install one of the packages we provide, or in [masterfiles in the CFEngine Community github repository](https://github.com/cfengine/core/tree/master/masterfiles "CFEngine github masterfiles"). We believe these improvements will make CFEngine behave more to the user expectations and relieve beginners from unnecessary pains. ## New classes, functions and variables Quite a few useful new variables are made available, most notably the $(sys.last\_policy\_update) which tells you the timestamp of the most recent policy change at the client. In the following example, we use the variables and show how they expand. bundle edit_line my_motd { vars: "interfaces_str" string => join(", ","sys.interfaces"); "ipaddresses_str" string => join(", ","sys.ip_addresses"); insert_lines: "This system is managed by CFEngine. The policy was last updated on $(sys.last_policy_update). The system has $(sys.cpus) cpus. Network interfaces on this system are $(interfaces_str), and the ip-addresses assigned are $(ipaddresses_str)."; } This results in the following file on an example system: This system is managed by CFEngine. The policy was last updated on Fri Mar 23 15:03:02 2012. The system has 4 cpus. Network interfaces on this system are wlan0, virbr0, and the ip-addresses assigned are 192.168.1.16, 192.168.122.1. The new and useful functions dirname(), lsdir(), and maplist() have also been implemented. ## Improvements to the embedded-db code CFEngine relies on an embedded db library to save state across runs, such as when cf-serverd and cf-agent makes connections to remote systems (aka. the last-seen database). However, it turned out that the db access would become a bottleneck on very busy systems, because CFEngine was storing information on every connection. By very busy, I mean around a thousand connections a minute. We needed to improve on this, because a thousand connections per minute is not close to what we want CFEngine to scale to. So we refactored the embedded database code, to allow a much higher degree of parallelisation and adjust the database structure to make the query patterns much more efficient. We also introduced self-healing of broken database files: they will now get moved away and recreated. To standardise CFEngine installations, we dropped support for the BerkeleyDB and the more experimental sqlite backends. As we announced this, Oracle actually [reached out](https://cfengine.com/forum/read.php?3,25234,25306#msg-25306 "CFEngine forum post by Oracle") and wanted to ensure continued support for BerkeleyDB in CFEngine. We will see if they offer help to maintain the backend. The remaining tokyocabinet and quick dbm backends are much faster and safer to use. Use tokyocabinet if you can, but if this does not compile on your system, you can use the more compatible quick dbm library instead. All our binaries are built with tokyocabinet. ## More details There are just too many improvements to cover in the 3.3.0 release, so we could only cover the highlights. There are some more details in the [ChangeLog-file at CFEngine's github repository](https://github.com/cfengine/core/blob/master/ChangeLog). Also, the bugtracker contains a list of all [issues that have been resolved for CFEngine 3.3.0](https://cfengine.com/bugtracker/changelog_page.php "Issues resolved for CFEngine 3.3.0"). ## Download today\! As usual, we relieve our community users with the hassle of compiling their own binaries, by providing free binaries for the following platforms. - Debian 5 and 6 - Fedora 14 and 15 - Red Hat Enterprise Linux 4, 5 and 6 - Suse 9, 10 and 11 - Ubuntu 8, 9, 10 and 11 Free registration at the engine room is required to [get the binaries](https://cfengine.com/downloads "Download CFEngine binaries"). These binaries are built according to CFEngine standards, all dependencies are embedded and support for the new SQL and virtualization promises is included. Download, install and test within minutes\! -------------------------------------------------------------------------------- title: "Meet the CFEngine Team - Diego Zamboni" url: /blog/2012/meet-the-cfengine-team-diego-zamboni/index.md date: "2012-03-19" --------------------------------------------------------------------------------
![](images/DiegoZamboni600x450-37d9cdd1.jpg)
Hello - my name is Diego Zamboni, and I am very happy to have this opportunity to introduce myself. If you read the [CFEngine forums](https://cfengine.com/forum/), you have probably seen me around, but it's nice to be able to step away from official business for a moment and just tell you about myself. I work for CFEngine AS since October of 2011, and my official title is "Senior Security Advisor". In that capacity, I work on advocating CFEngine as part of the security toolset that any sysadmin should have, and also on providing internal guidance to make CFEngine even more powerful as a security tool. I have recently also started working on interacting with, promoting and nurturing the CFEngine user community. It is often me who posts in the [@CFEngine\_news](https://twitter.com/#%21/cfengine_news) twitter account, in the CFEngine [Facebook](https://www.facebook.com/pages/cfengine/311003700627?ref=ts) and [LinkedIn](https://www.linkedin.com/groups?gid=136574) pages, and I'm often found on the [\#cfengine](https://web.libera.chat/?channel=#cfengine) IRC channel as well. I very much enjoy interacting with people, writing, public speaking and teaching, so I am very happy in this role as well. I have been interested in computing and technology for as long as I can remember. My first computer was the venerable [Timex-Sinclair 1000](https://en.wikipedia.org/wiki/Timex_Sinclair_1000), which I got when I was 11 years old (and which came with an amazing 2kB of RAM\!). Later I spent many, many hours playing with, tinkering with, and programming my [Commodore 64](https://en.wikipedia.org/wiki/Commodore_64), and later a [C128](https://en.wikipedia.org/wiki/Commodore_128). In fact, my first technical publication was in COMPUTE\!'s Gazette, which published (in its September ‘91 issue) a program I wrote for cataloguing floppy disks. When I was in [college](https://unam.mx/) studying computer engineering, I got a job as a sysadmin at the university's [supercomputer center](https://en.wikipedia.org/wiki/DGSCA). I was part of the team that managed both the Cray Y-MP4/464 supercomputer (how technology has evolved\! That big, expensive supercomputer had 512MB of RAM, 1/16th of the laptop in which I'm typing these words) running [UNICOS](https://en.wikipedia.org/wiki/UNICOS), and many other Unix systems running [Ultrix](https://en.wikipedia.org/wiki/Ultrix), [Irix](https://en.wikipedia.org/wiki/IRIX), [SunOS](https://en.wikipedia.org/wiki/SunOS), [NeXTSTEP](https://en.wikipedia.org/wiki/NeXTSTEP), and other operating systems. It was during these days that I first started thinking about the issues surrounding the management of a heterogeneous mix of systems, and when I first [read about](http://www.iu.hio.no/%7Emark/papers/paper1.pdf) and tried CFEngine, still back in version 1. It was also at this time that I got involved in computer security - there was a break-in into our Cray machine, and I participated in its investigation. I found the process fascinating. Afterwards I kept working in security, eventually founding the university's first incident response team and computer security group, which continue to exist to this day, in a [much more evolved form](https://www.seguridad.unam.mx/). Following my security path, I went to [Purdue University](https://www.cs.purdue.edu/), where I did my Ph.D. under the direction of [Gene Spafford](https://spaf.cerias.purdue.edu/narrate.html), one of the world's top security experts. After [graduating](https://spaf.cerias.purdue.edu/images/diego.jpg) from Purdue in 2001, I went to work at the [IBM Zurich Research Lab](https://www.zurich.ibm.com/), where I worked for eight years doing research in intrusion detection and prevention, malware analysis and containment, and other fun things. In 2009 I returned to Mexico, and in the process switched to a vastly different job as a consultant for [HP Enterprise Services](https://en.wikipedia.org/wiki/HP_Enterprise_Services). In 2010 I started writing, in my free time, a book about CFEngine. I had realized that CFEngine 3 was somewhat lacking in documentation (being relatively new, and very different from CFEngine 2), and I had been dreaming of writing a book for a long time, so the idea and the timing seemed right. Eventually I got my book proposal accepted by [O'Reilly](https://oreilly.com/). As a result of this, I started a conversation with [Mark Burgess](https://cfengine.com/markburgess/), which eventually led to me getting a new job\! I couldn't be happier: I am working with a technology that I love, and surrounded by amazing and brilliant colleagues. As for my book, titled "[Learning CFEngine 3](https://cf-learn.info/)", I am happy to say that, as you read this, it should be available for sale [from O'Reilly](https://shop.oreilly.com/product/0636920022022.do) and [amazon.com](https://www.amazon.com/Learning-CFEngine-3-Diego-Zamboni/dp/1449312209). If you have been looking to learn about CFEngine, or want to reinforce your understanding of how it works, please check it out\! I hope you will enjoy reading it as much as I enjoyed writing it. So, there you have it, now you know a little more about me. I would love to hear from you. Leave a comment here, or contact me on twitter, where I am [@zzamboni](https://twitter.com/zzamboni). I also keep a personal [website](https://zzamboni.org/) and [blog](https://zzamboni.org/post/). See you around\! -------------------------------------------------------------------------------- title: "NEW! CFEngine 3 Nova Evaluation Program" url: /blog/2012/new-cfengine-3-nova-evaluation-program/index.md date: "2012-03-19" --------------------------------------------------------------------------------
![](images/novaeval-55ff341a.jpg)
CFEngine is happy to see increased demand for our software, both the Open Source version, **[CFEngine Community](https://cfengine.com/community "CFEngine Community")**, and the Enterprise Software version, **[CFEngine 3 Nova](https://cfengine.com/enterprise "CFEngine 3 Nova")**. We recently moved the CFEngine Community source code to **[GitHub](https://github.com/cfengine "CFEngine Core on GitHub")** and plan to use that platform to interact more closely with the general community (watch this space for hot topics to come\!). We have also put up a test drive environment as a part of making CFEngine 3 Nova more accessible to potential customers. Qualified users\* will be able to access and test a fully operational version of our Enterprise Software for free, without having to install and set it up on their own system. Specifically, users will be able to: - View the status of all servers (e.g. availability, compliance, performance) - Run one of many out-of-the-box reports, such as: software installed (by machine), compliance, file diffs, promises not kept - Experiment with CFEngine Policy files - Drill-down and configure individual servers, files, services, etc. Get shell-access for manual verifications For a sneak preview, check out our video of the evaluation environment [here](https://cfengine.com/evaluation#video "CFEngine 3 Nova Evaluation Video"). We have also created a [written guide](https://cfengine.com/manuals/novaevaluationguide "CFEngine 3 Nova Evaluation Guide") to take you through the steps of your test drive and explain the different reporting capabilities of **CFEngine 3 Nova**. Corporate customers may also request **CFEngine 3 Nova** binaries to install on their own system for a test drive. Please [contact us](https://cfengine.com/contact "Contact CFEngine") for more information. Both test drive options give access to our commercial support ticketing system to help you on your way. \***CFEngine 3 Nova** is an Enterprise Software for corporations, and is not meant for private use, please see conditions on the [evaluation page](https://cfengine.com/evaluation "CFEngine 3 Nova Evaluation"). -------------------------------------------------------------------------------- title: "CFEngine is now on GitHub" url: /blog/2012/cfengine-is-now-on-github/index.md date: "2012-03-19" --------------------------------------------------------------------------------
![](images/github-cfengine-9e609ebe.png)
We are extremely happy to announce that the source code for [CFEngine Community](https://cfengine.com/community) is now [hosted on GitHub](https://github.com/cfengine/core/), one of the largest code-hosting services in the world. This move has many important advantages for both the CFEngine development team and for users. For one, it marks the switch from subversion to git as the source code management system, allowing for a much more agile development cycle and the use of more advanced tools. Being on GitHub allows users to more easily find the code and [follow its development](https://github.com/cfengine/core/commits/master), makes it very simple to check out the source code, and to communicate with the development team. [Check it out](https://github.com/cfengine/core/)\! -------------------------------------------------------------------------------- title: "Ten Reasons for 5-minute configuration update and repair" url: /blog/2012/ten-reasons-for-5-minute-configuration-update-and-repair/index.md date: "2012-03-19" --------------------------------------------------------------------------------
![](images/10reasons-2c132392.jpg)
How often should your configuration management system verify the integrity of your system? The default choices we've made by CFEngine are the results of almost 20 years of research into this area. Below you will find ten issues and references that explain why these choices are underpinned by the science. These ten things really all amount to the same thing: if you are playing ping-pong against the adversary of change, you need to be as quick on your feet as your opponent -- and faster 1. **Lost time is money**: Down-time of services, security related incidents and application mis-configurations that lead to incorrect behaviour, can all mean lost revenues. A fast response to misconfiguration, or better still *avoidance* in the first place, can be accounted for in real money. With a 5 minute check on system integrity, coupled with the ability to make repairs to thousands of machines in parallel, a company could save hundreds of thousands of dollars in lost revenue. This one isn't hard to understand, and doesn't need much science to argue its case -- but, at CFEngine, that doesn't stop us from being sure\! No surprises that it turns out that the human factor is the major cause of downtime. (See, for instance, [A Simple Way to Estimate the Cost of Downtime](https://static.usenix.org/event/lisa02/tech/full_papers/patterson/patterson_html/)) 2. **Minimize the risk of catastrophic loss before a change window**: Many industries try to limit the scope for human error by having change-windows during which all alterations and repairs must be made. The obsession with tracking who to blame for deviations from policy (liability) often overshadows the need to avoid deviations in the first place. If you are worried about the *risk* of system maintenance on service performance, take heart. Studies show that you can optimize (i.e. minimize) the risk of damage to a system, even with a small change window, if you arrange it to start repairs just after the peak of fault activity. But better still is to keep the deficit of unrepaired faults low in the first place with regular policy-approved maintenance. CFEngine 3 can easily do this because it has very little impact on systems. See [A Risk Analysis of Disk Backup or Repository Maintenance](https://www.sciencedirect.com/science/article/pii/S0167642306001262). 3. **A Red Queen Race - running to stand still**: In Biology, and in IT management alike, there is the concept of running to stand still. Lewis Carrol parodied this with the help of the Red Queen in *Through The Looking Glass*. > *"Well, in our country," said Alice, still panting a little, > "you'd generally get to somewhere else if you run very fast for a > long time, as we've been doing."* > > *"A slow sort of country\!" said the Queen. "Now, here, you see, > it takes all the running you can do, to keep in the same place. If > you want to get somewhere else, you must run at least twice as > fast as that\!"* The point is that, in evolving dynamical environments, change and counter-change must be matched, and must co-evolve to maintain a dynamical stalemate. If a configuration system is too slow in matching the moves of a fault-ridden environment, it will fail to maintain a system. Today, we continue to focus only on *build and deployment*, thanks to the ballooning scale of IT systems everywhere, it has become normal to pretend that system configurations remain constant after initial installation. In truth, a system is constantly degrading, like any paint job exposed to the weather. To maintain a dynamical balance (equilibrium) with the environment of users, we have to work fast enough to counter the \`weather' (*maintenance and audit*). If we then want to shift the balance for strategic gain, we need to make sure we can buy ourselves the time to do that, without being dragged back by firefighting (see maintenance theorem [On the Theory of System Administration](https://www.sciencedirect.com/science/article/pii/S0167642303000315), or the more detailed discussion in [Analytical Network and System Administration](http://www.amazon.co.uk/Managing-Human-Computer-Networks-Network-Administration/dp/0470861002/ref=sr_1_fkmr3_2?ie=UTF8&qid=1331982421&sr=8-2-fkmr3)). 4. **Be proactive to extend the Mean Time Before Failure (MTBF)**: A few years ago it was popular to speak about \`five-nines reliability' for all kinds of systems, meaning that the Mean Time Before Failure was very long\! Running preventative maintenance allows you to extend the MTBF almost indefinitely. Most problems can be avoided if we are quick. Wait too long and you allow a misconfiguration to occur; then an attempted roll-back might not even help you. Once the horse has bolted, it doesn't help to lock the barn door. Proactive maintenance is the answer to living in a dynamical social network of users and machines. Like washing your car to prevent it from rusting: negligence is corrosive to the system. (See [A Probabilistic Approach to Estimating Computer System Reliability](https://static.usenix.org/event/lisa01/tech/apthorpe/apthorpe_html/index.html)) 5. **Simplify scheduling by eliminating cron**: With 5 minute execution resolution, you can do away with configuring clunky cron altogether, for practically all applications. Instead of trying to configure as many schedulers as you have machines (something that requires changes of every machine's crontab individually), just use CFEngine 3's universal scheduler to offer a single point of schedule control through policy, giving much greater precision to select jobs for the right machines at the right times, and adding features like load-balancing to boot (see [Scheduling and Event Management](https://cfengine.com/manuals/st-schedule)). 6. **Tracking your true system behaviour**: Most configuration tools assume that computer systems are actually static: i.e. once deployed, a machine remains constant until an error is detected. Naturally this is wrong. Computers are changing very quickly all the time thanks to login and service usage. Suppose you want to audit change -- to track what is going on, you can sample the state Q of the system every dT seconds. If you want to know the rate (speed) of system change, then you need to sample twice as fast (2 points in this interval), i.e. dQ/dT = (Q2 - Q1)/dT. Similarly if you want to know whether the trend is increasing or decreasing, you need to sample twice as fast again: d2Q/dT2 = (dQ2 - dQ1)/dT2, so you need four times the sampling, or four times the rate of configuration verification. Nyquist's theorem says basically the same thing: to capture a variation properly we then have to sample the system at twice the frequency of the fastest Fourier component. This is why CFEngine 3's monitoring daemon samples every 2.5 minutes to capture 5 minute changes. That still only allows us to estimate trends over 15 minute intervals. Imagine, checking configuration at a slower rate\! If you only sample every hour, say, then you would have to wait 4 hours to see any pattern, and you would miss anything that happened in between (see [Nyquist-Shannon sampling theorem](https://en.wikipedia.org/wiki/Nyquist-Shannon_sampling_theorem)). 7. **Competition for system resources is a zero-sum game**: One of the reasons for an inability to function correctly is that multiple processes compete for resources, like trying to find a parking space on a Saturday morning at the mall. If any process goes out of control, consuming system resources (too much memory, CPU, or disk), then regardless of the intent, your system's integrity is basically under attack from the users. A resource consumed is a resource unavailable. The effects of this competition can be analyzed with game theory. Having a timely \`tit for tat' response for keeping a share is the best approach, just like a duel or a gunfight. Garbage collection is an obvious example -- something we take very much for granted until the garbage collectors go on strike. You need to be quick on the draw against a sudden attack, and you need to bail out buckets of water quickly on a sinking ship. Both of these are reasons to verify system integrity quickly and often (see [On the Theory of System Administration](https://www.sciencedirect.com/science/article/pii/S0167642303000315), or the summary in [Analytical Network and System Administration](http://www.amazon.co.uk/Managing-Human-Computer-Networks-Network-Administration/dp/0470861002/ref=sr_1_fkmr3_2?ie=UTF8&qid=1331982421&sr=8-2-fkmr3)). 8. **Systems are getting faster and therefore go out of control faster**: Unintended change occurs as a result of the processes that drive the system. Computers are getting faster every year, and thus errors can spiral out of control more explosively. Even if root causes only happen at the same rate, the avalanche of side-effects will expand at the fastest rate of the system. The fastest changes are the ones that we struggle to capture: - Servers: network traffic - Desktops: user behaviour and peripheral speeds - Mobiles: frequency of user activation - Compute nodes: disk, CPU and memory rates Network connections are throttled by typically 5-minute TCP time waits, so a 5 minute time matches well -- though web servers are often optimized for even faster turnaround cycles. Other system auto-correlation studies show significant resource changes at the time scale of 20 minutes and less. From the rates of change researchers and engineers can measure, to capture faults in a mission critical environment, being no slower than a 5 minute response is conservative (see [Measuring System Normality](http://dl.acm.org/citation.cfm?id=507054&dl=ACM&coll=DL&CFID=90423914&CFTOKEN=36362432)). 9. **Accurate because you can be (with CFEngine 3)\!** CFEngine's non-imperative, self-organizing execution optimizes at a faster rate of throughput than an imperative script for certain resource-intensive operations. That's because, by dropping a strict serial order, it has fewer constraints on its scheduling. Thus CFEngine can handle faster repair rates than serial tools. Intensive resource accesses, like scanning large filesystems and reading the process table (perhaps multiple times) are inherently serial in nature, etc, but only need only be executed once, in batch, for all similar configuration promises. It's rather like walking only once around the supermarket to do your shopping -- you can pick up what you need in any order, as you see it, instead of following your shopping list imperatives slavishly and having to walk back and forth across the store multiple times (see [Shortest Path Problem](https://en.wikipedia.org/wiki/Shortest_path_problem)). 10. **Because, with CFEngine 3, 5 minute updates are possible for as many as 5000 hosts per server**: CFEngine technology consumes very little CPU, or memory, and scales with little hardware -- i.e. scaling CFEngine is cheap. Even with a centralized architecture and a single hub/policy server, you can have 5 minute updating with a single server for up to 5000 machines. You can scale any technology you like, by brute force, (even fixing by hand\!) -- but you can't do it as consistently, reliably or as cheaply (for Total Cost of Ownership) than you can with lightweight automation (see [Scale and Scalability](https://cfengine.com/manuals/st-scale)). #### It's simple engineering The maintenance theorem is rather simple: it says that if something is attacking you quickly, you need to defend yourself at the same rate to stand a chance. At CFEngine, we began with daily updates in 1993, then hourly updates in 2004, then we went to 15 minute updates around 2000 and we are currently resting at 5 minute updates since around 2005. We think that 5 minute resolution is a compromise sufficient for most users at today's system rates, but we also know of a few that schedule checks every minute. Of course, not every maintenance task needs to be carried out every 5 minutes. Performing a full system backup would not even be possible at this rate (without mirroring technologies), but simple tasks like garbage collection and process integrity need to be checked often to avoid catastrophic loss. All of the points above are different ways of saying the same thing: "Speed Is The Product" (see [What makes clouds float and developers operative? Agility\!](https://cfengine.com/company/blog-detail/what-makes-clouds-float-and-developers-operative-agility/)). Try it for yourself. Does your configuration system run fast enough? -------------------------------------------------------------------------------- title: "REST API for Nova" url: /blog/2012/rest-api-for-nova/index.md date: "2012-02-20" --------------------------------------------------------------------------------
![](images/REST-79aabff0.jpg)
One feature our customers have requested for the next version of [CFEngine Nova](https://cfengine.com/nova) (available before summer 2012) is having the ability to collect reported data through a **REST API**. It is already possible to get at reporting data through the Mission Portal web-interface, as well as the command-line client cf-report. However, as we want to provide our customers with greater flexibility, we are now introducing the addition of REST (HTTP) as another interface. REST-based APIs are convenient for a number of reasons. First, they are commonplace in the industry and many people already know how they work. Secondly, they may integrate more easily with other tools that can call HTTP interfaces. In our first release of the REST API, we provide most of the reporting data available in the Mission Portal in read-only mode. Included are: - Class usage reports. - Basic information about hosts and when they were last seen. - Promise logs, summaries and compliance status. - Running setuid programs. - Installed software. - State of variables at hosts. - File changes and diffs. The API is versioned so that it may be extended or changed in the future based on customer feedback. It currently supports JSON as its message format. To get an idea of what you might use the REST API for, suppose you wanted to transfer information about promises not kept from Nova into some third-party reporting system. You could use an ETL tool like Pentaho Kettle to automate this process. Let's set up a simple transformation pipeline. No coding required\! 1. First, extract the data from the Nova REST API using an HTTP input loader. 2. Next, transform the data using a JSON input step and select the relevant fields. Alternatively, we could use a Javascript transformation for more fine-grained control. 3. Finally, push the resulting JSON document using an HTTP post step with the url of the third-party reporting service. ![REST API in CFEngine 3 Nova](images/CFEngine-REST-API.png) While this example is simple, it is possible to use [Pentaho Kettle](https://kettle.pentaho.com/) to combine several results from the REST API to make custom reports, as well as converting the data to different formats and write the output to multiple places. We hope that you will find the REST API a useful addition to Nova and that it will provide another avenue for integrating CFEngine with your existing infrastructure. Let us know how we can improve\! -------------------------------------------------------------------------------- title: "Meet the CFEngine Team - Eystein Måløy Stenberg" url: /blog/2012/meet-the-cfengine-team-eystein-maloy-stenberg/index.md date: "2012-02-14" --------------------------------------------------------------------------------
![](images/IMG_9464.JPG-f232dbe9.jpg)
My CFEngine journey started in 2008, when I was still a student. I was working on my master's degree in Mathematics, specialising in cryptography at the [University of Tromsø](https://uit.no/inenglish). I had moved to Oslo to write my master's thesis in cooperation with the [Norwegian Defence Research Establishment](http://www.ffi.no "Norwegian Defence Research Establishment"). At the same time, I took a course in Business leadership at the [University of Oslo](http://www.uio.no/english/ "University of Oslo"). As part of this course, we got the opportunity to be interns at technology start-ups in Oslo. My start-up of choice was, of course, CFEngine\! At the time, CFEngine had been an open source project for 15 years, and had a massive user-base. A couple months later, I was very excited to accept an offer as a software developer for the new company. Back then, CFEngine only had four employees: Mark Burgess (CTO), Thomas Ryd (CEO), Nakarin (Jeang) Phooripoom, and me. Since Mark was busy handling so many aspects of CFEngine, I had the responsibility to make the CFEngine software shine. It was quite a shock to go from being a student to suddenly developing software that is running on millions of computers across the globe\! My first main projects with the company were to port [CFEngine 3 to Windows](https://cfengine.com/blog/2009/cfengine-3-on-windows/), and add support for [POSIX and NTFS file ACLs](https://cfengine.com/manuals/NovaOwnersManual#File-Access-Control-Lists). These features were incorporated into the commercial [CFEngine Nova product](https://cfengine.com/nova), which had its initial release in April 2009. Since the early days, many things have changed a lot. We took on the first round of investment in February, 2011 from the [Norwegian investor Ferd](https://cfengine.com/press_releases/cfengine-investment). The Oslo office has moved locations three times to accommodate for the rapid expansion, and we have opened a new office in Palo Alto, California to meet the very strong demand for our products and services in the US market. We are now around 30 people working at CFEngine -- representing 12 nationalities (and still growing)\! I took the opportunity to move to the United States in May 2011, and now live in San Francisco. My new job is to meet customers that run CFEngine at scales of tens of thousands of hosts. I learn about the challenges that they face, suggest ways we can help them, and report their issues back to the development team. This is both interesting and challenging. In addition, I get to travel in the US, and see a lot of different cities and regional cultures. This country is so diverse\! I like to think of the US as a continent rather than a country. Indeed, it has more than 60x the population of Norway. We now have a team of software developers, but I still get to do development on CFEngine -- both the community and commercial versions. In the past few months, I have been optimising the Nova code, to push the scalability of CFEngine 3 Nova even further. We can now support thousands of hosts per Nova hub, with a five-minute update interval both for policy and reports. No other product can achieve anything close to this. As part of this work, I have been pushing the database technology we are using to the limit. All the reports in CFEngine Nova are stored in [MongoDB](https://www.mongodb.org/) -- an open-source document-oriented database. It turns out that [10gen](https://www.10gen.com), the developers of MongoDB, also have an office here in Palo Alto. I am visiting them regularly to understand how we can scale even further, without needing to throw more hardware at the solution. I am also working on incorporating Role-Based Access Control (RBAC) into the CFEngine Nova Mission Portal (GUI dashboard). We expect that this will make it into the Nova release due in spring 2012. I plan to write more about the details of RBAC in another blog-post. We are also planning to incorporate RBAC into the CFEngine language itself (the input side) in the not-too-distant future. Two larger topics on my mind that I'd like to see addressed in CFEngine are the global list-iteration issue ([https://cfengine.com/manuals/cf3-reference\#List-variable-substitution-and-expansion](https://cfengine.com/manuals/cf3-reference#List-variable-substitution-and-expansion)), and the automatic reloading of changed policies for cf-execd. All CFEngine daemons except cf-execd are reloading their policy on change, which is a bit inconsistent and may be a surprise to new users. Both these issues are very deep and risky changes, which is why they have not been fixed sooner. However, they are high up on the priority list now, so look for updates on them in the near future. Well, that is all I have for now. If you have use-cases for CFEngine that you would like to see covered, or ideas on how we can make CFEngine even better, please leave a comment below. I will try to address them with a reply or another blog entry. I also have some CFEngine-related posts on my [personal blog](https://www.blogcompiler.com). -------------------------------------------------------------------------------- title: "CFEngine on Android" url: /blog/2012/cfengine-on-android/index.md date: "2012-01-15" -------------------------------------------------------------------------------- {{< youtube d57lm4eoFKM >}} -------------------------------------------------------------------------------- title: "What makes clouds float and developers operative? Agility!" url: /blog/2012/what-makes-clouds-float-and-developers-operative-agility/index.md date: "2012-01-06" --------------------------------------------------------------------------------
![](images/agility-77385e24.jpg)
In an office, high above New York City, we are looking at the screen of a computer, discussing how the recent blog entry on [CFEngine, SysAdmin 3.0 and the Third Wave of IT Engineering](https://cfengine.com/blog/sysadmin-3.0-and-the-third-wave) applies to the challenges of institutional agility on Wall Street and beyond. \`Speed *is* the product,' says R. R is a CFEngine customer from a large, and heavily regulated organization, that has been choked with bureaucracy and process management. He has just explained how deploying a server has gone from taking 3-6 months, to taking just a few minutes, thanks to his new CFEngine-based process. The present moves faster than the past, and the future moves even faster than that. By casting off by-gone vestiges of industrialized mass production in favour of Third Wave individual customization, he has turned a bureaucratic mess into an inexpensive triumph. #### A prehensile tale of evolution Say the word \`agility' and my mind jumps to images of monkeys swinging from tree to tree, or along vines in the jungle. I associate agility with sufficient speed and pliability to master one's environment. Is that what we mean in IT infrastructure? In IT, the challenge of the environment is the jungle of changing demand, business imperatives, and the striving for competitive fitness in a cut-throat marketplace; it is network requests to servers, spanning a landscape of different operating systems and their curiosities; it is a backlog of security patches and bug-fixes that need to be navigated, without losing balance. Superficially, this is quite different from the primate jungle, but mastering this other world is still about having sufficient speed and dexterity. Agility is a function of our surroundings as well as our own capabilities. We cannot achieve agility without the tacit \`cooperation' of the environment. For a long time, system administration assumed that systems were not dynamical entities -- they were set up once and for all, and nothing really changed, except for the personal data you were working on. Today, we cannot turn away from the dynamical aspect of systems -- change is the norm, and we use IT in ways we never did in the past. All of this demands much more from infrastructure engineers. Van Doesburg said,\`every machine is the spiritualization of an organism'; today it is absolutely true. For Web companies, like social media sites that are the embodiment of entire systems, designed to bring new features at a rapid pace, the organism doesn't stand still for long. The need to be ahead in the game drives a scheme of \`continuous deployment', or what is being called \`DevOps' today. In biology, this running to stand still in the market has come to be called \`the Red Queen' hypothesis, from Alice through the Looking Glass: > "Well, in our country," said Alice, still panting a little, "you'd > generally get to somewhere else if you run very fast for a long time, > as we've been doing." > > "A slow sort of country\!" said the Queen. "Now, here, you see, it > takes all the running you can do, to keep in the same place. If you > want to get somewhere else, you must run at least twice as fast as > that\!" #### Knowledge and Dexterity For our Wall Street customer, the bleak Second Wave regulatory management techniques were trampling the natural flora of requirements that struggled to grow out of different subcultures in their organization. The ability to embrace that diversity not only removed the human conflict in the system (by far the most expensive and damaging aspect of IT management), it empowered the organization by *exploiting* rather than wiping out the specialist knowledge of each part. To be nimble enough for a Third Wave information-rich marketplace, we see that three things are needed: - Anticipation. - Action. - Adaptation. These are all enabled by knowledge. In other words, we need to see what is coming, be able to respond quickly or even proactively to it, and finally we have to tailor that response to the challenge of the day. This is not just a matter of buying faster machines, or hiring more people. Today's problem is one of information management. Every challenge requires of us: | Challenge | Tool | | --------------------------------- | ----------------------------------- | | To comprehend the challenge | Knowledge and insight | | To solve the challenge | An expressive language | | To respond to the challenge | A rapid, scalable deployment engine | | To confirm or verify the response | Up to date and relevant reporting | As we move into this age of information, the machinery we employ to this end has to be knowledge driven, lightweight and maintainable -- it has to transcend the industrial and support the \`organic' -- because if you are oversimplifying the reality of the jungle by insisting on the regularity of a parking lot, you will have a lot of expensive construction work to wipe it out and you will bear the long-term cost of the waste. At CFEngine we designed technology to support diversity for this reason. #### From packages to promises - a language for talking about systems Granularity is the key to agility. In the early 1990s, a major stride forward was made when package-based software installation was created. Early systems like HPUX no longer had to install \`everything or nothing' from a master tape. Packages replaced the \`golden image' method of steam-rollering systems into \`die Norm', allowing customization at the level of selecting applications. Today, by comparison, packages look like quite a blunt instrument: we not only need to modify the set of packaged components, but configure and tune the internal details of how these software components operate. In short, we must create the kind of seamless organism that the original provider could not possibly have imagined. This is what creates fitness-for-purpose in today's IT marketplace. Such creativity requires very specific adaptations that tie together an ecosystem of contributing elements: web service, directory service, storage management, security, etc. Pre-packaging fails the agility test as the details of local needs cannot be known in advance by the operating system vendor, and pre-packaged. A simple analogy makes it clear: a tin of soup, or a microwave meal might be a superficially fast or easy way to make dinner in a modern age, but this ease does not bring agility. The day you get a visitor with special dietary requirements (vegetarian or allergic etc) then the prepackaging is a major obstacle to adaptation: the recipe cannot be changed and repackaged without going back to the factory that made it. Thus packaging as a strategy, or even as a menu of packages, is merely an over-simplification, and we end up having to work around the technology with an intervention. For instance, to order the perfect steak, you need a *descriptive language* to explain the correct level of cooking, the side orders, and the choice of wine, etc. You neither want to settle for a prepackaged dinner, nor do you want to go into the kitchen and play with the utensils to make the food yourself. ``` # The CFEngine food replicator food: table_1.seat_3:: "lamb steak" cooked => "medium rare", seasoning => { "rosemary", "salt" }; table_1.seat_2:: "beef steak" cooked => "bloody", seasoning => { "salt" }; ``` CFEngine's modelling language offers this kind of surgical, precision-control over the smallest ingredients in a configuration, and even allows you to package your own containers or work with other suppliers' packages. This ensures that adaptability is not sacrificed for superficial ease. Rather than working with tins of soup, CFEngine deals with the raw ingredients to whip up a meal easily -- something like Star Trek's food replicators. Of course, this means you need to know a few basics about cooking to set up the initial choices, but once you do, you have a lasting investment that is agile and cheap for the long term. #### Cloud-hopping: from demolition to renovation IT-agility has found a natural partner in virtualized machine services, where capacity itself can be adjusted to changing demand on a continuous basis. For many companies, especially in the United States, public or private clouds are a rational approach to managing such resources. The technology, however, is still in its infancy for consumers. For what seems like a refreshing antidote to the \`tear down and rebuild' methods of iron age industrialization, the Cloud does little to rescue consumers from such techniques -- it only makes them look more affordable, by allowing it to happen with greater ease. Cloud does what *paging* did for disks, and what *packet transmission* did for networks: it breaks down infrastructure into small units that can be handled cheaply in a throw-away manner. If you lose a piece, it doesn't cost the world to replace it. If there is an error, throw the piece away and make a new one. But while this \`demolish and rebuild' approach sounds easy at the level of disk sectors and network packets it is a different matter altogether at the level of a working server or a building full of people. This is not agile management -- it is just cheaper brute force, and it results in loss of service. To manage the cloud service, companies like Amazon have embraced modern information rich approaches to their own infrastructure. Ironically, what is delivered to customers is pretty much an old industrial-age tin of soup they have to make do with, or modify on their own -- but this is about to change. Configuration management in the CFEngine way can take a \`tin of soup', tap it with the magic wand of policy, and make it transform itself into the desired meal without destroying the tin, and even while the meal is being eaten\! Moreover, CFEngine is a distributed agent approach to continuous change and maintenance that can arrange for different meals for different consumers without much overhead -- an obvious answer to the problem of one-size fits all. It brings customization *from within*, altering the raw materials, or mixing pre-packaged ingredients at any level of detail. The key point is: it is a *knowledge-based* approach to customization, and that is the only way to scale up agility without expensive brute force. #### Agility is knowledge: knowledge is the product With massive scale, operational complexity and a rapid pace being the order of the day, a knowledge-based approach enables *anticipation* of change, and makes it cheap to model the present and future to *adapt* a system in real time. Knowledge is what will take configuration technologies forward, in or out of the cloud. This is why CFEngine 3 was redesigned as a knowledge engine, as much as a configuration or change engine. Configuration itself is knowledge. For system infrastructure engineers, the job is increasingly going to be about content. Infrastructure engineers will be writers and teachers, learning to express intended behaviour simply and clearly with language, keeping knowledge explicit so it can be shared amongst the developers -- who make the wheels of commerce turn -- and the infrastructure engineers -- who underpin society by making the wheels. The future of agility lies in separating the *what* (knowledge) from *how* (action), i.e. separating slow change (infrastructure) from rapid change (application). This needs sharp, lightweight tools with fast reaction times. CFEngine 3 (Community and Nova) and their continuously improving range of capabilities is our answer to this challenge, and unlike package based approaches, the scope for improving our technology is still an open vista. -------------------------------------------------------------------------------- title: "CFEngine @ USENIX LISA 2011" url: /blog/2011/cfengine-usenix-lisa-2011/index.md date: "2011-12-13" --------------------------------------------------------------------------------
![](images/cfe_booth-d122b053.jpg)
The LISA conference, run by the [USENIX organization](https://www.usenix.org), is the largest conference specifically for system administration today. It has been 25 years in the running, and has been a cultural and professional center for the field of the system administration for most of that time. LISA is a unique haven for practitioners in the field, but it was also the place where modern theory emerged, driven largely by the efforts of a small group of individuals known as the ABC of system administration: Anderson, Burgess, Couch. CFEngine Founder Mark Burgess has been attending LISA since 1997. LISA originally stood for Large Installation Systems Administration, then it became simply "LISA" and most recently the long title has been revived. This year, LISA celebrated its 25th birthday, and chose the forward looking theme of DevOps, which many believe marks a change in the way system administration will be done in the future, characterized by greater agility. CFEngineers contributed, as always, in many sessions at the conference, from tutorials to plenum talks, and exhibition hall demonstrations, prominent in their distinctive Third Wave T-shirts. #### Keynote - DevOps LISA Keynote speaker Ben Rockwood, of Joyent, gave an inspiring presentation expanding on the idea of DevOps as mere application deployment (as is often assumed in discussions of configuration management tools) to provide a [historical and philosophical perspective spanning the 20th century](https://www.youtube.com/watch?v=3KpPBnEtRj4). Rockwood's view of DevOps was more in line with CFEngine's vision of the future of system administration, bringing a wider view on the subject, [though perhaps not everyone was as enamoured as we were](https://radar.oreilly.com/2011/12/usenixs-lisa-still-for-system.html). Rockwood traded remarks with CFEngine Founder Mark Burgess in questions, commenting on Alvin Toffler's Third Wave interpretation of society and DevOps. He generously credited Mark with 'kicking off' the technologies for DevOps, though was careful to point out that DevOps is not specifically about tooling. #### Scale and scalability For CFEngine, the message this year was all about scale and complexity. For some time now, we have been expounding the importance of running CFEngine to manage infrastructure at all scales, from the very small to the very large, or full-spectrum coverage for configuration management. On Tuesday evening we were delighted to see a room full of people at our BoF session. ![Standing room only at the CFEngine BoF LISA 11](images/Standing%20room%20only.jpg) - Sigurd Teigen presented the '[CFEngine on Android](https://cfengine.com/demos/cfengine-android)' proof of concept, showing how CFEngine could run on these still immature devices. - Jon Henrik Bjørnstad showed a live demo of CFEngine running on an embedded device that was built, configured and tested within less than a week by the technical team in Oslo. ![CFEngine running on Embedded Devices](images/LISABlog4.jpg) - Jon Henrik further showed how CFEngine 3 Nova can be used to [manage a simple private cloud](https://cfengine.com/demos/private-cloud-nova) based on KVM kernel virtualization on Linux, without any special technology other than a standard Linux distribution. This improves on basic private cloud deployment tools by adding CFEngine's special brand of self-healing monitoring and repair to virtual machine instance execution. - Finally, showing that we take usability and education seriously, Joe Netzel presented a preliminary draft of a [policy wizard](https://cfengine.com/policy_wizard) - a tool that guides you to create simple CFEngine 3 policies with a graphical user interface. #### CFEngine 3 and Business Alignment We wanted to meet our audience, and we were not disappointed. On Monday we had a very successful CFEngine 3 tutorial by Mark, that got people who were still running CFEngine 2 itching to [migrate to CFEngine 3](https://cfengine.com/23nova). The second tutorial, A Sysadmin's Guide to Navigating the Business World run by Mark Burgess and Carolyn Roland (NIST) was not specifically about CFEngine, but one of the key phenomena of the Third Wave is the dialogue between commerce and information. This packed tutorial was about opening system administration to the idea of working with business, as a strategic partner, rather than merely for it as a slave. At the exhibition, LISA attendees couldn't get enough of CFEngine -- and booth attendance overshadowed most of the other booths at the exhibition. Wednesday had people coming to greet us and hang out at our booth, wanting to see the embedded device and android demos. How to migrate from CFEngine 2 to CFEngine 3 was a popular question. ![The crowd at the CFEngine Booth @ LISA 11](images/LISABlog1.jpg) #### Dipping into The Third Wave On Wednesday, Founder and CTO Mark Burgess was introduced as a 'provocative thinker' to a packed room for an invited talk session. With quirky hand-drawn cartoons, Burgess presented \`3 Myths and 3 Challenges to Bring System Administration out of the Dark Ages', and argued that some basic changes were needed to move system administration from industrial-age over-simplification to a state of true information age engineering agility. #### A meeting of minds and friends Ultimately, LISA is a technofest, where friends and colleagues meet. Many of the attendees have known each other for years, and about two thirds each year are completely new. For the CFEngine family it was also an opportunity to huddle, uniting the US and Norwegian offices, and hanging out with our friends from Opscode, NIST, USENIX and Oslo University College, as well as newcomers that we bonded with in the hallways. It's only 12 months to go before @LISA12, next year in San Diego, and there are many more conferences before then\! -------------------------------------------------------------------------------- title: "Now in Beta: New CFEngine 3 Policy Wizard" url: /blog/2011/now-in-beta-new-cfengine-3-policy-wizard/index.md date: "2011-12-08" --------------------------------------------------------------------------------
[![](images/CFEngine%20Policy%20Wizard-646b3c5d.png)](https://cfengine.com/policy_wizard)
At Tuesday night's CFEngine Meet-the-Team BoF session at LISA, Joe Netzel, one of CFEngine's talented Systems Engineers, showed off a small app that he has been building in his spare time to help new users become more comfortable developing their own CFEngine policies. In Joe's own words: *"The **CFEngine Policy Wizard** attempts to bridge the gap between established concepts that are second nature to System Administrators -- such as setting file permissions, process management and software installation -- by offering a side-by-side comparative view of the CFEngine syntax."* We've now added his CFEngine Policy Wizard to our Website at [https://cfengine.com/policy\_wizard](https://cfengine.com/policy_wizard "CFEngine Policy Wizard") so you can play with it too -- and please let us know what you think. Of course, it's typical beta software, so the usual disclaimers and no guarantees apply. For full documentation about writing CFEngine policy files, please refer to the [CFEngine Reference Manual](https://cfengine.com/manuals/cf3-reference.html "CFEngine Reference Manual"). We want your feedback\! If you have questions or comments on the Wizard, leave them below or through [our bugtracker](https://cfengine.com/bugtracker/set_project.php?project_id=7). Happy Policy Writing\! The CFEngine Team -------------------------------------------------------------------------------- title: "A Preview of 'Project Constellation': The Network of Stars" url: /blog/2011/a-preview-of-project-constellation-the-network-of-stars/index.md date: "2011-12-06" -------------------------------------------------------------------------------- ![](images/star_agent-349ff840.jpg) ***Eystein Måløy Stenberg ****provides a sneak-peek into "Project Constellation", new CFEngine Nova technology under development that will expand the universe of configuration management by integrating network, server and mobile management, as well as support both geographic diversity and massive scale. * **The Single Star** CFEngine 3 supports a wide variety of network architectures today, in environments with both reliable infrastructure and unreliable or mobile connectivity. The most common architecture is the "Star Network", where you have one central hub responsible for sharing policies and collecting reports from all other hosts. This is the default network architecture adopted by [CFEngine 3 Nova](https://cfengine.com/nova). ![tl\_files/cfengine/img\_new/blog/starnetwork.png](images/starnetwork.png) The star network is popular because it is simple and ensures that a policy is consistent and easy-to-update across the hosts. This architecture can support thousands of hosts per hub, with a five-minute update interval. However, as an organization's infrastructure grows and becomes more complex, the star network set-up needs to scale as well. As we add clients, the hub will inevitably become a bottleneck for updates and collecting report information. If hosts are located in widely geographically distributed areas (e.g. different datacenters), network latency may become an issue. Organizations are often divided into departments that desire autonomy and independence from the rest of the organization to best manage their resources - yet there are probably common business goals that they need to coordinate and share. **The United Federation** To improve upon the star network, we can create multiple stars. Each star acts as an autonomous entity within the organization. Federation and autonomy are key concepts supported by [Promise Theory](https://en.wikipedia.org/wiki/Promise_theory), developed by CFEngine founder Mark Burgess. ![tl\_files/cfengine/img\_new/blog/multistarnetwork.png](images/multistarnetwork-9e70ebbe.png) However, with potentially large amounts of isolated islands of management, how can we ensure consistency across them and get a global overview? First, we realize that consistency does not have to mean homogeneity. CFEngine does not require everything to be the same -- that is old ["2nd Wave" sysadmin thinking](https://cfengine.com/blog/sysadmin-3.0-and-the-third-wave)\! Individual star networks might choose to subscribe to a few common parts of policy, but decide most of their needs themselves. The interesting question is ... How do we get an overview of what is going on in these potentially diverse environments? ** Moving to a Network of Star Networks** CFEngine 3 Nova enables detailed reporting in star networks. Can we simply dump information from each star hub into a "superhub"? There are a couple of issues that prevent us from doing this. The technical issue we encounter is scale. The amount of data that has to travel over the network and end up in a central database (detailed information about the entire enterprise\!) will become overwhelming. Administrators of legacy configuration management solutions frequently experience the need to manually manage the size and complexity of their "CMDB" databases and constantly add hardware as they swallow terabytes of space - creating an environment that is out of control. CFEngine 3's approach is to use intelligent summaries to pre-compress raw data into meaningful amounts. Privacy is another issue. We cannot assume that the managers of the star networks will allow detailed information about their environments to get transferred outside their control. We are developing technology to solve these difficult issues, by summarizing information, but leaving clues of where you can find more details. This has the effect of both heavily compressing and anonymizing it. With *Project Constellation*, we are working on advancing our Nova technology to enable federation and aggregation of star networks. A Constellation 'Mission Observatory' would give a global overview of the entire enterprise, at a level where the information is summarized and meaningful. Smart data-mining algorithms are key to achieving this -- an important aspect of CFEngine's focus on knowledge management. **Mining for information** In the [Third Wave of IT](https://cfengine.com/blog/sysadmin-3.0-and-the-third-wave), agile administrators don't want tons of meaningless data poured over them -- they want quick access to customized views that might change from week-to-week, or person-to-person. In the future, Constellation technology would enable users to subscribe to the information they need, i.e. adapt on demand to business goals, without massive overhead. Virtual views would allow users to subscribe to reports about just the promises that they are interested in. For instance, Web teams may only be interested in the configuration and running of the Web service and its content -- something which might be distributed all over the organization. We are also developing a new smart diagnostic engine to be a part of the CFEngine Knowledge Map, to help experienced and novice users navigate and learn new aspects about the system. **Want to be a part of this New Frontier?** We want to work closely with users to test and improve this coming technology. If you work with a large, distributed IT infrastructure and would like to learn more, please let us know at . We believe *Project Constellation* will further Nova's position as the most advanced configuration management solution available -- something to invest in for the decade, not just the next year. Keep your telescopes trained on [CFEngine](https://twitter.com/cfengine_news) for more news\! -------------------------------------------------------------------------------- title: "New Book: CFEngine 3 Beginner's Guide" url: /blog/2011/new-book-cfengine-3-beginners-guide/index.md date: "2011-12-02" --------------------------------------------------------------------------------
![](images/cfe3_beginners_guide02-554f45cf.jpg)
***Editor's Note: Packt Publishing has just come out with a new title, "[The CFEngine 3 Beginner's Guide](https://web.archive.org/web/20140802044056/http://www.packtpub.com/cfengine-3-beginners-guide/book)". In this guest blog, the book's author, [Rajneesh](https://web.archive.org/web/20130407023324/http://www.packtpub.com/authors/profiles/rajneesh), gives an overview of the subject, his motivations, and its intended audience. * *The publisher kindly offers blog readers access to a free e-chapter and a discount for purchasing the book.* ***\[The opinions expressed in this blog are the writer's own. CFEngine AS does not endorse or validate its content.\]*** CFEngine 3 Beginner's Guide About the book** CFEngine 3 Beginner's Guide is the only book dedicated to CFEngine 3, as of now. It covers all the aspects of installing, configuring and using CFEngine 3. I have tried to build this book upon tasks which System Engineers or System Administrators or anyone maintaining large or small heterogeneous networks perform on a daily basis. The effort we put in maintaining a sane state of systems is huge and CFEngine 3 is one such framework which may be used to make our lives easier. The book starts with examples for automating small day to day tasks such as configuring virtual IP addresses on the NIC, checking status of services and moves on to configuring and auditing the systems. As we move on we see how CFEngine may be used for monitoring systems and applications. I have also tried to explain how CFEngine may be used for reading logs and alerting if there is an anomalous system state. The last sections of the book dive into configuring your cloud infrastructure and the best practices. I have also included a small section of the CFEngine 'Nova' the enterprise version of CFEngine which provides enhanced monitoring, visualization and configuration utilities. All throughout the book, I have tried to explain the philosophy of ‘convergence' which forms the base of CFEngine. **What you will learn from this book?** The book is organized in a way so that, you learn installation, configuration and usage of CFEngine in a step-by-step manner. After going through the first two chapters you will be ready to install and configure CFEngine server and client. Post this, the third chapter dives into using CFEngine for system audits. Further chapters deal with various other aspects of System life cycle management where CFEngine may ease our job. Few such aspects are 1\. Security Audit or maintaining a "secure" system state. 2\. Using CFEngine as an advanced task scheduler. 3\. Logging and reporting using CFEngine 4\. Using CFEngine in advanced workflows to execute independent or interdependent tasks. After you learn how CFEngine may help us in complete system life cycle management, you will try to standardize a lot of such tasks and this is where the chapter on best practices is handy. It lists a few of the easy to follow best practices such as use of generic templates, use of a version control system, using CFEngine inbuilt functions and variables etcetera which help CFEngine to perform optimally. Apart from this I have also listed a few points to be considered while writing the promises so that they are easily understood by the team. **Who is this book written for?** I am sure the book will be useful for System Engineers and System Administrators who deal with a large number of systems on a regular basis. There are a lot of tasks which are repetitive and need to be performed on a large number of systems simultaneously or in an order - CFEngine can take care of both. We may execute tasks simultaneously or in an order using CFEngine. The book is also useful for folks who maintain the standard stack for all the hosts in a network. They may use the CFEngine framework not only to deploy the standard stack on large number of hosts but also to audit the hosts and check for anomalies. The framework is so powerful that it may be extended to perform all sorts of automation tasks, one such example is - backup administrators may configure backups using the CFEngine framework. **Why did I want to write this book?** When I started using CFEngine 2 a few years back and found it very useful. Although, version 2 had its own set of limitations, it served my purpose. And then came out the all new CFEngine 3 with loads of changes, functions and variables and learning it was again an uphill task. While I was going through the learning curve, I always looked out for literature which may help me understand the basics and the usage with easy to follow examples and explain the "whys" and "hows" of the CFEngine promises. The CFEngine documentation is a very good source but again the information was scattered and you needed to search for it. I wanted the information, easy to understand examples, functions, promises, and variables to be available at one place. The thought of gathering the information, topping it up with my own experiences and making it available to all who are new to CFEngine or are using CFEngine was always there on my mind. Finally, with the help of Packt publishing, I was able to give shape to the thought in the form of this book. Hope you like the effort. **Get 15% off\!** Packt Publishing is offering a 15% discount to any Yahoo\! Developer Blog reader, and the book is available in both eBook and traditional formats. To receive your discount, simply purchase [CFEngine 3 Beginner's Guide](https://web.archive.org/web/20140802044056/http://www.packtpub.com/cfengine-3-beginners-guide/book) through Packt Publishing, and provide the code **CFENGINE**15\*\*\*\* at checkout. (Note: customer must be logged in for code to work) **Want a Sample?** To help you get a feel for the content provided within this book, Packt is generous enough to provide sample content of what you can expect from reading this book: [Security Audit with CFEngine](http://www.packtpub.com/sites/default/files/4989OS-Chapter-5-Security-Audit-with-CFEngine.pdf?utm_source=packtpub&utm_medium=free&utm_campaign=pdf) -------------------------------------------------------------------------------- title: "CFEngine, SysAdmin 3.0 and the Third Wave of IT Engineering" url: /blog/2011/cfengine-sysadmin-3-0-and-the-third-wave-of-it-engineering/index.md date: "2011-11-02" --------------------------------------------------------------------------------
### PART I: the sysadmin poverty trap *The Third Wave of human society is an age where knowledge and information drive prosperity. Parallel to the changes that pushed humanity through major industrialization in manufacturing are developments happening in IT management today, some thirty years behind the manufacturing industry. CFEngine and its users have been a *primus motor* for these changes -- helping to transform old techniques into a knowledge-based approach to IT -- and we are celebrating this new chapter with the announcement of CFEngines Nova and Constellation.* I love great writers, and Alvin Toffler has been one of those few writers on the human condition who consistently impresses me with both careful thinking and excellent exposition. According to Toffler's prescient view of our human condition, Mankind's development can be divided into three major phases: - The 1st wave: the beginning of civilization through agriculture, or manually tending to the land. - The 2nd wave: Industrialization, automating production and machine-assisted strength. - The 3rd wave: Information Society, man and machine dividing labour optimally, and machine-assisted brains. In our \`cute' modern day language, we would probably say these phases represent civilization 1.0, 2.0 and 3.0 respectively. Suffice it to say, these three phases represent sweeping changes in the way humans approach the economics of prosperity. The lessons learnt from industrialization (or the 2nd Wave) were described in his book *Future Shock* (1969), and -- to anyone paying attention -- clear parallels can be seen in the journey towards a rational approach to Information Technology (IT). Just as human society still envelops everything from pre-agricultural tribes, farming communities, and factory sweat-shops to information-based commerce, in different parts of the globe, so IT today straddles all three \`waves' of development from manual chaos to goal-oriented self-repair in different organizations. #### Rehumanizing IT: Out of the IT poverty trap > *"The most powerful dehumanizing machine is not technology but the > social machine, i.e. The formation of command structures to make > humans emulate technology in order to build pyramids and > skyscrapers..." * > Lewis Mumford (1967) The Industrial Revolution that happened in the West was the way out of the poverty trap of manual labour for industrial nations. Prior to its arrival, humans worked reactively to cope with crisis after crisis, sometimes in self-cancelling ways (to use Toffler's language), and without a clear direction or goal. As the IT explosion began around the 1990s, the same thing happened in many organizations: overwhelmed by crises, ad hoc system administrators fought to keep their heads above water, so busy that they never won enough time to escape from this condition. In IT, alarm-based early warning systems have been making humans jump-to-it for more than 10 years, based on SNMP and related technologies. A feature of the second wave industrialization was the class division between rich land-owners and poor workers. This was really a division between infra-structure and its maintainers. The same division grew up in IT during what I am calling the 2nd Wave of IT management, in the division between the network and the computers (variously called \`servers' or \`workstations', etc). Network Managers (the land-owners) have been viewed as high ranking engineers, with certified qualifications, graduates from the Cisco University, etc. On the other hand, system administrators (or IT workers in more modern parlance) have been seen as low-status individuals merely performing factory monkey-work that could be learnt at a community college. The emancipation of system administrators to the level of engineers has been a long process involving motivated individuals. As one of those individuals, I can report that there is still a long way to go -- but today we are on the verge of a change thanks to the marriage between IT and commerce and its hype through concepts like the Cloud. It is these engineers who will usher in the 3rd Wave. As town planner Lewis Mumford remarked, industrialization is about making humans work like machines, in order to amplify its production of basic necessities. Industrialism was deeply de-humanizing, making humans work for the machinery, emulating it, to create a giant \`pump' for manufacturing. In the Third Wave, human freedoms become the driving force for technology: mobility and diversity become the keywords, and IT becomes an enabler of human freedom rather than suppressing it by forcing humans to work for it with clockwork factory-like procedures \[2\]. The Third Wave is really a transition from approaching certainty by over-simplifying through mass replication and artificial discipline, to tolerance and effective exploitation of a more organic or natural kind of growth through federation and contextualized knowledge. #### The 1,2,3 of IT management | 1st Wave | 2nd Wave | 3rd Wave | | --------------------------------- | ----------------------------------------------- | ---------------------------------------------------------- | | Tend the land | Industrialization | Knowledge society | | Ad hoc change | Processes/recipes | Goals and outcomes | | Ad hoc behaviour | Standardized \`canned' answers | Questions lead to emergent answers | | Bare hands | Shells, macro languages | Declarations of intent | | Ad hoc and reactive workflow | Imperative flowcharts and serial execution | Declarative thinking and parallelized background execution | | Ad hoc actions. | Batch processing, mass production | Tailored, custom action on demand | | Make the best of what you've got. | Consume and discard. | Recycling resources and renewable assets. | | Single purpose application | Multilayer software stack | Smart service oriented applications | | Push (by hand) | Push (assisted force) | Pull (subscribe voluntarily) | | Notice and respond | Regular checks | Continuous adaptive scheduling | | Ad hoc work | Roll-out | Incremental surgery | | Fire-fighting | Sound alarm and manually push the button | Proactive maintenance and reactive self-healing | | Ad hoc decision/control | Centralized decision/control | Decentralized decision/control | | Ad hoc discovery | Raw data collection | Contextualized knowledge | | Manual system administration | Scripting evolves to CFEngine 2 and derivatives | CFEngine 3 Nova/Constellation | The reincarnation of the 1st Wave system administrator in the 2nd wave is as an IT infrastructure engineer, and in the 3rd wave he or she is a kind of knowledge engineer. Thus the pre-requisite tools for IT management, in this future, will be: - A comprehension of our systems' many behaviours. - A comprehension of how close we are to our human goals. - The ability to promise desired outcomes across multifarious contexts. These are the pinnacles of a knowledge-based economy. Footnotes: 1. At Oslo, we built a course on this idea, and a colleague in Amsterdam did the same -- actually renaming their course SNE*System and Network Engineering* after a short spell out of the belief in this view of rigorous methods of engineering. 2. This was also the theme of Ayn Rand's famous novel Atlas Shrugged.
### PART II: from recipe to engineering *The Third Wave of human society is an age where knowledge and information drive prosperity. Similar changes are happening in IT management, where knowledge is becoming a premium commodity. This has led to a transformation of the role of the system administrator from administrator to system engineer and knowledge engineer. It is these engineers who will carry us into the 3rd Wave.* #### From System Administration 1.0 to 3.0 -- it's engineering In 1999 I was asked to write a book about system administration suitable for university curricula. At the time system administration was considered a non-academic field - something more like carpentry or cooking. As an academic, I disagreed with that view and had created a one semester course at the University College based on my understanding of the state of the art. ![4](images/principles.jpg)![3](images/practice.jpg) The second edition of the book*Principles of Network and System Administration*, still in use today, opened with the line: "Network and System Administration is a branch of engineering that concerns the operational management of human-computer systems". The reviewer at the time wrote back, saying: you know, I don't know, it's surely much too early to call system administration a kind of engineering -- but his reaction cemented in my mind the idea that it was simply *about time* that it was: all the pieces were there waiting to be assembled. Independently, Tom Limoncelli and Christina Hogan wrote a complementary book *The Practice of System and Network Administration* which described the high level strategies of being an engineer in a disciplined datacentre. Together, these two books documented the 1st and 2nd Waves of IT Management, preparing the way for the 3rd. However, they left out the embryonic changes that were taking place behind the scenes -- involving configuration management, model-based desired-state automation and self-healing. Ultimately though, I believe these books helped to change the mind-set of 1st Wave system administrators and usher in the transition to a knowledge-based approach to governing human-computer systems. | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ![2](images/analyticalbook.jpg)In *Analytical Network and System Administration* I made the leap towards understanding the engineering of combined human-computer systems -- this was the volume of work that led to Promise Theory. Many of these ideas have been are are used and evaluated in CFEngine 3, and have contributed to solving the major challenges that make CFEngine unique: federated and diversified scale and complexity. | #### 2nd Wave: hierarchy, rigidity and obedience The 2nd wave has brought many debatable and clunky efforts to imprint simple-minded replication onto IT management -- turning us humans into parts in the machines we have created for ourselves. There are now signs that we are moving on from these ideas. The fading affection for \`standardization' is an example. Instead of adapting technology to human diversity, humans have been asked to adapt to technology. Power that was once centralized into push-based systems that strong-arm (\`attack'\[1\]) their surroundings, demanding obedience. These ideas are gradually waning. As Toffler points out, even the industrial education was symptomatic of this obedience, requiring of the three R's: reading, 'riting and 'rithmetic in an overt curriculum to allow individuals to play their part. The covert curriculum was one of obedience, punctuality and repetition: act like a good machine. System administrators still willfully subordinate themselves to monitoring systems and alarms. Organization was, by default, made hierarchical, forcing items into mutually exclusive categories, subordinate to larger ones. Everything from family trees to biological species, to tables of contents, to computer file systems have been arranged in this \`one size fits all model'. The 2nd wave approach to knowledge management is to fit any data to a hierarchy, because this is the organizational model that humans choose to govern. #### From Steam Engine to ConFiguration Engine We don't have to think cyberpunk to see that traditional hierarchical machine-like systems from the 2nd Wave, even those such as the military, are evolving into a different kind of machine today, based more on intelligence and precision engineering than on simple-minded force and rank. If the military can do this, and maintain predictability, then surely others can afford a greater use of pre-planned intelligence too. In an industrialized (machine) process, one builds essentially a pump, an amplifier, a steam engine to power through the task. To take these power systems to the 3rd wave, one needs to measure that force and bring surgical precision to the table. Is there a contradiction then between this progression from industrialization and the massive proliferation and scale we clearly need in the IT industry today? Don't we need such brute force still? This is an interesting question indeed. For the time being, scale seems inevitable, but the landscape is more diverse (see the role of the Cloud below). And who knows what the size and capacity of the next generation of computer system will have? Perhaps an entire Amazon Cloud datacentre might fit into a small room so that everyone can have their own, cheaply. One answer here is that scale is not just about size, but rather how much effort and information are needed to describe and comprehend our systems -- to be sure that we control them, or at least to be able to ride on their backs. That cognitive cost does is not reduced by changes in hardware technology. #### Roll-out rocket launches How many sysadmins does it take to change a light-bulb? The answer could well be many\! The 2nd Waves approach to IT change management has been to demolish the existing system and rebuild a new one with the new feature (light bulb) in it. Tear down the building and make a new one\! Imperative, serialized recipes of industrial processes are simplistic and only work if they start from a blank slate -- so that is what people did. In the 3rd Wave, we tend to retrofit and recycle existing resources, making only the necessary changes. It is a more surgical approach to precision engineering. 2nd Wave configuration management prior to CFEngine was all about firing \`smart-bombs' (aka \`packages') onto systems that would explode onto the system, levelling the old configuration and rebuilding again from scratch. That was a destructive approach to management with high impact and system interruption. ![1](images/747.png) Worse than this, fear of change and the risks change carries lead to a defensive approach to change management, where sysadmins save up all changes to certain approved \`change windows' and then \`roll out' all changes in one go. This is basically rocket science: highly trained experts spend a long time building a contraption that can only be used once and only has one pre-programmed mission. It has a launch date and one chance to succeed. If it fails, you have to pick up the pieces and start again. This is high risk, and high cost. In the 2nd wave, engineers working with commercial forces were able to turn this vision of flight into a 747 -- a reusable craft that could be adapted to different cargoes and destinations, that could be re-used and maintained with standardized procedures. The mistake the IT industry still makes is to save up change and adaptation into huge, high-risk packages and then roll these changes out *en masse*, at industrial scale. The term \`roll-back' was usurped (inappropriately) from database transaction theory then to explain the back-pedalling that is attempted once mistakes are made. This approach is taken because the instruments of the 2nd Wave are rather blunt. System administrators still build systems like \`rockets' that require a lot of individual attention and expertise, and get launched on a particular date, going only to one destination, unsophisticated and inflexible. Second Wave transportation invented the 747, trains and buses for some basic flexibility in mass transport. In the 3rd Wave, we are ferried around in small go-anywhere, do-anything vehicles and these are self-maintaining without human assistance. Most companies today do IT change management like a rocket launch, instead of making small, specialized incremental changes very often to adapt to dynamical needs with agility. Companies like Google and Facebook straddle this divide between 2nd and 3rd wave, by combining industrial scale with agility, based on careful use of knowledge. It will take time for these methods to spread to the world at large. #### DevOps and The Cloud The 2nd wave is far from over. There are tools being written every day to pander to 2nd Wave nostalgia, tools that make better alarm systems, new SNMP face-lifts, or \`improved shells', new languages that perfect the details of defective models -- things that should have been left to posterity to judge. The Internet itself could be used in smarter ways. IPv6, for instance, is just a bigger pipe for second wave transport, but mobile 4G and service oriented adaptation are enablers of human freedoms -- and this is where the knowledge-based approaches will penetrate. The Cloud is another driving force that helps us towards a greater agility. Although curiously run as a 2nd Wave industrial operation, what Cloud Services provide is that magical infrastructure to support virtual agility. The Cloud does for computing what paging did for disks and what packets did for the network. It atomizes management to allow individuality and resource efficiency, by recycling. Instead of treating systems as \`all the same', configuration tools like CFEngine also atomize processes so that they can be self-healing, and use knowledge and carefully modelled intentions to engineer systems precisely -- a sharp instrument, not a blunt one. The DevOps movement that has recently come to attention is a natural reaction to slow 1st Wave thinking or inflexible 2nd Wave thinking. Impatient developers have begun to take control of their own operational deployments, to see their changes applied quickly, short circuiting the overloaded queue of the 1st Wave administrator and the inflexible \`everything the same' view of the 2nd Waver. What we want is not industrial configuration for the Cloud, but diversified precision engineering to support The Real Cloud -- which is all the smart-phones and tablets roaming around in the world in people's back pockets, bringing 3rd Wave human freedoms to the privileged generation. #### Infrastructure engineer System administration, IT operations, or whatever we call it today spans all three waves of IT development. The 2nd Wave system administrator was a machine operator, being fed by monitoring traces and reacting to alarms; the 3rd Waver is a knowledge-empowered infrastructure engineer, using intelligent planning and developing domain expertise. At CFEngine, we are privileged to work with these engineers, and to develop for the 3rd Wave. #### Footnote 1. If we want to speak in provocative terms, the management approach taken by push-based control systems is not functionally different from an external attack. This should be food for thought.
### PART III: the economy of de-standardization *The Third Wave of human society is an age where knowledge and information drive prosperity. Parallel to the changes that pushed humanity through major industrialization in manufacturing are developments happening in IT management today, some thirty years behind the manufacturing industry. The future of IT will have to break with past over-simplifications, in which Man was asked to work for and be part of the industrial Machine. Humans and technology will have to adapt, however uncomfortably, to support the basic human freedoms that we now crave.* #### Incomplete information - clinging on to certainty If one looks at the philosophies that swept through the 20th and 21st centuries, there is a transition of world views, from those based on belief and certainty to those based on making the best of what we know. We see it in the acceptance of methods like*probability* and *statistics* for dealing with unknowns, and in the move from a Newtonian clockwork universe to one dominated by fluctuation, relativity and quantum mechanics\[1\]. It's the same story with non-linear systems and \`chaos theory'. Unpredictability has finally been accepted, and seeking good-enough but still approximate knowledge about the state of the world is the way one succeeds. 2nd Wavers fight against this transition, clinging on to the simplicity of a familiar and predictable world, in spite of the limitations it brings. Unfortunately, the fiction of complete predictability can only be maintained at very high cost, by dominating the environment by brute force, as we did with steam engines and heavy machinery during the early industrial revolution. Eventually this cost becomes too high to support -- the real world is too complex to pattern with a single mold, and today's more prosperous and nuanced humans are less likely to accept restriction and limitation than in the past. In the 3rd Wave, system administrators, in their new clothing as infrastructure engineers, will eventually have to accept the same kind of transformation of world view. This change will sweep through everything we do, from the organization of systems, to the organization of workplaces, to the knowledge bases and information we use. The simplistic approach to categorization, by hierarchy and silo, will make way for more organic, emergent properties. A 3rd Waver will not accept a frozen, crystalline, mechanized view of the world. Instead goals will be seen to change in real time and reality will fluctuate around these goals. The best we can really achieve will be an *equilibrium* with our environments -- a slowly changing average behaviour with quasi-stability\[2,3\]. #### The end of the gold standard Ironically, the rise of literacy that accompanied the 2nd wave has allowed people to escape from the very shackles of simplistic, predictable thinking it advocates. This is how the 3rd Wave comes about. By raising prosperity in the wake of two World Wars, people have escaped from a limited lifestyle, and have sought to explore individual freedoms rather than grand visions of togetherness or compliance. These freedoms have been championed by and have driven commerce. Consider the changes that have taken place. Prior to the second world war, writers' visions were all about big civilizations and grand plans for humanity, but afterwards they were all about freedom. People spent their money on scooters and fridges, beauty and vacations -- people became tired of being machine parts. We see the same trends in IT with mobility, entertainment and communication. Just try and stop the iPhone from penetrating your organization\! It is not difficult to see why freedom is the natural next step. In the 1st Wave of humanity, people were isolated manual workers (working the land). They consumed what they produced because there was little infrastructure for scaling production or for sharing. People's lives were built around survival. There was some trade -- goods for goods. There was little need for security, as everyone needed to trust one another to survive. In IT, people managed their own systems and shared little; they had no need for security. This was cheap, but it was also a poverty trap. In the 2nd Wave, mass production and transportation drove sharing through markets and commerce, requiring continuous growth to survive in a competitive bubble. Security became a pressing issue because competition for high value was rife. Consumers were passive, choosing only what was offered or available, because there seemed to be little alternative. People's lives were based around systems and routines. The 2nd Wave invented gold-standard money as a universal trading currency -- a simplistic way of measuring value in a singular vision. The 2nd Wave was all about simple standards to make it easy for the machinery of society to scale up. This is cheap, but de-humanizing. In IT, all the same phenomena have basic analogues. In a 3rd Wave society, people rebel against the conformance of a programmed lifestyle. Economics becomes a curiosity shop of diversified on-demand rivalry -- driven now by consumers, not producers. Consumers are active; they demand individuality and customization. They request tailor-made services that differentiate them from others. One-size-fits-all is no longer an option. The gold standard is abandoned because, when consumers have the advantage, things are worth what people think they are worth, not what the town bully demands. In IT, the word standardization -- once everywhere (even as gold standard disks for the STIGs, etc.) -- is now mainly for the benefit of external regulators, government auditors, or multiple decentralized de facto police services that maintain predictability and balance. The challenge is to make this diversification of compliance cheap and scalable. Money itself is no longer fixed in value against gold, and the currency of trade has many different faces: we no longer work only for cash, there are other dimensions of value to consider, such as reputation, goodwill, confidence, freedom. We have become gamblers -- dealing with the incompleteness of information, instead of trying to suppress it. The Free/Open Source Software phenomenon is a good example of how the economy of trade works -- social recognition and goodwill have replaced hard cash in this area. #### Laws and norms In the 3rd wave, the governance of IT has to transform to be more like those \`wishful rules' of the human realm, because it is closer to human motivations than its industrial forerunner -- laws based more on trust, convention and voluntary cooperation, rather than programmed enforcement, and simplistic standardization. During the 2nd Wave, brute enforcement of behaviour has left an imprint on social norms, and these persist now mostly by convention. We have been trained to observe a \`civilized' behaviour, and continued enforcement is simply too expensive without the memory of basic cooperation -- the armies and invasions that brought about this compliance ultimately choke the economy with rising overhead. It will be interesting to see at what point diversity gives way to intolerable noise; this will doubtless be a highly subjective judgement -- to 2nd Wavers, any deviation from the programmed pattern would be unacceptable. In IT, the new norms are still forming, and lawlessness is still open and common, but we also see signs of the human and IT realms converging on traditions that we humans have already accepted. Our challenge at CFEngine lies in supporting this flexible reality, to bring 3rd Wave empowerment at low cost. Our increasing personal freedom suggests that we will increasingly insist on making our own goals to a much greater degree -- but governance is not going away: the recent financial collapse has shown us that independent forces of regulation are needed to prevent dog-eat-dog competition and open warfare from spiralling out of control. We might not need \`armies' in the traditional sense, but we will certainly need police -- i.e. policy verification agents. Compliance with standards and regulations will never disappear as there will always be a need to cooperate with what others expect of us; it's just that the authorities behind these standards will themselves diversify. Instead of just governments, independent bodies, companies will be the new governing entities too. There will be a remodelled Compliance 3.0 too. Ultimately, it does not matter whether the template for compliance is a law such as Sarbanes-Oxley, or if it is dictated by a need for customer transparency, or if it is merely a gentleman's agreement between companies. Demonstrating predictability in a noisy environment will be the name of the game in 3rd wave IT operations. #### Security How will we survive a transition to freedom and loss of certainty? Can we accept it? Already companies are bemoaning the arrival of iPhones and iPads and Android devices which cannot easily be secured to the level they are used to in 2nd Wave thinking. The situation is, in fact, beyond discussion: people will crave their freedom. We cannot stop it any more than King Canute could hold back the tide. Today we find it harder to control boundaries in the same way, with so many overlapping channels for communication. Will security even be a feature of the Third Wave? Of course it must be. Although security has often been dealt with like a standardized industrial process, \`locking things down'; but that is merely the way it has been handled during the second wave decades. It is a futile attempt to suppress uncertainty by brute force. The role of building trust through social norms has to be embraced, rather than trying to eradicate it -- IT will mimic what has happened in human society, because it is a human society. Security is more subtle in the Third Wave as boundaries are less well defined. That means that security perimeters have to be pulled back, closer to individual resources, rather than relying on impenetrable outer walls. The autonomous agent model is the only model that supports this kind of security -- and this immediately rules out push-based solutions. The 3rd wave organization will have to deal increasingly with emergent structure. Instead of pre-designing simplistic silos and walls, there will be much more cross talk to deal with, and organizations will divide along dynamically changing lines, that bend and flex with the modern freedoms we crave. #### The end of expertise? The 3rd wave will not be a paradise, however, there is no promise of a rose garden. In the 3rd wave, literacy is no longer a necessity. Machines can handle that too and humans could easily become dependent on a technology that no one but an elite understands. But the 3rd wave technology can also help us to think. If we use it wisely it can also help to keep humans in a decisive but curious, learning role. Qualification will come at a premium -- and until the 3rd Wave dominates, there will be a huge need for low-paid human IT administrators to work for the technology, like the sweat-shops of the 3rd World. Today's configuration management solutions are helping to spread 2nd wave industrial standardization to these old sweat-shops. And with new technologies like CFEngine 3 taking the first steps into knowledge-based approach to management, there are large savings to be made in terms of human labour and drudgery. The irony of the Third Wave is that, in spite of being based more on knowledge, standardized human competence will eventually become more sparse, because the industrial process-knowledge has been absorbed out of the people into the machinery. Is this a cause for concern? Individuality will replace standardized canned answers with deeper individual understandings. That makes communication a challenge but offers a deeper range of viewpoints. We cannot say this is wrong, or undesirable as it is already happening in free society. It is just evolution at work. System engineers will probably have to become more like pilots or doctors -- forced to know how to make a manual override of an autopilot that takes care of most basic operations, both to preserve their dignity as individuals, and also to allow legal-beagles to pin liabilities to a human in case of mistakes. But we know that this kind of transformation is true of something in every generation, and we adapt to it. Clearly there is a continued role for Open Standards and Open Source to play in the transformation to a rehumanized world -- an open process builds consensus through social interaction. This is what the human pre-frontal cortex was probably evolved to do -- so having passed through a time of playing \`machines' we can return to using our intelligence for its true purpose. Indeed, it has already enabled knowledge-based technologies to emerge (think of Facebook as a monitoring system, for instance). The 3rd Wave is not the end of expertise; it is the exploration of a more nuanced expertise, based on social diversity. #### The last days of the second wave > "The in-fighting among (second wave supporters) amounts to little more > than a dispute over who will squeeze the most advantage from what > remains of a declining industrial system." > A. Toffler, The Third Wave (1980) With CFEngine as our vehicle for exploring knowledge-based approaches to IT infrastructure, we have put a lot of work into finding ways to support social diversity. Cheap, scalable made-to-measure infrastructure makes smart use of *patterns* to reduce the amount of raw information. Similarly, the focus on *intention*, the*where* and the *why*, not merely the *what* and the *how*. Who needs knowledge management? Every infrastructure engineer does because we rarely touch infrastructure. It disappears into the background, out of sight, out of mind, It will take some time before the 3rd wave splashes onto the toes of even a majority of organizations in the West, let alone reach an understanding. We might never reach a stage of complete coverage, as the economics of the world themselves form something of a heat engine -- and a simple pump only works when rich and poor live side by side. Whatever the currency of that richness and poverty IT, we will need to be increasingly savvy about IT resources to keep it running, to cope with the deluge of information and manage the growth of IT into every part of the world around us. As in the human realm, the three waves of development are very much in existence together. The 2nd Wave still seems new to many companies and organizations. System administration, never mind system engineering has not come very far, except in the most advanced institutions. The 1st wave (manual work) is here -- and the 2nd (machine-assisted mass production) is still being promoted actively. The waves are far more entangled in IT than elsewhere, probably due to the accelerated pace of IT's incursion into society. Only the most forward-looking of companies are at the stage of considering the kind of differentiated, goal-oriented management of their IT systems, worthy of a 3rd generation approach. For now, CFEngine straddles the 2nd and 3rd Waves too, allowing industrial scalability at the same time as we advocate post-industrial customization and knowledge management wherever we go. We are providing a tool to help IT companies make the transition in the long term, from the old world to the new. No one suggests that such a transition will be easy, as Machiavelli put it: \`...the innovator has for enemies everyone who did well under the old regime, and only the luke-warm support of those who might do well under the new'\[4\]. But innovation stands still for no one. It is in our blood, even as we fear it. So watch out, here comes the CFEngine 3 family to hose away the cobwebs from industrial IT, and to gently cleanse the pores of 3rd Wave knowledge engineers. MB, Oslo, 22 Nov. 2011
-------------------------------------------------------------------------------- title: "CFEngine - The Third Wave of Configuration Management" url: /blog/2011/cfengine-the-third-wave-of-configuration-management/index.md date: "2011-11-01" --------------------------------------------------------------------------------
![](images/3_wave-b1f86a71.jpg)
If you have been anywhere near a Unix system in the past ten years, you will almost certainly have heard of CFEngine and its 'revolutionary, self-healing approach' to datacentre automation. However, what you might not know is that its current third incarnation *CFEngine 3* is both younger and more advanced than most of its imitators, harnessing the very latest ideas about system management -- and the difference is all about knowledge. In 2008, after five years of detailed study with existing users, I decided that a major change was needed in the industry, and formed the CFEngine company to set the standard for a Third Wave of Configuration Management technology. It was time to push the industry in the direction of knowledge-based IT management. From the inception, CFEngine's innovations were about increasing certainty and predictability in systems -- and that is the start of actual knowledge. But even after two generations of configuration management tools, system administration still needs to make a change -- and in more than just the scale and speed of its operations\[1\]. And in case you are wondering, yes, this all harks of Alvin Toffler's three waves of human society: - The first wave: Agriculture, tending the land. - The second wave: Industrialization, automating production. - The third wave: Information Society, man and machine dividing labour optimally. I've been arguing for some time now that this evolutionary sequence is in fact happening in IT management also\[2,3\]. The first wave was the introduction of system administrators, tending to fixes to the system with bare hands. This only scales to a few machines. The second wave involved scripting automation to mass produce simple replicas in a few sorts. The third wave is all about knowledge or intention-driven precision automation, delivering self-healing, surgical accuracy to system configuration. In the third wave, users have true desired-state management at low cost, allowing them to handle as much complexity as they need. In the third wave, humans do the thinking and make the decisions, and automation provides consistent, large-scale implementation. What can CFEngine users expect in the future? IT management still has a long way to go to ascend to the next level\[4\] -- it's more than mere automation of Web servers, or "DevOps" where the industry catches up on methods of inclusive autonomy. As IT grows in scale and complexity, the world is going to need agile methods that will work from the largest to the smallest devices, while keeping track of ever speedier business changes. That requires us to go beyond mere information, adding context and experience to bring about Knowledge. CFEngine 3's approach has been to build a model of knowledge, designed especially for system configuration, from the bottom up. This model, based on Promise Theory, provides a lingua franca (a set of policy \`assembly codons' for system change) that allows users to express desired state in a maintainable way -- not merely for one-off deployment. It is more like a genetic code that can be self-healing. Using this model, CFEngine can track knowledge about the system in a very pervasive way. All of that information is then assembled and indexed by intelligent algorithms and made available to system administrators so that they can make the best use of their skills, backed up by powerful automation. It's not a CMDB, but something more sophisticated and lighter-weight. We can call it the CFDB. Today, we are already far ahead of other products in the knowledge-based approach to desired-state. Our new Nova offering brings users closer to the documentation of their system and the software, and connects examples of policy, by cross-referencing information with a knowledge map. In addition it measures business value and connects the dots between management goals and technical operations -- giving system administrators the bridge they've needed all along to acquire a higher status within an organization. The CFDB semantic index offers insight into issues that are too complex to visualize without machine assistance. Because every computer can be introspective about its state, it can learn and make use of patterns of behaviour in itself and other machines -- allowing adaptive orchestration, so important for rapidly changing environments like the Cloud or mobile systems. CFEngine's machine learning capabilities have a long way to go before we can say they have exhausted the possibilities for exploiting assisted knowledge. The goal for now is to raise system administrators up to the next level by using their expertise, instead of burning through their time. What's on the horizon? Well, there's also the potential to harness human intelligence by using social networking concepts to make a kind of \`Facebook for the machines'. Anthropology suggests that our intelligence arose from social interaction. As a first step, Nova already incorporates some simple relational analysis, and provides users with collaboration tools so that multiple system administrators needn't tread on each others' toes when working on policy changes. So today, we greet the new star of the third wave: [*CFEngine 3 Nova*](https://cfengine.com/enterprise) -- coming soon to a datacentre near you\! And we promise even more surprises just around the corner, so be sure to keep your telescope trained on the launchpad of CFEngine's propulsion laboratory. 1. http://www.usenix.org/publications/library/proceedings/lisa98/full\_papers/burgess/burgess.pdf 2. 3. 4. -------------------------------------------------------------------------------- title: "What's new in CFEngine 3 Nova?" url: /blog/2011/whats-new-in-cfengine-3-nova/index.md date: "2011-10-25" --------------------------------------------------------------------------------
![](images/nova-277303bc.jpg)
Today CFEngine releases the latest version of its commercial enterprise software for configuration management and IT operations. In addition to all the usual power of scalable and convergent system configuration, CFEngine 3 Nova brings a higher level of information awareness to system administrators at very low cost. Resource efficiency takes on a new importance as users increasingly work in the Cloud where every CPU cycle is billed. The CFEngine mantra of "Lightweight\! Lightweight\! Lightweight\!" has pushed the new release even further in the direction of resource efficiency. Considerable improvements in network efficiency have also been achieved, allowing reporting of many thousands of hosts every 5 minutes to a single hub aggregator - far in advance of other solutions. ##### Some of the changes in *CFEngine 3 Nova* include: - An improved Ajax-based Web UI Mission Portal with LDAP/AD integration of login with roles - Improved reporting on all aspects of policy and host compliance. Flexible searching of reports in a brand new scalable interface supports thousands of hosts on a single hub. - Ability to add manual notes to the CFDB. - An improved Web-based policy editor in the Mission Portal for smart editing of subversion repositories. - Support for multiple host staging environments, such as Test, QA, and Production etc. Users can monitor the compliance of multiple stages of policy development in the Mission Portal to measure the effect of changes before committing to production. - Improved integration of policy with subversion repositories. Users can modify, update and approve revisions for the different staged environments. - Command line export of Mission Portal reports, allowing system administrators to search the database in the command line, using familiar Unix tools to query and filter data. - An improved PHP API to the CFDB now gives generic JSON return values so that users can customize the open interface and JQuery framework of the Mission Portal should they wish to do so. - CFEngine's unique Knowledge Map technology has been improved too to index and unify policy, documentation and code examples, as well as bringing a bank of concepts about configuration management to the attention of users. - Enterprise grade support for Mission Critical High Availability in the CFEngine control circuit. (see ) - Improved machine learning and anomaly monitoring for diagnostics and capacity planning. Additional sensors added to detect operating system performance and behavioural trends have been added, especially on Linux kernels. - Orchestration of processes involving several machines, especially useful in cluster and job processing environments. - Improved hypervisor integration. CFEngine's support for XEN/KVM virtual machine creation, on the fly, using the industry standard libvirt has been improved by stripping down this library. - Improved diagnostic capabilities when deploying CFEngine now allow users to debug deployment problems faster and more easily. - Elimination of MySQL dependences in the software. CFEngine 3 Nova now uses the NoSQL document oriented database MongoDB for all storage on its Mission Portal, providing rapid and efficient update and retrieval, as well as reducing the dependencies of the software even further. - CFEngine standard self-management policies have been modified to improve the separation between upgrades, CFEngine management and site policy. We wish all our users a stellar experience with Nova. #### Additional Nova Resources: - [Attend our November 2nd Webinar: *An Introduction to CFEngine 3 Nova*](https://web.archive.org/web/20111028231657/http://info.cfengine.com/NewNovaWebinar_Introduction-to-Nova_Registration.html) - [CFEngine 3 Nova product page](https://cfengine.com/enterprise) - [New Mission Portal screenshots](https://cfengine.com/nova-screenshots) - [New Nova Video Demo Series](https://cfengine.com/nova-videos) -------------------------------------------------------------------------------- title: "Untethered, CFEngine hitches a lift on Android" url: /blog/2011/untethered-cfengine-hitches-a-lift-on-android/index.md date: "2011-10-11" --------------------------------------------------------------------------------
![](images/android-fe6f422e.jpg)
#### *"If only you'd attached my legs, I wouldn't be in this ridiculous position." \- C3PO, Star Wars* Like most successful post-war technologies, the IT innovations that flourish today are those that bring freedom (in the broadest sense) to individuals. From the Italian scooter, to the American refrigerator, or the latest ability to communicate through an almost endless variety of interfaces and models, it is the desire to be set free that drives the commerce of technology. It might come as a surprise that support for this kind of freedom was built into the concept of CFEngine from the very beginning. From its genesis, CFEngine was tasked to handle such complexity -- to balance individual requirements with repeatability and predictable behavior. Today, some twenty years later, this same basic need for freedom remains: business advantage, after all, comes through uniqueness and differentiation of services, and this is what now drives IT customization. #### Configuration Management of mobile devices The latest chapter of this story is the touching of two branches of this freedom: the internet information ecology and mobile communications. We are talking, of course, about smartphones and their heavier brethren \`pads'. What about configuration management of these devices? Given that these devices run on small, specialized operating systems with little management, this idea might seem like a long shot. Your smartphone is an unruly colony of downloaded internet germs, but for an enterprise, corporate standards are a concern, i.e. the need to keep certain promises that maintain business integrity. Business has already become a highly distributed activity -- the freedom that personal electronic devices brings has changed the way in which many of us work. So far, however, little has been done to address the basic security issues -- except for a rough and primitive sandboxing of untrusted applications. Issues like the encryption of memory, pin code or password protection, software updates, application version control, are pressing. The sandbox model makes it harder for malware to propagate, but it also makes it harder for positive helper applications, like system maintenance to work. From a technological perspective, the management of mobile devices presents a number of challenges. For instance, a lot of security methodology begins with the notion of physical security -- but this is obviously not controllable for a mobile device. #### Androids without legs Phones are offline much of the time, not really suited to traditional models of push-based or bidirectional control, so CFEngine's lightweight opportunistic methods can be a key enabler here. Last year we showed how CFEngine could be run on Nokia's Linux phone () without modification. Today, it is the iPhone and Android operating systems that have achieved widespread popularity for their simple app design that integrates users with an ecosystem of apps and web services. Had these machines simply been generic Linux machines, it would be possible to provide full access management and security in much the same way as for laptops -- but the freedom to develop solutions has been severely curtailed in the smart phones. Even Open Source android does not allow the notion of a privileged monitor on which most management frameworks are built. I believe that smartphone providers will eventually have to incorporate management frameworks like CFEngine to verify aspects of corporate policy, but this will take time for the pendulum to swing back (as it always does) from closed, specialized devices (like phones) to more open multipurpose systems (like PCs). At CFEngine we are developing proofs of concept in this area today. #### App police Making CFEngine run on Android was not that hard. It is a small and compact piece of code. As a proof of concept, this has already been an important first step for our research and development. Although the capabilities are currently limited by the sandbox model, we can already demonstrate a few simple use-cases where agent-based management will work nicely, and we shall report more on these in the coming months. Battery life remains the main concern for applications that run often. CFEngine is easily small enough to run on an average smartphone (unlike most configuration management systems), and it can be run as an embedded application on Android, but without privileged access. To the phone, CFEngine is just another app, with no greater administrative power than 'Angry Birds'. This is presently a long-term hindrance. If a management "Police force" had no special powers, it would be pot-luck who could win in a competition between forces of good and evil. The current approach can only be an interim method however, and the signs are that this is already in a process of change (see ). CFEngine will continue to explore ways to exploit the options for enterprise management. Moreover, there is already room for using CFEngine's opportunistic methods to monitor and report on potential issues in a number of non-intrusive ways, bringing value to the partnership. ![CFEngine on Android](images/android_console.png) #### Mobility is the Real Cloud The meek might be lined up to inherit the Earth, but do we really want the IT world to be limited to mere \`smart phones' and pads alone? When we label something smart, it is inevitably because we don't fully believe that this accolade is beyond doubt. The point however is this: the real \`cloud' we should be talking about today is not what goes on in Amazon's basement, but rather the ecosystem of thin-client mobile devices, laptops and pads that interact with a magical but hidden infrastructure, where there is more intelligence and whose total configuration is the key. Managing this massive total ecology is what CFEngine is uniquely positioned to do. I think the technology for smart phones has a long way to go before it can reach the level of maturity required for scalable enterprise management, but the principles of distributed management are fairly well understood, based on opportunistic communication and autonomous operation. CFEngine embraced these principles long ago, and is therefore well positioned to deal with these devices. -------------------------------------------------------------------------------- title: "Your education is not wasted at CFEngine" url: /blog/2011/your-education-is-not-wasted-at-cfengine/index.md date: "2011-10-04" --------------------------------------------------------------------------------
![](images/cfe_team-0b8f4e57.jpg)
It is always a surprise to me how many people leave college with the feeling that they learned nothing useful during their time there -- with the attitude that theoretical learning and historical context (what distinguishes a college education from a mere training course) bear no relevance to the "real world". This, of course, is nonsense. Worse, many students go to work for organizations where the workplace culture turns them into battery hens, with every trace of individuality exorcised. Or perhaps industry norms and standards define the way employees have to behave. Where did the appreciation of education go, and what does it really mean? Like a few other modern companies, of which Google and IBM are famous examples, at CFEngine we have a high percentage of doctorates in the company, from the natural sciences, and university backgrounds from science, IT, and design and business. Currently, we have people from 10 countries in our office at Oslo -- with a rich variety of languages and backgrounds. Even that simple diversity changes the way we think about problems, as well as making us all more cosmopolitan citizens of the world. You don't have to sit with college books on your desk to benefit from an education. On the contrary, educated employees are people who feel they have permission to question the textbooks and think about problems where others might simply copy from a book. This kind of thinking is what has allowed CFEngine's software to be better, smaller and faster than programs that try to imitate it. It is what makes our workplace unusual and full of opportunity. What a college education encourages is how to think about problems, not what facts to memorize. #### What are you reading? Before we moved to larger premises, there was a sign on the door to our office saying "What are you reading?", encouraging everyone to expand their horizons as they left the office. I have always told students, friends and colleagues to read and listen to what others are saying. Everything we read, everything we have experienced makes each of us into a greater person. We benefit from the cultural and technical expertise of others in often unexpected ways. Is this really relevant to the world of business? Surely this has nothing to do with sales, for instance? Wrong\! Our customers are smart people. If you can entertain someone with your knowledge, you can hold their attention. If you can hold their attention, you can start a dialogue and win their trust. If you can win their trust, they will look to you for help and opportunities. This is how business gets started. In development it goes without saying that a deeper insight helps you to think of new approaches, and a knowledge of what has been done before points you towards solutions you can't keep in your head forever. #### Individual ownership of projects More importantly still is that the spirit of innovation that stems from careful thought and deep expertise is immensely satisfying, and makes for a happy working environment. The education and experience of our employees is of the greatest importance to us. Maybe you have something special about you, and you are looking for a way to put it to work? If you can entertain us with your ideas, we might be a willing audience\! -------------------------------------------------------------------------------- title: "CFEngine - From 2 to 3 and beyond..." url: /blog/2011/cfengine-from-2-to-3-and-beyond/index.md date: "2011-09-17" -------------------------------------------------------------------------------- CFEngine no longer supports the conversion tool for upgrades from CFEngine version 2 to 3. Manual intervention was still needed after its usage, and a simple, direct translation can be a poor choice that misses the opportunity for improvement. We recommend following the Upgrading from [CFEngine 2 to 3 guide](/manuals/cf3-upgrade.html), alternatively in combination with [Professional Services](/support-services) from CFEngine to provide a 'best effort' conversion. | CFEngines 1 & 2 | CFE3 Community Core | CFE Nova | |:-------------------------:|:--------------------------------:|:---------------------------------------------:| | World-wide deployment | Consistent extensible syntax | Introducing Knowledge Management | | Technology leader | Enhanced configuration modelling | Scalable reporting | | Brought convergent repair | Lists, patterns, methods | Native windows support | | | User extensible without scripts | SQL, LDAP integration | | | Fault tolerance features | CFDB searchable knowledge bank | | | Generalized package model | Spreadsheet model for content driven policies | | | | Simplified installation and upgrade | | | | Packaging | | | | Product integration | | | | Support | -------------------------------------------------------------------------------- title: "CFEngine releases community edition 3.2.0" url: /blog/2011/cfengine-releases-community-edition-3-2-0/index.md date: "2011-08-29" -------------------------------------------------------------------------------- ![](images/community-0028aa3a.jpg) As the summer draws to a close, CFEngine's core development team has been wrapping up some intense development to bring you version 3.2.0 of the CFE Community Edition. This release includes some major improvements to community users, and forms the foundation for our upcoming commercial release. See the blog post for a technical summary. We believe that CFEngine 3.2.0 is now the easiest and most scalable configuration management system to get running in multiplatform environments (Linux, Solaris, AIX, HPUX). Starting with the new bundled policy, just a single command on each host will bootstrap hosts together in a simple classical centrally managed star network, whether in the cloud or on bare metal. #### "Got no strings to hold things up" CFEngine maintains desired state even when network resources are scarce or unavailable, so it is ideal for mobile devices as well as fixed infrastructure. No client in the network depends on a central server to prop it up -- all clients can operate with no strings attached. This makes CFEngine the fastest and least resource consuming solution for configuration management by far, with the fewest dependencies. Cloud deployments work out cheaper, and network bandwidth is not clogged by irrelevant overhead. CFEngine is no mere deployment tool either -- it remains the only complete solution for continuous monitoring and repair, including security verification and compliance engineering. A CFEngine policy server can scale to several thousand machines, even when running five minute policy verifications -- thriving when other software would choke. Version 3.2.0 has improved the scalability of CFEngine's technology even further, making a solid and super-lightweight for our commercial releases -- which can now be upgraded automatically, making 3.2.0 a powerful tool for the server room, mobile systems or the cloud. Watch out for CFEngine Nova 2.1, building on 3.2.0 for all Unix, Mac and Windows systems. [DOWNLOAD CFEngine Community v. 3.2.0 here..](inside/myspace) -------------------------------------------------------------------------------- title: "Mission Critical IT - can you afford it?" url: /blog/2011/mission-critical-it-can-you-afford-it/index.md date: "2011-08-22" --------------------------------------------------------------------------------
![](images/mission_critical-28ae0d09.jpg)
*CFEngine is an inexpensive life-support system for complex and mission critical IT infrastructure...* Every year the world spends billions on risk avoidance -- safety and backup equipment, security systems and even insurance policies against loss and liability. For many of us, the risk of serious loss is quite small (though still sufficient to keep insurance companies in business) but in some industries the consequences of loss are so serious that even a small risk is unacceptable. Mission critical IT systems, i.e. those for which the absence of the system for even a moment could result in serious financial or human loss, are all around us, in power stations (e.g. nuclear), air-traffic control systems, flight navigation systems, medical life support, etc. Human lives, indeed society, depends on critical infrastructure to an ever increasing degree. #### Critical operations How do you prevent systems from failing? Ultimately, we can't. At least, we cannot prevent component parts from failing. Every technology, even our hearts and bodies, fail as we battle through the challenges of the environment -- but both biology and technology have come up with some ways to minimize the effects. - **Redundancy** -- having multiple backup systems that can take over if one part fails. - **Repair** -- proactive maintenance to avoid faults, and rapid detection of faults and repair before irreversible damage occurs. Once a fault has occurred, speed is of the essence. The industry speaks of two simple estimates: Mean Time Before Failure (MTBF) or how long IT is likely to function before there is a fault, and the Mean Time To Repair (MTTR) or roughly how long it will take to recover. Clearly, if you are on an aircraft flying towards a mountain, a thirty minute repair cycle is not helpful. In spite of the obvious importance of prevention and repair, nearly all IT management technologies are still about the initial deployment of machinery -- just set up your computers and hope for the best. At CFEngine, we have been making automated prevention and repair systems for many years -- indeed, many of the key principles used by the IT management industry were first introduced by CFEngine. Surprisingly though, automation is still not as widespread as it should be. Humans are notoriously unreliable under pressure, but some companies don't like machinery because if something goes wrong it is unclear who to blame. In some countries there is still a culture of repairing failure with a simple \`you're fired\!' -- perhaps because a culture of liability lawsuits weighs more in their minds than actual material failure? At CFEngine, we are not afraid of automation. Indeed, our mission is about rehumanizing IT, by judicious use of automation. As Alvin Toffler pointed out in the 1960s, replacing humans with technology is not dehumanizing, what is dehumanizing is asking humans to work like machines in the first place\! In mission critical systems, automation is the only way to keep IT operations running 24/7/365, and there are some key technology strategies for doing this: - Define a clear model of the Mission. - Avoid relying on anything we do not control. - No 'single points of failure' (employ redundancy). - Be as swift and agile as possible when implementing change. - Provide trustworthy insight about the state of the system. - Avoid interacting with any part of the system unless it deviates from the mission. We work continuously with key industry leaders to make CFEngine ever faster, lighter and more reliable in the face of complex and unpredictable infrastructure. You don't have to work in outer space to experience risk. Just having a complex organization with challenging communications can be reason enough for a serious system failure. Recently we have added a solution for high availability insight into the CFEngine Nova 2.1 product, through our Mission Portal or CFDB; now humans can play their proper role in early warning of rising issues. By making a fully redundant architecture for information updating, we can make reliable promises about availability of status information. This might be the lightest redundant solution to system monitoring available to users today, with even greater performance improvements added recently. Users get reliable and predictable insight, with fault recovery times of only a few minutes in case of failure -- something that cannot easily be matched by competing software. A final point is that CFEngine gives engineers real insight into the accuracy of measurements it presents. Most monitoring systems do not tell you how fresh the information you are seeing is, so humans can use their skills in forming technical judgements. Engineers have no use for pretty graphs that satisfy only their emotional needs. What is satisfying is that sound engineering principles have led to a solution with both low overhead and a relatively low price-tag. Today, the cost of creating mission critical infrastructure does not lie with the IT management technology -- you will spend far more on the human management regulation. #### Critical Cloud For some, Cloud Computing is the answer to scaling -- rent your computing power virtually on the web and recycle it without ever needing to own, just like car rental. The Cloud offers rapid response to demand, and such \`elastic' resources are certainly an important part of responding to peak demand, but merely deploying virtual machines quickly is not enough. To accomplish *mission critical stability* and keep risk at a minimum, you have to plan for the unexpected, and you cannot escape the need for prevention and repair with a short turnaround. ![hubs](images/redundhubs.png) Now with the release of CFEngine Nova 2.1 (September 2011) and its integrated solution for high availability management, users have software that is robust to network outages and integrates high quality insight for both system experts and higher level management, with failover redundancy and mission compliance monitoring. We believe that Nova 2.1 might well be the most robust cross-platform management technology, with the fastest response time and smallest impact that you can find in the industry today. This will not be the last time we improve CFEngine's robustness under pressure, even though we lead the Open Source technology race significantly in this area. Our continued R\&D shows that we are committed to our own critical mission: of making the impossible possible. -------------------------------------------------------------------------------- title: "CFEngine Completes Series A Investment" url: /blog/2011/cfengine-completes-series-a-investment/index.md date: "2011-04-04" -------------------------------------------------------------------------------- CFEngine is the leading vendor of open source configuration management solutions, used by the IT industry to keep datacenters running. Its products are installed on millions of servers world wide. The company has been profitable since its inception in 2008. "We expect to see fundamental changes in the IT Operations industry in the next few years", said Bjørn Erik Reinseth, Partner of Ferd Capital. "CFEngine is the thought leader among the new generation of solution providers and we believe they will be a game changer and capture a significant share of this market." Professor Mark Burgess, CTO and founder of CFEngine, created the technology as a completely new way to manage servers. "CFEngine is investing in the future of true IT automation, like no other company. We are pushing the limits of technology to provide the tools for the next decade and beyond, and we are growing to make that vision possible." Thomas Ryd, CEO of CFEngine, said "The investment will enable us to accelerate the growth of the company and invest in the success of our customers". The company's new US headquarter will be located in Palo Alto, California, and in addition a new office in the New York area is planned. Thomas Ryd plans to move to the US in the near future. Along with the investment there is a new Board of Directors consisting of Mark Burgess (CTO CFEngine), Christen Krogh (Chief Development Officer, Opera), Bjørn Erik Reinseth (Partner, Ferd), Dagfinn Teigland (Ferd) and Chairman Haavard Nord (former CEO, Trolltech). -------------------------------------------------------------------------------- title: "Test Driven System Administration" url: /blog/2011/test-driven-system-administration/index.md date: "2011-02-03" -------------------------------------------------------------------------------- CFEngine has been at the forefront of innovation in Configuration Management since the start of the 1990s, and CFEngine's CTO and Founder Mark Burgess has been writing a lot about the history of those ideas as well as the vision of configuration management going forward. CFEngine is not a company that rests on its laurels. This year our CTO will be giving a number of talks about the Future of Open Source Configuration Management, this will be worth catching: first at the SCALE 9x conference in Los Angeles, and later at the UKUUG Spring Conference in the UK, as well as a number of guest appearances. We recommend: - [Test-driven system administration](https://queue.acm.org/detail.cfm?id=1937179) - [Universality and IT management](http://www.iu.hio.no/%7Emark/blog_universal.html) - [Three myths holding system administration back...](http://www.iu.hio.no/%7Emark/blog_3myths.html) -------------------------------------------------------------------------------- title: "Loosening Orion's Belt -- CFEngine's Orion Cloud Pack" url: /blog/2010/loosening-orions-belt-cfengines-orion-cloud-pack/index.md date: "2010-12-21" --------------------------------------------------------------------------------
![](images/tree-79e8dea6-240457a7.png)
*Talk of the coming Cloud may be hanging on our horizon, but tonight, at Hr23.Day21.December::, planet Earth reaches the [Winter Solstice](https://en.wikipedia.org/wiki/Winter_solstice), when its axial tilt reaches its farthest point from the Sun and darkest night of the year for us in Oslo. As part of our celebration of the holidays around the solstice, we would like to lay a small package under the tree for all our community users, to brighten the sky with a taste of our Nova templates for Linux Managed Services.* #### The Orion Cloud CFEngine's Orion Cloud Pack was originally conceived to make the use of the EC2 Cloud simple for users of CFEngine ([see the demo](https://cfengine.com/pages/demos?view=Orion)), but the techniques that make Cloud Computing simple are not at all limited to that. With powerful configuration management, it really doesn't matter where you do your computing. Today, we are partnering with private cloud technologists [Open Nebula](https://www.opennebula.org), and [Canonical](https://www.canonical.com) of Ubuntu fame, to make this vision a reality using the most lightweight and forward-looking technology available today. Like Orion's belt itself, that illuminates the Orion Nebula, we envisioned these templates as 3 steps across the heavens to light up systems with a menu of the most popular managed services. We give this to the community now to lead the way in developing CFEngine solutions further. This can be a first step towards developing a full library of contributions for and by the CFEngine Community. Users of our commercial products will benefit directly from this openness, and will have the enhanced experience provided by the reporting and the Copernicus Knowledge Map. The CFEngine Cloud Pack is not about managing any particular cloud, but about making it easy to rebuild and relocate services anywhere, with a simple repeatable process that incorporates CFEngine's automatic self-healing. #### Get the software [Go to the Orion Cloud Pack](https://cfengine.com/cloud). CFEngine templates [on the source code downloads page](https://web.archive.org/web/20120206112140/http://source.cfengine.com/browse/copbl/trunk/OrionCloudExamples/). #### What lies in a name...? In case you haven't noticed, we use astronomical names at CFEngine for all our innovations. Orion, the hunter from Greek mythology, was taken by renaissance mythologist Natalis Comes to be an allegory for an approaching storm cloud. Better-known today, Orion is the name of one of the most famous and studied astronomical constellations in the night sky. It contains the three bright stars of Orion's belt (the three steps to the cloud) and the [Orion M42 Nebula](https://en.wikipedia.org/wiki/Orion_Nebula) (a gigantic dust cloud) beneath its belt. Now with the CFEngine Orion Cloud Pack, you too can get the Cloud under your belt\! ;-) The Orion Nebula is, in fact, part of a much larger cloud or nebula in the constellation of Orion [Molecular Cloud Complex](https://en.wikipedia.org/wiki/Orion_Molecular_Cloud_Complex). We chose the name Orion for our Cloud Pack (apart from the obvious puns) because we believe that Cloud Computing is only a stepping stone towards what we term *Molecular Computing*, in which many independent platforms and services bond together to form network patterns. These patterned systems act like molecules with new properties, bonded together by promises. This view follows naturally from a description of computing using Promise Theory, replacing traditional monolithic and hierarchical systems with a more natural form of engineering. -------------------------------------------------------------------------------- title: "Facebook 'likes' CFEngine!" url: /blog/2010/facebook-likes-cfengine/index.md date: "2010-06-24" -------------------------------------------------------------------------------- *CFEngine has played a major role in allowing Facebook to scale from one dormroom server to the largest datacenters of the world. Tom Cook explained it all with high velocity.* ![](images/facebook_likes_cfengine.png) #### "Velocity 2010 Conference" It's June, at the Velocity 2010 conference in Santa Clara, California, and Facebook Systems Engineer Tom Cook provides a glimpse inside the daily life of the Operations team. An important part of the job at Facebook is systems management. He tells us: Facebook chose CFEngine as their configuration management solution. It gives their engineers peace of mind. In his presentation, Cook traces Facebook's humble beginnings, from their launch on a single server at founder Mark Zuckerberg's Harvard University student dorm room just over 6 years ago to today, having over 400 million active users. (This figure has now increased to over 500 million this past July according to Facebook's website). This explosive growth has Facebook currently building a new data center in Oregon in addition to its other server sites in Virginia and California. These centers manage millions of applications and content, with users uploading over 3 billion photos each month. Facebook users spend an average of 16 million minutes a day on the site as well as sharing 6 billion pieces of content per week. Since Facebook is customized to each individual user, there is a large volume of information exchange. Facebook engineers face the challenge to service requests quickly and make changes while trying to avoid pitfalls such as downtime, miscommunication and undocumented changes. CFEngine updates all the servers while checking more than 1000 "promises" in a matter of seconds. #### "We Use CFEngine" CFEngine is probably the most popular data center automation solution available today. It is the universal glue that makes IT work. The larger the organization, the more complex the infrastructure. This is why CFEngine is especially popular among large organizations. As a large, expanding company, Facebook has been able to cope with their extremely rapid growth thanks to CFEngine. Facebook updates their servers every 15 minutes and needs a program that is fast. "CFEngine is a great option" says Cook. He mentions that unlike other sites who do a slow-roll or may experience downtime, they don't do it that way. "CFEngine evaluates a little over 100 policies and 1000 rules in around 30 seconds, which allows us to make changes quickly and push them into production". #### "No commit and quit" ![](images/facebook_login.png) Cook explains that the rapid servicing of information is an issue Facebook engineers are concerned with. So much is going on that their site needs a combination of teams and automated processes to oversee its total function. How is this done? First, the engineers see Facebook as a platform on which they can build; it acts as both a stable forum and an area to experiment with new features like Facebook Places. Second, Facebook engineers write, test, debug and deploy their own code. This means they are involved in the entire coding process from start to finish. Third, communication is key at Facebook and they have an internal news portal to share info and give updates and to keep up with what's going on. Thus, the engineering culture at Facebook is intense and engineers have a symbiotic relationship with one another and the Operations Team. Finally, they want work flexibility to allow them to make changes at any time of the day, logging everything. CFEngine, like Facebook, is committed to the development process from start to finish. We base solutions on the principle of automation for complex or large installations, and develop new code for this purpose. After years of research and development, CFEngine takes a pragmatic and long-term approach to automation. In our passion for developing new techniques and theoretical results, we have raised the industry bar for more than ten years. "How do we actually manage all those systems?" Cook asks. "There's a lot of churn on our network...we have to have some form of control at the end of the day to guarantee our people a stable place for development". The solution is the site's heavy use of automation as it saves time. Cook stresses that configuration management is "a must". Whether you have 2 or 2000 servers. Configuration management is one part of Facebook's two-part system management, the other is "on demand tools". For the latter, Facebook has written a program that suits their own needs. Cook says that configuration management systems "get you away from having to write shell scripts...error checking is handled for you." Cook points out that Facebook has one engineer per 1.1 million users to rely on for support. It would be impossible for any human to tackle this alone. #### Join the club Facebook's engineers are busy bees making the most communicative tool on the Internet run faster and smoother, with more than a little help from CFEngine. CFEngine has ensured that Facebook service is reliably available to its millions of users. No other system scales to tens of thousands of machines, and is just as \`at home' on laptops, desktops and embedded devices as it is on mainframes Do you like CFEngine? Then join the [CFEngine facebook page](https://www.facebook.com/pages/cfengine/311003700627?ref=ts). ## References [Facebook datacenter facts](https://www.datacenterknowledge.com/the-facebook-data-center-faq-page-2/) -------------------------------------------------------------------------------- title: "CFEngine in your phone" url: /blog/2010/cfengine-in-your-phone/index.md date: "2010-06-21" -------------------------------------------------------------------------------- *GNU/Linux is taking over the embedded device market, bringing Open Source flexibility and superior Unix management techniques to distributed, partially connected, and nomadic environments. Associate Professor Erik Hjelmås of Gjøvik University College in Norway has been experimenting with the latest CFEngine on a Nokia N900 mobile phone.* ![](images/N900-3.jpg) If there is one thing that characterizes the 21st century so far, it must be mobility: mobile phones, mobile internet, mobile people. The tools of system administration, on the other hand, by and large assume that infrastructure is fixed and rooted to predefined subnets and domains. Of course, this is not true. The Navy and any shipping company will tell you that maintaining multiple secure networks on a sea-going vessel is not something you do with the assumption of a reliable Internet connection. There might be limited local connections, but wide area links are either a few kilobytes per second or non-existent. CFEngine is a tool that was designed with unreliable infra-structure in mind, which could explain why it is popular with the military and with shipping. But now there is more. ## Location, Location, Linux\! As the GNU/Linux operating system spreads to every imaginable device from mobile routers to watches and tablets, the problem of operating system management rears its head. Philips televisions now come with a copy of the GNU Public License. Now it is the turn of mobile phones to take on the adaptability -- and corresponding complexity -- of a standardized classic operating system. ![](images/cfn900-2.png) The big question is: what on Earth would anyone want to do with CFEngine on a mobile phone. Is there really anything to configure? The answer is surprisingly prosaic: Erik Hjelmås explains, "Two to four times a year there are full updates that require you to flash the device and then reconfigure everything from the bottom up. Having automated configuration can be a life-saver." "If you forget to put the device lock on, and the phone is in your pocket, all kinds of stuff can happen with the desktop. I've often thought it would be great to just run CFEngine once and have everything \`self-heal' back to normal instead of having to search for icons and make shortcuts by hand." He adds, "This kind of device is used a lot as a video camera, so the storage promise is useful for monitoring disk space and cleaning up temporary files. I want a warning when the disk is getting full. Moreover, the N900 is based on Debian and has apt-get for package management. CFEngine can handle package updates." ![](images/cfn900.png) There are still some challenges in the mobile world, however. "The N900 does not have a cron, so cf-execd can arrange for jobs to be run on a regular schedule. So far though this causes the battery to run down a lot more quickly." The challenges of mass configuration are that it has to be lightweight to save energy, and self-healing to deal with unpredictable environments. That is CFEngine in a nutshell, or ... well, put your ear to your phone in the future and you won't hear the sea through a shell, but you might hear CFEngine's cleansing waves rising and ebbing to a relaxing schedule. -------------------------------------------------------------------------------- title: "CFEngine and Ubuntu go back to University" url: /blog/2010/cfengine-and-ubuntu-go-back-to-university/index.md date: "2010-06-14" -------------------------------------------------------------------------------- ### University of Oslo transforms desktops with CFEngine on Ubuntu #### CFEngine and Ubuntu go back to College ![](images/usit.png) *Users at Oslo University will soon be offered the Ubuntu operating system on their laptops and desktop machines, and it will come with fully automated CFEngine management.* The University of Oslo Computing Service (USIT) has always been quick to embrace developments in Unix system administration. It was in this milieu that CFEngine founder Mark Burgess originally conceived the automation software in 1993. Now CFEngine is the most popular solution for hands-free automation around the world, and USIT is forging ahead by bringing the Ubuntu (GNU/Linux) operating system to its users. Anders Bruvik of USIT explains: "We are currently in the final phase of a Ubuntu project. We are giving users at the University the opportunity to use Ubuntu on their desktops. Ubuntu is the preferred solution on the Linux platform as Windows and Mac OSX are on other platforms. Users at the University will be able to get whatever system they like." Many University users are likely to find an Ubuntu desktop an attractive proposition. GNU/Linux provides many of the tools that academics prefer, such as Latex text formatting and scientific number crunching \`out of the box'. Many of these programs were written principally for Unix-like systems. "It started with a desire to update our configuration management system on all platforms: Linux, Solaris, Mac OSX, etc. At the same time we are being asked to support Linux on the desktop, so we're killing two birds with one stone, so to speak. CFEngine runs on all these platforms, has good support, and is the all 'round winner on performance, functionality and re-usability. Ubuntu is the natural choice for desktops with its good driver support. It's easy to install and runs well on laptops." Bruvik explains that USIT's goal was initially just to simplify server administration using CFEngine, by building on its platform agnostic, self-healing approach to configuration. CFEngine allowed USIT to configure all servers, spanning multiple platforms, with a small set of rules, but then it dawned on the team that the same technology could easily be extended to the desktop. It's a radical move. Only a few years ago, it was nonsense to admit that GNU/Linux could match Windows or Macintosh on desktops. Bruvik adds, "CFEngine makes light work of any operating system, but Ubuntu is especially simple. On a desktop you've got to have the usability -- and that is where Ubuntu has succeeded in our view." ## CFEngine-Ubuntu partnership Last week the CFEngine and Ubuntu parent companies signed a partnership agreement to integrate and provide solutions for these popular tools. Anita Bilicka, CFEngine Community Liaison puts it like this: "CFEngine and Ubuntu are already a winning combination. Ubuntu is a very popular operating system in the cloud, for instance. We released our [Orion Cloud Pack](https://cfengine.com/cloud) with special support for Ubuntu, knowing it's such a popular choice. Now we see that there are strong arguments for building on that lead. Desktops are a pretty obvious candidate for this partnership too." Indeed, Ubuntu is well known for its user friendly desktop releases, but it now has also a very interesting server product that offers the chance of [complete hands-free life-cycle automation](https://cfengine.com/blog-reader/cfengine-ubuntu) from build to retirement. This is a holy grail for many Unix administrators. ## Future challenges Anders Bruvik adds, "We do still have some challenges before the big release for the new academic year. Configuring laptops with dynamic IP addresses and mobility has proved to be a challenge, and there is not really any best practice in this area yet. We're hoping that the collaboration between CFEngine and Ubuntu will help to develop some of these generic solutions for the wider community." Anita Bilicka confirms that CFEngine is already on the ball. "The next release of CFEngine Nova is introducing features that will make the mobility issue easier to handle. The world has already gone mobile -- and both Linux and CFEngine are everywhere." ## Back to College\! The [University of Oslo](https://www.uio.no/english) is one of the largest in Scandinavia, with close to 6,000 employees and 30,000 students. CFEngine was conceived of in 1993 at the Department of Physics, working closely with USIT's team -- several of whom are still present. CFEngine founder and CTO Mark Burgess smiles about the project, "For those who say that CFEngine is an academic project, we can now confirm that it's true\!" -------------------------------------------------------------------------------- title: "Zenoss and CFEngine integrate!" url: /blog/2010/zenoss-and-cfengine-integrate/index.md date: "2010-06-10" -------------------------------------------------------------------------------- _Zenoss is fast becoming a recognized leader for commercial open source monitoring. It offers a single model-based product to seamlessly manage physical, virtual and cloud based infrastructure. Its power lies in its flexible combination of network discovery and data presentation, and it makes an ideal partner for CFEngine's under-the-hood self-healing technology._ CFEngine's famous lightweight, hands-free approach to system administration grew significantly in 2008 with the release of its Nova product. CFEngine is designed to ensure compliance with a system model, and one of the features CFEngine Nova added was a simple reporting capability for the compliance level of the individual systems. So why CFEngine and Zenoss together? [![](images/nova.png)](https://cfengine.com/nova) CFEngine CTO Mark Burgess explains: "CFEngine is a very focused product. It deals with automated installation and repair, with compliance-specific reporting. CFEngine's monitoring capabilities are designed to use very focused information to provide intelligent feedback during repair. What Zenoss brings is the ability to view a wider range of devices than CFEngine is designed for. In that sense, the two products make perfect partners." Zenoss Community Manager Matt Ray adds, "We've started the integration at a simple level to demonstrate the principle, but since both tools are model driven and extremely flexible, we could do almost anything in the long run. We have a number of users who have both products, so we'll wait and see how this first community pack gets adopted. As more users provide feedback, we can imagine much more integration between the web interfaces for Zenoss and CFEngine." ![](images/zenoss2.png) Zenoss users can download the Zenoss community pack from the link below. CFEngine users will be able to add notifications directly to the Zenoss console in their system promises. For example, if you wanted to see when CFEngine repairs compliance violations for a particular system promise, you could inform Zenoss directly: ``` files: "/etc/passwd" comment => "Promise passwd security & tell zenoss if permissions were wrong", mode => mog("644","root","root"), action => tell_zenoss_repaired("passwd file had incorrect permissions"); ``` Burgess remarks, "Users disagree wildly about what they want to see reported. CFEngine was designed to handle situations without the need for human intervention, but many users want to see every repair anyway -- we all have a need to know what is going on. Good Knowledge management is really what this is all about. By integrating two open, model-driven tools no one is making that decision for you -- you can set up exactly what you need in your environment." Two of the most widely used Commercial Open Source products are now better able to support their overlapping communities. We wish our respective users success. #### References [CFEngine at the Zenoss website](https://web.archive.org/web/20141218134810/http://community.zenoss.org/docs/DOC-5897) [Zenoss blog article](https://web.archive.org/web/20100427041040/http://community.zenoss.org/blogs/zenossblog/2010/04/23/datacenter-barometer-cfengine-revs-up-configuration-management) -------------------------------------------------------------------------------- title: "Ubuntu partners with CFEngine to offer hands-free server administration" url: /blog/2010/ubuntu-partners-with-cfengine-to-offer-hands-free-server-administration/index.md date: "2010-06-10" -------------------------------------------------------------------------------- p\>*When Ubuntu LTS Server Edition made the radical decision to offer Long Term (five year) support for its server edition software, it sparked the potential to make a significant impact on data center management. Upgrading servers is one of the most disruptive operations in a data center because it means down-time. Now you might never have to take down a machine during its entire lifetime.* Perhaps a pipe dream? But, with a little luck, not so far from the truth as you might think. Today, thanks to package management, the bulk of patching and upgrading of a system can be done while the machine is live. This means the prospect that one might actually avoid taking down a machine to reinstall it for the entire life of the physical server is actually within reach, assuming that no critical kernel patches are needed. ![](images/UbuntuPartner.png) ## Continuous maintenance and repair brings you closer Such a long term approach requires some careful monitoring however. Machines can rarely survive for so long without some kind of pro-active maintenance. A traditional reactive model to incident handling will not cut it. To scale up to a large environment, CFEngine's unique self-repair approach comes into play. CFEngine is the original and still leading Configuration Management automation software, installed now on over a million machines world-wide and at some of the largest organizations. Originally dating back to 1993, it was completely renewed in 2008 in its 3.0.x series. The CFEngine software is available both in a fully Open Source Community Edition (3.0.5) and in enhanced Nova and Constellation commercial offerings. While the Community Edition is powerful enough in terms of capabilities, the commercial offerings aim to simplify configuration for enterprises, based on the concept of a Build Deploy Manage Audit lifecycle. In short, CFEngine allows you to start with a generic \`stem cell' machine (running, say, the Ubuntu Server edition), with the most basic installation. It can then customize and update the software without human intervention, based on policy. Unlike most management software, the story doesn't end with installation though. It continues for the lifetime of the system, with five minute checks for compliance. As Mark Burgess, CTO and Founder of CFEngine quips, *"You're not just getting a roll of astro-turf and hoping for the best, CFEngine is a 'round the clock groundskeeper that tends the garden and pulls the weeds."* The new partnership between Canonical and CFEngine will surely be of great interest to those cramped datacenters that need lightweight and versatile tools to eke the maximum amount of utilization from the increasingly costly cubic feet. [Visit the Canonical software partners page](https://canonical.com/partners) -------------------------------------------------------------------------------- title: "CFEngine shines through the cloud" url: /blog/2010/cfengine-shines-through-the-cloud/index.md date: "2010-06-09" -------------------------------------------------------------------------------- *Cloud woes? CFEngine made its [Orion Cloud Pack](https://cfengine.com/cloud) available last week, as a 1-2-3 quick-start for users of its CFEngine 3.0.x family of products. Continuing the astronomical theme of the company, the page sports the slogan "Get the cloud under Orion's belt" -- a reference to the M42 Nebula in which is located, where else, just under the famous three stars of Orion's belt.* ## 3-2-1 Launch\! "The Cloud Pack is a jump-start kit for the cloud," explains CEO Thomas Ryd. "The idea is to make it completely trivial to get started with managed services. CFEngine is unique in not only making installation easy, but also in bringing hands-free repair to the problem. It's especially important to have self-healing technology when your computer might be half-way across the world." ![](images/cloud_happy.jpg) "Our interest in the cloud comes as part of a larger strategy of supporting virtualization and outsourcing in a variety of ways of using CFEngine's approach to management. Businesses are looking for answers to questions about how whether they should insource or outsource IT departments today. CFEngine levels the playing field by making the location of the machines irrelevant. If you want to run a \`private cloud' or a desktop PC that is no problem." CFEngine author and founder Mark Burgess (Professor of Network and System Administration at Oslo University College) has been publicly critical of cloud computing, saying that the cloud is not a new technology. Has he now changed his mind? "The cloud is first and foremost an economic phenomenon," Burgess remarks, "not a technological one. What companies like Amazon have done nicely is to add the elasticity of the scaling, but it's all made by integrating established technology. The Cloud Pack makes this clear -- we put it together in response to the current excitement about the cloud, but it works equally well on your home PC." ## Industry partnerships CFEngine has chosen to partner with Canonical and its Ubuntu operating system, as well as with Eucalyptus and its private cloud framework -- both companies, who are supporting cloud services by making different aspects of the puzzle easy. Ryd explains, "Ubuntu is rapidly becoming the operating system of choice in the cloud today thanks to its user friendliness, and Eucalyptus is an enabling interface for running Amazon compatible services in a private environment, addressing the potential security fears that the public cloud services have sparked." Both companies were eager to collaborate with CFEngine to bring strong configuration management capabilities to the challenges of deploying massive systems. ## What's next? You can expect more versions of the Orion Cloud Pack, targeting different cloud providers: Amazon EC2, Eucalyptus, Ubuntu Enterprise Cloud, Rackspace, and more. -------------------------------------------------------------------------------- title: "CFEngine 3 is easier than you've heard" url: /blog/2010/cfengine-3-is-easier-than-youve-heard/index.md date: "2010-06-04" -------------------------------------------------------------------------------- *Psyching yourself up for an upgrade to CFEngine 3? You've probably convinced yourself that it is harder than it is. CFEngine allows you to decide between high and low level approaches. High level often means simplistic or inefficient, but low level can be overwhelming -- there is a balance to be struck. In this article, we start from the bottom up and list some of the most basic low level idioms you'll have used before in CFEngine 2, to show you how they can look in CFEngine 3. These are basic capabilities from which all high level approaches can be built, without the need for user-scripting. Then, using CFEngine 3's "[ACL paradigm"](https://cfengine.com/blog-reader/cfengine-acl) you will be able to scale these easily for most needs...* ## 1, 2, 3, Nova, Constellation\! If you already have a large CFEngine 2 installation, you might be a little afraid of upgrading to CFEngine 3, but take it from those who already did -- CFEngine 3 is simpler *and* more powerful than its predecessors. Don't focus on the differences, focus on the similarities and you'll be surprised how quickly it makes sense. The differences needn't be exaggerated, because CFEngine always emphasizes *principles* -- and the principles are the same. Why upgrade at all if you are using CFEngine 2? Because CFEngine 3 supports: - Sophisticated pattern matching and iteration - to scale better. - Knowledge modelling - to make sure that the intentions are clear, and offer reporting. - Mapping into a Graphical User Interface - for fancy commercial upgrade paths. Moreover, CFEngine 3 has a structure that is fully extensible for the future. So, how do others upgrade? Here are two approaches: - Low level: You can of course seek help from the CFEngine team to convert your old configuration into the new language (minus the new features and optimizations unique to CFEngine 3). - High level: Use the cloud? You could set up a green-field project using the Cloud as your testbed, and start with the [Orion Cloud Pack](https://cfengine.com/cloud) to get a service oriented overview, which can then easily be transferred back to private hardware. CFEngine Nova and Constellation customers have automatic access to the conversion tools. ## A short gallery of examples So let's get to it\! Here's how to achieve a few simple low level system repairs. Just put the following examples into the space provided, and this will be the skeleton for your configuration. The list is far from exhaustive, but it gives you some ideas. ```cf3 body common control { bundlesequence => { "main" }; inputs => { "cfengine_stdlib.cf" }; } bundle agent main { # insert examples here } ``` These "boiler plate" parts of the code are there to provide new extensibility and to support the knowledge management that is new in CFEngine. We won't mention those features here. CFEngine now makes extensive use of a [standard community library](https://cfengine.com/manuals/CfengineStdLibrary.html) of "body parts", or changable libraries of special words. Every blue word below can be changed by you to make the code easier to read and to hide decisions. The red words are part of the fixed syntax of CFEngine. The "rules" called "promises" have the generic form: ``` promise_type: "affected object" detail/attribute => value/template; ``` #### Creating file objects ``` files: "/home/mark/tmp/test_plain" comment => "Promise that a plain file exists with stated permissions", perms => mog("644", "root", "wheel"), create => "true"; "/home/mark/tmp/test_dir/." comment => "Make sure a directory exists", perms => mog("o=rx", "root", "wheel"), create => "true"; ``` Notice a new feature in CFEngine 3: comments. Comments that you add in code are not just thrown away, but appear in log messages and reports to annotate the *intention* behind the promise. *Why* are we doing this? This is part of Knowledge Management. #### Copying and synchronising files ``` files: "/home/mark/tmp/test_plain" comment => "Copy a plain file", copy_from => secure_cp("/tmp/file","serverhost"); "/home/mark/tmp/test_dir" comment => "Copy a directory and all of its contents", copy_from => secure_cp("/tmp","serverhost"), depth_search => recurse("inf"); ``` In this last example, you could easily redefine a new word to write even more simply: ``` "/home/mark/tmp/test_dir" copy_from => secure_cp("/tmp","serverhost"), depth_search => inf; ``` You have great freedom to make the code readable -- and this is encouraged. Careful choice of terminology is good Knowledge Management. #### Filling out a template file ``` files: "/etc/special.conf" comment => "Fill out a simple form containing variables", edit_line => expand_template("/src/template.in"), edit_defaults => empty; ``` If the template is small, you can also simply include in-line in the configuration. #### Adding and promising user attributes What about more high level issues? This doesn't have to be any harder. Managing users is notoriously tricky, but not with CFEngine's editing capabilities. These examples don't just add users, they verify and repair them to make sure the promises are kept. Ensuring users are properly configured: ``` files: "/home/mark/tmp/passwd" comment => "Prevent this user from logging in", edit_line => set_user_field("badwolf","7","/bin/false"); # Field 7 in /etc/passwd is the shell ``` Or groups: ``` vars: "userset" slist => { "user1", "user2", "user3" }; files: "/home/mark/tmp/group" comment => "Make sure the listed users in this group", edit_line => append_user_field("root","4","@(bundle.userset)"); ``` #### Integrity "tripwires" - change detection ``` files: "/home/mark/tmp" -> "me" changes => detect_all_change, depth_search => recurse("inf"), action => background; ``` #### Embed commands safely, with sandboxes and scheduling ``` commands: "/usr/bin/update_db" action => ifelapsed("480"); # Every 8 hours "/etc/mysql/start" contain => setuid("mysql"); ``` #### Process management ``` processes: "snmpd" signals => { "term", "kill" }; ``` #### Partition management ``` storage: "/home/mark/server_home" mount => nfs("myserver","/home/mark"); ``` #### Package management ``` vars: "allpacks" slist => { "apache2", "texinfo", "php5", "mplayer" }; packages: "$(allpacks)" package_policy => "add", package_method => generic; ``` ## Abstraction, abstraction, abstraction... Critics sometimes claim that CFEngine is only about low level configuration, and that there is no abstraction capability. This is wrong of course, but that is the subject of a different article. (For a taste, see [here](https://cfengine.com/pages/compare).) You can make CFEngine as high level as you like -- but watch out, as poor choices can lead to non-scalable strategy, or over simplification. ## Just a taste This is just a taste of what you'll find in the CFEngine 3 family. Most configurations can be upgraded automatically, without the newer features, of course. We haven't even begun to mention the benefits of using CFEngine Nova, and its graphical knowledge features. The examples here look simple, because they are quite \`ordinary'. Simplicity can be deceptive, when it is over-simplification. The true power of CFEngine is felt when you have truly special needs. That is when most systems let you down, because they are not generic, so you have to resort to shell commands or programming. The CFEngine language has many more tricks up its sleeves for the difficult cases, and you can always be sure that the final result will be predictable and constant no matter how many times you run CFEngine. -------------------------------------------------------------------------------- title: "CFEngine 3 Manages the Open Nebula Cloud at SARA" url: /blog/2010/cfengine-3-manages-the-open-nebula-cloud-at-sara/index.md date: "2010-06-04" --------------------------------------------------------------------------------
![](images/sara03-cc231990.jpg)
The Netherlands are well known for pushing the boundaries of IT to support scientific initiatives. The Dutch Supercomputing Centre SARA (Stichting Academisch Rekencentrum) is one of the top 500 supercomputing sites in the world and has been an ardent CFEngine user for many years. In keeping with this cutting edge profile, SARA has been turning to the latest developments in CFEngine to offer cloud services to Dutch researchers. ![](images/sara1.jpg) ## Through a nebula, darkly In 2009, SARA conducted a small-scale pilot experiment with five groups of scientific users to explore the possibilities of cloud computing in an HPC environment\[2\] -- a project funded by the European Big Grid\[3\] collaboration. After the initial phase, the experiment was judged a success and Big Grid moved to usher in a new phase of self-service computing for the Dutch scientific community. Today, still-experimental but much larger in scale, the SARA HPC cloud is being offered as a service to the Dutch scientific community of astronomers, particle physicists and many others. System expert Jeroen Nijhof told us about the project: "The new HPC cloud environment will provide researchers with a chance to operate within their very own virtual private HPC cluster that can host fully individual configurations, and will operate according to each scientific team's needs. The most attractive part of the offer is, of course, on-demand scalability." Participants can start with their own system images or create a virtual cluster from the bottom-up. Indeed, a user is able to create a copy of their current software environment (from small or personal machines) and weave it into an HPC cluster operating within the cloud, without expensive rewriting or editing of their development and production environments. ## HPC on demand -- getting a lot from a little The HPC cloud is based in Open Nebula\[4\] hosted on a cluster with 128 cores with the following characteristics: - 16 compute nodes with dual quad-core CPUs - 100 TB managed storage. - Host Software: - CFEngine 3 - OpenNebula - KVM Multicore/multiprocess, also MPI and OpenMP. - Virtual Private Compute Cluster: to start multiple VMs in their own private network (vlan) Nijhof adds, "In this beta phase we strive to offer a small production grade environment and we are continuously improving and expanding the technical environment. The software we use is OpenNebula and we have developed a web-based userinterface (ONE-MC) on top of that. We have developed additional software to manage clusters of virtual machines to enable Virtual Private Compute Clusters." Perhaps surprising to some, security is particularly important in a scientific setting so managing security is a major concern in the design. "We've put various mechanisms in place to assure that users are protected from the outside world, other users and vice-versa." "Users are limited as little as possible. Each will have their own Virtual Private Compute Cluster, that can be configured from scratch. They can actually start with an install cd of the operating system of their choice. It is also possible to start by uploading their own pre-configured virtual machine image. As a service we provide configuration templates and provide a community repository for virtual machines. In this repository people can share their own configuration templates and virtual machine images." Because the HPC infrastructure and computing environment are fully configurable, on demand, and to specific needs, users can save the time and effort it would take to port their applications to a specific HPC platform. In some cases such porting can be impossible because the source code is unavailable to the user. This is where virtualisation can provide a solution. Nijhof believes that the new HPC cloud infrastructure service will result in a shorter time from the posing of a scientific question to a forthcoming computational solution. "We're convinced that HPC cloud computing provides a flexible solution to scientist and provides added value to the HPC ecosystem. ## Agile Lifecycle Configuration with CFEngine So how does CFEngine fit into all this? Nijhof explains: "We invested time to search for a proper configuration management system because without it the project would fail due to the continuously improving and expanding goals. We chose CFEngine 3 based on previous experience with CFEngine 2 but we needed the flexibility and modularity that only the new CFEngine brings. We had a really tight schedule for implementing and managing the project. CFEngine 3 makes it really simple to add a new node to the cluster and manage its whole lifecycle. "We were also able to solve another problem. Realizing we did not have a testing environment due to lack of hardware, we made one from some production nodes and with the help of CFEngine 3. We could easily convert the testing nodes back to production or convert them back again. So we basically created a testing environment on demand. We divided the cfengine 3 policy into a global configuration and custom configuration. In that way it's very easy to add other projects besides our cloud into CFEngine 3 as well. Using *methods* was really a good solution to keep everything clean, simple and modular." Returning to the issue of security, Nijhof recognizes that scientific users need to be protected against threats, as scientific sites often attract hackers, and the integrity of scientific data are at stake. "The other big advantage of using CFEngine 3 not only for configuration management but for security checks. By monitoring the promises not kept we can easily see i.e. which files are modified and avoid accidents by proactively repairing possible holes." ## References \[1\] [Top 500 sites](https://web.archive.org/web/20080719235103/http://www.top500.org/site/1496) \[2\] [HPC Cloud project](https://web.archive.org/web/20110122053356/http://www.hpcinthecloud.com/offthewire/HPC-Cloud-Service-Available-for-Evaluation-92509064.html) \[3\] [Big Grid Project](https://www.biggrid.nl/) \[4\] [Open Nebula](https://www.opennebula.org/start) -------------------------------------------------------------------------------- title: "Self-Service Computing with Model Driven Configuration" url: /blog/2010/self-service-computing-with-model-driven-configuration/index.md date: "2010-05-30" -------------------------------------------------------------------------------- *More players are beginning to realize the benefits of the kind of model-driven configuration thinking that CFEngine introduced in the 1990s. Today, this kind of pre-programmed, policy-based configuration and *repair* are becoming essential partners for the commoditization of computing services in the Cloud. CFEngine has been there since before the beginning, and it still doesn't care where your computer is located.* CFEngine introduced the idea of what is now called \`model driven' configuration in the early 1990s. Although the name changes (e.g. policy driven, declarative, etc) the idea is still very simple: instead of describing procedural changes, you describe the desired state or outcome for your system. As CFEngine originator Mark Burgess has often said, "When you're cooking a lot of recipes look the same -- flour, eggs, sugar, water in some kind of order. If you don't know what it is you are aiming for (a cake or a loaf of bread), you can't really tell if you are succeeding. It's the same with configuration -- edit this file, install that package, but to what end?" By forcing yourself to document the intention of the system, by *promising* what it will actually do, system administrators can bring a new found clarity to the data center. As Burgess remarks, "The challenge of the decade is going to be Knowledge Management". Some recent news in this area: - [Rpath integrates with CFEngine](http://agile.dzone.com/news/rpath-adds-integrations-puppet?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+zones/agile+%28Agile+Zone%29) - [Zenoss integrates with CFEngine](http://community.zenoss.org/blogs/zenossblog/2010/04/23/datacenter-barometer-cfengine-revs-up-configuration-management?keepThis=true&TB_iframe=true&height=650&width=1350) - [Upcoming CFEngine Orion Cloud Pack simplifies utility computing](https://cfengine.com/cloud) -------------------------------------------------------------------------------- title: "Is CFEngine "Just an Academic Project"?" url: /blog/2010/is-cfengine-just-an-academic-project/index.md date: "2010-05-23" -------------------------------------------------------------------------------- It has become common recently to refer to CFEngine on the web as an "academic" project. Turns out, though, that this phrase turns out to mean quite different things depending on who's using it, and sometimes it doesn't really mean anything at all. For CFEngine users, the academic connotation is generally perceived as a good one, representing CFEngine's long standing commitment to both research and the educational environment. CFEngine is proudly academic in the sense that it was rationally and systematically designed and developed, with the specific choices based not on personal preferences or aesthetics but on well grounded, peer-reviewed and tested software engineering principles, things too often neglected in commercial software development. For critics of CFEngine, "academic" is most often used as a general purpose pejorative, as if being academic is somehow opposed to being substantial, committed, serious or real, as if CFEngine were only a university toy. In such usage, academic becomes a lazy substitute for the much harder task of performing substantive analysis of or comparisons to CFEngine itself. However you may feel about universities and their supposed ivory tower separation from the real world, at this point it is just silly to think of CFEngine in this way. Since its first release as Open Source code in 1994, CFEngine has been used by commercial, government and, yes, academic organizations throughout the world--and especially in large, mission critical environments in banking, aerospace, oil and gas, military, and so on. The fact is that CFEngine is deeply embedded in some of the largest and most sensitive production sites around the world, as well as being used successfully by scores of small and medium sized organizations of every kind. To his credit, CFEngine's originator, Professor Mark Burgess of Oslo University College, has kept CFEngine closely tied to what he terms "scientific principles" that have been the subject of research for almost twenty years. As he puts it, "What's important is the methods and the integrity of the project, not where it originates and develops. Science is about a culture of understanding, not of marketing and wild claims. There are certain ideas that you can only see when you are not inundated with the so-called real world. Without principles to guide you, you can't say you know what you're doing." Burgess himself has made many visits out into the real world during his career. "I like to go get my hands dirty and immerse myself in an issue and then retreat to a quiet place and reformulate it in abstract terms, stripping away irrelevant details to preserve the essence. That's what science does -- it idealizes messy reality into something we can repeat and learn from." At CFEngine headquarters, the company has built up a staff of like-minded individuals who are willing to ask the challenging questions, and who understand the value of models, experiments and their honest appraisal. They are proud of their academic roots and skills and they take CFEngine in new directions. I think we will all benefit from this fortuitous combination. -------------------------------------------------------------------------------- title: "CFEngine Watches the Tsunami Watchers" url: /blog/2010/cfengine-watches-the-tsunami-watchers/index.md date: "2010-05-07" -------------------------------------------------------------------------------- The Joint Australian Tsunami Warning Centre's [Earthquake Monitoring](https://www.ga.gov.au/earthquakes/) component at Geoscience Australia recently adopted CFEngine's newest worldwide software release to radically simplify the management of their test and production real-time monitoring systems. With so many earthquake events along the [Indo-Australian plate](https://en.wikipedia.org/wiki/Plate_tectonics), and the risk of equally catastrophic Tsunami waves, [Tsunami warnings](https://en.wikipedia.org/wiki/Tsunami) are a mission critical service in the area. The recent 7.4 magnitude [earthquake in Aceh, Indonesia](https://news.bbc.co.uk/2/hi/asia-pacific/8670593.stm) sparked fears of Tsunami along the recently hard-hit areas around the Indian ocean, though major Tsunami usually result from much larger events. Tsunami alerts have the potential to save lives and avert major damage to coastlines. Forewarned is forearmed. Although Tsunami Earthquake detection models can be remarkably accurate, testing of the automatic warning software is of vital importance to avoid both false and missing positives\[1\]. Leading the systems development effort, software engineer Michael Potter says: "I based my decision to use CFEngine on a visit to another earthquake monitoring center in the United States, who were using the older CFEngine 2 with great success in managing their system configuration. After careful analysis of the capabilities of CFEngine 3, we gradually introduced it to manage our test, and eventually our production, environments. We make extensive use of CFEngine's line editing capabilities, which are unique amongst configuration management systems." "We use CFEngine in the context of running operational production software systems, as opposed to system administration, and production software systems need a degree of automation in several of their components. CFEngine has allowed me to transform the way in which my team deploys configurations into our test and production environments. Previously there was a dependence on tar-balls and ad-hoc instructions to configure our environments - now we deploy a small, standard set of identical packages to all our servers in a single release, and let CFEngine work out what components need to be applied on any given server, including restarting different processes as necessary." ## Change management One of the effects of introducing CFEngine is that it became impossible for changes to be made directly to the production system, as CFEngine would always revert changes to the desired state. This prevented ad-hoc changes being made - a downside to earlier approaches. However it also meant that changes had to pass through a single point of management, which reduced the ability of operators to make legitimate short term changes. "The solution that CFEngine allowed was to provide a controlled cross-platform interface to the operators which let them update a special control file. CFEngine itself would then read and adapt it for the production system, on the fly, with its line-editing capability. Hence we now get to have our cake and eat it too: CFEngine prevents anyone from making permanent configuration changes directly to production systems, but our operators still have the degree of control they need by letting CFEngine itself make the changes in a controlled manner." The team uses line-editing heavily. A major benefit of this feature has been the automatic configuration of the test system. Previously the test systems did not fully replicate the production environments, and running something in the test environment did not provide the desired degree of confidence that it would also run in production. Using CFEngine however, engineers have been able to develop a system where they deploy precisely the same configurations and files to the test environment that are destined for production. CFEngine adapts these along the way, turning off the sending of critical alerts to the whole world in the case of a Tsunami Earthquake event. "We hope in future to let CFEngine manage a full, continuous integration suite, by stopping the system at certain times, removing and adding daily build packages, then running a test data suite." "Where do I see my future use of CFEngine heading? More automation, especially around testing, and more integration into our production systems -- particularly around providing more controlled interfaces to the production system, as well as detecting and acting upon certain events or scenarios where some change is required to keep the system running smoothly." -------------------------------------------------------------------------------- title: "U.S. Navy Onboard with CFEngine" url: /blog/2010/u-s-navy-onboard-with-cfengine-94/index.md date: "2010-05-07" -------------------------------------------------------------------------------- *The US Defense Department's principal operational processing center for automated numerical meteorological and oceanographic analyses and predictions FNMOC chooses CFEngine Nova as the solution for their Meteorology and Oceanography Center, ensuring reliable weather reports for the US fleet.* In September 2010, the Naval Inventory Control Point (NICP) purchased CFEngine Nova for The Fleet Numerical Meteorology and Oceanography Center (FNMOC) in Monterey, California. CFEngine is the world technology leader in data center automation. Its software is used on millions of machines worldwide and is known to be versatile, lightweight and faster than lightning. The FNMOC's primary mission is to provide relevant and timely worldwide Meteorology and Oceanography support, of the highest quality, to U.S. and coalition forces from their round the clock Operations Center. It is the Department of Defense's (DoD) primary production site for worldwide computer-generated operational, meteorological and oceanographic analysis. FNMOC produces and delivers weather, ocean and climate information for fleet safety and national Defense. Security is of utmost importance to the military and its classified networks, even for the weather forecast. CFEngine uses FIPS 140-2 validated encryption, which is necessary to run on DoD networks, and its self-healing technology provides a high level of stability. This means few vulnerabilities over the past 17 years, something which is highly valued in the industry. Being in the Navy is not just about sailing the seven seas. Onboard a ship, the reliability of weather forecasting is essential to the safety and security of U.S. Navy personnel and, of course, US defense. The FNMOC provides worldwide weather and oceanographic forecasts on high-performance supercomputers. They produce satellite processing, display capabilities and product distribution systems for the fleet. Versatility is thus a necessary component in managing FNMOC systems. It is the versatility and flexibility of the CFEngine technology which makes CFEngine popular for solving many data center issues. CFEngine is based on its own kind of forecasting models that explain how to automate complex and large installations, with data storms and flurries of their own. The sound theoretical basis ensures the integrity and reliability of services around the clock. CFEngine is probably the most popular data center automation solution available today assuring smooth sailing for its users. -------------------------------------------------------------------------------- title: "A Resource-Effective Solution" url: /blog/2010/a-resource-effective-solution/index.md date: "2010-03-10" -------------------------------------------------------------------------------- Jarle Bjorgeengen of the University of Oslo did a laboratory study showing the efficiency merits of CFEngine. The [report](https://www.usenix.org/publications/login/2010-02/index.html) is available to USENIX members in their ;login: magazine. Resource efficiency is a huge factor when talking about virtualization and configuration management in elastic Cloud Computing. -------------------------------------------------------------------------------- title: "Mark Burgess FLOSS Interview" url: /blog/2010/mark-burgess-floss-interview/index.md date: "2010-02-04" -------------------------------------------------------------------------------- Mark Burgess was interviewed (with a sore throat) on FLOSS weekly. Listen in [here](https://twit.cachefly.net/floss0106.mp3) (mp3 file). -------------------------------------------------------------------------------- title: "CFEngine raising the profile of system administrators" url: /blog/2010/cfengine-raising-the-profile-of-system-administrators/index.md date: "2010-02-04" -------------------------------------------------------------------------------- A new feature in the CFEngine Community Core is attracting some interest from system administrators. It is the simplest of ideas, but then such ideas are often the best. Following CTO Mark Burgess' recent blog with Carolyn Rowland on the [Business Value of System Administration](http://www.iu.hio.no/%7Emark/blog_busval.html), this new feature emerges as a simple way to document the business value accorded to the system administration job. The value\_kept, value\_repaired, value\_notkept settings fall under cfengine [transaction logging](https://cfengine.com/manuals/cf3-reference.html#action-in-_002a) and allow administrators to attach actual monetary (or other) values to promises kept, or issues repaired, or conversely measure the loss of non-compliance in dollar terms (choose your currency). This value is summed and recorded for each execution of CFEngine, and can be turned into graphs for your management reports. "If you combine this with system performance data, and other reports from CFEngine, you begin to build up a pretty compelling case for IT services value. Hopefully this will give skilled system administrators the leverage they need to advance in the view of the more removed managerial levels, guarding the purse strings," says Mark Burgess, author and company founder of CFEngine. System administration (the "IT services department") has long been viewed as little more than a cost centre by business leaders, rather than the creative solution it should represent. Burgess and Rowland's article suggests ways to combat that prejudice, and this feature was something that came out of this. The article summarizes the [USENIX LISA workshops](http://www.usenix.org/event/lisa09/workshops.html#bdim) on business alignment chaired by Mark Burgess. "Of course the amount of value one can create for an organization also increases when savvy administrators use a tool like CFEngine, and there is an additional value that we have not factored into the mix at all yet, but which is one of the biggest focuses for the company: knowledge management." Mark Burgess, adds, "This is part of our Continuing Research into system administration issues. Not everything is about technology, but you'd be surprised where a small technical feature can help." -------------------------------------------------------------------------------- title: "New Board Members" url: /blog/2010/new-board-members/index.md date: "2010-02-04" -------------------------------------------------------------------------------- Moving into a new year, the privately owned and funded CFEngine company has changed its board to include some power members of the Free and Open Software community. "The time has come to change the style of our board work, as we move into a different phase of growth," says CEO Thomas Ryd. Joining the board is Open Source superstar Haavard Nord, former co-founder and CEO of Trolltech, the Norwegian company behind the [Qt library](https://en.wikipedia.org/wiki/Qt_Development_Frameworks), and the basis for KDE. Trolltech (which translates roughly into "technology magic") was recently acquired by Nokia as a strategic part of their software development. Also joining the team is [Christen Krogh](https://www.opera.com/company/speakers/krogh/), Chief Development Officer at Opera Software, who has guided the development of one of the most innovative web browsers in the industry, dominating the games console and handheld market. Both of these complement the founder Mark Burgess's mindset nicely. "We found a good chemistry immediately and there is a lot of excitement in the company and the board about the potential of a company that already has the largest global footprint of any Open Source datacenter automation system." -------------------------------------------------------------------------------- title: "Florida Training Series" url: /blog/2010/florida-training-series/index.md date: "2010-02-01" -------------------------------------------------------------------------------- Regular training sessions are now available directly from CFEngine in Florida. Visit our [training page](https://cfengine.com/pages/training) for more details and regular updates. -------------------------------------------------------------------------------- title: "CFEngine 2 Conversion Tool" url: /blog/2010/cfengine-2-conversion-tool/index.md date: "2010-01-11" -------------------------------------------------------------------------------- Following the development of the community standard library, CFEngine has now released a conversion utility that transforms existing CFEngine 2 policies into a basic CFEngine 3 format. The output can be run in either the Community Edition, CFEngine Nova or any other version of CFEngine going forward. Earlier this year, CFEngine released an upgrade manual for community users. Now with the core transformation utility, commercial customers will be able to save potentially hundreds of hours of conversion time on a large installation of cfengine, moving to version 3. A complete conversion cannot be made automatically, because there are new features in the CFEngine 3 that would not be used if you simply trusted the automation. Also, one can reap many simplifications in CFEngine 3 in terms of expression and efficiency that were not possible in CFEngine 2. Maximizing the win from these features requires some post-processing. The conversion tool provides hints in comment form as you convert the code, to help you maximize the utility of the conversion however. Commercial users of the CFEngine Nova product will have instant gratification from the integrated Knowledge Map. ``` CFEngine Conversion Utility (beta) -> Matrix from /usr/local/sbin/cfconvert -> INPUTS from . /CfengineProjects/Test_Client/cfengine_2_config/ -> OUTPUTS at /tmp/cf_conversion_output -> Commencing pre-scan for common environment -> Pre-scan complete -> Scanning for recognizable control settings -> > convert control setting EmailMaxLines -> > convert control setting cfinputs_version -> > convert control setting smtpserver -> > convert control setting Inform -> > convert control setting AddInstallables -> > convert control setting workdir -> > convert control setting Syslog -> > convert control setting moduledirectory -> > convert control setting moduledirectory -> Start main promise bundle -> Import files detected -> delta-Transformation of "cfagent.global.conf" -> delta-Transformation of "cfagent.freebsd.conf" -> delta-Transformation of "cfagent.ntp.conf" -> delta-Transformation of "cfagent.named.conf" -> delta-Transformation of "cfagent.perfsonarServers.conf" -> delta-Transformation of "cfagent.perfsonar.conf" -> delta-Transformation of "cfagent.owmesh.conf" -> delta-Transformation of "cfagent.owamp.conf" -> delta-Transformation of "cfagent.perfsonarBUOY.conf" -> delta-Transformation of "cfagent.syslog.conf" -> delta-Transformation of "cfagent.bwctl.conf" -> delta-Transformation of "cfagent.perfsonarBUOY.conf" -> delta-Transformation of "cfagent.syslog.conf" -> delta-Transformation of "cfagent.pinger.conf" -> delta-Transformation of "cfagent.perfsonarBUOY.conf" -> delta-Transformation of "cfagent.owmesh.conf" -> delta-Transformation of "cfagent.perfsonarBUOY.conf" -> delta-Transformation of "cfagent.owmesh.conf" -> delta-Transformation of "cfagent.LSRegistration.conf" -> delta-Transformation of "cfagent.freebsd.i386.packages" -> Converting cfservd.cf -> Writing promises.cf -> 11238 lines of core transformed ``` -------------------------------------------------------------------------------- title: "Vim Plugins" url: /blog/2010/vim-plugins/index.md date: "2010-01-11" -------------------------------------------------------------------------------- Neil Watson has generously made some Vim plugins available for CFEngine 3. - [Abbreviations and other help for CFEngine 3 files](https://www.vim.org/scripts/script.php?script_id=2903) - [CFEngine 3 syntax highlighting](https://www.vim.org/scripts/script.php?script_id=2904) -------------------------------------------------------------------------------- title: "Sys Advent on CFEngine 3" url: /blog/2009/sys-advent-on-cfengine-3/index.md date: "2009-12-22" -------------------------------------------------------------------------------- Aleksey Tsalolikhin has written a helpful blog post introducing CFEngine 3. Read the [full article](https://sysadvent.blogspot.com/2009/12/day-24-config-management-with-cfengine.html). -------------------------------------------------------------------------------- title: "Open Promise-Body Library" url: /blog/2009/open-promise-body-library/index.md date: "2009-12-22" -------------------------------------------------------------------------------- CFEngine is built on promises. Promises were chosen as the model for CFEngine's configuration language, because they represent an expression of intention. But expressing your exact intentions in a safe and convergent way (according to the standards you expect from CFEngine) can sometimes be daunting and can result in haphazard nomenclature. Our strategy in CFEngine 3 was to make it easy to express clear and concise intentions without sacrificing any of the power of the tool. Body templates are used for this. The body of a promise tells you what it is about (think "body of document" or "body of contract") and it is composed of multiple issues that have different types. Each so-called constraint in the promise body has the form: left_hand_side => right_hand_side where the left hand side is a CFEngine reserved word, and the right hand side is something you define. The Open Promise-Body library is a collection of right-hand-sides that you can use to solve common problems and standardize your configuration into a set of universally used words. We are making this library an open document so that users can contribute their experiences and solutions to it. If you want to make internationalized translations, or extensions, you can. We encourage you to contribute back and participate in a discussion. | | | | ------------------------ | --------------------------------------------------------- | | The CFEngine input file: | | | HTML friendly version: | | -------------------------------------------------------------------------------- title: "A Big Thank-you to Our Users" url: /blog/2009/a-big-thank-you-to-our-users/index.md date: "2009-12-22" -------------------------------------------------------------------------------- CFEngine began in the early part of 1993. It was the first Open Source configuration management tool and quickly became an internationally used tool. Since then, CFEngine has been reinvented several times through our commitment to basic research, and today CFEngine is installed on over a million computers all over the world. ![](images/tree.png) We have put a lot of work into CFEngine over the past two years to truly solve the challenges faced by system administrators throughout the industry. The arrival of alternative configuration management systems has drawn some attention from CFEngine, but the feedback we get from our users is: \`We don't talk a lot about CFEngine because it *just works*. We couldn't run our business without it...' So we want to shift the attention back at our loyal CFEngine users. Today, there has never been a more exciting time to be using CFEngine. Major release 3, with its Free and Open Source Community Edition, and its enhanced commercial cousin CFEngine Nova, has once again set the bar for self-healing, reliable, convergent automation, and we want to help users to upgrade their older installations to the state of the art. So we are offering some of the fruits of our commercial development back to the community in four parts: - A new release of the CFEngine 3.0.3 source code. - A brand new, standardized upgrade guide for moving from CFEngine 2 to CFEngine 3. [(Upgrade guide)](https://cfengine.com/manuals/cf3-upgrade.html) - The CFEngine Community Open Promise-Body Library initiative, to enable users to exchange and develop an industry-wide *lingua franca* of idioms for CFEngine coding. [(CFEngine Community Open Promise-Body Library)](https://cfengine.com/manuals/CfengineStdLibrary.html) [(cfengine\_stdlib.cf)](https://cfengine.com/manuals/cfengine_stdlib.cf) - A solutions guide that uses the standard body templates for solving key problems in system administration. [(Solutions guide)](https://cfengine.com/manuals/cf3-solutions.html) At CFEngine, our aim is not to take away your job by automation, but to enhance the experience of applying your engineering skills with exciting, cutting edge technology. We hope you enjoy our latest releases, and we wish everyone seasonal greetings from the CFEngine team. -------------------------------------------------------------------------------- title: "System-Repairing Nanobot" url: /blog/2009/system-repairing-nanobot/index.md date: "2009-12-21" -------------------------------------------------------------------------------- A popular theme in science fiction writing today involves microscopic machines that build and repair other machines, clothing and even people. Today scientists are researching this very thing. Our future may be bright indeed. For system administration nano technology is already here. Software agents now exist that will search computer systems and repair defects or build missing components. CFEngine is such an agent. It is able to examine the overall system at a low level including files, processes and network packets and make builds or repairs based on what it finds. The result is a more self reliant system. Consider the average web server. What might the criteria be that describe a healthy web server? The Apache application package is installed. The Apache configuration files' ownership, permissions and content are correct. The Apache process is running with a minimum number of children. The SSL key file's ownership, permissions and content are correct. The content that Apache will serve has the correct ownership, permissions and content. The correct file systems are mounted. File systems should have a minimum amount of free space. The volume of inbound http or https traffic should fall within a desired range. System load should fall within an acceptable range. The volume of non http or https traffic should fall within an acceptable range. CPU temperature should fall within an acceptable range. Log files are rotated regularly to avoid dangerous growth. Temp files are purged regularly to avoid dangerous growth. Other services such as ftp should not be running. A historical approach to this problem would be one of passive monitoring and reactive humans. A monitoring system, often a heterogeneous collection of tools, keeps watch on the system and reports to a central office if anything is found to be outside of the norm. A human is tasked with examining and correcting the problem. How long does this take? Our CFEngine nanobot is able to monitor these criteria and often take action without the need for a human. For example. If file ownerships or permissions are wrong then correct them. If file contents are wrong then copy the correct file from a repository or rebuild the file on the fly. If the number of Apache processes falls outside of the allowed range then restart Apache. If the Apache package is not installed then install it. If file systems are missing then attempt to mount them. If file systems have less than the minimum amount of free space delete certain files, switch over the service to another host or report the incident. If http traffic is outside of the desired range then restart Apache, increase the number of children, start another load balanced host or report the incident. If undesired services or processes are found running on the hosts kill them and report the incident. Our CFEngine nanobot is capable of taking corrective action immediately after detecting a problem. How long will this take compared to the traditional approach of alerting a human and waiting for intervention? It may be a while yet before nanobots can scrub your clothes clean as you wear them but having a nanobot autonomously repair your software system is possible now. -------------------------------------------------------------------------------- title: "Windows Event Logs" url: /blog/2009/windows-event-logswindows-event-logs/index.md date: "2009-12-13" -------------------------------------------------------------------------------- Over the last few months the CFEngine AS development team has added better integration for CFEngine Nova with windows (without need of Cygwin). The most recent development is support for Windows event logs. Event logs are the Windows counterpart to syslog from Unix. The main difference is that event logs aim to group similar log messages, giving each group an event id. A program that creates logs, such as CFEngine Nova, must define the possible event ids, and their meaning. In many applications, only one event id is defined, a generic log message. However, CFEngine Nova defines the following range of event ids, which allows for automatic handling of log messages. | Description | Event ID | Type | | -------------------------------------------- | -------- | ----------- | | Promise kept | 100 | Information | | Promise repaired | 101 | Information | | Promise not repaired due to warn only policy | 102 | Error | | Promise not repaired due to error | 103 | Error | | Report promise | 104 | Information | | Generic information | 105 | Information | | Generic verbose | 106 | Information | | Generic warning | 107 | Warning | | Generic error | 108 | Error | The Nova event logs can be found under the "System" logs. Almost all monitoring products for Windows support reading event logs, and they can thus monitor logs from CFEngine Nova as well. This makes it possible to do more advanced querying on the status of a machine running CFEngine Nova, e.g. to show all promises that have not been kept in a certain time interval. However, we recommend using the Knowledge Map to do more advanced things, as it is specifically made for this purpose and supports all operating systems that CFEngine runs on. ![Promise repaired in Event Viewer](images/winevent-repaired-acl-closeup.png) ![Promise not kept in Event Viewer](images/winevent-notkept-storage.png) -------------------------------------------------------------------------------- title: "Size of CFEngine Installations" url: /blog/2009/size-of-cfengine-installations/index.md date: "2009-11-24" -------------------------------------------------------------------------------- How big are existing CFEngine installations? Until recently, this question was mostly of interest to people researching approaches to automation and scaling. It is also challenging to obtain good data from users as many users under-quote the number of machines they manage. Often this is to avoid attracting attention to themselves. Nevertheless, since forming the cfengine company, we have been collecting data from CFEngine users, weeding out obvious nonsense, such as users who report 9999 systems, or 123456. The results reveal a picture that makes a lot of intuitive sense, so we can be confident that it is correct. Simplified into orders of magnitude, we have: | | | | | - | ----- | ---- | | 1 | 10 | 1294 | | 2 | 100 | 344 | | 3 | 1000 | 66 | | 4 | 10000 | 8 | On a log-log plot this is almost a straight line, indicating a natural power law, of the kind a Professor of Networking would expect. This is a sign of classic organic growth, and shows that big organizations tend to get bigger by accretion, while the majority of organizations are small. -------------------------------------------------------------------------------- title: "Elastic Virtualization" url: /blog/2009/elastic-virtualization/index.md date: "2009-11-16" -------------------------------------------------------------------------------- In spite of being publicly sceptical of cloud computing as a technological concept CFEngine author Mark Burgess announced that CFEngine will support machine and service virtualization within its framework of convergent promises, in order to facilitate the kind of adaptive utility computing that is a growing presence in datacenters. "Elastic adaptation is the one aspect of what is being called Cloud Computing that I see as a real technological challenge," he added. "I have done some research on this with Alva Couch of Tufts University, while he visited Oslo on sabbatical last year, and we have some interesting twists to bring to the table -- without letting the cat out of the bag yet." Initially, CFEngine will interface with popular virtual machine APIs, such as VMware, KVM and others in order to provide basic VM management in a simple way. CFEngine already runs on VMWare ESX Servers, and CFEngine Nova (the commercial version of CFEngine 3) will include special features to integrate with the VMware APIs directly, providing convergent automation to the world of virtualization. This step brings two of the most popular server management solutions a step closer together. See also [The Cloud Minders](https://www.usenix.org/publications/login/2009-04/pdfs/burgess.pdf) by Mark Burgess in the [April 2009 issue of Login](https://www.usenix.org/publications/login/2009-04/index.html). -------------------------------------------------------------------------------- title: "CFEngine 3 on Windows" url: /blog/2009/cfengine-3-on-windows/index.md date: "2009-11-16" -------------------------------------------------------------------------------- Following a sustained effort by the programming team at CFEngine AS, CFEngine Nova (the commercial version of CFEngine 3) will run natively on Windows NT platforms (not merely emulated under the Cygwin framework), with first release just into the new year 2010. Support has been added for registry management and Windows Access Control Lists, as well as integration with Event Manager and other goodies. The plans over the next year include further integration of CFEngine with Active Directory and its group policies. CFEngine still has something to offer Windows users, even with the new tools that Microsoft is bringing to Windows 2008. One thing is integration of Windows resources into the CFEngine Knowledge Map, but also there is the ability to manage security through ACLs, and implement group policies convergently over time (not just one-off), as well as to integrate with a major Unix management system in a universal framework. Early rumours of the release have already led to a flurry of interest for the upcoming software release from a number of companies internationally. -------------------------------------------------------------------------------- title: "List Manipulation" url: /blog/2009/list-manipulation/index.md date: "2009-11-10" -------------------------------------------------------------------------------- New features have been added to the CFEngine development tree, both Community Edition and CFEngine Nova, for list manipulation. The grep() function takes a list and extracts elements that match a given regular expression, returning a sublist. The join() function, which complements the splitstring() function, takes a list and returns a scalar string with elements separated by a delimiter. These functions provide Perl-like operations that reduce the dependence on external scripting. List manipulation is a key enhancement in CFEngine 3, and forms a core paradigm for reducing configuration complexity. ### Upgrade path A cfengine2 to cfengine3 conversion codebook was promised today for community users. The book should be available before Christmas 2009 and will constitute a cookbook of recipes for translating code into the new form. CFEngine Nova supported customers can look forward to a program to assist in the conversion of old CFEngine policies into the superior CFEngine 3 language, speeding things up further. No direct conversion of policies is possible to users, since there are new features in the cfengine 3 language, including auto-documentation of the CFEngine code that must be added by CFEngine engineers. The comments and dependency data enhance reporting of error and log messages and allow reports to be added in the commercial version. These annotations document the intentions behind policy and form the basis of the CFEngine Nova Knowledge Map. Also on front page... ### Partnering with industry CFEngine is now partnering with key industry re-sellers to bring professional support to users of the commercial and community editions. By supporting a complete datacentre package through leading industry suppliers, we are working to simplify CFEngine datacentre deployments for new and existing users. The future of CFEngine's world-spanning footprint depends on our two-pronged approach: constant simplification of the solution, and scaling up of support around the world. Today, we are bridging services across 3 continents to help users as effectively as we can. ### Virtually complete, CFEngine Nova CFEngine is deployed in the world's largest datacentres, the homes of basic Internet services, Cloud Computing and Application Service Providers. Now efforts are underway to better support the key virtualization technologies that are making inroads into hosting centres, to allow better utilization of servers, with the popular technologies including Red-Hat KVM, VMware, and Xen. Investigate the features in the CFEngine 3 Community Edition, or learn about our simplified BDMA framework of CFEngine Nova. -------------------------------------------------------------------------------- title: "Mark Burgess Presents at Google" url: /blog/2009/mark-burgess-presents-at-google/index.md date: "2009-10-26" -------------------------------------------------------------------------------- ### Mark Burgess Presents at Google By Brendan Strejcek - Mon, 26 Oct 2009 Mark Burgess recently gave a presentation on Promise Theory and CFEngine 3 at Google on his way to LISA 08. [Login](https://cfengine.com/login) | [Register](https://cfengine.com/register) -------------------------------------------------------------------------------- title: "CFEngine at OS Bridge" url: /blog/2009/cfengine-at-os-bridge/index.md date: "2009-06-20" -------------------------------------------------------------------------------- Brendan Strejcek talks CFEngine and configuration management with the founders of other open source tools at Open Source Bridge in Portland: https://www.youtube.com/watch?v=ybwdt-Ps3Mc -------------------------------------------------------------------------------- title: "CFEngine 3 Tutorial in Linux Pro Magazine" url: /blog/2009/cfengine-3-tutorial-in-linux-pro-magazine/index.md date: "2009-04-01" -------------------------------------------------------------------------------- The cover feature of the April 2009 issue of Linux Pro Magazine is a tutorial I wrote for introducing CFEngine 3. You can download the [full text of the article in PDF format](https://web.archive.org/web/20120131085918/http://www.linux-magazine.com/w3/issue/101/Cfengine.pdf). -------------------------------------------------------------------------------- title: "Promising a Rose Garden?" url: /blog/2007/promising-a-rose-garden/index.md date: "2007-01-01" -------------------------------------------------------------------------------- "A Promising Approach to Management" is the subtitle of an essay written by Mark Burgess that develops the philosophy that has made CFEngine a flexible and successful solution for autonomic computer management. But there is more to this essay than a cute jingle. According to its author, we should think less about the details of our technologies and more about the promises we make. Download the [full PDF](https://web.archive.org/web/20151115113625/http://www.ustream.tv/recorded/49677732) of the essay. -------------------------------------------------------------------------------- title: "CFEngine Webinars" url: /webinars/index.md --------------------------------------------------------------------------------

Webinars

We host regular webinars to highlight new features, show off projects from the community or explore interesting use cases for CFEngine. To join the next webinar, register here.

-------------------------------------------------------------------------------- title: "Dispatching Ansible with CFEngine" url: /webinars/agent-is-in-065/index.md date: "2026-09-24" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Demo of CFEngine 3.28" url: /webinars/agent-is-in-063/index.md date: "2026-07-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "An MCP for Mission Portal?" url: /webinars/agent-is-in-062/index.md date: "2026-06-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Use our lightweight C library libntech to make a handy json tool" url: /webinars/agent-is-in-061/index.md date: "2026-05-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Improved package management on Enterprise Linux" url: /webinars/agent-is-in-060/index.md date: "2026-04-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "From monolith to modules with cfbs" url: /webinars/agent-is-in-058/index.md date: "2026-02-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Using an Ansible module with CFEngine" url: /webinars/agent-is-in-057/index.md date: "2026-02-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Profiling cf-agent" url: /webinars/agent-is-in-056/index.md date: "2025-12-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "evaluation_order in body file control" url: /webinars/agent-is-in-055/index.md date: "2025-11-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Sneak peek of CFEngine 3.27" url: /webinars/agent-is-in-054/index.md date: "2025-10-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine and decomputing" url: /webinars/agent-is-in-053/index.md date: "2025-09-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "body fsattrs and the immutable bit" url: /webinars/agent-is-in-052/index.md date: "2025-08-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Data-Driven Configuration with classfilterdata()" url: /webinars/agent-is-in-051/index.md date: "2025-07-31" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "AI Roundtable" url: /webinars/agent-is-in-050/index.md date: "2025-06-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Demo of CFEngine 3.26.0" url: /webinars/agent-is-in-049/index.md date: "2025-05-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Using CFEngine for Android application inventory & management" url: /webinars/agent-is-in-048/index.md date: "2025-05-13" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Preview of cfbs analyze" url: /webinars/agent-is-in-047/index.md date: "2025-03-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Classification using genders" url: /webinars/agent-is-in-046/index.md date: "2025-02-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Deduping the Internet with CFEngine" url: /webinars/agent-is-in-045/index.md date: "2025-01-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Cody and Nick's Christmas Special" url: /webinars/agent-is-in-044/index.md date: "2024-12-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Nick & Cody's Thanksgiving Day Special" url: /webinars/agent-is-in-043/index.md date: "2024-11-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Fireside Chat w/ Bas Van der Vlies" url: /webinars/agent-is-in-042/index.md date: "2024-10-31" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Policy language support in Zed, and other editors" url: /webinars/agent-is-in-041/index.md date: "2024-09-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Windows module workshop with CFEngine" url: /webinars/agent-is-in-040/index.md date: "2024-08-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Editing /etc/default/grub and similar files" url: /webinars/agent-is-in-039/index.md date: "2024-07-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Sneak peek of CFEngine 3.24" url: /webinars/agent-is-in-038/index.md date: "2024-06-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Windows package management" url: /webinars/agent-is-in-037/index.md date: "2024-05-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Upgrading to Enterprise" url: /webinars/agent-is-in-036/index.md date: "2024-04-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Automatic grouping of hosts using select_class" url: /webinars/agent-is-in-035/index.md date: "2024-03-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Host visibility with containerized CFEngine" url: /webinars/agent-is-in-034/index.md date: "2024-02-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Interview with Mark Burgess" url: /webinars/agent-is-in-033/index.md date: "2024-01-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Fireside chat w/ Mike Weilgart" url: /webinars/agent-is-in-032/index.md date: "2023-12-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Sneak peek of CFEngine 3.23" url: /webinars/agent-is-in-031/index.md date: "2023-11-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Profiling CFEngine policy" url: /webinars/agent-is-in-030/index.md date: "2023-10-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Basic Docker inventory with CFEngine" url: /webinars/agent-is-in-029/index.md date: "2023-09-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Testing CFEngine policy" url: /webinars/agent-is-in-028/index.md date: "2023-08-31" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine Q&A: Policy questions" url: /webinars/agent-is-in-027/index.md date: "2023-07-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Demo of CFEngine 3.22" url: /webinars/agent-is-in-026/index.md date: "2023-06-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Migrating to cfbs" url: /webinars/agent-is-in-025/index.md date: "2023-05-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Pretty printer (cffmt) demo with Miek Gieben" url: /webinars/agent-is-in-024/index.md date: "2023-04-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Detecting previously hidden malware with Invary & CFEngine" url: /webinars/agent-is-in-023/index.md date: "2023-03-31" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Hackathon: Termux services on Android" url: /webinars/agent-is-in-022/index.md date: "2023-02-23" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Troubleshooting with cf-support" url: /webinars/agent-is-in-021/index.md date: "2023-01-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "The 2022 CFEngine holiday security calendar" url: /webinars/agent-is-in-020/index.md date: "2022-12-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Sneak peek of CFEngine 3.21" url: /webinars/agent-is-in-019/index.md date: "2022-11-22" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Policy examples" url: /webinars/agent-is-in-018/index.md date: "2022-10-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "STIG compliance for CFEngine" url: /webinars/agent-is-in-017/index.md date: "2022-09-29" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine Enterprise for ARM64" url: /webinars/agent-is-in-016/index.md date: "2022-08-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Extending autorun" url: /webinars/agent-is-in-015/index.md date: "2022-07-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Sneak peek of CFEngine 3.20" url: /webinars/agent-is-in-014/index.md date: "2022-06-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Custom promise types" url: /webinars/agent-is-in-013/index.md date: "2022-05-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Spacemacs for CFEngine" url: /webinars/agent-is-in-012/index.md date: "2022-04-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Infrastructure hardening with CFEngine & Lynis" url: /webinars/agent-is-in-011/index.md date: "2022-03-31" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Event-driven CFEngine" url: /webinars/agent-is-in-010/index.md date: "2022-02-24" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Introduction to cf-secret" url: /webinars/agent-is-in-009/index.md date: "2022-01-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Security hardening holiday calendar" url: /webinars/agent-is-in-008/index.md date: "2022-01-03" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Creating a module with CFEngine Build" url: /webinars/agent-is-in-007/index.md date: "2021-11-26" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Running CFEngine on IoT (Part 2)" url: /webinars/agent-is-in-006/index.md date: "2021-10-28" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Running CFEngine on IoT (Part 1)" url: /webinars/agent-is-in-005/index.md date: "2021-09-30" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine Build system (cfbs)" url: /webinars/agent-is-in-004/index.md date: "2021-08-27" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "What's new in CFEngine 3.18?" url: /webinars/agent-is-in-003/index.md date: "2021-08-04" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Community user demo with Jeff Carlson" url: /webinars/agent-is-in-002/index.md date: "2021-06-25" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Debugging with CFEngine policy analyzer" url: /webinars/agent-is-in-001/index.md date: "2021-05-21" -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine White Papers" url: /white-papers/index.md --------------------------------------------------------------------------------

White papers

Learn, analyze and compare

Our white papers analyze CFEngine and its use in detail. We show real and reproducible results, with links to Open Source repositories of test scripts and policies.

-------------------------------------------------------------------------------- title: "Efficient synchronization of tabular data" url: /white-papers/efficient-synchronization/index.md date: "2023-11-14" --------------------------------------------------------------------------------
White paper

Efficient synchronization of tabular data

CFEngine is a configuration management tool that offers a domain-specific language for automating system administration tasks while abstracting away platform-specific differences. Exclusively available to CFEngine Enterprise customers is an additional feature — a reporting system. This system collects invaluable information about managed hosts, by mirroring their states onto a centralized hub. These states are stored as tabular data structures and are synchronized using a log-based synchronization technique. The existing implementation of the reporting system is not optimized for minimal bandwidth consumption, primarily due to scalability limitations in its design. It's worth noting that bandwidth consumption is a crucial concern for CFEngine customers. In response, this paper summarizes an enhanced log-based synchronization technique introduced by Lars Erik Wik, designed to address this challenge.

{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "Ansible and CFEngine scalability" url: /white-papers/ansible-and-cfengine-scalability/index.md date: "2021-01-12" --------------------------------------------------------------------------------
White paper

Ansible and CFEngine Scalability

Scalability is an important feature of any infrastructure management solution. Either the to-be-managed infrastructure is big already or it is expected to grow as the business grows. Over time more and more resources are needed for CI/CD pipelines and more customers use the product(s). Generally, growing a business means more traffic and requests need to be handled by the infrastructure. Hence, scalability is an important metric for comparing infrastructure management tools when deciding which one to use. Or which ones.

{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "Ansible and CFEngine" url: /white-papers/ansible-and-cfengine/index.md date: "2020-09-09" --------------------------------------------------------------------------------
White paper

Ansible and CFEngine

Ansible and CFEngine are two configuration management tools and at first glance they look like competitors - two tools dealing with the same problem, in very different ways. But are they? Maybe they are actually not dealing with the same problem and are not as incompatible as it seems.

{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "solutions" url: /solutions/index.md -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Solutions - Compliance automation - CFEngine" url: /solutions/compliance-automation/index.md date: "2021-01-04" --------------------------------------------------------------------------------

SOLUTIONS

Automate your compliance

Improve your compliance levels through the power of CFEngine's automation. No matter the compliance framework you use, CFEngine provides systems administrators & compliance professionals with the flexibility & visibility needed in order to meet those requirements and easily verify compliance attainment in real-time.

Contact us for info

Continuous infrastructure hardening

Manual system configuration isn't scalable and can lead to an overflow of backlog tasks and lack of compliance. CFEngine not only automates many of these repetitive, day-to-day tasks, but continuously checks systems every 5 minutes to ensure they are always enforced & compliant.

Track changes as they happen

As systems & files change over time, CFEngine provides fine-grained visibility to detect, monitor & track changes that may impact your environment's integrity and leave your infrastructure vulnerable to attacks.

Real-time compliance stats

System configuration is only half the battle. Transparency into your infrastructure's status is critical to verifying compliance and efficiently completing successful audits. Through CFEngine's web UI, Mission Portal, you can quickly verify compliance with pre-built and customizable dashboards & reports.

The true cost of non-compliance

Non-compliance costs more than twice the cost of maintaining compliance.

- Ponemon Institute & Globalscape
-------------------------------------------------------------------------------- title: "Solutions - Infrastructure automation - CFEngine" url: /solutions/infrastructure-automation/index.md date: "2021-01-04" --------------------------------------------------------------------------------

SOLUTIONS

Automate your infrastructure

Infrastructure has increasingly become heterogeneous over the years with the emergence of public and private clouds, containers, hybrid environments, and IoT. This continuous evolution creates a new set of problems for IT Managers & Systems Administrators who, regardless of the IT stack, need complete solutions that increase operational efficiency to meet the needs of the business.

Contact us for info

Know more, react faster.

Organizations often use multiple point solutions meant to solve one specific problem, leading to fragmented information and making it difficult to rely on the data. CFEngine is used as a single source of truth and its rich dataset can be integrated via API to other 3rd party data systems.

Do more with less

CFEngine enables you to minimize or even eliminate manual, time consuming tasks like patch management, system provisioning, access management and other tedious system configuration tasks allowing you to accomplish more in less time with fewer resources.

Future-proof your infrastructure

As new systems are deployed and changes are made, CFEngine helps you maintain consistency across your infrastructure by preventing configuration drift through continuous system checks and automated remediation.

The impact of automation on IT operations

61% of IT professionals say that manual processes are the biggest barrier to productivity.

- Freeform Dynamics, July 2017

AUTOMATION IN ACTION

Reduce costs and boost efficiency through infrastructure automation

LinkedIn, one of the world's leading online professional networks, has enjoyed explosive growth, scaling from 300 machines in 2010 to over 40,000 servers. As a result, the company turned to CFEngine to scale and automate its infrastructure throughout its entire lifecycle - from build and deploy to continuous operations

Download the LinkedIn case study
-------------------------------------------------------------------------------- title: "Solutions - Security automation - CFEngine" url: /solutions/security-automation/index.md date: "2021-01-04" --------------------------------------------------------------------------------

SOLUTIONS

Automate your security

Infrastructure security is no longer an issue we think about a couple times a year. As frequent changes to infrastructure are made and new systems deployed, it is critical to prevent configuration drift to harden infrastructure and meet internal & external requirements.

Contact us for info

Reduce human error

CFEngine helps keep systems up to date and minimizes manual activity by automating system updates and remediating drift across your entire infrastructure.

Maintain consistency

Whether you're deploying new systems at rapid scale or simply trying to gain more control of existing systems, CFEngine helps drive consistent configuration throughout your infrastructure at all times.

Actionable data, not big data

Eliminate the noise of "big data" and expose actionable data by creating custom alerts & dashboards that are relevant to you.

IBM Cyber Security Intelligence Index

Over 95% of all cybersecurity incidents recognize human error as a main contributing factor.

- IBM Security Services
-------------------------------------------------------------------------------- title: "CFEngine Case Studies" url: /case-studies/index.md --------------------------------------------------------------------------------

Case studies

See how our customers benefit from the power of automation

Case studies show how CFEngine is solving real world problems for our users.

-------------------------------------------------------------------------------- title: "HIPAA Vault case study" url: /case-studies/hipaa-vault/index.md date: "2021-01-04" --------------------------------------------------------------------------------
Case study

CFEngine Enables HIPAA Vault to Provide Compliant Hosting

HIPAA Vault has used CFEngine Community since 2013 and upgraded to Enterprise to increase visibility and security. Learn more by downloading the case study.

"We chose CFEngine over others because it is the most trusted and reliable solution in the market. We are more interested in dependability than having the new shiny tool that just hit the market."
Gil Vidals
Founder & CEO, HIPAA Vault
{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "LinkedIn case study" url: /case-studies/linkedin/index.md date: "2021-01-04" --------------------------------------------------------------------------------
Case study

Learn How CFEngine Helps LinkedIn Automate IT Infrastructure

With CFEngine, LinkedIn automates 100k+ servers while reducing costs and boosting efficiency. Learn more by downloading the case study.

"Our operations team will make around 5-10 production changes per day. CFEngine provides the automation framework and gives us the ability to continue to scale operationally."
Mike Svoboda
Systems and Automation Engineer, LinkedIn
{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "Locaweb case study" url: /case-studies/locaweb/index.md date: "2021-01-04" --------------------------------------------------------------------------------
Case study

Locaweb Leverages CFEngine to Automate 30k+ Servers

Locaweb decreases infrastructure deployment times by over 95% while maintaining quality of service and increasing infrastructure visibility. Learn more by downloading the case study.

"The beauty of becoming highly automated with CFEngine is that it allows us to have full control of the whole stack and move very fast. The perceived quality of Locaweb is judged by our IT Operations. To ensure the highest level of quality, we rely on consistent and full control, which is what CFEngine gives us"
Sal Jamil
CTO, Locaweb
{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "Percolate case study" url: /case-studies/percolate/index.md date: "2021-01-04" --------------------------------------------------------------------------------
Case study

CFEngine helps Percolate reduce IT Ops cost by 10x

With 50% growth annually, Percolate chose CFEngine to help scale infrastructure resources without increasing staff. Learn more by downloading the case study.

"Unlike other platforms based on Ruby, Python or Perl, CFEngine has provided Percolate with a lightweight solution based on ‘C' with minimal dependencies that can easily scale to thousands of machines with minimal resource consumption."
Laurent Raufaste
Director of Infrastructure, Percolate
{{< recaptacha case-study-form >}} -------------------------------------------------------------------------------- title: "CFEngine Downloads" url: /downloads/index.md --------------------------------------------------------------------------------

Downloads

-------------------------------------------------------------------------------- title: "Download CFEngine Community" url: /downloads/cfengine-community/index.md date: "2021-01-04" --------------------------------------------------------------------------------

CFEngine Community

CFEngine Community Edition is the Open Source foundation of CFEngine's innovative configuration management technology that helps systems administrators automate and ensure the availability, security and compliance of mission-critical applications and services. CFEngine Community is licensed under the Gnu General Public License GPL.

CFEngine Enterprise offers many advanced features and functionality that transform the Community Edition into an enterprise class software product. A feature comparison guide is available here . You can also download a free 25 license version of the Enterprise Edition.

Please see the supported versions page for the release schedules and the difference between LTS and non-LTS releases.

Previous releases

Please see previous releases if you are looking for a different version.

CFEngine repositories

If you want to get packages from our repositories you can find instructions for how to configure your system here.

Nightlies

Nightly builds are published on a separate page.

Source code

If you are looking for the source code for all CFEngine releases the tarballs are available for download .

-------------------------------------------------------------------------------- title: "Download CFEngine Community - Nightly" url: /downloads/cfengine-community-nightly/index.md date: "2021-01-04" --------------------------------------------------------------------------------

CFEngine Community Nightly

Artifacts from our nightly builds will appear here, if our automated build infrastructure considers them of a good enough quality.

Note that despite some builds here are marked as LTS, these packages are not guaranteed to work in any way. In fact, they might not install or even not exist at all.

-------------------------------------------------------------------------------- title: "Download CFEngine Community - Previous releases" url: /downloads/cfengine-community-previous/index.md date: "2021-01-04" --------------------------------------------------------------------------------

CFEngine Community - Previous releases

-------------------------------------------------------------------------------- title: "Download CFEngine Community - Source code" url: /downloads/cfengine-community-source/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Download source code

Here you will find the source code to officially released versions of CFEngine. If you want access to the latest patches and bleeding edge development, see our GitHub page. CFEngine Community is licensed under the Gnu General Public License GPL.

Please see the supported versions page for the release schedules and the difference between LTS and non-LTS releases.

Current releases

Previous releases

-------------------------------------------------------------------------------- title: "Download CFEngine Enterprise" url: /downloads/cfengine-enterprise/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Download CFEngine Enterprise

Packages for Enterprise versions are available to download below.

If this is your first time downloading, please see the getting started guide in our documentation for more information on how to install and use CFEngine.

Please see the supported versions page for the release schedules and the difference between LTS and non-LTS releases.

 

Previous releases

Please see previous releases if you are looking for a different version.

-------------------------------------------------------------------------------- title: "Download CFEngine Enterprise - Previous releases" url: /downloads/cfengine-enterprise-previous/index.md date: "2021-01-04" --------------------------------------------------------------------------------

CFEngine Enterprise download – Previous releases

The latest version of CFEngine Enterprise can be found on the download page for the current version .

Please note that the Policy Server package will automatically create two system users during installation, cfapache and cfpostgres, if they do not already exist.

-------------------------------------------------------------------------------- title: "Quick installation of CFEngine Enterprise" url: /downloads/quick-install/index.md date: "2021-01-04" --------------------------------------------------------------------------------

Quick installation of CFEngine Enterprise

Getting started tutorial

Recommended for new users

If you're a new user, or haven't already read it, we highly recommend following along with the getting started tutorial in our documentation. Through it, you'll learn how to install CFEngine, explore the Web UI and reports for the first time, download ready to use modules, and more:

Getting started with CFEngine.

The quick install shell script

The quick install script can be useful for more experienced users who are familiar with CFEngine and just want to install the latest LTS release on a couple of Linux machines. If you are interested in installing CFEngine on other platforms, or other versions of CFEngine, see cf-remote or the enterprise download page. Follow the steps below to install and bootstrap CFEngine:
1

Check the requirements

CFEngine hub

  • 64-bit machine with a recent version of Linux
  • 2gb of memory
  • 100mb of disk space per host you plan to connect to
  • Ports 5308 and 443 must be open & hostname must be set

CFEngine client

  • 32/64-bit machine with a recent version of Linux
  • 20mb of memory
  • 20mb of disk space per host you plan to connect to
  • Port 5308 must be open

Find the IP address of both the client and the hub because you'll have to put them into the commands wherever we refer to HUB and CLIENT.

Note: If your hosts have multiple interfaces (multiple IP addresses and multiple networks) it matters which one you use. When we add these IP addresses to the configuration below, it works like a firewall - allowing those IPs and blocking other IPs. In practice, this means that if you are testing with 2 hosts in for example AWS, you'll want to use the private IP address for both the hub and the client for it to work correctly.

2

Set up the hub

Run the following command to download and automatically install the CFEngine hub package:

wget https://s3.amazonaws.com/cfengine.packages/quick-install-cfengine-enterprise.sh  && sudo bash ./quick-install-cfengine-enterprise.sh hub

If your hub and client are on different /16 subnets, meaning the first 2 numbers in their IP addresses (IPv4 dot-decimal notation) are not identical, you will have to specify which IP addresses/subnets are trusted. See the documentation for more details, or simply replace HUB and CLIENT with their respective IP addresses and run this command on the hub:

echo '{"variables": {"default:def.acl": ["HUB", "CLIENT"]}}' | sudo tee /var/cfengine/masterfiles/def.json

With the hub package of CFEngine correctly installed and configured, you must start CFEngine, by bootstrapping the hub to itself. Replace HUB with the appropriate IP address and run this command:

sudo /var/cfengine/bin/cf-agent --bootstrap HUB
3

Set up the client

Run the following command on any machine you'd like to connect to the hub; it will download and automatically install the agent.

wget https://s3.amazonaws.com/cfengine.packages/quick-install-cfengine-enterprise.sh  && sudo bash ./quick-install-cfengine-enterprise.sh agent

Now bootstrap the agent to the hub. Just like above, replace HUB in the following command:

sudo /var/cfengine/bin/cf-agent --bootstrap HUB
4

Set up and log in to CFEngine Mission Portal

In any modern browser, visit the IP address of your hub from step 2 above. You should see the Mission Portal first-time setup page where you need to set a secret code to proceed with the setup. The code was printed in the output of the Hub installation command you ran on the hub in step 2. To obtain a new code, you can run the following command on the hub machine:

sudo cf-hub --new-setup-code

Screenshot of the Mission Portal first-time setup page where you need to set a secret code to proceed with the setup.

Once you have set the secret code, you will be prompted to create an admin user.

Screenshot of the Mission Portal complete first-time setup page where you need to set admin credentials.

What's next?

-------------------------------------------------------------------------------- title: "CFEngine Videos" url: /videos/index.md --------------------------------------------------------------------------------

Videos

A peek into the world of CFEngine

We publish videos of our newest features, conference talks, and developer demos.

-------------------------------------------------------------------------------- title: "Introduction to CFEngine Build" url: /videos/cfgmgmtcamp-2023-introduction-to-cfengine-build/index.md date: "2023-02-07" description: Lars Erik Wik and Ole Herman S. Elgesem talked about CFEngine Build at CfgMgmtCamp 2023. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine Build in Mission Portal" url: /videos/build-in-mp/index.md date: "2022-07-27" description: Find modules and make changes to your infrastructure, all from within the Web UI. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Part 1 - Installation" url: /videos/getting-started-1/index.md date: "2022-03-16" description: Installing CFEngine on an Ubuntu 20 Linux virtual machine in Digital Ocean. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Part 1.5 - Local virtual machine" url: /videos/getting-started-1-5/index.md date: "2022-03-16" description: Using Vagrant and VirtualBox to run VMs locally instead of in the cloud. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Part 2 - Modules from CFEngine Build" url: /videos/getting-started-2/index.md date: "2022-03-16" description: Creating a project, finding and adding modules from CFEngine Build and deploying to the hub. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Part 3 - Reporting & Web UI" url: /videos/getting-started-3/index.md date: "2022-03-16" description: Making changes across your infrastructure and seeing information about all your hosts in our graphical user interface and reports. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "The CFEngine Build System - cfbs" url: /videos/cfbs/index.md date: "2021-08-15" description: The new tool, cfbs, allows you to download and manage CFEngine policy, modules, and dependencies. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Testing the latest CFEngine features locally" url: /videos/vagrant-quickstart/index.md date: "2021-06-05" description: Our vagrant quickstart environment allows you to locally test the latest features in nightly CFEngine builds. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Trigger agent runs and report collection from Mission Portal" url: /videos/trigger-agent-runs-and-report-collection/index.md date: "2021-03-31" description: The host info page now has buttons to trigger policy evaluation and report collection for a host, making it easier to debug and iterate quickly. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Publishing pip packages with GitHub Actions" url: /videos/github-actions/index.md date: "2021-02-11" description: This video demonstrates how easy it is to make new releases of a python package on GitHub, and publish it to pypi.org. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFEngine Inventory in Ansible" url: /videos/cfengine-inventory-in-ansible/index.md date: "2020-11-03" description: Some of CFEngine's API endpoints are directly compatible with Ansible, making it easier to use the 2 together. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Introducing Compliance Reports" url: /videos/compliance-reports/index.md date: "2020-08-28" description: Compliance reports allow you to create custom, high level reports, based on company or regulatory requirements. You can test the beta version today, either in CFEngine 3.16, or by using the nightly packages. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Secret encryption in CFEngine" url: /videos/cf-secret/index.md date: "2020-05-29" description: cf-secret enables end-to-end file encryption in CFEngine. It allows you to easily encrypt a secret, such as an API token, for a specific host in your infrastructure. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "High Availability (HA) Reporting" url: /videos/high-availability-enterprise-reporting/index.md date: "2020-03-13" description: CFEngine HA allows reporting and policy distribution to keep working, even if your hub goes offline. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Modernizing the CFEngine Project" url: /videos/modernizing-cfengine/index.md date: "2020-02-12" description: Vratislav Podzimek and Ole Herman S. Elgesem talked about CI, Travis, Jenkins, GitHub CodeQL, AddressSanitizer, valgrind, codecov, git submodules, and more, at CfgMgmtCamp 2020. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Deploying using cf-remote" url: /videos/deploying-using-cf-remote/index.md date: "2019-04-30" description: This video shows how to remotely deploy CFEngine via ssh, using the cf-remote command line tool. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "CFToolkit: New tooling in CFEngine 3.10, 3.12, and beyond" url: /videos/cftoolkit/index.md date: "2019-02-15" description: Ole Herman S. Elgesem showed off new tooling in CFEngine at CfgMgmtCamp 2019. -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- title: "Introducing the Policy Analyzer" url: /videos/policy-analyzer/index.md date: "2019-01-16" description: Do more from Mission Portal with this powerful and simple-to-use extension. --------------------------------------------------------------------------------