We recently announced the general availability of CFEngine Enterprise 3.6.1. One of the key capabilities added to this maintenance release is a supported upgrade process. In today’s post, I’ll walk you through an outline of the upgrade procedure, which will hopefully provide a good starting point for you to map out the entire process for your CFEngine deployment. Note that the examples here assume starting the upgrade from a CFEngine 3.5.x install, but the same steps are applicable to version 3.0 as well. Don’t forget to refer to our online documentation for the complete set of steps.
Or what we should mean by Distributed Orchestration Orchestrating complicated distributed processes is an unfamiliar aspect of computing that leads to all kinds of confusions. We are not taught how to do it in college, so we end up trying to apply any methods we are taught, often in inappropriate ways. Promise theory paints a very simple picture of distributed orchestration. Rather than imagining that a central conductor (controller) somehow plays every instrument by remote magic wand, in an algorithmic fashion, promise theory says: let every player in an ensemble know their part, and leave them all to get on with it. The result is an emergent phenomenon. The natural dependences on one another will make them all play together. Over-thinking the storyline in a distributed process is the easiest way to get into a pickle. This is a key point: how we tell the story of a process and how it gets executed are two different things. Modern programming languages sometimes pretend they are the same, and sometimes separate the two entirely. Scroll back time to 1992, and the world was having all the same problems as today, in different wrapping. Then there was cron, pumping out scripts on hourly, daily and weekly schedules. This was used not only for running jobs on the system but for basic configuration health checks. Cron scripts were like a wild horde of cats, each with their own lives, hard to bring into some sense of order. In the early 90s the acme of orchestration was to sort out all your cron jobs to do the right thing at the right time. The scripts had to be different on the machines too because the flavours of Unix were quite different – and thus there was distributed complexity. Before CFEngine people would devise devious ways of creating one cronfile for each host and then pushing them out. This was considered to be orchestration in 1992. One of the first use cases for CFEngine was to replace all of this with a single uniform model oriented language/interface. CFEngine was target oriented, because it had to be repeatable. Convergence . In this article I explain why virtual environments and containers are basically this issue all over again. Another tool of this epoch is the make for building software from dependencies. In 1994, Richard Stallman pointed out to me that CFEngine was very like make. Indeed, this ended up influencing the syntax of the language. The Makefile was different, it was the opposite of a script. Instead of starting in a known state and pushing out a sequence of transitions from there, it focused on the end state and asked how can I get to that desired end state? In math parlance, it was a change of boundary condition. This was an incredibly important idea, because it meant that – no matter what kind of a mess you were in – you would end up with the right outcome. This is far more important than knowing where you started from. Makefiles did not offer much in the way of abstraction; you could substitute variables and make simple patterns, but this was sufficient for most tasks, because patterns are one of the most important mechanisms for dealing with complexity. Similarly, make was a serial processor running on a single machine, not really suitable for today’s distributed execution requirements. The main concession to parallelism was the addition of “-j” to parallelize building of dependencies. What was really needed was a model based approach where we could provide answers to the following questions: what, when, where, how and why. So now we come to the world of today where software is no longer shackled to a workstation or a server, but potentially a small cog in a large system. And more than that - it is a platform for commerce in the modern world. It’s not just developers and IT folks who care about having stuff built - it’s everyone who uses a service. Many of the problems we are looking to solve can be couched in the model of a deployment of some kind. Whether it is in-house software (“devops”), purchased off-the-shelf software (say “desktop”) or even batch jobs in HPC clusters, all of these typically pass through a test phase before being deployed onto some infrastructure container, such as a server, process group, or even embedded device. Alas the technologies we’ve invented are still very primitive. If we look back to the history of logic, it grew out of the need to hit objects with projectiles in warfare. Ballistics was the cultural origin of mathematics and logic in the days of Newton and Boole. Even today, we basically still try to catapult data and instructions into remote hosts using remote copies and shells. So if a script is like a catapult, that takes us from one decision to the next in a scripted logic. Another name for this triggered branching process is a chain reaction (an explosion). A Makefile is the opposite: a convergent process like something sliding easily down a drain. The branching logic in a script leads to multitudes of parallel alternative worlds. When we branch in git or version control systems we add to this complexity. In a convergent process we are integrating possible worlds into a consistent outcome. This is the enabler for continuous delivery. So developers might feel as though they have their triggered deployments under control, but are they really. No matter, we can go from this… To this … This picture illustrates for me the meaning of true automation. No one has to push a button to get a response. The response is proactive and distributed into the very fabric of the design – not like an add-on. The picture contrasts how we go from manual labour to assisted manual labour, to a proper redesign of process. Automation that still needs humans to operate it is not automation, it is a crane or a power-suit. CFEngine’s model of promises is able to answer all of the questions what, when, where, how and why, at a basic level and has been carefully designed to have the kind of desired-end-state self-healing properties of a drain. Every CFEngine promise is a controlled implosion that leaves a desired end-state. Today, configuration promises have to be supported across many different scales, from the smallest containers like a user identity, to processes, process groups, virtual and physical machines, local networks, organizational namespaces and even globally spanning administrative domains. How do we do that? The simple answer is that we always do it “from within” – through autonomous agents that collaborate and take responsibility for keeping desired-end-state promises at all levels. Traditionally, we think of management of boxes: server boxes, rack boxes, routing boxes, etc. We can certainly put an agent inside every one of those processing entities… But we also need to be able to address abstract containers, labelled by the properties we use in our models of intent – business purpose. These are things like: linux, smartos, webservers, storage devices, and so on. They describe the functional roles in a story about the business purpose of our system. This brings up an important issue: how we tell stories. Despite what we are taught in software engineering, there is not only one version of reality when it comes to computer systems. There is the story:
This blog is the result of joint technology collaboration between Arista Networks and CFEngine Inc.
Introduction It is an accepted fact that the disciplines of system administration and network administration are traditionally very disjoint: different teams, different knowledge, different tools and different goals. For example, does the networking team know which nodes and applications are affected by a change on a specific switch? Is the network topology actually the way it says it is in the design diagrams? Unfortunately, network design diagrams and server CMDBs quickly grow out of sync with reality in today’s fast-changing world. This makes questions like the above virtually impossible to answer, thus leaving luck to play an important role in what changes lead to downtime, loss of revenue and late-hour emergency fixes. However, times are changing and one of the promises of the DevOps movement is enabling better collaboration across these teams. The first step towards collaboration is to enable holistic visibility of the current configuration of the environment, which includes applications, servers and the network connecting them. A popular use case for CFEngine today is to update server CMDBs with the current information. CFEngine can do this as it is running on all the nodes and provides a server-side API for the inventory - as it was seen 5 minutes ago. The missing piece is to also extend this and enable this in the network layer. In this blog series, a joint-effort between Arista and CFEngine focused on network automation, we will look at how to discover and report complete inventory information with CFEngine running on Arista switches (this part), as well as how to easily extend the information using the new Arista APIs (part two).
If you’ve seen what we have delivered with CFEngine Enterprise 3.6.0 earlier this summer, I’m sure you realized what a feature-rich release this turned out to be. But we didn’t stop there. The CFEngine team has spent the past few weeks validating additional platforms, building a simplified upgrade process, and in the process, even added a few small enhancements to the mix! Today, we’re announcing CFEngine Enterprise 3.6.1 - a maintenance release with a number of goodies that I’m sure you will appreciate. Expanded Platform Coverage CFEngine Enterprise 3.6.1 now supports the following host platforms to provide you with comprehensive coverage within your IT infrastructure.
Thanks to all of you who tuned in for our special webinar this week ‘Drive Financial Services Innovation Through IT Automation’. For those of you who could not attend, the recording is now available below - please be sure to watch and learn how CFEngine Enterprise 3.6.0 can enable your IT organization to automate applications and infrastructure specifically for challenges facing financial services firms:
You can also check out the slides here. Happy viewing!
Today’s financial services firms are operating in an IT environment of ever-increasing complexity, unprecedented regulatory requirements, and a need to move faster than ever before. In order to effectively compete in this new reality, firms are turning to IT automation as a way to scale their systems without sacrificing security, uptime, or compliance. Join us for a special webinar this Thursday July 17th to learn about the new CFEngine Enterprise 3.6.0 and how it can enable your IT organization to automate applications and infrastructure specifically for the challenges facing financial services firms today. Attend this live event to learn how CFEngine Enterprise 3.6.0 can enable your financial services organization to:
As any of you know, CFEngine was a proud Gold Sponsor of the Silicon Valley 2014 DevOps Days Conference last month in Mountain View, along with other industry-leading sponsors such as IBM, CA Technologies, Dell, and Red Hat. If you were unable to attend this sold-out event, we are pleased to announce that a recording of CFEngine CTO Mark Burgess’s talk on ‘The Future of Configuration Management’ is now available online! You can watch Mark’s talk here. To learn more about DevOps Days and find a future event near you, please check out the conference website. We hope to see you at the next event!
Thanks to all of you who tuned in for our ‘Introduction to CFEngine Enterprise 3.6.0’ Webinars last week. For those of you who could not attend, the recording is now available below - please be sure to watch and learn about CFEngine Enterprise 3.6.0 can help you:
Proactively alert on policy drifts in your environment through a simple-to-use, configurable dashboard Gain comprehensive visibility into your IT infrastructure with enhanced compliance and extensible inventory reporting Leverage new language abstractions to improve administration productivity Achieve painless integration with other IT systems using native JSON support
Join CFEngine partner Vertical Sysadmin for a free webinar this Thursday - Getting Started with CFEngine: Basic Concepts and Q&A. This event will cover topics from high-level system architecture to the anatomy of writing a promise to help you get productive quickly with CFEngine. Specific items to be covered include:
Basic terms - Promises, Bundles, Bodies, and Classes Language concepts Common promise types, attributes, and syntax Design Center sketches GUI features of CFEngine Enterprise Date: Thursday, July 10, 2014 Time: 9:00 am to 10:00am London UK, 4:00pm to 5:00pm Singapore (SGT) Register today!
Networking has become a utility in our daily lives, but it is still evolving as a platform for modern commerce. CFEngine has now taken networking beyond simple automation, by bringing modern self-repairing, model-oriented approaches to WebScale management at the IP layer. Network interface configuration is nothing new in the world of configuration management. CFEngine versions 1 and 2 were plugging the obvious hole in the out-of-the-box experience during the mid 1990s, before operating systems finally got their acts together. In 2007, CFEngine even explored Layer 3 WAN management (IP routing by any other name) as a member of the EMANICS network of excellence - but, back then, the market was not ready. The dominant networking vendors were still largely concerned with proprietary architecture, steeped in feature-based competition. However, the more recent arrival of Linux as a network operating system has changed that. The adoption and development of the Linux kernel by several companies has pulled down the vendor wall to let in Open Source. Significantly, for the industry, the growth fueled by Open Source has led to the birth of modern datacenter engineering, and DevOps. Recently, with the discussion of Software Defined Networking (SDN), and the arrival of DevOps in the networking space, several tool providers have waded into the world of Layer 2 networking - bonds, bridges and VLANs - which is the comfort zone of most system administrators. However, the historically complex Layer 3 routing services have thus far been neglected. In order to change this, CFEngine founder Mark Burgess has been working with Dinesh Dutt of Cumulus Networks to come up with a model of Layer 3 networking based on CFEngine’s Promise Theory approach to scalable management.
