This document is an abbreviated version of the CFEngine tutorial (http://cfengine.com/manuals/cf3-tutorial.html).
CFEngine was designed to enable scalable configuration management, for the whole system life-cycle, in any kind of environment. Almost every other system for configuration assumes that there will be a reliable network in place and that changes will be pushed out top-down from an authoritative node. Those systems are useless in environments like
CFEngine does not need reliable infrastructure. It works opportunistically in almost any environment, using few resources. It has few software dependencies. So, not only does it work in all of the traditional fixed-plan scenarios, but it is capable of working in totally ad hoc deployment: an temporary incident room, a submarine drifting on and off line, a satellite or a robot explorer.
One could argue `well I don't need that kind of system, because my network is reliable'. However, your network is not as reliable as you think, and mobility is an increasingly important topic. Even with a very strong redundant network, the services that support the network can be paralyzed by any of a number of failed dependencies or mishaps. It is crucial in a modern pervasive environment that systems remain available, fault tolerant and as far as possible independent of external requirements. This is how to build scalable and reliable services.
| CFEngine works in all the places you think it should, and all the new places you haven't even thought of yet. How do we know? Because it is based on almost 20 years of careful research and experience. |
One of the hardest things in management is to make everyone aware of their roles and tasks, and to be able to rely on others to do the same. Trust is an economic time-saver. If you can't trust you have to verify, and that is expensive.
To improve trust we make promises. A promise is the documentation of an intention to act or behave in some manner. This is what we need to learn to trust systems, no matter whether they are machines or humans.
One CFEngine user once said to me, that the thing that had helped him the most in deploying CFEngine was its design based around voluntary cooperation. “Our main problems were not technical but political – getting everyone to agree in all of our departments around the world”. This was because, for all the technology, it is people who make the decisions and people need to feel that the system is empowering rather than disempowering them.
|
CFEngine works on a simple notion of promises. Everything in CFEngine can be thought of as a promise to be kept by different resources in the system. Combining promises with patterns to describe where and when promises should apply is what CFEngine is all about. |
Humans are good at making decisions and awful at reliable implementation. Machines are pitiful at making decisions and very good at reliable implementation. It makes sense to let each side do the job that they are good at.
The main problem in managing systems is a loss of self-discipline. Discipline does not imply that orders have to be barked from a central command. It only requires that every part of the system knows its job and carries it out seamlessly and flawlessly.
Skilled workers tend to think that it is enough to be smart. In fact this is wrong: smart people tend to be problem solvers and will happily solve the same problem many times, wasting time and effort. Moreover, human intervention is often based on panic and lack of understanding so every time someone logs onto a system by hand, they jeopardize everyone's understanding of the system. Only the self-discipline of stable procedures leads to predictability.
Ad hoc changes are bad because:
People often rile against automation saying that it dehumanizes their work. In fact the opposite is true: forcing humans to do the work of machines, in repetitive and reliable ways is what dehumanizes people. The only way to make progress with a bad habit is to recognize it and be willing to abandon the habit.
CFEngine is a framework. It is not so complex, but it is certainly extensive. Often when trying to describe CFEngine, it seems that there is too much to tell and it is hard to convey in a simple way what the software can do. The picture below shows a few ways in which you can think of CFEngine.
For many users, CFEngine is simply a configuration tool – i.e. software for deploying and patching systems according to a policy. Policy is described using promises – indeed, every statement in CFEngine 3 is a promise to be kept at some time or location. More than this, however, CFEngine is not like most automation tools that `roll out' an image of some software once and hope for the best. Every promise that you make in CFEngine is continuously verified and maintained. It is not a one-off operation, but an encapsulated process that repairs itself should anything deviate from the policy.
That clearly places CFEngine in the realm of automation, which often begs the question: so it's just another scripting language? Certainly CFEngine contains a powerful scripting language, but it is not like any other. CFEngine is not a low level language like Perl, Python or Ruby; it is a language of promises, in which you express very high level intentions about the system and the inner details figure out the algorithms needed to implement the result.
Above all, CFEngine is aimed to promote human understanding of complex processes. Its promises are easily documentable using comments that the system remembers and reminds us about in error reporting. It hides irrelevant and transitory details of implementation so that the intentions behind the promises are highlighted for all to see. This means that the knowledge of your organization can be encoded into the CFEngine language.
| WHY DOES KNOWLEDGE MATTER? 1. Technical descriptions are hard to remember. You might understand your configuration decisions when you are writing them, but a few months later when something goes wrong, you will probably have forgotten what you were thinking. That costs you time and effort to diagnose. 2. Organizations are fragile to the loss of those individuals who code policy. If they leave, often there is no one left who can understand or fix the system. Only with proper documentation is it possible to immunize against loss. |
CFEngine comprises a number of components. In this chapter we'll consider how to build them and what they are for.
A CFEngine system is something like an orchestra. It is composed of any number of computers (players), each of which has its own copy of the music and knows what to play. It might or might not have a conductor to help coordinate the individual parts – that's up to you.
CFEngine's software agents are independent components that run on each individual computer. They can communicate if they need to, as depicted in the figure below. This means you don't have to arrange risky login credentials to run your network – and if something goes wrong with the communications network, CFEngine is where it needs to be to repair or protect the system during the outage.
If the network is not working, CFEngine just skips these parts and continues with what it can do. It is fault tolerant and opportunistic.
cron). It also works as a wrapper, executing and collecting the
output of cf-agent and E-mailing it if necessary to a system
account.
cf-serverd and
request that it execute cf-agent with its existing policy. It
can thus be used to simulate a push of changes to CFEngine hosts, if
their policy includes that they check for updates.
This section explains how CFEngine will operate autonomously in a network, under your guidance. If your site is large (thousands of servers) you should spend some time discussing with CFEngine experts how to tune this description to your environment as scale requires you to have more infrastructure, and a potentially more complicated configuration. The essence of any CFEngine deployment is the same.
There are four commonly cited phases in managing systems, summarized as follows:
These separate phases originate with a model of system management based on transactional changes. CFEngine's conception of management is somewhat different, as transaction processing is not a good model for system management, but we can use this template to see how CFEngine works differently.
In CFEngine, what you build is a template of proposed promises for the
machines in an organization such that, if the machines all make and
keep these promises, the system will function seamlessly as
planned. This is how it works in a human organization, and this is how
is works for computers too.
| ROLL-OUT and ROLL-BACK? You should not think of CFEngine as a roll-out system, i.e. one that attempts to force out absolute changes and perhaps reverse them in case of error. Roll-out and roll-back are theoretically flawed concepts that only sometimes work in practice. With CFEngine, you publish a sequence of policy revisions, always moving forward (because like it or not, time only goes in one direction). All of the desired-state changes are managed locally by each individual computer, and continuously repaired to ensure on-going compliance with policy. |
CFEngine does not make absolute choices for you, like other tools. Almost everything about its behavior is matter of policy and can be changed. However, a structure for use, like the following, is recommended (see the following figure).
In order to keep operations as simple as possible, CFEngine maintains
a private working directory on each machine referred to in
documentation as WORKDIR and in policy by the variable
$(sys.workdir). By default, this is located at
/var/cfengine or C:\var\CFEngine. It contains everything
CFEngine needs to run.
The figure below shows how decisions flow through the parts of a system.
In this introduction, we shall assume that there is only one central
policy distribution server, a specially-appointed server which is
referred to simple as the policy server.
Once a client machine has a copy of the policy, it extracts only those promise proposals that are relevant to it, and implements any changes without human assistance. This is how CFEngine manages change.
|
WHY DO THIS? CFEngine tries to minimize dependencies by decoupling processes. By following this pull-based architecture, CFEngine will tolerate network outages and will recover from deployment errors easily. By placing the burden of responsibility for decision at the top, and for implementation at the bottom, we avoid needless fragility and keep two independent quality assurance processes apart. |
To emphasize the fact that CFEngine is not an imperative programming language, and to keep closely to the nomenclature of Promise Theory, CFEngine uses two concepts throughout: bundles and bodies.
Promises are the fundamental statements in CFEngine. Promises are the policy atoms. If there is no promise, nothing happens.
However, promises can become quite complicated and readability becomes an issue, so it is useful to have a way of breaking them down into independent components. The structure of a promise is this:
<body>) tags in HTML, for example.
A promise body is a list of declarations of the following form:
CFEngine_attribute_type => user_defined_value or template
The CFEngine reserved word body is used to define
parameterized templates for bodies to hide the details of complex
promise specifications. For complex body lists, you must fill in a
body declaration as an `attachment' to the promise, e.g.
files: "/tmp/promiser" # Promiser perms => myexample; # The body is just one line, # but needs an attachmentThe attachment is declared like this, with a `type' that matches the left hand side of the declaration in the promise:
body perms myexample
{
mode => "644";
owners => { "mark", "sarah", "angel" };
groups => { "users", "sysadmins", "mythical_beasts" };
}
The structure is this:
promiser
LVALUE => RVALUE
..
body LVALUE RVALUE
{
LVALUE => RVALUE;
LVALUE => RVALUE;
}
|
Another way of looking at it is this:
promiser
CFEngine_word => user_defined_value
..
body CFEngine_word user_defined_value
{
CFEngine_word => user_defined_value;
CFEngine_word => user_defined_value;
...
}
|
Body attachments are required items. You cannot choose to put the attachments in-line. This is a lesson that was learned from CFEngine 2. Readability is quickly lost if too many details are placed in-line.
Some promises in CFEngine are implicit and hard-coded into the program.
For example, the fact that CFEngine looks for a number of files to read and
execute them in a sequence cannot be changed.
However, you can change the behavior of such promises by setting control
parameters. These are formally parts of the `promise body', so we use the body structure to set them. Each agent, (CFEngine software component) has a special body whose name is control, used for setting these parameters. For cf-agent and cf-serverd we can have:
body agent control
{
bundlesequence => { "test" };
}
body server control
{
allowconnects => { "127.0.0.1" , "::1", @(def.acl) };
}
A bundle is a simple concept. A bundle is merely a collection of promises in a `sub-routine-like' container. The purpose of bundles is to allow you greater flexibility to break down the contents of your policies and give them names. Bundles also allow you to re-use promise code by parameterizing it.
Like bodies, bundles also have `types'. Bundles belong to the agent that
is used to keep the promises in the bundle. So cf-agent has bundles
declared as
bundle agent my_name
{
}
The cf-serverd program has bundles declared as:
bundle server my_name
{
}
and so on.
Variables and classes defined inside bundles are not directly visible outside. All variables in CFEngine are globally accessible, however if you refer to a variable by ‘$(unqualified)’, then it is assumed to belong to the current bundle. To access any other (scalar) variable, you must qualify the name using the name of the bundle in which it is defined: ‘$(bundle_name.qualified)’.
Some promise types, like var, classes may be made
by any agent. These are called common promises. Bundles of type common
are special. They may contain common promises.
Classes defined in common bundles have global scope.
The syntax of CFEngine follows a simple pattern in all cases and has a few simple rules:
bundle agent-type identifier
{
...
}
body constraint_type template_identifier
{
...
}
matching and expanding on a reference inside a promise of the form ‘constraint_type => template_identifier’.
cfengine_word => user_defined_template(parameters)
user_defined_template
builtin_function()
"quoted literal scalar"
{ list }
In each of these cases, the right hand side is a user choice.
Once you have learned this pattern, it will make sense anywhere in the program. The figure below illustrates this pattern. Some words are reserved by CFEngine, and are used as types or categories for talking about promises. Other words (in blue) are to be defined by you. Look at the examples and try to identify these patterns yourself.
Everything in CFEngine 3 can be interpreted as a promise. Promises can be made about all kinds of different subjects, from file attributes, to the execution of commands, to access control decisions and knowledge relationships.
This simple but powerful idea allows a very practical uniformity in CFEngine syntax. There is only one grammatical form for statements in the language that you need to know and it looks generically like this:
type:
classes::
"promiser" -> { "promisee1", "promisee2", ... }
attribute_1 => value_1,
attribute_2 => value_2,
...
attribute_n => value_n;
We speak of a promiser (the abstract object making the promise), the promisee is the abstract object to whom the promise is made, and then there is a list of associations that we call the `body' of the promise, which together with the promiser-type tells us what it is all about.
| The promiser is always the object affected by the promise. |
Not all of these elements are necessary every time. Some promises contain a lot of implicit behavior. In other cases we might want to be much more explicit. For example, the simplest reports promise looks like this:
reports:
"hello world";
And the simplest commands promise looks like this
commands:
"/bin/echo hello world";
This promise has default attributes for everything except the `promiser', i.e. the command string that promises to execute. A more complex promise contains many attributes:
# Promise type
files:
# promiser -> promisee (no curly braces needed if only one)
"/home/mark/tmp/test_plain" -> "system blue team",
# attribute => value
comment => "This comment follows the rule for knowledge integration",
perms => owner("@(usernames)"),
create => "true";
The list of promisees is not used by CFEngine except for documentation, just as the comment attribute (which can be added to any promise) has no actual function other than to provide more information to the user in error tracing and auditing.
You see several kinds of object in this example. All literal strings
(e.g. "true") in CFEngine 3 must be quoted. This provides
absolute consistency and makes type-checking easy and error-correction
powerful. All function-like objects (e.g. users("..")) are
either built-in
special functions or parameterized templates which contain the `meat'
of the right hand
side.
The words commands, and files are built-in promise
types. Promise types generally belong each to a particular component
of CFEngine, as the components are designed to keep different kinds of
promises. A few types, such as vars, classes and
reports are common to all the different component bundles. You
will find a full list of the promise types that can be made by the
different components in the reference manual.
When writing promises, get into the habit of giving every promise a comment that explains its intention.
Also, give related promises handles, or labels that can be used to refer to them by.
files: "/var/cfengine/inputs" handle => "update_policy", comment => "Update the configuration from a master server", perms => system("600"), copy_from => mycopy("$(master_location)","$(policy_server)"), depth_search => recurse("inf"), file_select => input_files, action => immediate;If a promise affects another promise in some way, you can make the affected promise one of the promisees, like this:
access: "/master/cfengine/inputs" -> { "update_policy", "other_promisee" }, comment => "Grant access to policy to our clients", handle => "serve_updates", admit => { "217.77.34.*" };
Conversely, if a promise might depend on another in some (even indirect) way, document this too.
files: "/var/cfengine/inputs" comment => "Update the configuration from a master server", handle => "update_policy", depends_on => "serve_updates", perms => system("600"), copy_from => mycopy("$(master_location)","$(policy_server)"), depth_search => recurse("inf"), file_select => input_files, action => immediate;
Get into the habit of adding the cause-effect lines of influence. Enterprise editions of CFEngine will track the dependencies between these promises and map out impact analyses.
CFEngine decisions are made behind the scenes and the results of certain true/false propositions are cached in Booleans referred to as `classes'. There are no if-then-else statements in CFEngine; all decisions are made with classes.
CFEngine runs on every computer individually and each time it wakes up the underlying generic agent platform discovers and classifies properties of the environment or context in which it runs. This information is effectively cached and may be used to make decisions about configuration.
Classes fall into hard (discovered) and soft (user-defined) types. A single hard class can be one of several things:
ultrix, sun4, etc.
www.sales.company.com and
www.research.company.com have the same unqualified name – www.
Monday, Tuesday,
Wednesday, ..).
Hr00,
Hr01 ... Hr23).
GMT_Hr00, GMT_Hr01 ...
GMT_Hr23).
This is consistent the world over, in case you need virtual
simultaneity of change
coordination.
Min00, Min17 ... Min45).
Min00_05,
Min05_10 ... Min55_00)
Q1, Q2, Q3, Q4).
Day1, Day2, ... Day31).
January, February, ... December).
Yr1997, Yr2004).
Night,Morning,Afternoon,Evening, which fall
into six hour blocks
starting at 00:00 hours.
ipv4_192_0_0_1,
ipv4_192_0_0, ipv4_192_0, ipv4_192).
To see all of the classes define on a particular host, run
host# cf-promises -v
as a privileged user. Note that some of the classes are set only
if a trusted link can be established with cfenvd, i.e. if both
are running with privilege, and the /var/cfengine/state/env_data
file is secure. More information about classes can be found in
connection with
allclasses.
User-defined or soft classes are defined in bundles. Bundles of type
common yield classes that are global in scope, whereas in all
other bundle types classes are local. Soft classes are evaluated when
the
bundle is evaluated. They can be based on test functions or simply from
other classes:
bundle agent myclasses { classes: "solinus" expression => "linux||solaris"; # List form useful for including functions "alt_class" or => { "linux", "solaris", fileexists("/etc/fstab") }; "oth_class" and => { fileexists("/etc/shadow"), fileexists("/etc/ passwd") }; reports: alt_class:: # This will only report "Boo!" on linux, solaris, or any system # on which the file /etc/fstab exists "Boo!"; }
Classes may be combined with the operators listed here in order from highest to lowest precedence:
So the following expression would be only true on Mondays or Wednesdays from 2:00pm to 2:59pm on Windows XP systems:
(Monday|Wednesday).Hr14.WinXP::
Consider the following more advanced example. Promises in bundles of type ‘common’ are global in scope – all other promises are local to the scope of their bundle.
body common control { bundlesequence => { "g","ls_1", "ls_2" }; } ################################# bundle common g { classes: # The promise "zero" is always satisfied , and is global in scope "zero" expression => "any"; } ################################# bundle agent ls_1 { classes: # The promise "one" is always satisfied , and is local in scope to ls_1 "one" expression => "any"; } ################################# bundle agent ls_2 { classes: # The promise "two" is always satisfied , and is local in scope to ls_2 "two" expression => "any"; reports: zero.!one.two:: # This report @b{will} be generated "Success"; }
Here we see that class ‘zero’ is global while classes ‘one’ and ‘two’ are local. The report `Success' result is therefore true because only ‘zero’ and ‘two’ are in scope in the ‘ls_2’ bundle (and the class expression for bundle ‘ls_2’ requires that both ‘zero’ and ‘two’ be true and that ‘one’ not be true).
CFEngine is controlled by a series of locks which prevent it from checking promises too often, and which prevent it from spending too long trying to verify promises it already verified recently. The locks work in such a way that you can start several CFEngine processes simultaneously without them interfering with each other. You can control two things about each kind of action in the action sequence:
You can set these values either globally (for all actions) or for each action separately. If you set global and local values, the local values override the global ones. All times are written in units of minutes. Global setting is in the control body:
body agent control { ifelapsed => "60"; # one hour }
or locally in the transaction bodies:
body action example { ifelapsed => "90"; # 1.5 hours }
These locks do not prevent the whole of cf-agent from running, only atomic promise checks. Several different atoms can be run concurrently by different cf-agents. The locks ensure that atoms will never be started by two cf-agents at the same time, or too soon after a verification, causing contention and wasting CPU cycles.
A key difference in CFEngine 3 compared to earlier versions is the presence of types. Types are a mechanism for associating values and checking consistency in a language. Once again, there is a simple pattern to types in CFEngine.
The principle is very simple: types exist in order to match like a plug-socket relationship. In the examples above, you can see two places where types are used to match templates:
bundle TYPE name # matches TYPE to running agent
{
}
lvalues => rvalue constraints:
body TYPE name # matches TYPE => name in promise
{
}
CFEngine variables have two meta-types: scalars and lists. A scalar is a single value,
a list is a collection of scalars. Each scalar may have one of three types:
string, int or real. Typing is dynamic, so these are
interchangeable in many instances. However arguments to special functions check legal
type for consistency.
Integer constants may use suffixes to represent large numbers.
Variables (or "variable definitions") are also promises – the promise to represent their values. We can write these in any promise bundle. CFEngine recognizes two variable object types: scalars and lists (lists contain 0 or more objects)1, as well as three data-types (string, integer and real). Typing in CFEngine is dynamic, as in Perl and other scripting languages. Thus variables of any data-type may be used as strings.
Scalar variables hold a single value. The are declared as follows:
bundle <type> name
{
vars:
"my_scalar" string => "String contents...";
"my_int" int => "1234";
"my_real" real => "567.89";
}
The ‘<type>’ indicates that any kind of bundle applies here. Scalar variables are referenced by ‘$(name)’ (or ‘${name}’) and they represent a single value at a time.
List variables hold several values. The are declared as follows:
bundle <type> name
{
vars:
"my_slist" slist => { "list", "of", "strings" };
"my_ilist" ilist => { "1234", "5678" };
"my_rlist" rlist => { "567.89" };
}
An entire list is referred to with the at symbol ‘@’, but it does not usually make sense to use this reference in a string. For instance
reports:
cfengine_3::
"My list is @(my_slist)";
means nothing and cannot be expanded (it does not generate an error, but instead inserts the text @(my_slist) into the string); but if we use the scalar reference to a list variable, CFEngine will iterate over the values in the list essentially making this into a list of promises.
To summarize:
vars: "longlist" slist => { @(shortlist), "plus", "plus" }; "shortlist" slist => { "you", "me" };
The declaration order does not matter – CFEngine will execute the promise to assign the variable ‘@(shortlist)’ before the promise to assign the variable ‘@(longlist)’.
Instead of doing this in some arbitrary way, with possibility of name collisions, CFEngine asks you to make this explicit. There are two possible approaches.
The first uses parameterization to map a global list into a local context.
# # Show access of external lists. # # - to pass lists globally, use a parameter to dereference them # body common control { bundlesequence => { hardening(@(va.tmpdirs)) }; } ######################################################### bundle common va { vars: "tmpdirs" slist => { "/tmp", "/var/tmp", "/usr/tmp" }; } ########################################################## bundle agent hardening(x) { classes: "ok" expression => "any"; vars: "other" slist => { "/tmp", "/var/tmp" }; reports: ok:: "Do $(x)"; "Other: $(other)"; }
This alternative uses a direct `short-circuit' approach to map the global list into the local context.
# # Show access of external lists. # body common control { bundlesequence => { hardening }; } ######################################################### bundle common va { vars: "tmpdirs" slist => { "/tmp", "/var/tmp", "/usr/tmp" }; } ########################################################## bundle agent hardening { classes: "ok" expression => "any"; vars: "other" slist => { "/tmp", "/var/tmp" }; "x" slist => { @(va.tmpdirs) }; reports: ok:: "Do $(x)"; "Other: $(other)"; }
cf_nullAs of CFEngine core version 3.1.0, the value ‘cf_null’ may be used as a NULL value within lists. This value is ignored in list variable expansion.
vars: "empty_list" slist => { "cf_null" };
Array variables are written with ‘[’ and ‘]’ brackets, e.g.
bundle agent example { vars: "component" slist => { "cf-monitord", "cf-serverd", "cf-execd" }; "array[cf-monitord]" string => "The monitor"; "array[cf-serverd]" string => "The server"; "array[cf-execd]" string => "The executor, not executioner"; commands: "/bin/echo $(component) is" args => "$(array[$(component)])"; }
Arrays are associative and may be of type scalar or list. Enumerated arrays are simply treated as a special case of associative arrays, since there are no numerical loops in CFEngine. Special functions exist to extract lists of keys from array variables for iteration purposes.
Thus one could have written the example above in the form of the following example:
bundle agent array { vars: "v[index_1]" string => "value_1"; "v[index_2]" string => "value_2"; "parameter_name" slist => getindices("v"); reports: Yr2008:: "Found index: $(parameter_name)"; }
If you are looking for loops in CFEngine then we need to reprogram you a little, as you are thinking like a programmer! CFEngine is not a programming language that is meant to give you low level control, but rather a set of declarations that embody processes. It's the difference between the gears on a bicycle and the automated transmission in a transporter.
Loops are executed implicitly in CFEngine, but there is no visible mechanism for it – because that would steal attention from the intention of the promises. The way to express them is through lists.
Loops are really a way to iterate a variable over a list. Try the following.
body common control { bundlesequence => { "example" }; } ########################################################### bundle agent example { vars: # This is a list "component" slist => { "cf-monitord", "cf-serverd", "cf-execd" }; # This is an associative array "array[cf-monitord]" string => "The monitor"; "array[cf-serverd]" string => "The server"; "array[cf-execd]" string => "The executor, not executionist"; reports: cfengine_3:: "$(component) is $(array[$(component)])"; }The output looks something like this:
/usr/local/sbin/cf-agent -f ./unit_loops.cf -K
R: cf-monitord is The monitor
R: cf-serverd is The server
R: cf-execd is The executor, not executionist
You see from this that, if we refer to a list variable using the scalar reference operator ‘$()’, CFEngine interprets this to mean “please iterate over all values of the list”. Thus, we have effectively a `foreach' loop, without the attendant syntax.
The following promise types may be used in any bundle:
varsclassesreportsThese additional promise types may be used only in agent bundles
commandsdatabasesfilesinterfacesmethodspackagesstorageThese promise types belong to other components:
accesscf-serverd.
measurementscf-monitord (CFEngine Nova and above).
rolescf-agent remotely, in cf-serverd.
topicscf-know.
occurrencescf-know.
If you are impatient to get hands-on experience, now might be a good time to take a break from Concepts and try out your first promises (http://cfengine.com/manuals/cf3-tutorial.html#First-promises. Still, since knowledge management is an integral part of CFEngine, we strongly recommend to read the following section on this very issue sooner rather than later.
A unique aspect of CFEngine, that is fully developed in the commercial editions of the software, its ability to enable integrated knowledge management as part of an automation process, and to use its configuration technology as a `semantic' documentation engine.
Knowledge management is the challenge of our times. Organizations frequently waste significant effort re-learning old lessons because they have not been documented and entered into posterity. Now you can alleviate this problem with some simple rules of thumb and even build sophisticated index-databases of documents.
The learning curve for configuration management systems has been the brunt of frequent criticism over the years. Users are expected to either confront the informational complexity of systems at a detailed level, or abandon the idea of fine control altogether. This has led either to information overload or over-simplification. The ability to cope with information complexity is therefore fundamental to IT management
CFEngine introduced the promise model for configuration in order to flatten out this learning curve. It can lead to simplifications in use, because a lot of the thinking has been done already and is encapsulated into the model. One of its special properties is that it is both a model for system behaviour and a model for knowledge representation (this is what declarative languages seek to be, of course). More specifically, it incorporated a subset of the ISO standard for `Topic Maps', an open technology for semantic indexing of information resources. By bringing together these two technologies (which are highly compatible), we end up with a seamless front-end for sewing together and browsing system information.
Knowledge management is a field of research in its own right, and it covers a multitude of issues both human and technological. Most would agree that knowledge is composed of facts and relationships and that there is a need both for clear definitions and semantic context to interpret knowledge properly; but how do we attach meaning to raw information without ambiguity?
Knowledge has quite a lot in common with configuration: what after all is knowledge but a configuration of ideas in our minds, or on some representation medium (paper, silicon etc). It is a coded pattern, preferably one that we can agree on and share with others. Both knowledge and configuration management are about describing patterns. A simple knowledge model can be used to represent a policy or configuration; conversely, a simple model of policy configuration can manufacture a knowledge structure just as it might manufacture a filesystem or a set of services.
Knowledge only truly begins when we write things down:
The trouble is that writing is something people don't like to do, and few are very good at. To an engineer, it can feel like a waste of time, especially during a busy day, to break off from the doing to write about the doing. Also, writing requires a spurt of creative thinking and engineers are often more comfortable with manipulating technical patterns and notations than writing fluent linguistic formulations that seem overtly long-winded.
CFEngine tries to bridge this gap by making documentation simple and part of the technical configuration. CFEngine's knowledge agent then uses AI and network science algorithms to construct a readable documentation from these technical annotations. It can do this because a lot of thought has already gone into the meaning of the promise model.
The beginning of knowledge is to annotate the technical specifications. Remember that the point of a promise is to convey an intention. When writing promises, get into the habit of giving every promise a comment that explains its intention. Also, expect to give special promises handles, or helpful labels that can be used to refer to them in other promise statements. A handle could be something dumb like `xyz', but you should try to use more meaningful titles to help make references clear.
files: "/var/cfengine/inputs" handle => "update_policy", comment => "Update the CFEngine input files from the policy server", perms => system("600"), copy_from => rcp("$(master_location)","$(policy_server)"), depth_search => recurse("inf"), file_select => input_files, action => immediate;If a promise affects another promise in some way, you can make the affected one promise one of the promisees, like this:
access: "/master/CFEngine/inputs" -> { "update_policy", "other_promisee" }, handle => "serve_updates", admit => { "217.77.34.*" };
Conversely, if a promise might depend on another in some (even indirect) way, document this too.
files: "/var/cfengine/inputs" handle => "update_policy", comment => "Update the CFEngine input files from the policy server", depends_on => { "serve_updates" }, perms => system("600"), copy_from => rcp("$(master_location)","$(policy_server)"), depth_search => recurse("inf"), file_select => input_files, action => immediate;
This use of annotation is the first level of documentation
in CFEngine.
The annotations are used internally by CFEngine to provide meaningful
error messages with context and to compute dependencies that reveal
the existence of process chains. These can be turned into a topic map
for browsing the policy relationships in a web browser, using
cf-know.
| The CFEngine Knowledge Map is only available in commercial editions of the software, where the necessary support to set up and maintain this technology can be provided. |
CFEngine's model of promises can also be used to promise information
and its relevance in different contexts. The Knowledge agent cf-know
understands three kinds of promise.
topics:things:occurrences:CFEngine is capable of automating the documentation of a policy, using basic annotations provided above, as a knowledge map. They require very little effort from the user. If you are using the Community Edition of CFEngine, you can develop a topic map, but we do not support the backend technology without a commercial license. In either case, once you become familiar with the use of Topic Maps, you will want to extend your knowledge manually to incorporate things like:
So let us spend a while showing how to encode knowledge in
topic maps
using cf-know.
The kind of result you can expect is shown in the pictures below. The
example figures show typical pages generated by the knowledge agent
cf-know. The first of these shows how we use the technology to
power the web knowledge base in the commercial CFEngine product.
In this use, all of the data are based on documentation for the CFEngine software, and most of the relationships are manually entered.
For a second example, consider how CFEngine can generate such a knowledge map analysis of its own configuration (self-analysis). The data in the images below describe the CFEngine configuration promises. One such page is generated, for instance, for each policy promise, and pages are generated for reports from different computers etc. You can also create you own `topic pages' for any local (enterprise) information that you have.
In this example, the promise has been given the promise-handle
update_policy, and the associations and the lower graph shows
how this promise relates to other promises through its documented
dependencies (these are documented from the promisees and
depends_on attributes of other promises.).
The example page shows two figures, one above the other. The upper figure shows the thirty nearest topics (of any kind) that are related to this one. Here the relationships are unspecific. This diagram can reveal pathways to related information that are often unexpected, and illustrates relationships that broaden one's understanding of the place the current promise occupies within the whole.
Although the graphical illustrations are just renderings of semantic associations shown more fully in text, they are useful for visualizing several levels of depth in the associative network. This can be surprisingly useful for brainstorming and reasoning alike. In particular, one can see the other promises that could be affected if we were to make a change to the current promise. Such impact analyses can be crucial to planning change and release management of policy.
|
A knowledge base is a slightly improved implementation of a Topic Map which is an ISO standard technology. A topic map works like an index that can point to many different kinds of external resources, and may contain simple text and images internally. So you use it to bind together documents of any kind. A CFEngine knowledge base is not a new document format, it is an overlay map that joins ideas and resources together, and displays relationships. |
Topic maps are really electronic indices, but they form and work like webs. A topic is the technical representation of a `subject', i.e. anything you might want to discuss, abstract or physical e.g. an item of `abstract knowledge', which probably has a number of concrete exemplars. It might be a person, a machine, a quality, etc.
Topics can be classified into boxes called topic-types so that
related
things can be collated and unrelated things can be separated, e.g.
types allow us to distinguish between rmdir the Unix utility
and rmdir the Unix system-call.
Each typed topic can further point to a number of references or exemplars called occurrences. For instance, an occurrence of the topic `computer' might include books, web documents, database entries, physical manifestations, or any other information. An occurrence is a reference that exemplifies the abstract topic. Occurrence references are like the page numbers in an index.
A book index typically has `see also' references which point from one topic to another. Topic Maps allow one to define any kind of association between topics. Unlike an ordinary index, a topic map has a rich (potentially infinite) variety of cross reference types. For instance,
topic_1 ``is a kind of'' topic_2
topic_1 ``is improved by'' topic 2
topic_1 ``solves the problem of'' topic_2
The topic map model thus has three levels of containers:
Contexts map conveniently into CFEngine classes. Topics map conveniently into promisers. Occurrences also map to promisers of a different type. These three label different levels of granularity of meaning. Contexts represent a set of topics that might be relevant, which in turn encompass a set of occurrences of resources that contain actual information about the topics in that context. The primacy of topics in this stems from their ability to form networks by association.
The classic approach to information modeling is to build a hierarchical decomposition of non-overlapping objects. Data are manipulated into non-overlapping containers which often prove to be overly restrictive. Topic maps allow us to avoid the kinds of mistakes that have led to monstrosities like the Common Information Model (CIM) with its thousands of strictly non-overlapping type categories.
Each topic allows us to effectively `shine a light' onto the occurrences of information that highlight the concepts pertinent to the topic somehow.
You can use cf-know to render a topic map either as text (for
command line
use) or as HTML (for web rendering). We begin with the text rendering
as it requires less
infrastructure. You will just need a database.
Try typing in the following knowledge promises:
body common control
{
bundlesequence => { "tm" };
}
###################################################
bundle knowledge tm
{
topics:
"server" comment => "Common name for a computer in a desktop";
"desktop" comment => "Common name for a computer for end users";
programs:: # context of programs
"httpd" comment => "A web service process";
"named" comment => "A name service process";
services::
"WWW" comment => "World Wide Web service",
association => a("is implemented by",
"programs::httpd",
"implements");
# if we don't specify a context, it is "any"
"WWW" association => a("looks up addresses with",
"named",
"serves addresses to");
occurrences:
httpd::
"http://www.apache.org"
represents => { "website" };
}
###################################################
body association a(f,name,b)
{
forward_relationship => "$(f)";
backward_relationship => "$(b)";
associates => { $(name) };
}
The simplified things interface is similar, but uses fixed relations:
bundle knowledge company_knowledge
{
things:
regions::
"EMEA" comment => "Europe, The Middle-East and Africa";
"APAC" comment => "Asia and the Pacific countries";
countries::
"UK" synonyms => { "Great Britain" },
is_located_in => { "EMEA", "Europe" };
"Netherlands" synonyms => { "Holland" },
is_located_in => { "EMEA", "Europe" };
"Singapore" is_located_in => { "APAC", "Asia" };
locations::
"London_1" is_located_in => { "London", "UK" };
"New_Jersey" is_located_in => { "USA" };
networks::
"192.23.45.0/24" comment => "Secure network, zone 0. Single octet for corporate offices",
is_connected_to => { "oslo-hub-123" };
CFEngine can analyze the promises you have made, index and cross reference them using the command:
# cf-promises -rNormally, the default policy in Nova or Constellation will perform this command each time the policy is changed.
cf-knowCFEngine's knowledge agent cf-know allows you to make promises
about knowledge and its inter-relationships. It is not specifically a
generic topic map language: rather it provides a powerful configuration
language for managing a knowledge base that can be compiled into a
topic map.
To build a topic map from a set of knowledge promises in knowledge.cf, you would write:
# cf-know -b -f ./knowledge.cf
The syntax of this file is hinted at below. The full ISO standard topic map model is too rich to be a useful tool for system knowledge management. However, this is where powerful configuration management can help to simplify the process: encoding a topic map is a complex problem in configuration, which is exactly what CFEngine is for. CFEngine's topic map promises have the following form:
bundle knowledge example
{
topics:
topic_type_context:: # canonical container
"Topic name" # short topic name
comment => "Use this for a longer description",
association => a("forward assoc to","Other topic","backward assoc");
"Other topic";
occurrences:
Topic_name:: # Topic
"http://www.example.org/document.xyz" # URI to instance
represents => { "Definition", "Tutorial"}; # sub-types
}
The association body templates look like this:
body association a(f,name,b) { forward_relationship => "$(f)"; backward_relationship => "$(b)"; associates => { $(name) }; }
|
Promise theory adds a clear structure to the topic map ontology, which is highly beneficial as experience shows that weak conceptual models lead to poor knowledge maps. |
We can model topic maps as promises within CFEngine; the question then remains as to how to use topic maps to model configurations so that CFEngine users can navigate the documented promises using a web browser and be able to see all of the relationships between otherwise isolated and fragmentary rules. This will form the basis of a semantic Configuration Management Database (sCMDB) for the CFEngine software. The key to making these ends meet is to see the configuration of the topic map as a number öf promises made in the abstract space of topics and the turning each promise into a meta-promise that models the configuration as a topic with attendant associations. Consider the following CFEngine promise.
bundle agent update { files: any:: ``/var/cfengine/inputs'' -> { ``policy_team'', ''dependent'' }, comment => ``Check policy updates from source'', perms => true, mode => 600, copy_from => true, copy_source => /policy/masterfiles, compare => digest, depth_search => true, depth => inf, ifelapsed => 1; }
This system configuration promise can be mapped by CFEngine into a
number of other promise proposals intended for the cf-know
agent. Suppressing some of the details, we have:
type_files:: "/var/cfengine/inputs" association => a("promise made in bundle","update","bundle contains promise"); "/var/cfengine/inputs" association => a("specifies body type","perms","is specified in"); "/var/cfengine/inputs" association => a("specifies body type","mode","is specified in"); "/var/cfengine/inputs" association => a("specifies body type","copy_from","is specified in"); # etc ... occurrences: _var_CFEngine_inputs:: "promise_output_common.html#promise__var_CFEngine_inputs_update_cf_13" represents => { "promise definition" };Note that in this mapping, the actual promise (viewed as a real world entity) is an occurrence of the topic `promise'; at the same time each promise could be discussed as a different topic allowing meta-modeling of the entity-relation model in the real-world data. Conversely the topics themselves become configuration items or `promisers' in the promise model. The effect is to create a navigable semantic web for traversing the policy; this documents the structure and intention of the policy using a small ontology of standard concepts and can be extended indefinitely by human domain experts.
|
You will find extensive help, examples and documentation as part of the commercial CFEngine support. Visit the website http://www.cfengine.com for more details. |
Need help getting started?
For a complete overview:
[1] Arrays can be scalars or lists of the RHS (rvalues). An array is really just a pattern in the names of the LHS (lvalues), not a separate type. [2] Here, CFEngine differs from the topic map standard in allowing contexts
to be overlapping sets, rather than mutually exclusive `types'.
CFEngine is guided by Promise Theory in this respect in order to enable
distributed cooperation and the development of a free and emergent ontology.
Table of Contents
Footnotes