Monitoring is the sampling of system variables at regular intervals in order to present an overview of actual changes taking place over time. Monitoring data are often presented as extensive views of moving-line time series. Monitoring has the ability to detect anomalous behaviour by comparing past and present.
The term reporting is usually taken to mean the creation of short summaries of specific system properties suitable for management. System reports describe both promises about the system, such as compliance, discovered changes and faults.
The challenge of both these activities is to compare intended or promised, behaviour with the actual observed behaviour of the system.
The traditional view of IT operations is that configuration, monitoring and reporting are three different things that should not be joined. Traditionally, all three have been independent centralized processes. This view has emerged historically, but it has a major problem. Humans are needed to glue these parts back together.
Monitoring as an independent activity is inherently non-scalable. When numbers of hosts grow beyond a few thousands, centralized monitoring schemes fail to manage the information. Tying configuration (and therefore repair) to monitoring at the host level is essential for the effective management of large and distributed data facilities. CFEngine foresaw this need in 1998, with its Computer Immunology initiative, and continues to develop this strategy.
CFEngine's approach is to focus on scalability. The commercial editions of CFEngine provide what meaningful information they can in a manner that can be scaled to tens of thousands of machines.
| If you have regular reporting needs, we recommend using our commercially supported version of CFEngine (CFEngine Nova or above), as you will save considerable time and resources in programming, and you will have access to the latest developments through the software subscription. |
No promises made in CFEngine imply automatic aggregation of data to a central location. In commercial CFEngine versions, e.g. CFEngine Nova, an optimized aggregation of standardized reports is provided, but the ultimate decision to aggregate must be yours.
Monitoring and reporting capabilities in CFEngine depend on the software version include:
cf-report are also available for Nova users to browse network-wide data.
The following list of reports are only available in full in commercial editions of CFEngine. Some sample reports are provided in the Community Edition.
CFEngine's default behaviour is to report to the console (known as standard output). It's default behaviour is to report nothing except errors that are judged to be of a critical nature.
By using CFEngine with the inform flag:
# cf-agent -I # cf-agent --informyou can alter the default to report on action items (actual changes) and warnings.
By using CFEngine with the verbose flag:
# cf-agent -v # cf-agent --verboseyou can alter the default to report all of its thought-processes. You should not interpret a message that only appears in CFEngine's verbose mode as an actual error, only as information that might be relevant to decisions being made by the agent.
CFEngine allows you to use
reports promises to
make reports of your own. A simple example of this is shown below.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
reports:
cfengine_3::
"$(sys.date),This is a report"
report_to_file => "/tmp/test_log";
}
We can apply this idea to make more useful custom reports. In this example, the agent tests for certain software package and creates a simple HTML file of existing software.
body common control
{
bundlesequence => { "test" };
}
#
bundle agent test
{
vars:
"software" slist => { "gpg", "zip", "rsync" };
classes:
"no_report" expression => fileexists("/tmp/report.html");
"have_$(software)" expression => fileexists("/usr/bin/$(software)");
reports:
no_report::
"
<html>
Name of this host is: $(sys.host)<br>
Type of this host is: $(sys.os)<br>
"
report_to_file => "/tmp/report.html";
#
"
Host has software $(software)<br>
"
ifvarclass => "have_$(software)",
report_to_file => "/tmp/report.html";
#
"
</html>
"
report_to_file => "/tmp/report.html";
}
The outcome of this promise is a file called /tmp/report.html containing output like this:
<html>
Name of this host is: atlas<br>
Type of this host is: linux<br>
Host has software gpg<br>
Host has software zip<br>
Host has software rsync<br>
</html>
The mechanism shown above, can clearly be used to create a wide variety of report formats, but it requires a lot of coding and maintenance by the user.
CFEngine Nova simplifies this kind of report generation by enabling
and updating many out-of-the-box reports directly from the
cf-report agent.
|
CFEngine generates information internally that you might want to use
in reports. For example, the agent cf-agent
interfaces with the local light-weight monitoring
agent cf-monitord so that system state can be reported simply:
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
showstate => { "otherprocs", "rootprocs" };
}
A corollary to this is that you can get CFEngine to report system anomalies.
reports:
rootprocs_high_dev2::
"RootProc anomaly high 2 dev on $(mon.host) at approx $(mon.env_time)
measured value $(mon.value_rootprocs)
average $(mon.average_rootprocs) pm $(mon.stddev_rootprocs)"
showstate => { "rootprocs" };
entropy_www_in_high&anomaly_hosts.www_in_high_anomaly::
"High entropy incoming www anomaly on $(mon.host) at $(mon.env_time)
measured value $(mon.value_www_in)
average $(mon.average_www_in) pm $(mon.stddev_www_in)"
showstate => { "incoming.www" };
This produces standard output of the form:
R: State of otherprocs peaked at Tue Dec 1 12:12:21 2009 R: The peak measured state was q = 98: R: Frequency: [kjournald] |** (2/98) R: Frequency: [pdflush] |** (2/98) R: Frequency: /var/cfengine/bin/cf-execd|** (2/98) R: Frequency: COMMAND |* (1/98) R: Frequency: init [5] |* (1/98) R: Frequency: [kthreadd] |* (1/98) R: Frequency: [migration/0] |* (1/98) R: Frequency: [ksoftirqd/0] |* (1/98) R: Frequency: [events/0] |* (1/98) R: Frequency: [khelper] |* (1/98) R: Frequency: [kintegrityd/0] |* (1/98) |
Finally, you can quote lines from files in your data for convenience.
body common control
{
bundlesequence => { "report" };
}
###########################################################
bundle agent report
{
reports:
linux::
"/etc/passwd except $(const.n)"
printfile => pr("/etc/passwd","5");
}
######################################################################
body printfile pr(file,lines)
{
file_to_print => "$(file)";
number_of_lines => "$(lines)";
}
This produces output of the form
R: /etc/passwd except R: at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash R: avahi:x:103:105:User for Avahi:/var/run/avahi-daemon:/bin/false R: beagleindex:x:104:106:User for Beagle indexing:/var/cache/beagle:/bin/bash R: bin:x:1:1:bin:/bin:/bin/bash R: daemon:x:2:2:Daemon:/sbin:/bin/bash |
Logs can be attached to any promise. In this example, an executed shell command logs a message to the standard output. CFEngine recognizes the
stdout
filename for Standard Output, in the Unix/C standard manner.
bundle agent test
{
commands:
"/tmp/myjob",
action => logme("executor");
}
############################################
body action logme(x)
{
log_repaired => "stdout";
logstring => " -> Started the $(x) (success)";
}
In this next example, a file creation promise logs different outcomes (success or failure) to different log files.
body common control
{
bundlesequence => { "test" };
}
bundle agent test
{
vars:
"software" slist => { "/root/xyz", "/tmp/xyz" };
files:
"$(software)"
create => "true",
action => logme("$(software)");
}
#
body action logme(x)
{
log_kept => "/tmp/private_keptlog.log";
log_failed => "/tmp/private_faillog.log";
log_repaired => "/tmp/private_replog.log";
log_string => "$(sys.date) $(x) promise status";
}
This generates three different logs with outputs in of the form:
atlas$ more /tmp/private_keptlog.log Sun Dec 6 11:58:16 2009 /tmp/xyz promise status Sun Dec 6 11:58:43 2009 /tmp/xyz promise status |
CFEngine interfaces with the system logging tools in different ways. Syslog is the default log for Unix-like systems, while the event logger is the default on Windows. You may choose to copy a fixed level of CFEngine's standard screen messaging to the system logger on a per-promise basis.
body common control
{
bundlesequence => { "one" };
}
bundle agent one
{
files:
"/tmp/xyz"
create => "true",
action => log;
}
body action log
{
log_level => "inform";
}
Total auditing of a system is a surprisingly difficult thing to do, and it is extremely resource intensive. The followers of an audit trail are often paranoid by nature and are seldom satisfied with the level of detail they find. However, the times we really need an audit are rare, but the cost is ever present. The price of certainty is high.
| Spend a moment considering this: if you want to describe every change of state that happens on a computer, then you need to remember old state and compare it to new state. Then you have to record the differences. So you need more than the entire size of your computer's normal resources to do this. Your storage efficiency will always be less than 50% and your processing efficiency will be less than 50% on every audited item. Is this worth the effort? Perhaps your resources would be better spent keeping targeted backups and simply rebuilding contaminated systems. |
Switch on auditing like this:
body agent control
{
auditing => "true";
}
If you decide to go for full auditing, CFEngine will not collect and centralize the reports as they will be too large for this to be a scalable operation. Still, you can view them in a web browser on the local host, or copy them manually to a suitable location.
Doing a change detection scan is a convergent process, but it can still detect changes and present the data in a compressed format that is often more convenient than auditing. The result is less precise, but there is a trade-off between precision and cost.
To make a change tripwire, you use a files promise, something like this:
body common control
{
bundlesequence => { "testbundle" };
}
#
bundle agent testbundle
{
files:
"/home/mark/tmp" -> "me"
changes => scan_files,
depth_search => recurse("inf");
}
# library code ...
body changes scan_files
{
report_changes => "all";
update_hashes => "true";
}
body depth_search recurse(d)
{
depth => "$(d)";
}
In CFEngine Nova, reports of the following form are generated when these promises are kept by the agent:
Change detected File change Sat Dec 5 18:27:44 2009 group for /tmp/testfile changed 100 -> 0 Sat Dec 5 18:27:44 2009 /tmp/testfile Sat Dec 5 18:20:45 2009 /tmp/testfile |
These reports are generated automatically in CFEngine Nova, and are integrated into the web browsable knowledge map. Community edition users have to extract the data and create these themselves.
In commercial versions of CFEngine, you can extract data from the system in more sophisticated ways from files or pipes, using Perl Compatible Regular Expressions to match text. The
cf-monitord
agent is responsible for processing measurement promises.
In this example, we count lines matching a pattern in a file. You might want to scan a log for instances of a particular message and trace this number over time.
bundle monitor watch
{
measurements:
"/tmp/file"
handle => "line_counter",
stream_type => "file",
data_type => "counter",
match_value => scanlines("MYLINE.*"),
history_type => "log";
}
#
body match_value scanlines(x)
{
select_line_matching => "^$(x)$";
}
See the CFEngine Nova documentation for more possibilities of measurement promises.
In the commercial editions of CFEngine much more extensive and searchable reporting is available.
The preferred approach to querying information on a hub is to use the web interface in the Mission Portal. This gives the greatest flexibility in both search and presentation of data. Given the extensiveness of the Mission Portal user interface, the details are covered in a separate document.
Users with login access to the hub can also use the command line tool
cf-report to extract a limited view of the data.
Currently supported reports include:
compliancedead-clientsfile_changesfile_diffslast-seenpromisessetuidsoftwaresummaryvarsSome special command line options are supported in the commercial versions.
cf-report, and must always be specified.
If only a host-key is specified, CFEngine returns with the last known location and identity of the host. (Note that, in the following examples, the SHA keys are reduced for readability).
host# cf-report -q --hostkey SHA=bd6dfcc2... -> Hostname: hub.test.cfengine.com -> Recent IP Addresses: 10.0.0.29To dump all values from all hosts:
cf-report -q --show promises cf-report --query-hub --show promisesYou can select a single host for a particular report:
cf-report -q --hostkey SHA=c40fb732c6e5... --show varsOr you can select a CFEngine class of hosts that will be selected to report
cf-report -q --show summary --class-regex linux cf-report -q --show summary --class-regex SuSE cf-report -q --show summary --class-regex NewYorkHere are some examples using filters to 'grep' out certain items:
cf-report -q --hostkey SHA=c40fb732c6... --show vars --filter date cf-report -q --filter "mail.*" --hostkey SHA=bd6dfccb1a... --show setuid cf-report -q --show promises -p knowledge_files_db_stamp
Table of Contents