CFEngine, SysAdmin 3.0 and the Third Wave of IT Engineering

PART I: the sysadmin poverty trap

The Third Wave of human society is an age where knowledge and information drive prosperity. Parallel to the changes that pushed humanity through major industrialization in manufacturing are developments happening in IT management today, some thirty years behind the manufacturing industry. CFEngine and its users have been a primus motor for these changes – helping to transform old techniques into a knowledge-based approach to IT – and we are celebrating this new chapter with the announcement of CFEngines Nova and Constellation.

I love great writers, and Alvin Toffler has been one of those few writers on the human condition who consistently impresses me with both careful thinking and excellent exposition. According to Toffler’s prescient view of our human condition, Mankind’s development can be divided into three major phases:

• The 1st wave: the beginning of civilization through agriculture, or manually tending to the land.
• The 2nd wave: Industrialization, automating production and machine-assisted strength.
• The 3rd wave: Information Society, man and machine dividing labour optimally, and machine-assisted brains.

In our `cute’ modern day language, we would probably say these phases represent civilization 1.0, 2.0 and 3.0 respectively. Suffice it to say, these three phases represent sweeping changes in the way humans approach the economics of prosperity.

The lessons learnt from industrialization (or the 2nd Wave) were described in his book Future Shock (1969), and – to anyone paying attention – clear parallels can be seen in the journey towards a rational approach to Information Technology (IT).

Just as human society still envelops everything from pre-agricultural tribes, farming communities, and factory sweat-shops to information-based commerce, in different parts of the globe, so IT today straddles all three `waves’ of development from manual chaos to goal-oriented self-repair in different organizations.

Rehumanizing IT: Out of the IT poverty trap

*“The most powerful dehumanizing machine is not technology but the social machine, i.e. The formation of command structures to make humans emulate technology in order to build pyramids and skyscrapers…” * Lewis Mumford (1967)

The Industrial Revolution that happened in the West was the way out of the poverty trap of manual labour for industrial nations. Prior to its arrival, humans worked reactively to cope with crisis after crisis, sometimes in self-cancelling ways (to use Toffler’s language), and without a clear direction or goal. As the IT explosion began around the 1990s, the same thing happened in many organization: overwhelmed by crises, ad hoc system administrators fought to keep their heads above water, so busy that they never won enough time to escape from this condition. In IT, alarm-based early warning systems have been making humans jump-to-it for more than 10 years, based on SNMP and related technologies.

A feature of the second wave industrialization was the class division between rich land-owners and poor workers. This was really a division between infra-structure and its maintainers. The same division grew up in IT during what I am calling the 2nd Wave of IT management, in the division between the network and the computers (variously called `servers’ or `workstations’, etc). Network Managers (the land-owners) have been viewed as high ranking engineers, with certified qualifications, graduates from the Cisco University, etc. On the other hand, system administrators (or IT workers in more modern parlance) have been seen as low-status individuals merely performing factory monkey-work that could be learnt at a community college.

The emancipation of system administrators to the level of engineers has been a long process involving motivated individuals. As one of those individuals, I can report that there is still a long way to go – but today we are on the verge of a change thanks to the marriage between IT and commerce and its hype through concepts like the Cloud. It is these engineers who will user in the 3rd Wave.

As town planner Lewis Mumford remarked, industrialization is about making humans work like machines, in order to amplify its production of basic necessities. Industrialism was deeply de-humanizing, making humans work for the machinery, emulating it, to create a giant `pump’ for manufacturing. In the Third Wave, human freedoms become the driving force for technology: mobility and diversity become the keywords, and IT becomes an enabler of human freedom rather than suppressing it by forcing humans to work for it with clockwork factory-like procedures [2].

The Third Wave is really a transition from approaching certainty by over-simplifying through mass replication and artificial discipline, to tolerance and effective exploitation of a more organic or natural kind of growth through federation and contextualized knowledge.

The 1,2,3 of IT management

The reincarnation of the 1st Wave system administrator in the 2nd wave is as an IT infrastructure engineer, and in the 3rd wave he or she is a kind of knowledge engineer. Thus the pre-requisite tools for IT management, in this future, will be:

• A comprehension of our systems’ many behaviours.
• A comprehension of how close we are to our human goals.
• The ability to promise desired outcomes across multifarious contexts.

These are the pinnacles of a knowledge-based economy.

Footnotes:

1. At Oslo, we built a course on this idea, and colleague in Amsterdam did the same – actually renaming their course SNESystem and Network Engineering after a short spell out of the belief in this view of rigorous methods of engineering.
2. This was also the theme of Ayn Rand’s famous novel Atlas Shrugged.

PART II: from recipe to engineering

The Third Wave of human society is an age where knowledge and information drive prosperity. Similar changes are happening in IT management, where knowledge is becoming a premium commodity. This has led to a transformation of the role of the system administrator from administrator to system engineer and knowledge engineer. It is these engineers who will carry us into the 3rd Wave.

From System Administration 1.0 to 3.0 – it’s engineering

In 1999 I was asked to write a book about system administration suitable for university curricula. At the time system administration was considered a non-academic field - something more like carpentry or cooking. As an academic, I disagreed with that view and had created a one semester course at the University College based on my understanding of the state of the art.

The second edition of the bookPrinciples of Network and System Administration, still in use today, opened with the line: “Network and System Administration is a branch of engineering that concerns the operational management of human-computer systems”. The reviewer at the time wrote back, saying: you know, I don’t know, it’s surely much too early to call system administration a kind of engineering – but his reaction cemented in my mind the idea that it was simply about time that it was: all the pieces were there waiting to be assembled.

Independently, Tom Limoncelli and Christina Hogan wrote a complementary book The Practice of System and Network Administration which described the high level strategies of being an engineer in a disciplined datacentre. Together, these two books documented the 1st and 2nd Waves of IT Management, preparing the way for the 3rd. However, they left out the embryonic changes that were taking place behind the scenes – involving configuration management, model-based desired-state automation and self-healing.

Ultimately though, I belief these books helped to change the mind-set of 1st Wave system administrators and usher in the transition to a knowledge-based approach to governing human-computer systems.

In Analytical Network and System Administration I made the leap towards understanding the engineering of combined human-computer systems – this was the volume of work that led to Promise Theory. Many of these ideas have been are are used and evaluated in CFEngine 3, and have contributed to solving the major challenges that make CFEngine unique: federated and diversified scale and complexity.

2nd Wave: hierarchy, rigidity and obedience

The 2nd wave has brought many debatable and clunky efforts to imprint simple-minded replication onto IT management – turning us humans into parts in the machines we have created for ourselves.

There are now signs that we are moving on from these ideas. The fading affection for `standardization’ is an example. Instead of adapting technology to human diversity, humans have been asked to adapt to technology. Power that was once centralized into push-based systems that strong-arm (`attack’[1]) their surroundings, demanding obedience. These ideas are gradually waning.

As Toffler points out, even the industrial education was symptomatic of this obedience, requiring of the three R’s: reading, ‘riting and ‘rithmetic in an overt curriculum to allow individuals to play their part. The covert curriculum was one of obedience, punctuality and repetition: act like a good machine. System administrators still willfully subordinate themselves to monitoring systems and alarms.

Organization was, by default, made hierarchical, forcing items into mutually exclusive categories, subordinate to larger ones. Everything from family trees to biological species, to tables of contents, to computer file systems have been arranged in this `one size fits all model’. The 2nd wave approach to knowledge management is to fit any data to a hierarchy, because this is the organizational model that humans choose to govern.

From Steam Engine to ConFiguration Engine

We don’t have to think cyberpunk to see that traditional hierarchical machine-like systems from the 2nd Wave, even those such as the military, are evolving into a different kind of machine today, based more on intelligence and precision engineering than on simple-minded force and rank. If the military can do this, and maintain predictability, then surely others can afford a greater use of pre-planned intelligence too.

In an industrialized (machine) process, one builds essentially a pump, an amplifier, a steam engine to power through the task. To take these power systems to the 3rd wave, one needs to measure that force and bring surgical precision to the table.

Is there a contradiction then between this progression from industrialization and the massive proliferation and scale we clearly need in the IT industry today? Don’t we need such brute force still? This is an interesting question indeed. For the time being, scale seems inevitable, but the landscape is more diverse (see the role of the Cloud below). And who knows what the size and capacity of the next generation of computer system will have? Perhaps an entire Amazon Cloud datacentre might fit into a small room so that everyone can have their own, cheaply.

One answer here is that scale is not just about size, but rather how much effort and information are needed to describe and comprehend our systems – to be sure that we control them, or at least to be able ride on their backs. That cognitive cost does is not reduced by changes in hardware technology.

Roll-out rocket launches

How many sysadmins does it take to change a light-bulb? The answer could well be many! The 2nd Waves approach to IT change management has been to demolish an the existing system and rebuild a new one with the new feature (light bulb) in it. Tear down the building and make a new one!

Imperative, serialized recipes of industrial processes are simplistic and only work if they start from a blank slate – so that is what people did. In the 3rd Wave, we tend to retrofit and recycle existing resources, making only the necessary changes. It is a more surgical approach to precision engineering.

2nd Wave configuration management prior to CFEngine was all about firing `smart-bombs’ (aka `packages’) onto systems that would explode onto the system, levelling the old configuration and rebuilding again from scratch. That was a destructive approach to management with high impact and system interruption.

Worse than this, fear of change and the risks change carries lead to a defensive approach to change management, where sysadmins save up all changes to certain approved `change windows’ and then `roll out’ all changes in one go. This is basically rocket science: highly trained experts spend a long time building a contraption that can only be used once and only has one pre-programmed mission. It has a launch date and one chance to succeed. If it fails, you have to pick up the pieces and start again. This is high risk, and high cost. In the 2nd wave, engineers working with commercial forces were able to turn this vision of flight into a 747 – a reusable craft that could be adapted to different cargoes and destinations, that could be re-used and maintained with standardized procedures.

The mistake the IT industry still makes is to save up change and adaptation into huge, high-risk packages and then roll these changes out en masse, at industrial scale. The term `roll-back’ was usurped (inappropriately) from database transaction theory then to explain the back-pedalling that is attempted once mistakes are made. This approach is taken because the instruments of the 2nd Wave are rather blunt.

System administrators still build systems like `rockets’ that require a lot of individual attention and expertise, and get launched on a particular date, going only to one destination, unsophisticated and inflexible. Second Wave transportation invented the 747, trains and buses for some basic flexibility in mass transport. In the 3rd Wave, we are ferried around in small go-anywhere, do-anything vehicles and these are self-maintaining without human assistance.

Most companies today do IT change management like a rocket launch, instead of making small, specialized incremental changes very often to adapt to dynamical needs with agility. Companies like Google and Facebook straddle this divide between 2nd and 3rd wave, by combining industrial scale with agility, based on careful use of knowledge. It will take time for these methods to spread to the world at large.

DevOps and The Cloud

The 2nd wave is far from over. There are tools being written every day to pander to 2nd Wave nostalgia, tools that make better alarm systems, new SNMP face-lifts, or `improved shells’, new languages that perfect the details of defective models – things that should have been left to posterity to judge.

The Internet itself could be used in smarter ways. IPv6, for instance, is just a bigger pipe for second wave transport, but mobile 4G and service oriented adaptation are enablers of human freedoms – and this is where the knowledge-based approaches will penetrate.

The Cloud is another driving force that helps us towards a greater agility. Although curiously run as a 2nd Wave industrial operation, what Cloud Services provide is that magical infrastructure to support virtual agility. The Cloud does for computing what paging did for disks and what packets did for the network. It atomizes management to allow individuality and resource efficiency, by recycling.

Instead of treating systems as `all the same’, configuration tools like CFEngine also atomize processes so that they can be self-healing, and use knowledge and carefully modelled intentions to engineer systems precisely – a sharp instrument, not a blunt one.

The DevOps movement that has recently come to attention is a natural reaction to slow 1st Wave thinking or inflexible 2nd Wave thinking. Impatient developers have begun to take control of their own operational deployments, to see their changes applied quickly, short circuiting the overloaded queue of the 1st Wave administrator and the inflexible `everything the same’ view of the 2nd Waver.

What we want is not industrial configuration for the Cloud, but diversified precision engineering to support The Real Cloud – which is all the smart-phones and tablets roaming around in the world in people’s back pockets, bringing 3rd Wave human freedoms to the privileged generation.

Infrastructure engineer

System administration, IT operations, or whatever we call it today spans all three waves of IT development. The 2nd Wave system administrator was a machine operator, being fed by monitoring traces and reacting to alarms; the 3rd Waver is a knowledge-empowered infrastructure engineer, using intelligent planning and developing domain expertise. At CFEngine, we are privileged to work with these engineers, and to develop for the 3rd Wave.

Footnote

1. If we want to speak in provocative terms, the management approach taken by push-based control systems is not functionally different from an external attack. This should be food for thought.

PART III: the economy of de-standardization

The Third Wave of human society is an age where knowledge and information drive prosperity. Parallel to the changes that pushed humanity through major industrialization in manufacturing are developments happening in IT management today, some thirty years behind the manufacturing industry. The future of IT will have to break with past over-simplifications, in which Man was asked to work for and be part of the industrial Machine. Humans and technology will have to adapt, however uncomfortably, to support the basic human freedoms that we now crave.

Incomplete information - clinging on to certainty

If one looks at the philosophies that swept through the 20th and 21st centuries, there is a transition of world views, from those based on belief and certainty to those based on making the best of what we know. We see it in the acceptance of methods likeprobability and statistics for dealing with unknowns, and in the move from a Newtonian clockwork universe to one dominated by fluctuation, relativity and quantum mechanics[1]. It’s the same story with non-linear systems and `chaos theory’. Unpredictability has finally been accepted, and seeking good-enough but still approximate knowledge about the state of the world is the way one succeeds.

2nd Wavers fight against this transition, clinging on to the simplicity of a familiar and predictable world, in spite of the limitations it brings. Unfortunately, the fiction of complete predictability can only be maintained at very high cost, by dominating the environment by brute force, as we did with steam engines and heavy machinery during the early industrial revolution. Eventually this cost becomes too high to support – the real world is too complex to pattern with a single mold, and today’s more prosperous and nuanced humans are less likely to accept restriction and limitation than in the past.

In the 3rd Wave, system administrators, in their new clothing as infrastructure engineers, will eventually have to accept the same kind of transformation of world view. This change will sweep through everything we do, from the organization of systems, to the organization of workplaces, to the knowledge bases and information we use. The simplistic approach to categorization, by hierarchy and silo, will make way for more organic, emergent properties.

A 3rd Waver will not accept a frozen, crystalline, mechanized view of the world. Instead goals will be seen to change in real time and reality will fluctuate around these goals. The best we can really achieve will be an equilibrium with our environments – a slowly changing average behaviour with quasi-stability[2,3].

The end of the gold standard

Ironically, the rise of literacy that accompanied the 2nd wave has allowed people to escape from the very shackles of simplistic, predictable thinking it advocates. This is how the 3rd Wave comes about. By raising prosperity in the wake of two World Wars, people have escaped from a limited lifestyle, and have sought to explore individual freedoms rather than grand visions of togetherness or compliance. These freedoms have been championed by and have driven commerce.

Consider the changes that have taken place. Prior to the second world war, writers’ visions were all about big civilizations and grand plans for humanity, but afterwards they were all about freedom. People spent their money on scooters and fridges, beauty and vacations – people became tired to being machine parts. We see the same trends in IT with mobility, entertainment and communication. Just try and stop the iPhone from penetrating your organization! It is not difficult to see why freedom is the natural next step.

In the 1st Wave of humanity, people were isolated manual workers (working the land). They consumed what they produced because there was little infrastructure for scaling production or for sharing. People’s lives were built around survival. There was some trade – goods for goods. There was little need for security, as everyone needed to trust one another to survive. In IT, people managed their own systems and shared little; they had no need for security. This was cheap, but it was also a poverty trap.

In the 2nd Wave, mass production and transportation drove sharing through markets and commerce, requiring continuous growth to survive in a competitive bubble. Security became a pressing issue because competition for high value was rife. Consumers were passive, choosing only what was offered or available, because there seemed to be little alternative. People’s lives were based around systems and routines. The 2nd Wave invented gold-standard money as a universal trading currency – a simplistic way of measuring value in a singular vision. The 2nd Wave was all about simple standards to make it easy for the machinery of society to scale up. This is cheap, but de-humanizing. In IT, all the same phenomena have basic analogues.

In a 3rd Wave society, people rebel against the conformance of a programmed lifestyle. Economics becomes a curiosity shop of diversified on-demand rivalry – driven now by consumers, not producers. Consumers are active; they demand individuality and customization. They request tailor-made services that differentiate them from others. One-size-fits-all is no longer an option.

The gold standard is abandoned because, when consumers have the advantage, things are worth what people think they are worth, not what the town bully demands. In IT, the word standardization – once everywhere (even as gold standard disks for the STIGs, etc.) – is now mainly for the benefit of external regulators, government auditors, or multiple decentralized de facto police services that maintain predictability and balance. The challenge is to make this diversification of compliance cheap and scalable.

Money itself is no longer fixed in value against gold, and the currency of trade has many different faces: we no longer work only for cash, there are other dimensions of value to consider, such as reputation, goodwill, confidence, freedom. We have become gamblers – dealing with the incompleteness of information, instead of trying to suppress it. The Free/Open Source Software phenomenon is a good example of how the economy of trade works – social recognition and goodwill have replaced hard cash in this area.

Laws and norms

In the 3rd wave, the governance of IT has to transform to be more like those `wishful rules’ of the human realm, because it is closer to human motivations than its industrial forerunner – laws based more on trust, convention and voluntary cooperation, rather than programmed enforcement, and simplistic standardization. During the 2nd Wave, brute enforcement of behaviour has left an imprint on social norms, and these persist now mostly by convention. We have been trained to observe a `civilized’ behaviour, and continued enforcement is simplify too expensive without the memory of basic cooperation – the armies and invasions that brought about this compliance ultimately choke the economy with rising overhead. It will be interesting to see at what point diversity gives way to intolerable noise; this will doubtless be a highly subjective judgement – to 2nd Wavers, any deviation from the programmed pattern would be unacceptable.

In IT, the new norms are still forming, and lawlessness is still open and common, but we also see signs of the human and IT realms converging on traditions that we humans have already accepted. Our challenge at CFEngine lies in supporting this flexible reality, to bring 3rd Wave empowerment at low cost.

Our increasing personal freedom suggests that we will increasingly insist on making our own goals to a much greater degree – but governance is not going away: the recent financial collapse has shown us that independent forces of regulation are needed to prevent dog-eat-dog competition and open warfare from spiralling out of control. We might not need `armies’ in the traditional sense, but we will certainly need police – i.e. policy verification agents.

Compliance with standards and regulations will never disappear as there will always be a need to cooperate with what others expect of us; it’s just that the authorities behind these standards will themselves diversify. Instead of just governments, independent bodies, companies will be the new governing entities too. There will be a remodelled Compliance 3.0 too. Ultimately, it does not matter whether the template for compliance is a law such as Sarbanes-Oxley, or if it is dictated by a need for customer transparency, or if it is merely a gentleman’s agreement between companies. Demonstrating predictability in a noisy environment will be the name of the game in 3rd wave IT operations.

Security

How will we survive a transition to freedom and loss of certainty? Can we accept it? Already companies are bemoaning the arrival of iPhones and iPads and Android devices which cannot easily be secured to the level they are used to in 2nd Wave thinking. The situation is, in fact, beyond discussion: people will crave their freedom. We cannot stop it any more than King Canute could hold back the tide.

Today we find it harder to control boundaries in the same way, with so many overlapping channels for communication. Will security even be a feature of the Third Wave?

Of course it must be. Although security has often been dealt with like a standardized industrial process, `locking things down’; but that is merely the way it has been handled during the second wave decades. It is a futile attempt to suppress uncertainty by brute force. The role of building trust through social norms has to be embraced, rather than trying to eradicate it – IT will mimic what has happened in human society, because it is a human society.

Security is more subtle in the Third Wave as boundaries are less well defined. That means that security perimeters have to be pulled back, closer to individual resources, rather than relying on impenetrable outer walls. The autonomous agent model is the only model that supports this kind of security – and this immediately rules out push-based solutions.

The 3rd wave organization will have to deal increasingly with emergent structure. Instead of pre-designing simplistic silos and walls, there will be much more cross talk to deal with, and organizations will divide along dynamically changing lines, that bend and flex with the modern freedoms we crave.

The end of expertise?

The 3rd wave will not be a paradise, however, there is no promise of a rose garden. In the 3rd wave, literacy is no longer a necessity. Machines can handle that too and humans could easily become dependent on a technology that no one but an elite understands. But the 3rd wave technology can also help us to think. If we use it wisely it can also help to keep humans in a decisive but curious, learning role.

Qualification will come at a premium – and until the 3rd Wave dominates, there will be a huge need for low-paid human IT administrators to work for the technology, like the sweat-shops of the 3rd World. Today’s configuration management solutions a helping to spread 2nd wave industrial standardization to these old sweat-shops. And with new technologies like CFEngine 3 taking the first steps into knowledge-based approach to management, there are large savings to be made in terms of human labour and drudgery.

The irony of the the Third Wave is that, in spite of being based more on knowledge, standardized human competence will eventually become more sparse, because the industrial process-knowledge has been absorbed out of the people into the machinery.

Is this a cause for concern? Individuality will replace standardized canned answers with deeper individual understandings. That makes communication a challenge but offers a deeper range of viewpoints. We cannot say this is wrong, or undesirable as it is already happening in free society. It is just evolution at work.

System engineers will probably have to become more like pilots or doctors – forced to know how to make a manual override of an autopilot that takes care of most basic operations, both to preserve their dignity as individuals, and also to allow legal-beagles to pin liabilties to a human in case of mistakes. But we know that this kind of transformation is true of something in every generation, and we adapt to it.

Clearly there is a continued role for Open Standards and Open Source to play in the transformation to a rehumanized world – an open process builds consensus through social interaction. This is what the human pre-frontal cortex was probably evolved to do – so having passed through a time of playing `machines’ we can return to using our intelligence for its true purpose. Indeed, it has already enabled knowledge-based technologies to emerge (think of Facebook as a monitoring system, for instance).

The 3rd Wave is not the end of expertise; it is the exploration of a more nuanced expertise, based on social diversity.

The last days of the second wave

“The in-fighting among (second wave supporters) amounts to little more than a dispute over who will squeeze the most advantage from what remains of a declining industrial system.” A. Toffler, The Third Wave (1980)

With CFEngine as our vehicle for exploring knowledge-based approaches to IT infrastructure, we have put a lot of work into finding ways to support social diversity. Cheap, scalable made-to-measure infrastructure makes smart use of patterns to reduce the amount of raw information. Similarly, the focus on intention, thewhere and the why, not merely the what and the how. Who needs knowledge management? Every infrastructure engineer does because we rarely touch infrastructure. It disappears into the background, out of sight, out of mind,

It will take some time before the 3rd wave splashes onto the toes of even a majority of organizations in the West, let alone reach an understanding. We might never reach a stage of complete coverage, as the economics of the world themselves form something of a heat engine – and a simple pump only works when rich and poor live side by side. Whatever the currency of that richness and poverty IT, we will need to be increasingly savvy about IT resources to keep it running, to cope with the deluge of information and manage the growth of IT into every part of the world around us.

As in the human realm, the three waves of development are very much in existence together. The 2nd Wave still seems new to many companies and organizations. System administration, never mind system engineering has not come very far, except in the most advanced institutions.

The 1st wave (manual work) is here – and the 2nd (machine-assisted mass production) is still being promoted actively. The waves are far more entangled in IT than elsewhere, probably due to the accelerated pace of IT’s incursion into society. Only the most forward-looking of companies are at the stage of considering the kind of differentiated, goal-oriented management of their IT systems, worthy of a 3rd generation approach.

For now, CFEngine straddles the 2nd and 3rd Waves too, allowing industrial scalability at the same time as we advocate post-industrial customization and knowledge management wherever we go. We are providing a tool to help IT companies make the transition in the long term, from the old world to the new.

No one suggests that such a transition will be easy, as Machiavelli put it: `…the innovator has for enemies everyone who did well under the old regime, and only the luke-warm support of those who might do well under the new’[4]. But innovation stands still for no one. It is in our blood, even as we fear it. So watch out, here comes the CFEngine 3 family to hose away the cobwebs from industrial IT, and to gently cleanse the pores of 3rd Wave knowledge engineers.

MB, Oslo, 22 Nov. 2011