Semantic Werks

“Improving change tolerance through Capabilities-based design: an empirical analysis“, by Ramya Ravichandar, James D. Arthur, Shawn A. Bohner and David P. Tegarden. J. Softw. Maint. Evol.: Res. Pract. 2008; 20:135–170

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes :).

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao’s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.

A software system is a human-created artifact. And yet, even though humans are in control at every stage in the process, these systems often exhibit unanticipated effects. Most attention is focused on undesirable effects, such as the Ariane 5 failure. Presumably there are also unanticipated and unexpected positive effects that we aren’t told of, for the obvious reason that things work as expected, and there is no reason to question why.

Since software is human-controlled, there should be no obvious reason to compare it to natural systems like gene networks or chemical reactions. However, as Stuart Kauffman notes in “At Home in The Universe”, humans are restricted to guessing about the future. As such, we cannot anticipate all potential scenarios. This makes human-created artifacts subject to the same randomness that many natural processes are. Consider the biological notion of ‘fitness landscape’. For a given fitness landscape there is a global maximum that describes the best adapted genotype. Clearly, we should strive to have all software reach this global maximum. But, as I have mentioned, we cannot control all the variables, since we can’t anticipate all occurences and changes in the environment. This means we can’t know ahead of time what the environment will be like when we build our software. If we knew, then we would clearly strive to build to this global maximum. Instead, we attempt to build to some pre-specified local maximum, that for us is best-adapted to our environment (see Jackson). However, even this is unlikely. For example, we may abandon certain features as deadline pressures mount. One way of looking at this is that we are changing the environment that we are targeting (abandoning requirements of the environment).

How might software mimic dynamical systems, like biological evolution? Evolution is essentially a process of striving towards these peaks in a fitness landscape. Evolving software should seek to improve its adaptation to a given set of environmental constraints (fitness pressures). The landscape shifts as new environmental constraints and conditions become known, so our software has to adapt itself (reproduce? variation? genetic drift? somehow adapt). Successful software will move to a new local maximum on the new fitness landscape. Constantly shifting environments might reflect an unstable ecology, in which maxima are impossible to find (rugged). Other landscapes might be extremely stable. For example, a long-running satellite might have very few environmental changes. We can use this landscape metaphor to model why different software seems to have different requirements.

I want to digress for a minute to discuss criticality. In small systems, we might not see any complexity effects. For example, code to post blog entries might seem easy to get mostly right (to hit a good local maximum). Why all the fuss? My contention is that at some threshold, which will change depending on what the environment is, software shifts from simple to complex. This notion is similar to laminar vs. turbulent flow in fluids. At some threshold, what was a simple-to-model system becomes nasty and impossible to predict. We need simulations and approximation algorithms to understand it. Well, I think the same is true of certain software systems. They may cross this threshold and have unpredictable effects - and this in a system in which we know all the inputs and all the outputs. Even with human control, in other words, we cannot truly control the outcomes.