Semantic Werks

“Improving change tolerance through Capabilities-based design: an empirical analysis“, by Ramya Ravichandar, James D. Arthur, Shawn A. Bohner and David P. Tegarden. J. Softw. Maint. Evol.: Res. Pract. 2008; 20:135–170

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes :).

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao’s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.

James D. Herbsleb and Eiji Kuwana. Preserving knowledge in design projects: what designers need to know. CHI ‘93: Conference on Human factors in computing systems, 7–14 , Amsterdam, April 1993.

I hadn’t read this paper until it was recommended to me by an advisor, but it is now one of my favourites. It describes an experiment the authors ran on three industrial software projects. They had access to the requirements and design phase meetings from these companies (NTT, Andersen, and MCC), and coded the discussions based on three variables: the target of the question (subject of the question, categorized as evolve, task assignment, interface, realization, and identity), the attribute of the question (who, what, how, why, when), and lifecycle stage (req, design, impl, maintenance).

The results were somewhat surprising. While most questions referred mainly to requirements or design stages (not surprising, since these were planning meetings), few questions sought the intentions behind a decision. Many more questions were oriented to ‘what’ was being changed/affected (~60%), and ‘how’ that was to happen/happening (30%).

Practitioners might not find this surprising; after all, pragmatically those two questions are most relevant to “getting ‘er done”. However, at the time (1993) there was a lot of work on design-rationale tools, which specifically targeted the ‘why’ questions. This is also a central argument behind early-requirements analysis: that understanding intentionality will improve how well a tool performs for the users. Now, I think an important distinction is between answering ‘why’ a particular implementation is necessary, or why the change was made, versus a better understanding of user goals. User goals describe ‘what’ the tool should satisfy, while design-rationale is answering ‘why’ a certain change is being made.

The authors present four possibilities for ‘why’ questions having such low representation:

1. the information is perceived to be unimportant or avoided by people conscious they are not domain experts;
2. these questions are important, but these meetings are not the venue for addressing them, or the context is clear to the participants;
3.  ’what’ and ‘how’ questions provide enough information to elicit or understand the ‘why’ questions;
4. it is hard to answer these questions with current tools;
5. (least likely) while low in frequency, these questions are greater in importance: this suggestion seems unsupported by the data; e.g., the minutes didn’t reflect that such questions are more important.

Another critique that can be made, in a similar vein, is regarding the outcomes of these projects-to-be. I would imagine that studying the design meetings of ultimately failed projects doesn’t tell us much about what ought to be done in these meetings. Perhaps they focused to such a great extent on the implementation details that they omitted other important considerations.  However, the fact the paper has used three fairly distinct datasets strengthens the results considerably.

The paper concludes with an observation that usage scenarios are very important according to the data; this supports the agile focus on use cases and user stories.

All in all an excellent paper, grounded in empirical and externalizable data. The one regret I have is that (to my knowledge) the data is not publicly available. It seems like a rich source of data for a wide variety of questions. While I understand the privacy and IP problems involved, one would think that there is either a way to anonymize it, or that 10 years later, it would be of very little relevance.