Semantic Werks

“Improving change tolerance through Capabilities-based design: an empirical analysis“, by Ramya Ravichandar, James D. Arthur, Shawn A. Bohner and David P. Tegarden. J. Softw. Maint. Evol.: Res. Pract. 2008; 20:135–170

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes :).

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao’s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.

@inproceedings{wu_using_2002,
address = {Paris, France},
title = {Using Graph Patterns to Extract Scenarios},
url = {http://swag.uwaterloo.ca/\~j25wu/papers/iwpc02-wu.pdf},
booktitle = {International Workshop on Program Comprehension},
author = {Jingwei Wu and Ahmed E. Hassan and Richard C. Holt},
month = jun,
year = {2002},
pages = {239-247}
}

Grok is a tool from Waterloo that can do algebraic manipulations of graph structures. In this paper, the graph structures are source code relationships like ‘calls’ or ‘uses’. They first extract ’scenarios’ from the software model, which seems similar to my desire to extract high-level goals from requirements specifications or documentation. Again, the challenge is a reconstruction one, and seems very similar to software histories. Since we don’t have access to the decisions and thought-processes at the time of design, we must reconstruct these (albeit imperfectly) from original documents. Although imperfect, it isn’t much different than a historian analyzing Napoleon’s decision to attack Russia in 1812.

Source code extraction has better fidelity. After all, most version control systems allow you to checkout code from a specific date, preserving perfectly the state of the software at that point in time. However, this can only ever reproduce the facts on the ground at that point in time (e.g., bugs, design, architecture) but not what is, in my opinion, the more interesting rationale insights (why this design?). Similarly, we know Napoleon ended the invasion with massive casualties — but have to settle for reasonable estimates as to why this disaster (or triumph) happened.

The target system here is Gnome’s Nautilus. The authors first generate facts about the source code, then manipulate this ‘factbase’ with Grok in order to produce something that can be used for architecture recovery. Using this knowledge, they then generate scenarios (e.g. interprocess communication via CORBA) and then validate with facts extracted. There seems to be a certain degree of ‘find the facts which fit our scenario’, and one has to wonder to what extent the extraction process was ‘massaged’ to produce the graph that supported the scenario. Perhaps a more reasonable approach would have been to use a portion of the code to generate the patterns, and then search the remainder for similar matches (supervised classification). I think this issue is a common one in software engineering science. The Grok pattern matching extension seems very similar to the RDF query language SPARQL.

[One problem] was identifying a useful set of scenarios, because substantial knowledge about the important tasks of the system were required. Our experience showed that such knowledge could be accumulated during architecture extraction. In our analysis, the trial-and-error method was applied to extract useful scenarios. We carried out many iterations of defining patterns before we got useful matches. (p. 8/9)

The ability to query a factbase is very powerful, especially as a precursor to generate visualizations. In general, with large knowledge bases, it is silly to merely generate a large graph visualization without first manipulating the data to support user-defined queries. This paper confirms the utility of a graph-matching approach to pattern extraction, but highlights the difficulty that remains, namely, gaining sufficient understanding of the system to be able to pose lucid queries.

@inproceedings{marco_lormans_monitoring_2007,
address = {Amsterdam},
title = {Monitoring Requirements Evolution using Views},
booktitle = {European Conference on Software Maintenance and Reengineering (CSMR'07)},
author = {Marco Lormans},
month = mar,
year = {2007},
pages = {349--352}
}

This is a paper that is quite closely related to my research, in fact, probably the closest I have yet found. Both Lormans and myself are interested in the topic of requirements management. Requirements management generally refers to keeping track of changing requirements, storing and updating existing requirements, and tracing from requirements to code/tests/implementation. Traceability is by far the most active research domain in requirements management. Traceability seems to have the most to offer practitioners, but I think one thing traceability fails to address is change in requirements themselves. I am working on a theory for why requirements change, and how that change manifests. Traceability is a technique for updating the Machine (software) when a requirement is changed.

My claim is that system maintenance is easier when maintainers can work in the domain language, rather than the Machine language. Consider a system that evolves as follows: a dialysis system needs to handle a new scenario where patients can take the unit home. It will be simpler for a systems analyst to describe the scenario in the domain or problem language (Patient, unit, blood, kidney, home) than the machine’s language (Class Patient, Class Monitor, Package org.test, etc.). Of course, object-oriented modeling is intended to remove some of that cognitive dissonance, but there will always be things the Machine needs to know that don’t exist in the domain. Thus, being able to describe one’s changes in the language of the domain is very important.

Lohrmans describes his system for doing this, the Requirements Engineering System (there has to be a more descriptive name!). I found the proposal (it was presented at a doctoral symposium) weighted towards generating traceability links and automatic generation of changes from work products like requirements analyses. Like my work, Lormans’s also focuses on post-implementation changes and system maintenance. He emphasizes requirements views for structuring concerns, including views such as coverage or lifecycle. I found the description of the work skimmed over the issue of maintaining various versions of requirements models. It isn’t apparent how RES will store requirements models that it generates, or how it can show changes to stakeholders. My opinion is that this problem will require some careful thinking on visualizations for change.

The work on requirements traceability with information retrieval techniques like Latent Semantic Indexing is interesting and valuable, particularly in large products, but given all IR techniques have incomplete recall, I have to wonder whether important non-functional requirements might be omitted. If we structure our requirements in a tree, such as goal models do, the roots of the tree are the more important goals — the high-level requirements. If one of these is missed, then isn’t the entire model very indeterminate? Furthermore, if the IR technique is relying on statistics like co-occurrence, I would imagine it is possible that certain high-level goals could be so obvious to system stakeholders that they are not that frequent in work products like requirements documents. For this reason I’m more fond of semi-automated techniques.

A big plus in Lormans’s work is its integration with several multi-year EU research projects. This has provided a lot of empirical data for him to leverage, and is something I need to achieve as well. The MOOSE project is one such project.