Semantic Werks

Reading Robert Yin’s book on case studies has really opened my eyes. To date the most contentious aspect has been his distinction between statistical generalization and analytic generalization. The former is what is done in surveys and experiments: using a sample to make statements about the population. The latter is what he argues is done during a case study. A case study is not a sample, but rather part of an argument.

To properly frame the debate, I think it is important to start with the question of epistemology. That is, what is it we are seeking to do when we do research? In my opinion, it is to motivate the rational beliefs we can hold about a given observable phenomenon in the universe. For example, do requirements evolve? If they evolve, what is being done about it? Etc., etc.

Once we have a list of questions we are interested in — why is this apple hitting my head while the moon is always up in the sky? — we can begin to try and answer them. This is where the scientific method chosen is important. In phenomena whose causes can be controlled for, an experimental approach is useful. It allows us to generalize our results to a wider population. But a lot of phenomena in the world are not easily controllable. In this case, we need a technique that can allow us to still draw some conclusions about the phenomenon of interest. Otherwise, we would just throw up our hands in disgust. Is agile software development better than waterfall? Choosing a few good (representative) cases might well let us make some conclusions.

I think one of the important things to remember is that while case studies seem easy to do — just pick a subject and write up a history of the project — they are in reality much more complex to set up than their experimental cousins. This duality is what has led many to discount their scientific — epistemological — value. Most of what is called ‘case study’ in the literature is really more akin to ‘proof-of-concept’.

This post initially was going to be commenting on a recent report on requirements and business analysis. However, along the way I came to read up more on the ‘CHAOS’ reports, and a fascinating discussion about their methodology (and business model), which I think ties in directly with the latest report.

The new report is from IAG, a company that sells requirements consulting services. They released a technical whitepaper on requirements use in business. Titled “The Impact of Business Requirements on the Success of Technology Projects”, the report concludes that 68% of companies will have their projects fail due to bad requirements practices.

This is a company with a vested interest in these results (it’s akin to Merck reporting on a new painkiller). As with most of these reports, such as the Standish CHAOS report, there is no methodology writeup, the participants are unidentified, and there is no assessment of validity and reliability.

They mention a survey of over 100 enterprises, with business application projects costing(?) over $250k. The survey instrument is not available publicly. As a result, the report is replete with statistics and numbers of unknown origin. For example, “over 70% of companies in the upper third of requirements discovery capability reported having a successful project”. What is the upper third of ‘requirements discovery capability’? How is membership assessed? What is a successful project? The rest of the report, it seems to me, can be summed up thusly: “companies that use our Requirements Discovery Process™deliver better projects on-time and under-budget”. Hardly surprising.

See also:

• A report on the reliability of the oft-quoted Standish survey.
• An attempt to extract methodology from Standish and their rebuttal to methodological questioning (which seems to amount to “Tough”).
• A well-known commentator on the usefulness of IT Consultants.

In ObservedRequirement, Martin Fowler makes the following comment: “the word ‘requirement’ is inherently waterfallish” (in relation to a quote about BRUF). A lot of these claims are unsupported by rigorous empirical study: either they refer to closed-data, proprietary reports (Standish), or base their experiences on anecdote. Does agile scale to projects that have more than 10 people involved and exist for several years? Could the Air Force be agile? We don’t have very good answers to these questions, yet. Nonetheless, the ‘agile’ movement has a lot of experienced, seasoned developers and consultants behind it, and it would be foolish to assume it was irrelevant because the data wasn’t there.

Fowler’s solution to adaptive software is to study what people are doing, and update the system accordingly. There are two problems with this. One, it assumes the software you build has people as direct users. Plenty of software is only indirectly used by people; for example, software to connect wireless modems, software to run microwaves. In general, the software Fowler seems to be talking about is one niche in the entire ecology: admittedly a big and sexy one. The second problem is that it falls into the task-artifact cycle [1], that is, it will only reveal those problems made manifest by that artifact. Logs will never be able to show what problems people had that were caused by missing features, or those things which they wished they could do.

If we want to build evolving software, this approach would be useful to tell us what bottlenecks there were, what paths people took through our software, and so on. But presumably we would have to build this agility into the tool to begin with: the monitoring framework, the reporting functionality, the self-* systems autonomous computing talks about. In this case, aren’t we now doing stable software engineering? It may be cumbersome to elicit all the requirements at once, and likely not even possible, but perhaps what people like Suzanne Robertson are arguing for is to try and get as many as possible early on. If someone tells us to do something, do we not want a sense of the purpose, a teleological explanation? The knowledge to help motivate our efforts, that even if we start by digging holes in an empty field, we may be building the Taj Mahal?

[1] See Vicente, Cognitive Work Analysis, p. 103-104. The term is often used positively, in the sense that good iterative design can accommodate the changing requirements involved. However, Vicente makes the case for escaping the tyranny imposed by the initial design: “Prototyping and usability testing could iterate an existing design, but couldn’t suggest wholly new directions (Beyer and Holtzblatt).