Semantic Werks

Reading Robert Yin’s book on case studies has really opened my eyes. To date the most contentious aspect has been his distinction between statistical generalization and analytic generalization. The former is what is done in surveys and experiments: using a sample to make statements about the population. The latter is what he argues is done during a case study. A case study is not a sample, but rather part of an argument.

To properly frame the debate, I think it is important to start with the question of epistemology. That is, what is it we are seeking to do when we do research? In my opinion, it is to motivate the rational beliefs we can hold about a given observable phenomenon in the universe. For example, do requirements evolve? If they evolve, what is being done about it? Etc., etc.

Once we have a list of questions we are interested in — why is this apple hitting my head while the moon is always up in the sky? — we can begin to try and answer them. This is where the scientific method chosen is important. In phenomena whose causes can be controlled for, an experimental approach is useful. It allows us to generalize our results to a wider population. But a lot of phenomena in the world are not easily controllable. In this case, we need a technique that can allow us to still draw some conclusions about the phenomenon of interest. Otherwise, we would just throw up our hands in disgust. Is agile software development better than waterfall? Choosing a few good (representative) cases might well let us make some conclusions.

I think one of the important things to remember is that while case studies seem easy to do — just pick a subject and write up a history of the project — they are in reality much more complex to set up than their experimental cousins. This duality is what has led many to discount their scientific — epistemological — value. Most of what is called ‘case study’ in the literature is really more akin to ‘proof-of-concept’.