A good read from Hakan Erdogmus in IEEE Software this month. He tries to draw the distinction between those who conduct empirical research on software engineering versus those who prefer to adopt the Shoeless Joe philosophy: “Build it, and they will come”.
I like the distinction, although I wasn’t clear whether he was lumping researchers who prefer to code in with industry types who are actively experimenting on the job, so to speak. At a recent talk, someone asked the presenter a question about his experimental setup, how hard it was going to be to ‘prove’ A was better than B, etc. This about what was clearly a case study investigation! So part of the challenge seems to be that empirical software engineering is a really young discipline.
I think a common belief in ‘academe’ is one Hakan lists as a common objection to empirical research:
We don’t need empirical evidence to know whether something works. If it’s widely used [or cited --ed], it works. If nobody uses it, it doesn’t work.
Uh-huh. Right. Here are two counter-examples to this ‘argument by groupthink’ approach. One is Thalidomide. Hugely popular, widely used, and utterly disastrous. Maybe some more empirical research ought to have been done on it. So popularity isn’t (sufficient) evidence for effectiveness. While Thalidomide is an extreme example, the software industry is particularly prone to ’solutions’ hyped as extremely effective. Where is the software version of the FDA to protect us?
Second: Let me, and erstwhile (current?) presidential candidate Ralph Nader relate a little story. Once there was this rather profitable industry group that made powerful, gasoline-fueled motorized buggies known as cars. Since your average motorized buggy could go 120 mph easily, if something were to go wrong — and they often did — the occupant could end up looking like hamburger. At the same time, some scientists had come up with a device that would greatly reduce the hamburger effect by restraining occupants. It was cheap, it saved lives, and it was completely ignored until manufacturers were hauled before courts and legislatures. What is the software version of the seatbelt?
Hakan rightly criticizes empiricists who are too insistent on verifiable and repeatable data before accepting new techniques. In my view, this is most commonly manifested in the rejection by academia of any idea arising in industry. There seems to be a strong parallel here to the dichotomy between science and engineering knowledge I discussed earlier.