Semantic Werks

I’ve tried to resist being swayed by the ’softer’ side of software science: business practices, management science, process improvement, etc., all of which I feel are challenging to evaluate scientifically.

However, a recent paper,  A Replicated Survey of IT Software Project Failures, suggests that if we want software projects to succeed in IT, than these aspects are perhaps the best place to focus our attention.The paper is a refreshing improvement on the Standish reports.

Of the top five reasons a project was cancelled, three were management-related: Senior management not sufficiently involved (33%), too many requirements and scope changes (33%), and lack of necessary management skills (28%). The other two were project over budget (which presumably has many possible causes), and a lack of necessary technical skills (22%).

As software engineering researchers, I would argue that a lot of our work is focused on the last reason (technical skills) or less important causes (e.g. technology problems). There is a large research community dedicated to requirements research, but arguably this community is seen as less important (compare, for example, papers in ICSE 2008: of  103 papers, only 6 were requirements related).

These results also suggest what I’ve suspected: the choice of technology (C#, Java, SQLServer, AJAX, etc) is less important than getting a good team together, with properly scoped requirements and a sound leadership vision. As Karl Pilkington would say, the rest is just ‘pfaffing about’.

Thanks to Jorge and Lorin Hochstein for the pointer.

This post initially was going to be commenting on a recent report on requirements and business analysis. However, along the way I came to read up more on the ‘CHAOS’ reports, and a fascinating discussion about their methodology (and business model), which I think ties in directly with the latest report.

The new report is from IAG, a company that sells requirements consulting services. They released a technical whitepaper on requirements use in business. Titled “The Impact of Business Requirements on the Success of Technology Projects”, the report concludes that 68% of companies will have their projects fail due to bad requirements practices.

This is a company with a vested interest in these results (it’s akin to Merck reporting on a new painkiller). As with most of these reports, such as the Standish CHAOS report, there is no methodology writeup, the participants are unidentified, and there is no assessment of validity and reliability.

They mention a survey of over 100 enterprises, with business application projects costing(?) over $250k. The survey instrument is not available publicly. As a result, the report is replete with statistics and numbers of unknown origin. For example, “over 70% of companies in the upper third of requirements discovery capability reported having a successful project”. What is the upper third of ‘requirements discovery capability’? How is membership assessed? What is a successful project? The rest of the report, it seems to me, can be summed up thusly: “companies that use our Requirements Discovery Process™deliver better projects on-time and under-budget”. Hardly surprising.

See also:

• A report on the reliability of the oft-quoted Standish survey.
• An attempt to extract methodology from Standish and their rebuttal to methodological questioning (which seems to amount to “Tough”).
• A well-known commentator on the usefulness of IT Consultants.