Semantic Werks

As part of my research, I try to keep up with the trends and tools in software development (it has nothing to do with procrastination, either).

I’m beginning to wonder if maybe I’m missing a large chunk of people in my reading. According to the blogs and sites I follow, agile is far superior to other methodologies, Ruby and Python are orders of magnitude more productive, REST architectures are preferred, and, importantly for me in particular, requirements are best understood as user stories, and not as lists of MUST and SHOULD items. Let’s call this the agile world.

But I feel like there’s this other world, the world I hear about when I read academic papers, or read IBM/Oracle/MSFT blogs, or hear about work at large companies. This is the world of J2EE, of COBOL, of UML models, of reams of paper documentation, of Rational Rose, of ESB, of SOA, of WS-*. This, for lack of a better term, is the waterfall world.

Is it just me, or is this waterfall world not getting the message that people like David Anderson and Martin Fowler are spreading? Is there a reason for this? Frequently argued is the point that these systems are orders of magnitude more complex, older, and larger, and therefore ‘agile’ methodologies just won’t work. And true, some of the agile proponents are tiresome ideologues who talk but don’t listen.

So while it is appealing to self-identify as the little boy pointing out the emperor’s lack of apparel, maybe I’m out of touch. Maybe there really is a need for complex modeling initiatives, for up-front requirements documentation. I’m not talking about safety-critical software either — that’s a whole different kettle of fish, and a straw-man for the purposes of this discussion. No, I’m specifically referring to large corporate IT departments, process control software, line automation, etc.

I guess I just don’t know how to experience the waterfall world without being involved in it first-hand. In the agile world, there is hype aplenty — hype sells books and seminars and certifications — but , at the risk of sounding repetitive, experience reports and anecdote just won’t cut it (for either side). It’s extraordinarily difficult in the software community to get at any objective source of information — except surveys, typically.

So who’s up for a PhD establishing the benefits of agile and its ecosystem over waterfall and its suite of tools? Can such a thing even be established?

This is a bit of a contrarian’s view of new software methodologies. First let’s acknowledge that to date, no methodology has reduced the essential complexity in software development, and there is still no silver bullet. That said, it seems obvious that software development has its flaws, although not as bad as Standish tried to insinuate. In other words, there does seem to be plenty of accidental complexity introduced with software development methodologies. It is increasingly clear that so-called ‘waterfall’ development methodologies are replete with accidental complexity (and in my view, accidental complexity is synonymous with waste).

There have been a number of improvements suggested. Two popular methodologies (paradigms, really) are Scrum and Kanban (where Kanban stands for pull-based processes). Both seem to be moving to widespread adoption in the industry (Scrum is ahead of Kanban in this regard, I think). And now the real problem starts. As Tom Grant mentions, this is when the bad implementations start. Someone attends a conference or reads a blog, thinks, ‘map the value stream! we can do that!’ and introduces it to his manager. He’s persuasive, and it’s Friday so the manager wants to go home, so a pilot is approved. The champion may even have some form of certification. The result is a half-assed implementation with no senior management buy-in. What is in reality a complex and holistic methodology, requiring paradigm-shifting levels of commitment from the company, ends up failing miserably. There’s push-back against the process — people start to ask where the high-level design is, a crisis arises pulling staff off the project, etc. This is not uncommon in any methodology — the original ‘waterfall’ paper didn’t actually say “lock in requirements first, then develop, then test, and never talk to the customer”

One of the main problems behind this is a lack of objective information on the methodology. Issue 1 is that while nearly every report you read about the process is positive (of the ‘we implemented 2 week Scrum sprints and reduced delivery time by 123%’ variety), they are often biased and don’t control for confounding factors. These aren’t meaningful case studies, but rather experience reports. They typically don’t have a central proposition defined in advance, they don’t explore rival explanations, they don’t generalize to a theory, and they fail to address threats to validity. I’d caution against reading too much into these reports.

For example, we can’t ignore the issue of expertise. For instance, Paul Graham argues that the success of his Lisp-based website, ViaWeb, shows that Lisp is as good as any other language for web development. However, we have no good examples of follow-on successes, and I suspect that what really happened was a) a smart LISP programmer releasing b) at a fortuitous moment. In the same way, the success of Toyota and TPS doesn’t prove that this system is better than any other. What it does show is that Toyota have an incredibly disciplined and focused organization. To my mind, the pillars of TPS — statements like ”go-see” and continuous improvement — are so abstract that they can basically be translated as ‘always do your best’. They don’t strike me as radically different than what any organization that is driven and motivated would come up with. So if Toyota took market share from GM, was it due to TPS or really just the case of a smaller, aggressive company taking over a sedentary and moribund established player? The gas crisis in the 70s certainly played a part, too.

The second issue behind the failure of new methodologies is financial. Consultants and experts with have a vested interest in adoption, not success. Scrum trainers optimize to generate new Scrum masters, not successful Scrum implementations. Certifications prove nothing beyond an ability to sit through a two-day session. This effect contributes to the problem that mushy middle adoption often ignores the core importance of a tool or methodology. There is a focus on the artifacts — the whiteboard display, the continuous flow mapping, the calculation of velocity — over the huge challenge of getting organizational backing for the process. Moving to a lean paradigm for software development is company-changing. It doesn’t seem to me like something that can be done halfway.

Context is important. This is something that thought-leaders usually mention in their writing, but is somehow lost on the wider audience. What worked in Microsoft will almost certainly not work exactly the same way in your organization. It might be a difference in product maturity, developer skill, management interest, etc.

Lesson: organizational maturity seems to be the chief determinant. It’s like arguing that what matters in childhood education is whether you use phonics or ‘look-say’ techniques to teach reading. This focus ignores the massive contributing factors of teacher skill and parental involvement. I’d say that the best predictor of success in adopting a ‘new’ methodology or  paradigm is the nature of the organization, and not the specifics of that methodology.

p.s. I never once mentioned ‘engineering’ in this post. :)

Having just read a book on Toyota Production System (TPS), and following various developments in lean software development, I was curious to know what Toyota itself was doing with respect to software. Software forms a large part of new vehicles (I’ve heard figures like 50% of new car development costs), so this seems like a good place to use notions like continuous improvement, visual management, waste reduction, etc.

I gather as part of a “Lean” tour of Japan, various Lean software thought-leaders (Mary Poppendieck, among others) had occasion to meet with the head of Toyota’s software division. From the description of the meeting, it sounded like Toyota were doing fairly standard BRUF development — analyze, plan, implement, test, deliver. Some people call this ‘waterfall’ but personally, I don’t think that term has ever been well-defined.  Certainly I doubt anyone has ever said, “Hey, let’s do waterfall on this project!”

So from the initial description, fairly standard development, but, being Toyota, there’s more to the story. Mary Poppendieck elaborated on her understanding, suggesting that in the process control domain, the three month cycle is quite fast (close to some Scrum sprint lengths of 1 month, anyway). There is also some interesting follow-up discussion here and here. The takeaway seems to be that ‘context is everything’.  The other takeaway might be “Toyota isn’t perfect”.

via @agilemanager and @alinahsu

Software research has always been a strange creature. Born in the 60s as computer pioneers realized the immense challenge software posed (where previously most efforts had centered on hardware engineering and design), software covers such a broad scope of domains and possibilities that categorizing it is like categorizing transportation conveyances. Are you building an ocean tanker? A cheap, self-propelled city vehicle? Something to destroy buildings?

That’s why most arguments about software tend to feature wagon-circling by members of particular domains. Are we engineers or craftspeople? The guy who builds websites for local veterinarians is appalled by the demands RUP places on his development process. He finds Java and Tomcat ridiculously over-engineered. At the same time, the multinational IT support person can’t fathom how anyone would develop without a fully elicited requirements model or class diagram. She looks at Ruby and sees a newfangled bastardization of functional programming and Perl.

These aren’t the only camps, either. It often seems to break down over project size, potential harm, and importance. NASA has very few defects in its code, but spends billions of dollars to get to that point. Not something 37 Signals is interested in doing.

This is why I find the term ’software engineering’ misleading. Sure, back in the 60s, 70s, 80s, venues like ICSE were largely about giant defence projects, things like Star Wars (see here for a great rant about how futile that effort was, though). Software could be engineered, right? Throw peons at the project, a bunch of money, and with a healthy management team the project would succeed.

Since the dawn of the open-source movement, I would say, and the rise of personal software in the late 80s, research on large-scale software was increasingly less useful to a large number of practitioners. This led to a number of non-academics proposing truly innovative and successful methodologies — Scrum, FDD, XP, Lean/Kanban — because, in my view, the researchers just weren’t able to see the change coming. Partly this is because as a professor, you need secure funding, and small firms cannot provide this. Inevitably you end up working on Oracle instead of MySQL, for example.

The exciting things is that doing research in software fascinates me precisely because of this diversity. But that research must be targeted appropriately. Don’t insist on the primacy of UML for every possible project — it just doesn’t make sense. Don’t see statements like “people over process” and assume that process is irrelevant.

I think in order to more fully embrace this awesome scope, to make research (more) relevant, we ought to stop pretending we all do engineering the way they had hoped — although not all of them — at the NATO conference in 1968. We need to more carefully identify the relevance of humans in the software loop. We need to accept the inevitability of change and inconsistency in software and organizations. We need to figure out what’s useful for the ten-person company, as well as the thousand person company. We ought to just call ourselves software scientists, and go from there.

Currently, it seems to me that practitioners of all stripes get their information thusly: first, via colleagues or anecdote; secondly, from websites like Slashdot or Digg; thirdly, from vendors and PR agencies; fourthly, from industrial research firms like Gartner or Forrester; and finally, lastly, from academic research papers. I would really like to move academic research up this list.

There’s a proposal out to put bike lanes on Bloor Street, a major east-west arterial in Toronto. My feeling: waste of time.

Yesterday I rode from downtown Toronto at College and Spadina to Bloor and Lansdowne. There are official, marked bike lanes all the way along College (less busy than Bloor) to about Dufferin. And they are totally useless. There must have been at least a dozen trucks, couriers, and cars parked in the lane, forcing the bicyclist to merge into traffic. The lanes are downright dangerous at right-turns, because cars just make the turn anyway. And cyclists seem to ignore them, as I passed two groups riding two abreast. Further point of information:  I was seriously injured while riding in a bike lane. So I’m not the biggest fan.

I actually felt safest on Lansdowne, which I gather has been criticized for its lack of ‘official’ bike lanes (it has a bike symbol but no solid white line). There, there are no cars parked in the north-bound lane, and the road is plenty wide enough for all but large trucks to share with cyclists.

I gather the cycling advocates like any idea that gives more visibility to cyclists, but this idea of a bike lane on Bloor would divert money from more useful ideas. Here’s one: completely close off streets to non-local automobile traffic (say, for example, King Street). Last time I checked there were about 5 good ways of getting across Toronto in a car: why can’t one be dedicated to transit and bikes?

I guess the only other idea I had was to actually enforce the Traffic Code, namely towing people who park in an official lane. Solid white means don’t cross, but this law, along with no-idling, no running red lights, etc., don’t seem to be enforced. Courier companies, I’m pretty certain, see traffic tickets as a cost of doing business (and likely can deduct them on their taxes anyway!).

Update: Apparently some advocates are getting annoyed too…

The new Railpath, looking north from the Dundas overpass

Today I biked from my place at Spadina and Richmond to Lansdowne and Dupont. My idea was to explore the new Railpath project, a bike path that should go from Dupont and Dundas W to Dundas and Lansdowne, and, hopefully, all the way to downtown (seems unlikely). Sadly, as the photo shows, this path is not yet complete.

This  left me on the newly remodeled Lansdowne, which is a great improvement — as Toronto streets go, which means the chance of death is just below 10%.

Anyway, eventually (when?) you should be able to cycle fearlessly along this path. Coming up from Lansdowne (from Queen), take a left on Dundas to the bike path, then veer right at the second road. This path will start just before the final bridge on Dundas.

I sure hope they manage to secure the rights to the rest of the corridor — it goes all the way downtown to Union Station. If you look at the cycling map, there is a big black hole of cycling-safe roads in the box between King and College, and Spadina to Roncesvalles. I don’t understand why we need four major east-west car corridors (King, Queen, College, Dundas) and no cycling routes. Whiskey Tango Foxtrot.