I read a while back on Jeff Atwood’s blog about a paper written by Tom DeMarco. Tom was one of the forces behind the development of structured analysis. Many years after bringing structure and metrics to software development, Tom DeMarco, one of the big proponents of Software Engineering, has stated that he no longer believes in software engineering. You can boil his paper down to a advocacy of iterative, agile-type development.

I agree with his viewpoint. Traditional software development fails so frequently and so dramatically that it shows a fundamental disconnect between reality and practice. An organic approach is far more realistic.

When I discussed this with LiquidHub’s Chief Strategy Office, Ray Bordogna, he countered that in his experience one of the chief reasons behind so many development failures is the inability to capture and define requirements. His point was that it didn’t matter what approach you took if you didn’t know what you were building.

He makes an excellent point. But the requirement gathering process is wedded to the methodology. If you develop detailed design documents and expect to throw them over the wall to a pack of hungry developers, you will fail more often then you will succeed. If you build the design in tandem with the development group you will succeed more often then you will fail.

So many projects fail to capture requirements in a traditional approach that you have to wonder why that is. There are many smart people in the businesses we deal with, but the problem is endemic. So maybe capturing exhaustive requirements for a non-trivial problem domain isn’t very realistic. If it’s not, then the only reasonable option is to build up a solution from known states.

Mike Hannon, a Senior Associate at LiquidHub, told me that he doesn’t think requirements should be agile. That you should know what you’re building and not have it shift underneath you. He’s right too. But he’s right in the specific sense, not in the overall sense. Requirements shouldn’t change with the winds while you’re building software. But they should be fluid while you’re not.

In the end it shows how much further this industry still has to travel. We’re still talking about the fundamentals of software development because we’re struggling to control a process that is virtual and complex. So for now, it’s a science, it’s an art but it’s not engineering.