By now, it's pretty much confirmed that, in the world of software development, Agile processes are here to stay. At their best these process offer shorter development times, higher quality, and a generally happier bunch of programmers. In fact, you cannot review industry want ads without seeing the word "Agile" sprinkled liberally all over the place. Yup, Agile is here to stay.
Why, then, do we not find all corporate IT departments converting to cube-less, team-centered Scrum factories at breakneck speed? For the record I have actually visited such a place and, I must say, it is pretty cool. But they had a unique set of circumstances (which I'll discuss in a bit) that made the global adoption of Agile more attainable.
But, to the original question: with all its highly touted success, why isn't conversion to Agile an easier sell?
One reason: MANAGEMENT!
Managers are often risk averse and cost conscious. Aversion to risk means they probably don't get warm and fuzzy over development methodologies that are light on hard delivery dates. Remember, for better or worse, most managers are accustomed to IT project plans that GUARANTEE delivery by a date certain. Just because those dates are almost never met doesn't mean managers blame the process—they blame IT who can't deliver on time! Try telling them that these same IT "pros" can improve things with some new methodology that doesn't require delivery dates but will make them happier really soon...
And their cost consciousness means the idea of tearing down perfectly good cubes and replacing them with chaotic wide open team areas where messy developers run the show... well, it gives them the willies. And, if you think they balk at that, try telling them they now need two programmers (for paired programming) where they used to have one. Somewhere a Lean management consultant is having a mild coronary at the mere thought.
Remember the company I mentioned before that successfully implemented Agile? What was it about them that made their efforts a success?
If you read between the lines you'll see all the above points indicate management involvement. In fact, without the dedication of the CIO, it's likely this conversion would not have worked. This same company had attempted Agile "projects" before without success. To really do this required a change in culture (ex. developers NOT getting pulled off projects to do production support) that couldn't be attained one project at a time. And that kind of change could not have happened without high level management support.
I know. This sounds like another anti-management diatribe for which we programmers are famous. But it's not. Let me be clear—management is not the problem! But they are the target audience. It is up to those of us who believe in Agile to make a compelling, coherent case FOR Agile.
And that case is this: Agile processes, when applied correctly to the right problems, can do what management wants most—reduce risk and cost.
If we make that case with plenty of documentary evidence, the solution will sell itself.