Guest Post: Beyond Agile

And now for something completely different. This week, we offer a guest post from my Ovum colleague and agile methodology expert Michael Azoff.

Software development is more art than science: more about sociology than computer science — Agile has demonstrated that. The dream of computer scientists back in the 1970s, the era of the birth of computing, was that all you needed was a perfect specification and that programmers simply had to implement that spec. And what was implied? Maybe one day, you could automate that step and remove the need for a human programmer. Of course that dream didn’t happen and could never be fulfilled. The reasons are twofold: change and people.

* Change: because you can never nail down a spec perfectly upfront for most projects. Change is introduced during the lifetime of the project, so even if you had that perfect spec it can easily go stale.
* People: because for anything but the smallest projects, you need a team or multiple teams. And when people interact, there is scope for miscommunication and misunderstandings.

It is not a joke when project leaders looking back on large scale project failures say that rather than the hundred developers that were used, if they could rewind history and try again, they would pick the ten ablest developers and get the job completed and in short time.

Fast forward to today: Agile methodology has reached beyond the innovators and visionaries and has arguably gone mainstream. In practice that means various contortions and customizations of Agile methodologies exist, entwined with other processes and methodologies found in organizations, including neo-waterfall.

Neo-waterfall is an interesting case. I use that term because I do not believe developers ever did strict waterfall — if they did, the job would never get accomplished. So there was even a hint of agile in classic waterfall. Developers generally do what is necessary to get the job done and present the results to management in whatever form management expects it. Some form of iteration is essential, call it rework or doing it twice or whatever, because most software requirements are unique and getting it implemented perfectly right first time is difficult.

So now we have a situation where Agile adoption has reached the masses and organizations are ready to try it alongside other options, or, in some cases using only Agile and nothing else. The question is where do we go from here? Have we now solved the software development problem? (To recognize that there is a problem, read Fred Brooks’ The Mythical Man-Month). First of all, the overall (research and anecdotal) evidence is in favor of Agile: it is a step in the right direction (actually major strides forward). Agile methodologies solve development project management problems better than other known methodologies and processes.

However, Agile is not the end of the software development road. There is a “beyond Agile.” The idea is to retain the strengths of Agile and improve its weaker areas.

On the strengths side: the values and principles as expressed in the Agile Manifesto; the philosophy of adaptability and continuous learning (there is good overlap with Lean thinking here); the embracing of change; the emphasis on delivering business value; the iteration heart beat; the retrospectives for making continuous learning happen; the use of testing throughout the lifecycle; gaining feedback from users; getting the business involved; applying macro-management to the team, with a multi-skilled team self-organizing. The list continues: pair programming, test-driven development etc.

However, what will change in ‘beyond Agile’ are the areas where Agile has addressed itself less well. So the emphasis in early phase Agile has been on small teams where possible. The problem is that some enterprise projects are very large scale and need a lot of teams on a global basis. Various Agile development groups have addressed these issues but there is no consensus. The use of architecture and modeling also varies across these approaches. I expect some new form of Agile-friendly architecture and modeling will emerge. Certainly, the technology needs to improve: nothing quite beats the adaptability, the versatility (the agility) of programming languages — and creating software by drawing UML diagrams alone is dreadfully dull.

Another fault line has to do with QA and testing. Listening to some developers talk about how they had to bypass the (non-Agile) QA facility within their organization because it became a bottleneck, where they took on the job of QA themselves. That illustrates how QA and development have become separated in some organizations. ‘Beyond Agile’ I envisage will see QA and testing (the whole range, not just developer testing) become better integrated with development. While Agile developers have embraced quality and testing, the expertise in traditional QA and testing should not be lost.

Managing Agile stories, vast numbers running into thousands, and dealing with interdependencies and the transformation from business orientation to technical orientation — this is another area that could benefit from refinement.

While Agile expands its reach beyond development into operations in DevOps, and into business development (where Lean thinking is already established), the question is whether in the future the practices will be recognizably Agile or follow a new development wave. My hunch is that it will be recognizably rooted in our current understanding of Agile. It would be just fine if Agile became so established and traditional that we called it simply ‘software development’, without further distinction.

Event notice: Special 10th anniversary: The Agile Alliance’s Agile2011 Conference, Aug 8-12, 2011, will be revisiting Salt Lake City, Utah, where the Agile Manifesto was written back in 2001. I’m told the original signatories of the Agile Manifesto will be on stage to debate the progress of Agile.

3 thoughts on “Guest Post: Beyond Agile”

  1. An interesting read with some excellent points – thank you.

    I find it interesting, though, that in one paragraph you talk about “project leaders looking back on large software failures say that, rather than the hundreds of developers that were used…they would pick the 10 ablest developers and get the job done in short time”, and later you state, “The problem is that some enterprise projects are very large scale and need a lot of teams on a global basis.”

    (BTW, I believe the first statement came from the post-mortem of the US Air Force’s SAGE project in the late 50’s and early 60’s.)

    Is that not a contradiction? Just because a ‘system’ may be large doesn’t mean that the group of people working on it must be as well. Conway’s Law is one that should be actively broken, not followed to the letter!

    Yes, scaling Agile has been discussed since I first heard about XP in 2000. I have experience with the approach that Craig Larman and Bas Vodde have put forth in their Scaling Lean & Agile books, and it does work reasonably well. Yes, there are products and situations that require large teams. Yes, organizational circumstances may result in global dispersion of the people who will do the work.

    However, if you work from the perspective that those situations are the exception and not the rule, e.g. use a small number of small, co-located teams, then you will reap the benefits of Agile without having to deal with many of the problems associated with anything “big” in the IT industry. In other words, don’t start small, start tiny and only scale when there is a distinct reason to do so rather than taking the opposite approach of assuming that you have to have a large team to build something large.

  2. You’re right Dave. There is a contradiction – let’s examine it: Why do frustrated project managers say that they could get the job done quicker with just ten of the best (and let’s assume they’re Agile!). The answer is communication – and its breakdown and pitfalls in large scale projects. Agile (eg Scrum) solves this communication problem in the small scale.

    The problem of course is when tiny or small is not enough, sometimes the scale of the beast is such that you need a lot of developers – what then? You mentioned Larman & Vodde and to quote their book: “Large scale Scrum is Scrum…it is not new and improved Scrum…it is regular Scrum”. (You need to preface it with “In our opinion…” – there is no single source of truth here, like Moses on Mount Sinai). They go on to describe the pattern of a Scrum team and process and then repeat it multiple times, with some join refinements to deal with the new larger scale. Jeff Sutherland does something similar with Scrum of Scrums, Scrum of Scrums of Scrums etc. ThoughtWorks does (or at least did in one case) something a little different in its conquer then divide approach. The crux therefore is how to deal with communication in large scale Agile projects: different Agile approaches work for different organizations, skill levels, types of projects etc.

    You mentioned Larman & Vodde pattern worked for you, so yours and others’ experiences will build to a body of evidence. I think we’re still at a stage of experimentation — and it may be that the ‘Agile’ answer is to breakdown a large scale project into smaller ones (I’ve read some that advocate that) and then you just have a sequence of manageable projects that at a later stage you can join up. You kind of touched on that in your last paragraph.

Leave a Reply

Your email address will not be published. Required fields are marked *