08.25.09
Posted in Agile Development, Application Lifecycle Management (ALM) at 3:06 am by Tony Baer
For those of us on the outside looking in, it might be tempting to pigeonhole the agile software development community as a highly vocal but homogeneous minority that gained its voice through the Agile Manifesto. In actuality, the manifesto was simply a call to action to, in essence, value working software over working plans. The authors didn’t pretend to monopolize knowledge on the topic and purposely made their manifesto aspirational rather than prescriptive.
And so, not surprisingly, in a world of software development methodologies, there is no such thing as an agile methodology. There are many. There is Scrum, so named for the quick huddles in Rugby that call for fast decisions based on a basic assessment of conditions on the ground. Proponents call scrum responsive while critics assail it for lacking the big picture. In actuality, properly managed scrums do have master plans for direction and context. Other variants such as eXtreme Programming (XP) drive development through testing to ensure 100% test coverage. In actuality, agile methodology is a spectrum that can borrow practices from scrum, XP or other methodologies as long as it meets the goal of working software with minimal red tape.
More recently, debate has emerged over yet another refinement of agile – Lean Development, which borrows many of the total quality improvement and continuous waste reduction principles of lean manufacturing. Lean is dedicated to elimination of waste, but not at all costs (like Six Sigma). Instead, it is about continuous improvement in quality, which will lead to waste reduction. It aims to root out rework, which is a great waste of time and money, and a process that subtracts rather than adds value to the product. Lean techniques were at the crux of new enhancements to Danube’s agile planning offering which we discussed a few weeks back.
The best known proponent of lean development, Mary Poppendieck, makes the metaphor that software development is very much like a manufacturing process. Having spent serious time studying and writing about lean manufacturing, and having spent time with the people who implemented lean at the pioneering NUMMI auto plant nearly 20 years ago, we can very much relate to Poppendieck’s metaphors.
In essence, developing software is like making a durable good like a car, appliance, military transport, machine tool, or consumer electronics product. In essence you are building complex products that are expected to have a long service life, and which may require updates or repairs. We also contend that the metaphor means that software should also carry a warranty, where the manufacturer vouches for the product’s performance over a specific operating life.
But not surprisingly, there is hardly any consensus over lean in the agile community. We came across a well-considered dissent by Pillar Technology consultant Daryl Kulak that criticizes the very idea of comparing software development and manufacturing in the same sentence. Kulak maintains that unlike manufacturing, software is not comprised of the kinds of uniform discrete parts that are found in manufacturing. There’s a bit of irony there, as proponents of SOA, and before that component-based development, often called for just that. But Kulak is correct that most software is not uniform discrete parts, at least according to common methods of software construction and design. And besides, there is always some local environmental variable that will cause variations in how software is configured to run on different targets.
That’s a key part of Kulak’s argument. “We don’t discover its actual character until we’ve started building it.” He also argues that the core systems-oriented principles that are at the heart of lean could lean to “ ‘mechanical Agile,” where we view our teams as machines and our people as a kind of robot. This type of team can only follow orders and cannot innovate or adapt.”
Kulak and others have good reason for their cynicism, as it wasn’t that long ago that CFOs actually believed the promises of outsourcers that software factories would make software development predictable in time and cost. That had great appeal during the post 2000 recession. Obviously – in retrospect – software development is not so cut and dried – business needs are always changing (that’s why we have agile) and software deployment environments always have their unique idiosyncrasies.
(There has been a more recent refinement of the software factories idea that focuses, not so much on offshore factories. Instead the emphasis is more on patterns and industrializing aspects of software development, e.g., the portions of an app that could be exposed as a reusable service. But that doesn’t eliminate people from the equation, it amps up the architecture.)
But back to core issue, lean cynics miss the point. When we visited the NUMMI plant, we were struck by how lean actually spurred production line workers to get creative and develop solutions to problems that they carefully tracked on the plant floor. We saw all those hand-drawn Pareto charts on the walls, and in fact the reason why we were there was to study implementation of the first computerized production management system that would help people track quality trends more thoroughly. What we saw was hardly a mechanical process, but one that encouraged people to ask questions, then base their decisions on a combination of hard data and personal experience. It was actually quite a step up from traditional command and control production line management.
There’s no reason why this approach shouldn’t work in software. This doesn’t mean that lean approaches will work in all software projects, as conducting value stream analyses could get disruptive for projects, such as the ever evolving Facebook apps of the Obama 2008 campaign. But the same goes for agile – not all projects are cut out for agile methodologies.
So to software developers, stop thinking that you’re so unique. Obviously, lean manufacturing methods won’t apply verbatim to software development, but look at the ideas behind it. Stop getting hung up on minor details, like software isn’t a piece part.
Update: The NUMMI auto plant in Fremont, California has unfortunately become a casualty of the restructuring of GM and contraction in the US auto market. So it would be tempting to state that NUMMI was a case where the surgery was successful but the patient died. The lesson for software developers is that process only ensures that the product will be developed; it doesn’t ensure success of the product.
Permalink
08.19.09
Posted in .NET, Application Development, Cloud, IT Infrastructure, Java, OS/Platforms, Virtualization at 1:05 am by Tony Baer
With the ink not yet dry on VMware’s offer to buy SpringSource, it’s time for SpringSource to get back to its regularly scheduled program. That happened to be SpringSource’s unveiling of the Cloud Foundry developer preview: This was the announcement that SpringSource was going to get out before the program got interrupted by the wheels of finance.
Cloud Foundry, a recent SpringSource acquisition, brings SpringSource’s evolution from niche technology to lightweight stack provider full circle. Just as pre-Red Hat JBoss was considered a light weight alternative to WebSphere and WebLogic, SpringSource is positioning itself as a kinder and gentler alternative to the growing JBoss-Red Hat stack. And that’s where the VMware connection comes into play, but more about that later.
The key of course is that SpringSource rides on the popularity of the Spring framework around which the company was founded. The company claims the Spring framework now shows up in roughly half of all Java installations. Its success is attributable to the way that Spring simplifies deployment to Java EE. But as popular as the Spring framework is, as an open source company, SpringSource monetizes only a fraction of al Spring framework deployments. So over the past few years it has been surrounding the framework with a stack of lightweight technologies that complement it, encompassing the:
• Tomcat servlet container (a lightweight Java server) and the newer DM server that is based on OSGi technology.
• Hyperic as the management stack;
• Groovy and Grails, which provides dynamic scripting that is native to the JVM, and an accompanying framework to make Groovy programming easy; and
• Cloud Foundry, which provided SpringSource the technology to mount its offerings in the cloud.
From a mercenary standpoint, putting all the pieces out in a cloud enables SpringSource to more thoroughly monetize the open source assets that otherwise gain revenue stream through support subscriptions.
But in another sense, you could consider the SpringSource’s Cloud Foundry as the Java equivalent of what Microsoft plans to do with Azure. In both cases, the goal is Platform-as-a-Service offerings based on familiar technology (Java, .NET) that can run in and outside the cloud. Microsoft calls it Software + Services. What both also have in common is that they are still in preview and not likely to go GA until next year.
But beyond the fact that SpringSource’s offering is Java-based, the combination with VMware adds yet a more basic differentiator. While Microsoft Azure is an attempt to preserve the Windows and Microsoft Office franchise, when you add VMware to the mix, the goal on SpringSource’s side is to make the OS irrelevant.
There are other intriguing possibilities with the link to VMware such as the possibility that some of the principles of the Spring framework (e.g., dependency injection, which abstract dependencies so developers don’t have to worry about writing all the necessary configuration files) might be applied to managing virtualization, which untamed, could become quite a beast to manage. And as we mentioned last week in the wake of the VMware announcement, SpringSource could do with some JVM virtualization so that each time you need to stretch the processing of Java objects., that you don’t have to blindly sprawl out another VM container.
Permalink
08.11.09
Posted in .NET, Agile Development, Application Development, Cloud, IT Infrastructure, Java, Middleware, OS/Platforms, Virtualization at 1:30 am by Tony Baer
VMware’s proposed $362 million acquisition of SpringSource is all about getting serious in competing with Salesforce.com and Google App Engine as the Platform-as-a-Service (PaaS) cloud with the technology that everybody already uses.
This acquisition was a means to an end, pairing two companies that could not be less alike. VMware is a household name, sells software through traditional commercial licenses, and markets to IT operations. SpringSource is a grassroots, open source developer-oriented firm whose business is a cottage industry by comparison. The cloud brought both companies together that each faced complementary limitations on their growth. VMware needed to grow out beyond its hardware virtualization niche if it was to regain its groove, while SpringSource needed to grow up and find deeper pockets to become anything more than a popular niche player.
The fact is that providing a virtualization engine, even if you pad it with management utilities that act like an operating system, is still a raw cloud with little pull unless you go higher up in the stack. Raw clouds have their appeal only to vendors that resell capacity or enterprise large firms with the deep benches of infrastructure expertise to run their own virtual environments. For the rest of us, we need a player that provides a deployment environment, handles the plumbing, that is married to a development environment. That is what Salesforce’s Force.com and Google’s App Engine are all about. VMware’s gambit is in a way very similar to Microsoft’s Software + Services strategy: use the software and platforms that you are already used to, rather than some new environment in a cloud setting. There’s nothing more familiar to large IT environments than VMware’s ESX virtualization engine, and in the Java community, there’s nothing more familiar than the Spring framework which – according to the company – accounts for roughly half of all Java installations.
With roughly $60 million in stock options for SpringSource’s 150-person staff, VMware is intent on keeping the people as it knows nothing about the Java virtualization business. Normally, we’d question a deal like this because the company’s are so dissimilar. But the fact that they are complementary pieces to a PaaS offering gives the combination stickiness.
For instance, VMware’s vSphere’s cloud management environment (in a fit of bravado, VMware calls it a cloud OS) can understand resource consumption of VM containers; with SpringSource, it gets to peer inside the black box and understand why those containers are hogging resource. That provides more flexibility and smarts for optimizing virtualization strategies, and can help cloud customers answer the question: do we need to spin out more VMs, perform some load balancing, or re-apportion all those Spring TC (Tomcat) servlet containers?
The addition of SpringSource also complements VMware’s cloud portfolio in other ways. In his blog about the deal, SpringSource CEO Rod Johnson noted that the idea of pairing VMware’s Lab Manager (that’s the test lab automation piece that VMware picked up through the Akimbi acquisition) proved highly popular with Spring framework customers. In actuality, if you extend Lab manager from simply spinning out images of testbeds to spinning out runtime containers, you would have VMware’s answer to IBM’s recently-introduced WebSphere Cloudburst appliance.
VMware isn’t finished however. The most glaring omission is need for Java object distributed caching to provide yet another alternative to scalability. If you only rely on spinning out more VMs, you get a highly rigid one-dimensional cloud that will not provide the economies of scale and flexibility that clouds are supposed to provide. So we wouldn’t be surprised if GigaSpaces or Terracotta might be next in VMware’s acquisition plans.
Permalink
08.03.09
Posted in Agile Development, Application Development, Application Lifecycle Management (ALM) at 4:04 pm by Tony Baer
Although this month our first order of business is to demolish the governance silos separating managing the application lifecycle (ALM) from IT Service Management (ITSM), and then of course achieve world peace, it’s impossible to ignore the Agile software development world’s upcoming annual event. Sadly, for the third straight year we’ll miss the Agile software conference. We’re getting into the stream of tools briefings that for the most part revolve around the common theme of addressing some hurdle to scaling up or out agile practices.
This week we’ve been briefed by VerisonOne which has chosen to focus on adding support for lean software development variant of agile in its Summer 2009 release. It’s very easy to get confused here, as on the face of it, agile is a lean approach to software development if you define lean as lighter weight process that stems to slice through the red tape.
But if you adhere more loosely to the definition of lean that emerged out of the just-in-time, continuous improvement-oriented practices of the manufacturing world, there are methods that depart from your typical scrum or form of extreme development. The core thread of lean is eliminating waste through value stream mapping (essentially, the workflow that a product undergoes, documenting the steps and cycles times for adding value) so you can decide where to attack waste and bloat. Then, as with core agile practices, you assume planning goals are moving targets; the subtle distinction in lean is that you make key decisions as late as possible, then deliver as fast as possible (before the goalposts change).
Having a background in manufacturing, and having visited the famous GM/Toyota NUMMI assembly plant joint venture as they were implementing their first computerized production and quality tracking systems 20 years ago, we can relate to the idea of lean. In the software development world, Mary and Tom Poppendieck have become the best-known evangelists; suffice it to say that in the agile world there is plenty of debate on the merits of lean: that lean require more systematic planning and upfront analysis of waste compared to traditional agile, vs. whether the upfront work leads to analysis paralysis. Clearly, the ability to analyze waste up front does require more of a big picture view than many software development practitioners either have in their veins or are trained for. We’ll pass the buck here by saying that the choice of methodology should depend on the project.
VersionOne has attacked the problem by adding features such as visualizing a value stream with the ability to drill down on each step to apply, in effect, what we’d call development policies. That is, you may decide that here are certain parameters, such as amount of work-in-process (uncompleted work), that must be enforced so that development teams do not get in over their heads with too many deliverables at any one time. It can also reveal findings on cycle time that can help you either whittle away at the scope of a particular development iteration, and of course, critical metrics such as defect density so you can connect the dots between development steps, code characteristics, project scope, and quality issues.
What would be nice would be to have some simulation capabilities, so you could test drive changing planning assumptions to see how it might impact developer productivity or the ability to meet delivery commitments. Alas, that feature will have to wait, but it’s always helpful when a tools provider steps into the debate over software development methodology, as it takes those debates out of the realm of theory.
Permalink
Posted in Agile Development, Application Development, Application Lifecycle Management (ALM) at 12:30 pm by Tony Baer
Publishing of the Agile Manifesto back in 2001 formalized a belief in the development community that the business goals of software development projects were in actuality moving targets. Since then, agile has emerged as mostly a bottom-up movement among true believers. In a handful of cases, such as within software vendors (whose business is software), agile has become a matter of enterprise urgency. In scattered cases, such as at BMC, successes of enterprise-scale agile development (e.g., multiple teams across multiple sites). But for the most part agile has been perceived as a cottage industry in the developer community that often views itself in a David vs. Goliath situation.
There is little question that agile makes a good fit for the kind of incremental development that adds or extends existing software portfolios. If the problems are well-contained, they are well-suited for the compact teams that are typically involved with agile projects. Furthermore, as incremental software, extensions and add-ons do not it against what is agile’s greatest weakness: the ability to maintain consistent direction and big picture vision in a methodology where the assumptions are revisited with each iteration or release cycle.
All that is an urban myth however as true agile development does require the basic scoping and planning required for larger projects. While the business environment is hardly static these days, no CEO or CFO in his or her right mind would sign their name to an open ended project that might evolve in random directions. Admittedly the approach to planning for an agile project might differ from a more conventional one, but planning, scoping, and high end requirements planning are essential so that the agile project – as changing as it is – operates under the right assumptions and project leaders adequately manage customer expectations. In agile speak, by the way, such mega planning cycles are appropriately called “epics.’
The other side of the coin is that for agile methodology to measurably impact software development, it must be able to scale up to the point where multiple teams can coexist on larger projects. Just because software development is incremental today doesn’t mean that large projects are a thing of the past. On the other hand, while it is critical not to pigeonhole agile into small projects, there is no single methodology that fits all projects. In some cases, you may have a large project that remains better suited for waterfall.
Over the past few years, agile planning tools providers have been gradually adding features that extend the rings of agile planning past the small group holed up by the water cooler. And in coming weeks leading up to the Agile 2009 conference, you’re likely to see other announcements from the likes of Rally, ThoughtWorks and others.
This week it’s Danube’s turn. They’ve come up with a clever search and aggregation facility within their project planner to identify and group project goals and objectives that gets you beyond the rigid hierarchical schemes of most project planners – agile or otherwise. The rationale for such need is that multiple release cycles may share different mixes of common planning goals, and that in one release cycle, the order of priority of those goals might vary. For instance, in planning a customer portal, the requirement of concealing the customer’s actual identity is always important, but it may be more important when it comes to page navigation or entry of credit card numbers than where it comes to setting a page display or shipping preference. Within a conventional hierarchical planning scheme there is no way to distinguish that.
Danube’s innovation typifies the state of agile planning – tools providers are beginning to add the bells and whistles that can improve the ability to communicate and coordinate across larger or more complex projects. But it also reveals that the solution is hardly a done deal. For instance, as Danube can now facilitate a more sophisticated means for managing requirements and goals for more involved agile projects, it lacks the next logical piece of the puzzle: adding traceability. As you consider goals for each iteration, it helps to have context, and in regulated situations, it is essential to understand where and why those requirements or goals originated, and how and when they were or are redressed. Danube has change logs, but on the next release, we’d like to see them raise the ante with some ability to track the bread crumbs that will inevitably accumulate during the course of an agile project.
Permalink