[This article is a guest post by Rajesh Raheja, who is a senior director of development in the Oracle Fusion Middleware SOA group, responsible for developing SOA governance tools and providing engineering guidance to key customers and partners for Oracle's Application Integration Architecture. He is a Certified ScrumMaster and a Stanford Certified Project Manager. You can visit his blog or connect on Twitter @RahejaRajesh.
Note: The views expressed on this blog are his personal views and do not necessarily reflect the views of his employer.]
Agile faces some key challenges to overcome when dealing with adoption in the enterprise. Projects in the enterprise come in various flavors – from developing minor utilities to large enterprise business applications. The latter kind of projects have characteristics that are a bit different from other projects. For example, integrating multiple heterogeneous applications to satisfy an end-to-end business process, requires a lot of time doing functional analysis as well as testing. A rough estimate pegs approximately 70% of the project time (of around 9-15 months per release) on just analysis and testing.
Although a waterfall based approach would be simpler, it would not have the improved visibility and the ability to deliver incremental value, both of which are needed to reduce risk in such complex projects. Looking back at projects which started off excited about using Agile, many reverted to some mix of waterfall and Agile (i.e. “iterfall”) models, highlighting issues with “strict Scrum” (our preferred choice of Agile methodology). This reflects the realities that Agile has to face in the enterprise.
Here are the top 3 challenges faced:
1. Complex inter-dependencies between projects
This was by far the biggest issue faced. Enterprise projects are complex with many layers of inter-dependencies, and this is not addressed effectively in Scrum. For example, an application integration project that sends customer orders between a CRM and ERP fulfillment system would have the following tasks done by each team i.e. the CRM, ERP team and middleware/integration teams.
- Configuration of new data model attributes
- Modifications to API and web services
- User interface enhancement to add new fields to existing screens
- Web service configuration to pass the new data elements to middleware
- Data migration or clean up scripts
- Regression testing
The integration team needs to wait on the application teams to develop the above. These application teams handle many unrelated projects and do not necessarily use Scrum (there goes the Scrum of Scrums). Visibility into deliverables is typically at a milestone which is usually communicated upfront. Testing activities are challenging even without Scrum, because even if one application is ready, the end-to-end testing cannot happen till the other application is also ready.
The Scrum approach of dealing with this by avoiding dependencies just does not cut it and immediately relegates Scrum to the “only use for small non-critical projects” bucket. To gracefully handle this issue, most teams I worked with ended up either customizing their approach to “Scrum-like” or “iterfall”.
2. Handling of Specialized and Global Project Resources
Scrum emphasis on self-organizing teams works great when teams are small and co-located. But it gets challenging when the reality is that teams today are global and geographically distributed across two or three continents.
On top of that, Scrum did not seem to cater to specialized resources very effectively. For example, in major organizations, there is often a shared community of expert resources such as architects, usability engineers, QA, performance and scalability team; who serve many project teams. These resources have to be committed well in advance and have specific deliverables, which also becomes an additional dependency to track. It cannot simply be assumed that these specialized resources would be available for every sprint, forget about every stand-up meeting.
3. Sprint Overhead
The concept of sprints – a time boxed period in which user stories are planned, executed and made production-ready – was initially very alluring, but later became one of the reasons why some teams came up with their own approaches to bypass the overhead caused by it.
Sprint overhead is primarily caused by valid user stories that always exceeded sprint lengths. No, these were not epics, and had been broken down to the level acceptable by the product owner as providing real business value. For complex projects, even a three or four week sprint could not fit an end-to-end feature story due to the dependencies. Some teams initially resorted to creative hacks e.g. artificial break up of stories (“Architecture – Part 1”), which really defeated the core purpose. Those who did not resort to such hacks were penalized with inaccurate velocity calculations and perception of poor performance since many sprints had incomplete stories with zero velocity. Instead of helping, this need to satisfy strict Scrum logistics interfered with the development of such projects. An unfortunate side-effect of trying to finish off stories in unreasonable sprints was lack of good documentation; with business knowledge sometimes being sacrificed in the name of lean documentation.
This reminds me of a popular Mexican fresh grill slogan:
“Food cannot be made at microwave speed.”
Similarly, it became apparent that complex enterprise projects cannot be built with fixed sprint lengths. Moreover, product roadmaps are built years in advance where the supposed flexibility of dropping features is sometimes not even an option. Overall, business process analysis, attribute mapping, functional interoperability design and architecture design needs flexibility and adaptability in the Agile methodology.
As readers may notice a lot of the emphasis above is on “strict Scrum.” Teams that managed to overcome these limitations did so by adapting Scrum (which I will describe in future posts). Upon hindsight, some projects may have been much better suited to Kanban, something I am now trying on a smaller project. Some practitioners consider Kanban as “watered down Scrum” and others think “Scrum, but” and “Kanban-like” to be the cause of failed Agile projects, but in my experience, it is because of these derivative approaches that complex enterprise agile projects succeeded, without which most would have moved back to waterfall due to the execution risk posed by the strict Scrum limitations. As pointed out by another practitioner, we need to leave the “turf wars” behind and use the right tool for the right job. Agile is just emerging and will need to be flexible in order to move beyond the hype and gain true relevance in the enterprise.
Would you like to contribute as an Agile Scout Guest Poster? Let us know.