Waterfall, Agile, and Organizational Focus

A colleague asked me the following question, which I thought was great and could be helpful to others to discuss.

Question: "Many of our IT projects use a project management style that is more Waterfall than not. This creates all the problems associated with a Waterfall process, such as very long delivery times and poor fit for requirements.

Some attempts have been made at introducing an Agile methodology. The most common reason these didn't work out seems to be that cross-functional teams compete with the regular/operational work duties of team members. People claim they aren’t able to attend meetings.

When multiple members are regularly not coming to meetings, the results are predictable: frustration, failure to meet deadlines, poorer deliverables. The end result seems to be a return to a Waterfall approach that silos stakeholders, PMs, and developers.

Is there a best practice way of dealing with what must be a common occurrence?"

Response: In my career the "best practice" response I've seen to this has been something like:

• Commit the organization to growing a Lean/Agile approach over time
• Hire a Lean/Agile coach to provide thorough training and coach some teams in Agile practices
• Use a continuous improvement cycle to help the initial teams improve while spreading the approach to more teams

I work in an IT organization where that approach was started over five years ago. I would say success is very mixed. On the plus side, Agile has stuck around, and the teams using it seem happy with the control and flexibility it gives them. Ongoing challenges in my organization are similar to what my colleague describes in his:

• Difficulty creating and meeting estimates for long-term work
• Difficulty keeping up with business priorities
• Difficulty getting business participation in the Product Owner role

What would be a standard "best practice" followup? Something along the lines of: do more training and coaching. Help teams improve their practices. Start smaller, demonstrate success, then grow the new practices.

Those are reasonable tactics, but I think there might be organizational root causes to consider as well. I think it has to do with the organization's basic ability to truly commit to Lean.

A basic tactic of Lean is to identify and eliminate instances of waste. A basic form of waste is work in progress. One way this this translates into IT is trying to do too many things at once compared to the resources available. (Another way to say this is that Lean is about respecting individuals, which includes enabling them to do their best work.)

Inability to focus and retain resources on top priorities is a problem that spans Waterfall and Agile approaches. Lack or loss of resources is a classic "top ten" reason for project failure in traditional project management. Another example: most organizations can name "bottleneck" resources who are clearly overloaded and (unwillingly) put outcomes at risk. And in Agile, because of the emphasis on continuous teamwork, it becomes especially apparent which individuals have too many competing duties.

From what I've seen and read, success with Lean and Agile requires a leap of faith. The organization's leaders, customers, and teams must truly believe that in the long run, focusing resources is better because:

• Completing a few high priority outcomes sooner (and moving on to the next ones) is more valuable than making slow progress on numerous competing priorities (and falling short in most)
• People are more productive when they do less switching between jobs, so total productivity increases
• Quality increases when people have enough focus to improve how they work

The value proposition is straightforward, but the reality for organizations is quite difficult. That's where it's important to be honest about the organization's ability to commit. Can the organization truly:

• Make and stick to a long-term commitment that affects everyone -- are there relevant examples of doing this?
• Ensure that decision-makers prioritize some goals and defer others -- when their success so far has been based on appearing to satisfy or appease everyone?
• Bring along everyone who will be impacted -- not just implementation teams but also operations and the customer?
• Push through a difficult learning process, a lot of contentious tradeoffs, and an initial apparent reduction in productivity to get to the benefits of a new way of working?

If we're being honest, I suspect the correct answer in many organizations is: no, not yet. In their book The Toyota Way to Service Excellence, Liker and Ross reflect that in most organizations Lean transformation will fizzle out or turn into a set of formulaic, low-value tools. The basic premises of focus and respect for people don't fit how those organizations define success in their culture.

What this suggests to me is looking for opportunities to shift an organization's culture over time and strengthen "muscles" over time.

One opportunity, as old-fashioned as it may seem, is to get better at Waterfall -- because many of the same muscles will help strengthen Agile teams later. For example:

• Improve resource tracking to provide better information about how many and which projects can realistically be done
• Improve project prioritization to ensure the most important projects get the most attention
• Proactively identify and resolve resource contentions
• Improve estimation and tracking of projects to provide a "feedback loop" for resource management and prioritization
• Improve communication with stakeholders to demonstrate where progress is being made and why other goals are being deferred

For IT organizations, the cloud and DevOps offer another kind of opportunity at the moment, because they shift how resources are used and how teams work. If teams for new services can be encapsulated and grown with limited dependency on existing operations, and those services can maintain focus, then gradually the culture of an organization can change as other services fall away. Of course, that's a long game with serious management commitment and risk.

What other opportunities have people seen work to overcome barriers to organizational focus in either Agile or Waterfall? I'd love to see your comments.