How to scale agile software development without reinventing the waterfall

Founder and CEO of Inflectrawith 25 years of experience in the technology industry.

In the software industry, which I’ve been in for 25 years, there’s a saying that often applies: “What’s old is new again.” Today’s messaging tools like Slack and Teams, for example, have a lot in common with previous generations of messaging like AIM, ICQ and YTALK to do.

Likewise, the latest trend in the world of software lifecycle methodology is “scaling Agile”. (I will use this terminology “scale agile” versus “scale agile” as the latter implies a specific framework or methodology). However, in the rush to remove some of the (valid) limitations of current agile methods, there is a risk that we will simply reinvent the waterfall. In this article I will examine why this can cause problems.

Why was Waterfall replaced?

Waterfall, the original software development methodology, was based on a series of linear stages of taking a customer’s requirements, putting them into writing, and then sequentially designing, developing, and testing them. That sounded good in theory, but each phase took several months. Also, because the design was mostly incomplete and the requirements unclear, testing was rushed and the final product was flawed and delayed. Also, when the product was ready, the customer realized that it wasn’t what he wanted.

The Agile Manifesto, announced in 2000, aimed to change the way software was built. With the emphasis on real-time communication and working software over formal documentation, this represented a sea change in the way projects were managed.

One of the key practices that emerged in the various Agile methodologies (e.g. Scrum and eXtreme Programming) was to have short iterations or sprints (two to three weeks) in which you delivered incrementally working software. This allowed the client to provide timely feedback, significantly reducing the conceptual, technical, and programmatic risks to a successful outcome. In addition, working software was available at all times, so that changes in direction could be made without great “lower costs”.

Why do we want to scale Agile?

One of the assumptions behind the “Agile Manifesto” and much of the early literature (e.g Extreme Programming Explained: Embracing Change by Kent Beck) was that Agile worked best when:

• They had a small team of about 10 cross-functional people.

• The team worked at a common location and could work together personally.

• Customers were on site and able to provide real-time feedback.

• The individual requirements (called “user stories”) were completely independent of one another.

• The project team could work independently from other teams.

However, as IT has become critical to the functioning of the entire world (software eats everything), the complexity of projects has increased. In a given automobile there may be thousands of software products that need to be integrated and coordinated. The increase in remote work also means that teams are often not in the same location or even in the same time zone. Increased security vigilance means that we need to manage the compliance aspects of our software products and ensure that all software code has been adequately tested and reviewed.

As a result, companies using methodologies such as Scrum, XP, and Kanban struggle to deliver value at scale. Therefore, a way needs to be found to be agile on a larger scale.

What are the pitfalls of agile scaling?

Consider a retail bank. You have many teams working on different systems and products. Teams may be working on their T24 core banking system, their customer website, their online account management portal, their in-branch teller platform, and consumer mobile banking apps for iOS and Android phones. Each of these products would involve multiple separate Agile teams with their own backlogs, sprint plans, and release schedules.

Suppose the company’s strategic goal is to “redefine banking” to become the most customer-friendly bank in the industry. How does the organization prioritize all work in the bank and establish strategic direction from the top down? How to do that without creating a complex hierarchical top-down project plan reminiscent of the “bad old days” of the waterfall?

A major danger would be the creation of one giant enterprise-wide requirements backlog, with all requirements, functions, and needs managed in a single, monolithic structure. Likewise, it leads to micromanagement when the CIO/CTO manages the entire effort as a single, tightly coupled effort.

Unfortunately, I’ve seen this in action where a developer updated a task and immediately received a call from the CTO asking why they were jeopardizing the entire project schedule.

How do you achieve agile scaling?

The key is understanding the different levels involved and making sure the information and teams are organized accordingly. There are actually three key levels to agile scaling:

•Portfolio (or department): This level is responsible for determining the key strategic deliverables that the organization cares about and mapping them to high-level milestones. For example: “We aim to launch our new consumer-friendly concept for retail banking in the third quarter of 2024.”

• Program (or solution): This level captures the strategic goals and determines what cross-functional capabilities are needed to make the vision a reality. These features span multiple products and are mapped to their own overarching timeline. For example, one feature could be “the ability to view account balances across all channels through Q1 2024.”

• Product (or project): Each product is responsible for providing discrete functions that provide part of the overall functionality. For example, the web portal, iOS app, and Android app product teams must each have specific requirements (“user stories”) in their backlog that must be in place for the omnichannel experience to be available.

With this model, you don’t have a single monolithic project plan, instead each layer works autonomously to deliver the functionality needed, with the organization being able to measure value and prioritize resources to achieve the strategic objective. In this way, teams are agile but working to support the larger vision.

The Forbes Technology Council is an invitation-only community for top-notch CIOs, CTOs, and technology executives. Am I Qualified?

Leave a Comment