What Agile Software Development Really Means
Is Agile a fad? Is it a "methodology"? A "process"? A "framework"? What the heck is it?
Furthermore, why are there so many "flavors" of Agile? Why is everybody doing their own thing?
And finally, why does Agile work great for some people, but others really struggle with it, have great difficulty adopting it, or suffer poor results from it?
Fear not, because in this very article I will explain, once and for all, what Agile is really about, which will in turn will explain all of the above.
To be more precise, I don't have enough space here to go into all the ins and outs of what we nowadays consider as "Agile", but I can explain the main, overriding idea and philosophy behind it. Once you firmly grasp this, everything becomes clearer about how Agile is supposed to work, how to figure out how make it work, and what it essentially is. Ready? Here it is:
Agile is about reducing inefficiency and increasing quality under dynamic conditions.
That's a mouthful which does not necessarily make sense at first. Let's tackle it in parts. Think about the common hype you hear around Agile: "Get things done in half the time at half the cost". OK, that's a little bombastic, but it clearly gets to the heart of the first thing Agile is trying to do; to minimize time and cost in development.
The origin of this is clearer if you know something about Agile's history. In its early days of development, Agile took its inspiration from the efficiency revolution that happened in automotive manufacturing in the 1970' and 80's. This began with the now-infamous Toyota Production System and later evolved into what was called "Lean Manufacturing". If you ever wondered how Japanese car companies managed to trounce American car companies so thoroughly in those and subsequent decades, there is your answer.
The whole drive behind Lean Manufacturing is to eliminate waste. The first step, obviously, is to identify where waste occurs, which is exactly what the Japanese did, anywhere and everywhere they could. They recognized that stocking up huge volumes of inventory in advance of knowing actual demand led to waste; this led to the innovation of Just-in-Time inventory. They realized that giving their workers too much work at times resulted in waste; as did giving them too little work at other times. In addition to creating a smooth workflow, they emphasized timely adaptations to situations on the manufacturing line--and, more broadly, to market changes in general.
So this inspired the thinking--can we improve outcomes in software development by adopting a similar philosophy? We'll return to this in a minute.
Now I also said that Agile is about increasing quality, because this is the second half of the automotive revolution I'm referring to. You see the success in Japanese automotive manufacturing had a lot to do with one man: W. Edwards Deming. One of the earliest of what we now call "management consultants", Deming proposed that, in order to sustain and achieve long-term efficiency, you had to focus on quality. When he could not get any US companies to buy into this, he went to Japan to help them with their reconstruction effort after World War II. In so doing, he instructed them on how to build some of the highest quality products the world had ever seen--at lower prices and higher volumes than anyone had ever seen.
Not only did Deming promote quality, he advocated the process of constantly looking for ways to improve quality further. He recognized that quality was the ultimate "eliminator of waste", because when things were done right, they did not result in multiple fixes and revisits later due to issues down the line, which bled time and cost.
A key innovation is that the type of efficiency that Deming pursued was a holistic, human centered form of efficiency. Others thought efficiency was just about "saving money", or "time to market", something that happened through feats of accounting or engineering. Not Deming. He realized that efficiencies had to be created by people. For that to happen, people had to be engaged and empowered. That's why, when Deming listed out the important principles in producing high levels of efficiency and quality, he emphasized not just the importance of quality, but also of the individual, and of organizational culture. Some of Deming's "14 Points for Total Quality Management":
Build quality into a product throughout production.
Work to constantly improve quality and productivity.
Drive out fear; create trust.
Teach and institute leadership to improve all job functions.
Strive to reduce intradepartmental conflicts.
Remove barriers that rob people of pride of workmanship.
Include everyone in the company to accomplish the transformation.
This legacy of ideas is reflected today when we talk about the importance of culture when it comes to Agile. In today's terms we talk about:
Collaboration
Engagement
Sustainable Pace
Continuous Improvement
..and so on.
Furthermore, at the heart of Agile ideas are those time-tested principles of efficiency and quality. Efficiency over what? That's a good question, because efficiency is not an absolute--it's context dependent.
The popular refrain is that Agile is more efficient than "Waterfall". What is Waterfall? That's the "old model" of software development that attempted to build software using an "assembly line" process and a top-down, bureaucratic form of management. It is commonly said that Waterfall proves very inefficient, and is a costly way to develop software.
This is largely true, but we need to understand why. Because the truth is, in some contexts, the "assembly line" is the most efficient way to get things done; there's a reason it exists--after all it did power the industrial revolution. Top-down management is highly efficient in some contexts as well--it is a model inherited from the military, where it has been successful for thousands of years.
The reason Agile exists, however, is as a solution to a some fundamental realizations about software development, including:
Software requirements tend to be emergent and to change often;
The knowledge and expertise for software implementation is at the bottom rather than at the top
Given this context, what we have discovered is that Waterfall is not the most efficient way to get things done. It is based on a model where those at the top of the organization hold all the knowledge, and can plan things out for everyone else to follow. Success is then a matter of everyone below following the plan to the letter. Now for situations where a project can be planned out in comprehensive detail by those at the top, where the plan is unlikely to change, and where the individuals performing the work just have to follow the plan rather than constantly interact with each other--then yes, Waterfall is actually a very efficient way to get things done.
All we're saying is that this is not typical of most software development projects. On the contrary, many of them are quite complex and dynamic. No one person knows how to build the solution; it takes a team of people to figure out. The ones who can properly vet, estimate, and plan significant parts of the work are the developers rather than the managers. Users and stakeholders change requirements as their needs change and evolve. Unanticipated technical hurdles are found, which puts everything on hold as they are worked through. And so on and so forth.
This is why Agile is commonly associated with practices. These practices are clever ways that people have found to improve efficiency and productivity under these conditions. For example, some of the ideas behind common Agile practices include:
Frequent Interactions
Developing software, especially under difficult conditions, requires good, frequent communication to keep everyone on the same page, to coordinate efforts, and to manage expectations. Working as separate departments which don't talk to each other, or relying on documentation to communicate, is far less efficient and productive than having real conversations and face-to-face interactions. Thus, many Agile practices have the agenda of bringing people together to have the right conversations, such as the Daily Scrum or the use of User Stories.
Iterative Development
Because things may change direction over time, or new paths may emerge, it makes sense to work in small batches, or "iterations/sprints", on what we can see most clearly in the moment. If we look up after completing one iteration to find that requirements have changed, we can work on the new requirements in the next iteration. We can also present the results of our iteration to get user feedback to check that we're on the right track. The iterative cycle gives us opportunities to "inspect and adapt". This is more efficient than toiling "heads down" on something for a long time, only to find out after the fact that work has to be redone because things changed or that user doesn't like how things turned out. Furthermore, an iteration allows us to practice "getting to done" on a defined amount of work within a defined amount of time. This is important, because it takes some practice before we really understand how long things actually take and how many steps are really involved to "get to done". By using iterations to practice, we can hone in a more realistic assessment of the overall project.
Feedback Loops
The iterations we discussed are one type of feedback loop; it allows us to measure and improve our performance, to get feedback from users, assess any changes in context or requirements, and so on. But the concept of the feedback loop underlies other aspects of Agile. The Daily Scrum is a feedback loop for the daily ins-and-outs of the project; Sprint Reviews are a feedback loop for the deliverables of the sprint; Sprint Retrospectives are a feedback loop for the sprint performance and an opportunity for the team to practice Continuous Improvement; Code Reviews are a feedback loop for the development team to assess each others work for quality and accuracy; and so on. A team that is committed to producing the highest possible quality and providing the users with exactly what they need will implement many feedback loops to ensure they meet this goal.
Small, Cross-functional Teams
Software development is a team sport, one that is exceedingly difficult to play with a bureaucratic structure. In contrast, small teams representing a cross-section of the disciplines and knowledge that need to go into the software tend to function more efficiently. A team of 10 or less people that has within it the right expertise, a single leader, and a process of working side-by-side with each other, can easily outperform the efforts of a much larger group of people operating across a system of departments and hierarchies. Things simply get done better when you get everybody in the same room, talking to each other, and working with each other.
Self Organization
In software development, it's really the developers who understand best how to estimate the work, and how to actually do it, not the managers above them. Therefore instead of "command-and-control" we need to empower our team to self-organize to determine, amongst themselves, how the work will actually be done, and who on the team will do what. For a manager to decide this arbitrarily hurts more than it helps. Therefore, while a team still benefits from coaching and direction, empowering it to self-organize, estimate, and commit to its own work tends to make more sense than the other way around.
Visibility
We cannot address inefficiencies and quality problems unless we can first identify them. That only happens when we make them highly visible, because many inefficiencies are hidden, or only known by a select few. Universal understanding and awareness is critical for keeping everyone on the same page and to keep expectations aligned. Thus, a key principle in Agile is to make things visible and transparent. Task boards, Product Backlogs, and Daily Scrums are all tools to make things visible to everyone on the team. Everyone should know where we're going, why we're going there, where we've been, and how things are going now. Most importantly, if there are issues that are slowing people down, or decisions that need to be made, or processes that are not working well, or anything that needs resolution or escalation--the goal in Agile is to make it visible quickly, so it can be remediated quickly. Hence a common saying is that Agile practices will not solve every problem, but will raise the visibility of problems that need to be solved. This is an important aspect, since many organizations suffer from problems that hamper their teams but are ignored or "swept under the rug".
Other practices focus on driving quality and long-term efficiency. These are more on the engineering side of the equation, but are an important part of Agile practice. A few examples include:
Design Patterns
Design Patterns are "best practice" approaches for building certain types of software solutions. Such best practices reflect a higher level of skill and quality in development, even for routine development tasks. For example, both a 3-star Michelin chef and a fast-food cook can cook eggs, but the quality between the two may be worlds apart. Just like the chef, the skilled developer following design patterns may take more time and be more methodical about their craft, but the results are worth it in the long run, in the form of higher quality code that is more flexible and less costly to maintain down the line.
Test-Driven Development
The idea behind test-driven development is to define up front specific tests the software should pass to verify that it is working correctly, then write code to expressly pass those tests. This is more efficient than a developer writing code to simply pass off to a tester, but the tester sending things back for rework after they find it fails certain test criteria. Why not expose this test criteria up front, and have the developer incorporate them into verifying their work is complete? An associated practice is Unit Testing, which is to build "self-diagnostic" checks into every piece of functionality a developer creates, to verify that it is working as intended. This is helpful because when parts of a system are modified, or new parts added, the introduction of new permutations can cause old parts to behave unexpectedly, or break altogether. This necessitates the need for extensive regression testing, or going back and re-testing existing functionality. This job can be reduced if Unit Tests exist, because they act as "warning lights", failing for parts of the system which no longer behave as they used to. This allows developers to quickly troubleshoot and repair the application.
Continuous Integration
When work is divided out across several developers, we think we are being efficient. The problem arises when it's time to join together pieces that have been worked on independently. One person's code might not integrate well with another's; it may break it, overwrite it, be too different in its design, or cause unexpected behaviors. If the combined code pieces don't work, we have to troubleshoot whose code is causing the problem. The longer we wait before we integrate the code, the more likely it won't work right the first time, and the more difficult it will be to track down the cause. Continuous Integration is the process of frequently integrating code to catch such issues while they're still small and easy to identify. For example, the code integration could happen daily, with whatever code developers have written for the day. The recommendation is actually to integrate code multiple times a day; this means each developer "checks in" small increments of code as they complete them and a "build" of the system with all the latest checked in code is performed every few hours. If the build fails, the software performing the integration notifies the developer who last checked in code, so that they know something in their code "broke the build" and needs to be fixed. Again, working in small steps helps capture problems early.
There are plenty of other practices, but they are variations on one of these themes--or at least they should be. In fact, this is how we should be thinking about Agile practices; not by thinking of them as "rules", but really understanding how they add to our efficiency, productivity, or the quality of our deliverable. That is--given our context.
Our assumption here is that we want to be able to quickly and efficiently accommodate to changing information or conditions. This is where "Agile" gets its name. The word agile is an adjective, not a noun. Its dictionary definition is "quick and well coordinated movement". In the context of software projects, we take it to mean "able to respond quickly and efficiently to change." Or in other words, the ability to adapt to changing circumstances without costing enormous time and effort.
This is ultimately how you determine if you're really Agile or not. Can you quickly and easily respond to change? Great! Feel free to describe yourself as "agile". Not there yet? Then you're really not "agile", no matter how many of the Agile practices you claim to follow or how many Agile books you've read. It boils down to measurable results.
What's the whole uppercase "a" about then? Well, its partly due to all this being seen as a movement--a "grass-roots" one--by individuals across the development community to improve the way we develop software. There are those who even think that the term "Agile" may someday fade into obscurity, as its ideas are accepted and adopted to the point where they become "business as usual".
There are also those, however, who have tried to make a little money by selling "Agile" as a packaged "product" to organizations. They come up with their own name, processes, practices, and even trademarks, insisting that they've done all the thinking for you and all you have to do is implement their out-of-the-box solution.
This is not necessarily bad, but it can give the wrong impression of what Agile is about. The fact is, there is no "one solution" for everybody. Everybody has different inefficiencies, different contexts, differing working styles, different cultures, and so forth. Now there are some practices that tend to be easily adapted to most situations; but they tend to flexible and principle-based, rather than prescriptive. A good example is Scrum, a set of popular Agile practices that are relatively easy to understand and implement, while providing some flexibility. Many organizations find success by mixing, matching, and tailoring known Agile practices to fit their unique situations. Who is to say you can't develop your own Agile practices? If it is maximizing your efficiency, productivity, quality, or ability to respond to change, then it's on the right track.
Thus, the many "flavors" of Agile that are out there. Still, it is very important not to stray from the fundamentals. In recent years, there has been concern in the Agile community that the popular focus on Agile has been too much on efficiency and productivity, and that people have forgotten about quality, which is equally fundamental.
This has led to the proliferation of what some call "Bad Agile", or "frAgile", which are implementations of Agile that yield short term results, but implode on themselves over the long run. What happens is that organizations move to a collaborative, iterative style of working, adopt a few Agile practices, and start churning out code at an unprecedented rate. The problem is they go too fast, because they don't pay enough attention to prioritizing quality every step of the way. As a result, several iterations after the first, residual quality problems start bogging the team down. Now the team has to spend half their time compensating for prior quality issues instead of focusing all of their time on new development. Going down this path can ultimately lead to worse performance than before the team tried to become "Agile".
Take a lesson from W. Edwards Deming: quality is the key to long-term and sustainable efficiency. Recognize the importance of culture in becoming Agile. This is another key reason organizations struggle with it. In order to give it a shot, you may have to ditch micro-management, silos, gatekeepers, and other rigidities. If you feel taken aback, consider that when US auto manufacturers hired Deming to show them how to beat the Japanese, Deming eventually gave up, citing that US company culture was too set in its ways to be able to adopt a better way of working. Don't be those guys.
Ultimately, Agile is a way of thinking. It's fundamentally simple, yet it also goes against our conditioned behaviors. Let me give you an example. We're all aware of what we should do to be successful and professional, right? We should be on-time, prepared, take notes, be organized, etc. But I'm sure we all know at least one person who regularly breaks the rules. We might even work for one. These are folks who:
Procrastinate until the last minute
Are messy and tend be somewhat unprepared and disorganized
Hold overly long meetings that go off topic
Spend a lot of time doing "busy work" that could be avoided with better planning and organization.
Etc.
They know what they should be doing, they just don't actually do it. This is another reason Agile can have difficulty taking off. Sad to say, but one or two key individuals who don't want to change their habits can derail the entire efforts of the group. This is especially true if they have built a "comfort zone" around the status quo.
I believe we all stumble in this area from time to time--including myself. For example, sometimes I:
Don't want to change my mind or my approach even when I see it's not working because I really committed myself to it.
Don't want anybody to see what I'm working on until it's "perfect".
Think that what I'm doing isn't valuable to anybody because it doesn't have all the things I want personally.
Try to do everything at once instead of focusing on a little at a time.
Repeat the same time-consuming routines of out of habit instead of devoting time to make things more efficient going forward.
The more you learn about Agile, the more you'll realize why such thoughts affect our ability to practice it. While this discussion of Agile could go on and on, my purpose here is simply to stimulate your thinking. I have observed that people often have serious confusions about Agile, and that the information they hear tends to be partial. If you believe Agile to be some complicated or even arbitrary "methodology" I can understand why you would be wary of it.
If you haven't done so already, I highly recommend reading the Agile Manifesto. The creation of this seminal document by members of the development community is considered one of the watershed events in the Agile movement. Make sure you don't miss the Twelve Principles of Agile Software. As you read them, think about what kind of inefficiencies and needs the writers might have faced that led them to these principles.
Don't feel bad if there are still parts of Agile you don't fully grasp. Not only does it take a while to learn, there is always something new to learn. I'm learning new things myself all the time, and I've been studying Agile for over ten years. I expect the longer I keep at it, the more distinctions I'll make. Hopefully I've given you a few insights you can use to think through and understand Agile yourself. It's not a fad after all--spread the word!