Agile Development
Introduction
The Agile methodology is probably the most well known and widely used software development methodology. It is an iterative approach to software development that emphasizes collaboration between cross-functional teams and the ability to respond to change quickly. It is more adaptable than earlier methodologies like the Waterfall model, which is a more linear approach to software development.
Waterfall Methodology
The Waterfall methodology was derived from manufacturing processes used to create large military and industrial systems. It is a linear, sequential approach to software development that is divided into distinct phases, where each phase must be completed before moving on to the next phase. Being designed more for physical engineering, it maps poorly the design engineering of software development.
Special Cases for Waterfall
There are some special cases where the Waterfall methodology is still a reasonable choice.
- Embedded Systems: In embedded systems, the hardware is often fixed and the software must be designed to work with that hardware. It might even be impossible to change the software once it has been deployed.
- Safety-Critical Systems: In safety-critical systems, the software must be designed to be as reliable as possible. This often means that the software must be thoroughly tested before it is deployed. Things like proofs of correctness, code analysis, and detailed security audits all require a finished specification to be effective.
- Regulated Industries: In regulated industries, the software must be designed to meet certain standards. This often means that the software must be thoroughly documented and tested before it is deployed.
- Cross Company Projects: If parts of your project are being built by outside contractors who are hard to contact, it may be easier to specify everything up front. This is especially true if the project is being built by multiple companies in different countries.
These cases are a tiny minority of software projects, but they do exist. The waterfall methodology should never be your default choice, but it is a reasonable choice in some cases.
Unfortunately, many companies still use the Waterfall methodology for all of their projects. This is despite there being clear evidence that it produces lower quality software, is more expensive, and takes longer to deliver. The Agile methodology is a better choice for most software projects.
Why is Waterfall Still Used?
The main reason that the waterfall methodology is still used is that is makes so much sense on paper. There are clear timelines and a clear separation of responsibilities. Of course it makes sense to gather requirements before you design the system. Of course it makes sense to finalize the design before you start coding. Of course it makes sense to finish implementation before you start testing.
In an ideal world, the waterfall methodology would be the best choice. Unfortunately, the real world is not ideal. The waterfall methodology is based on the idea that you can know everything about a project before you start. It relies on humans making no mistakes in order for the project to be successful. It is a fragile methodology that is easily broken by the real world.
With this model the most important decisions for a project are made at the beginning of the project. This is when the requirements are gathered and the design is created, but this is when we have the least information about the project.
No matter how skilled your team is, no matter how much experience they have, they will make mistakes. Even if they communicate perfectly with the users, there will still be misunderstandings simply because the users do not know what they want. There's no way to make this approach work consistently in the real world.
You will always end up returning to an earlier phase of the project to fix a mistake. This is called rework, and it is the most expensive part of any project. The longer you wait to find out that you have made a mistake, the more expensive it is to fix that mistake. The waterfall methodology is designed to hide mistakes until they are so costly that they cannot be fixed. If rework is inevitable, then you should plan for it from the beginning.
Agile Manifesto
History
In the 1990s, developers were becoming increasingly dissatisfied with the prevailing heavyweight software engineering methodologies. These waterfall style approaches focussed too much on planning, structure, and contracts. Developers wanted methods that allowed them to spend more time creating things and began experimenting with more flexible approaches.
A number of software engineering books and papers were published during this period that laid the groundwork for the Agile methodology. These books included:
- 1991: "Rapid Application Development" by James Martin
- 1994: "Unified Software Development Process" by Ivar Jacobson, Grady Booch, and James Rumbaugh
- 1994: "Dynamic Systems Development Method" by the DSDM Consortium
- 1995: "Scrum Development Process" by Ken Schwaber
- 1996: "Extreme Programming Explained" by Kent Beck
- 1997: "Feature Driven Development" by Stephen Palmer and John Felsing
All of these approaches were trying to do the same thing, to flatten the cost of change. They wanted to reduce the cost of changes late in the project. They all recognized that the best way to reduce the cost of change was to find out that you have made a mistake as soon as possible.
In 2001, a group of 17 software engineers met at a ski resort in Snowbird, Utah to discuss more lightweight, flexible, and efficient ways to develop software. They published the Agile Manifesto, which outlined the core values and principles of the Agile methodology. The manifesto itself is quite short. It identifies four core values and twelve principles that are intended to guide the development of software.
Core Values
- Individuals and Interactions over Processes and Tools
- Working Software over Comprehensive Documentation
- Customer Collaboration over Contract Negotiation
- Responding to Change over Following a Plan
To quote the manifesto:
"While there is value in the items on the right, we value the items on the left more."
It's critical to note that the Agile methodology is not suggesting that you should ignore the items on the right. Those items are still important, but they are not as important as the items on the left. The authors recognized that the prevailing emphasis on the items on the right was causing problems, and that the items on the left were being neglected.
Agile Principles
In addition to the core values, the Agile Manifesto also includes twelve principles that are intended to guide the development of software. These principles are:
1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
There are two important parts to this principle. The first is that the customer is the most important part of the project. The second is incremental delivery. Customers should receive project deliverables that provide value throughout the project's lifecycle and not just at the end.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Changing requirements are more than just inevitable, they are necessary for the project to be successful. The project team should be able to respond to changes in requirements quickly and efficiently. You'll hear developers complain about scope creep throughout your career, but not all scope creep is bad. Sometimes the customer only realize what they want after they experienced a partial system, and sometimes the business landscape changes so quickly that the project needs to change with it.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Frequent deliverables mean that you get faster feedback, faster bug fixes, and better software in the long run. Most successful large software companies have increasingly embraced this approach and many are capable of releasing multiple times per week, or even per day. This is not just about getting the software to the customer faster, it's about getting feedback from the customer faster.
4. Business people and developers must work together daily throughout the project.
Having the developers and the business experts siloed from each other will lead to the creation of something that no one wants. The developers need to understand the business needs and the business needs to understand the technical limitations. This is not just about having a good relationship between the two groups, it's about having them work together on a regular basis.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The agile methodology is very anti-micromanagement. It is based on the idea that the people who are doing the work are the best people to decide how to do the work. This is not to say that the developers should be left to their own devices, but that they should be given the freedom to make decisions about how to best accomplish their goals. Their basic idea was that if you got the right people in the right positions, they would be able to figure out how to get the job done.
6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
This one feels a bit dated in the age of COVID-19, but the basic idea is still sound. Video calls are better than phone calls, phone calls are better than emails, and emails are better than documents. The more direct the communication, the less likely there are to be misunderstandings. By removing the barriers to communication, you make misunderstandings less likely. Most studies that I've seen have shown that the difference in efficiency between face-to-face and video calls is not that large, but the difference between video calls and anything else is huge.
7. Working software is the primary measure of progress.
The goal of any project is to deliver working software. Everything else is secondary. This is not to say that documentation, planning, and other things are not important, but that they are not the goal of the project. This also means that measuring team productivity by lines of code, hours worked, or any other metric that is not directly related to working software is a terrible idea. These don't measure progress, they measure activity.
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Overworked developers are bad for everyone. They might burn out, they might quit, or they might just make mistakes. The goal of the project is to deliver working software, and the best way to do that is to have a team that is able to work at a sustainable pace. You want to build a repeatable cycle of work that can be maintained indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
The team's goal needs to be focussed on improving the final product and delivered those improvements consistently over time. High quality, maintainable, extensible, and testable code is what teams need in order to maintain their speed. Better designed and maintained software is easier and faster to expand. The simple fact is that by focussing on quality, better software gets made in less time.
10. Simplicity - the art of maximizing the amount of work not done - is essential.
Any process or effort that does not directly contribute to the final product should be avoided.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
Empower teams to structure and organize themselves and let them collaborate and innovate as they see fit. The impact of this approach has been clearly demonstrated in the State of Devops reports year after year. Independent teams, that don’t need approval from outside, get more work done with fewer bugs.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
A successful team needs to regularly focus on improving their processes and practices. This is not just about fixing bugs, but about improving the way that the team works. If we aren't taking a critical look at our approach, how can we identify flaws or find better ways to work?
Agile Overview
That was a lot of theory, so let's narrow it down to the key points.
Incremental Improvement
With Agile, you are constantly improving both the product itself and your understanding of the requirements. At the end of each increment, you should have a working product that can be shown to the customer. This allows the customer to provide feedback on the product, which can then be used to improve the product in the next increment.
This allows for a fail fast approach, where you can quickly identify and correct mistakes. If a requirement is wrong, we'll know. If a technology choice isn't viable, we'll know. If a feature isn't useful, we'll know. We get to identify our mistakes without having to wait until the end of the project.
Efficient Communication
Agile methodologies emphasize communication between team members and stakeholders. This ensures that everyone is on the same page and that the project is moving in the right direction. If a developer has a question, they can ask it. If a stakeholder has a concern, they can voice it. If a team member has an idea, they can share it. Streamlining communication is right at the heart of Agile.
Quality Focus
In order to maintain velocity teams need to never compromise on the quality of the software they are delivering. Most tools and techniques that have become associated with Agile development are motivated by this need for maintaining quality. Things like unit testing, continuous integration, test driven development, pair programming, design patterns, and regular code refactoring are all intended to enhance product quality.
If the complexity of the system gets out of control, then development will slow down.
Agile Advantages
- Continuous Customer Contact: The customer is directly involved in more of the decisions, ensuring that the product meets their needs.
- More Adaptable to Change: Because we're working in small increments, we can change direction quickly. Adding new features, changing requirements, or fixing bugs is easier and less disruptive.
- Faster Delivery: By working in small increments, we can deliver working software faster. Every study confirms incremental approaches are faster than waterfall.
- Lower Project Risk: Even if we make a major mistake, we've only lost a small amount of work. If for some reason the project turns out to be a bad idea, we will find out sooner and be able to stop the project before too much time and money has been spent.
Even if you follow all of these recommendations exactly the process still won’t guarantee success. A significant portion of all projects end up being scrapped because they failed to provide any real value. The advantage Agile provides for these cases is the ability to identify those failures earlier in their lifecycle.
Not every team, company, or project should adopt Agile. There are cases where another approach should be chosen.
- The project is small and well understood: If you know exactly what you need to build and it's not going to change, then Agile is overkill. This case is rare, but it does exist.
- The stakeholders don't want Agile: If the stakeholders don't want Agile, then Agile is not going to work. This methodology requires a high level of trust and collaboration between the team and the stakeholders. If that trust and collaboration doesn't exist, then you have to pick something else.
- The company can't support Agile: This might happen because of insufficient training, lack of support from management, or a culture that is not conducive to Agile. Perhaps there are significant barriers to communication due to time zones, language barriers, or other factors. Or perhaps the organization is subject to regulations that make Agile difficult or impossible.
Adopting Agile
There are a lot of different ways that an organization can adopt Agile. There are also a lot of wrong ways to adopt Agile. The most important thing to remember is that Agile is not a set of rules that you can just follow to get a successful project. It is a set of principles that you can use to guide your project. You need to understand the principles and then figure out how to apply them to your specific project.
Some groups have tried to add structure to Agile in order to make it more palatable to organizations that are used to more structured methodologies. Most of these are not very good. They tend to add a lot of overhead to the process and make it harder to get the benefits of Agile.
The Scaled Agile Framework (SAFe) is the most popular of these, but it is also the most criticized. Its creators had the best of intentions, but they ended up creating a methodology that is too complex and too rigid to be Agile. It sounds good to business leaders, but it doesn't help teams to develop software more efficiently.
I've met a number of engineers who simply hate Agile, and consider it a failed methodology. Some common complaints are:
- The process is painful and bureaucratic: This complaint usually comes from Agile being incorrectly applied. Agile is supposed to be lightweight and flexible, but many organizations have added a lot of overhead to the process. This is not Agile's fault, it's the fault of the organization.
- How can you plan your software, communicate deadlines, or estimate costs?: This is a legitimate concern, and should not be dismissed. No one likes hearing "I don't know" when they ask when you'll be done. Agile is not a silver bullet, and it doesn't solve all of the problems of software development. It is a methodology that is designed to help you develop software more efficiently, but it is not a magic wand that will make all of your problems go away. But consider this: if every study says taking this approach is faster, then maybe not knowing the exact date is a price worth paying.
- Agile is idealistic and doesn't work in the real world: I find this also tends to come from Agile being incorrectly applied. When Agile is applied correctly, it works very well. When it is applied incorrectly, it doesn't work at all. The problem is that it is very easy to apply Agile incorrectly. However, it has been my experience that trying and failing with Agile is still better than any other methodology.
Scrum
There are many developers with the mistaken impression that Scrum and Agile are the same thing. This is mostly the result of a successful marketing campaign by the Scrum Alliance. Thanks to their efforts, Scrum has become the most popular approach to Agile. Think of Scrum as a framework for adopting Agile rather than Agile itself. Agile would still exist without Scrum, but Scrum would not exist without Agile.
Scrum is popular for a number of reasons:
- It is easy to understand and implement.
- There are well established training programs and certifications.
- The structure makes it appealing to business leaders.
Scrum Basics
The basic approach for Scrum is to organize development into a series of short increments known as Sprints. Each Sprint is typically one to four weeks long, and at the end of each Sprint the team delivers a working product. The team then reviews the product with the stakeholders and uses the feedback to plan the next Sprint.
I've found two week sprints to work best for me. One week sprints always feel rushed, and four week sprints always feel like they drag on with dead time at the end. Two weeks is long enough to get some real work done, but short enough that you can quickly change direction if you need to.
Your mileage may vary, but I would recommend starting with two week sprints and adjusting from there.
Each sprint is a sort of mini waterfall project, with all of the phases of the project being completed in that time frame. You are planning, designing, coding, testing, and deploying in each sprint. This is a big part of why Scrum is so popular. It is a very structured approach to Agile that is easy to understand and implement.
Scrum Roles
There are three main roles in Scrum:
- Product Owner: The person responsible for defining the features of the product and deciding on the release date and content. They own the vision of the product and are responsible for communicating that vision to the team.
- Scrum Master: The person responsible for ensuring that the team is following the Scrum process and removing any obstacles that are preventing the team from being successful. They help to facilitate communication and resolve conflicts.
- Development Team: The group of people responsible for doing the actual coding work. The ideal size for a development team is between five and nine people. The team should be self-organizing and cross-functional, meaning that they have all of the skills necessary to deliver the product. They should be as independent as possible. If they need to get approval from outside the team to make a decision, then that will slow progress.
Meetings
Scrum is organized around a series of meetings that are intended to keep the team on track and ensure that everyone is on the same page. Note that this is the main complaint that I hear from engineers about Scrum. They feel that there are too many meetings and that the meetings are not productive. This is a legitimate concern, and it is something that you need to be aware of when you are adopting Scrum. The meetings are intended to be short and focused, but they can easily become long and disruptive without careful management.
The main meetings are:
Sprint Planning
For each sprint the team will start with a Sprint Planning meeting where the team gets together to decide on the work to be done in this sprint. The backlog of work should already be prioritized and estimated, so the team can decide on the work to be done based on the priority and the teams capacity. Sometimes the product owner will need to answer questions or provide clarification, but the goal is to get the team to commit to the work that they will do in this sprint.
The total amount of work to be done in the sprint is determined by the team's historic productivity. Despite what some managers seem to think, the goal should not be to constantly increase the amount getting done or to work anyone into the ground. Consistent delivery is the goal.
Backlog Refinement
The product backlog is a list of all of the work that needs to be done on the project, or at least all that work that is known at this time. Each item will more the project forward in some meaningful way. New items will be added when the product owner creates user stories, or when bugs are reported. Developers might also add items when they find technical debt or other work that needs to be done.
The product owner is responsible for maintaining the backlog, but the team is responsible for estimating the work and breaking it down into smaller tasks. This is an ongoing process that should be done throughout the project. Adding new items, adjusting priority, or adding detail to existing items should be done as needed.
The goal of the backlog refinement meeting is to do any manipulations of the backlog that require the team to be present. This might include breaking down large items, estimating new items, or major re-prioritization. If the only time the backlog is looked at is during refinement meetings, then the team is not doing enough refinement and the meetings are likely to be long, frustrating, and unproductive.
Many teams treat backlog refinement as a one hour meeting that happens once a week, or perhaps once per sprint. This approach tends to fail badly in practice. The backlog should be refined as needed, and the team should be able to do this without a formal meeting. If the team is not able to do this, then meetings will need to be scheduled more frequently, making them a drain on productivity.
The main activity in the backlog refinement meeting is to estimate the work that needs to be done for each item in the backlog. Exactly how this is done will depend on the team. They might use story points, hours, t-shirt sizes, or some other metric. Every team has their own approach, so plan on adapting to the team's approach during onboarding.
No matter which approach is used, the goal is to get a rough idea of how much work each item will take, with an emphasis on rough. Accurate estimates are not possible, so don't waste time trying to get them. The goal is to get a rough idea of how much work each item will take, so that the team can decide on the work to be done in the sprint.
I go into more detail on estimating in the Estimation section.
Daily Standup
The daily standup is a short meeting held each day. It is called a standup because the idea is that the meeting should be short enough that everyone can stand for the duration. Each team member should answer three questions:
- What did they work on yesterday?
- What are they planning to work on today?
- Are there any obstacles that are preventing them from making progress?
That's it. The goal is to maintain coordination between team members and to identify any obstacles that are preventing progress. The meeting should be short and focused, and should not be used to solve problems. If a problem is identified, it should wait until the end or be taken offline and resolved outside of the meeting in a separate discussion called a sidebar or parking lot.
Don't let the meeting get bogged down in details. If a team member starts to go into too much detail, the Scrum Master should step in and remind them to keep it short. If a problem is identified, the Scrum Master should make a note of it and follow up with the team member after the meeting. I've had to call out team members and even managers for getting off track in standups. It's not fun, but it's necessary to avoid wasting time. Every team I've worked with has appreciated the shorter meetings.
There are a number of difficulties with virtual standups. The most common is that they tend to be longer and less focused than in-person standups. This is because it is harder to read body language and harder to interrupt someone who is going on too long.
Some teams have chosen to move to a text based standup, where each team member writes their answers to the three questions in slack or some other chat program. I consider this to be a waste of time. The temptation to gloss over everyone else’s status is just too high for most developers to resist, and you end up with a scenario where only the scrum master and product owner know what the team is doing.
Some team's have gone in the opposite direction with this meeting and leave the team Zoom call open for the entire morning and use it to socialize and build team comradery. I’ve heard some positive opinions about this approach, but it just sounds like torture to me.
Sprint Review
At the end of each sprint the team will hold a Sprint Review meeting where they get together with the project stakeholder to demonstrate the progress that has been made. The goal is to get feedback on the work that has been done and to decide on the work to be done in the next sprint.
A secondary goal is to familiarize developers who didn't work on the feature with the feature. This is important because it allows the team to be more flexible in the future. If a developer who worked on the feature is out sick, then someone else can pick up the work.
Sprint Retrospective
The last meeting that characterizes Scrum is the Sprint Retrospective which is held at the end of every sprint. The purpose of the meeting is to discuss the Sprint process and not the product. The team reviews what went well during the sprint in terms of collaboration, planning, process, and tools. They also discuss what didn't go well during the last sprint and what the team can do differently to prevent similar issues in the future. If the daily standups got sidetracked with off topic discussions for example, this is the time to bring it up.
It's important to remember that all retrospectives should be blameless. The purpose of this meeting is never to assign blame for some issue, it is just focussed on how to prevent similar problems in the future. Be proactive in acknowledging your own failures and always assume your fellow developers are acting with the best intentions. This approach ensure that mistakes are addressed before they can compound into system failures.
Scrum Overview
Scrum is a popular methodology for a reason. It is easy to understand and implement, and it is very effective at delivering working software quickly. It is not without its flaws, but it is a good choice for many projects. It emphasizes a cycle of planning, executing, and reviewing that is very effective at delivering working software.
It loops the software development lifecycle phases. Each sprint delivers meaningful progress, before resetting and starting the cycle again. This is a very effective way to develop software, and it is the reason that Scrum is so popular.
There are some downsides to Scrum. The biggest one is probably the number of meetings. If the meetings are not managed well, they can seriously hamper productivity. Even if the meetings are kept short and focused, they can still be a drain on productivity if not timed well. It takes time for developers to get into the flow of coding, and if they are constantly interrupted by meetings, they will not be able to get as much done.
Kanban
Kanban is a lightweight alternative to Scrum that is often used in Agile development. It was first used by Toyota in the 1940s as part of their production process. It is a visual system for managing work as it moves through a process. The goal is to limit the amount of work in progress and to identify bottlenecks in the process.
Kanban Board
The most important part of Kanban is the Kanban board. This is a visual representation of the work that needs to be done. The board is divided into columns that represent the different stages of the process. Each item of work is represented by a card that moves through the process from left to right. The exact set of columns will depend on the team, but the most common columns are:
- Backlog: This is the list of all of the work that needs to be done. It is not prioritized, and it is not estimated. It is just a list of work that needs to be done.
- Ready: This is the list of work that is ready to be done. It is prioritized and estimated, and the team can start working on it at any time.
- In Progress: This is the list of work that is currently being worked on. The team should limit the amount of work in progress to avoid bottlenecks.
- Done: This is the list of work that has been completed. It is important to move work to this column as soon as it is done to avoid confusion.
- Blocked: This is the list of work that is blocked for some reason. The team should work to unblock this work as soon as possible.
Kanban Principles
Kanban is not intended to be a precisely defined process like Scrum. Instead, it can easily be adapted to meet the needs of your company, team, or project. The core principles of Kanban are:
- Visualize the Workflow: The team needs to map out their workflow and break it down into its core stages. Work items are represented as cards and all team members are aware of what stage each item is currently in. This visibility improves communication and collaboration, as well as helping to identify bottlenecks in the process.
- Limit Work in Progress: If teams have too many tasks in progress at once, there tends to be more context switching and higher cognitive load. This can lead to lower quality work and longer lead times. By limiting the amount of work in progress, teams can focus on completing tasks and delivering value.
- Focus on Flow: Flow refers to how an item moves through the process. With good flow, items more steadily and predictably through the process. By focusing on flow, teams can identify bottlenecks and work to remove them.
- Continuous Improvement: The team should work to continuously improve the process. This might involve changing the process, adding new columns to the Kanban board, or adjusting the work in progress limits. If you aren't measuring your process, you can't improve it.
Kanban is an extremely lightweight and flexible process. It can be great for teams that are comfortable with the approach. It also has low overheads in terms of time spent on non-development tasks, which can make it very appealing. The lack of structure is really Kanban's biggest weakness in many environments. It doesn't smoothly handle incremental delivery and the lack of timelines associated with the phases can make planning difficult.
I use a Kanban board for all of my personal projects. It's a great way to keep track of what I'm working on and to make sure that I'm making progress. I keep a separate digital board for each category of work that I do, and I move tasks along the board as I work on them.
My current set of boards are:
- Teaching: This is where I keep track of grading, lesson planning, and other teaching related tasks.
- Hobby: I do a lot of painting, modeling, and woodworking projects, so I keep track of what I'm working on here.
- Work: I keep a board for my general work tasks. This doesn't include any software development tasks, those are tracked in Jira. I use this for things outside of my normal work tasks, like continuing education, mentoring, and general support tasks.
Conclusion
Agile is the most popular software development methodology in use today. When applied correctly it is a powerful tool for delivering high quality software quickly. When applied incorrectly it can be a frustrating and bureaucratic mess. The key to success with Agile is to understand the principles and to adapt them to your project.