The Agile Methodology
The Agile methodology offers an alternative to a traditional project management. It is used mostly for the rapid development of software and contrasts with the traditional Waterfall. software development approach which is more suited to engineering and construction. It really useful where there is uncertainty and ambiguity around what is required and allows a complex solution to evolve iteratively. It is great for pure software development and for the development of configurable systems such as CRMs or ERP systems.
For more discussion on the Software Development Cycle or the Waterfall approach please see the information at the bottom of this article.
Scrum
Scrum often mentioned in the same breath as Agile and is probably the most flexible and popular Agile approach. Scrum is about self managed development teams who work rapidly and efficiently together, incrementally building properly tested functionally over very short timescales. It is a way of dealing with complex work and breaking it down into manageable chunks.
As with all Agile approaches it focuses on early product delivery, often in a minimal viable form. This means a basic functioning solution can be rapidly built and brought into service instead of trying to achieved a large release all in one go. Incremental improvements are then added and these can continue indefinitely if necessary. The idea is frequent releases or ‘iterations’ rather than a single all encompassing project i.e. think it, build it, ship it, tweak it.
Scrum was developed by by Hirotaka Takeuchi and Ikujiro Nonaka as is a cross functional ‘Lean’ approach and takes the term Scrum from Rugby, the idea is that the team are working together like a rugby scrum.
Sprints
Sprint is a Scrum term and also referred to as an ‘iteration’ and are ‘time boxed’ mini-projects of 2-4 weeks duration. Each Sprint contains independent analysis, design, implementation, testing and planning of next steps. The sprint length is defined by the team who commit to delivering the agreed scope in this time period. The recognised industry standard is about 2 weeks, which means that a new sprint should be starting approximately every two weeks.
Scrum is a flexible methodology but not within the sprints themselves, these are fixed. Once defined the sprint scope does not change, any changes required will need to go onto the Product Backlog (see below) and a future Sprint. This is important because disrupting the teams efforts wastes time, is demoralising and breaks the principles of Scrum.
The idea of the Sprint is that within the Sprint period a usable and shippable product is produced (ready to release but possibly not installed). Note that it is not the whole solution that is being delivered but a standalone element of the solution .
Artifacts
- Product Backlog
The Product Backlog is a list of requirements or products that comprise the programme, this can include wishlist items as well as hard requirements. These are managed and prioritised by the Product Owner (see below). Typically the product backlog items are written as a User Story or Use Case scenarios. Items at the top of the backlog catalogue should have a detailed user story, whereas those of lower priority just need an intelligible description, as they are unlikely to be worked on for some time.
There are different categories of backlog items and these include user stories, bugs, chores (essential for operation), Epics (big stories containing themes) and prototypes, features, themes and spikes (research task – maybe to understand the story better).
The story needs to contain everything necessary to make the solution usable including such things as a user interface, back end functionality database work, security, regulatory compliance etc, with a restricted amount of functionality included. The scope needs to be kept small so that it can be contained within a a sprint and may be a subset of the overall deliverable. An example is adding the address information rather than all personal information, which can be expanded on later.
Prioritisation is normally in terms of the most valuable and essential items first. If it is on the backlog it may get done but if it is not will not be included in the project. The backlog is visible to everyone and anyone is entitled to add to the backlog. At this stage products in the backlog should be viewed as an opportunity and not a commitment to deliver.
- Story
The Story is what needs to be achieved and what would a good solution look like from a user perspective. It is written in a conversational style e.g. “As a customer I want to be able to add my address so that I can register it on the website”. By keeping a story open the Sprint Team are able to be more innovative in how they develop the solution.
The user story should be independent, negotiable, of value, sized right and stable. These stories are generally written on cards 75 x 125 mm with a pointed marker pen. The story should be concise and to the point and fit on the card. The acceptance criteria for the user story is generally written on the back of the card. The user story is not a commitment but a promise of something to be further refined later. Creating the user story should not take more than a few days or it is likely to be too big.
- Team/Sprint Backlog
The Sprint Backlog is a list of work that has been been committed to for the current Sprint i.e. it is a subset of the product backlog and represents the products that will be delivered in the defined Sprint period. The Backlog products are then broken down by the team into individual tasks (these are sub-steps of the story). They are normally tracked to completion on a visual management as ‘Not Started’ In Progress’ and ‘Completed’
Roles
- Product Owner
The Product Owner represents the stakeholders and is responsible for return on investment through prioritisation of the product backlog. They hold the product development vision and act as the decision maker on requirements and focus on what needs to be done. They should be engaged with the Sprint Team on an ongoing basis, but not interfere in their work.
- Scrum Team
The Scrum Team is a cross functional team held accountable for delivering shippable product increments in time boxed periods called Sprints. They collaborate closely together as a team to deliver the product. Ideally they would be used to working together, but as a minimum should be able to work as ‘team players’.
There are no external reporting lines into this team and they effectively operate autonomously but in an aligned way. The Scrum team are typically a mixed team of users, BAs developers and testers all working closely together in the same location as one cross functional and self managing team. Typically the team is between 4 and 9 people in size and very often operate outside a 9:00- 17:00 routine.
- Scrum Master
The Scrum Master is usually a technical individual not a manager and certainly not a project manager, in fact they are not there to manage at all, but to act as a facilitator for the team. They are responsible for the execution of the Sprint and handle issues outside the daily Scrums. The Scrum Master protects the team from distraction and interruption and deals with issues, often they also provide technical guidance and keep an eye on the Sprint progress.
Meetings
- Sprint Planning Meeting
The Sprint Planning Meeting is typically a half day where the Product Owner and team prioritise and logically sequence Backlog Products.
The purpose of the sprint is to provide something that is’good enough’ that can be added to later, this is a principle of Scrum. The intention is not to ‘gold plate’ the solution because this is extra work and of questionable added value. Refinement can be done later, remember this is an iterative evolving process and the key is to get something usable up and running quickly and dynamically.
The meeting starts with the Product Owner coming to the meeting and telling the team what he/she would like then to do next and discuss it further. They then leave the team to continue their planning and to avoid influencing the team in any way. The team may also choose to go over the Story with users and discuss the finer points of the the functionality they would like to see in a separate meeting. This meeting would be ‘owned’ by the users. Note stakeholders only participate if invited and members of other sprint teams may also be invited to attend to discuss dependencies.
Facilitated by the Scrum Master the team plan the Sprint, estimate the amount of work that needs to be completed and commit to which products (stories) will be included, based on priority and synergy. They then break the products down into tasks which can be distributed within the team. The team votes on what they have committed to within the sprint period, if the confidence level is low then some products may need to be taken out again. This is an important decision point because the team is committing to delivering all the committed products within the time period.
The team estimates using Story Points and these these are used to rank the stories in terms of size and complexity, with all subsequent stories measured relative to the first one. Points cards can be purchased to assist this process. The team also calculates the capacity which is how much work the team can fit into the sprint including meetings, days off and an allowance for issues (10-15%). They also need to consider legacy implications of what is being done e.g. they may spend 50% of time on new work and the rest of the time on technical architecture, refactoring existing code and preventing building of technical debt issues.
The team define the acceptance criteria (definition of done) e.g. the sprint must be code reviewed, tested, QA reviewed, Product Owner approved.
On completion of the planning the team presents the commitments to the Product Owner who can reprioritise if he/she feels that a more critical item needs to be included instead of other functionality.
- Daily Scrum
This is an official 15 min daily morning stand up meeting. Scrum team members update each other on what they did yesterday, what they will do today, what they plan for tomorrow and any issues that need resolving (if a team member is missing the Scrum Master stands in for them).
Tasks and their status are normally posted and updated on a board by the owners, so that these can be used as a point of reference and discussion. Note there is no reporting to a higher authority and no one is ‘Boss’.
Issues and resolutions are discussed as a team. The meeting length is ‘time boxed’ so bigger issues must be discussed offline between the impacted members.
- Scrum of Scrums
If the sprint forms part of a wider programme a Scrum of Scrums will be required. A 15 minute daily or as a minimum weekly meeting should be attended by the Scrum Master or other Sprint Team representative. It is run on the same basis as the Scrum Meeting but at a programme level and to a summarised level of detail. Focus is mainly on collaboration, dependencies and issues e.g. the web server is down and slowing development.
- Sprint Review
This is where the potentially shippable solution is demoed, research is presented or other deliverable is shown to the Product Owner, stakeholders and users. This is the stage feedback is provided to the team. It is interesting to note that reality users often need to see the wrong product before they can advise on what they really need, this is OK and makes the development process much more rapid. Any new requirements are added to the Product Backlog for later enhancement.
The Product Owner (who represents the customer) will decide which items are complete and which did not meet acceptance criteria. Because of ongoing dialogue they should already know what has been achieved and any issues.
Completion of the Sprint should be fun and it is important to ‘Celebrate Success’ in some way e.g. team have coffee and doughnuts, mine’s a sugar glazed crispy creme.
Note that the Scrum of Scrums above will also have its own Programme Review similar to the Sprint Review where each team will present to each other. This ensures all teams get the ‘ bigger picture’.
- Sprint Retrospective
This is a debrief of what went well/not so well in the Sprint and how could things be done better next time. It is a learning experience for the team and may result in some items actually being added to the backlog. The idea is to bring continuous improvement into the Sprint process. This is often referred to as Kaizen which means continuous improvement in Japanese and is a term borrowed from manufacturing.
Only Scrum Team members should attend the retrospective. It can be offsite and also be used as a team bonding exercise.
- Backlog Grooming
Review candidates for future Sprints in the context of the current understanding.
Controls
- Metrics
Scrum metrics are things like the number of sprints, sprint cycle time, stories committed to in the sprint, team capacity in terms of work it can do and velocity.
Velocity is a particularly important concept and relates to how much work is being done in each Sprint. It is measured in terms of Story Points completed. This is really useful because if we delivered 15 points in the last Sprint we can be pretty confident that this is our capacity for the next sprint. As the programme progresses an average velocity can be calculated and this can be used as an ongoing benchmark. Note that the velocity is only calculated for stories that were accepted as complete by the Product Owner.
Examples of sprint Metrics might be:
- Velocity – Planned and Actual
- Number of stories – planned and accepted
- Quality – Unit Testing percentage, number of tests completed, number of defects, refactors
Note: metrics are also required for the programme and these are usually an aggregate of the Sprint team metrics. e.g. the Programme velocity will be the total velocities of all of the Sprint Teams.
- Burndown charts
Burndown charts plot Story Points remaining against days on a line graph and show how much time is remaining to complete the work required. It is shows the burndown of the story points burning down to zero. The trend allows us to evaluate whether a Sprint will deliver on time or if there are issues.
At programme level a similar line chart can be created that plots actual and planned Story Points remaining against iterations (Sprints) using a line graph. This shows progress of the whole programme against what was planned. Again potential overrun can be evaluated using the actual trend line.
- Task Tracking
Task Tracking is usually presented as ‘Post It’ notes on a Visual Management Board.
- Feature Progress Report
The Feature Progress Report shows current progress across multiple stories and is typically updated daily using ‘story completed’ status. It shows how many stories are completed v what was planned in bar graph format and can be ‘traffic light’ coded to show status.
- Predictability
Predictability is a measure of whether Sprint Team delivered within expected bounds e.g. were 80% of stories signed off within each sprint. This can be presented in a line graph format and can be used at programme level to show team predictability. Ideally the results should be within a tight band. Big changes in performance mean that the team’s estimating is unpredictable and cannot be relied upon, but with a new programme is something that will improve over time. Predictability means that status reporting can be provided within a predictable level of confidence.
The Software Development Life Cycle (SDLC)
The software development cycle has the following elements
- Initiation – define the need or opportunity
- Project Definition – define the scope, cost benefit, risks, planning, resources
- Requirements Analysis
- User Requirements – functional requirements specification – use cases
- System Requirements
- Analysis and Design – transfer detailed requirements into a complete systems design. Define features and operations in detail. This can include screen layouts, business rules, process diagrams, pseudocode and other documentation.
- Development – convert systems design into a system including coding, environment design and databases. Includes test cases and procurement of components or services
- Integration and Testing – run test cases to test code and integrations work correctly and test that the system fully meets the requirements specification – testing independently performed by QA and usability checked via UAT – performed in a segregated testing environment
- Implementation and Training – implementation into the production environment – follow a cutover plan
- Operations and Maintenance – post installation review, initiate support tasks to operate and maintain the system – write documentation to support the system or change, handle upgrading and patching etc
- Decommissioning – proper discarding of data (or archiving), hardware and software usually after transition to a new system
Both Agile and Waterfall execute to this life cycle except that Waterfall does this sequentially whereas Agile executes this in chunks called iterations or in Scrum speak ‘Sprints’.
*Waterfall Overview
A Waterfall project is similar to an engineering project like building a bridge. Each stage has a formal start and end date and is normally managed using traditional Project Management techniques.
It is easy to implement and manage, and is great if you know exactly what is required and has its place for certain types of software project e.g. a data migration. However, it relies heavily on being able to have complete knowledge of all the requirements up front. This is very seldom the case and makes this approach difficult to execute in practice. It is especially difficult when the development is breaking new ground, which often results in a sub-optimal solution that requires substantial rework.
Each phase is held by different teams and there are potentially significant handovers which means everything has to be documented. There can be difficulties in communication between project phases and there are lots of potential failure points in the process. Changes are very difficult to achieve especially of they lie at the core of the development, because everything is based on the originally highly detailed specification and a significant rework may result. Large waterfall projects are often characterised as not living up to expectation, being delivered very late and way over budget or or failing completely.
Agile v Waterfall
In Agile the development process is cyclical so that changes and tweaks can be included in the development before the whole solution is released rather than waiting to the end. It should be noted that Agile is still a planned approach, although it is iterative and flexible in nature.
One of the big aspects of Agile delivery is that benefit is delivered to the customer throughout the project. More and more functionality is available to the business as each iteration or Sprint is delivered. Value actually being delivered from an early stage is also good for the overall business case.
With Waterfall nothing is available until the very end when everything becomes available in a big bang. Agile is far more evolutionary and allows for an ongoing path way instead of trying to achieve everything in a single step change. With agile the business is engaged the whole time and because new functionality is being made available every few weeks there is plenty opportunity for direct feedback.
Waterfall is also inherently more risky because of the loss of engagement with the ‘customer’. The longer the project goes on the more likely that the ‘world has moved’ on within the business and the specification no longer matches the real world requirement.