There are many ways to organize software development. Today, the infamous waterfall model has mostly been replaced by some flavor of agile development. However, this change hasn’t been a panacea. Still, software developers are plagued by lack of time, unclear requirements, code bases filled with barely legible legacy code and processes which only get in the way. How can we find the root cause for these problems?

Since the downfall of the waterfall model, it has become out of fashion to think of different phases in software development. In Scrum, for example, there are no more phases: Instead, you have sprints which combine all the old phases of the waterfall model in a single short time span. To recap, the waterfall model had these phases:

  1. Requirement gathering and analysis
  2. System design
  3. Implementation
  4. Testing
  5. Maintenance

All these activities still exist when you’re using an agile development model. When doing a software development project, the project will eventually end and you’ll enter a permanent maintenance phase. If you are continuously improving a product, maintenance is a permanent background task. Implementation is done within a sprint. The system design can either be done beforehand or within the sprint / during the implementation via continuous design. Requirement gathering and analysis should be done before the sprint starts to make sure that the implementation doesn’t get blocked because of open questions. Testing is done as part of feature development and/or in specialized regression tests.

So, what does this all mean? The point I’m trying to make is that the old waterfall phases can be helpful when searching for the root cause of problems. As developers we can get so caught up in the daily grind that we lose sight of the big picture. For example, say that you are implementing a new feature and everything takes much longer than expected because you stumble from open question to open question. Finding answers takes time and suddenly you are not sure anymore that you can deliver in this sprint (or at all). Here, the very first phase didn’t go well: The requirements were too poorly understood to even start development of this feature. Why was this the case? Maybe the product owner (or whoever defines features in your process) didn’t spot all the open questions. Maybe he knew, but decided to start development anyway because the feature was important and things would work themselves out over time. Our maybe the customer didn’t clearly express his ideas when he made the request in the first place. In any way, the cause of the problem is found upstream in the development process. If this happens often, there is something wrong with your requirements engineering process (or the people doing it).

Another example for an upstream problem is a lack of focus in your development. You might’ve noticed that your priorities shift all the time in unpredictable ways. Things that were critical last month are suddenly unimportant even though they were only partially addressed (or not at all). Or you might have to juggle too many conflicting interests raised by equally powerful stakeholders. This can lead to confusing changes in priority from sprint to sprint depending on which stakeholder has the upper hand. Or you might lack a mid-term plan. Do you have a rough idea what you plan to develop in the next six months? I know that this seems very waterfally and lacking this kind of planning is fine if you work in a small company or in a very customer centric project. However, it is troubling if you’re developing the next version of some standard software product, especially if you only ship a few times each year. All of these problems are caused by bad planning. You lack clear priorities and a vision of what you want to build into your software. So again, the root cause can be found upstream of development.

Many code bases are in a terrible state and get worse with every line of code which gets changed. To keep code from deteriorating over time, constant refactoring needs to be applied and new code needs to be written properly. It is the developers’ task to do this, but often they are under pressure, badly trained or a combination of the two. Sometimes refactoring tasks are too big to get done as part of a related feature (or bug fix). Then, they need to be planned and executed at an appropriate point in time. Balancing these refactoring tasks and feature development is the job of the product owner in a Scrum process. If refactoring is always pushed back indefinitely, then the code base will never improve. So, if you find yourself working in a messy code base, you might want to check in with your product owner and discuss priorities. Note, that this is not an excuse to write bad code. The majority of blame for bad code clearly lies with the developers.

Last, we briefly have to talk about staffing. I strongly believe that people are more important than processes. With a tight knit and competent team, any process can be done in an efficient manner. But with a dysfunctional team, even the best process will not work properly. At the end of the day, the development managers are responsible for staffing and hence, they have a crucial job in software development. I’ve outlined in my developer matrix how big the differences between developers can get. So, hiring the right developers is essential. Of course, a similar spread exists in the more “bureaucratic” roles. Let’s take the role of a product owner as an example. A product owner has a lot of responsibility in a Scrum process and needs to make hard decisions on a daily basis. He needs to know both the product and the team well to make a meaningful plan. A bad product owner will place a huge burden on the development team as his tasks get shifted downstream. Managers should make very sure that the right person is doing this job. Naturally, they also have to hire the right amount of people at the right time. Hiring too few employees is less dangerous than hiring too many. As Fred Brooks pointed out decades ago: Adding people to a late development project only makes it later. So, even when managers are “only” dealing with people issues, they still fulfil a crucial role.

That concludes this article. I hope you get the general idea. When you face your next problem during software development, take a moment and search for the root cause upstream. It might be enlightening.