Complexity

Complexity is one of those areas that seems like it should be fairly easy to understand, but ironically I’ve found the opposite. And this makes sense – there is so much in the IT industry that is abstract already that adding complexity to the mix is difficult.

Yet it is important to talk about complexity, because it is usually a core issue at the heart of most software projects. Complexity has direct impact on the requirements, resources and risk levels of a project, and a good understanding of complexity allows us to engage in more educated discussions around quality and schedules. So why is it so hard?

Before we explore that I’d like to step back a second and identify the types of complexity we usually encounter in a project.

Domain Complexity

This is complexity in the problem we are trying to solve – in the workplace it would be the business steps or process we are working with. Typically a large amount of the analysis or requirements phase will be on understanding and simplifying this type of complexity. It’s also a core part of the development process – our aim is often to reduce a problem to it’s most generic form.

It’s critical for a project to understand the complexity here – but also important to note that the final solution may not incorporate the full extent of the complexity. For example complex mathematical models may support a business process, whilst the application will only implement the final step of a derived formula.

This has some interesting implications for quality assurance and testing – but is also a post for another day.

Technical Complexity

The technical complexity is not necessarily directly related to the domain complexity, but relates to the complexity that will be present in the final solution. It is heavily impacted by:

• The chosen design and architecture.
• The existing environment and code base.
• Any concurrency that must be accounted for in the design.
• The scalability of the environment and solution.
• The variation in state, and navigation/flow of the solution (or, more simply – how many different ways can I jump around and cause things to change).

The interesting thing about technical complexity is that it often exponentially increases. Individually a single feature may be fairly simple – but as we start to consider how many related issues affect it the problem grows. Take a simple web form, add in some complex state, throw in some concurrency, add a little latency – and suddenly what seemed a simple feature may now be generating half of the defect reports for your system.

So why is complexity hard?

Technical complexity can be hard to identify, because of the number of abstract concepts that are involved. The detail behind a problem is often vastly more complicated then a user may imagine. The issue however, is more not as simple as this.

Historically our industry has focused on gathering business requirements – this is a core concept behind waterfall methodologies. The reasoning behind this is that the IT team is delivering a product for the business, and the technical complexity is not important in that concept. In this model complexity is reduced behind the veil.

However in the last few decades we have seen systems become significantly more complex (in all areas), and the boost in computer powers has seen us start to solve more complex problems. Our users also expect us to focus on making systems more intuitive – in short, the problems we are solving today have a (generally) higher level of detail. A significant part of the industry is exploring other methodologies such as agile, but even in agile environments there can be a preference to focus solely on identifying the business problems.

Managing Complexity

Complexity in a project represents risk – so it’s very important that the level of complexity is managed and communicated. As a developer I have to ensure that not only am I managing my work, but I need to prepare both stakeholders and testers so that they have an expectation of the complexity.

Beyond risk, complexity represents work. Because of this it is critical that complexity is identified early in the process. This is a core driver behind the move towards agile-style methodologies in our industry – it provides a good opportunity for analysts, developers and domain experts to work together to determine where the work exists in the project. The work is a good early indicator of where the complexity is in the solution.

Agile style methodologies encourage the development of user stories, which map out the proposed solution. A user story breakdown that is the result of collaboration should highlight where the complexity of the solution is, and highlights where the effort for the project can be most effectively targeted – not just in development but also in both the analysis and testing.

Overlapping Complexity

Pay close attention to areas where both domain complexity and technical complexity are present! These are the areas that will require the most attention to detail early in the project. It is commonly said that the earlier defects are caught the cheaper it is to fix and recover. Defects in this area are much more likely to be high severity – they are at the core problem you are likely to solve.

Sometimes bugs arise from issues that only “only” technically complex – perhaps an issue exists for a very specific case that 99.9% of users won’t encounter. Businesses make decisions all the time to live with these edge cases. However if an issue exists close to the heart of the issue you are trying to solve it is much more likely to be a costly problem.

So what can we do to reduce this risk? Resources is the obvious answer, however if we identify the complexity early we also have an opportunity to look at ways of finding alternate solutions. Does the system need to be modelled exactly as requested – or can a simple change in design to the user interface separate the problems? Given the choice I would usually take the option to separate and reduce complexity – as it reduces the problem and allows us to focus resources in other areas- at the end of the day success of a project is measured by what and when we deliver.

Photo Credit: 55Laney69