The brains of human beings have a finite capacity. Our minds can only handle so much system complexity at a time. This is certainly true in my case, and unfortunately, the situation is not improving as I get older.
The complexity of even small, modern software applications is simply enormous, and complexity grows non-linearly as the size of a software system increases. In fact, Gerald Weinberg argues that complexity grows, at least as fast as the square of the size of the program. He calls this the size/complexity dynamic of software development.
The complexity of a piece software derives from two different systems or models if you prefer that term:
The Problem domain
The first system is that of the real world problem we are working with. In enterprise systems, this is the concepts within the scope under consideration, their structure or organisation, and the way various people in different roles both inside and outside of the enterprise use them to do business. More generally, software designers refer to this system as the problem domain. Peter Coad defines the problem domain for a piece of software as the field of endeavor under consideration. The more concepts involved, and the more ways they can be combined, structured and used, the more complex the problem domain becomes.
There are a number of different ways to analyse a problem, but object-oriented and component-based approaches have dominated in enterprise software development for approximately the last fifteen years. After all, every team working in .Net, Java, or Cocoa has some sort of object model at the heart of their software, and I have found no better technique than Peter Coad's 'modeling in color' for the initial, object-oriented analysis and design of enterprise software. This is especially true when that analysis further informed by the patterns and strategies Nicola, Mayfield, and Abney's, Streamlined Object Modeling and Eric Evans's Domain-Driven Design.
The Technical Architecture
The second system is the technological one used to provide the solution required, the number and nature of the technology components involved. Even as early as the 1970's Fred Brooks argued in his first essay within The Mythical Man-Month that software crafted for a specific individual to use is at least three times simpler than the same software turned into a product so that it is applicable to hundreds, thousands or even millions of people. He also argued that the same software turned into a reusable component of a larger system is roughly three times as complex again.
Today a web-based system designed to serve multiple concurrent users, even if it is only a handful at a time and not the millions required by enterprises like Amazon, eBay, Google, Facebook, Microsoft, IBM, or Apple, etc., has far more technical complexity than an old traditional single-user application of comparable functionality.
The two main techniques we have for handling complexity are:
Decomposition and integration
Decomposition and integration is the breaking down of a problem into smaller and smaller pieces until each piece can be solved easily and then putting these small solutions together to form solutions to the bigger problems until the overall solution is achieved.
Abstraction is the ignoring of details of a problem that are not relevant to what we are currently doing. This makes it easier to work with the details that are. Abstraction enables us to solve a problem in simpler, more general terms first and subsequently address specifics of the problem in detail
Other equally complex systems are involved in the development of any non-trivial piece of software. These are complex purely because their fundamental components are human beings. They include:
- The system of humans, the project team, that are designing and
building the system, and the organisation or structure of the project
the roles and responsibilities, and the ways the team members
communicate and interact with each other. Given enough time two friends
garage could produce
very large and complex software systems by solving one small
problem after another. Adding more people to the team allows us to
solve many of the small problems in parallel and deliver large
systems quicker. The more people we add, the more we can do in
parallel. However, the catch is the more people we
have working in parallel, the more we are likely to bump into
communication and coordination problems. If there are significant
dependencies between the problems being solved, then we have to
work hard at putting the right communication channels in place so
everyone has the information they need at the right time. This turns
the problem into primarily one of managing communication.
- The system of humans, the customer team, that are specifying and
sponsoring the construction of the software. Similarly, and sometimes
even more so, the various egos and personalities within the customer
organisation and the way they work or do not work together forms a
system that is complex to navigate and negotiate with.
In fact, the human systems are generally more complex to deal with than the problem domain and technology systems. Hence, Jeff De Luca's first law of information technology: I.T. is 80% psychology and 20% technology.