Reduce technical debt

What is technical debt?

"...we accumulate the learnings we did about the application over time by modifying the program to look as if we had known what we were doing all along and to look as if it had been easy to do..." ~ WardCunningham[2]

Ward Cunningham's seminal perspective on technical debt pertains to how we accumulate learnings about an application over time. The challenge lies in modifying the program to reflect as if we had known what we were doing all along and that it had been easy to do.

In software-intensive systems, technical debt is traditionally understood as a collection of design or implementation constructs that are expedient in the short term but create a technical context that can make future changes more costly or impossible[1]: It is when small changes take big efforts, and big changes require small miracles. But can we explore it from a different perspective? Let's consider technical debt through the lens of a knowledge-centric approach.

Technical debt from a knowledge-centric perspective

From a knowledge-centric perspective, technical debt can be seen as the cost we pay when our initial decisions, due to lack of knowledge or deliberate strategy, lead to rework down the line.

Technical debt can be broadly categorized into two types: intentional and unintentional.

1. Intentional Technical Debt: This type of technical debt arises from deliberate decisions where short-term gains are prioritized over long-term code quality. A development team might choose to implement a quick but less optimal solution to meet a deadline or to quickly deliver a feature to the market. They know that the solution isn't the best in terms of maintainability or scalability, but they make a conscious decision to accept the technical debt and deal with it later. This is akin to taking out a loan knowingly, with a plan to repay it in the future.

1. Unintentional Technical Debt: This type of technical debt arises from a lack of knowledge or experience, resulting in suboptimal design or coding practices that later need to be corrected. It's like incurring a debt unknowingly, only to discover it later when the interest (in the form of rework) has compounded.

From a knowledge-centric perspective, both types of technical debt involve a replacement of existing knowledge — either the knowledge was insufficient at the time of the initial decision (unintentional), or the existing knowledge was deliberately set aside for short-term gains (intentional).

Technical Debt is a Vestige from an Evolutionary Process

Software development's true nature is a journey of exploration and investigation, as an individual seeks to fill in the gaps in their understanding and gain the knowledge necessary to complete a task.

This process, which we call Knowledge Discovery, starts with identifying the knowledge to be discovered, which can be thought of as the question "what do we need to know?" This represents the gaps in understanding or information that an individual or group believes they lack. By recognizing what is unknown, i.e., the questions without answers, the process of knowledge discovery aims to fill in these gaps by finding answers through research, investigation, and other means.

At its core, knowledge discovery is an evolutionary process. It mirrors biological evolution through variation and selection: generating ideas (variation) and subjecting them to criticism (selection). This iterative approach ensures that only the most robust ideas survive, driving progress toward better understanding and solutions.

Here's how this analogy works:

1. Variation of Ideas: In software development, new ideas, features, or architectural approaches emerge as variations. These are often implemented under constraints, such as time, resources, or incomplete understanding of future needs.
2. Selection of the Best: Over time, as requirements and technologies evolve or the product matures, certain approaches prove more effective, efficient, or scalable than others. This leads to the selection and refinement of the "best" solutions, akin to natural selection.

This perspective shifts the focus away from finding definitive solutions and toward fostering an environment where better questions can emerge - questions that challenge assumptions, illuminate unknowns, and lead to breakthrough discoveries.

How to reduce the technical debt?

Managing and reducing the two types of technical debt might require different strategies. For intentional technical debt, it's about strategic planning and timely repayment. For unintentional technical debt, it's about improving knowledge acquisition and application processes to make better decisions in the first place. In both cases, the goal is to minimize the waste or rework necessary to deliver the desired software, thereby "maximizing the work not done".

This human-centric view lends itself well to understanding the nature of 'waste' and 'technical debt' in software development. Typically, 'waste' might refer to unnecessary features. However, when viewed from a knowledge-centric standpoint — where software development operates on knowledge much like a car runs on fuel — the definition of 'waste' evolves. We define 'waste' as a situation where existing knowledge (akin to laid bricks) is replaced with new information. It's the "death by a thousand cuts", where each cut is a piece of knowledge replaced.

This waste is primarily due to rework, which comes in two forms:

1. Changes to 'What' should be built, involving altering or removing features, which, in turn, changes the code and the user-facing functionality.
2. Changes to 'How' to build the 'What,' otherwise known as refactoring. This involves altering the code without changing the user-facing functionality.

Rework, thus defined, is distinct from design iterations intended for rapid learning, where design decisions are viewed as experimental and subject to change. It's also different from fast project cycles designed to accelerate customer feedback.

Therefore, the concept of 'technical debt', when viewed through a knowledge-centric lens, becomes a measure of the efficiency of our knowledge discovery and application process. It's about reducing the 'waste' or rework necessary to deliver the desired software. To put it another way, "minimizing waste" equates to "maximizing the work not done". Technical debt, then, is the cost we pay when our initial decisions, due to lack of knowledge, lead to wasteful rework down the line. The less we need to refactor, the more efficient we are, and the less 'debt' we accumulate. But to refector we need because of the very nature of software development!

Adding to the understanding of technical debt from a knowledge-centric perspective, it becomes clear that managing technical debt effectively often involves balancing between iterative and incremental development.

In software development, if we are unable to safely reiterate the existing design (refactor), then incrementing (adding functionality) may lead to the degradation of code and overall progress. Everything becomes progressively worse — it's slower, bug-ridden, complicated, unattractive, unpredictable — a host of issues arise. The takeaway is clear: there is no free lunch in the world of software development. If you can't iterate, you should avoid incrementing as well. If you do choose to increment, you must be prepared to iterate.

Applying a knowledge-centric view to this challenge brings further clarity. Typically, in the software development landscape, 'incremental' and 'iterative' refer to adding new features and refining existing ones, respectively. But from a knowledge discovery standpoint, these terms acquire a different nuance. 'Incremental' can be defined as a process where new knowledge is added without discarding existing knowledge — akin to laying bricks without removing any. Conversely, 'iterative' involves replacing existing knowledge with new information.

So, in terms of knowledge discovery, 'incremental' development involves consistently adding to the existing knowledge base, while 'iterative' development signifies the continuous cycle of refreshing knowledge, replacing old with new. This balancing act between iterative and incremental development is crucial in managing and minimizing technical debt, thereby maximizing the efficiency of the knowledge discovery and application process.