How to measure Predictability in Software Development

Abstract

Predictability is a key aspect of project management, as it allows organizations to plan and allocate resources effectively. By analyzing the predictability of a project, organizations can identify potential risks and take proactive measures to mitigate them.

The primary resource in software development is knowledge, which acts as the fuel that drives the software development engine. The true nature of software development is a progression from some knowledge about an idea to full comprehension of a finished product ready for delivery. Essentially, software development is a process of knowledge accumulation.

The learning curve shows the rate of knowledge accumulation about "what to do" and "how to do it" in order to deliver working software. The learning curve serves as a leading indicator for project predictability by measuring the rate of knowledge acquisition, enabling proactive management of potential issues and alignment with project expectations.

Unlike traditional metrics such as throughput and lead time, which are only measurable after work items are completed and delivered, the shape of the learning curve becomes visible just a few weeks into the project. This early visibility allows managers to anticipate problems well before any deliverables are affected.

We measure the predictability of a software development project as its ability to meet its expected knowledge discovery rate within a set timeframe.

By comparing the rate of knowledge acquisition of a project with its forecasted learning rate, we can gauge how closely the process aligns with our expectations. If a project is not acquiring knowledge at an expected rate, it may indicate underlying issues. Conversely, if a team acquires the necessary knowledge at the expected rate, the project is considered predictable.

Predictability is the most important metric for software engineering teams and the broader business organization.

A predictable engineering team allows the organization to depend on its metrics to realistically forecast delivery dates, which:

• Builds trust
• Sets and meets realistic customer expectations.
• Enables marketing and sales to accurately plan their activities related to the product.
• Positively impacts financial planning and risk management.

For developers, this predictability means working at a sustainable pace that avoids creating a culture of burnout.

Learning Curve in Software Development Projects

"All software is simply a representation of knowledge and it is the acquisition of this knowledge that is the real activity of software development." ~ Phillip Glen Armour [1]

The primary challenges in digital engineering projects stem from gaps in essential knowledge areas. These include:

• Technological Proficiency: Understanding current technologies, programming languages, and frameworks.
• Domain Knowledge: Acquiring knowledge about the project's domain, including industry-specific requirements and constraints.
• Business Acumen: Knowledge of the specific business needs and requirements that drive project goals.
• System Integration: Knowledge of integrating and updating both old and new systems.
• Functional Expansion: Knowledge about how to effectively add new functionalities within the constraints of existing systems.
• Emerging Technologies: Awareness and understanding of new and emerging technologies.
• Regulatory Compliance: Knowledge of the legal and regulatory requirements that impact the project.
• Security: Understanding security requirements and best practices.

Traditional methods for monitoring the progress of new technology development tend to fall short due to their focus on lagging indicators. These indicators, such as velocity, throughput, and lead time for newly developed features, user stories, and components, often overlook the crucial aspect of knowledge acquisition in software development. Recognizing this gap, our approach marks a significant paradigm shift towards a knowledge-centric perspective in managing software development projects. We utilize the learning curve as a key metric, moving beyond conventional metrics to capture the essence of knowledge growth and application within project teams.

A learning curve represents the rate at which knowledge is acquired over time in a project or learning environment[2]. In the context of software development projects, learning curves are particularly important due to the rapid evolution of technologies and methodologies.

The learning curve serves as a leading indicator for project predictability by measuring the rate of knowledge acquisition, enabling proactive management of potential issues and alignment with project expectations.

In economics, learning curves often relate to production efficiency, showing how costs decrease as cumulative production experience increases. In software development, while the focus is on knowledge acquisition, similar principles apply – as more knowledge is gained, the team's efficiency and productivity typically improve.

To effectively establish learning curves, we utilize the Cumulative Growth Rate (CR) of knowledge discovered. CR quantifies the rate of knowledge acquisition over time, calculated using log returns, which measure the relative growth in knowledge between two time points. Importantly, CR effectively normalizes growth by taking into account variations in project sizes, durations, and initial knowledge levels. This normalization makes CR a relative measure, enabling fair and equitable comparisons across diverse projects.

Application in Project Management

We measure the predictability of a software development project as its ability to meet its expected knowledge discovery rate within a set timeframe.

By comparing the rate of knowledge acquisition of a project with its forecasted learning rate, we can gauge how closely the process aligns with our expectations. If a project is not acquiring knowledge at an expected rate, it may indicate underlying issues. Conversely, if a team acquires the necessary knowledge at the expected rate, the project is considered predictable.

If a team is not acquiring knowledge at an expected rate, it may indicate underlying problems within the project. This insight allows managers to investigate and address the root causes proactively with interventions such as:

• Targeted Training: Focused training efforts to address specific knowledge gaps.
• Addressing the Mismatch Between Capabilities and Challenges: Reallocation of developers to optimize learning and talent utilization.
• Collaboration Improvement: Process improvements to enhance knowledge transfer and sharing.
• Shortening feedback loops: Shortening feedback loops to accelerate learning and problem-solving.
• Reducing rework: Reducing rework by addressing knowledge gaps early on.
• Reducing Multitasking: Reducing multitasking to improve focus and knowledge retention.

Case Study

Below is a project forecast produced using Reference Class Forecasting (RCF).

On the forecast graph, the thick blue line represents the cumulative growth rate of the knowledge expected to be discovered for the successful delivery of the project. The x-axis displays the timeline of the new forecasted project, and the y-axis shows the cumulative growth of the knowledge to be discovered

The green line represents the most likely completion date, the dashed red line indicates the earliest possible completion date, and the dashed blue line designates the latest possible completion date. This range of outcomes helps you prepare for different eventualities and plan your project more effectively.

At the bottom of the diagram, the histogram provides a visual representation of the likely completion dates and their probabilities, allowing you to assess the overall distribution of possible completion times.

For finding out how closely the new project aligns with the forecast we use the Predictability Index \(P\). In the diagram below we have presented the Predictability Index between the forecast and the executed project.

On the x-axis we have weekly dates from the start of a project, while on the y-axis we have the predictability index \(P\) for each week. The predictability index \(P\) ranges from 0 to 100. A predictability index of 100 indicates that the executed project is perfectly aligned with the forecast, while a predictability index of 0 suggests that the executed project is significantly diverging from the forecast. By monitoring the predictability index over time, project managers can identify early signs of potential issues and take corrective actions to realign the project with the forecasted learning trajectory.

The diagram shows the Predictability Index for a project is much less than 100 in the beginning. This indicates that the project was not aligning with the forecast as expected. We can say the project was unpredictable and required immediate attention to realign it with the forecasted learning trajectory.

When evaluating the predictability of a project's Cumulative Growth Rate (CR) of knowledge discovered in relation to its forecast, categorizing the relationship into three cases can be very insightful. Each case has different implications for project management and strategic decision-making. Here’s a breakdown of each scenario and what they might indicate:

1. The current project's knowledge growth rate surpasses that of the forecast.
   • Predictability index is typically lower because the trajectories do not align closely with the forecast.
   • Could be indicative of a highly efficient process or superior expertise. That is not necessarily a good thing if we expect the project to uncover some new knowledge and innovate.
   • Management Strategy: Might focus on maintaining momentum, scaling successful strategies, or even re-evaluating projections and resource allocations to accommodate faster growth.
2. The project's knowledge growth rate lags behind the forecast.
   • Predictability index is typically lower because the trajectories do not align closely with the forecast.
   • Could be indicative of unforeseen challenges, less effective execution, or external factors impeding knowledge discovery.
   • Management Strategy: May require intervention to diagnose issues, reallocate resources, adjust strategies, or revise goals to reflect realistic outcomes based on current progress.
3. The project's knowledge growth rate progresses at a similar pace and pattern to the forecast.
   • Predictability index is high, reflecting a strong alignment with expected trajectories based on historical data.
   • Indicates that the project is on track as planned and following a successful path.
   • Management Strategy: Continued monitoring and adherence to the established plan. This scenario is ideal from a predictability standpoint as it suggests stability and manageability.

Understanding these scenarios helps in not only assessing current project status but also in planning future actions.

Importantly, both scenarios where the current project's knowledge growth rate surpasses or lags behind the forecast would typically result in lower predictability indexes. .

Conclusion

Throughout this article, we have delved into the concept of predictability in software development project management. We've examined how learning curves can serve as a predictive tool, offering crucial insights into a team's adaptation to new technologies and challenges.

The case study presented demonstrated a situation where a new project did not align with the forecast as expected, indicating its unpredictability. This highlights the need for immediate attention to realign projects with the expected learning trajectory, ensuring predictability and successful outcomes.

Future work could further refine this method, exploring its application in different contexts and its integration with other project management tools and strategies.

Appendix