Project management

The Nature of Project Management

The very definition of a project implies a journey into the unknown. At the outset, many aspects of the project's outcome and the work required to achieve it remain undefined and uncertain. Acknowledging that any project will inevitably face knowledge gaps is crucial for effective project management. Addressing these gaps effectively is the largest part of project management and is key to ensuring predictable delivery in software development projects.

Unlike physical resources, knowledge is both the tool and the product of project management. The effectiveness with which a team acquires, applies, and transfers knowledge directly influences the predictability and success of a project. By effectively managing knowledge—understanding what is known, identifying what needs to be learned, and applying this evolving understanding—teams can navigate uncertainties, solve complex problems, and achieve project goals efficiently. Thus, knowledge is not merely an asset but the fundamental driver of project progress and success.

Central to this perspective is understanding the knowledge gap between what a developer knows and what they need to know to effectively complete tasks. Bridging this gap involves discovering and applying new knowledge.

Knowledge gaps can arise because the project team may lack the necessary knowledge and expertise required to address specific technical challenges or industry-specific requirements.

However, it's important to recognize that knowledge gaps also frequently represent opportunities for innovation, where the team must develop novel solutions to meet both business objectives and technological advancements.

Embracing this dual nature of knowledge gaps—as both hurdles to be overcome and as catalysts for creative problem-solving—enables teams to navigate the complexities of software development with a proactive and adaptive mindset.

In project management, we encounter things that we know and things we don't know. What we do know is the knowledge we possess and are cognizant of, serving as the bedrock for our decisions. Simply put, we are familiar with both the questions and their respective answers.

A large part of project management is managing the unknowns. The outcome of the project and the work required to produce the outcome is unknown at the start. We can distinguish between three categories of missing knowledge:

• Known Unknowns: We recognize gaps in our knowledge and actively seek answers. Essentially, we are aware of the questions we need answered. This echoes Socrates’ famous saying: "I know that I do not know."
• Unknown Unknowns: These are the elusive blind spots we are unaware of, often the culprits behind unforeseen challenges. In other words, we're oblivious to both the questions and their answers.
• Unknown Knowns: These represent our subconscious or tacit knowledge—information we know but may not recognize we do. It's often overlooked, yet immensely valuable. Here, we are acquainted with the answers but not necessarily the questions they address.

Dealing with “known unknowns” is the field of risk management. Dealing with “unknown unknowns” is the field of product management.

Once we are aware that each project contains “Unknown Unknowns”, it becomes evident that a process is needed to:

1. show us what we know, what we don't know and what we don't know we don't know
2. discover these Unknown Unknowns” and turn them either into “Known Unknowns” or “Known Knowns”

We call this process “knowledge discovery”.

Here is a process that will help manage knowledge discovery in a project:

1. Understand your project's complexity
2. Staff the team with the talent needed to fill most of the knowledge gaps
3. Continuously fill the remaining knowledge gaps
4. Monitor progress and adapt strategies as needed

This iterative process aims to acquire new and missing knowledge with respect to the project, through experiments and training.

Understand your project's complexity

The traditional view is that project complexity refers to the degree of difficulty, uncertainty, and interdependence involved in managing and completing a project[3][4].

But what does "difficulty" mean in the context of software development projects? Does it imply developers face challenges while typing?

This is where the knowledge-centric approach to software project estimation and management comes in, which focuses on the acquisition of knowledge as the primary activity of software development[8].

Project complexity refers to the amount of knowledge that needs to be discovered, i.e., the missing information acquired by the team throughout the project lifecycle for the successful completion of the project.

The project is a system composed of the people who will deliver (Capability) and the people who define what to be delivered (Client). The Client is responsible for defining "what to do," while the Capability is responsible for defining "how to do the what."

The complexity of a project can be influenced by both Client and Capability. We group the factors into two clusters based on their ownership and influence by either the Client or the project team (Capability). Here's the breakdown:

Client-owned factors influencing project complexity:

• Requirements complexity: The number, degree of uncertainty, and intricacy of project requirements, as well as their potential for change, can impact the knowledge discovery process. In simpler terms, this factor asks whether the client has a clear understanding of what needs to be done. Projects with numerous or complex requirements, or with clients who are less certain about their desired outcomes, may necessitate more knowledge discovery and learning.
• Client collaboration: In addition to the internal characteristics of projects, it is important to consider the contexts in which projects are executed. The performance of the same project team may vary depending on whether the client is a startup or a Fortune 500 corporation, as the level of collaboration with stakeholders can differ significantly. For example, working with a small startup typically implies that questions are answered promptly and communication is more agile. On the other hand, working with a large organization like a bank often involves slower communication and more bureaucratic processes, which can impact the project's overall efficiency and progress.

Capability-owned factors influencing project complexity:

• Technology expertise: This includes the intricacy of the algorithms, technologies, tools, and frameworks used in the project, as well as the overall architectural design. A project with more advanced or unfamiliar technologies will require more knowledge discovery and learning.
• Codebase knowledge: This factor enables project managers to assess the potential risks, challenges, and dependencies associated with building upon an existing codebase or starting from scratch. It plays a significant role in determining project complexity, particularly when the team needs to familiarize themselves with an existing codebase. The time and effort required to understand the existing code, its architecture, and design patterns can slow down the project's progress, especially in the initial stages. Furthermore, dealing with an existing codebase may present additional challenges, such as technical debt, outdated dependencies, or a lack of proper documentation. These challenges can increase project complexity and affect overall project estimation and management processes. On the other hand, starting from scratch might necessitate more time for initial setup, architecture design, and development, but it also provides an opportunity to leverage the team's prior knowledge and expertise, potentially streamlining the development process.
• Domain Knowledge: Every software project operates within a specific domain, whether it's healthcare, finance, retail, or any other industry. Domain knowledge encompasses understanding the unique requirements, constraints, and best practices of that industry. Acquiring this knowledge can be particularly challenging for teams working in unfamiliar domains. The lack of domain expertise can lead to misunderstandings, misaligned priorities, and ultimately, a product that does not meet user needs or regulatory standards.

  The problem domain or industry the software is being developed for can influence the amount of knowledge that needs to be acquired. For example, a project in the healthcare industry may involve understanding specific medical terminology, regulations, and processes, which adds to the complexity.

• Team collaboration: The individual knowledge, skills, and expertise of team members, as well as the interdependence between teams or groups possessing specific information required for a project to progress, can influence the amount of knowledge that needs to be acquired during a project. For example, a project staffed primarily with junior developers will likely require more knowledge discovery and learning than a project with a more balanced mix of junior and senior developers. Having an experienced architect on the team can provide a valuable knowledge base and guidance, enabling developers to access the information they need more quickly. Furthermore, effective collaboration and communication between different teams or groups are essential for sharing vital information and addressing project challenges efficiently. Additionally, having dedicated QA team members can help ensure that any incorrect knowledge acquired and applied is identified and addressed more rapidly. This faster feedback loop can prevent the propagation of issues stemming from misunderstandings or lack of knowledge, ultimately improving the project's overall efficiency and quality. In this context, the interdependence among team members and groups plays a crucial role in managing project complexity and ensuring successful outcomes.
• Development process: The way the discovery of missing information is managed by the project team. When it comes to discovering missing information, Agile and Waterfall development processes handle this aspect differently:
  • Agile methodologies embrace change and encourage continuous learning and adaptation throughout the project. The iterative nature of Agile development enables teams to discover missing information and adjust their plans accordingly as the project progresses. By breaking the project down into smaller, manageable tasks, teams can identify gaps in their knowledge and understanding more quickly and address them before they become major issues. Frequent communication and collaboration among team members and stakeholders further facilitate the identification and resolution of missing information.
  • In the Waterfall development process, the discovery of missing information can be more challenging due to its linear and structured nature. The process relies heavily on upfront planning, with the assumption that all requirements and information are gathered and documented before the project begins. As a result, discovering missing information later in the development process can lead to delays, increased costs, and the need for significant rework. Teams following the Waterfall process may find it more difficult to adapt to change and acquire missing information, as doing so may require revisiting earlier stages in the project lifecycle.

It's important to note that while these factors can be categorized based on ownership and influence, there will still be some degree of interdependence between them. Collaboration and communication between the client and the project team (Capability) are crucial in managing project complexity effectively.

In this article we explore how to estimate project complexity using a knowledge-centric approach.

Understand your project's knowledge gaps

The main tool in understanding our project's complexity is the Complexity Profile (CP), detailed further here.

The Complexity Profile (CP) is a tool that categorizes knowledge into three types that the project team must discover to complete a task, user story, or epic:

• Knowledge to be discovered on what to do: This pertains to the overall definition of the task, user story, or epic.
• Knowledge to be discovered on how to do it: This involves the specific steps or methods required to complete the task, user story, or epic.
• Knowledge to be discovered on what to do and how to do it: This encompasses both the overall definition and the specific steps or methods needed to complete the task, user story, or epic.

Project complexity is influenced by both Client and Capability factors, including requirements complexity, client collaboration, team collaboration, technology expertise, domain expertise, and the development process. By considering these factors and assigning a quantity to each quadrant, the CP provides an idea of how much knowledge is required for each aspect of the project.

The most valuable knowledge to acquire is knowledge that no one has, but not all missing knowledge is valuable.

The need to know a project's complexity arises in two instances: before starting it and once execution begins.

Before Starting the Project

Before commencing work on a new project, we can comprehend its complexity using the process detailed here.

Figure 1 presents the initial Complexity Profile (CP) of our project, representing the perplexity of both Capability and Client about the project as a whole.

Upon establishing the initial complexity profile of our project, we can manage the project's complexity (knowledge gaps) through the following steps:

1. Identify knowledge gaps: Determine areas where Client or Capability may lack expertise or where knowledge gaps may exist. This facilitates planning for additional training, research, or external support to bridge these gaps.
2. Compare with past projects: Review completed projects with similar complexity levels and identify patterns or commonalities. This helps better understand potential challenges, risks, and opportunities your team may encounter during the project and how to address them.

Starting the Project

For continuously understanding knowledge gaps during our project's execution, we will use the process explained in detail here.

At the end of the first iteration of this process, we will have a detailed operational complexity profile. Figure 2 presents the operational Complexity Profile (CP) of our project, representing the perplexity of both Capability and Client about the user stories.

Staff the team with the talent needed to fill most of the knowledge gaps

Understanding your project's complexity would show the expertise you need to acquire certain knowledge areas. Each of these areas presents unique challenges that must be addressed to ensure project predictability.

In general, the team should include professionals with strong technological proficiency, who stay updated with the latest technologies, programming languages, and frameworks, through regular training and hands-on experimentation. Domain experts are crucial for understanding industry-specific requirements and constraints, ensuring the project aligns with industry standards. Business analysts or stakeholders with business acumen are needed to align the project with business goals, target audience needs, and market dynamics. Integration specialists are essential for seamlessly integrating new solutions with existing systems, maintaining compatibility and data integrity. Compliance experts are necessary to navigate legal and regulatory requirements, conducting regular audits and ensuring adherence to standards. Finally, security professionals are vital for implementing robust security measures, performing regular audits, and staying ahead of potential threats. By assembling a team with these diverse skills, project managers can effectively address knowledge gaps and enhance the predictability of software development projects.

We need different people to work on different knowledge gaps. The difference is the level of perplexity they can handle both from Capability and Client perspectives. For instance:

• If the problem is clear then we need a junior person, who can read a recipe book and follow it. She will need stable requirements and all the exact tools and technologies from the book.
• If the problem is complex we need a senior person, who can cope without a recipe, with changing requirements and with whatever is available in terms of tools and technologies.

When we need to adapt we need more diversity in the system because then we've got more evolutionary capacity. It's not a coincidence that for the living organisms the mutation rates increase under stress - more diversity more, adaptability. The requisite diversity is the optimal level of diversity for the context we are in. Diversity means inefficiency an organization needs to have. If overfocused on efficiency the organization loses adaptive capacity.

Diversity can help us answer the question posed by Brooks' Law[9]. The question is, "if I have a project that is running late, should I add people to it to speed it up?" Our answer is, "it depends on how you manage knowledge discovery." If the project scope is partitioned into work items classified using the Complexity Profile explained here, then you'll have Clear and Complicated work that can be given to the newcomers. That will free up your present developers to do more Complex work clarifying. Then the answer is "yes." The answer would be "No" if the addition of people will require that your current staff take time away from being productive to instruct the newcomers, resulting in a net drop in productivity. Hence, Brooks' Law: adding manpower to a late project makes it later.