Improve productivity and happiness on a software development project

Establish an iterative Knowledge Discovery Process

Abstract

"All software is simply a representation of knowledge and it is the acquisition of this knowledge that is the real activity of software development." ~ Phillip Glen Armour [8]

The goal of this article is to help the leaders of software development organizations improve their productivity and make developers happy. For that we will put on the lens of Information theory to differentiate between knowledge discovery and knowledge application in defining an effective software development process. The primary purpose of this process is to show us where we have lack of knowledge. Not only to show us what we know and what we don't know as much as to show us what we don't know we don't know. We acknowledge that the true role of the development process is to acquire knowledge, and the most valuable knowledge is knowledge we do not already have[8]. Armed with the new understanding we'll take an end-to-end view on the way we manage a software development project.

After that, all the way through the project, day by day, we should be investing effort to identify the knowledge gaps about which we feel most perplexed and how that perplexity is hampering our project the most. As software development is a highly cognitive and mentally demanding activity, it's important to understand how to measure the happiness of developers. A state of "flow" is a key indicator of optimal experience and happiness in work.

Finally, we will show how to integrate the knowledge discovery process into the overall project delivery. We can do that using the Kanban[3] method and especially Ustream Kanban[4].

How could we know if our efforts at increasing our knowledge are successful? By using all the reporting functionality KEDEHub provides us with. We can check the results on a daily basis if needed. The best is to check on a weekly basis and report every Monday.

Flow

“A bad system will beat a good person every time.” —W. Edwards Deming

Software development is a process of going from 0% knowledge about an idea to 100% knowledge of a finished product ready for delivery to a customer. In its simplest form, the job of software development is to reduce the number of unknowns to zero by the time the system is delivered. It is a process of knowledge acquisition and ignorance reduction[8]. Hence we model the software development process as a knowledge discovery process. The goal is to iteratively identify knowledge gaps, reduce our perplexity, increase our knowledge and close the identified knowledge gaps.

It is also a continuous process rather than a punctuated process. This means that moment by moment on the project, ignorance is identified, questions are asked, answers are obtained and validated, and knowledge is discovered[8]. We need to find the knowledge to be discovered for each user story. That knowledge is the difference between the missing information and our prior knowledge.

Quite commonly, we underestimate how much knowledge we have to discover to deliver the functionality to the customer, or overestimate our prior knowledge, which is the same thing.

People in general and software developers in particular are happy when working in a state of Flow. Flow experiences lead to an increase in productivity. When the flow conditions have been met they create a flow loop between action and feedback that allows for continuous and effortless tuning of performance while taking action. Flow loops make an activity worth doing for its own sake[2].

Entering flow depends on establishing a balance between perceived individual skills and perceived task challenges. Said in another way - capability should match work complexity.

After that the developer must receive immediate feedback that allows them to continuously adjust their performance as they tackle these challenges. This kind of feedback communicates how well they are performing and how they can improve their performance.

By measuring how well project policies and processes meet the flow conditions we can engineer a project organization that facilitates flow.

From a project management perspective there are two areas we need to create.

  1. Conditions for flow. That means breaking down the project into disjoint units of work. For each task the three flow conditions should be in place.
  2. Action and performance feedback. Tasks distribution so that each of the developers has a proper challenge for their skills. A feedback mechanism should be in place to tell developers how well they are doing.
The aim of what follows is to shed light on how to design systems that facilitate flow.

Creating conditions for Flow

“The goal is to make full use of the workers' capabilities by building up a system that will allow the workers to display their full capabilities by themselves.” ~ Y. SUGIMORI, K. KUSUNOKI, F. CHO & S. UCHIKAWA (1977) Toyota production system and Kanban system

In order to have conditions for flow for each user story we need to find a balance between their perceived individual capabilities and the complexity. This balance should be tailored to the individual developer, with a slight preference for challenging tasks. To determine this balance, an assessment of the developer's individual capabilities and the complexity of the task is necessary. This assessment requires accurate and reliable information about both of these factors.

There are two types of knowledge that a software developer must discover in order to complete a task:

  • Knowledge to be discovered on what to do: This refers to understanding the task and its objectives
  • Knowledge to be discovered on how to do it: This refers to knowing the methods and having the skills needed to complete the task

We refer to the developer's perception of the task complexity as "required knowledge." Bear in mind that the perception of knowledge to be discovered may vary among developers and could evolve over time for a single developer as they refine their skills and task understanding. It is also important to recognize that a developer's perception of required knowledge may not always align perfectly with the actual requirements of the task, and it may be necessary to provide additional training or support to bridge any gaps.

Prior knowledge on what to do and how to do it has already been discovered. This "prior knowledge" provides a starting point for the software developer, but additional knowledge must still be acquired in order to complete the task.

If there was a balance between individual capability and work complexity can be quantified by measuring the knowledge discovered after successful work completion.

It is more convenient to represent the four combinations of required and prior knowledge in order to complete a task in a matrix, which we will refer to as Complexity Profile (CP), which is explained here.

The diagram is intended to help visualize the balance between individual capabilities and work complexity that is necessary for a developer to be in a state of flow when working on a task. Each of the four subspaces in the diagram represents a combination of required and prior knowledge, which we refer to as knowledge to be discovered, or the information the developer needs to gain in order to complete a task.

The less knowledge that needs to be discovered, the happier the developer will be. However, it is important for there to be some knowledge that needs to be discovered in order to maintain interest and engagement.

To achieve a balance that is on a personal, individual level, with a slight tilt towards challenges, the developer needs to pick a task with a required-prior knowledge combination from one of the yellow quadrants that has a missing information between 2 and 4 bits. That means, there are up to 16 alternatives and it requires at least 1 and up to 4 questions asked in a row, in sequence, linearly.

Understand your project's knowledge gaps

The main tool in understanding our project's complexity is the Complexity Profile (CP), detailed further here.

The Complexity Profile (CP) is a tool that categorizes knowledge into three types that the project team must discover to complete a task, user story, or epic:

  • Knowledge to be discovered on what to do: This pertains to the overall definition of the task, user story, or epic.
  • Knowledge to be discovered on how to do it: This involves the specific steps or methods required to complete the task, user story, or epic.
  • Knowledge to be discovered on what to do and how to do it: This encompasses both the overall definition and the specific steps or methods needed to complete the task, user story, or epic.

Project complexity is influenced by both Client and Capability factors, including requirements complexity, client collaboration, team collaboration, technology expertise, domain expertise, and the development process. By considering these factors and assigning a quantity to each quadrant, the CP provides an idea of how much knowledge is required for each aspect of the project.

The need to know a project's complexity arises in two instances: before starting it and once execution begins.

Before Starting the Project

Before commencing work on a new project, we can comprehend its complexity using the process detailed here.

Figure 1 presents the initial Complexity Profile (CP) of our project, representing the perplexity of both Capability and Client about the project as a whole.

Figure 1: Complexity Profile (CP)

Upon establishing the initial complexity profile of our project, we can manage the project's complexity (knowledge gaps) through the following steps:

  1. Identify knowledge gaps: Determine areas where Client or Capability may lack expertise or where knowledge gaps may exist. This facilitates planning for additional training, research, or external support to bridge these gaps.
  2. Compare with past projects: Review completed projects with similar complexity levels and identify patterns or commonalities. This helps better understand potential challenges, risks, and opportunities your team may encounter during the project and how to address them.

Starting the Project

For continuously understanding knowledge gaps during our project's execution, we will use the process explained in detail here.

At the end of the first iteration of this process, we will have a detailed operational complexity profile. Figure 2 presents the operational Complexity Profile (CP) of our project, representing the perplexity of both Capability and Client about the user stories.

Figure 2: Operational Complexity Profile (CP)

Filling the knowledge gaps

"Lack of knowledge…that is the problem. You should not ask questions without knowledge. If you do not know how to ask the right question, you discover nothing." ~ W. Edwards Deming

An operational complexity profile is a maneuverable space. Each movement is a clarification. Clarification is a knowledge discovery process. All possible movements between domains are presented below.

Figure 3: Movements between domains
Each clarification represents accumulation of knowledge about what is the problem and how to solve it.

For example, if we want to move a user story from cell CC to cell AA we have several alternative routes:

  • Linearly CC -> CA -> AA which means we clarify first from capability then from client perspective
  • Linearly CC -> AC -> AA, which means we clarify first from client then from capability perspective
  • Diagonally from CC to AA, which means we clarify both from capability and from client perspectives

We also see the transition states between the cells. They are presented in stripes. An user story is in a transition state when it has left a cell but hasn't got into a new cell yet. That means work is being done to clarify the user story.

To clarify a complex user story we employ numerous experiments in parallel which will provide the missing information we need.

Each experiment should be:

  • Coherent - to the individual and not the group as a whole, although the group must accept that it is a valid view point.
  • Safe-to-fail. That means if it fails we'll be able to recover.
  • Finely grained, tangible
To agree that our hypothesis is coherent is not the same as to agree that it is true. Any coherent hypothesis about how things could work gets a small amount of resources to run a short-cycle, safe-to-fail experiment. We manage the experiments as a portfolio. Some of the experiments must be oblique, some naïve, some contradictory, some high-risk/high-return types. When we run several probes in parallel, they will change the environment anyway so the logic is that all experiments should be roughly of the same size and take roughly the same time. In practice those experiments generally merge, mutate or fail and a solution or solutions emerge.

As an example, here is a set of axes for experiments that could provide enough knowledge for both Client and Capability perspectives:

  • The particular technologies to be used. That is very common in software development where technologies are constantly changing and maturing.
  • The way of articulating the problem to be solved — a better model could make the solution obvious.
  • The people from the client company to build relationships with.
  • The people in the team — their aspirations or fears, their motivation, their relationships with one another and out into the wider organization.
  • The client company's organizational constraints i.e. policies, rules, procedures.
  • The culture of the client company.
  • The third party dependencies and associated risks.
An experiment on one of the axes will affect the other axes. There might be a concern that there are insufficient coherent hypotheses, or more importantly insufficient contradictions. If that is the case then we should be concerned that we may have insufficient range of experiments and may be at risk of missing a key weak signal, or signals.

For example, if from a Capability perspective a user story is complex we may decide to do a Proof of Concept (PoC). We do that by preparing a concept with a couple of alternatives. Then the card will follow CC -> AC, and stay in transition until the PoC provides some missing information. If the results from the PoC have narrowed the perplexity we have made the user story complicated and the card will make AC -> AA. Another example, if a user story is complex from client perspective then the Capability could work with the Client by presenting alternatives. This is what software development experts do - provide alternatives not only about how to do but also what to do. When providing alternatives to the client about what to be done we have the chance to move the work through the CP in a direction that best suits our capabilities. One may say that clients who don't know what they want are the best!

Knowledge discovery funnel

"The organization and the people within it need to discover a quantity of knowledge that they do not have, and factor that knowledge into something that works." ~ Phillip Glen Armour[8]

We see that the complexity profile serves us as a map to show how we can move to reach our target level of perplexity. It helps us act in order to move toward reduced perplexity.

This dynamic can be presented in a simpler way as a movement through a funnel as shown on Figure 4.

Figure 4: Knowledge discovery funnel

We have rotated the complexity profile 45 degrees so that the lines of movement from complex to complicated and from complicated to clear are horizontal.

Only complicated and clear user stories can exit the funnel.

Clear stories let delivery to focus on capturing and codifying prior knowledge.

The flow through the funnel should not be in a rush. Especially if the knowledge gaps are numerous, complex, and interrelated. It must enable efficient and effective learning to close the identified knowledge gaps. Otherwise, bad decisions would be made and cause rework that would be very costly to the project. Delaying decisions will help you deal with making critical decisions with insufficient knowledge and simultaneously keep multiple options open until knowledge gaps are closed.

Delay critical decisions until the knowledge is discovered.

Synthesis

In philosophy, synthesis refers to the process of combining different concepts to create a new, more comprehensive understanding of reality.

  • Objects (substances) refer to the basic building blocks of reality. They are independent entities that exist and persist through time, and have properties that determine their nature and identity.
  • Properties, on the other hand, are qualities or attributes that belong to objects. They are what make objects what they are and distinguish them from other objects. Examples of properties include color, shape, size, and texture.
  • Relations refer to the ways in which objects are connected to one another. Relations can be between objects or between objects and properties. Examples of relations include causality, spatiotemporal proximity, and functional dependency.

By considering the objects, properties, and relations that exist in the problem domain, software developers can synthesize a more comprehensive solution that better captures the underlying reality of the problem.

In the first step of synthesis we take the thing we want to understand as a part of a larger whole, hence it is put together with other things, not taken apart. We need to consider the parts not on their own, but in relation to the whole. We need to identify the contours of a system.

“Matter tells space how to curve, and curved space tells matter how to move.” ~ John Wheeler
We describe the space itself, not its features. Features of the system will emerge from the space if you did this. In the second step of synthesis, we explain the behavior of the larger whole. So if we are trying to understand a business, we have to first explain the market system the business is a part of. In the third step of synthesis, the behavior or properties of a part is explained by revealing its role or function in the system that contains it.

Synthesis is the process of making complex things complicated.

As an example, consider the problem domain of a new e-commerce web site. The objects in this domain include products, customers, orders, and payment methods. These objects have properties such as product name, price, quantity, customer name, address, payment type, and so on. The relations between these objects include:

  • Products and customers: A customer can purchase one or more products.
  • Products and orders: An order contains one or more products.
  • Orders and customers: An order is made by a customer.
  • Orders and payment methods: An order requires a payment method to be completed.

By considering the objects, properties, and relations in this problem domain, a software developer can synthesize a new e-commerce web site that better captures the underlying reality of the problem. For example, the developer may decide to implement a feature that allows customers to easily view their purchase history, based on the relation between orders and customers. Or the developer may decide to implement a feature that displays related products, based on the relation between products.

Analysis

Analysis is a systematic and detailed examination of a subject or issue, with the aim of breaking it down into its component parts, understanding their relationships and interconnections, and examining their properties and characteristics. The goal of analysis is to gain a deeper understanding of the subject being studied and to identify patterns, structures, and relationships that are not immediately apparent.

In the context of software development, analysis refers to the process of evaluating and breaking down a problem or system into its component parts in order to understand its requirements, constraints, and design considerations. This information can then be used to inform the design and development of a new system or solution that better meets the needs and requirements of the problem domain.

Analysis means obtaining answers to known questions.

Using analysis we make the complicated user stories clear or pick one of the good alternatives.

Let's consider the objects "Products" and "Customers" in the problem domain of an e-commerce web site, and the relation between them, which is "A customer can purchase one or more products."

To analyze these obkects and their relation, we can start by asking questions about their properties and the ways in which they interact. For example:

  • What properties do products have? Some properties may include name, description, price, image, category, brand, and so on.
  • What properties do customers have? Some properties may include name, email, address, payment information, purchase history, and so on.
  • How does the relation between products and customers affect the design of the e-commerce web site? The relation between products and customers may influence the way that the site is structured, the features that it provides, and the information that is displayed to users.

Based on this analysis, we can synthesize a solution that better captures the underlying reality of the problem domain. For example, the solution may include features such as:

  • A product catalog that allows customers to easily browse and search for products.
  • A customer account system that stores and manages customer information, such as purchase history and payment information.
  • A checkout process that allows customers to purchase products and manage their orders.

By considering the objects, properties, and relations in the problem domain, and synthesizing a solution based on this analysis, we can create an e-commerce web site that provides a better experience for customers and meets their needs more effectively.

Synthesis vs. Analysis

Analysis yields information about the structure of a system, and how it works. That's knowledge, know-how, not understanding. If we apply analysis to a system, we begin by breaking it apart and we lose all of its essential properties. We cannot explain the behavior of a system by analysis. Analysis cannot produce understanding of systems. We cannot examine a system by looking at its parts, you must look at it as a part of a larger whole. We can reveal its structure and all individual actions, but we can't say why it works the way it does. Explanation never lies inside the system, it lies outside.

Unfortunately, analysis and thought are frequently treated as synonyms, but analysis is only one way of thinking; synthesis is another. We should use the synthesis to achieve a whole system's perspective. Synthesis does not mean “studying a whole as a synthesis of its parts” but “studying a whole as part of a larger system.” Synthesis yields understanding, analysis yields knowledge, and it was that distinction that was critical for the emergence of the systems thinking. It uses both, but to understand systems, particularly those that involve people, synthetic thinking is required.

Discovery kanban board

For our products we strive to have the proper combination of complex, complicated and clear options.

We can manage the movement of user stories on a kanban board as presented on Fig. 16.

Figure 5: Discovery kanban board

On the board we have the Knowledge discovery funnel as the Discovery kanban board[4]. The requests for new things to be developed enter here. We'll call them options in order to show that it is not mandatory for them to be delivered at all.

Options become promises after the Commitment point. Options may be abandoned but promises must be delivered.

Cards in orange represent complex options. Each complex option needs to go through synthesis and analysis in order to enter the Backlog part of the board. Complex options enter the funnel and get out as either complicated or clear following the below workflow on Figure 6.

Figure 6: Knowledge Discovery process

Here is how the process works. First step is to take one option and categorize it. Categorization means we place it at the proper cell in the complexity profile of the Discovery board. As we saw earlier the complexity profile acts like a funnel.

Cards in green are clear options and can directly enter delivery.

Cards in amber are complicated options. It is not mandatory for all complicated options to become clear. A complicated option may enter delivery when it is considered that the analysis can be done during the delivery. That could happen for two reasons:

  1. When an option is clear from client perspective and complicated from capability perspective because there are only two alternatives how to implement it. The choice of an alternative can be left to the delivery system to make.
  2. When the option is complicated from client perspective because there are small details to be decided e.g. text message content, colors etc. Those can be cleared during delivery.

An option cannot enter the delivery system directly from Synthesis. A complex option should never exit the funnel, because it will clutter the delivery process. The delivery system is not supposed to do the work of the discovery system, because the delivery system has to work only on clear goals.

It is also possible on the complexity profile cards to move “backwards” say from complicated to complex. For example, we have a complicated option with a couple of alternatives about what to do. The client is considering which alternative to choose when new information from the market arrives and suddenly none of the alternatives is viable anymore. The client is back to the spot where they don't know what they want. The option is again complex.

Kanban system

The discovery part of the board which manages the Knowledge discovery funnel in a kanban system on its own.

The WIP limits are set at the top of each column. The numbers show the minimum number of options that need to be present at any given time. For instance, the column Synthesis has to have at least 20 options. They could be a hundred but never less than twenty. We see that the minimum values get smaller as we move to Analysis and Clear. This is because we are operating a funnel. We need a lot of options to enter the funnel because only a fraction of them will leave it as delivery promises. We see that the funnel starts at 20, then becomes 10 and ends in 2-5 options. Why do we want to have 20 complex and only between 2 and 5 clear options? Because clear things are obvious for everybody else and hence not valuable to work on.

We have a pull system on the Discovery side. That means if we don't have at least 20 complex options we have to do some work and find new ones. But the low limit on Clear will prevent us from picking easy-to-find but non-valuable clear options. We don't want non-valuable, clear options to flood the delivery system with non-valuable delivery requests.

Experiments

Both analysis and synthesis of an option may require some experiments or probes e.g. a Proof of Concept (PoC).

How do we show on the board that we probe for an option? We put the option card in a transition cell between CC and AA on the Discovery board. The experiments are tracked on a Experiments kanban board. If the experiment provides the missing information then the option card is moved into an AA cell on the Discovery board. If not then it stays in transition. It could be the case that it is decided to abandon the option. That is presented as the “Discarded options” area of the kanban board.

Action and performance feedback

After a clear goal for a user story has been established i.e. what to do and how to do it, it is important to ensure that each individual developer has a balance between their abilities and the complexity of the user story. This balance should be on a personal, individual level, with a slight tilt towards challenges. To achieve this, we aim to have a missing information between 2 and 4 bits, which corresponds to user stories that are considered "complicated" and require some analysis by the developers.

To ensure that the balance is achieved, it is important to have a way to measure the complexity of a user story and personal abilities using bits of information. However, measuring the perplexity a developer feels when looking at a user story is difficult, if not impossible.

One way to evaluate whether a developer's abilities and work complexity were in balance is by measuring the Knowledge Discovery Efficiency (KEDE) after the work has been completed, as explained here By analyzing KEDE, we can determine whether the project management process is making developers happy. If it is not, we can make changes to our process accordingly.

Development kanban board

We can visualize and manage the Knowledge Application work on a kanban board.

Figure 7: Development kanban board

Experiments kanban board

How do we show on the board that we probe for an option? For that we use the Experiments kanban board presented at the bottom. This kanban board can be physically detached from the Delivery kanban board. A new experiment card is placed in the Problems column in the lower left hand corner of Figure 7. The experiment card follows the Problems - Options - Experiment - Review path.

Here is how the experiments are worked and tracked on the Experiments kanban board. We can use a great approach called POPCORN Flow[5].

Once we pick the problem we plan to focus on, we move the card to the “Options” column, which is next to the right of the “Problems” column. It is unlikely that a problem is so simple that there is only one option available to solve it. Thus, the problem that made its way into the "Options" column should be split into several possible options for solving it.

We then choose one option that seems to promise the best outcome. We move that option to the "Experiment" column to the right. Each experiment should be defined with three attributes:

  1. Action: which action is the experiment proposing?
  2. Duration: how long is the experiment expected to run?
  3. Expectation: What is the expected desired outcome of the experiment?

We can have additional columns like for the committed, worked on and finished experiments. We don't show that here because that is another large topic.

When the experiment is finished, it moves to the column to the right – “Review”. During the review, we ask some questions, such as:

  • What did we expect to happen (i.e., the hypothesis)?
  • What had actually happened?
  • What did we learn?
  • What opportunity do we perceive?
If the review provides enough information for the experiment card then the parent option card is moved into Analysis cell on the Discovery board.

The problems, options and experiments must obey the constraints of the Work in Progress (WiP) for the delivery team.

End-to-end kanban board

We can manage the movement of user stories on a End-to-End kanban board as presented on Figure 8.

Figure 8: End-to-End kanban board

The left hand Discovery part of the board tracks exploring options. The right hand Delivery part of the board tracks promises to deliver.

On the board an option can leave the discovery part and enter the delivery part either from Analysis or from Clear. Cards in green are clear and can directly enter delivery.

Let's not forget that in the delivery part we are dealing with promises. Promises should not be canceled because the delivery system has a limited and expensive capacity. In the discovery part we are dealing with options. Options by definition can be discarded. That is presented as the “Discarded options” area of the kanban board.

Staffing

The process presented here would shine a bright, glaring light on what people did not know. This could quite uncomfortable for many and they could responded as many of us would do, by avoiding it altogether and working on something on which they could "make progress," on something they knew. For most of us, given a straight choice of working on something that is easy, something that plays to our strengths and demonstrates how competent we are, and working on something that is hard, something we have no clue about, and ultimately shows us and everyone else just how little we understand about the system, we are probably going to choose the former[8]. This is not only a natural, human tendency; it is also exacerbated by some of the pressures we experience in development, the desire to demonstrate "progress" and most of all by the organization's and management's expectation of producing a product[8].

The bottom line is always that processes do not do the work, people do. Not only do they come with their own proprietary thought processes and experience, they also come with a host of other characteristics: motivation, ethical basis, fortitude, helpfulness, resourcefulness, rapidity of thought, capacity to give and receive feedback, etc. These characteristics may be skills or they may be capabilities. These are not the same thing. People are created differently and through application of their skills, through life experience, and through their inclinations, end up with different capabilities. But the source of the problem is not in differences. In fact, it is the differences rather than the similarities that make project teams effective[8].

We need different people to work on different entries in the complexity profile. The difference is the level of perplexity they can handle both from Capability and Client perspectives. For instance:

  • if the problem is clear then we need a junior person, who can read a recipe book and follow it. She will need stable requirements and all the exact tools and technologies from the book.
  • If the problem is complex we need a senior person, who can cope without a recipe, with changing requirements and with whatever is available in terms of tools and technologies.

When we need to adapt we need more diversity in the system because then we've got more evolutionary capacity. It's not a coincidence that for the living organisms the mutation rates increase under stress - more diversity more, adaptability. The requisite diversity is the optimal level of diversity for the context we are in. Diversity means inefficiency an organization needs to have. If overfocused on efficiency the organization loses adaptive capacity.

Diversity can help us answer the question posed by Brooks' Law[9]. The question is, "if I have a project that is running late, should I add people to it to speed it up?" Our answer is, "it depends on how you manage knowledge discvery." If the project scope is partitioned into work items classified using the Complexity Profile explained here, then you'll have Clear and Complicated work that can be given to the newcomers. That will free up your present developers to do more Complex work clarifying. Then the answer is "yes." The answer would be "No" if the addition of people will require that your current staff take time away from being productive to instruct the newcomers, resulting in a net drop in productivity. Hence, Brooks' Law: adding manpower to a late project makes it later.

Initial vs. Continuous Discovery

The Initial Knowledge Discovery runs without Delivery i.e Knowledge application. It may result in a Product Vision, Product Strategy, and initially validated Business Model that has a chance of achieving the Product-Makret Fit. The Initial Knowledge Discovery can serve as a base for investment decisions.

After the Initial Knowledge Discovery we establish a regular feedback loop to continuously revisit and update the categorization of the user stories in progress and categorize the newly added user stories.

It needs to be emphasized that we don't get a full process specification, and analyze it all before starting delivery. Instead, we do the Initial Knowledge Discovery oly to have the categories for the project defined. For that we need a small subset of the requirements for the project. After that we run a continuous iterative process that flows through Knowledge Discovery and Delivery user stories in small batches.

If the project team is a Scrum team then the batch could be a Sprint backlog.

Product management

The Knowledge Discovery can be run by Product managers and Product owners. Then it can be called Product Discovery.

For each option in the Product Options part of the End-to-end board the following questions need to be asked.

  1. Value. Will it create value for the customers?
  2. Usability. Will users figure out how to use it?
  3. Viability. Can our business support it?
  4. Feasibility. Can it be done?
  5. Ethic. Should we do it?

Dual-Track Agile

Jeff Patton and others in the Agile product space have been big proponents of an approach called Dual-Track Development or Dual-Track Agile[6]. In that setup, there are two streams that run in parallel:

  1. Product Discovery - to discover the product to build
  2. Product Delivery - to deliver that product to the market
Product Discovery results in a validated Product Backlog. In particular, high-risk assumptions are tested before the implementation. We use Product discovery work to identify real customer needs and really test that we’ll get the outcome we hope for[7].

The difference with the approach presented in this article is that here we don't have two activities in parallel. Instead we have two activities Discovery and Deliver running sequentially, but at their own pace and with their own variation.

Works Cited

1. Bakardzhiev, D. (2016). Adaptable or Predictable? Strive for Both - Be Predictably Adaptable! InfoQ. https://www.infoq.com/articles/predictably-adaptable/

2. Schaffer, O. 2013. Crafting Fun User Experiences: A Method to Facilitate Flow. Human Factors International Whitepaper. Retrieved from: https://scholar.google.com/scholar?cluster=9760324565241258858

3. Anderson, D. J., & Carmichael, A. (2016). Essential Kanban Condensed. Lean-Kanban University.

4. Steyaert, P. (2018). Essential Upstream Kanban (Illustrated edition). Lean-Kanban University.

5. Q&A with Claudio Perrone on PopcornFlow

6. Dual Track Development is not Duel Track

7. The Mindset That Kills Product Thinking

8. Armour, P.G. (2003). The Laws of Software Process, Auerbach

9. Brooks, F. P. (1995). The mythical man-month: Essays on software engineering. Addison-Wesley

How to cite:

Bakardzhiev D.V. (2022) Improve productivity and happiness on a software development project. https://docs.kedehub.io/kede-manage/knowledge-discovery-management.html

Getting started