Improve productivity and happiness on a software development project

Flow

“A bad system will beat a good person every time.” —W. Edwards Deming

Software development is a process of going from 0% knowledge about an idea to 100% knowledge of a finished product ready for delivery to a customer. In its simplest form, the job of software development is to reduce the number of unknowns to zero by the time the system is delivered. It is a process of knowledge acquisition and ignorance reduction[8]. Hence we model the software development process as a knowledge discovery process. The goal is to iteratively identify knowledge gaps, reduce our perplexity, increase our knowledge and close the identified knowledge gaps.

It is also a continuous process rather than a punctuated process. This means that moment by moment on the project, ignorance is identified, questions are asked, answers are obtained and validated, and knowledge is discovered[8]. We need to find the knowledge to be discovered for each user story. That knowledge is the difference between the missing information and our prior knowledge.

Quite commonly, we underestimate how much knowledge we have to discover to deliver the functionality to the customer, or overestimate our prior knowledge, which is the same thing.

People in general and software developers in particular are happy when working in a state of Flow. Flow experiences lead to an increase in productivity. When the flow conditions have been met they create a flow loop between action and feedback that allows for continuous and effortless tuning of performance while taking action. Flow loops make an activity worth doing for its own sake[2].

Entering flow depends on establishing a balance between perceived individual skills and perceived task challenges. Said in another way - capability should match work complexity.

After that the developer must receive immediate feedback that allows them to continuously adjust their performance as they tackle these challenges. This kind of feedback communicates how well they are performing and how they can improve their performance.

By measuring how well project policies and processes meet the flow conditions we can engineer a project organization that facilitates flow.

From a project management perspective there are two areas we need to create.

1. Conditions for flow. That means breaking down the project into disjoint units of work. For each task the three flow conditions should be in place.
2. Action and performance feedback. Tasks distribution so that each of the developers has a proper challenge for their skills. A feedback mechanism should be in place to tell developers how well they are doing.

Creating conditions for Flow

“The goal is to make full use of the workers' capabilities by building up a system that will allow the workers to display their full capabilities by themselves.” ~ Y. SUGIMORI, K. KUSUNOKI, F. CHO & S. UCHIKAWA (1977) Toyota production system and Kanban system

In order to have conditions for flow for each user story we need to find a balance between their perceived individual capabilities and the complexity. This balance should be tailored to the individual developer, with a slight preference for challenging tasks. To determine this balance, an assessment of the developer's individual capabilities and the complexity of the task is necessary. This assessment requires accurate and reliable information about both of these factors.

There are two types of knowledge that a software developer must discover in order to complete a task:

• Knowledge to be discovered on what to do: This refers to understanding the task and its objectives
• Knowledge to be discovered on how to do it: This refers to knowing the methods and having the skills needed to complete the task

We refer to the developer's perception of the task complexity as "required knowledge." Bear in mind that the perception of knowledge to be discovered may vary among developers and could evolve over time for a single developer as they refine their skills and task understanding. It is also important to recognize that a developer's perception of required knowledge may not always align perfectly with the actual requirements of the task, and it may be necessary to provide additional training or support to bridge any gaps.

Prior knowledge on what to do and how to do it has already been discovered. This "prior knowledge" provides a starting point for the software developer, but additional knowledge must still be acquired in order to complete the task.

If there was a balance between individual capability and work complexity can be quantified by measuring the knowledge discovered after successful work completion.

It is more convenient to represent the four combinations of required and prior knowledge in order to complete a task in a matrix, which we will refer to as Complexity Profile (CP), which is explained here.

The diagram is intended to help visualize the balance between individual capabilities and work complexity that is necessary for a developer to be in a state of flow when working on a task. Each of the four subspaces in the diagram represents a combination of required and prior knowledge, which we refer to as knowledge to be discovered, or the information the developer needs to gain in order to complete a task.

The less knowledge that needs to be discovered, the happier the developer will be. However, it is important for there to be some knowledge that needs to be discovered in order to maintain interest and engagement.

To achieve a balance that is on a personal, individual level, with a slight tilt towards challenges, the developer needs to pick a task with a required-prior knowledge combination from one of the yellow quadrants that has a missing information between 2 and 4 bits. That means, there are up to 16 alternatives and it requires at least 1 and up to 4 questions asked in a row, in sequence, linearly.

Filling the knowledge gaps

"Lack of knowledge…that is the problem. You should not ask questions without knowledge. If you do not know how to ask the right question, you discover nothing." ~ W. Edwards Deming

An operational complexity profile is a maneuverable space. Each movement is a clarification. Clarification is a knowledge discovery process. All possible movements between domains are presented below.

Each clarification represents accumulation of knowledge about what is the problem and how to solve it.

For example, if we want to move a user story from cell CC to cell AA we have several alternative routes:

• Linearly CC -> CA -> AA which means we clarify first from capability then from client perspective
• Linearly CC -> AC -> AA, which means we clarify first from client then from capability perspective
• Diagonally from CC to AA, which means we clarify both from capability and from client perspectives

We also see the transition states between the cells. They are presented in stripes. An user story is in a transition state when it has left a cell but hasn't got into a new cell yet. That means work is being done to clarify the user story.

To clarify a complex user story we employ numerous experiments in parallel which will provide the missing information we need.

Each experiment should be:

• Coherent - to the individual and not the group as a whole, although the group must accept that it is a valid view point.
• Safe-to-fail. That means if it fails we'll be able to recover.
• Finely grained, tangible

As an example, here is a set of axes for experiments that could provide enough knowledge for both Client and Capability perspectives:

• The particular technologies to be used. That is very common in software development where technologies are constantly changing and maturing.
• The way of articulating the problem to be solved — a better model could make the solution obvious.
• The people from the client company to build relationships with.
• The people in the team — their aspirations or fears, their motivation, their relationships with one another and out into the wider organization.
• The client company's organizational constraints i.e. policies, rules, procedures.
• The culture of the client company.
• The third party dependencies and associated risks.

For example, if from a Capability perspective a user story is complex we may decide to do a Proof of Concept (PoC). We do that by preparing a concept with a couple of alternatives. Then the card will follow CC -> AC, and stay in transition until the PoC provides some missing information. If the results from the PoC have narrowed the perplexity we have made the user story complicated and the card will make AC -> AA. Another example, if a user story is complex from client perspective then the Capability could work with the Client by presenting alternatives. This is what software development experts do - provide alternatives not only about how to do but also what to do. When providing alternatives to the client about what to be done we have the chance to move the work through the CP in a direction that best suits our capabilities. One may say that clients who don't know what they want are the best!

Knowledge discovery funnel

"The organization and the people within it need to discover a quantity of knowledge that they do not have, and factor that knowledge into something that works." ~ Phillip Glen Armour[8]

We see that the complexity profile serves us as a map to show how we can move to reach our target level of perplexity. It helps us act in order to move toward reduced perplexity.

This dynamic can be presented in a simpler way as a movement through a funnel as shown on Figure 4.

We have rotated the complexity profile 45 degrees so that the lines of movement from complex to complicated and from complicated to clear are horizontal.

Only complicated and clear user stories can exit the funnel.

Clear stories let delivery to focus on capturing and codifying prior knowledge.

The flow through the funnel should not be in a rush. Especially if the knowledge gaps are numerous, complex, and interrelated. It must enable efficient and effective learning to close the identified knowledge gaps. Otherwise, bad decisions would be made and cause rework that would be very costly to the project. Delaying decisions will help you deal with making critical decisions with insufficient knowledge and simultaneously keep multiple options open until knowledge gaps are closed.

Delay critical decisions until the knowledge is discovered.

Synthesis

In philosophy, synthesis refers to the process of combining different concepts to create a new, more comprehensive understanding of reality.

• Objects (substances) refer to the basic building blocks of reality. They are independent entities that exist and persist through time, and have properties that determine their nature and identity.
• Properties, on the other hand, are qualities or attributes that belong to objects. They are what make objects what they are and distinguish them from other objects. Examples of properties include color, shape, size, and texture.
• Relations refer to the ways in which objects are connected to one another. Relations can be between objects or between objects and properties. Examples of relations include causality, spatiotemporal proximity, and functional dependency.

By considering the objects, properties, and relations that exist in the problem domain, software developers can synthesize a more comprehensive solution that better captures the underlying reality of the problem.

In the first step of synthesis we take the thing we want to understand as a part of a larger whole, hence it is put together with other things, not taken apart. We need to consider the parts not on their own, but in relation to the whole. We need to identify the contours of a system.

“Matter tells space how to curve, and curved space tells matter how to move.” ~ John Wheeler

We describe the space itself, not its features. Features of the system will emerge from the space if you did this. In the second step of synthesis, we explain the behavior of the larger whole. So if we are trying to understand a business, we have to first explain the market system the business is a part of. In the third step of synthesis, the behavior or properties of a part is explained by revealing its role or function in the system that contains it.

Synthesis is the process of making complex things complicated.

As an example, consider the problem domain of a new e-commerce web site. The objects in this domain include products, customers, orders, and payment methods. These objects have properties such as product name, price, quantity, customer name, address, payment type, and so on. The relations between these objects include:

• Products and customers: A customer can purchase one or more products.
• Products and orders: An order contains one or more products.
• Orders and customers: An order is made by a customer.
• Orders and payment methods: An order requires a payment method to be completed.

By considering the objects, properties, and relations in this problem domain, a software developer can synthesize a new e-commerce web site that better captures the underlying reality of the problem. For example, the developer may decide to implement a feature that allows customers to easily view their purchase history, based on the relation between orders and customers. Or the developer may decide to implement a feature that displays related products, based on the relation between products.

Analysis

Analysis is a systematic and detailed examination of a subject or issue, with the aim of breaking it down into its component parts, understanding their relationships and interconnections, and examining their properties and characteristics. The goal of analysis is to gain a deeper understanding of the subject being studied and to identify patterns, structures, and relationships that are not immediately apparent.

In the context of software development, analysis refers to the process of evaluating and breaking down a problem or system into its component parts in order to understand its requirements, constraints, and design considerations. This information can then be used to inform the design and development of a new system or solution that better meets the needs and requirements of the problem domain.

Analysis means obtaining answers to known questions.

Using analysis we make the complicated user stories clear or pick one of the good alternatives.

Let's consider the objects "Products" and "Customers" in the problem domain of an e-commerce web site, and the relation between them, which is "A customer can purchase one or more products."

To analyze these obkects and their relation, we can start by asking questions about their properties and the ways in which they interact. For example:

• What properties do products have? Some properties may include name, description, price, image, category, brand, and so on.
• What properties do customers have? Some properties may include name, email, address, payment information, purchase history, and so on.
• How does the relation between products and customers affect the design of the e-commerce web site? The relation between products and customers may influence the way that the site is structured, the features that it provides, and the information that is displayed to users.

Based on this analysis, we can synthesize a solution that better captures the underlying reality of the problem domain. For example, the solution may include features such as:

• A product catalog that allows customers to easily browse and search for products.
• A customer account system that stores and manages customer information, such as purchase history and payment information.
• A checkout process that allows customers to purchase products and manage their orders.

By considering the objects, properties, and relations in the problem domain, and synthesizing a solution based on this analysis, we can create an e-commerce web site that provides a better experience for customers and meets their needs more effectively.

Discovery kanban board

For our products we strive to have the proper combination of complex, complicated and clear options.

We can manage the movement of user stories on a kanban board as presented on Fig. 16.

On the board we have the Knowledge discovery funnel as the Discovery kanban board[4]. The requests for new things to be developed enter here. We'll call them options in order to show that it is not mandatory for them to be delivered at all.

Options become promises after the Commitment point. Options may be abandoned but promises must be delivered.

Cards in orange represent complex options. Each complex option needs to go through synthesis and analysis in order to enter the Backlog part of the board. Complex options enter the funnel and get out as either complicated or clear following the below workflow on Figure 6.

Here is how the process works. First step is to take one option and categorize it. Categorization means we place it at the proper cell in the complexity profile of the Discovery board. As we saw earlier the complexity profile acts like a funnel.

Cards in green are clear options and can directly enter delivery.

Cards in amber are complicated options. It is not mandatory for all complicated options to become clear. A complicated option may enter delivery when it is considered that the analysis can be done during the delivery. That could happen for two reasons:

1. When an option is clear from client perspective and complicated from capability perspective because there are only two alternatives how to implement it. The choice of an alternative can be left to the delivery system to make.
2. When the option is complicated from client perspective because there are small details to be decided e.g. text message content, colors etc. Those can be cleared during delivery.

An option cannot enter the delivery system directly from Synthesis. A complex option should never exit the funnel, because it will clutter the delivery process. The delivery system is not supposed to do the work of the discovery system, because the delivery system has to work only on clear goals.

It is also possible on the complexity profile cards to move “backwards” say from complicated to complex. For example, we have a complicated option with a couple of alternatives about what to do. The client is considering which alternative to choose when new information from the market arrives and suddenly none of the alternatives is viable anymore. The client is back to the spot where they don't know what they want. The option is again complex.

Kanban system

The discovery part of the board which manages the Knowledge discovery funnel in a kanban system on its own.

The WIP limits are set at the top of each column. The numbers show the minimum number of options that need to be present at any given time. For instance, the column Synthesis has to have at least 20 options. They could be a hundred but never less than twenty. We see that the minimum values get smaller as we move to Analysis and Clear. This is because we are operating a funnel. We need a lot of options to enter the funnel because only a fraction of them will leave it as delivery promises. We see that the funnel starts at 20, then becomes 10 and ends in 2-5 options. Why do we want to have 20 complex and only between 2 and 5 clear options? Because clear things are obvious for everybody else and hence not valuable to work on.

We have a pull system on the Discovery side. That means if we don't have at least 20 complex options we have to do some work and find new ones. But the low limit on Clear will prevent us from picking easy-to-find but non-valuable clear options. We don't want non-valuable, clear options to flood the delivery system with non-valuable delivery requests.

Action and performance feedback

After a clear goal for a user story has been established i.e. what to do and how to do it, it is important to ensure that each individual developer has a balance between their abilities and the complexity of the user story. This balance should be on a personal, individual level, with a slight tilt towards challenges. To achieve this, we aim to have a missing information between 2 and 4 bits, which corresponds to user stories that are considered "complicated" and require some analysis by the developers.

To ensure that the balance is achieved, it is important to have a way to measure the complexity of a user story and personal abilities using bits of information. However, measuring the perplexity a developer feels when looking at a user story is difficult, if not impossible.

One way to evaluate whether a developer's abilities and work complexity were in balance is by measuring the Knowledge Discovery Efficiency (KEDE) after the work has been completed, as explained here By analyzing KEDE, we can determine whether the project management process is making developers happy. If it is not, we can make changes to our process accordingly.

Development kanban board

We can visualize and manage the Knowledge Application work on a kanban board.

End-to-end kanban board

We can manage the movement of user stories on a End-to-End kanban board as presented on Figure 8.

The left hand Discovery part of the board tracks exploring options. The right hand Delivery part of the board tracks promises to deliver.

On the board an option can leave the discovery part and enter the delivery part either from Analysis or from Clear. Cards in green are clear and can directly enter delivery.

Let's not forget that in the delivery part we are dealing with promises. Promises should not be canceled because the delivery system has a limited and expensive capacity. In the discovery part we are dealing with options. Options by definition can be discarded. That is presented as the “Discarded options” area of the kanban board.

Staffing

The process presented here would shine a bright, glaring light on what people did not know. This could quite uncomfortable for many and they could responded as many of us would do, by avoiding it altogether and working on something on which they could "make progress," on something they knew. For most of us, given a straight choice of working on something that is easy, something that plays to our strengths and demonstrates how competent we are, and working on something that is hard, something we have no clue about, and ultimately shows us and everyone else just how little we understand about the system, we are probably going to choose the former[8]. This is not only a natural, human tendency; it is also exacerbated by some of the pressures we experience in development, the desire to demonstrate "progress" and most of all by the organization's and management's expectation of producing a product[8].

The bottom line is always that processes do not do the work, people do. Not only do they come with their own proprietary thought processes and experience, they also come with a host of other characteristics: motivation, ethical basis, fortitude, helpfulness, resourcefulness, rapidity of thought, capacity to give and receive feedback, etc. These characteristics may be skills or they may be capabilities. These are not the same thing. People are created differently and through application of their skills, through life experience, and through their inclinations, end up with different capabilities. But the source of the problem is not in differences. In fact, it is the differences rather than the similarities that make project teams effective[8].

We need different people to work on different entries in the complexity profile. The difference is the level of perplexity they can handle both from Capability and Client perspectives. For instance:

• if the problem is clear then we need a junior person, who can read a recipe book and follow it. She will need stable requirements and all the exact tools and technologies from the book.
• If the problem is complex we need a senior person, who can cope without a recipe, with changing requirements and with whatever is available in terms of tools and technologies.

When we need to adapt we need more diversity in the system because then we've got more evolutionary capacity. It's not a coincidence that for the living organisms the mutation rates increase under stress - more diversity more, adaptability. The requisite diversity is the optimal level of diversity for the context we are in. Diversity means inefficiency an organization needs to have. If overfocused on efficiency the organization loses adaptive capacity.

Diversity can help us answer the question posed by Brooks' Law[9]. The question is, "if I have a project that is running late, should I add people to it to speed it up?" Our answer is, "it depends on how you manage knowledge discvery." If the project scope is partitioned into work items classified using the Complexity Profile explained here, then you'll have Clear and Complicated work that can be given to the newcomers. That will free up your present developers to do more Complex work clarifying. Then the answer is "yes." The answer would be "No" if the addition of people will require that your current staff take time away from being productive to instruct the newcomers, resulting in a net drop in productivity. Hence, Brooks' Law: adding manpower to a late project makes it later.

Initial vs. Continuous Discovery

The Initial Knowledge Discovery runs without Delivery i.e Knowledge application. It may result in a Product Vision, Product Strategy, and initially validated Business Model that has a chance of achieving the Product-Makret Fit. The Initial Knowledge Discovery can serve as a base for investment decisions.

After the Initial Knowledge Discovery we establish a regular feedback loop to continuously revisit and update the categorization of the user stories in progress and categorize the newly added user stories.

It needs to be emphasized that we don't get a full process specification, and analyze it all before starting delivery. Instead, we do the Initial Knowledge Discovery oly to have the categories for the project defined. For that we need a small subset of the requirements for the project. After that we run a continuous iterative process that flows through Knowledge Discovery and Delivery user stories in small batches.

If the project team is a Scrum team then the batch could be a Sprint backlog.