How to Measure Developer Cognitive Load

What is Cognitive load?

Cognitive load refers to the total amount of mental effort required for a task, such as software development[6]. Rooted in cognitive psychology, this concept is closely related to cognitive control, the processing capacity of the conscious mind, and working memory capacity.

The processing capacity of the conscious mind refers to the amount of mental resources that are available to process and understand information. While a great deal occurs below the threshold of our awareness, and this has an impact on how we feel and what our life is going to be like, in order for something to become encoded as part of your experience, you need to have paid conscious attention to it[9].

Working memory, often referred to as the "mental scratchpad", is a cognitive system that allows individuals to temporarily store and manipulate information in their minds. It plays an important role in a variety of mental tasks, including problem-solving, decision-making, and learning.

In 1956, cognitive psychologist George Miller proposed that the number of items humans can actively hold in their conscious awareness at once is limited to about seven[4]. However, more recent research suggests that the limit may be closer to four or five items[8]. This has been interpreted to mean that the human brain can hold about 2 to 3 bits of information in working memory at a time, where each bit represents a binary decision.

The precise limit is determined by each individual's memory capacity, with activity from low-capacity individuals reaching this plateau sooner than that from high-capacity individuals[5]. Cognitive capacity is directly related to cognitive ability[6][7].

The capacity of working memory is limited to a few items, with estimates ranging from three to seven items.

The cognitive load can increase significantly when individuals are presented with a large number of options, a phenomenon known as "choice overload" or the "paradox of choice."[11]. This leads to hindered decision-making and a reduced likelihood of selecting rational and objective options[2][3].

Measuring Cognitive Load

"The more possibilities we are confronted with, the more perplexed and challenged we are likely to be.” ~ Aurelian Craiutu, Faces of Moderation

To quantify how a large number of options affects cognitive load, we can consider the "search space" of options contributing to the "perplexity" of the decision task. The larger the search space, the more cognitive resources are required to make a decision.

Perplexity, commonly used in natural language processing and machine learning, refers to the uncertainty or unpredictability of a model's output. In a broader sense, perplexity can be understood as a measure of the complexity, difficulty, uncertainty, or unpredictability of a problem or task. A person might be in a state of perplexity when trying to solve a difficult puzzle or find an object in a large search space. A more complex or difficult task might have a greater number of possible states or options to consider, which might increase the feeling of uncertainty or confusion.

Perplexity might be thought of as the average number of potential solutions or answers that are available within a particular domain.

The size of the search space refers to the number of potential solutions or answers that are available within a particular domain.

To quantify perplexity, we need to know how many options a software developer must consider and remember to successfully complete a task. This can be challenging, if not impossible, since it involves understanding the individual's thought process!

However, we can measure the knowledge discovered after completing the work. This knowledge equals the difference between the prior knowledge a software developer had and the knowledge required to develop the work item.

Let's look back at the game where we searched for a gold coin in a number of boxes. The number of boxes is the size of the search space. Hence if we know the number of questions asked H we can calculate the number of boxes:

That is what information theory calls “perplexity”. It is good to point out that perplexity cannot be zero. At the minimum it can be one.

Below is an animated example of calculating perplexity when we search for a gold coin hidden in 1 of 64 boxes.

From a human perspective, perplexity is determined by the number of possible answers to a question.

Knowledge Discovery Efficiency (KEDE) is a measure for understanding the knowledge gap that a software developer needs to bridge in order to successfully complete a task. It quantifies the knowledge that was lacking before the developer began their task - this is the knowledge that they needed to acquire. This knowledge gap is of utmost importance, as it directly influences the developer's experience, impacting their happiness and productivity. KEDE values can be anywhere from 0 to 100. A KEDE near 0 implies situations where the knowledge gap is enormous, like when creating completely new knowledge - think Albert Einstein or pioneering startups like Paypal. A KEDE value near 100 signifies situations where there's virtually no knowledge gap, as if we're looking at an omniscient entity. An expert full-time software developer, who mostly applies prior knowledge but also creates new knowledge when required, would have a KEDE of about 20. The less knowledge to be discovered the lower the perplexity for the human involved.

Taking into consideration how KEDE correlates with perplexity, we can say that::

For instance, if a knowledge worker has KEDE=100, it means she had to find a coin in only one box.

This situation reflects a perplexity of 1, indicating there was only one possible answer to the question: "Where is the gold coin?" This is the minimum value.

However, if a knowledge worker has KEDE=20, it implies she had to find a coin among 16 boxes, indicating a tolerable perplexity level.

But if a knowledge worker had KEDE=1, equivalent to answering 99 binary questions, it means she had to find a coin among approximately 6.3x10²⁹ boxes!

To contextualize, there are an estimated 10²² to 10²⁴ stars in the Universe[1]. The number of boxes was significantly greater than the number of stars in the Universe! We can infer that the software developer faced an exceedingly high level of perplexity.

Recall that when we have n equally sized boxes the average number of questions needed to find the coin location is:

Thus, by using KEDE, we can quantify in bits of information the cognitive load a software developer experienced while working.

With KEDEHub we are able to quantify in bits of information the cognitive load a knowledge worker faces while working on a problem.

Practical examples of how to measure cognitive load

Assessing a developer's cognitive load involves examining it over time. The following diagram depicts a developer's weekly cognitive Load over a selected period across all projects.

The x-axis shows the week weeks, and the y-axis denotes weekly cognitive Load values. The blue dot for each week represents a developer's cognitive Load. The dark blue line signifies a developer's average weekly cognitive Load. Comparisons with the company average can be made by examining cognitive Load for all contributing developers during this period, depicted by light gray dots. The black line signifies these developers' average weekly cognitive Load.

Examining the frequency distribution of a developer's averaged cognitive Load over time can be informative. A histogram, as shown below, can illustrate this distribution.

The x-axis displays the weekly KEDE, while the y-axis shows the frequency of each specific value. The colored histogram represents the developer's cumulative cognitive Load. The blue vertical line denotes the developer's average weekly cognitive Load for the selected period, while the green dashed line indicates the median weekly cognitive Load.

Both diagrams show that there were weeks, with a KEDE=10, when the selected developer experienced a cognitive load equivalent to searching in just 512 boxes. However, the developer's capability typically ranged between 2 and 4. This suggests that their cognitive load was akin to searching through approximately 16,777,216 boxes. Even though this cognitive load is considerable, it's still much lower compared to the average load within the company, where KEDE=1 corresponds to a number of boxes that exceeds the total number of stars in the Universe! Furthermore, the company's median cognitive Load is less than 0.5, indicating that half of the time, the cognitive load was intolerable!

As an initial measure, it would be beneficial to examine why the selected developer has a significantly better work experience than the rest of the company. Directly asking the developer could be a good approach, but for a deeper understanding of a developer's knowledge discovery efficiency, one should analyze its distribution across various projects. The diagram below shows a stacked bar chart of the developer's weekly efficiency divided by projects during a selected period.

The x-axis denotes week dates and the y-axis indicates weekly cognitive Load. Each bar represents a week's cognitive Load for the developer, divided into segments according to the proportion of weekly cognitive Load for each project.

For this specific developer, their cognitive load appears to remain constant, regardless of the project. As a result, it may be necessary to conduct a comprehensive examination of the company as a whole to pinpoint patterns or trends affecting their job satisfaction.

In response, we recommend that the company's leadership investigate the factors contributing to such extreme cognitive loads for the average developer. Based on my experience as an engineering manager, these factors could include insufficient training, prolonged feedback loops, unclear requirements, etc.