Knowledge Discovery Efficiency (KEDE) and Multi-scale Law of Requisite Variety

Information-Theoretic Formulation

For many purposes the variety may more conveniently be measured by the logarithm of its value. If the logarithm is taken to base 2, the unit is the bit of information. In practice, we use the Shannon information entropy, denoted by H. For a quantifiable variable, entropy is just another measure of variance[1].

Without regulation, the entropy of the system is H(X). The presence of the regulator should ideally remove uncertainty from X. Thus, the difference between H(X) and H(X∣Y) represents how much uncertainty the regulator removes:

Here, I(D;R) is the mutual information or the channel capacity, representing how effectively the regulator Y reduces uncertainty in the regulated environment D. The conditional entropy H(D|R) measures the uncertainty remaining in X after considering the actions or states of regulator R.

• If H(X∣Y) is large, then after the regulator has acted or observed, significant uncertainty about the controlled environment remains. This indicates poor regulation or insufficient regulator variety.
• Conversely, if H(X∣Y) is low (ideally approaching zero), then the regulator effectively predicts or controls the states of the regulated environment D.

In other words, the conditional entropy of the controlled system given the regulator should be as small as possible ideally zero - implying perfect regulation.

The law of Requisite Variety says that Y 's capacity as a regulator cannot exceed Y 's capacity as a channel of communication. Thus, all acts of regulation can be related to the concepts of communication theory by noticing that the “disturbances” correspond to noise, and the “goal” is a message of zero entropy, because the target value E is unchanging.

Defining Knowledge to be Discovered

Knowledge to be Discovered H(X|Y) is the gap in internal variety that had to be compensated by selection.

Thus, the variety of the disturbances, is defined as the "required knowledge" or total unconditional missing information H(X), The regulators variety, or "prior knowledge" is defined as the mutual information I(X:Y), which equals the "requisite knowledge" we discussed earlier. The "knowledge to be discovered" is the conditional information H(X|Y) - what the regulator lacks about the task before starting it i.e. the gap in internal variety that had to be compensated by selection.

• Imagine a scale: on one side is the required knowledge and on the other is the prior knowledge.
• The gap between required and prior knowledge is the Knowledge to Be Discovered.
• They will be balanced if they are equal, implying that the knowledge to be discovered equals zero.
• When balanced, no further knowledge is needed; when imbalanced, knowledge acquisition is required.

Anchoring KEDE to Natural Constraints

In our model, N is always the theoretical maximum output rate in an unconstrained environment, and S is the observed output rate under specific conditions over a given interval.

Both N and S can always be discretized—or “binned”—in a way that preserves the total information rate, regardless of whether the output arises from natural processes, human behavior, or machines. By choosing a bin width Δt small enough (e.g., milliseconds), the range of possible tangible outputs within each bin shrinks dramatically. This reduced range leads to less uncertainty in each bin, which compensates for the smaller time interval. Yet the ratio

remains an accurate measure of information rate.

Interpreting Complexity Across Scales

Complexity is not an intrinsic, single-number property of a system. Rather,the information required to describe a system depends on the level of detail or scale at which the system is viewed. When we examine a system at very fine detail, we may see a large number of possible microstates; when we examine it at coarse scale, many microstate differences collapse into a smaller set of distinguishable macrostates. Complexity is therefore a function of scale, not a scalar[4].

To formalize this intuition, we consider systems composed of many components. If the components are people, then a task may require a certain (minimum) number of people acting together[4]. These components can be grouped into larger parts, and these partitions represent different scales by which the system may be described.

Any multi-scale structure can be represented by a sequence of nested partitions. The partition P_n contains exactly n parts, and the partitions must be nested. A nested sequence of partitions means each partition refines the previous one. Thus, each part in partition P_i is a subset of some part in partition P_i-1. This nesting ensures that as we move to finer scales (higher n), we gain more detail without losing the structure defined at coarser scales (lower n)[5].

This definition has strict properties:

• when n=1 then the system is one undivided whole (coarsest).
• when n=|X| then each component stands alone (finest).
• as n increases, the description becomes more detailed.

Scale is not:

• size of a group
• number of components in a part
• physical distance
• resolution
• granularity
• anything continuous

• the number of groups in a particular element $P_i$ of a nested partition sequence $P={\left\{P_i\right\}}_{i=1}^{\infty }$,
• which defines how components of a set X are grouped,
• and includes all correlations exposed only because of the grouping at that level.

"Coarse-graining" means increasing the level of abstraction or simplification in a system, making the basic units larger and fewer, which reduces computational complexity but sacrifices detail. Fine scale means higher n, more groups, small group size, finer description. Large scale means lower n, fewer groups, large group size, coarser description. Fine-scale means independent behaviors. Large-scale means coordinated behaviors. Each latger scale reduces the number of distinct components by grouping them, thereby coarse-graining them. This hierarchical coarse-graining continues until we reach the coarsest scale, where all components are treated as a single unit.

This notion of scale is distinct from the amount of variety (complexity) available at that scale. Given a nested partition sequence P of a system X at scale n , ${\tilde{S}}_X^P\left(n\right)\equiv n{S}_X^P\left(n\right)\equiv {\sum }_{\chi \in P_n}H\left(\chi \right)$ captures the total (potentially overlapping) information of the system parts when described through the partition P_n. Note that $S_X^P\left(n\right)$ is the average amount of information necessary to describe one of the n parts[5].

The complexity profile $C_X^P\left(n\right):{\mathbb{Z}}^{+}\to [0,\infty )$ at scale n of a system X viewed through a nested partition sequence P, is $$C_X^P\left(n\right)={\tilde{S}}_X^P\left(n\right)-{\tilde{S}}_X^P(n-1)$$ with the convention that ${\tilde{S}}_X^P\left(0\right)=0$. The complexity profile assigns a particular amount of information to the of a system X at each scale n. For $n>\left|X\right|$ where $C_X\left(n\right)=0$. The complexity at each scale measures the amount of additional information that becomes available when different parts of the system are considered separately[5].

This multi-scale view reveals deep structural constraints:

1. At fine scales, complexity reflects the variety of individual components..
2. At intermediate scales, complexity reflects partially redundant or partially coordinated behaviors.
3. At large scales, complexity reflects coherent, system-level patterns that require many components to act in concert.

At fine scales, descriptions capture the variety of individual components; at larger scales, they capture the structured behaviors that emerge only when components act together. With coarse-graining at larger scales, you lose fine-grained distinctions, so the information cannot increase. it can only stay the same or decrease.

The scale-dependent framework rests on three structural steps:[5]:

• Partition the system into components and define how these components may be grouped at different scales.
• Construct a nested sequence of partitions, ensuring that each coarser partition is a grouping of the finer one.
• Measure complexity at each partition level by quantifying how much information remains after coarse-graining.

Through this construction, the complexity profile makes visible where a system’s information resides—whether predominantly at fine scales, at intermediate scales, or in large-scale coordinated behaviors.

The complexity profile also obeys a sum rule: $${\sum }_{n=1}^{\infty }{C}_X\left(n\right)={\sum }_{x\in X}H\left(x\right)$$ Because the complexity at each scale measures the amount of additional information present when different parts of the system are considered separately, the sum of the complexity across all the scales will simply equal the total information present in the system when each component is considered independently of the rest[5].

The sum rule is a structural conservation law for descriptions of a single system. It ensures that information is only redistributed across scales, but not created or destroyed. This reflects a fundamental tradeoff: large-scale structure can exist only if small-scale degrees of freedom are constrained[5].

Generalizing Ashby to a Multi-Scale Setting

Ashby's classical Law of Requisite Variety states that a regulatory system must possess at least as much variety as the environmental conditions that require distinct responses. In other words, the conditional entropy of the controlled system X given the regulator Y should be as small as possible ideally zero - implying perfect regulation.

Consider two individuals who cannot coordinate. At a fine scale, they have more than enough behavioral variety as each person can act independently. But at the larger scale required to move a couch, they lack the necessary coordinated variety. Classical Ashby deems the system “sufficient,” but in practice it fails. The limitation is one of scale.

This motivates the Multi-Scale Law of Requisite Variety[4]:

For a system to effectively regulate its environment, its complexity must equal or exceed the environment's complexity at every scale.

To make this statement precise, the system and environment are each represented as a set of components equipped with corresponding nested partitions. At each scale n, the conditional entropy: ${H\left(X\right|Y)}_n$ , quantifies the missing information about the environment's state X at scale n given the knowledge of the system Y at scale n. Note: The parentheses refer to the same entropy measure applied to the same underlying variables. The subscript n refers to the scale at which the description is evaluated, not a new random variable. Thta is because H(X|Y) is a function of the joint distribution of X and Y, which can be analyzed at different scales using the nested partitions defined earlier. The distributions X and Y never change. Only the description changes with scale.

If this coarse-grained conditional entropy is zero: ${H\left(X\right|Y)}_n=0$ at scale n, then the system Y can fully distinguish all environmental states at that scale i.e. it perfectly regulates the environment X at that scale. If it is greater than zero, some information about X at scale n is missing from Y, indicating imperfect regulation at that scale.

This is the multi-scale operational version of Ashby's Law of Requisite Variety.

But computing conditional entropy requires knowing the joint distribution p(x,y). In many practical situations, this distribution is unknown or difficult to estimate. We typically know:

• what the environment does on its own (its complexity at various scales)
• what the system does on its own (its complexity at various scales)

Using the complexity profile $C_X^P\left(n\right)$ at scale n, we compute complexity as a function of scale. When the system matches the environment — meaning each system component has a distinct response for each environmental condition relevant to it—the following necessary condition must hold:

and if at any scale $C_Y\left(n\right)\le C_X\left(n\right)$ , the system cannot fully regulate the environment, where $C_Y\left(n\right)$ and $C_X\left(n\right)$ are the scale-dependent complexities of the system and environment at scale n, respectively.

Thus, a system cannot “add” large-scale capability without sacrificing some small-scale variety. The multi-scale law requires that the distribution of complexity across scales — not merely its total amount — must be appropriate for the task or environment.

This result is formalized in Theorem 1 of [5], which proves that if ${H\left(X\right|Y)}_n=0$ is even possible under some joint distribution of X and Y, then the inequality holds for every possible nested partition sequence, i.e. for every admissible way of defining scales. It also extends to every subset of components (Corollary 1), ensuring that not only the system as a whole but every subsystem must be at least as complex as the corresponding part of the environment.

Thus, $C_Y\left(n\right)\ge C_X\left(n\right)for\; all\; scales\mathrm{n;}$ is a necessary condition for ${H\left(X\right|Y)}_n=0$. Conversely, if at any scale $C_Y\left(n\right)\le C_X\left(n\right)$, then zero conditional entropy is impossible, meaning the system can never fully distinguish environmental states at that scale — indicating that the system cannot fully regulate the environment.

In summary:

• The complexity profiles $C_Y\left(n\right)$ and $C_X\left(n\right)$ tell us whether it is possible for the system Y to distinguish environmental states X at scale n, without knowing the joint distribution. Thus, they measure capacity i.e. how much information is available at each scale.
• The conditional entropy ${H\left(X\right|Y)}_n$ tells us whether the system Y actually distinguishes environmental states X at scale n. Thus, it measures performance i.e. how much information is actually utilized at each scale.

The complexity profile has a sum rule because it measures conserved internal information. Conditional entropy has no sum rule because it measures relational uncertainty, which is not conserved across scales. Coarse-graining can make conditional entropy decrease, increase, vanish, or appear. Correlations with the environment can disappear at some scales while appearing at others This distinction is absolutely central to the multi-scale law of requisite variety. Failure to regulate at one scale cannot be compensated by success at another. If conditional entropy did obey a sum rule, then:

• If a system perfectly regulated its environment at some scales (zero conditional entropy), it would have to be less effective at other scales (higher conditional entropy) to conserve total conditional entropy.
• This would imply a tradeoff where excelling at regulation in one area necessitates failure in another, which contradicts the goal of effective regulation across all scales.

Implications

The multi-scale formulation strengthens Ashby’s original insight. A system is not effective simply because it has high total variety; it must have variety in the right places. A regulatory system, an organization, or a biological organism must align its complexity profile with the demands of its environment:

• granular variety where fine responsiveness is needed,
• coordinated large-scale complexity where collective action or coherent strategy is required,
• and appropriate structure at the intermediate scales where patterns of interaction matter.

By making scale explicit, the multi-scale law provides a precise mathematical criterion for diagnosing mismatches between systems and environments—showing when regulation or control is impossible, regardless of mechanism.