Gaming KEDE

How we prevent gaming?

"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." ~ Charles Goodhart

Goodhart's law states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it.

Since KEDE reflects the capability of software development organizations in terms of how efficiently they discover and apply knowledge it is possible KEDE be used as an input to performance reviews and salary negotiations. People would say that because KEDE calculation has as its input number of symbols typed per hour it will be trivially easy to be gamed by the software developers. The argument would be that a developer who gets paid per symbol typed can easily earn an entire year's salary in a single day without creating any business value whatsoever.

Indeed a developer could type

i = i + 1 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0;

instead of

i++;

A developer could even copy and paste a huge amount of code, commit it and claim high productivity. For instance adding on a regular basis a page of “dead” code that does nothing would gain 0.01 KEDE. Or maybe 10 pages in order to gain 0.1 KEDE? Or a developer could commit all the code of an open source framework such as React.

Here are some of the ways such a behavior can be identified:

• There are tools on the market for finding dead, unreachable, irrelevant, duplicate code.

• There is the practice of code review and many organizations do it.

• We calculate KEDE on a daily and weekly basis hence the developer would need to do the same copy paste on a daily basis. Does this sound probable?

• We calculate KEDE not for a single developer only but for the whole team. If one of the team members suddenly is several times more productive than the other wouldn't that be very suspicious and sound an alarm?

• If individual managers conspire with developers to game their team's KEDE it will be visible to the rest of the company.

• We have historical KEDE data for hundreds of thousands of software developers for the last decade and some more. If somebody today suddenly starts beating that historical performance that would sound an alarm.

• KEDE calculation rests on the natural limitation of the maximum typing speed a human being could achieve. That sets a hard ceiling on software development performance.

Some people could say that a whole company could engage in an effort to game their overall KEDE standing. Indeed, that is possible, but again how probable is it?

• If the company is not brand new then there will be historical data hidden in their version controlled source code bases. We can calculate KEDE for all of the company's history. Then a sudden increase in knowledge discovery efficiency will be easily detected.

• In order to show to its clients their KEDE a company will definitely want to have its records audited by an external independent party. Same like ISO audits, CMMI appraisals etc. Such an independent audit will reveal if fraud was committed.

We see that a brute force approach for gaming KEDE is difficult. What about if developers adopt subtler approaches to boost their KEDE? Some of the ways are:

• Moving the same code from one repository to another. Many organizations do that anyway for reasons not related to KEDE calculation.

• Duplicating ode changes into another repository or the same repository. This can happen when a commit is rebased, amended or cherry-picked. Then the new commit has a different commit identifier but contains the same code changes as the original.

• Committing say 10 pages of code today. Then tomorrow delete the 10 pages from yesterday and commit the same 10 pages. This way at the end of the project the developer would have added only 10 pages of code but enjoyed a positive KEDE throughout.

Such cases should be found automatically before calculating KEDE and KEDEHub does it.