VP Engineering performance Report
How to measure performance of a VP of Engineering
Motivation
More too often the performance of a technology company is judged using only metrics such as increase in the throughput of new features, increase in velocity of story points, reduction of the average time needed to deliver a feature.
Each of those approaches accounts for some important facet of software development. However all of the approaches can actually be applied to manual work just as well! That is because they don't measure what is specific to software development only - its essence, which is the acquisition and application of knowledge.
Productivity of software developers is measured with their efficiency in acquisition and application of knowledge.
KEDEHub allows you to measure the productivity of your organization using the one and only scientifically backed metric - knowledge discovery efficiency (KEDE). A knowledge discovery process is the combination of knowledge acquisition and its application to produce working software. KEDE quantifies how efficiently your organization is utilizing its potential. KEDE says that if a team is more efficient in terms of knowledge discovery it should produce more working software for the same time.
KEDE is calculated per software developer. However, no individual software developer ever works in isolation but always interacts with others. Many people overestimate how much of the knowledge discovery efficiency is a function of the skills and abilities of the individual developers and underestimate how much is set by the systems they operate in.
Individual KEDE is a function of processes and policies of the organization the software developer works in.
In what follows we present an example of a performance review that could have been prepared using KEDEHub and the observed knowledge discovery efficiency (KEDE) for a company.
The company we will look into is PyTorch Lightning[1]. The manager we will try to profile is their VP of Engineering[2].
Context
PyTorch Lightning was started by William Falcon while completing his Ph.D. AI research at NYU CILVR and Facebook AI Research, with the vision of making it a foundational part of everyone's deep learning research code. The framework was designed for professional and academic researchers working in AI, making state of the art AI research techniques, such as TPU training, trivial. PyTorch Lightning is a lightweight machine learning framework that handles most of the engineering work, leaving you to focus on the science. It's more of a PyTorch style-guide than a framework, setting a standard on how to structure your deep learning PyTorch code. Lightning ensures that when your network becomes complex your code doesn't. PyTorch Lightning was released in March 2019 and made public in July 2019. This core group was quickly augmented by an extended community of over 180 contributors.
More or less the same people take part in Grid.ai[3]. Grid.ai's code is not open source. That makes it difficult to say how the efforts of the group are shared between the two projects.
Initial state
Below we have KEDE presented for the 12 months period from June 2019 until June 2020. With average KEDE=0.8 we can say that the company had hard times developing the product. When other organizations have KEDE below 1 they usually report that:
- constantly changing requirements
- new and unstable technology
- no prioritization in place
Of course all of that could have been present because the company developed a new breakthrough technology.
In order to better understand the performance let's look at the flagship project: pythorch-lighting.
Below we have KEDE presented for the same 12 months period from June 2019 until June 2020. The average KEDE=0.8 is the same as the average KEDE for the entire company.
End state
The VP Engineering[2] joined the team in June 2020 and left after 6 months in December 2020.
Below we have KEDE presented for the 6 months period from June 2020 until the end of December 2020. The average KEDE=1.2 was a 50% improvement in the company's knowledge discovery efficiency.
We can only speculate what has changed. One possibility is that the VP Engineering brought to the company knowledge they didn't have before. Another possibility is that he re-organized the work in such a way so that the knowledge flew much smoother.
Let's look at the flagship project: pythorch-lighting.
Below we have KEDE presented for the 6 months period from June 2020 until the end of December 2020. The average KEDE=1.0 is less than the KEDE for the whole organization. It was still a 25% improvement compared with the 12 months before.
Conclusion
Physicians use a thermometer in order to get an understanding about what is going one inside the black box that is a human organism. You can use KEDEHub the same way - to get visibility into the black box that is an organization developing software.
The present day attitude toward measuring knowledge work productivity very much resembles how temperature was understood 400 years ago. It is argued that the manager's touch captures information richer than any tool. However, we don't agree. It is time to start measuring the productivity of knowledge work by its essence, which is the acquisition and application of knowledge. That is what the one and only scientifically based metric - knowledge discovery efficiency (KEDE) is used for!
Managers and decision-makers , who want to unleash the untapped potential in their organization, can use KEDEHub to get visibility into the black box that is an organization developing software with our scientific and non-intrusive method.
References
Getting started
