Member-only story
3 pitfalls to watch out for when creating your own metrics
As more tools are available to businesses to make data driven decisions, many people are tempted to invent their own metrics or KPIs to measure success and progress. This is especially true in the case of software companies, where many leaders have an engineering mindset and are interested in coming up with metrics of their own.
While many people are aware of fallacies to watch out for when they’re analyzing data, few are paying attention to them when they’re designing new metrics to guide their teams. Here are 3 I’ve learned to watch out for.
Survivorship bias
Wikipedia defines survivorship bias as “the logic error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility”. The canonical example is from WWII — looking at the damages from warplanes that flew back to the base might falsely conclude that the heaviest damaged parts of the planes are the areas that require fortification.
In software engineering, we might be tempted to measure the number of bugs reported by customers alone as…
