How to Design Better Metrics
Metrics are a powerful tool; they help you measure what you care about. Having lofty goals is great, but to know if you're making progress, incentivize your team and create accountability, you need to be able to express them in numbers.
But that's easier said than done. There are dozens of metrics that seemingly measure the same thing, and new trendy metrics are invented every day. Which ones should you use and what should you avoid at all costs? This article will help you decide that.
Over the last decade I have been living and breathing metrics and have found that there are a few general principles that distinguish good metrics from bad metrics:
Principle 1: A metric should be a good proxy of what you're trying to measure
You typically cannot directly measure the exact thing you care about.
Let's say my goal was to measure the quality of my newsletter posts; how do I do that? "Quality" is subjective and there is no generally-accepted formula for assessing it. As a result, I have to choose the best (or least bad) proxy for my goal that I am actually able to measure. In this example, I could use open rate, likes etc. as proxies for quality.
This is closely related to what people often called the "relevance" of the metric: Does it create value for the business if you improve the metric? If not, then why measure it?
For example, let's say you work at Uber and want to understand if your supply side is healthy. You might think that the number of drivers on the platform, or the time they spend online on the app, is a good measure.
These metrics are not terrible, but they don't really tell you if your supply side is actually healthy (i.e. sufficient to fulfill demand). It could be that demand is outpacing driver growth, or that most of the demand growth is during the mornings, but supply is growing mostly in the afternoons.
A better metric would be one that combines supply and demand; e.g. the number of times riders open the app and there is no driver available.
Principle 2: The metric should be easy to calculate and understand
People love fancy metrics; after all, complex Analytics is what you pay the data team for, right? But complicated metrics are dangerous for a few reasons:
