Three lessons from a metrics-driven operator

Recently Neil Roseman gave a fascinating talk about metrics to the Pike13 team. Neil spent ten years at Amazon.com, where he served as VP of Engineering, and subsequently held a similar role at Zynga.com. Although very different businesses, both Amazon and Zynga are metrics-obsessed companies, and Neil was known as a metrics-driven operator.

Neil shared a few interesting insights from these experiences into metrics-driven cultures. In particular, I took away three key messages:

1. It’s an amazing time to be able to track everything.

Storage is cheap and tools are practically free. Neil referenced W. Edwards Deming’s famous quote, “You can’t manage what you don’t measure.” At both Amazon and Zynga there’s a culture of collecting data about everything. As long as the act of collecting data doesn’t create significant drag on actually getting things done, it’s always worth holding onto data. Since we often “don’t know what we don’t know” there’s great value in having access to the data whenever you need it.

2. Focus on the inputs not the outputs.

When collecting data in a metrics-driven culture, it’s critical to understand the difference between inputs and outputs. For example, Net Promoter Score (NPS) is a metric that is very closely tracked at Amazon. But it’s viewed as an output—changes in NPS are driven by things like customer service response time and platform availability (which, in turn, are driven by other factors). Neil’s point was that if you want to improve NPS, it’s important to identify the factors that affect it and to measure those.

3. Don’t underestimate the importance of good judgment.

Neil quoted Freewheelin' Franklin of The Fabulous Furry Freak Brothers, who said, "Dope will get you through times of no money better than money will get you through times of no dope." Neil suggested that we substitute judgment for dope and data for money. In other words, judgment will get you through times of no data better than data will get you through times of no judgment. Even in an age when it’s easy to track everything, metrics-driven decisions still ultimately come down to having good judgment and avoiding common mistakes like confirmation bias. Recognize the wisdom in the old adage, “If you torture your data long enough, it will confess.” If you manipulate the data long enough, you’ll find the result you want and make inferences that lead you in the wrong direction.