Managing Metric Madness
The Black Belt proudly displayed a huge spreadsheet printout. “It’s the first time we’ve been able to see all our metrics in a single place,” he announced. “Everyone was surprised at how many we have.”
Including me. There were well over 100 metrics on the spreadsheet. I wondered if all of them were important. Even if they were, it would be beyond the ability of mere humans to understand the relationships among so many variables. Quite simply, there were too many metrics to manage.
Metrics are, of course, essential to Six Sigma. It’s no secret that we need metrics to manage our businesses; we can’t improve a process that we can’t measure. Most managers and leaders are trying to make sense of an overwhelming number of “important” performance measures. They usually have very limited success. What’s to be done?
The first step in corralling the metrics monster is to understand metrics. A metric is part of an operational definition. A definition isn’t complete until we’ve identified how we’ll measure what we’re defining.
It’s also critical that the measurement is valid. This means that it measures what we think it measures, which is more difficult than you might think. Take something as simple as the diameter of a hole. Any experienced mechanical inspector will tell you that there are many answers to the question “What size is that hole?” Holes aren’t perfect cylinders. They’re not round on either end, and they aren’t the same size throughout their depth. We need to know why we’re measuring the hole in the first place. Do we need minimum or maximum clearance? A press fit? Access through the hole? The validity of the hole-size metric depends on our answer.
Metrics should also be reliable. Perfectly reliable metrics always give the same result when what’s being measured is the same, even when assessed by different people or at different points in time. In some cases it’s possible to quantify unreliability, such as with a formal measurement systems analysis. In other cases, such as a customer experience or a destructive measurement, this is problematic. Despite the challenges, thought should always be given to making the metric as reliable as is economically possible.
Not all metrics are created equal. In nearly all cases there are a critical few metrics that can be used to characterize an outcome. These critical-to-quality metrics can be identified using subject-matter expertise, logic, common sense or observation. Their importance can be assessed using statistical methods ranging from a simple Pareto analysis to complex components of variance analysis.
Sometimes it’s not clear which of the many metric candidates are critical and, despite your best efforts, you find yourself with dozens of measurements. Don’t despair. It’s possible to reduce the number of measures you’ll focus on to a manageable few by considering the interrelationships among the metrics. If X1 and X2 have a correlation coefficient of 0.99, then knowing X1 means that you know X2. Why track both? Track the one that’s easiest or cheapest to obtain.
You can also use more advanced statistical methods to find the critical few. Factor analysis or principal components analysis tells you how to reduce a large number of variables to a smaller number of factors. These factors, in a sense, explain the variation of the variables. The factors should bear a logical relationship to the variables and make sense. For example, a customer survey might ask questions such as:
• Was the product what you expected?
• Was the invoice price the same as the quoted price?
• Was the delivery date the same as what the sales agent stated?
If statistical analysis indicated that responses to all these questions were related, then it might be possible to treat them all as measurements of a single construct that might be called, say, “honesty.” The honesty score could be monitored on a dashboard instead of the three separate questions.
There are a limited number of reasons for being interested in any metric. I suggest that metrics are only of interest if they have one or more of these properties:
• They’re an important outcome.
• They’re an input that affects an important outcome.
• They measure something you can control or something that can be compensated for by something you can control.
This implies that we have a solid model of cause and effect. We know that a given change will produce the desired result because we understand the mechanism that makes this happen. In other words, we have transfer functions linking inputs to outcomes. Many transfer functions are a result of knowing the subject matter, such as engineering or marketing know-how. Many others remain to be discovered by applying Six Sigma.
Thomas Pyzdek, author of The Six Sigma Handbook (McGraw-Hill, 2003) and producer of the podcast “Six Sigma Pointers,” provides consulting and training to clients worldwide. He holds more than 50 copyrights and has written hundreds of papers on process improvement. Visit him at www.pyzdek.com.
|
