Managers are commonly fed a diet of report-card data. These data have usually been aggregated into summaries, averages, and totals to characterize the big picture. As useful as such summaries can be, they can also be an obstacle to an effective analysis. Here, we’ll learn how to avoid this obstacle.

Our example will come from a plant producing steering wheels. At the end of the production line, the inspectors would either pack the wheels for shipment to the assembly plant or place them on a line going to the rework department. Each day, Andy would visit the inspection station to conduct a 60-piece audit. When one of the inspectors rejected a wheel, she would tell Andy why she had rejected that wheel. Andy would add a tick mark to his sheet, and after 60 wheels had passed he would return to the quality office across the street and record the data.

The chart Andy produced for management consumption was an np-chart showing the total number of wheels rejected for each audit. This report-card chart is shown in Figure 1.

Figure 1: Number rejected per 60-piece audit

The central line of 22 shows that about one-third of the wheels were going to rework. The spike and the run near the upper limit at the end suggest that occasionally more than one-third of the wheels went to rework. So although this report card tells a bad story, it fails to tell us what problems are causing the rework.

The tally sheet Andy used to create the chart in Figure 1 is summarized in Figure 2. There were seven specific reasons for rejection, plus a catch-all “other” category.

Figure 2: Breakdown of rejections per audit

Inspecting the values in Figure 2 will reveal that the spike on Audit 8 was mostly foreign material. While some other things may also be found in the table, the best way to understand the values in Figure 2 is to place each row on its own chart. Figure 3 shows the np-charts for each row of running records from Figure 2.

Figure 3: np-charts for the eight reasons for rejection

In Figure 3, we find 16 points above the upper limits. These are 16 signals of excessive numbers of problems. The last two audits contained two signals each. Figure 4 shows the report-card chart of Figure 1 with the 14 audits that contained one or more signals of excessive problems highlighted.

Figure 4: Number rejected per 60-piece audit

Although Figure 1 did show a couple of signals, it hardly looked like 14 of the 21 audits contained signals of excessive numbers of problems of one sort or another. Aggregating the data into totals hides most of these signals. And this is the inherent problem of report-card charts, whether they involve counts or measurements. As we combine data from different problems, categories, or departments, we’ll aggregate the noise from each, and signals will get buried in the totals.

The more highly aggregated the values, the greater the likelihood that the chart will appear to be predictable. Report-card charts might provide a useful record of what was, as in about one-third of the wheels went to rework last month, but they’re not very useful in reducing the amount of rework. They can tell you how much trouble you’re in, but they don’t show you how to get out of trouble.

Post-hoc improvement efforts

In this case, the daily audits were used in lieu of recording the results of the 100% inspection operation. In addition to being used to generate a report card for management, the categories of Figure 2 were used to generate a Pareto chart to highlight the problems during the past month. A meeting would be held to discuss the Pareto, and occasionally some action might be taken.

In Figure 2, the leading problems in July were foreign material and damaged edge. Stress was in fourth place, and sinks was in last place. However, in August, sinks and stress were the top two categories in the Pareto. When your Pareto chart categories play musical chairs from month to month, it’s common to find that each category will show a history of being operated unpredictably (as seen here in Figure 3).

The September meeting to discuss the August Pareto focused on the two leading categories: sinks and stress. When I asked what the causes of sinks and stress were, no one was sure. At that point, I suggested that they weren’t going to find out what was happening by collecting data at the end of the line. They were going to have to go out and watch the process in real time to see where the sinks and stress were being generated. So, as each wheel came out of the press, they placed a tag on it identifying the press and time of day. Then, in the rework department, they discovered that all the sinks came off one press, and all the stress came off the other press.

While both presses appeared to be running at the nominal temperature, one was actually running cold, and the other was running hot. When they recalibrated the thermocouples, the problems with sinks and stress disappeared.

Problems don’t get solved until we know when and how they occur. This requires context. Data collected at the end of the line are often devoid of this contextual information. That’s why separating our nonconforming product into bins and performing a post-hoc analysis isn’t a reliable way to improve a process.

The importance of feedback

After the success with the thermocouples, they started posting the report-card chart in the production department. This chart included the table of counts for each reason for rejection. By suppressing the zero counts, they made the table quasi-graphic, allowing everyone to see what was happening. Over the next few months, the reasons for rejection grew from eight to 16, as shown in Figure 5.

Figure 5: Number rejected per 60-piece audit

When asked why they had two categories for flow-lines, their answer was textbook perfect: “We have two categories because we think there are two different assignable causes.” And that’s the secret for using count data effectively. You have to disaggregate the counts to the point that you can identify the different assignable causes.

Back in Figure 2, the category of foreign material included lint and vapor drip. (In fact, the spike in Figure 1 came from 30 wheels with vapor-drip spotting.) But in Figure 5, we see that lint and vapor drip had become separate categories from foreign material in the resin.

As the categories got focused, it became easier to take corrective actions, and the number of rejected wheels began to drop. The limits shown in Figure 5 came from the previous month. But 12 of the first 13 audits came in below the central line, so they decided to recompute the limits. These new limits and the next 15 audits are shown in Figure 6.

Figure 6: Number rejected per 60-piece audit

And then they went below the lower limit again. Unprecedented levels of quality! Then, on the last two days there, someone had the bright idea to save a penny by giving only one razor blade to each insert trim operator each shift! Bad insert trim made the folly of this idea apparent very quickly.

Finally, when everyone wears cotton gloves the problem of lint shouldn’t be surprising. But look at the running record for lint in Figure 7. Andy noticed that he got a lot of lint only when his audit included reworked wheels. Because rework consisted of rubbing a wheel with an acetone-soaked cheesecloth, Andy had the lint tested. And yes, the lint was coming from rework, not the cotton gloves everyone wore. They changed to using organdy cloth in rework, and lint disappeared. (Andy made sure each of the last 10 audits contained reworked wheels.)

Figure 7: Number rejected per audit for lint

Effective data analysis

Whether working with counts or measurements, data collected at the end of the line may provide report cards, but they will seldom contain the information needed to improve things. As shown here, when we aggregate data up into totals or averages, we accumulate noise at the same time. This accumulated noise will tend to hide the individual signals, making report-card charts appear more predictable than the processes they represent.

In fact, there are consultants who make use of this fact by insisting on using highly aggregated data so they can be soothsayers to management. So don’t judge operations by the report-card values alone.

One of the secrets of effective data analysis is disaggregation. As we disaggregate our report-card data, we move closer in time and place to where things happen. As we gain context, we obtain the leverage needed to understand what’s happening and how to fix problems.

Sorting through the wheels in the rework department didn’t help them find the source of stress and sinks. Rather, it was the tags on the wheels that provided the necessary context.

No data have any meaning apart from their context.