An engineer once told me, “I work on project teams that have an average half-life of two weeks, implementing solutions with an average half-life of two weeks.” Time after time, and in place after place, our improvement efforts often fall short of expectations and fade away. In this article, I will explain why this happens and what we can do about it.

Many improvement programs today are built around the use of experimental techniques. These techniques are proven ways of obtaining a lot of information in a short time. However, the problems keep recurring. To appreciate why this happens, we need to consider the relationship between experimental techniques and the types of problems we face.

Three types of problems

Myron Tribus (former dean of continuing engineering education at MIT) once described the problems we face as falling in the space defined by the three axes of Figure 1 (below). Tribus elaborates that we are taught how to handle problems as if they lie along only one axis or another, even though, in practice, our problems tend to involve either two or three axes at once.

Figure 1: Three different types of problems

The first axis is the axis of “unknown mechanisms.” How can we prove that if we do “this” then we will get “that”? The scientific method uses experimentation and nontrivial replication to answer questions about unknown mechanisms.

During the past 100 years, the mathematical foundation of statistically designed experiments has taken us from experimenting one factor at a time to using multifactor experiments and highly efficient screening designs. The successes of this approach in agriculture and biomedical research have created the discipline of statistical science.

Clearly, when seeking specific answers to a set of specific questions, designed experiments are unsurpassed. As statisticians began to work with other fields, the emphasis upon experimental methods increased until some saw it as the only approach to use with any problem. But the problems of the second and third axes create obstacles for a purely experimental approach.

The problem of too many variables

The second axis is “too many variables.” Whether we are setting up a new production process or simply seeking to improve an existing process, we will always have to work with a laundry list of cause-and-effect relationships. Typically, this laundry list will contain anywhere from several dozen to several hundred variables. Moreover, given the way cause-and-effect diagrams tend to grow over time, we have to admit that many of the variables on our laundry list will initially be unknown. Perhaps the last item on the laundry list should always be “other unknown causes.”

Figure 2: The laundry list of causes; variables that affect a product characteristic

Of course, our hope is that we can divide the laundry list into the critical few, each of which has a large effect on the product; and the trivial many, each of which has only a small effect on the product. If we can do this, then we can control the critical few and ignore the trivial many.

Figure 3: Our hope for the laundry list: Variables that affect a product characteristic

Even the smaller laundry lists will contain more variables than we typically study in an experiment. Our search for the critical few will usually begin with past experience. We will choose some variables that are likely to be in the critical few, then carry out some experiments to verify our choices. We then define what levels to use for these controlled inputs when we go into production.

So, while experimentation may be used to confirm and consolidate our choices regarding which process inputs to control, the problem of too many variables tends to overwhelm a completely experimental approach. Our choices will usually include most of the critical few cause-and-effect relationships. But what happens when one or more of the factors that we didn’t choose to study happens to have a dominant effect? If it isn’t studied, it will not be part of the controlled inputs. As an uncontrolled variable, it will be free to vary. Although it may not change while the project team is studying the process, it is almost certain to change in production. As this cause varies, its dominant effect will take the process on walkabout. And after a couple of weeks, a new project team will be formed.

“But can’t we use screening designs to separate the critical few from the trivial many?”

Only to a limited extent. Screening designs are very efficient, but they still get large as the number of variables increases. Moreover, no matter what experimental design we may use, we can only study those variables from the known side of the laundry list in Figure 2. We can never experiment with the unknown cause-and-effect relationships. When an unknown variable happens to be one of the critical few, neither guesswork nor experimentation is going to reveal this dominant variable, and our set of process inputs will be incomplete.

The problem of variation in results

“We controlled all the input variables precisely, but we still had too much variation. Where did all the variation come from?”

When we control an input variable at a fixed level, it will no longer contribute to the variation in the product characteristic. Rather, it will tend to restrict the range of possible product characteristics that the other, uncontrolled variables can create. A second controlled input will further restrict this range of outcomes. As long as controlling an additional input variable will either reduce the range of process outcomes or shift the process average closer to the target, it will be a candidate for the set of controlled process inputs. Thus, the overall set of levels chosen for the controlled inputs will effectively determine the process average by restricting the range of outcomes. This progression is shown in Figure 4.

Figure 4: The effect of controlling dominant process inputs

To the extent that we can hold the process inputs at their chosen levels, the control factors will contribute little or no variation to the process outcomes. Virtually all of the variation in the process outcomes will be due to the set of uncontrolled variables from our laundry list.

This fact of life has two consequences. First, we will find it very difficult to reduce the variation by experimenting with the controlled input variables. Second, variation comes from those variables that are routinely overlooked. This is why variation is frequently not on the table until it causes a problem.

One manager at a company making jet engines told me that they didn’t just have a team; they had a task force on vibration. He went on: “I’ve been here 30 years, and every year we have had to form a task force to work on vibration.” In a jet engine, vibration is a complex byproduct of variation in the many parts, even though all these parts meet their specifications. Variation is inevitable, and it is always a problem. If we want to know anything about variation, we are going to need to know about the uncontrolled variables.

But the number of uncontrolled variables will quickly outstrip our ability to perform experimental studies. How, then, will we ever estimate the effects of each of the uncontrolled variables? Fortunately, we don’t need to do so. We only need to separate those variables with a dominant effect from those with smaller effects.

“But even this separation requires that we can observe the effects of all of the uncontrolled variables.”

Yes, but we don’t have to observe each of these effects individually, as would be required by an experimental approach. Instead, we can observe all these effects simultaneously with an observational approach.

As long as all of the uncontrolled variables have small effects, their combined effects will, over time, produce a steady and recurring amount of routine variation. No one effect will stand out from the sum of all of the other effects, and the resulting routine variation will basically be background noise.

When one of the uncontrolled variables has a dominating effect, this effect will stand out from the routine variation. This will result in changes in the stream of process values from time to time, and these changes will be the evidence that a cause with a dominating effect is present.

So, by shifting to an observational approach, we can characterize the process behavior as being either routine or changing. This allows us to simultaneously consider the combined effects of all of the uncontrolled variables, whether they are known or unknown. By characterizing the variation in the product stream as either changing or routine, we gain the ability to know when to look, and when not to look, for another variable to add to the set of controlled inputs.

Beyond experimentation

When our project team is working to fix a problem, we always need to identify what variables to control and what levels to use for these variables. For problems with unknown mechanisms that lie along the first axis, an experimental approach has proven to be effective. Experiments allow us to ask specific questions and obtain specific answers.

However, production is harder than research. As we leave the laboratory, we come into contact with other cause-and-effect relationships that were not included in the experiments, and the problem moves to the second axis, where there are too many variables. Here, an experimental approach cannot cover all of the possibilities. In an experiment, there are only three things we can do with an input variable: We can study it, we can hold it constant, or we can ignore it. And our laundry list guarantees that the last category will be the largest.

“So if we can’t use experiments to study all of the possible input variables, how can we determine if our set of controlled process inputs is sufficient for production?”

Here, we will need a way to observe the combined effects of all of the uncontrolled cause-and-effect relationships, known and unknown. This observational approach will need to allow us to judge whether the original set of controlled inputs is complete or incomplete.

“But the process outcomes vary. How can we judge if we need additional controlled inputs when the results vary?”

The fact that variation comes primarily from the uncontrolled variables is simply one more obstacle to conducting experiments. Almost all production problems involve both the second and third axes. The large number of variables, plus the presence of unknown variables, plus the fact that the results vary, will combine to limit what can be achieved with an experimental approach. As noted, only an observational approach will have the completeness that is required to study the effects of all of the known and unknown variables. And the premier observational approach is a process behavior chart (aka control chart).

The limits on a process behavior chart describe the combined routine variation of all of the common causes, both known and unknown. In this way they define the process potential.

The running record shows the actual performance of the process. By combining both process potential and process performance in a single graph, a process behavior chart provides a way to judge and characterize the process. As long as the running record operates within the limits, we have no evidence of exceptional variation. When a point goes outside a limit, it is taken as evidence of an assignable cause. By studying the context of the point outside the limit, we will usually discover a variable with a dominant effect. By making this variable part of the controlled inputs to the process, we reduce the variation in the product stream and gain leverage for adjusting the process average.

Thus, the process behavior chart complements an experimental approach by handling problems along the second and third axes that are difficult or impossible to address experimentally.

The assumption behind projects

Once we have chosen a few variables and carried out our experiments, and perhaps even made a pilot run, it’s time to turn the process over to the production department. Production is told what variables to control and what levels to use in running the process, and the project team moves on to fight the next fire. The assumption behind this project approach to problem solving is that once the process is fixed, it should stay fixed. However, since the bulk of the process variation comes from the uncontrolled variables, this assumption is never true. Your process is always going to change, and the average half-life between these changes is two weeks!

These changes occur in two ways. The set of uncontrolled causes may contain an unknown cause with a dominant effect, or a cause with a small effect may morph into one with a large effect. Either way, the uncontrolled variable with a large effect becomes a source of exceptional variation. Such variables are called assignable causes, and all assignable causes need to be added to the set of controlled process inputs. When we do this, we gain additional leverage to adjust the process average, and we remove a dominant chunk of variation from the process outcomes.

How do we discover these assignable causes? Aristotle told us how to do this: The way to discover an assignable cause is to pay attention to those points where the process changes. Because these changes can occur at any time, you will need to be watching the process in real time to know when a change occurs. That is exactly what a process behavior chart was created to do.

Summary

The assumption behind project-based improvement is that a process will stay fixed once it is fixed. This is contrary to experience and the laws of nature. Entropy alone will guarantee that your process will change.

Running a series of experiments will allow us to get specific answers to specific questions. But experiments are of limited value when we don’t know what questions to ask. This is what makes experiments so effective with problems along the first axis, but less effective with problems on the second and third axes.

Our cause-and-effect diagram (laundry list) is never complete. There will always be unknown causes. Experiments can only be used with the known causes, so here we need to shift from experimentation to observation. We need a way to study the system as a whole to determine whether there are any causes with dominant effects that are not yet in the set of controlled inputs.

Controlled process inputs directly affect the process average, but only indirectly restrict the process variation. Experimenting with the set of controlled variables will not have much effect on the process variation.

Virtually all of the process variation comes from the uncontrolled causes. Since many of these uncontrolled causes are unknown, the only comprehensive approach to reducing variation is to use a process behavior chart and wait for the assignable causes to make their presence known.

So, while experiments work with problems that lie along the first axis, problems that lie along the second or third axis require a process behavior chart. Without the completeness of this observational approach, you will continue to “solve” the same problem every year as the unknown assignable causes take your process on walkabout.

While project teams may always be needed to fight fires, the sustained use of process behavior charts in production will dramatically reduce the number of fires. You will no longer need to guess which problem to study or which variables to experiment with. You will no longer need to follow some elaborate framework for problem solving. Just listen to the voice of the process, identify assignable causes when they make their presence known, and then make these causes part of the set of controlled process inputs. As has been proven over and over, this sustained use of process behavior charts will allow you to reduce variation, operate your process up to its full potential, increase quality, increase productivity, and improve your competitive position.