One thing burned into the brains of those who survive a statistics class is that you have to specify an alpha-level before you do anything statistical. And when it comes to statistical inference, they are correct. But just what does the alpha-level represent? What does it mean in practice? Read on to find out.

The decision problem

With experimental data, the essence of statistical analysis is the decision regarding the relationship between a factor and some response variable. For example, if changes in the level of Factor A do not result in changes in the level of the Response Variable Y, then Factor A does not have an effect upon Variable Y. (This is usually called the null state.) If changes in Factor A do result in changes in the level of Variable Y, then Factor A does have an effect upon the response variable. (Call this the alternate state.) The experimenter must decide which state of nature exists. Thus we have the decision matrix shown in figure 1. On the left side we have the two possible decisions we can make, and across the top we have the two states of nature. The resulting two-by-two grid contains two correct decisions and two incorrect decisions.

 …