Pitfalls in Experimentation and Data Interpretation, Part 3

Bad news sells and bigger is better

Ah, the scientific method. How elegant, how useful—and how easily ignored. The process of studying a problem, formulating a hypothesis, running a controlled experiment, analyzing the resulting data, and then making an objective decision is so quickly cast aside in the interest of quick-and-dirty data collection and analysis, which are too often faulty.

In part two of this series we looked at reliability testing for the M16 rifle, and test marketing for New Coke, prime examples of experiments that weren’t representative of reality. We also looked at the related pitfall of using a biased sample. In part one of this series, we looked at the dangers of not running an experiment or not verifying that your supplier ran an experiment (the perils of buying snake oil).

…

Want to continue?

By logging in you agree to receive communication from Quality Digest. Privacy Policy.

Create a FREE account

Forgot My Password

Comments

Pitfalls

Thank you for a very informative series on data interpretation. In my 30+ years in the product safety and performance testing field I have encountered many of these pitfalls and often done the experimentation and analysis to try and straighten out misconceptions. I guess this has made me a skeptic whenever I see reports of claims made by "scientists" in the press. Of course, anyone who brings up these types of data analysis errors in relation to the issue of global climate is now labeled a "science denier". So be it. But for those willing to look more deeply into the issue, I think you will find many examples of "False Correlation", "Using Biased Samples", "Bad News Sells", "Not Running Experiments", and "Extrapolation". Currently there is a lot of flap about the "pause" in global warming that has lasted more than a decade. One segment of the anthropomorphic global warming science community has been very busy trying to explain it away while another segment is denying its existence. In my view there is very little actual science in the so-called scientific consensus on this issue.

Lessons from our past

Great article with real examples of the effects of overlooking key considerations for experiments. Sad to say but our organizationn is guilty of a few of the pitfalls mentioned. I will definitely share this with my team to hopefully improve future experiments results.