Big Data Have Arrived

But do you know their quality and origin?

There has been an explosion in new technology for acquiring, storing, and processing data. The “big data” movement (and its resulting sub-industry, data mining) is becoming more prevalent and having major effects on how quality professionals and statisticians do their jobs.

Big data are a collection of data sets that are too large and complex to be processed using traditional database and data-processing tools. Any change this big will require new thinking. However, one thing won’t change and now becomes more important. My respected colleagues Ron Snee and Roger Hoerl call this an “inquiry on pedigree,” which asks if you know the quality and origin of your data to answer the following questions:
• What was the original objective of these data, if any?
• How were these data defined and collected?
• What was the state of the processes that produced these data—both the data process itself and the process by which data were collected?

…

Want to continue?

By logging in you agree to receive communication from Quality Digest. Privacy Policy.

Create a FREE account

Forgot My Password

Comments

Problems with getting Accurate and Complete Data

Mr. Balestracci has hit the nail on the head - no data analytics solution will be able to yield valid results unless diligence has been done on the data. Is the data complete, self-consistent, and accurate? Many of the test and quality datasets that our team reviews have vast disparities in data formats and schemas before being synchronized. A lot of companies try to implement a system themselves, only to find that the data is not properly formatted and scrubbed to provide the analytics that they need. Even some commercial companies are more "glitz" than substance.

IntraStage's customers are incorporating manufacturing data from multiple sources and formats - manufacturing floors, contract manufacturers, suppliers, field data, failure analysis, and engineering - to provide a full view of quality across the entire product lifecycle. Check out www.intrastage.com to learn more about how test and quality analytics are being used to improve yield and product quality by our customers today.