| by Michael Bird
Laboratory accreditation isn't a new idea. Neither is compliance to ISO/IEC 17025, the governing standard for calibration and testing laboratories. Although ISO/IEC 17025 can be a formidable and complex standard that requires a collection of interwoven documents, it is one that we've come to depend upon as a critical tool for quality calibration.
Twenty-five years ago, there were few procedures or policies supporting instrument calibration. Little consideration was given to the measurement standards used in calibrations. Instead, the emphasis was on the person performing the calibration rather than on the quality of the system. Sometimes that wasn't enough.
Slowly, the calibration process began to evolve, first by emphasizing measurement standards and then by formally documenting the results. In its infancy, the quality system concept was sound. Although veteran calibration technicians could be heard grumbling in their labs, the system worked. It created a standardized approach to calibration and advanced the process from a simple art to the science of measurement.
Then the technical world took a surprising turn. Suddenly, quality system documentation was more important than the science and knowledge it supported. For some reason, it was decided that a proper operating procedure could act as a surrogate for knowledge and experience, and that "simple" tasks such as calibration, testing or even engineering were matters easily replaced by a procedure written by a knowledgeable person. Emphasizing the quality system rather than an individual's knowledge had a chilling effect on industry. At some point the knowledgeable person disappeared, and it was thought that the quality system alone could provide true quality products and services. Documentation superseded performance.
Unfortunately, nothing could be further from the truth. A couple of decades ago, we entered an age of calibration confidence companies, what I call "lick 'em and stick 'em" outfits that look awfully good on paper but lack the technical knowledge required for a true quality assessment. During the mid-'90s accreditation roared to the rescue, sorting out these less-than-desirable business entities and helping us navigate through a minefield of calibration wanna-bes.
At that time I worked for a large pharmaceutical company. Part of my job was to qualify vendors for subcontracted calibrations. At first we took care to audit each and every one of these companies by witnessing their operations first-hand. I met many knowledgeable people to whom I entrusted the integrity of our company's product. However, as time went on and staff was reduced, we relied increasingly on phone and fax audits. These worked surprisingly well, and I learned to sniff out the unscrupulous with amazing accuracy. But time was always a precious commodity, so our company began selecting ISO/IEC 17025-accredited laboratories as our preferred vendors. Again, this policy served us extremely well until about 2002, when I began to suspect that some of the accredited companies weren't what they claimed to be on paper. I even caught one vendor calibrating pipettes in the back of a 1962 Rambler station wagon, despite assurances of a "best measurement uncertainty" of ±0.002 µl and up-to-date accreditation.
Talk about an epiphany. Some ISO/IEC 17025-accredited companies were simply placing stickers on instruments without calibrating them. I received certifications with an accreditation-body logo and data attesting that the instrument was accurate to parts per million and, of course, the assurance of traceability to NIST and all the other effects that made it look like a quality calibration.
In response to feeling apparently solid ground caving in beneath me, I attended training sessions for ISO/IEC 17025 offered by various accreditation bodies and discovered the root of the problem: ISO/IEC 17025 guidelines focus only on documentation. No amount of documentation will replace the ability to evaluate a test report or, more important, an individual's or management's true intentions.
Take the 1962 Rambler and pipettes as an example. With an uncertainty of ±0.002 µl, the mass of this volume at a "maintained temperature of 20° C" is roughly ±0.002 mg, or two counts on a six-decimal-place balance. The accuracy alone of a 1 mg weight is ±0.002 mg. Given the factors required to calculate a pipette's uncertainty, this uncertainty statement--approved by an ISO/IEC 17025 auditor--is obviously absurd. An experienced calibration professional would have spotted this.
ISO/IEC 17025 Remote Auditing
Big lab
Retlif Testing Laboratories, a large testing lab with facilities in New York and New Jersey, is a $6-7 million dollar business. The company spends $50,000 yearly to maintain its ISO/IEC 17025 accreditation, according to Walter Poggi, Retlif's president. Although Poggi isn't completely sold on Bird's self-certification/remote-auditing approach, he does see value in it.
"The cost of accreditation somehow needs to be controlled, whether through competition or a more logical approach," says Poggi. "Maybe an on-site visit every other year, or internal audits and sending to the auditor to review them. There should be some type of combination that could make lab accreditation more affordable. We're not getting the growth in small labs today as we did during the '80s, and the problem is the cost of accreditation."
Small lab
CPM Labs is one of those small labs for which the lower cost of remote administration would help. The four-person calibration lab, based in Rancho Cordova, California, has been in business for 24 years. The company has been unable to bid on a few jobs for lack of ISO/IEC 17025 accreditation and is now seeking it. CPM estimates that certification costs will represent 25-30 percent of its discretionary budget.
Craig Howell, CPM's president, acknowledges that in the long term accreditation will probably mean more business, but it's difficult to ignore more immediate uses for the money.
"I could invest this money in tools and training, things that would make me money in the near term," says Howell. "For instance, I could purchase a Fluke calibrator that's worth about $50,000 a year in extra electronic equipment calibration business. We've also had a big increase in business lately, and I would like to hire another employee--we have enough work--but instead I'm saving that money for accreditation."
ILAC
The International Laboratory Accreditation Cooperation (www.ilac.org) evaluates lab accreditation bodies worldwide in much the same way ANAB might evaluate and accredit ISO 9001 registrars in the United States. Members of ILAC have been assessed by the organization and other members (i.e., peers) against the requirement of ISO/IEC 17011, the general requirements for accreditation bodies. ILAC takes a dim view of remote auditing.
"The ISO/IEC 17011 international standard requires accreditation bodies to perform on-site assessments before accreditation can be granted or continued (ISO/IEC 17011, subclause 7.7)", ILAC wrote Quality Digest . "ILAC member laboratory accreditation bodies always perform on-site visits to assess compliance to the international standard, ISO/IEC 17025…. ILAC would never recognize and the ILAC members would not allow such a practicing accreditation body to be part of their fraternity (i.e., ILAC). Furthermore, ILAC is not considering any shift to its current practice of on-site assessments."
|
There is nothing wrong with the ISO/IEC 17025 standard itself. The problem lies in an accreditation process that doesn't ensure quality workmanship and, because of its cost, endangers the livelihood of thousands of small calibration labs. Based on my years of experience as a calibration professional (and ISO/IEC 17025 auditor) I have identified a few of the key issues.
• Auditor competency. The ability to evaluate a test report comes from experience and the extensive knowledge required to gain it. Training sessions can teach only the mechanics of what to look for on a test report. A two-day, five-day or even ten-year ISO/IEC 17025 training session can't replace years of field experience. Auditors must have honest-to-goodness calibration lab experience to spot bad practice within "good" documentation.
• Consulting vs. auditing. As with ISO 9001, ISO/IEC 17025 auditing guidelines prohibit the auditor from also assisting in any meaningful way with a company's processes. I disagree. If certain deficiencies are noted during an audit, and the assessor is aware of the solution, he or she shouldn't be prohibited from telling the assessed entity what's needed to fix the problem. Does this cross the line between assessor and consultant? Of course it does, and it's the proper action to take. When a traditional accreditation organization notes a deficiency, it leaves the premises, files a report and recommends a "qualified" consultant who will help the company. The consultant racks up a nice bill, and the assessors are called back to the scene of the deficiency. Many dollars later the situation is resolved and the laboratory is accredited.
Isn't recommending a third-party consultant to resolve audit deficiencies just as much a conflict of interest as the auditor giving advice? When an ISO/IEC 17025 auditor recommends a third-party consultant to assist in an organization's accreditation, and that consultant must be paid to come up to speed on what the auditor already knows about the company, does this provide value to an assessed company, or is it just an additional source of income for the "experts?" Why not let an informed assessor resolve the issue? This would help bring the cost of accreditation down to the point where small businesses could afford it.
• On-site vs. online audits . Let's face it: Every day an assessor stays on site, he or she costs the business productivity and time, in addition to the cost of accreditation, travel and hotel expenses. Most of an assessor's time is spent studying documents and then relaying questions regarding them to the company. Why can't this be done remotely?
Evaluation should begin by having the assessed company send the ISO/IEC 17025 questionnaire and appropriate sets of documents to an assessor with calibration experience. The more involved the assessed company becomes in preparing and collecting these materials, the more knowledgeable it becomes about the standard. In effect, the assessed company is accomplishing a self-accreditation. The assessor can look over the documents and then communicate by phone, fax, e-mail--or any of the other 21st-century methods--to determine the efficacy of the assessed quality system. That way, a company's true quality is evaluated by an experienced and knowledgeable assessor, someone who can look at a test report and tell whether data are falsified--someone who can tell, just by talking to the assessed, whether the company cares about quality.
At this point, a determination can be made about the necessity of an on-site visit. If the company has done its homework, then the on-site visit can be minimized and costs kept down. There are a number of ways to evaluate the adequacy of a company's technical ability, the least of which is eyewitness surveillance. Looking closely at documentation, questioning procedures or checking on standards are all effective methods. During my ISO/IEC 17025 assessor days, I found that an on-site visit was mostly a dog-and-pony show. In the end, the show meant little; most deficiencies were caught on paper.
The accreditation world must make an effort to preserve the small calibration and testing lab because here's where you'll find breadth and depth of knowledge greater than any calibration mill. These companies, though they may lack accreditation, know the true meaning of quality and customer service. Yet many are losing accounts, in some cases to unqualified ISO/IEC 17025-accredited companies.
The onus is on national and international organizations that mutually recognize accreditors to evaluate and recognize accreditation organizations by their personnel, reputations and true intent. At the same time, these international organizations must acknowledge the value of performing off-site audits, provided the assessors have the proper experience. Neglecting to do this will jeopardize small businesses as well as the true intent of ISO/IEC 17025.
By depending too heavily on process and documentation, we fail to see the value that exists whenever a quality calibration is performed. A quality system should serve as a bridge between documentation and performance. Changing our approach to accreditation will align performance more consistently to global standards and values.
Management's intent, good or bad, is always embedded throughout an organization. If a company was a quality lab before a certification or accreditation, it will continue to be one after certification or accreditation. Documentation aside, do these labs care about quality? By the same token, quality labs that are looking to become accredited must evaluate the accreditation organizations and their personnel.
Michael Bird is president of International Laboratory Assessment and Accreditation (www.ilaaonline.org) . He's a professor of physics at the College of Lake County in northern Illinois. Bird earned a master's degree in physics from the University of Utah, and a bachelor's degree in physics and math from Elmhurst College.
|
