by Sidney Vianna

Variation reduction is a key concern for the aerospace industry. Standards have even been established to manage variation reduction of key characteristics contained in flight hardware (e.g., AS9103).

It is not, therefore, surprising that, after several years of deployment of the industry-controlled other party (ICOP) scheme, we come to the realization that there are unacceptable levels of variation in audits performed by certification body (CB) auditors. This leads to concern over how much confidence can be placed in some of the existing aerospace quality management certificates.

At the time of this writing, the Online Aerospace Supplier Information System (OASIS) database reported:

• 8,546 organization sites registered to AS/EN 9100

• 153 organization sites registered to AS/EN 9110

• 467 organization sites registered to AS/EN 9120

• 95 accredited certification bodies

• 740 Aerospace Experienced Auditors

The aviation, space, and defense (AS&D) ICOP scheme was developed as a means for industry stakeholders to rely on quality management system certificates as a component of their supplier approval and surveillance processes. The goal was to establish a process whereby customers could rely on an AS/EN 9100 (or 9110 and 9120) certificate to signify that the certified supplier has an aerospace quality management system (AQMS) that:

• Is robust enough to identify customer and regulatory requirements

• Consistently delivers products compliant to such requirements

• Allows appropriate actions to be taken based on customer and regulatory feedback

• Provides for continual improvement of its operation, system, process, and products

Unfortunately, there is evidence that a number of certified suppliers have poor quality performance. For example, instances have been reported where AS/EN 9100-certified organizations have allowed nonconforming material to enter the supply chain (i.e., “quality escapes”), creating serious problems for their customers and the end-users of the products. Other examples include certified suppliers that continuously fail to improve their low supplier ratings, fail to implement effective corrective actions, or have ongoing unacceptable delivery performance.

A number of people, working under the auspices of the International Aerospace Quality Group (IAQG), are concerned with the audit variation and substandard organizations attaining and maintaining certification. They have embarked on several initiatives to improve the outcome and strengthen both the processes and the confidence that can be placed on the certificates issued in the AS&D sector.

Underestimating auditor-days
Some of the concerns associated with audit variation and poor process performance have been identified. Due to the commercial aspect and competitive nature of the certification business, it has been observed that CBs tend to estimate minimum acceptable auditor-days, which are sometimes below the threshold and are inadequate for the performance of the audits. Allowing adequate audit time is critical for auditors to assess a representative sample of the processes that comprise the quality system under evaluation, so inadequate audit time is a major concern. If auditors are not given enough time to follow the necessary audit trails or investigate, in depth, all of the critical processes within the system, erroneous conclusions might be reached about the effectiveness and compliance of the system being audited.

The estimation of the audit time required by CBs when conducting audits under the ICOP scheme follows the same protocol described in Annex 2, page 34 of the International Accreditation Forum (IAF) Guidance Document on the Application of ISO/IEC Guide 62:1996. Incidentally, ISO/IEC Guide 62:1996 has been replaced by ISO/IEC 17021:2006, and all CBs are required to have their accredited management systems compliant with the 2006 document by Sept. 15, 2008. However, the IAF guidance document is still in place.

The methodology for estimation of auditor-days takes into account the size, in terms of number of employees, maturity of the system, and complexity of the organization, and also considers possible exclusion of processes. This time is augmented by specific requirements contained in the AS9104 standard, which contains requirements for aerospace quality management system certification and registration programs. The estimation process does allow for some subjective assessments, and it has been observed that some CBs, to maximize their competitiveness, are quoting unacceptably low audit-days. To address this, it is expected that the upcoming AS9104-1 document will increase the number of audit-days required, as well as tighten up the allowances for audit-day reductions.

Soft grading
Another aspect under scrutiny is the practice of “soft grading” nonconformities. Soft grading refers to situations when CB auditors have found that a nonconformity exists, but fail to report the finding as a nonconformity and categorize it appropriately. The auditor may rationalize the finding as isolated or document it incorrectly as an opportunity for improvement. AS9104 describes the definition of a nonconformity and the need for adequate categorization as follows:

“3.9 Nonconformity:

The absence of, or the failure to implement and maintain, one or more quality management system requirements, or a situation that would, on the basis of available objective evidence, raise significant doubt as to the quality of what the organization is supplying.

“Major: The absence of, or total breakdown of a quality management system element specified in the AQMS standard or any nonconformities where the effect is judged to be detrimental to the integrity of the product, processes, or service.

“Minor: A single system failure or lapse in conformance with a procedure relating to the AQMS standard.

“Note: A number of minor nonconformities against one requirement can represent a total breakdown of the system, and this can be considered as a major nonconformity. Any finding (e.g., comments, observations) that equates to the above definition shall be documented as a nonconformity, categorized as major or minor, and dealt with in accordance with 8.3.”

The implications of soft-grading nonconformities are serious. If the finding was only noted as an observation or opportunity for improvement, the organization being assessed would not need to perform effective root cause analysis and implement corrective actions to prevent the recurrence of such nonconformities.

In practical terms, this allows known system deficiencies to linger, with the risk of performance degradation. Soft grading also results in higher audit scores, computed via the AS9101 checklist. Soft grading may also be misused to allow for expedited certification of registrants. Remember that one of the requirements contained in the AS9104 standard states that no certificates shall be issued until corrective action has been accepted and verified for all nonconformities identified in certification and recertification audits.

One might then ask, “Why would some CB auditors soft-grade nonconformities and be part of this insidious practice?” Once again, a possible answer can be derived from the competitive nature of the certification business. Because CBs are selected and contracted directly by the registrants, they too have to deliver on the “customer satisfaction” expectations of their clients. Some misguided clients apply undue pressure on the CB auditors not to report nonconformities, trying to artificially boost their audit scores and prevent the initiation of corrective action requests, which could be time-consuming and require significant investments to perform root cause analysis and implement fixes. However, both parties promoting and allowing soft grading fail to realize that, by being accomplices in this practice, they are undermining stakeholder confidence in the ICOP process.

Possible elimination of the categorization aspect of nonconformities is being discussed, as well as the stoppage of the scoring feature, contained in the AS9101 (AS9111 and AS9121) checklist.

Auditor competence
Auditor competence is being closely examined as a possible contributor to some of the concerns under the ICOP scheme. Even though there are explicit requirements describing the minimum requirements for aerospace professionals to qualify as an Aerospace Experienced Auditor, experience shows that auditors differ considerably in areas that are crucial for the execution of an effective audit: knowledge, communication skills, thoroughness, tenacity, resistance to intimidation, ability to identify and focus on areas of higher risk, and so forth. These are all critical aspects of auditing. Unfortunately, there is a large variation in performance within these areas. The IAQG is considering alternative means for the demonstration and attestation of auditor competence. We might see enhancements to the body-of-knowledge requirements for Aerospace Experienced Auditors. The auditor approval and competence maintenance processes are to be strengthened. Further, it is likely that the oversight activities of CBs and auditors will be increased, since the percentage of witnessed auditors is small at present.

In the meantime, in the United States, under the leadership of the Americas Aerospace Quality Group (AAQG), the third aerospace CB auditor workshop will be held in Long Beach, California, July 14-15, 2008. The first two annual events were held in Milwaukee in 2006 and Atlanta in 2007. These workshops are aimed at providing face-to-face feedback to CB auditors about areas that have been observed as being in need of improvement during the oversight process performed in the previous year. Besides the soft-grading issue, auditors have been and will continue to be reminded of critical aspects of audits, such as:

• Adequate audit planning, which must take into account feedback originating from the customers of the certified organization

• Performing process-based audits

• Verification of closure and the effectiveness of corrective actions

• Importance of delving into customer and contractual requirements

• Timeliness and completion of the audit results in the OASIS database

Both the European (EAQG) and Asia- Pacific (APAQG) Aerospace Quality Groups are very interested in the Americas aerospace auditor workshop experience and are considering replicating these workshops. The quest to improve auditor competence and consistency is, without question, a worldwide concern.

A recent feature has been incorporated into the OASIS database that allows for any organization to provide feedback about an issue concerning supplier performance. The feedback is routed to the CB responsible for issuing the certificate and to the supplier in question. Such feedback should be used by the CB as a relevant input, as part of the planning of an upcoming audit, or even to trigger a special surveillance audit, should serious questions be raised about certification validity. Notwithstanding how the issue will be handled, the people behind the ICOP scheme expect that the CBs will pay close attention to the feedback being provided and act in an appropriate and timely manner. According to sources connected with the OEM members of the AAQG, the provision of feedback to CBs is expected to increase significantly in the near future.

Future steps
So, at this major crossroad, how will the ICOP scheme evolve? There are indications that the IAQG response to most of these concerns will be twofold. First, the requirements associated with the ICOP scheme will probably become more refined and detailed to minimize potential loopholes that some believe exist within the current requirements. The upcoming additional controls should not create unnecessary bureaucracy and result in overly prescriptive methods, but level the playing field and increase consistency. The second aspect is to strengthen the oversight process and keep all participants accountable. If CBs and auditors blatantly and willfully disregard the rules, there will be consequences for both. It should be mentioned that, since the inception of the ICOP process, a number of CBs have had their accreditation suspended and a smaller number have had their accreditation withdrawn.

Certified suppliers must also remember to contribute to the health of the ICOP scheme and to play their role to maintain the integrity and value of the process. Failure to maintain the ICOP scheme’s effectiveness will probably mean that suppliers will be exposed, as in the past, to numerous repetitive and inconsistent second-party audits. Although customers will always have the right to perform on-site audits of their suppliers, if greater confidence can be placed on a quality management system certificate, customer auditors will focus their time and effort on issues that are specific to their contracts, such as process and product evaluations.

Suppliers must demand that their respective CBs perform the work according to the rules and demand that audits satisfy duration requirements and scope. Auditors must be competent, knowledgeable, and thorough. An auditor who keeps presenting a clean bill of health to a substandard supplier is akin to a doctor who fails to diagnose and report early on a growing malignant tumor. Although the mistaken diagnosis may be welcomed in the short term, it will have dire consequences for the patient in the future.

About the author
Sidney Vianna is the director of aviation, space, and defense for DNV Industry North America. He is a nonvoting member of the Registration Management Committee, under the Americas Aerospace Quality Group.