Three basic evaluation methods exist for any work activity: inspection, compliance auditing and management auditing. The first method, inspection, measures a process's output against certain characteristics. These characteristics, generally identified as form, fit and function, are specified, and the process output either possesses those characteristics or it doesn't. As a result, an inspection's outcome is always binary: pass or fail.

 In contrast, compliance audits check on the implementation of written manuals, procedures and work instructions. The compliance audit evolved in the 20th century as business practices became more complex. The first use of compliance auditing appeared in financial transactions, because tax collectors and bank examiners needed assurance that the financial data were correct. This concept of verifying compliance was picked up by the quality profession in the 1960s and applied to the military and the nuclear power industry. Compliance audits are still used in high-risk activities, where there is a desire to verify that the activities are being performed in strict compliance to approved requirements. Third-party registration audits, regulatory inspections and most supplier audits measure compliance. The application of a compliance audit results in stability and assurance that rules are being followed.

 The management audit is a more recent concept. It focuses on results, evaluating the effectiveness and suitability of controls by challenging underlying rules, procedures and methods. Management audits, which are generally performed internally, are compliance audits plus cause-and-effect analysis. When performed correctly, they are potentially the most useful of the evaluation methods, because they result in change.

Compliance Audits vs. Management Audits

Whether performing a compliance or a management audit, auditors must obey four basic rules. First, audits must provide information for a defined need, that is, the customer's need. Second, auditors must be capable of performing their duties. Third, audits must measure performance against agreed criteria. Fourth, audit conclusions must be based on fact.

Rule 1: Serve your customers

 Audits provide information. All affected parties need to know if product, process and system controls are present and being applied, and obviously it doesn't hurt to know whether these controls actually work. An auditor evaluates the controls against requirements and produces a report. If controls are present and working, all parties' confidence in the process is increased. If controls are missing or not working, then resources can be applied to fix the problems.

 Auditors serve three customers: the auditee, the client and the organization. Auditees' primary goal may be to simply pass the audit, but auditees trying to derive the most benefit from the audit will also want to know whether the organization is functioning effectively. In this case, an auditor's outside perspective can be quite valuable. The client (the person who commissions the audit), in contrast to the auditee, is accountable for the auditors' actions and reports. Committees cannot generally perform this function; an audit boss should schedule the audits and make assignments. Finally, auditors must serve the organization's needs. Business values are important and the auditors can assist by determining whether the enterprise is actually achieving its goals.

Rule 2: Use qualified people

 Auditors must be able to carry out their assignments in an impartial and objective fashion. This means that they cannot have a vested interest in the activity being audited. If they developed the rules, they cannot impartially evaluate the effectiveness and application of those rules. Although an auditor can never be totally independent of the auditee, some separation must be maintained. It's fine to audit within your group, but you can't audit your own job.

 Auditors must also be capable of doing their jobs. They need certain emotional, intellectual and mechanical skills, which they can obtain by attending a course, reading a book or observing others. Often, all three methods are used. In addition to knowing how to conduct an audit, auditors must be familiar with the technical processes being examined. A good way to demonstrate this familiarity is to flowchart the activity to be audited--if a person can't flowchart it, he or she can't audit it. Finally, auditors need to be able to communicate well, both orally and in writing.

Rule 3: Measure against agreed criteria

 Auditors are not allowed to make up the rules--they must audit against performance standards that are already in place and accepted by the auditee. This is the planning part of the plan-do-check-act loop. The highest level of requirements includes corporate policies, management system standards and regulatory requirements. Usually originating from outside the auditee's organization, these requirements establish the goals and objectives to be achieved. National and international standards, such as QS-9000 and ISO 9001, fall into this highest category. Next comes the local approach, often called a quality manual or quality plan, for implementing these high-level requirements. It gives the framework for achieving the concepts and should be fairly compact. This document is then followed by a number of process-specific procedures. Further detail can be provided in work instructions, such as drawings, traveler sheets and sampling plans. One of an auditor's challenges is to obtain and become familiar with the many levels of requirements forming the basis for the audit.

Rule 4: Use facts to form conclusions

 Auditing is fact-based; conclusions are drawn from the data. Facts can be good (a requirement was met) or bad (a requirement wasn't met), but no judgment or opinion should taint them. These facts, also known as objective evidence, can come from five sources. They can be physical properties, such as flow rates and dimensions; sensory-derived input from seeing, hearing, smelling or tasting; documents or records; information drawn from interviews with auditee staff members; or patterns such as percentages or ratios. Auditors use checklists and other tools to determine the facts to be gathered, and then they perform the fieldwork to gather these facts.

 The output of the audit process, be it a management or compliance audit, is a report. The client (audit boss) receives the report from the auditor and delivers it to the auditee. To prepare a report, the auditor must take all of the positive and negative facts and make some sense of the data. In other words, the auditor must analyze the data.

 The first step is to list all of the positive and negative observations (data), then sort those data into controls or problem areas. Generally, there will be a large number of negative observations associated with just a few control items. This natural chunking of the data allows the auditor to see the patterns, rather than the individual events. For a compliance audit, these patterns are then reported as either conformities or nonconformities.

 Management audits require some additional work. The auditor needs to identify the pain associated with those groups of bad facts. (It's important to identify business problems, such as scrap, rework and overtime, as pain.) Then the auditor combines the missing control (the system error that's causing the problems) and the business pain into one statement, called a finding. The finding will reveal cause-and-effect patterns occurring within processes. Because the business pain is identified, there will be a tremendous desire to do something about it.

 By associating the negative facts with missing or weak controls, the auditor rises to the system level of analysis. This has lasting value, because the system affects the process, which affects the product or service.

Instilling a desire to improve

 Audits measure actions against requirements; they examine the product, process or system against performance standards. This has value when the requirements have been thoroughly tested and scientifically proven, but, unfortunately, this is rarely the case.

Management Auditor's Rules

Most manuals, procedures and work instructions are imperfect; they're the result of a small number of individuals assembling some rules with limited resources. By focusing on results, the management audit can determine whether those plans and approaches are any good. If they aren't, the developers and users are compelled to improve their methods because they can see the adverse consequences of not doing so. When employees and managers begin to see audits as opportunities to improve, they begin to see auditors not as police officers but as productive members of the organization.

About the author

  Dennis R. Arter is an independent consultant and trainer. He instructs large and small firms in the fields of management auditing and quality systems. Arter has served clients in the fields of government, manufacturing, chemicals, energy, research, aerospace, and food processing and is the author of the book Quality Audits for Improved Performance (ASQ Quality Press, 1994).

Arter is an ASQ Fellow and an active member of the Society's Standards Group, Customer Supplier Division and Quality Audit Division. He is responsible for coordinating all quality, environmental, dependability and statistics standards within the ASQ. He was on the team that developed the ASQ Certified Quality Auditor program and holds a CQA charter certificate. He managed the team that developed the Quality Audit Handbook, published by Quality Press in 1997. E-mail him at darter@qualitydigest.com .