Solid Proof- Assess reliability of audit evidence for effective risk management

In general, the audit process is similar across most industries. From financial audits to quality audits, the following key principles apply:

- Auditing is conducted against an agreed-upon standard.
- Auditors assess controls for adequacy and compliance.
- Audits collect evidence.

The information obtained is used to assess risk and plan for risk mitigation. ISO 9001:2008 and ICH Q91 provide guidance for the overall process of evaluating a supplier. ISO 9001 is a generic, internationally accepted quality management system standard that is relevant to most businesses. The ICH Q9 guide, "Quality Risk Management," provides guidance for managing risk based on the same risk management principles that are effectively used in many areas of business and government.

The ICH Q9 guide states: "Risk evaluation compares the identified and analyzed risk against given risk criteria. Risk evaluations consider the strength of evidence for all three of the fundamental questions."2 These fundamental questions are:

- What might go wrong?
- What is the likelihood (probability) it will go wrong?
- What are the consequences (severity)?

To effectively evaluate risk, we need an understanding of the reliability of the audit evidence obtained. But how do we best assess the reliability of audit evidence?

After reviewing quality auditing texts and articles, I found few resources that answer this question. The most relevant information came from financial auditing best practices, specifically one article that provided a guide for assessing the reliability of financial audit evidence.3 As a quality assurance specialist, I made some changes to the document to create the following guide, which will be useful for assessing the reliability of quality audit evidence.

Guide to assessment. First, consider six categories of evidence from which a quality auditor can choose:

- Confirmations.
- Documentation.
- Analytical evidence.
- Inquiries of the supplier.
- Retest.
- Observations.

To assess the reliability of the evidence obtained, we must consider the relevance, sufficiency and competence of the evidence collected. The following guidelines can help define these attributes.

1. Objectivity. Is the evidence objective or subjective? Objective evidence is achieved when two or more independent auditors are likely to arrive at the same result.

2. Documentation. Documented evidence, such as records, provides proof of compliance to procedures and is more reliable than verbal evidence.

3. Externality. Third-party evidence may be more reliable than evidence from within the organization being audited.

4. Sample size. Larger samples may be more reliable than smaller samples.

5. Sampling method. Was it appropriate?

6. Corroboration. Corroborated evidence is the same or similar to evidence from two or more independent sources. It may be more reliable than uncorroborated evidence.

7. Timeliness. Timely evidence is typically more reliable than evidence produced after a delay.

8. Authoritativeness. Evidence obtained from the machine operator may be more reliable with regard to how well a particular machine works than evidence from the engineer who built the machine. Consider the supplier’s evaluation history. What authority performed the audit or certification?

9. Directness. Interviewing and observing the operator perform the task may be more reliable than reviewing the work order steps. Also, an original document is more reliable than its copy.

10. Adequacy of controls. Evidence from a system or process adequately controlled is more reliable than evidence from a poorly controlled or questionable system or process.

Adapted from best practices from the financial industry, these guidelines can be useful for any organization in its quality audit processes and can benefit its overall risk management strategy. Only with reliable audit evidence can it assess risk and mitigate it effectively.

References and Note
ICH Q9 is a Federal Drug Administration standard on quality risk management developed by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Visit www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm128053.pdf (case sensitive) for details.
"ICH Harmonized Tripartite Guideline: Quality Risk Management Q9," International Conference on Harmonization of Technical Requirements for Registration for Pharmaceuticals for Human Use, November 2005, www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q9/Step4/Q9_Guideline.pdf (case sensitive).
Richard L. Ratliff and I. Richard Johnson, "Evidence - Audit Evidence - Includes Guidance on Audit Evidence," Internal Auditor, August 1998.

Article Credits: QP