This report summarizes research evidence on audits of Occupational Health and Safety (OHS) management systems. The focus is on determining what is known about the reliability and validity of these audit instruments. Besides reviewing literature in occupational health and safety, the review team also looked at findings from the process safety field.
Background
OHS auditing is a process for assessing the compliance of an OHS program with legislation and regulations and verifying that it conforms to established guidelines or best practices in occupational health and safety.
The growing number of international and national standards and agreements that incorporate environmental, health and safety aspects of production has encouraged widespread development and use of OHS audits. Auditing is now seen as an effective method for ensuring compliance and improving the performance of prevention systems.
But despite the widespread and expanding use of audits in Canada and internationally, there has been no synthesis of the literature on their effectiveness or on the reliability and validity of audit instruments.
Following a feasibility study, IWH researchers decided that a full systematic literature review on the effectiveness, reliability and validity of OHS auditing was not appropriate, because the evidence base was scant and many of the studies were of limited quality. Instead, they carried out a traditional, narrative literature review in order to summarize the available information for stakeholders.
What is a narrative review?
A narrative review provides a general overview of the research literature in a specific area. Unlike a systematic review, it does not systematically appraise the quality of articles nor does it formally synthesize results. Narrative reviews may be more appropriate when research in a particular area is scarce.
Narrative reviews can also help to provoke thinking on a topic and to identify gaps in the research. However, because they deal mainly with theory and general information, they tend to be less helpful to decision-makers seeking answers to specific intervention-related questions.
What is an OHS management audit?
An “audit” is a detailed examination or analysis of strengths and weaknesses. Most experts emphasize that OHS auditing is more than a hazard identification exercise – it should also involve a comprehensive examination of the whole OHS management system itself.
Researchers have described six types of OHS audits:
- OHS audits on specific topics (such as human factors or hazardous substances)
- plant technical audits
- site technical audits
- compliance or verification audits (such as compliance with legal or internal standards)
- validation audits (such as the design of OHS management systems themselves)
- management safety audits
For this review, an OHS audit was considered to be a systematic assessment of an OHS management system, defined as: "the integrated set of organizational elements involved in the continuous cycle of planning, implementation, evaluation, and continual improvement, directed toward the abatement of occupational hazards in the workplace."
What measurement concepts relating to audits were considered?
Various measurement concepts can be applied to OHS management system audits. The following were considered relevant for this review and are defined in the context of audits:
- variation in responses (the variation in audit results in relation to the possible range of results for a selection of workplaces)
- interrater reliability (the consistency of audit results when carried out by different auditors or auditing teams)
- test-retest reliability (the consistency of audit results when carried out twice separated in a relatively short time period)
- responsiveness (the extent to which audit scores can show change when there is meaningful change in the OHS management system)
- content validity (the extent to which the audit questions or criteria are complete according to the definition of a particular OHS management system)
- construct validity (the extent to which relationships between audit measures and other OHS-related measures (e.g., injury rate, safety climate) have been tested and the findings are consistent with those expected by theory)
What factors might influence the reliability and validity of audits?
From the existing literature on audits, the reviewers compiled a list of factors that could affect the reliability and validity of OHS audits.
- Auditor-related factors include issues of competence, bias and independence of the individuals carrying out the audit.
- Process-related factors include: the theoretical basis for the audit; the existence of a coherent and comprehensive audit framework; the existence of clear standards for comparison; the use of multiple information sources; how you choose who to speak to and where to look when auditing a workplace; the weighting of various audit components; quality control issues; and details about procedures and objectivity.
What research was included in this narrative review?
The search strategy targeted management audits. It was developed from a small sample of known relevant articles in MEDLINE, which is a bibliography of journal articles from the broader medical literature.
Literature search
Search strategies were developed and then applied similarly to five electronic databases covering the fields of medicine, management, economics, and occupational health and safety.
Study relevance
The relevance of titles and abstracts was determined by applying two criteria:
- The publication had to contain in formation on the reliability or validity of OHS management system audits.
- The publication had to be a journal article, a book, a conference proceeding or a report; magazine articles or newsletter publications were excluded.
Review process, data extraction and report generation
Both authors independently reviewed all retrieved publications. The content of the publications was then discussed and a general, common understanding of the findings was established. Each author then extracted pertinent information from his/her assigned articles and summarized it.
Findings
The results of the review are summarized below in two sections. The first section focuses on instruments designed for the audit of OHS management systems. The second section is concerned with audit instruments intended for the safety management systems of high-hazard and high reliability operations. While the second group of instruments were not specifically concerned with OHS, they were included because they overlapped conceptually with the first group and came from a different disciplinary stream.
Researchers found and reviewed 11 distinct audit instruments designed for the audit of OHS management systems. They are:
- the Diekemper and Spartz (D&S) method
- the Method for Industrial Safety and Health Activity Assessment (MISHA)
- the International Safety Rating (ISR) system, both a generic version and one for the mining industry
- the Complete Health and Safety Evaluation (CHASE), a group of related audits developed through a collaboration of academics and private industry in the United Kingdom
- an adaptation of the Program Evaluation Profile (PEP) originally developed by the U.S. Occupational Safety & Health Administration (OSHA)
- two different Goodyear Tire and Rubber Company audits
- an audit instrument for small- and medium-sized organizations based on the Australian and New Zealand OHS management system standard (AS/NZS)
- the American Industrial Hygiene Association (AIHA) ISO 9001 harmonized OHS management system audit
- the AIHA Universal OHSMS Assessment Instrument
Researchers also found and reviewed 14 articles concerned with management systems auditing in high-hazard (or “high reliability” operations). They identified a number of audit methods have been used in high-hazard industries and high reliability operations. They are:
- MANAGER, a process safety management system evaluation technique developed in 1986 for use in quantitative risk assessment in the chemical industry
- Process Risk Management Audit (PRIMA), an auditing tool for the quantitative assessment of process safety management systems originally developed in the U.K.
- the assessment of the management of maintenance in the chemical industry (not named)
- the Safety Management Assessment System (SMAS)
- the I-Risk audit
Summary of evidence on reliability and validity of OHS management audits
Overall few studies were found which examined the reliability and validity of OHS management audits. Here are some key findings:
- Only the American Industrial Hygiene Association instruments could be considered to have strong content validity. The reports on other instruments lacked this information.
- There was some evidence of construct validity for the four instruments that measured improvement in audit scores following an intervention on the OHS program. However, the amount of information provided was usually insufficient for judging the role auditor bias might have played
- The evidence provided by the investigation of the construct validity of the ISR audit is weak at best. No other studies attempted to validate audit results against a quantitative injury rate criterion
- No research looked intentionally at the responsiveness of audit instruments to changes in an organization’s OHS program. But some studies provided data that allowed IWH reviewers to calculate effect sizes, which ranged from medium to medium-large
- Interrater reliability was the measurement property studied most. It was found to be lower than expected for the D&S method, MISHA method and the AIHA Universal OHSMS audit
Summary of evidence on reliability and validity of safety management system audits in the high-hazard and high reliability operations
- There was surprisingly little research in the area of management systems in high-risk operations. Despite this lack of evidence, audits are widely used in this sector, and, in fact, are required by numerous regulations governing high-hazard processes.
- The studies provided some evidence of content validity. All audit methods were well supported by past reviews of accident causation and most authors presented theoretical frameworks.
- Some evidence of construct validity was provided by studies of MANAGER and PRIMA through the comparison of audit findings to other safety performance indicators
- Although a number of studies reported results of audits conducted by independent assessors, no formal tests of reliability were reported
Discussion
In general, researchers found relatively few research studies on the reliability and validity of OHS management audit instruments. Given the common use of these tools, there is ample room in the literature for more information about their measurement properties. For example, interrater reliability was studied in only a preliminary manner in the literature concerned with audits in the high-hazard sector.
Certainly there are obstacles to conducting validity studies to compare audit scores against a criterion like injury rate. Resource availability is one challenge, since audits often require several days on site. Availability and comparability of criterion data across work sites can sometimes be an issue, especially for in high-hazard organizations. There are likely to be fewer obstacles to the study of other measurement properties such as content validity, interrater reliability and responsiveness.
Limitations of this narrative review
One limitation is that the literature search was not exhaustive, so relevant information may have been missed. However others have also found the research literature on OHS management audit reliability and validity to be sparse. There might be relevant information outside the research literature, but no attempt was made to access it.
The reviewers are less confident that the evidence available on audits in high-hazard sector organizations is thoroughly represented here. If this type of audit had been of primary interest to us, the bibliographic databases would have been selected to include more engineering sources.
Another potential limitation is that this was not a systematic review. After the feasibility study, researchers decided that a full systematic review on the effectiveness, reliability and validity of OHS auditing was not appropriate, mainly because the evidence base was scant and many of the studies were of limited quality. Instead, they chose to carry out a narrative review intended to provide potentially useful information to stakeholders. The literature search was nevertheless quite thorough and the same search strategy was systematically applied to all databases.
Conclusions
There is little published research information on the measurement properties of OHS management audits. The evidence that is available is often weak in quality. Among the studies discussed here, few had the primary intent to specifically look at the measurement properties of the instruments.
The findings from this narrative review raise questions about the OHS management system-based audit instruments which are in common use. It appears that a good deal of effort goes into developing the content for many of the audit tools reviewed. Unfortunately, a lot of this effort is ill-documented by researchers. It would be helpful if authors provided more detail about the conceptual models and definitions that guided their work, as well as the process used to draw upon expert opinion.
Few researchers have studied audit results alongside outcome criteria. A database with both quantitative audit scores and OHS outcomes would provide the basis for the weighting used in scoring different sections of a quantitative audit instrument.
Reports of audits being validated using outcome measures like injury rates are rare in the literature. Yet this is an important approach to audit tool validation. While there are real difficulties in carrying out such validation in the high-hazard sector, the challenges are less formidable in other industries. Analyses of audit results and injury rates could not only help the process of validation but also assist in audit development.
Construct validity has also been demonstrated in studies where the audit score was found to increase following an intervention on the OHS management system. There is still a need for more studies of construct validity, especially those which establish a relationship between audit scores and other measures of OHS performance such as safety climate.
In the few cases where interrater reliability has been systematically examined, it has often been found to below. This was found even when the tool had superior content validity. Low interrater reliability is not a large concern when audits are used only for initial diagnostic purposes. It is a concern when they are used to measure ongoing progress in the development of an OHS management system, and when they are used to certify a certain level of OHS management system quality. There should therefore be a greater expectation for reports on the reliability of audits used for such purposes―both in research studies and in the “real world.”