Three recent studies inform our understanding of the effectiveness of targeted inspections on workplace health and safety.
The province of Ontario annually spends more than $30 per worker on services related to the prevention of occupational injury and illness, of which approximately 50 per cent is allocated to labour inspection and enforcement services (OHSCO, 2010). Assessing the appropriate level of investment in labour inspection services is challenging, as is designing efficient and effective operational strategies by which these services are delivered to enterprises or economic sectors.
Three recent North American studies reported on the impact of labour inspections on the subsequent incidence of work-related injury and illness. All three studies described the outcomes of labour inspection programs that attempted to target inspection services on higher risk workplaces.
The targeting of regulatory inspection and enforcement activities on enterprises, sectors or activities with high concentrations of risk or regulatory non-compliance is an increasingly common principle in regulatory practice. In The Regulatory Craft, Malcolm Sparrow speaks about the importance of “systematic identification of important hazards, risks, or patterns of non-compliance” (Sparrow, 2000, p. 100). Among the justifications for targeting regulatory enforcement resources to the highest risk or lowest compliance are the scarcity of enforcement resources and the goal of maximizing regulatory efficiency.
This Issue Briefing summarizes the methods and findings of these three well-designed studies and provides an interpretation of the differences in findings across them.
Background
The economic burden of preventable occupational illness and injury may represent as much as three per cent of gross domestic product in developed economies (Leigh, 2011). The role of regulatory enforcement in worker health protection is debated. Some argue that regulation and enforcement are unnecessary, that competition among employers in a market free of regulation will naturally lead to the appropriate level of investment in occupational health and safety (OHS) to protect the health of workers. Others argue that, because firms do not bear all of the costs of occupational injury or illness (employer costs may represent no more than 10 to15 per cent of the true costs), firms in an unregulated market would fail to invest sufficiently in worker health protection. In this view, regulatory intervention is justified to correct market failure.
The effectiveness of regulation and regulatory enforcement is also debated. A 2007 systematic review authored by Institute for Work & Health staff found strong evidence that citations or penalties assessed during a labour inspection successfully reduce the incidence of work injury following inspection (Tompa, 2007). The same review concluded that the evidence for a general deterrence effect (where firms improve OHS practices based on knowledge of the consequences of inspection) was much more limited.
The three studies summarized in this Issue Briefing extend the evidence concerning the effectiveness of OHS inspections and enforcement by focusing on the impact of targeting high risk sectors or enterprises.
The three targeted OHS inspection programs
Targeted inspection programs in three North American jurisdictions were the subjects of the recent studies. These programs are briefly described here.
Ontario High Risk Firm Initiative
The province of Ontario has a labour force of approximately six million workers. In 2004, the Ministry of Labour (MOL) increased its complement of labour inspectors from 200 to 400, resulting in a doubling of the number of annual inspections under the Occupational Health and Safety Act (OHSA) to 90,000 per year. From 2004 to 2008, the MOL adopted a labour inspection and enforcement strategy known as the High Risk Firm Initiative. It directed approximately 25 per cent of OHSA inspections to the 10 per cent of workplaces, across all economic sectors, that were the poorest performers, based on the incidence and cost of work-related compensation claims over three prior years.
California High Hazard Enforcement Program
The state of California has a labour force of approximately 15 million workers. The California Division of Occupational Safety and Health (Cal/OSHA) inspects about 8,000 to 10,000 establishments per year, out of more than 700,000 in the state.
One enforcement strategy implemented by Cal/OSHA is the High Hazard Enforcement Program (HHEP). This program is focused on 13,000 employers (two per cent of all employers in California), representing 500,000 employees (three per cent of all employees in California) in 20 high risk sectors. High risk sectors are defined by a ‘DART’ (Days Away from work, Restricted work or job Transfer) rate equal to or greater than 200 per cent of the DART rate for all private employment sectors in California. To elaborate, DART refers to cases with days away from work, a job transfer or modified duties arising from a non-fatal occupational injury or illness. Over a 10-year period, HHEP conducted roughly 1,700 inspections among 65,000 firm-years of economic activity.
Washington State Division of Occupational Safety and Health
The state of Washington has a labour force of about 3.2 million workers. It is the only American state that administers both a workers’ compensation insurance scheme as well as a state-administered occupational health and safety program. The Division of Occupational Safety and Health (DOSH), within the Department of Labor & Industries, administers the Washington Industrial Safety and Health Act (WISHA) by developing and enforcing rules that protect workers from hazardous job conditions. Inspectors visit about 7,000 workplaces each year and additionally conduct roughly 2,500 consultations with employers who request assistance in complying with OHS requirements.
The DOSH inspection program is based on targeted inspections, anchored to criteria that emphasize high risk sectors, the experience rating status of individual employers and the absence of recent inspection attention. DOSH program performance information indicates that firms selected for targeted inspection and enforcement have workers’ compensation claim rates that are approximately 2.5 times higher than the average compensation claim rate in Washington State.
Study designs and outcomes
Ontario Under the High Risk Firm Initiative, 10 per cent of employers were selected for targeted OHS consultation services from one of Ontario’s publicly funded health and safety associations or labour inspection services. Employers were selected on the basis of a weighted ranking of compensation claim incidence rates and claim costs over the three years prior to the program year.
The two per cent of firms with the worst rankings were targeted to receive intensive inspection (four inspections in the course of a year) by MOL inspectors. After exclusions, a portion of the next eight per cent of firms were randomly assigned to one of three groups: firms allocated to consultation services, firms allocated to labour inspection and firms allocated to a control group. The latter firms were not targeted for consultation or inspection, but may have received attention through other program mandates, such as a response to an incident or complaint in the case of inspection, or a request for service in the case of consultation.
A recent study documented outcomes over a 21-month period following an inspection in the 2006 program year in the manufacturing sector (Hogg-Johnson, 2012). Firms with fewer than six full-time-equivalent workers (FTEs), firms targeted in the previous two program years or firms participating in a voluntary incentive program were excluded from all three study groups. A total of 2,153 manufacturing firms were randomized across the three study groups: consultation (N=600), inspection (N=619) and the control group (N=934). The total compensation claim incidence rate in this selected group of manufacturing firms was 14.2 per 100 FTEs in 2005. Statistical methods were used to model claim and disability day rates by study group and year, while controlling for firm characteristics.
Manufacturing sector firms were found to be very similar across the three study groups on the basis of their size, years in business, geographic region, rates of work injury claims, and rates of disability days over the three-year period prior to the 2006 program year. In the 21-month period following the 2006 program year, the rates of work injury claims declined by approximately 10 to 12 per cent in all three study arms. Rates of disability days also declined by equivalent amounts across all three study arms. The Ontario study found no difference in outcomes over the follow-up period between firms allocated to labour inspection and firms allocated to the control group.
California A recent study documented the outcomes of labour inspections conducted under California’s HHEP over the decade from 1996 to 2006 (Levine, 2012). Under HHEP, the state labour inspection service randomly selected firms from a list of all employers in 20 high risk industry sectors. The California study sample was primarily made up of small and medium-sized enterprises (the average number of employees was 34). The estimated annual injury rate was 10 per 100 FTEs.
Each firm selected for inspection was retrospectively matched to a firm eligible for, but not selected for, inspection. For both the 400 inspected firms and the 400 matched control firms, information on workers’ compensation claims over a four-year period following the inspection year was used to estimate the effects of inspection on injury frequency and costs.
The California study concluded that the annual frequency of work injury in inspected firms declined by 9.4 per cent more than it did in eligible firms that were not inspected.
Washington From 1999 to 2008, the impact of DOSH inspections in the state of Washington was monitored by Safety & Health Assessment & Research for Prevention (SHARP), a research and evaluation unit within the Department of Labor & Industries. An earlier evaluation, describing the impact of labour inspections over the period 1997 to 2000 (Baggs, 2003), was recently updated to describe the period 1999 to 2008 (Foley, 2012).
The SHARP study was based on a quasi-experimental design, comparing outcomes in the year following an inspection program year among firms selected for labour inspection with those among firms not selected for labour inspection. Firms inspected in the two years prior to the program year were excluded from the evaluation analysis. Firm outcomes were described for fixed-site employers (e.g. in agriculture, manufacturing, wholesale trade, retail trade, finance, insurance, services and general government) and non-fixed-site employers (e.g. in construction).
The study sample was further restricted to those firms with single business locations reporting employment of at least 10 FTEs per year and having continuous workers’ compensation coverage over the study period. The study sample represented approximately 15 per cent of all state-fund insured employers having continuous workers’ compensation coverage over the study period and documented the outcomes associated with approximately 15 per cent of all DOSH inspection visits.
In the earlier evaluation (1997 to 2000), approximately five per cent of fixed-site employers and 15 per cent of non-fixed-site employers received a labour inspection. In the year following the 1999 program year, the incidence rate for fixed-site employers who were inspected declined from 4.18 to 3.03 compensation claims per 100 FTEs, compared to a reduction from 1.41 to 1.36 compensation claims per 100 FTEs among fixed-site employers who were not inspected. Adjusting for firm size and for prior claim rate history, fixed-site employers who were inspected had a 22.5 per cent decline, compared to a decline of seven per cent in non-inspected fixed-site employers. Non-fixed-site inspected firms had a 12.8 per cent decline, compared to 7.4 per cent for uninspected non-fixed-site employers.
The updated evaluation pooled the results of all 10 previous annual studies from 1999 to 2008. The inclusion and exclusion criteria for the 10 annual studies were consistently applied. In the case of fixed-site industries, DOSH enforcement inspections were associated with a 4.3 per cent larger decrease in the incidence of compensation claims compared to fixed-site employers who did not receive a labour inspection. In the case of non-fixed-site industries, DOSH enforcement inspections were associated with a 3.1 per cent larger decrease in the incidence of compensation claims compared to non-fixed-site employers who did not receive a labour inspection.
In the year following inspections, fixed-site employers and non-fixed-site employers with one or more citations had a 20 per cent reduction in compensable claims (excluding non-traumatic musculoskeletal disorders), compared to firms that were not inspected. (The report did not include information on the proportion of inspections that resulted in a citation.)
Discussion
Two of the three well-designed studies summarized in this Issue Briefing—those looking at the programs in California and Washington—reported a greater reduction in work injury among firms selected for labour inspection services, compared to firms not selected for inspection. The Ontario study found no difference in work injury outcomes between inspected firms and non-inspected firms.
Two of the three studies summarized in this Issue Briefing used random assignment or selection methods to choose firms to receive labour inspection services. In Ontario, high risk firms were randomly assigned to labour inspection. In California, high risk sectors were the sampling frame for randomly selecting firms to receive labour inspection. The third study in the state of Washington used a quasi-experimental design where high risk firms were compared to the average experience of their sector.
Random assignment is the optimal study design for controlling threats to valid inference, including the potential influence of regression to the mean. (Regression to the mean refers to the phenomenon in which the average score of a group of poor performers on an initial assessment will tend to move towards the mean score of all tested subjects on a second assessment.) Regression to the mean is a substantial threat to valid inference in studies of targeted service delivery.
The Washington State program selected firms for labour inspection based on a history of higher compensation claim rates and compared outcomes to a group of firms with average compensation claim rates. Despite efforts to use statistical methods to adjust for the effects of regression to the mean, the results of the Washington State study remain vulnerable to the error of attributing changes over time to the impact of inspection when the changes may have occurred without inspection.
How might we interpret the contrasting results of the California and Ontario studies? Both studies were comparable in their reliance on workers’ compensation records as a source of information on firm performance following inspections. The length of follow-up was sufficient in both studies—four years in the California study and two in the Ontario study.
While both studies used random assignment or selection methods to allocate firms to labour inspection, there was a subtle difference. In Ontario, high risk firms were identified under the targeting program; in California, high risk sectors were identified and firms randomly selected within these sectors. This difference may have resulted in a stronger influence of the regression to the mean in Ontario compared to California.
One difference between Ontario and California was the intensity of the labour inspections. The Ontario MOL conducted 15 inspections per 1,000 workers. The rate of inspections in Washington State was two per 1,000 workers and 0.5 inspections per 1,000 workers in California. A potential consequence of the greater inspection intensity in Ontario is a higher probability of an enterprise experiencing an inspection.
The Ontario study documented that approximately 35 per cent of employers in the consultation arm, the inspection arm and the control arm received an inspection (a proactive visit) or an investigation (in response to an incident or complaint) in the year prior to the 2006 program year. These inspections and investigations were conducted outside the selection methods of the High Risk Firm Initiative. It was also the case that 22 per cent of firms, in both the consultation and the control group arms, received one or more labour inspections in the 2006 program year, which were also conducted outside the selection criteria of the High Risk Firm Initiative.
In the language of randomized trial design, the control groups in Ontario were partially ‘contaminated’ by exposure to labour inspection services. It is plausible that the higher intensity of labour inspections in Ontario ‘diluted’ the impact of the exposure to targeted inspections and enforcement in the 2006 program year.
There are other potential explanations for the contrasting results of the California and Ontario studies. It is possible that the methods for identifying high risk firms in Ontario were less accurate than the method used to identify high risk sectors in California. The fidelity of inspection service delivery also differed between California and Ontario. In California, seven per cent of 409 firms selected for inspection did not receive an inspection. In Ontario, 22 per cent of 619 firms allocated to the inspection group did not receive an inspection.
Conclusion
This Issue Briefing summarizes three studies of specific approaches to targeting high risk workplaces in the enforcement of OHS standards. Two of the studies found a positive impact on the incidence of work injury following inspections. Two of the studies were able to evaluate the impact of inspections on the incidence of work injury in a rigorous randomized assignment or selection design: one study reported a positive effect, while the other study reported no effect. However, in the jurisdiction with no effect, there was a much greater likelihood that firms not targeted for inspection had been recently inspected outside of the targeting program.
The California and Ontario studies demonstrated that random selection or assignment is feasible in the delivery of labour inspection services. Evaluation designs that incorporate randomized selection or assignment are an exceptionally strong method for evaluating the impact of programs, and they should be used to inform future assessment of the impact of targeting occupational health and safety inspection and consultation services.
These studies inform our understanding of the effectiveness of targeted inspections. They also leave us asking for more insight into how workplaces respond to regulatory inspections and how workplaces incorporate the lessons learned from these inspections.
References
- Baggs J, Silverstein B, Foley M. Workplace health and safety regulations: Impact of enforcement and consultation on workers’ compensation claims rates in Washington State. American Journal of Industrial Medicine, 2003; 43(5):483-494. doi:10.1002/ajim.10209.
- Foley M, Fan ZJ, Rauser E, Silverstein B. The impact of regulatory enforcement and consultation visits on workers’ compensation claims incidence rates and costs, 1999-2008. American Journal of Industrial Medicine, 2012 Jun 19. doi:10.1002/ajim.22084.
- Hogg-Johnson S, Robson L, Cole DC, Amick III BC, Tompa E, Smith PM, Van Eerd D, Mustard C. A randomised controlled study to evaluate the effectiveness of targeted occupational health & safety consultation or inspection in Ontario manufacturing workplaces. Occupational and Environmental Medicine, 2012 Aug 23. doi:10.1136/oemed-2011-100333
- Leigh JP. Economic burden of occupational injury and illness in the United States. Milbank Quarterly, 2011; 89:728-772.
- Levine D, Toffel M, Johnson M. Randomized government safety inspections reduce worker injuries with no detectable job loss. Science,2012; 336 (6083):907-911. doi:10.1126/science.1215191.
- Occupational Health and Safety Council of Ontario. System performance measurement report 2008, Final report, 2010.
- Sparrow M. The regulatory craft. Washington, D.C.: Brookings Institution Press, 2000.
- Tompa E, Trevithick S, McLeod C. Systematic review of the prevention incentives of insurance and regulatory mechanisms for occupational health and safety. Scandinavian Journal of Work, Environment & Health, 2007; 33(2):85-95.