As workplaces grow more aware about the toll of work-related psychological conditions, they may want to put in place policies and programs that address work-related risk for these conditions. But to identify workers at risk, workplaces need a tool that can reliably and accurately measure those risk factors.
A tool developed by the Public Services Health and Safety Association (PSHSA) was designed to do just that. It’s a survey that asks workers 20 questions related to five dimensions (about the worker and workplace) that together can contribute to the risk of work-related psychological conditions. For the initial tool development, a team at PSHSA and the Conference Board of Canada, now renamed Signal49, aimed to have a tool short enough that it’s easy and practical to use, but multi-faceted or holistic enough that it can capture the many factors that can play a role in raising or lowering the risk of occupational stress injuries—in particular among first-responder organizations.
Once it was developed, PSHSA wanted to make sure that the tool was tested to show that it works well as a measure; it also wanted to hear about ways it might be further improved. That was why the health and safety association turned to a research team at the Institute for Work & Health (IWH). The PSHSA asked IWH to review the development of the original tool, suggest improvements, and then conduct an evaluation of the tool’s performance. The team also wanted to demonstrate whether the score produced by the survey-based tool was associated with different mental health outcomes—both at the time the survey was taken, and approximately 12-months later.
The IWH study is now published in the peer-reviewed journal, Annals of Work Exposures and Health (doi:10.1093/annweh/wxag068). It finds that the tool is a valid, reliable instrument. The five dimensions it measures—participation in mental health activities; workplace supports, programs and policies; psychologically safe workplace; leadership; and mental health stigma—are correlated with workers’ risk of burnout, level of psychological distress and self-reported work-related psychological injury. What’s more, the correlation with burnout and psychological distress was found not just at the time of the survey but also 12 months later. In other words, the tool has predictive validity, which means it can identify workers who are at increased risk of these conditions in the future.
“It’s always difficult to measure prevention,” says Henrietta Van hulle, interim CEO of PSHSA. “Many organizations have been putting in place different programs, different peer support models, different return-to-work support models, different leadership practices. But it’s always very difficult to tell if these actually make a difference.”
The fact that a study has now validated the tool is “really beneficial to us,” adds Van hulle. It means PSHSA can confidently encourage workplaces to use it to measure their psychosocial work environment, to pinpoint areas where they need to make improvements and to use it again later to measure progress.
What the study set out to find
When the IWH team discussed collaborating with PSHSA on this study, “we made it clear we would really put the measure through its paces, and that this wouldn’t be a rubber stamp process,” says Dr. Peter Smith, IWH president and senior scientist, and lead researcher of the study. As part of the evaluation of the tool’s psychometric or measurement properties, the team also decided to run a head-to-head comparison of the tool and two more established measures of workplace psychosocial environments. The two other measures studied were the 12-item Psychosocial Safety Climate measure and the 11-item Psychological Job Quality Index. Both have been validated in a 2022 Australian study to be leading indicators of psychological injury, but they measure slightly different factors related to worker mental health.
The IWH team also suggested adding questions about mental health stigma to the existing tool. Stigma is a linchpin factor in that “if there’s a lot of stigma about mental health at your workplace, people will suffer in silence, leadership won’t hear about the mental health challenges, and people won’t talk about the impact of work demands on their mental health,” says Smith. He notes that the added questions give the measure the unique ability to assess mental health stigma alongside other workplace factors.
How the study was conducted
To conduct the evaluation study, Smith’s team asked a sample of about 1,200 Canadian working adults, from a broad cross-section of industries and types of workplaces, to complete all three tools. Participants were also asked questions designed to measure their levels of burnout and psychological distress. They were also asked whether they had a mental health or cognitive condition or disability that has made working difficult in the past year, and whether such condition was a result of their work.
After the first survey conducted in August and September 2024, the team then asked a small subset of the participants, about 120, to complete the tools again one month later. This second time checked for what’s called test-retest reliability—that is, whether the survey questions produced the same answers across multiple completions when there was no change in the work environment. The team were also able to repeat the survey in about 700 participants from the original group approximately 12 months later. This third check was aimed at assessing the tools’ predictive ability. In addition, the team conducted many other tests with the data to see whether particular items were redundant and whether the items measured the dimensions they were designed to measure.
“We did a variety of tests with this tool,” says Smith. “And in each case, it passed.”
The study found that overall, the PSHSA tool worked just as well as the two more established measures. In some cases, it worked even better. That is, the correlations it found between poor scores—indicating psychologically high-risk environments—and the outcomes of burnout, psychological distress and work-related psychological injury were stronger than those picked up by the other measures.
And although the measure was initially developed for first responder workplaces, the IWH study confirmed that the tool can be used in a broad cross-section of industries and organization types in Canada.
However, the study did note one limitation of the tool, which was the high level of missing responses for some items. Some were items related to workplace support: “We have an impactful peer support program in place” and “We have valuable critical incident protocols and appropriate debriefing processes in place.” And some were items about mental health stigma: “Most people think that people with a mental health condition could snap out of it if they wanted to.”
These workplace support questions may be missing responses because people don’t often know about the kinds of workplace programs that are in place unless they have used them, says Smith. As for the high level of no responses to the questions about stigma, people may not feel that they can say what most people think. Rephrasing the questions to be “most people around me…” or “most people I work with” may help get response rates higher, Smith adds.
“We gave it a thorough psychometric testing, and it’s proven to be a measure that can identify five different aspects of the workplace psychosocial environment in a fairly efficient way,” says Smith.
Smith credits the PSHSA for being willing to put the tool to the test. “It takes courage and a lot of confidence in your instrument, to approach an independent research organization like IWH and say, ‘Can you evaluate it?’ It should happen all the time, but it doesn’t,” says Smith.
The tool is now available through PSHSA. To learn more or request access, please contact the PSHSA through the website [Assess Occupational Stress Injuries & Build Resilience | PSHSA]. For workplaces wondering if they should use the tool, Van hulle says they might start by looking at data from their employee assistance programs and from their short-term and long-term disability plans. “What are those numbers telling you?” says Van hulle. “Do you know where your baseline is? Do you think you're doing enough? Do you want to demonstrate that you are making a difference year over year?” That's where the tool fits in, she says.
“It’s a validated tool, tested against other leading psychosocial risk indicators. So with this tool, employers aren’t just guessing at their risk level or relying on lagging indicators like disability claims after the fact. It'll tell you where you are now, give you the opportunity to put preventative measures in place, and then let you reassess on a regular basis. It turns a vague sense that something should be done into an actual baseline, a plan, and a way to track whether it’s working,” says Van hulle.