The psychosocial work environment, in broad terms, refers to the organizational and social conditions in which employees perform their work. Because these conditions can affect the mental and physical health of employees, employers are increasingly aware of the need to identify and manage these psychosocial conditions in their workplaces.
To do that, employers need tools—namely, surveys. Looking through these surveys, one typically finds questions on a wide range of dimensions that make up the psychosocial work environment: workload, psychological demands, job security and more.
When surveys measure such diverse dimensions, it’s important that the survey score for any one dimension is truly a measure of that dimension and not of others, says Institute for Work & Health (IWH) President and Senior Scientist Dr. Peter Smith. If this isn’t the case, workplaces using the survey will not be able to rely on its results to pinpoint the dimensions of the psychosocial work environment that require attention—nor will they be able to determine whether efforts to improve a specific dimension are leading to measurable change.
In a recent study, Smith and John Oudyk, occupational hygienist at Occupational Health Clinics for Ontario Workers (OHCOW), examined one of the most well-known psychosocial work surveys in Canada—Guarding Minds @ Work (GM@W).
The team found it unable to discriminate among different dimensions. The questions making up the scales that measure the 13 different dimensions don’t map onto those dimensions as we expected,
says Oudyk.
The tool remains one of the most recognized measures of its kind in the Canadian employment landscape,
adds Smith. And while it may give an employer a general indication of whether their psychosocial work environment is healthy or unhealthy, if they wanted to use the tool to drill down on specific dimensions of the work environment, our findings show they will likely find it limiting.
The study was published in October 2021 in the journal Quality & Quantity (doi:10.1007/s11135-021-01269.6).
A tool to measure 13 dimensions
The GM@W survey, developed by a research team at Simon Fraser University, was launched in 2009. It is financially supported by Canada Life (formerly Great-West Life) through an organization called Workplace Strategies for Mental Health. The survey was updated in 2012, 2016 and 2020. The 2016 version includes 65 items designed to measure 13 dimensions of the work environment that have the potential to affect worker mental health.
These dimensions are: psychological support; organizational culture; clear leadership and expectations; civility and respect; psychological competencies and requirements; growth and development; recognition and reward; involvement and influence; workload management; engagement; balance; psychological protection; and protection of physical safety. The survey asks five questions within each of these 13 dimensions. (The 2020 update of the survey expands the number of questions to 73 items.)
This 13-factor framework was incorporated into the National Standard of Canada for Psychological Health and Safety in the Workplace, published in 2013 by the Mental Health Commission of Canada and the Canadian Standards Association (CSA). The standards and related implementation guide recommended using the GM@W survey, even though little information was available on its psychometric properties. (The psychometric properties of a tool indicate its reliability, validity and responsiveness, including its ability to measure what it is designed to measure.)
The joint IWH and OHCOW study set out to help fill this gap. The study was one of the first to look at the psychometric properties of the GM@W survey, says Smith. It was conducted as part of research examining the psychometric properties of the GM@W survey and the Copenhagen Psychosocial Questionnaire (COPSOQ). The latter is used in StressAssess, another tool to measure the psychosocial work environment. It was developed by OHCOW and other stakeholders working under the name Mental Injury Tool Group, with Smith providing analytical support.
When we go into a workplace and they’re asking about our questionnaire, we usually get asked two questions. The first one is, ‘Does your survey measure the CSA 13 factors?’ and the second one is, ‘Is our survey a validated survey?’
says Oudyk in explaining the motivation behind the study.
How the study was done
To conduct their study, Smith and Oudyk commissioned EKOS Research Associates to contact randomly selected Ontario workers from an existing panel of willing survey participants. Conducted in February and March 2020, the survey resulted in nearly 1,000 workers from various industries completing the GM@W questionnaire online. Smith and Oudyk then analyzed the survey results, using standard research methods for assessing a tool’s validity and reliability. For example, they examined correlations across the survey’s items—both within the same dimension (i.e. correlation among the five items within each dimension) and across the 13 different dimensions.
While it is expected that different dimensions of the psychosocial work environment will be related to each other, in theory, the five items that pertain to a given dimension should be much more highly correlated with each other than they are with items that pertain to other dimensions. However, for most of the survey’s 65 items, Smith and Oudyk found the correlation among the items within the same dimension was similar to their correlation to items outside their dimension.
This limitation is important, Smith notes. A workplace could target a number of dimensions to improve its psychosocial environment and, by extension, worker mental health. But, due to resource constraints, an employer might choose to focus on particular dimensions at different times.
For example, if a workplace sets out to improve civility and respect and uses the survey to measure progress on that front a year later, it’s a challenge if the scores for civility and respect are all mixed up with the ones about workload management or recognition and reward. An employer wouldn’t be able to get a clear signal of what’s happening to civility and respect.
Why psychometric studies matter
Workers and workplaces are increasingly identifying the importance of the psychosocial work environment and its relationship to both job satisfaction and health. The GM@W website and the National Standard of Canada for Psychological Health and Safety have been pivotal in raising the profile of the psychosocial work environment and in highlighting the importance of measuring and addressing the psychosocial work environment as part of a healthy workplace,
says Smith.
That said, we believe any measure should be independently assessed, on multiple occasions, to ensure its psychometric properties are valid and reliable, including its ability to distinguish between multiple dimensions within the psychosocial work environment,
he continues. When using a measure that specifies, defines and provides individual items to assess 13 dimensions of the psychosocial work environment, a workplace will reasonably assume that these dimensions, while related, can be distinguished from each other. Based on our analyses, we don’t believe this is currently the case when using the GM@W instrument. And it’s important that workplaces and workers understand this.
Since this study was conducted, some of the 65 original questions have been changed or dropped, says Oudyk. However, based on our team’s analysis of the 45 items that were included in both the 2016 and 2020 versions—analysis that were not included in the paper—the same issues around the survey’s inability to discriminate among dimensions remain,
he says.
IWH shared its findings with the GM@W research team. In response, the team sent a joint statement authored by Drs. Merv Gilbert and Dan Bilsker, directors of Vancouver Psychological Health + Safety Consulting Inc., and Dr. Martin Shain, principal of the Neighbour at Work Centre. The statement said the IWH/OHCOW study reflects a misunderstanding of the nature and intent
of the 13 dimensions described in the survey and standard. When they were developed, there was no expectation that the factors would be statistically distinct,
the GM@W team wrote. Rather, they were viewed as categories of psychosocial risk mitigation that impact the psychological well-being and safety of workers.
The full response from the GM@W team is available online (see: https://d3mh72llnfrpe6.cloudfront.net/wp-content/uploads/2022/05/12224233/development-and-utility-of-guarding-minds-002.pdf. The team noted, as well, that its analysis of the survey’s psychometric properties is available upon request.