If you’ve ever taken part in a questionnaire—a political poll, a customer satisfaction survey or a research study—you might not have given much thought to the types of question you were asked, how they were worded or how many there were. But researchers spend a great deal of time thinking about and creating the questions used in a study. In fact, this is an entire field of research called psychometrics.
Psychometrics is the field of study that looks at the design, delivery and interpretation of tests that measure human responses. Typically, these tests measure our knowledge or abilities (e.g. an IQ test), our personality and behaviour (e.g. whether we’re more introverted or extroverted) or our attitudes and beliefs (e.g. how we feel about our level of health or the support we get in our workplace).
In health research, for example, psychometric testing is used to create measures that assess pain, fatigue, distress, anxiety, alertness, mobility, agility—the list goes on. In organizational research, psychometric testing is used to create measures that assess worker, supervisor and organizational experiences and behaviours, such as job satisfaction, perceived job characteristics (e.g. job control, work overload), organizational commitment, job stress, job roles, work-family balance/conflict, leadership styles, person-organization fit, and so on.
Psychometrics uses mathematics and statistics, as well as lots of input from individuals to whom the measure is given, to ensure a measure works the way it’s intended to. It makes certain the questions asked cover a range of possible perspectives and that they get enough detail without becoming too repetitive. It ensures the questions asked give rise to results that are valid, reliable and responsive.
Validity
Psychometrics assesses a tool’s validity by looking for evidence that indicates the tool measures what we think it should. For example, we might think a measure asking people about how important physical activity is to them is only valid if those individuals who say physical activity matters actually exercise more than those people who say physical activity doesn’t matter. We might think it isn’t valid if there are important aspects of physical activity that the questionnaire fails to include. That would be a question about content validity, just one of many different types of validity to consider.
Reliability
A tool is assessed for its reliability by determining if people give consistent answers to questions when asked those same questions under similar circumstances. For example, in developing a measure on the commuting difficulties workers face, you would run statistical analyses to find out if the questions given to the same group of workers on different occasions (but close in time) produce roughly the same results. That’s an example of test-retest reliability. Some measures ask others to rate or evaluate another person’s physical or psychological behaviours or health. A measure would be considered reliable if different observers score the same way. That’s an example of inter-rater reliability.
Responsiveness
And then there’s the question of the tool’s responsiveness. Psychometrics looks at its ability to measure meaningful change. That is, if a person’s situation, skills or beliefs change, is the tool sensitive enough to detect this change, and how much change has to take place before the measure will detect it? For example, if a new workplace wellness program is introduced and the program is effective, can we capture changes using a health measure? What about if the change is small—is this just random error or is it meaningful and “real”?
There’s a great deal to be discussed when creating, applying and evaluating the many different measures used in research. Hopefully, this summary gives you an appreciation of the effort that researchers put into designing a questionnaire.
Source: At Work, Issue 79, Winter 2015: Institute for Work & Health, Toronto