Suppose you’re the superintendent of a school district and you want to improve the math scores of the Grade 3 students in your catchment who write compulsory province-wide exams. You hire a consulting math expert to help. The consultant starts by administering a math test to find out which students are most in need.
All 1,000 Grade 3 students in your district take the test, and the consultant chooses the 50 students with the lowest scores to receive a remedial math program. Once the program is complete, the 50 students take a second test, and their scores, on average, show a healthy improvement. On this basis, you roll out the remedial program to all Grade 3 math students in the district who are performing below par.
When the board-wide exam takes place later that year, you’re disappointed. The students’ scores are not much better than they were the previous year—and they certainly didn’t improve to the degree you expected based upon the results of the 50 poorest performing students.
What went wrong? You might want to consider the possibility of a statistical phenomenon called regression to the mean.
Regression to the mean refers to the tendency of results that are extreme by chance on first measurement—i.e. extremely higher or lower than average—to move closer to the average when measured a second time. Results subject to regression to the mean are those that can be influenced by an element of chance. When chance or fluke gives rise to extreme scores, it’s unlikely those extreme scores will be repeated on a second try.
In our school district, for example, the kids who scored the poorest on the first math test likely included some who normally know the answers but, by chance, did not that day. Perhaps they were tired, sick, distracted, etc. These kids were going to do better on the second test whether they received the remedial program or not, bringing up the average score among the 50 poorest performers.
You can see why researchers have to consider regression to the mean when they are studying the effectiveness of a program or treatment. If they don’t, they may wrongly conclude that their intervention is responsible for an improvement when, in fact, regression to the mean is at play. This is especially the case when program effectiveness is based on measurements of people or organizations at the extremes—the unhealthiest, the safest, the oldest, the smartest, the poorest performing, the least educated, the largest, etc. The ones on the low extremes are all likely to do better the second time around, and those on the top are likely to do worse—even without the intervention.
Steps to account for regression to the mean
Researchers can take a number of steps to account for regression to the mean and avoid making incorrect conclusions. The best way is to remove the effect of regression to the mean during the design stage by conducting a randomized controlled trial (RCT). Because an RCT randomly assigns study participants to a study group (which receives the program or treatment) or a control group (which does not), the change in the control group provides an estimate of the change caused by regression to the mean (as well as any placebo effect). Any extra improvement or decline in the study group compared to the control group (as long as it is statistically significant) can be attributed to the effect of the program or treatment.
Researchers can also take multiple baseline measurements when selecting people or organizations to be part of a study group. They can then select participants based on the average of their multiple measurements, not just on a single test.
Scientists can also identify and account for regression to the mean when analyzing their results. This involves complicated statistical calculations too difficult to describe here.
Regression toward the mean is a statistical occurrence that can get in the way and distort researchers’ measurements. That’s why it has to be taken into account, in the design of the study or in the analysis of findings.
Source: At Work, Issue 78, Fall 2014: Institute for Work & Health, Toronto