Tuesday, February 07, 2012
Regression to the mean
After a long time, i came across a mathematical concept called "Regression to the mean". Basically the phenomenon is this : An extreme outcome will be wrongly attributed to a set of variables rather than the fact that it is serendipitous. Given this attribution, the prediction on the next outcome will have a systemic bias.
Lets take an example : What is your prediction of the score on day 2 of a golf tournament given that the score on day 1 is known. Suppose you are predicting the scores of the highest and the lowest player.
If we are solving this equation with the following variable :
Performance = Talent + Luck
The only thing that we should expect in this scenario is :
- The highest scoring player should likely to do well on day 2 as well, but his score is likely to be lower than day 1. Perhaps the score on day 1 was driven by an exceptional run of good luck which is unlikely to hold on day 2.
- The lowest scoring player should is likely to do below-average on day 2, but his score is likely to be higher that on day 1. Perhaps the score on day 1 was driven by an exceptional bout of bad luck which will turn on day 2.
This phenomenon is called regression to the mean as in most instances the above happens (player 1 gets lower score and player 2 gets a higher score - thus moving closer to the mean).
This regression finds echos in the attribution-error concept of psychology. The attribution error is succinctly described in wikipedia as "the tendency to over-value dispositional or personality-based explanations for the observed behaviors of others while under-valuing situational explanations for those behaviors."
Basically we 'substitute' the question that was being asked and answer the other question. In predicting an outcome, we substitute the question with 'what we know about the person' rather than a more objective appraisal.
