Statistics Lesson (for Bill Gates)

Marginal Revolution had a very good article about "The Small Schools Myth" recently:
http://www.marginalrevolution.com/marginalrevolution/2010/09/the-small-schools-myth.html

The question is: Do small schools really do a better job than large schools? The answer is no, not if you understand the statistical principles behind the data. Unfortunately, Bill Gates may have wasted a lot of money because he didn't understand these principles.

The above article linked to a chapter from Howard Wainer's book, "Picturing the Uncertain World."
http://press.princeton.edu/chapters/s8863.pdf

What does it mean? The short version is this: With smaller samples you will tend to see higher highs and lower lows than you would with larger samples. People tend to focus on the "higher highs" that show up in smaller samples (student results from small schools) while ignoring the "lower lows" that also result from these smaller samples.

From the Marginal Revolution article: "Thus, for purely random reasons we would expect small schools to be among the best performing schools in any given year. Of course we would also expect small schools to be among the worst performing schools in any given year! And in fact, once we look at all the data this is exactly what we see."

Wainer's book goes on to illustrate this with other examples such as kidney cancer rates, crime rates, and sex differences.

You may not care about statistics. But this is a good concept to grasp. It will help us avoid overreacting to all the data that we must digest and interpret constantly, on the job and in our personal life.