Monday, January 03, 2005

Fun with Surveys (part 2)

First, knowing that Republicans are more likely than Democrats to curtail civil liberties tells us nothing. Nothing. More likely by what percent? 1%? 10%? 80%?. No fair researching: what would your guess be? Put more formally, is the difference between republicans and democrats statistically significant? The article never says or suggests that it is. Click Read more.

Clicking on the link to the survey found in the article is only slightly more enlightening. It does confirm that they are looking at raw survey responses rather than OLS regressions or even difference of means tests, which verifies the thought that we are not talking about statistical significance here. This is a common trick in the academic community.

Here is another example. A famous report by James Coleman found that Catholic school students score significantly better than public school students on standardized tests. Follow up research showed that while the difference was significant (due to an enormous sample size), it was a difference of 1.87 questions correct out of 50 (totally from memory so this could be wrong). When you have a huge sample size you can cloak the truth by hiding behind significance. When you have a small sample size you can cloak the truth behind raw differences that have no significance. That is what this study seems to have done. Next page.


Post a Comment

<< Home

Listed on Blogwise