Sample Size and Statistical Significance
- By: Chris Liss
- On: 6/2/2009 12:35:00 PM
- View Comments : 8
Related: Chris Liss
When you're looking at a set of data - a player's at-bats, a pitcher's innings, whatever - and you see something in it that appears to be an outlier, you're faced with the question of whether there's a cause to that anomaly or whether it's just random fluctuation. In other words, is Justin Verlander the same guy he was last year, but for one month, he faced a combination of free swingers, favorable shadows over the batter's box, hitters who faced Tim Wakefield the day before, or is there a more central cause - that Verlander's stuff is harder, better located and/or nastier?
And in considering that question, the sample size of the data is important because over a short period of time, random fluctuation is more likely to occur. Put differently, you might flip an evenly weighted coin 10 times and get 8 heads. You will never in a trillion years flip it 10,000 times and get 8,000 heads. Okay, most of us grasp that.
The question is - at what point do we conclude that the coin is *not* evenly weighted? Well, it's not simply a matter of how many times you flip the coin, i.e., sample size. It's *also* a matter of the *magnitude of the deviation*. If you get 16 heads and four tails, I wouldn't be so sure the coin is unevenly weighted. But if you get 20 heads and no tails, it's almost certainly so. (Less than 1 in a million odds of that). In both cases, the sample size is quite small, but in the latter, it's more than sufficient. When the coin lands at a an 80-percent heads clip, you need a larger sample to determine the coin is rigged because the magnitude of the deviation from the baseline (50/50) is less. And if the coin lands heads at a 55-percent clip, you need an even larger number of flips to determine whether it's rigged.
So understand that the sample size is only one of two factors in determining the significance of the outlier. The other is the magnitude.
That's why when you see Verlander strike out 60 batters in 44 IP or Joe Mauer - f*** it, I'll mention him, hit 11 home runs, you cannot simply say, "it's only one month, I'm not a believer" without also considering the magnitude of the deviation.
Is 60K in 44 IP a big enough magnitude to make one month significant. Is 11 home runs? In my opinion, yes. But whatever your opinion, you must address both factors if you're going to get a good gauge of whether it's dumb luck or a new baseline.
reddit
Comments
On: 6/2/2009 5:24:00 PM
On: 6/2/2009 7:26:00 PM
On: 6/3/2009 9:08:00 AM
And to that end, Colin's point is a good one. Sure, there is less than 1% chance that a 1-in-20 Mauer would hit 12 in 102. But considering all of the 250 hitters with 100 ABs this season, there's a 72% chance that at least one of them would do something that rare (ie 0.5%). So that it happens to be Joe Mauer is surprising, but that any random player would perform so far outside his skill set is not that surprising.
If I had to guess at Mauer's HR rate the rest of the year, I would say somewhere around 1 in 20. That would make him a .320-25-115 C over a full season. I think that's worth a first round pick, but more like a late first-rounder.
On: 6/3/2009 3:53:00 PM
On: 1/12/2011 11:28:00 PM
___________________________
wool rugs
Leave a comment
Commenting is restricted to registered users only. Please register or login now to submit a comment.