tag:blogger.com,1999:blog-5888411.post113163933003907538..comments2024-03-23T05:20:41.018-06:00Comments on Dan Agonistes: Searching for SignificanceDan Agonisteshttp://www.blogger.com/profile/07863051818485888739noreply@blogger.comBlogger1125tag:blogger.com,1999:blog-5888411.post-1131688133256114072005-11-10T22:48:00.000-07:002005-11-10T22:48:00.000-07:00I liked the article, but I just want to mention th...I liked the article, but I just want to mention that you aren't allowed (statistically) to do this many 95% significance tests at a time. Basically, you have a random chance of making the wrong conclusion 5% of the time, and when you do thousands of tests, you expect to find extreme examples among them. For instance, with the ANderson/Anderson matchup, you expect to find results that extreme or more (about) 1 in 10000 times. Given that you did 30,000 tests, you might expect one like this anyway. <BR/><BR/>It would be possible to determine how many matchups you tested could even give a result this low, like you demonstrated in the 5 AB example. For instance, if only 1500 matchups could achieve a p-value this low, the extremeness of the result is more noteworthy than if 15,000 could. This context for the tests is important when drawing conclusions.<BR/><BR/>Again though, I liked the article. I agree that it would be better to do with a different measure than BA. I remember Carlos Delgado hitting, I think, 5 straight HRs of Sosa(on the Braves this year). That sort of dominance is missed by using BA.Anonymousnoreply@blogger.com