Umpires and QuesTec

Several readers have been asking about the recent study that was reported to show umpire bias by race known as the Hamermesh study. Phil Birnbaum and Mitchel Lichtman have been doing great work in that regard already so I have little to add other than providing a few links for those interested:

The original study

The Time Magazine piece

Phil's first take - he questions the author's findings of statistical significance by examining the core table (table 2) from the original study

Phil's follow-up - where he uses to conclude that perhaps and at most 1 in 700 pitches is biased

And even more by Phil - here he uses several different tests of significance and it appears there is no racial bias

MGL's own study - here he uses a much simpler approach and comes to the tentative conclusion that there are not racial differences that are statistically significant. Update on 8/19: MGL posted some updates to his study here and here and comes to the opposite conclusion. He also notes there is a good discussion of the study at The Sports Economist.

One of the side topics that have arisen here is the affect of QuesTec on called strikes. The authors of the Hamermesh study found that for both white and minority pitchers, in non-QuesTec parks pitchers received a higher percentage of strikes when the race of the pitcher and umpire matched than they did in QuesTec parks. White pitchers did not experience this difference when the umpire was non-white although minority pitchers still did.

This provides an opportunity to look at the PITCHf/x data from this season in QuesTec and non-QuesTec parks to get a more granular feel for what the overall difference might be. While we have data for only 9 of the 11 parks where QuesTec is installed, we still end up with almost 35,000 pitches in QuesTec parks and 63,000 in non-QuesTec parks to analyze. When we do so by comparing the location of the pitch to the strike zone (defined by the PITCHf/x operator for each plate appearance) and give the umpires a 1 inch buffer zone to correspond with the limits of the system, we find the following:

Park      Pitches   CS%   CB%  Agree%
QuesTec     34427 .8252 .9433   .8790
Non-QuesTec 62862 .8052 .9488   .8772

By way of explanation CS% is the called strike percentage defined as the percentage of actual pitches in the strike zone that were actually called strikes. CB% is the called ball percentage defined as the percentage of pitches that were actually out of the strike zone that were called balls and Agree% is the overall percentage of pitches on which PITCHf/x (given the buffer zone) and the umpire agreed.

By simply examining the confidence intervals it appears that umpires do indeed call more pitches in the zone strikes at QuesTec parks than at non-QuesTec parks. The difference is statistically significant at .05 at amounts to 1 pitch in 50. However, at QuesTec parks umpires don't do as well at identifying balls and end up calling more of them strikes to the tune of 1 in 180 pitches. This result too is statistically significant at .05 indicating that perhaps the biggest effect of QuesTec is simpy to call more strikes.

Because the factors are working in opposite directions when we add them up the Agree% fails to meet the .05 test. Overall then, if we attribute the entire difference to whether the umpire is in a QuesTec park or not we're talking about a difference of 1 pitch in 550. Of course there may be other factors at work here including the calibration of the system at particular parks that may play a role which I haven't examined.