Umpires and QuesTec

Several readers have been asking about the recent study that was reported to show umpire bias by race known as the Hamermesh study. Phil Birnbaum and Mitchel Lichtman have been doing great work in that regard already so I have little to add other than providing a few links for those interested:

The original study

The Time Magazine piece

Phil's first take - he questions the author's findings of statistical significance by examining the core table (table 2) from the original study

Phil's follow-up - where he uses to conclude that perhaps and at most 1 in 700 pitches is biased

And even more by Phil - here he uses several different tests of significance and it appears there is no racial bias

MGL's own study - here he uses a much simpler approach and comes to the tentative conclusion that there are not racial differences that are statistically significant. Update on 8/19: MGL posted some updates to his study here and here and comes to the opposite conclusion. He also notes there is a good discussion of the study at The Sports Economist.

One of the side topics that have arisen here is the affect of QuesTec on called strikes. The authors of the Hamermesh study found that for both white and minority pitchers, in non-QuesTec parks pitchers received a higher percentage of strikes when the race of the pitcher and umpire matched than they did in QuesTec parks. White pitchers did not experience this difference when the umpire was non-white although minority pitchers still did.

This provides an opportunity to look at the PITCHf/x data from this season in QuesTec and non-QuesTec parks to get a more granular feel for what the overall difference might be. While we have data for only 9 of the 11 parks where QuesTec is installed, we still end up with almost 35,000 pitches in QuesTec parks and 63,000 in non-QuesTec parks to analyze. When we do so by comparing the location of the pitch to the strike zone (defined by the PITCHf/x operator for each plate appearance) and give the umpires a 1 inch buffer zone to correspond with the limits of the system, we find the following:

Park      Pitches   CS%   CB%  Agree%
QuesTec     34427 .8252 .9433   .8790
Non-QuesTec 62862 .8052 .9488   .8772

By way of explanation CS% is the called strike percentage defined as the percentage of actual pitches in the strike zone that were actually called strikes. CB% is the called ball percentage defined as the percentage of pitches that were actually out of the strike zone that were called balls and Agree% is the overall percentage of pitches on which PITCHf/x (given the buffer zone) and the umpire agreed.

By simply examining the confidence intervals it appears that umpires do indeed call more pitches in the zone strikes at QuesTec parks than at non-QuesTec parks. The difference is statistically significant at .05 at amounts to 1 pitch in 50. However, at QuesTec parks umpires don't do as well at identifying balls and end up calling more of them strikes to the tune of 1 in 180 pitches. This result too is statistically significant at .05 indicating that perhaps the biggest effect of QuesTec is simpy to call more strikes.

Because the factors are working in opposite directions when we add them up the Agree% fails to meet the .05 test. Overall then, if we attribute the entire difference to whether the umpire is in a QuesTec park or not we're talking about a difference of 1 pitch in 550. Of course there may be other factors at work here including the calibration of the system at particular parks that may play a role which I haven't examined.

Playing Catch Up

After a week communing with family and nature up in Estes Park Colorado we're back and struggling to catch up. Here are a few new things...

Baserunning Metrics - as promised the leaders and trailers in baserunning are now up on Baseball Prospectus.

Stringing - The Rocky Mountain News wrote a little article on Gameday and a little about being a stringer with a few quotes from yours truly. The printed version had a nice picture of Mike Hageman who is the veteran among our crew at Coors Field.

Enhanced Gameday Links - Nice summary on a Reds blog of some of the analysis done using the PITCHf/x data.

Umpires - Speaking of which, my column last week was on umpire accurracy and named names. It's interesting that in that analysis I identified Doug Eddings as the second most pitcher-friendly umpire (albeit based only on 603 called pitches) behind Jeff Nelson as he gave the pitcher 38 more calls than hitters accounting for 6.3% of called balls and strikes. On Saturday night I took eleven family members to the Rockies/Dodgers game where Eddings was behind the plate and it certainly appeared that Eddings was calling a very low strike zone much to the pitcher's advantage.

The Umpires Strike Back

My column today on Baseball Prospectus takes an initial look at the oft-said belief that hitters with better plate discipline and pitchers with better command end up getting the benefit of the doubt from the man behind the plate. I recall first hearing this idea in the late Ron Luciano's book The Umpire Strikes Back that I read back in 1983 or so. There he says the following regarding pitchers.

During a game an umpire gets into a groove with a pitcher. People like Catfish Hunter [pictured above] and Ron Guidry are always going to be around the plate, so an umpire gets into the habit of calling strikes. Even when they miss the plate, it's usually a situation pitch intended to setup the batter for the next pitch or entice him to swing at a pitch outside the strike zone that he can't hit solidly. The umpire becomes so used to calling strikes that it's difficult to call a ball. Strike one, strike two, foul ball, it's close to the plate, strike three.

Then there are pitchers like Ed Figueroa. He was all over the place. One pitch would be high, the next pitch would be in the dirt, the third pitch would be in the concession stand. He would throw three pitches outside the strike zone, then nip the corner of the plate by a quarter inch and expect the umpire to be ready to call a strike.

Within certain limits we can use the PITCHf/x data to try and get a read on this by measuring the number of called strikes and called balls for pitchers and hitters and how many of each went in favor and opposed to the player. By adding these up and we can then calculate a percentage of pitches for each player. Overall, what we find is that umpires, within the limits of the system, seem to get the calls correct 9 out of 10 times with pitchers getting the small upper-hand. It's also the case that left-handed hitters incur a 10% penalty on called strikes over their right-handed brethren.

You'll have to read the article to see all of the conclusions but suffice it to say that Luciano, if he was speaking for all umpires, may have overstated his case.

Umpire Stats

Also was alerted to this interesting analysis of umpires based on data from BP. This analysis is just for 2006.

In and of itself this doesn't tell us much since you would expect there to be some spread here due to randomness. Nor is it surprising that, for example, some of the same umpires show up in high SO/9 quadrant as well as the high percentage of called strikes quadrant since they are clearly related. It would be interesting to see whether there is any trend that holds over from year to year (for example for Randy Marsh who has a very low % of called balls and a high % of called strikes) and then to quantify the effect if any. The author notes that he is looking into this so stay tuned.

And while the two measures that are shown are clearly related it could be that some umps are more hesitant to ring batters up or make a ball four call and so that could explain why an umpire appears to be a "hitter's ump" in one graph and a "pitcher's ump" in another (again Randy Marsh is an example). Interesting stuff.