Scouting vs. Sabermetrics

Here's an excellent roundtable discussion hosted by The Numbers Game author Alan Schwarz on Baseball America's site. It's styled as a debate on the merits of the "Moneyball" philosophy versus traditional scouting. On the sabermetric side are Gary Huckaby of Baseball Prospectus and Voros McCraken both of whom do analysis for major league clubs. On the scouting side are Eddie Bane, the Angels scouting director, and Gary Hughes the Cubs assistant General Manager.

As a Cubs fan and a proponent of sabermetric analysis I became more and more frustrated by the responses of Gary Hughes as I continued to read. Here's just three short snippets to wet your appetite (or turn your stomach depending on your viewpoint)...

On Evaluating Hitting Prospects

"ALAN SCHWARZ: OK, so it's the trading deadline, and you want to evaluate another team's Double-A right-field prospect. Everyone agrees that he has considerable skills, and you're going to scout him for three games. How will you evaluate what kind of asset he might be for your big league club a few years from now?

GARY HUGHES: You'll have a history coming in, but you'll evaluate his five tools. You'll compare what you have on your own club. You'll think about what your immediate needs are and what your long-term needs are. And you'll make your decision based on your feeling.

EDDIE BANE: The first thing I do when I get to the ballpark is, I don't care about his right-field play. I don't care about his running speed. I want to see him hit. If he don't hit, I don't have to stay three days. I'm going to pick up the stat sheet--I'm going to look at the strikeouts and walks. I'm going to look at the batting average. I'm going to know all that stuff because I've been on the computer. But if I don't think this guy can hit for the Anaheim Angels, the other stuff is secondary.

ALAN SCHWARZ: But what would you have to see to be encouraged?

GARY HUGHES: The swing, the approach at the plate, the show of fear.

EDDIE BANE: If you show fear, you're gone.

VOROS McCRACKEN: How would someone show fear?

GARY HUGHES: There would be a little give at the plate.

EDDIE BANE: You give on a pitcher with a decent slider . . .

VOROS McCRACKEN: That happens to everyone--everyone gets their knees buckled every once in a while. So if you rule a guy out that gets his knees buckled, that seems extreme. You'd need to see him show fear a bit more consistently. I'm not sure . . .

EDDIE BANE: I am sure. Because if I see fear in a hitter, I'm not ever coming back. I don't see fear in good big league hitters. I know that they get fooled and they'll bail on balls. But for me, that's a different term than fear."

For me, this exchange was disheartening because it gets to the heart of the debate over the meaning of baseball statistics that I touched on in my review of The Thinking Fans Guide to Baseball . Here you have Gary Hughes and Eddie Bane saying that over a three game stretch they would rule out a guy based on his reaction to a nasty slider! While they give a slight nod to looking at the stat sheet they are clearly more trusting of what their eyes tell them. The underlying reason for that, which they emphasize in another part of the debate (and which was echoed by Leonard Koppett in his book), is that they view statistics as encapsulating the past but their observation as encapsulating the future.

Here's the problem with that approach in a nutshell. Three games (12 plate appearances) is an incredibly small sample size in which to tell anything definitive about a hitter. I would argue that a player's past performance is almost always a better indicator of how he'll perform in the future. This coupled with the biases of human perception such as the tendency to magnify singular events and forget a slew of "non-events" makes the observational approach even less reliable. This is not to mention the frustration with the fact that a team might be about to make a $300,000 decision on a guy and yet doesn't have the discipline to avail themselves of all the tools (and most of them are free) out there.

On DIPS

ALAN SCHWARZ: One thing that Eddie and Gary, you might not be aware of, is that a few years ago Voros came up with something called Defense Independent Pitching Stats, which . . .

EDDIE BANE: Alan, you said, "You guys may not be aware." That's one of the things we're battling. We are aware. I read these guys' stuff all the time.

ALAN SCHWARZ: I said, "May not be aware." Gary, have you ever heard of DIPS?

GARY HUGHES: No.

ALAN SCHWARZ: OK then! (Laughter)

Ok, so here you have an assistant GM of a major league ballclub who has no idea what DIPS is. Even though I know that many "baseball men" are not comfortable with numbers I can't fathom that a front office wouldn't at least be conversant about something like DIPS, which has been around for 5 years, and which had a major impact on the baseball research community. This is their industry and could potentially provide them competitive advantage and yet they've never heard of it? If you're not a Cubs fan feel lucky.

In another part of the conversation Schwarz floats the idea that perhaps the GM role be a platoon of sorts with sabermetric and scouting functions that coalesce to make decisions. This certainly appears to be the trend as more and more teams hire statistical consultants. However, since they are consultants they likely have less input than full-time employees. While I've argued in the past that an analyst's role is best served by a degree of separation from the day to day workings of the team, this is one area where that model might be less than optimum.


On Minor/Major League Equivalency

ALAN SCHWARZ: That gets us to this question--do you guys think Triple-A stats can predict player performance in the majors?

GARY HUGHES: I don't know. I can't answer that. That's not my thing.

VOROS McCRACKEN: I think to the extent that that's your answer, that you don't really know . . .

GARY HUGHES: I don't think you know.

VOROS McCRACKEN: I don't know. But I do have an idea. I have looked at stats for tons of Triple-A players, and what they've done in the major leagues, and I think with this sort of information, I don't think that "I don't know" should be the final answer. I think, "I don't know, and I would like to find out" would be the better approach. I'm not sure that's always been the approach. I would say that you know almost as much about what a guy's going to do in the big leagues from his Triple-A stats as you do from his major league stats.

Once again here is an assistant GM admitting that he has no idea how to evaluate minor league statistics! One wonders what meetings with Jim Hendry and Gary Hughes are like when they're deciding who to invite to add to the 40-man roster or invite to spring training or take in the Rule 5 draft.

And unlike DIPS, the recognition of a relationship between minor and major league statistics has a history going back 20 years to the work of Bill James as codified in one of principles in Sabermetrics 101:

"Performance at the major league level can be predicted by performance at the minor league level and to a lesser degree in other leagues including college, the Japanese, and Mexican leagues"

There's lots more in the article which you'll have to read to believe.