Productive Outs

This season ESPN and the Elias Sports Bureau have begun tracking "Productive Outs". These are defined as:

"A Productive Out, as defined and developed by ESPN The Magazine and the Elias Sports Bureau: when a fly ball, grounder or bunt advances a runner with nobody out; when a pitcher bunts to advance a runner with one out (maximizing the effectiveness of the pitcher's at-bat), or when a grounder or fly ball scores a run with one out. "

It has been suggested that tracking productive outs is part of a backlash against the Moneyball movement (emphasize on base percentage, don't throw away outs on sacrifice bunts and stolen bases) and a way to track that "small ball" is a winning strategy particularly when it counts - in the post season. This appears to be the case given the tenor of the article by Buster Olney of ESPN. My minor issues with productive outs include:

• The definition is a little strange in that it does include pitchers bunting with one out but not position players. It doesn't seem to make sense to exclude very poor hitting position players (or any position players for that matter) since poor hitting is why a pitcher would sacrifice with one out in the first place.
• I'm not sure why moving a runner into scoring position with 1 out is not included. I've seen many managers employ just such a strategy even though it is self-defeating.
• I assume for the analysis below (and for common sense sake) that a batter hitting into a double play with nobody out would not get 2 productive outs for his efforts whether a runner scored or was moved into scoring position or not.
• Ground outs with runners on as well as sacrifice flies are already counted as RBIs and so the most productive outs included in this new statistic are already being accounted for.

Which Outs Are Productive?
The first big problem with outs as a positive statistic is that they generally decrease the run potential (the average number of runs scored after a particular situation obtains within an inning) and/or the odds of scoring a single run - which makes them a negative statistic. Therefore some of the outs labeled as productive outs are not productive in any sense. For example, assume that a runner is on first with nobody out. The run potential in such a situation is .953 (according to the table provided by Tangotiger including data from 1999-2002). If the batter makes a productive out the run potential moves down to .725.

Fine, you say, but doesn't it increase the odds of scoring that single run? Nope. Before the productive out the odds of scoring at least one run was 43.7% (again using a table from Tangotiger). After the out 40.6%. Not exactly productive. Of course, this sort of analysis doesn't hold for all base-out situations. For example, with a man on second and nobody out the run potential is 1.189 and the probability of scoring 63.2%. If a grounder moves the runner to third the run potential goes down to .983 but the probability of scoring increases slightly to 66.2%. So the obvious question becomes, can we decide which outs are truly productive and only count those? Sure.

In order to calculate which outs are actually productive you simply look at the before and after state for the batter making the productive out. As in our previous example, before the productive out the run potential was .953 and after .725 so the run potential was decreased by .228 by the productive out. If the batter drove in the run via a ground out or sacrifice fly a 1 would be added to the run potential of the final state. The same can be done for the probability of scoring at least one run.

The table below calculates the before and after differences for both run potential and the probability of scoring a single run for the situations covered by the definition of productive outs for non-pitchers (for pitchers I'll grant that in most cases it makes sense for them to bunt but that will have to wait for a later post). Negative numbers are bad, positive numbers are good.

Runners/Outs	Run Potential	Prob of Scoring 1 Run
1/0	-.228	-.031
2/0	-.464	+.030
12/0	-.106	+.054
13/0 (to 2nd)	-.437	-.181
13/0 (scores)	-.179	1.000
23/0	-.069	1.000
3/1	+.134	1.000
13/1	+.102	1.000
23/1	-.080	1.000

The two rows in bold therefore represent situations where a productive out (given an average hitter) is actually detrimental to the team by both decreasing the run potential and the probability of scoring. In two other situations, runner on third and one out and first and third one out, the productive out is truly productive (both values are positive). So to be more accurate Elias and ESPN should drop the first pair of situations from their stat and only include the second pair. But what about the four mixed states?

In the third set of situations that include runner on 2nd and nobody out, 1st and 2nd nobody out, first and third nobody out, and 2nd and 3rd one out, the run potential decreases while the odds of scoring a run either increases or is 100% since the run scored. This leads to the second big problem with productive outs.

Outs are Context Sensitive
The actual utility of "productive outs" is heavily dependant on the context in which they are made. For example, a productive out in the first inning of a 0-0 game is far less valuable than the same productive out when your team is tied in the bottom of the 9th. The difference is even greater when compared with a productive out that occurs in a 10-1 game in the 5th inning. Basically, because an out is never the desired outcome, any outs that are made fall on a continuum of negative outcomes. Of course, one could argue along the same lines with any counting statistic such as hits or doubles. Certainly a double is more valuable with men on base than with nobody on. But the difference here is that a double is purely positive and so always has positive value in terms of run potential and scoring probability. With making outs the situation dictates the utility.

To find out that Alex Sanchez is 14 of 20 in productive out situations is all fine and good and it does tell me that he has been effective in moving runners over. But it tells me nothing as to how valuable those outs actually were relative to other outcomes or more importantly how many times he came through with hits ("productive hits") in such situations (I assume a productive out opportunity is only recorded when the batter makes an out).

To illustrate the context sensitive nature of productive outs one can look at a Win Expectancy chart like that provided by Tangotiger in the productive out situations and perform the same calculation we did before (A Win Expectancy table shows the probability of the home team going on to win a game when faced with a particular situation based on outcomes of actual games. For example, the odds of the home team winning when the game is tied in bottom of the 7th with 1 out and a runner on first is 59%) All situations shown in the table below are for the home team.

Runners/Outs	Bottom 7th Down 1	Bottom 9th Tied
1/0	-.032	-.012
2/0	-.022	+.023
12/0	+.001	+.023
13/0 (to 2nd)	-.066	-.090
13/0 (scores)	+.015	1.000
23/0	+.036	1.000
3/1	+.054	1.000
13/1	+.061	1.000
23/1	+.033	1.000

As you would expect in these situations the productive outs are more productive (especially when tied in the bottom of the 9th of couse), except for with a runner on 1st and nobody out and runners on 1st and 3rd with an out that moves the trailing runner to second without scoring the runner from third. But that is exactly the point - their value fluctuates with the situation. I have no doubt that moving runners along can be important in helping win baseball games at critical junctures. I just don't think it has any meaning when tracked as a general counting statistic.

Is Making Productive Outs a Skill?
Related to the whole point of tracking productive outs is the notion that there is a skill to them. Certainly bunting, which is a part of productive outs, is a skill but do we really think that ground outs that advance or score runners with the infield playing back can be thought of as a skill? Same goes for fly balls hit just deep enough or to the weak throwing outfielder or with the fast runner on third.

As you peruse the list on ESPN it does appear that hitters who strike out less frequently and are good bunters wit good speed are generally higher on the list (Sanchez, Taguchi, Womak, Clayton, Perez, Pierre, Miles). But my guess is that a big part of the reason for this is the fact that these are hitters who can't get the big hit that maximizes runs and so are utilizing their strengths by bunting and slapping the ball in the infield. Looked at in this light productive outs are a negative statistic. This leads us to the final criticism.

Do Productive Outs Correlate with Winning?
In the final analysis productive outs should be measured on whether or not they help the team win. As I mentioned previously I believe productive outs are strategic and not general purpose and so a team that has alot of them will - all other things being equal - perform worse than a team that has few.

Earlier this season Larry Mahnken over at Hardball Times did an analysis of this question that backed this up and found that:

• "The 15 teams above average in POP have scored 4.77 runs per game, have a .334 POP and a .468 winning percentage.
• The 15 teams below average in POP have scored 4.74 runs per game, have a .270 POP and a .532 winning percentage.
• The top five teams in POP have scored 4.33 runs per game, have a .351 POP and a .392 winning percentage.
• The bottom five teams in POP have scored 4.74 runs per game, have a .230 POP and a .534 winning percentage."

More to the point, he found that the correlation of productive out percentage to wins was only .476. On the contrary, a few years back Stats, Inc. did a study of games from 1993-1997 that showed that teams with the higher OPS (on base plus slugging percentage, a quick and accurate sabermetric measure) in games won those games 85.2% of the time.

But you say, what about productive outs in the post season? There the run environment is contrained and so the value of productive outs should be greater. This was the contention in the original article on ESPN as well. In that article Olney discussed how the Marlins won last year's World Series largely because they led in productive outs 9-5 and that in 62.5 percent of the post season series since 1969 the leader in productive outs has gone on to win the series.

From a purely strategic perspective Kevin Mello and Vijay Mehotra did a study published in the February issue of By The Numbers on run potential and scoring probabilities in all World Series games from 1982-2002. Their conclusion is that one-run strategies are generally not more helpful in the World Series than in the regular season. However, using their table I calculated the following run potential and scoring probabilities for productive outs.

Runners/Outs	Run Potential	Prob of Scoring 1 Run
1/0	-.018	+.08
2/0	-.160	+.053
12/0	-.400	+.030
13/0 (to 2nd)	-.050	-.147
13/0 (scores)	-.240	1.000
23/0	+.110	1.000
3/1	+.170	1.000
13/1	+.130	1.000
23/1	-.420	1.000

As you can see, the value of productive outs is higher in the World Series because the run environment is contrained (they found that scoring drops about 13% in the World Series) partially because of the better pitching but also in part because teams tend to play one-run strategies in the post-season. This has the effect of turning the runner on first and nobody out situation into a slightly positive situation for scoring a run and (strangely) the second and third nobody out situation into a positive for maximizing runs. The general rule holds, however, that giving up the first out is a bad play if your aim is to maxmize scoring.

So what of the argument that the 9-5 advantage in productive outs by the Marlins led to their 6 game victory in 2003? If one looks a little deeper at the 2003 World Series you quickly discover that it likely wasn't that the Marlins small-ball strategy paid off in a big way but rather that the Yankees failed to get hits with runners on base.

First, the Marlins scored 17 runs in the series and the Yankees 22. Using Bill James Run Created formula (RC = (H+BB+HB-CS-GIDP)*(TB+.26*(BB-IBB+HB)+.52*(SB+SF+SH))/(AB+BB+HB+SF+SH) you find that the Marlins should have scored 16.9 runs and the Yankees 27.7. The Marlins hit their mark while the Yankees underperformed by almost a run per game. A quick look at retrosheet reveals that the Marlins hit .232 for the series, .214 with runners on base, and .233 with runners in scoring position. The Yankees on the other hand hit .261 in the series, .182 with runners on base, and .140 (7 for 50) with runners in scoring position. Over a span of just six games, three of which were lost 3-2, 4-3, and 6-4, that pretty well accounts for the outcome of the series.

My guess is that the correlation between winning in the post season and the number of productive outs referenced by Olney simply reflects the fact that winning teams get more runners on base, have more productive out opportunities, and therefore usually end up making more productive outs. The 2003 World Series was likely an exception. A slog through the play-by-play files found on retrosheet should definitively answer the question. A project for another day.

Conclusion
So what does it all mean? Productive outs are an interesting idea but one that hasn't been fully thought out by its creators. Rating players by their Productive Out Percentage (POP) is likely meaningless because the values of the outs are so situationally dependant. If it is going to be tracked it should at the least be modified to exclude:

• The runner on first and nobody out situation
• The runners on first and second and nobody out where the out does not score the run

However, I predict that after a brief interest it will fade away much like Number of Bases Touched in the 1870s and Game Winning RBIs from the 1980s because it does not correlate to winning.

Making productive outs is not a general purpose skill but rather a strategy to be employed very narrowly. As a result it does not have the "house advantage" that new Dodger GM Paul DePodesta wrote about in an article I discussed earlier this year:

"I was on a quest to find relevant relationships. Usually it wasn't as simple as 'if X then Y.' I was looking for probabilistic relationships. I christened the new model in the front office: 'be the house.' Every season we play 162 games. Individual players amass over 600 plate appearances. Starting pitchers face 1,000 hitters. We have plenty of sample size. I encouraged everyone to think of the house advantage in everything we did. We may not always be right but we'd be right a lot more often than we'd be wrong. In baseball, if you win about 60% of your games, you're probably in the playoffs."