On the Merits of Probability

Some have called the Dodgers/Padres game on September 18, 2006 "the game of the century" for its unparalleled excitement and finish in the midst of a pennant race. Down 9-5 entering the bottom of the ninth inning the Dodgers Jeff Kent, J.D. Drew, Russell Martin, and Marlon Anderson hit consecutive homeruns off of first Jon Adkins and then Trevor Hoffman to not the game at 9-all. In the top of the 10th the Padres once again struck for a run on a Josh Bard single to take the lead. But in the bottom of the 10th with Kenny Lofton aboard and one out, comeback player of the year Nomar Garciapara sent the Dodgers faithful home with a 11-10 victory powered by his homerun to left field. Oh and it put the Dodgers in first place to boot.

The following graph shows the Win Expectancy (WX) for the Dodgers during the game and highlights some of the key plays along the way. The table that follows includes each play and how that play either increased or decreased the WX for the Dodgers.

                                                                              Score
Inning Outs Batter               Event Text            Start     End    Diff  LA  SD
  1     0   Dave Roberts         43/G                  0.500   0.522   0.022   0   0
  1     1   Brian Giles          K                     0.522   0.538   0.016   0   0
  1     2   Adrian Gonzalez      S8/L                  0.538   0.527  -0.012   0   0
  1     2   Mike Piazza          D8/L.1-H              0.527   0.419  -0.108   0   1
  1     2   Russell Branyan      W                     0.419   0.410  -0.009   0   1
  1     2   Mike Cameron         T9/L.2-H;1-H          0.410   0.245  -0.165   0   3
  1     2   Geoff Blum           S9/L.3-H              0.245   0.187  -0.058   0   4
  1     2   Josh Barfield        8/F                   0.187   0.198   0.011   0   4
  1     0   Rafael Furcal        S5/BG                 0.198   0.227   0.029   0   4
  1     0   Kenny Lofton         S8/G.1-2              0.227   0.276   0.049   0   4
  1     0   Nomar Garciaparra    64(1)3/GDP.2-3        0.276   0.185  -0.091   0   4
  1     2   Jeff Kent            D8/F.3-H              0.185   0.252   0.066   1   4
  1     2   J.D. Drew            K                     0.252   0.223  -0.028   1   4
  2     0   Jake Peavy           K                     0.223   0.237   0.014   1   4
  2     1   Dave Roberts         K                     0.237   0.248   0.010   1   4
  2     2   Brian Giles          S7/G                  0.248   0.240  -0.007   1   4
  2     2   Adrian Gonzalez      K                     0.240   0.254   0.014   1   4
  2     0   Russell Martin       13/G                  0.254   0.232  -0.023   1   4
  2     1   Marlon Anderson      HR/9/F                0.232   0.317   0.086   2   4
  2     1   Wilson Betemit       43/G                  0.317   0.300  -0.017   2   4
  2     2   Brad Penny           K                     0.300   0.289  -0.011   2   4
  3     0   Mike Piazza          53/G                  0.289   0.307   0.018   2   4
  3     1   Russell Branyan      K                     0.307   0.320   0.013   2   4
  3     2   Mike Cameron         S7/G                  0.320   0.311  -0.010   2   4
  3     2   Geoff Blum           CS2(26)               0.311   0.329   0.018   2   4
  3     0   Rafael Furcal        HR/8/F                0.329   0.438   0.109   3   4
  3     0   Kenny Lofton         K                     0.438   0.410  -0.028   3   4
  3     1   Nomar Garciaparra    9/F                   0.410   0.390  -0.020   3   4
  3     2   Jeff Kent            D8/L                  0.390   0.417   0.027   3   4
  3     2   J.D. Drew            DGR/7/L.2-H           0.417   0.538   0.121   4   4
  3     2   Russell Martin       1/L                   0.538   0.500  -0.038   4   4
  4     0   Geoff Blum           6/P                   0.500   0.528   0.028   4   4
  4     1   Josh Barfield        E6/TH/G               0.528   0.498  -0.030   4   4
  4     1   Jake Peavy           4/P                   0.498   0.533   0.035   4   4
  4     2   Dave Roberts         CS2(26)               0.533   0.561   0.028   4   4
  4     0   Marlon Anderson      S9/G                  0.561   0.601   0.041   4   4
  4     0   Wilson Betemit       K+SB2                 0.601   0.583  -0.019   4   4
  4     1   Brad Penny           6/L                   0.583   0.541  -0.041   4   4
  4     2   Rafael Furcal        43/G                  0.541   0.500  -0.041   4   4
  5     0   Dave Roberts         K/B                   0.500   0.530   0.030   4   4
  5     1   Brian Giles          9/F                   0.530   0.553   0.022   4   4
  5     2   Adrian Gonzalez      S6/G                  0.553   0.537  -0.016   4   4
  5     2   Mike Piazza          W.1-2                 0.537   0.509  -0.027   4   4
  5     2   Russell Branyan      W.2-3;1-2             0.509   0.472  -0.037   4   4
  5     2   Mike Cameron         9/F                   0.472   0.567   0.095   4   4
  5     0   Kenny Lofton         K                     0.567   0.537  -0.030   4   4
  5     1   Nomar Garciaparra    D8/L                  0.537   0.592   0.055   4   4
  5     1   Jeff Kent            63/G                  0.592   0.547  -0.046   4   4
  5     2   J.D. Drew            IW                    0.547   0.558   0.011   4   4
  5     2   Russell Martin       5(2)/FO/G.1-2         0.558   0.500  -0.058   4   4
  6     0   Geoff Blum           D9/L                  0.500   0.409  -0.091   4   4
  6     0   Josh Barfield        K                     0.409   0.472   0.063   4   4
  6     1   Terrmel Sledge       43/G.2-3              0.472   0.517   0.045   4   4
  6     2   Dave Roberts         K                     0.517   0.576   0.059   4   4
  6     0   Marlon Anderson      S9/L                  0.576   0.625   0.049   4   4
  6     0   Wilson Betemit       W.1-2                 0.625   0.699   0.073   4   4
  6     0   Oscar Robles         FC1/SAC/BG.2-3;1-2    0.699   0.787   0.088   4   4
  6     0   Rafael Furcal        42(3)/FO/G.2-3;1-2    0.787   0.706  -0.081   4   4
  6     1   Kenny Lofton         12(3)3/GDP            0.706   0.500  -0.206   4   4
  7     0   Brian Giles          E5/G                  0.500   0.443  -0.057   4   4
  7     0   Adrian Gonzalez      3/SAC/BG.1-2          0.443   0.466   0.023   4   4
  7     1   Mike Piazza          IW                    0.466   0.441  -0.025   4   4
  7     1   Josh Bard            54(1)3/GDP            0.441   0.590   0.148   4   4
  7     0   Nomar Garciaparra    7/L                   0.589   0.551  -0.039   4   4
  7     1   Jeff Kent            S8/L                  0.551   0.591   0.040   4   4
  7     1   J.D. Drew            6(1)/FO/G             0.591   0.541  -0.050   4   4
  7     2   Russell Martin       13/G                  0.541   0.500  -0.041   4   4
  8     0   Mike Cameron         9/L                   0.500   0.548   0.048   4   4
  8     1   Geoff Blum           W                     0.548   0.499  -0.049   4   4
  8     1   Josh Barfield        D9/L.1-H;B-3(TH)      0.499   0.197  -0.301   4   5
  8     1   Todd Walker          S8/L.3-H              0.197   0.137  -0.061   4   6
  8     1   Dave Roberts         K+SB2                 0.137   0.146   0.009   4   6
  8     2   Brian Giles          WP.2-3                0.146   0.143  -0.003   4   6
  8     2   Brian Giles          9/F                   0.143   0.167   0.024   4   6
  8     0   Marlon Anderson      T9/L                  0.167   0.300   0.133   4   6
  8     0   Wilson Betemit       S8/G.3-H              0.300   0.405   0.105   5   6
  8     0   Olmedo Saenz         K                     0.405   0.316  -0.089   5   6
  8     1   Rafael Furcal        7/F                   0.316   0.234  -0.082   5   6
  8     2   Kenny Lofton         D9/L.1-3              0.234   0.334   0.100   5   6
  8     2   Nomar Garciaparra    K                     0.334   0.166  -0.168   5   6
  9     0   Adrian Gonzalez      S7/L                  0.166   0.142  -0.024   5   6
  9     0   Manny Alexander      14/SAC/BG.1-2         0.142   0.150   0.008   5   6
  9     1   Josh Bard            D8/F.2-3              0.150   0.104  -0.046   5   6
  9     1   Mike Cameron         IW                    0.104   0.103  -0.001   5   6
  9     1   Geoff Blum           WP.3-H;2-3;1-2        0.103   0.047  -0.055   5   7
  9     1   Geoff Blum           8/SF.3-H;2-3          0.047   0.036  -0.011   5   8
  9     2   Josh Barfield        S9/L.3-H              0.036   0.018  -0.018   5   9
  9     2   Jack Cust            3/G                   0.018   0.019   0.002   5   9
  9     0   Jeff Kent            HR/8/F                0.019   0.043   0.023   6   9
  9     0   J.D. Drew            HR/9/F                0.043   0.094   0.051   7   9
  9     0   Russell Martin       HR/7/F                0.094   0.206   0.112   8   9
  9     0   Marlon Anderson      HR/9/F                0.206   0.642   0.436   9   9
  9     0   Julio Lugo           8/F                   0.642   0.583  -0.059   9   9
  9     1   Andre Ethier         6/P                   0.583   0.536  -0.048   9   9
  9     2   Rafael Furcal        9/F                   0.536   0.500  -0.036   9   9
  10    0   Dave Roberts         8/L                   0.500   0.560   0.060   9   9
  10    1   Brian Giles          D7/L                  0.560   0.442  -0.118   9   9
  10    1   Adrian Gonzalez      IW                    0.442   0.421  -0.021   9   9
  10    1   Paul McAnulty        8/F                   0.421   0.523   0.102   9   9
  10    2   Josh Bard            S9/L.2-H;1-3          0.523   0.167  -0.356   9  10
  10    2   Mike Cameron         W.1-2                 0.167   0.155  -0.012   9  10
  10    2   Geoff Blum           9/F                   0.155   0.206   0.051   9  10
  10    0   Kenny Lofton         W                     0.206   0.338   0.132   9  10
  10    0   Nomar Garciaparra    HR/7/F.1-H            0.338   1.000   0.662  11  10

In the aftermath of that game fellow Baseball Prospectus author Will Carroll and I engaged in a dialogue on the merits and usefulness of measures such as Win Probability Added (WPA) and Win Expectancy Added (WXA). And so for your reading pleasure think of this as a primer on the subject as Will raises legitimate criticisms and throws me a few bones along the way...


[WCarroll] Ok, this WPA thing is beyond me, Dan. I realize that I'm the guy in the group that can't do the complex math, but this reeks of the type of things that statheads get hated for. On the one hand, it reduces "clutch" to mathematical terms, which seems counter to most orthodox analysis and on the other, it makes timing more important than skill. If we look at the amazing Dodgers game this week, Marlon Anderson comes out as the hero. I realize he went five for five with a pair of homers, but why was his homer - the fourth in sequence - any more important than the first one? Jeff Kent's homer came four runs down and started this thing, but he gets almost no credit. And what about the fact the two of the homeruns came off of a superior pitcher in Trevor Hoffman? How does that work?

[DFox] Oh Will. Statheads love to get hated for stuff like this and so I doubt they’ll much sleep over it. But seriously all of the objections you cite are perfectly legitimate. But first, let’s keep in mind that what Win Probability Added (or Win Expectancy Added, which is slightly different although those differences are not important at the moment) is trying to do. At its core it is a technique that’s been around for more than 30 years that simply attempts to quantify how far a player’s actions in a particular game push his team toward a win or a loss. The assignment of those probabilities, and this is the core of all the objections, are based on a matrix that indicates just how probable it is that a team will win given a series of specific game states taking into account outs, inning, score and so on. By crediting (or debiting) the player for a change in the game state we assign them with a certain amount of WPA or WXA (granted, the simplistic way that most folks do this today is to assign all the credit to the pitcher and batter while leaving out the rest of the defense entirely).

Now, because the probabilities used take into account the inning and score primarily, it will always be the case that an event that ties the game or puts the team head will have a much larger magnitude than the same event that occurs in a different context. That’s just the nature of the technique.

[WCarroll] So you're admitting this is flawed? The idea that the timing of a play has as much to do with the sequence is flawed to me. Now, the likelihood of the back times four homers occurring is so low as to be near zero and not worth calculating, it still seems to me that the sequence of events is ignored here. Anderson's homer is a reduced value without Kent's or Martin's and not accounting for that seems to call the technique into question.

[DFox] In the case of Kent versus Anderson Kent’s homerun came at a time when the Dodgers had just a 1.9% chance of winning, being down by four runs in the bottom of the ninth as they were. Although he hit the homerun to make it 4-1 the Dodgers still had just a 4.3% chance of winning and therefore we credit Kent with a 2.3% change or .023 in WXA terms. Anderson on the other hand hit his second homerun when the Dodgers had a 20.6% chance of winning (the two intervening homers raising the odds by just over 5 and 11 percent respectively) and pushed them to 64.2% thereby assigning him a WXA of .436. Of course, Anderson’s homerun wasn’t more important in the big sense of contributing to the win, but it was the event that pushed the Dodgers over the top in terms of their odds of winning the game.

[WCarroll] I can see where the math is going, but Kent's home run is so necessary to the process that it seems he should get more credit than just making a four run game into a three run game with a swing of the bat. The event driven model doesn't account well for the actual nature of the game. Did J.D. Drew’s home run cause the pitching change and if so, where is that factored in?

[DFox] I would certainly agree with you that Kent’s homerun was absolutely necessary to the process. However, it can’t really be argued that immediately after the homerun the Dodgers had a greatly improved chance of winning. They didn’t. They still were down three runs in the bottom of the ninth with no runners on base and the technique credits him appropriately. But this, I think, gets to the heart of one the objections you raised in your initial question regarding two of the homers coming against a tougher pitcher than the other two. The way in which WPA and WXA are calculated do take into account a good portion of the context in which the play was made (and WXA takes in more by including the run environment in a theoretical framework) – but not all of it. While you could conceivably adjust the probability of winning for each batter/pitcher matchup along with a host of other variables including defensive personnel and positioning, weather, tendencies of the manager, and what the batter had for breakfast, the ability to use the technique would drop sharply. In the end these methods provide a model of the game and like all models are imperfect. It’s a bit of a balancing act between greater precision on the one hand and usability on the other.

As to the question of the pitching change anyone watching the game could see that the Drew homerun on the back of the Kent dinger “caused” the pitching change. But again, that is not factored into the equation since the models most folks use don’t take into account differences in batter/pitcher matchups nor the relative strength of the respective team bullpens. For example, although the Dodgers had a 5.1% chance of winning after Drew’s homerun one could argue that with Hoffman still available in the pen their chances were actually smaller than that.

[WCarroll] Everyone's looking for a quantification of clutch and in this analysis, I'm not convinced that the methodology does anything more than make nice graphs and flawed conclusions.

[DFox] But it does allow us to make pretty graphs and that should count for something shouldn’t it? :)

Although originally the Mills Brothers who pioneered this concept (albeit in a slightly different form) in a 1970 book titled Player Win Averages: A Computer Guide to Winning Baseball Players had intended for it to be used to measure clutch ability, this doesn’t do it for the simple reason that it doesn’t correct for the quality, or leverage, of a player’s opportunities. Anderson and Kent did not have the same level of opportunity to accumulate WXA in this game. I think of it more as measuring the total contribution of a player towards winning and losing given the situations they were placed in and so your caution against flawed conclusions is well taken. Of course the more games you aggregate the more the quality of the opportunities tends to even out leveling the playing field (for most hitters anyway, for relief pitchers and especially setup men and closers it’s a different story).

Part of the confusion I think comes from our terms. I’ll admit to having written in the past that we can use WX to quantify "clutch performances" but that is a different thing than “clutch ability”. The former is an acknowledgement that an individual play like Anderson’s second homerun was a very important play and WX can be used to get a feel for how improbable those "clutch" performances are. The latter, however, is about the inherent ability of a player to perform above his normal level when the game is on the line.

To measure clutch ability what analysts have done as you know is look for differences in performance in situations termed clutch and non clutch and see if those differences persist across seasons or careers. What they’ve found in analysis like that in The Book is that there may indeed be a small clutch ability but that ability is basically drowned out by the normal variability inherent in the game. In other words, it may be there but it doesn’t matter much. Now, if it had been shown that there is a wide variance in ability between players in clutch situations then WPA and WXA would be much more useful in measuring that ability all other things (like opportunities given larger sample sizes) being equal. But since that is not the case WXA is obviously not going to be able to capture it.

[WCarroll] Sure. I'm not going to disagree with the theory and I'm certainly not going to critique the math, but what this amounts to is a grand equation that seems to be Game Winning RBI. Back to back to back to back is improbable, sure, but let's look at the situation. What if the hitter before Anderson gets out? Hits a double? How does that change things from this standpoint. In one case, they win in a different, slightly less unique way (or tie that is) and in another, Anderson is penalized for something he has absolutely no control over.

Forget the red herring of clutch. What we have here is a measure of timing, of coincidence, and have disconnected talent from the discussion. Anderson is not a better player than any of the other three. He's been on a hot streak since coming west, but few would argue that he may have been the weakest player on Little's lineup card. For one night, he hits well -- 5 for 5 is nice -- and hits in an interesting way, but he's still the weakest player in the lineup. He just has a better story to tell his grandchildren some day.

[DFox] I think your intuition about WPA or WXA being a fancy form of GWRBI makes for a good analogy. I’m certainly not saying that using WXA for a single game allows you to make a case that Marlon Anderson or Neifi Perez for that matter are actually better players than J.D. Drew or Brandon Inge. And you’re completely on track that if the hitter before Anderson makes an out, or does anything but hit a homerun, it changes the potential WXA for Anderson’s plate appearance. And yes, Anderson has no control over it and as mentioned previously the performance analysis community in general doesn’t think that Anderson has much control over how he performs (relative to his true talent level) when in the situation dictated by the previous hitters in the inning and in the game.

In the end WXA is a measure of timing and coincidence and when taken in very small doses (one very exciting game for example), it is largely disconnected from talent. That’s why when taken over the course of a season or career, if you rank players by their WXA Drew will still beat Anderson and Inge will still beat Perez (actually, almost everyone whose ever played will beat Perez). That said, I believe it’s still an interesting analytical tool when looking not at projecting or evaluating talent, but when looking at the flow of individual games and crediting players over the long haul. For example, I wouldn’t be worried about using seasonal WXA as a tool for input into the discussion of MVPs, Rookie of the Year, or Cy Young awards. In fact, to me, WXA would provide a good mix of who’s the best player and who was the most valuable (leaving questions of team quality aside since WXA doesn’t account for that).