Touching BasesOctober 01, 2009
Bullpen Management
By Jeremy Greenhouse

It's the manager's job to get the most out of his players. With regards to the bullpen, this means optimally inserting relievers depending on such factors as the current baserunners, batter, and score. Hence, relief contributions tend to be measured by Win Probability Added. Having a bullpen ace makes the managers job easier in that he doesn't even have to think about whom to give his highest-leveraged innings. LOOGYs are always nice too. FanGraphs has a statistic that compares a player's WPA in high-leverage situations vs. his WPA in low-leverage situations, to see how relatively Clutch that player is. Looking at the bullpen as a collective unit, we can more or less make the assumption that a Clutch bullpen has been managed well, which is to say that better relievers are pitching in well-deserved, higher-leveraged innings.

I collected all data from FanGraphs since 1979 on team bullpens. Here it is in the form of a Google Motion Chart. What you will see is this year's team bullpen's Clutch score plotted against their WPA/LI, which is a measure of how well the bullpen performed, treating high-leverage and low-leverage situations as equal.

While this data could be used to rank managers historically, I've chosen to focus only on this year for now. The Yankees, Red Sox, and Twins have been best at deploying their top relievers at opportune times thanks to three of the top closers in the game in Mariano Rivera, Jonathan Papelbon, and Joe Nathan. Meanwhile, the Pirates blow everybody else out of the water in bullpen mismanagement. Let’s see how they’ve gone about this. Sorted by the average leverage index for each reliever, here are the WPA figures for all Pirate relievers with at least 20 innings pitched.

Name pLI WPA
Matt Capps 1.6 -2.68
Sean Burnett 1.23 -0.04
John Grabow 1.23 1.19
Joel Hanrahan 0.92 1.04
Jesse Chavez 0.9 -0.83
Steven Jackson 0.88 -0.2
Jeff Karstens 0.78 -0.75
Evan Meek 0.65 0.14

Providing Matt Capps with the highly leveraged innings was a decent idea to start the season, but he’s been struggling this year. His walk rate has skyrocketed and his .370 BABIP isn’t helping matters. Meanwhile, Joel Hanrahan has been phenomenal with the Pirates so far, boasting a double-digit K/9 mark without having allowed a homer, yet he has been riding the pine when it’s mattered most.

A look at the Yankees, who have had the "Clutchiest" bullpen in baseball this year.

Name pLI WPA
Mariano Rivera 1.71 3.58
Phil Hughes 1.43 2.4
Phil Coke 1.29 0.88
Alfredo Aceves 1.07 1.92
Brian Bruney 0.99 1.02
David Robertson 0.68 -0.3
Brett Tomko 0.66 -0.55
Jonathan Albaladejo 0.65 -0.22
Jose Veras 0.62 -0.2
Edwar Ramirez 0.44 0.15

As a Yankee fan, it kills me that Phil Coke pitches more important innings than David Robertson. Other than that quibble, it’s hard to argue with what Joe Girardi’s done with the bullpen pieces he’s been given. Ramirez, Veras, and Albaladejo are clearly the three worst relievers to have seen time in the Yanks’ pen, and Girardi did a good job hiding them. Mo and Hughes make it easy at the top.

Equal time to the Red Sox, who have a highly-touted bullpen which has performed at a merely average level when factoring out leverage.

Name pLI WPA
Jonathan Papelbon 2.19 4.83
Hideki Okajima 1.31 1.78
Ramon Ramirez 1.23 0.47
Manny Delcarmen 1.11 -0.31
Daniel Bard 0.93 -0.25
Justin Masterson 0.75 0.32
Takashi Saito 0.68 -0.26

I’m sure Sox fans would rather see Bard in higher-leverage situations, but besides that minor note, I would think the Nation would also be satisfied with Francona’s usage of the pen.

Having an elite closer the likes of Mo and Papelbon doesn't make the decisions that go into bullpen management that cut and dry, though. Two teams at the bottom of the rankings who have terrific closers are the Dodgers and Royals. How did they go about possibly mismanaging their bullpens?

Name pLI WPA Name pLI WPA
Jonathan Broxton 1.85 2.87 Joakim Soria 2 3.23
George Sherrill 1.71 1.08 John Bale 1.24 -1.36
Cory Wade 1.66 -0.95 Jamey Wright 1.04 -0.13
Hong-Chih Kuo 1.55 0.6 Juan Cruz 1.02 -0.81
Ramon Troncoso 1.34 2.16 Kyle Farnsworth 0.92 -2.18
Ronald Belisario 1.19 -1.17 Roman Colon 0.82 -0.84
Jeff Weaver 1.14 0.53 Robinson Tejeda 0.69 0.1
James McDonald 0.91 0.14 Ron Mahay 0.6 -1.22
Brent Leach 0.86 -0.16
Guillermo Mota 0.71 -0.97

First, it’s clear the Dodgers had a tremendous bullpen, while the Royals, well, not so much. The main problem for the Dodgers appears to be Joe Torre’s reliance on Cory Wade to start the year. With so many other terrific options, Torre waited too long to pull the plug on Wade, who's been in the minors for the last couple of months.

As for the Royals, at least Hillman was able to get Soria right. I’d love to know what the Royals saw in John Bale to make them think he was one of their top relievers. And how do they go out and sign Kyle Farnsworth to an $9-million deal, have him pitch better than they would have expected—better than he’s pitched in years—but put him in the least meaningful innings that he’s ever pitched? To be fair, his high-leverage stint against the Yankees Tuesday night didn’t work out too well. The Royals are also burying Robinson Tejeda at the bottom of their bullpen chain, which hasn’t worked out too well for them. And free Carlos Rosa!

In addition to using the best relievers in the most critical situations, managers also have to find a way to get the most out of their relievers by playing to their strengths. Which brings me to platoon splits. Failed starters can always get jobs as relievers if they have the ability to shut down same-handed batters.

The Braves have had a superb bullpen this year, and their Clutch score might be penalizing them for being equally awesome in both high- and low-leverage situations. Part of the reason for their success was their closer-by-committee tandem of righty Rafael Soriano and lefty Mike Gonzalez. For the following table, I went to Baseball Reference's splits pages and found how often each reliever the platoon advantage as well has how much better he fared when facing same-handed batters. Baseball Reference calls the split that compares a pitchers production to himself tOPS+, and for pitchers, lower is better, so Peter Moylan's ptnOPS+ of 65 would mean that Moylan allows an opposing OPS 35% worse against right-handed batters. Therefore, Bobby Cox should try to have Moylan face mainly righties.

Name Throws Ptn% ptnOPS+ Name Throws Ptn% ptnOPS+
Peter Moylan R 61% 65 Eric O'Flaherty L 50% 85
Jeff Bennett R 58% 69 Mike Gonzalez L 34% 78
Manny Acosta R 54% 95
Buddy Carlyle R 53% 114
Rafael Soriano R 52% 49
Kris Medlen R 46% 150

I was surprised to see that Soriano and Gonzalez, who do exhibit traditional platoon splits, have not been given the advantage of facing same-handed batters that often. Instead, it appears that O’Flaherty and Moylan have been used as the righty and lefty specialists while Bobby Cox has opted to allot Soriano and Gonzalez the eighth and ninth innings.

Running the numbers for the Nationals, nothing of note really came up.

Name Throws PTN% ptnOPS+ Name Throws PTN% ptnOPS+
Jay Bergmann R 64% 76 Ron Villone L 50% 94
Jorge Sosa R 60% 65 Sean Burnett L 50% 101
Julian Tavarez R 60% 64 Joe Beimel L 40% 98
Saul Rivera R 58% 92
Logan Kensing R 55% 110
Kip Wells R 55% 72
Joel Hanrahan R 50% 97
Mike MacDougal R 49% 83
Tyler Clippard R 47% 173

The fact that Mike MacDougal, he of the 32/38 K/BB ratio, is closing this year in Washington should say all you need to know about the state of the Nats' bullpen. But hey, they won the Harper lottery.

This type of analysis is essentially made for Tony La Russa, so I’ll put both parts together to try to grade his management.

Name pLI WPA
Ryan Franklin 1.94 2.23
Kyle McClellan 1.45 0.89
Jason Motte 1.04 0.17
Dennys Reyes 1.04 0.26
Trever Miller 0.89 1.2
Chris Perez 0.84 0.38
Blake Hawksworth 0.79 1.24
Brad Thompson 0.5 -0.32

Franklin has emerged as a reliable bullpen ace, and La Russa thankful for that fact. Coming into the year, the likes of Jesse Todd, Jason Motte, and Chris Perez were names you heard vying for that closer job. After Franklin, though, La Russa has had struggles. He’s given high-leverage appearances to Motte, who has not been one of his better relievers. Hawksworth also may be a guy who's emerging that La Russa can start to trust more.

Name Throws PTN% ptnOPS+ Name Throws PTN% ptnOPS+
Chris Perez R 70% 88 Trever Miller L 62% 30
Jason Motte R 61% 71 Dennys Reyes L 60% 60
Kyle McClellan R 57% 121
Ryan Franklin R 54% 109
Blake Hawksworth R 53% 65
Brad Thompson R 52% 99

La Russa does a fantastic job of platooning. Both lefties he’s utilized out of the pen have had the benefit of facing a majority of same-handed batters. Trever Miller has put up great numbers this year, and La Russa would be well-served to use him as the southpaw in a righty-lefty combination with Kyle McClellan who has been holding his own as La Russa's go-to guy after Franklin. There is a dilemma in the case of Miller, who is truly exceptional against lefties to the tune of 37 strikeouts to six walks this year. So in a relatively close game, should La Russa bring him in once the starter is out and a lefty is up to ensure quality innings from Miller, or should La Russa at times wait and hope that Miller might have the chance to face a couple lefties in the 8th or 9th when the leverage is highest, but risk not pitching Miller at all?

There’s a lot more than what I've addressed that goes into bullpen management, but I think bullpen management is one of the more important in-game aspects of managing a baseball team, and it might be a subject that's possible to objectively grade.


Nice job, Jeremy. I always wonder about the approach of matching LI to effectiveness. Specifically, I wonder if it says more about reliever consistency than bullpen management.

For instance, at what point should the Pirates have given up on Capps and moved to other relievers? Relievers don't pitch many innings, and judging them on even half a season's performance seems overly harsh. As you mentioned, his BABIP was way up this year -- should he have been "demoted" because of that?

Bottom line, it's a lot easier to manage your bullpen when you have a Rivera out there.

Good information. It's harsh to accuse Trey Hillman of bullpen mismanaged when everybody he put in (except for Soria) just sucked sideways. And they were especially bad during the six or so weeks that Soria was unavailable. Now, if you want to accuse Dayton Moore of bullpen mismanagement, that I can buy.

And Farnsworth? Well, maybe it's true that it's the best he's pitched in years, maybe it's not, but he blew every high leverage situation in which he was used earlier in the year, thus living up to his reputation and earning Hillman's distrust.

Thanks Studes. I hope the motion chart was to your liking as well. I agree that having an elite closer makes managing the bullpen easier, but it still means the manager is recognizing his best reliever. I can't answer when the Pirates should have not used Capps, but I'm confident they should have done something differently.

glp, I continue to be impressed by Royals fans coming to the defense of their team.

Interesting analysis. But if your goal is to evaluate managers' decision-making, you ideally shouldn't base that on outcomes. Even a manager who perfectly deployed his relievers will often see a mismatch between LI and WPA, just due to bad luck. And WPA is particularly influenced by luck, even more than, say, ERA. Using the best possible estimate of the pitchers' talent AT THE TIME of the decision -- like a Marcel or PECOTA forecast -- would be a better way to assess managers. Or, if you're committed to using current year performance, use something like FIP that strips out some of the luck in WPA. (For example, Farnsworth's 3.09 FIP tells us more about how well he's pitched than his -2.18 WPA.)

Guy, I tried to do that with regards to individual pitchers such as Farnsworth. I was hoping that the overall team metrics would have a large enough sample to strip away some luck.

I agree with Guy. Jeremy, I'm not saying that having an "elite" closer makes the difference, it's having a consistent closer, consistent year after year, in small sample sizes and large, that makes the difference.

What if Capps' true talent projection was about the same as Hanrahan's for most of the season? (I'm guessing that was probably true for most of the season, given Hanrahan's first half with the Nationals). I agree things didn't turn out well for the Pirates, but I think that's primarily a function of reliever inconsistency and not bullpen mismanagement.

Another thing to consider is relative quantity of high leverage situations. With the Yankees, for example, it could be true that fewer high leverage innings allow Girardi to deploy his better relievers more efficiently, whereas the Royals and Pirates wind up with too many high leverage innings to cover. While leverage and WPA are useful in judging an individual's performance (but by no means conclusive), they are kind of lacking when it comes to evaluating a manager's bullpen usage.

I also agree that FIP, TRA, and/or WPA/LI would be a better side by side comparison with pLI, because the WPA is directly affected by the pLI. That is to say, given two relievers with identical performances, the higher leveraged reliever will have a higher WPA, so your charts would make him look better even though he wasn't.

The chart is pretty sweet though, I really like the idea of comparing WPA/LI to the Clutch metric.