It's the manager's job to get the most out of his players. With regards to the bullpen, this means optimally inserting relievers depending on such factors as the current baserunners, batter, and score. Hence, relief contributions tend to be measured by Win Probability Added. Having a bullpen ace makes the managers job easier in that he doesn't even have to think about whom to give his highest-leveraged innings. LOOGYs are always nice too. FanGraphs has a statistic that compares a player's WPA in high-leverage situations vs. his WPA in low-leverage situations, to see how relatively Clutch that player is. Looking at the bullpen as a collective unit, we can more or less make the assumption that a Clutch bullpen has been managed well, which is to say that better relievers are pitching in well-deserved, higher-leveraged innings.
I collected all data from FanGraphs since 1979 on team bullpens. Here it is in the form of a Google Motion Chart. What you will see is this year's team bullpen's Clutch score plotted against their WPA/LI, which is a measure of how well the bullpen performed, treating high-leverage and low-leverage situations as equal.
While this data could be used to rank managers historically, I've chosen to focus only on this year for now. The Yankees, Red Sox, and Twins have been best at deploying their top relievers at opportune times thanks to three of the top closers in the game in Mariano Rivera, Jonathan Papelbon, and Joe Nathan. Meanwhile, the Pirates blow everybody else out of the water in bullpen mismanagement. Let’s see how they’ve gone about this. Sorted by the average leverage index for each reliever, here are the WPA figures for all Pirate relievers with at least 20 innings pitched.
Providing Matt Capps with the highly leveraged innings was a decent idea to start the season, but he’s been struggling this year. His walk rate has skyrocketed and his .370 BABIP isn’t helping matters. Meanwhile, Joel Hanrahan has been phenomenal with the Pirates so far, boasting a double-digit K/9 mark without having allowed a homer, yet he has been riding the pine when it’s mattered most.
A look at the Yankees, who have had the "Clutchiest" bullpen in baseball this year.
As a Yankee fan, it kills me that Phil Coke pitches more important innings than David Robertson. Other than that quibble, it’s hard to argue with what Joe Girardi’s done with the bullpen pieces he’s been given. Ramirez, Veras, and Albaladejo are clearly the three worst relievers to have seen time in the Yanks’ pen, and Girardi did a good job hiding them. Mo and Hughes make it easy at the top.
Equal time to the Red Sox, who have a highly-touted bullpen which has performed at a merely average level when factoring out leverage.
I’m sure Sox fans would rather see Bard in higher-leverage situations, but besides that minor note, I would think the Nation would also be satisfied with Francona’s usage of the pen.
Having an elite closer the likes of Mo and Papelbon doesn't make the decisions that go into bullpen management that cut and dry, though. Two teams at the bottom of the rankings who have terrific closers are the Dodgers and Royals. How did they go about possibly mismanaging their bullpens?
First, it’s clear the Dodgers had a tremendous bullpen, while the Royals, well, not so much. The main problem for the Dodgers appears to be Joe Torre’s reliance on Cory Wade to start the year. With so many other terrific options, Torre waited too long to pull the plug on Wade, who's been in the minors for the last couple of months.
As for the Royals, at least Hillman was able to get Soria right. I’d love to know what the Royals saw in John Bale to make them think he was one of their top relievers. And how do they go out and sign Kyle Farnsworth to an $9-million deal, have him pitch better than they would have expected—better than he’s pitched in years—but put him in the least meaningful innings that he’s ever pitched? To be fair, his high-leverage stint against the Yankees Tuesday night didn’t work out too well. The Royals are also burying Robinson Tejeda at the bottom of their bullpen chain, which hasn’t worked out too well for them. And free Carlos Rosa!
In addition to using the best relievers in the most critical situations, managers also have to find a way to get the most out of their relievers by playing to their strengths. Which brings me to platoon splits. Failed starters can always get jobs as relievers if they have the ability to shut down same-handed batters.
The Braves have had a superb bullpen this year, and their Clutch score might be penalizing them for being equally awesome in both high- and low-leverage situations. Part of the reason for their success was their closer-by-committee tandem of righty Rafael Soriano and lefty Mike Gonzalez. For the following table, I went to Baseball Reference's splits pages and found how often each reliever the platoon advantage as well has how much better he fared when facing same-handed batters. Baseball Reference calls the split that compares a pitchers production to himself tOPS+, and for pitchers, lower is better, so Peter Moylan's ptnOPS+ of 65 would mean that Moylan allows an opposing OPS 35% worse against right-handed batters. Therefore, Bobby Cox should try to have Moylan face mainly righties.
I was surprised to see that Soriano and Gonzalez, who do exhibit traditional platoon splits, have not been given the advantage of facing same-handed batters that often. Instead, it appears that O’Flaherty and Moylan have been used as the righty and lefty specialists while Bobby Cox has opted to allot Soriano and Gonzalez the eighth and ninth innings.
Running the numbers for the Nationals, nothing of note really came up.
The fact that Mike MacDougal, he of the 32/38 K/BB ratio, is closing this year in Washington should say all you need to know about the state of the Nats' bullpen. But hey, they won the Harper lottery.
This type of analysis is essentially made for Tony La Russa, so I’ll put both parts together to try to grade his management.
Franklin has emerged as a reliable bullpen ace, and La Russa thankful for that fact. Coming into the year, the likes of Jesse Todd, Jason Motte, and Chris Perez were names you heard vying for that closer job. After Franklin, though, La Russa has had struggles. He’s given high-leverage appearances to Motte, who has not been one of his better relievers. Hawksworth also may be a guy who's emerging that La Russa can start to trust more.
La Russa does a fantastic job of platooning. Both lefties he’s utilized out of the pen have had the benefit of facing a majority of same-handed batters. Trever Miller has put up great numbers this year, and La Russa would be well-served to use him as the southpaw in a righty-lefty combination with Kyle McClellan who has been holding his own as La Russa's go-to guy after Franklin. There is a dilemma in the case of Miller, who is truly exceptional against lefties to the tune of 37 strikeouts to six walks this year. So in a relatively close game, should La Russa bring him in once the starter is out and a lefty is up to ensure quality innings from Miller, or should La Russa at times wait and hope that Miller might have the chance to face a couple lefties in the 8th or 9th when the leverage is highest, but risk not pitching Miller at all?
Nice job, Jeremy. I always wonder about the approach of matching LI to effectiveness. Specifically, I wonder if it says more about reliever consistency than bullpen management.
For instance, at what point should the Pirates have given up on Capps and moved to other relievers? Relievers don't pitch many innings, and judging them on even half a season's performance seems overly harsh. As you mentioned, his BABIP was way up this year -- should he have been "demoted" because of that?
Bottom line, it's a lot easier to manage your bullpen when you have a Rivera out there.
Posted by: studes at October 1, 2009 4:46 AM
Good information. It's harsh to accuse Trey Hillman of bullpen mismanaged when everybody he put in (except for Soria) just sucked sideways. And they were especially bad during the six or so weeks that Soria was unavailable. Now, if you want to accuse Dayton Moore of bullpen mismanagement, that I can buy.
And Farnsworth? Well, maybe it's true that it's the best he's pitched in years, maybe it's not, but he blew every high leverage situation in which he was used earlier in the year, thus living up to his reputation and earning Hillman's distrust.
Posted by: glp at October 1, 2009 8:37 AM
Thanks Studes. I hope the motion chart was to your liking as well. I agree that having an elite closer makes managing the bullpen easier, but it still means the manager is recognizing his best reliever. I can't answer when the Pirates should have not used Capps, but I'm confident they should have done something differently.
glp, I continue to be impressed by Royals fans coming to the defense of their team.
Posted by: Jeremy Greenhouse at October 1, 2009 9:52 AM
Interesting analysis. But if your goal is to evaluate managers' decision-making, you ideally shouldn't base that on outcomes. Even a manager who perfectly deployed his relievers will often see a mismatch between LI and WPA, just due to bad luck. And WPA is particularly influenced by luck, even more than, say, ERA. Using the best possible estimate of the pitchers' talent AT THE TIME of the decision -- like a Marcel or PECOTA forecast -- would be a better way to assess managers. Or, if you're committed to using current year performance, use something like FIP that strips out some of the luck in WPA. (For example, Farnsworth's 3.09 FIP tells us more about how well he's pitched than his -2.18 WPA.)
Posted by: Guy at October 1, 2009 10:29 AM
Guy, I tried to do that with regards to individual pitchers such as Farnsworth. I was hoping that the overall team metrics would have a large enough sample to strip away some luck.
Posted by: Jeremy Greenhouse at October 1, 2009 11:52 AM
I agree with Guy. Jeremy, I'm not saying that having an "elite" closer makes the difference, it's having a consistent closer, consistent year after year, in small sample sizes and large, that makes the difference.
What if Capps' true talent projection was about the same as Hanrahan's for most of the season? (I'm guessing that was probably true for most of the season, given Hanrahan's first half with the Nationals). I agree things didn't turn out well for the Pirates, but I think that's primarily a function of reliever inconsistency and not bullpen mismanagement.
Posted by: studes at October 1, 2009 12:37 PM
Another thing to consider is relative quantity of high leverage situations. With the Yankees, for example, it could be true that fewer high leverage innings allow Girardi to deploy his better relievers more efficiently, whereas the Royals and Pirates wind up with too many high leverage innings to cover. While leverage and WPA are useful in judging an individual's performance (but by no means conclusive), they are kind of lacking when it comes to evaluating a manager's bullpen usage.
Posted by: Will at October 1, 2009 6:18 PM
I also agree that FIP, TRA, and/or WPA/LI would be a better side by side comparison with pLI, because the WPA is directly affected by the pLI. That is to say, given two relievers with identical performances, the higher leveraged reliever will have a higher WPA, so your charts would make him look better even though he wasn't.
Posted by: Benjamin at October 1, 2009 6:29 PM
The chart is pretty sweet though, I really like the idea of comparing WPA/LI to the Clutch metric.
Posted by: Benjamin at October 1, 2009 6:34 PM