This is the third part of what has been a three part series on the MLB Draft. Part one created a model for the expected value of each draft pick, while part two calculated probabilities of becoming a certain caliber player, as well as expanded on the conclusions in part one.

Today's article focuses on individual teams and how much control they have over the draft process. Is drafting more or less a complete crapshoot, or does the success of a draft vary greatly depending on the front office and the team who is doing the drafting (and oftentimes developing the players as well). Is there much to distinguish a great drafting franchise from a poor one, or is the difference mostly due to luck?

To review, the data I had at hand was gleaned from Sean Smith's WAR database and Baseball Reference, and contained overall picks #1-50, as well as a handful of picks after that (every 5th pick through #100, every 10th pick through #500, and every 25th pick through #1000). While not every pick for every team is covered, this data gives each team a sample of well over 100 draft picks, including all of a team's very high selections. Data used in this study, will focus only on each player's "first six year" WAR, since the team only gains from drafting a valuable player during the years in which it does not have to pay market value. The data is also limited to those players drafted in 2001 or earlier, since more recently drafted players have not had a chance to come up and show their full value.

Draft WAR By Team

So, how did the teams fare? For what it's worth, the table below shows team's drafts based on the sample of picks which I have (which includes all top picks and a smattering of picks after that).

As you can see, the Red Sox are the clear #1, while the Padres, Cubs, and Rangers rank near the bottom. As a Cubs fan, the news comes as no surprise, since for nearly all of my first several years of following the team (I started following in 1987), the Cubs never seemed to have a home grown player contribute to meaningfully to the team. Likewise, it seems as though the Red Sox have had an endless array of talent coming up through their farm system.

Of course, this still doesn't account for the fact that teams have undoubtedly changed a great deal since 1965, and the philosophy and scouting behind a team's drafting and development strategy when the draft first began likely bears no resemblance to the operation of today. Additionally, the WAR Above Average per Pick value is tough to extrapolate to the entire draft since the data I have is heavy on top picks and those top picks have higher WAR and a higher variability in WAR.

While the numbers are interesting, and give a snapshot of how teams have done with their past drafts (again, this is only a sample of picks, not all picks - perhaps another study has shown WAR by team for all picks - if so, that would be superior to the above table), we can't fully get at the question of how large a difference there is between a smart drafting organization and a poor drafting organization without fuller data and a more refined unit of analysis.

Draft WAR by General Manager

Perhaps more relevant than a team's drafting record is the record of individual general managers. For study this I compared 10 current general managers with substantial draft records dating to back before the 2001 draft. I went back and obtained all picks (not just the sample I previously had) for each of these GM's during their tenure so that I had a substantial amount of data to work with.

Comparing each GM's actual WAR to the expected WAR from the model and then comparing the GM's to each other, gives us an idea of how successful each GM has been relative to the others. The table below shows each GM and his drafting record.

As you can see, of the 10 GM's studied, Billy Beane is unsurprisingly at the top of the heap, followed closely by Walt Jocketty, former GM of the Cardinals and current GM of the Cincinnati Reds. Bringing up the rear are Brian Cashman of the Yankees and Brian Sabean of the Giants.

So, Beane has had good drafts and the Sabean has had bad drafts. Is this a real difference, or is this a simple artifact of luck? To investigate this, we first calculate the weighted variance of the GM's WAR Above Average per pick. This observed weighted variance is .036. Then we calculate the expected weighted variance if all teams were equally good at drafting (with an expected WAR Above Average value of 0 and a SD of 2.0, which is the SD of WAR Above Average over all picks). This expected variance is .013. Taking the square root of the difference of the two variances gives an estimate of the standard deviation of the true drafting talent across GM's. (Observed Variance - Expected Variance due to Noise = True Variance). Calculating this with our numbers tells us that the true distribution of GM talent (including scouting, development, etc.) has a standard deviation of .150 WAR per pick.

With each team making about 45 picks per year, this means that the SD of the GM talent over an entire draft is a staggering 6.75 WAR. Basically a good GM will net his team an extra 6 or 7 wins above that of an average GM in a single draft. An outstanding GM (top 3% of all GM's) can net his team 13 wins above that of an average GM. Of course the signs can be reversed when talking about poor GM's. This distribution shows just how valuable a good GM can be. As we can see here, the difference in draft quality is more due to skill than chance (though of course, chance plays a major role), and a good GM and scouting system can make all the difference.

According to Moneyball, Billy Beane at one point was to be essentially traded for Kevin Youkilis. While Youkilis has become an outstanding player, the trade would not have been a good one. Beane, in just 4 years of the draft between 1998-2001, brought the A's essentially the equivalent of a Hall of Fame player, giving the A's 46 extra WAR over what the average GM would have been able to acquire. This advantage was gained on his drafting skills alone, not even accounting for his ability to make expert trades or sign free agents. Of course, time will tell how Beane's drafts will turn out during the years that followed the proposed trade, but the point is made - GM's have an enormous impact on a team's successes, even when considering their ability to draft alone.

Even when we scale back the WAR Above Average per Pick by about 25% to account for the regression effect (.15 estimated true standard deviation divided by the .20 observed standard deviation = .75), we still find that Beane is good for about 9 extra WAR per draft, while Sabean and Cashman are losing their teams about 9 WAR per draft.

Unfortunately, because draft picks take so long to develop, it makes it difficult to tell in "real time" how a GM is doing. However, this short study of 10 current long-time GM's shows us just how valuable a good GM can be.