This is a great piece. I would love to see this idea applied to previous seasons to see how closely the expected jump in wins correlates to actual wins. Obviously there are factors independant of the farm system which contribute to wins and losses, but it would still be interesting. I know I will be following the standings at the end of this upcoming baseball season to see if the Rangers come out 1 or 2 wins higher this year compared to last year.

Interesting piece...A few questions:

-Are you using OLS? If so, did you explore other options (e.g., beta regression) given that the DV is not continuous? Try it with wins and use poisson or NB?
-What happens if you change your assumptions regarding normal distribution of rankings? Did you run it straight up? What do those results look like?
-What do the CI's look like for the win predictions?

This result also seems to pass the smell test: a great organization should be able to churn out 85 wins over and above what other competitors should be able to find in average systems and on the free agent market. Works for me.

Thanks for the comments.

The standard errors on predicted WPCT's are quite high due to random error, so don't expect the standings to necessarily match up with the results here.

I did not try other methods besides OLS. I think there are some other methods out there that would analyze this data more efficiently, but this was my first stab at it. Using an Heirarchical Model or some sort of advanced time-series approach may be useful.

Sky:

If you're just using OLS, then you should be dropping Market and Salary from your models.

Previous and current year salaries will only have an indirect effect through previous and current year WPCT. Likewise, market size will only have an indirect effect via salary and then WPCT.

Unless you're using a hierarchical model that's going to measure the effect of Market on Salary, then Salary on current and previous WPCT, and then those variables along with the farm system on future WPCT, you're just testing the same causal effects multiple times.

Finally, you may consider just sticking with rankings rather than normalizing. Normalization is an additional assumption you're throwing into the model. Nate Silver uses rankings rather than a normalized index in his very successful Secret Sauce model and that works fine.

Very interesting study and results. Thanks for the good job!

How many wins are the Giants looking to add on top from 2010-2018?

Looks like it will peak in 2013-14 about a little more than 3 wins. What does that mean exactly, they will add 3 wins over 2012? (meaning it is cumulative) Or just 3 wins over 2009?

Thanks.

JD, If market size and salary have only an indirect effect (through current WPCT) on future WPCT, then those variables would no longer appear significant when current WPCT is included in the model.

The fact is that salary and market size are significant predictors of future WPCT, ***even when current WPCT is accounted for***. That's why they are still in the model.

If Silver is using straight rankings in an OLS model I would really question his methodology.

What's wrong with straight rankings? Is there evidence that suggests that your assumption of normality is more appropriate? Both work as independent variables in an OLS framework. It is your dependent variable that is violating one of the core assumptions of OLS. (Plus it's really easy to re-run the model and see if there's much of a difference)

OGC, It means that the Giants system will give them a 3 win boost in 2013, also a 3 win boost in 2014. It's not cumulative though, so the farm system helps them the same in both 2013 and 2014 - three wins in each season. Overall the Giants can expect about a total 16 win boost over the next nine seasons due to their farm system.

Sparky & others, due to the Central Limit Theorem, things made up of many small variables (such as a talent on a farm team for an MLB team) will necessarily be normally distributed. That's why I transformed it the way I did. Farm talent isn't linear distributed like using straight rankings would assume.

Also, WPCT is close enough to being continuous that its no problem to treat it as such.

I'm not saying that it is/is not normally distributed, I'm just saying you provide no evidence that it is (and that it is simple enough to run it both ways to let us know what the difference is, if any). Similarly, I'm no statistician so I might be off base, but I'm having difficulty understanding how the central limit theorem justifies the assumption that the quality of farm systems are normally distributed.

Sky,

I agree with JD that the conversion to z-scores is an unnecessary assumption. I think the cleanest test would be a non-parametric approach like spearmans rank correlation. I think SPSS and the like allow you to enter in covariates, so you should be able to run the same exact test as above. It will be more conservative, but I doubt it will change your results.

It's probably worth doing because (I'd argue) the normality assumption is probably wrong. You would want the distribution of farm values (across teams) to be normally distributed. You're right that if any given teams farm value is a sum of independent random variables, that teams farm value distribution will be normal. But that doesn't mean that the distribution will be normal across teams. If there are systematic biases in how talent is distributed between farm systems, then you would expect a very non-normal distribution. But again, my guess is that the more conservative nonparametric tests will support your conclusions nonetheless.

Thanks for the publicity, Sky. I do want to point out that the rankings you linked to on SI.com are NOT Baseball America's rankings, however. They were mine. I don't do BA's farm system rankings myself; those are a collaborative effort. In fact, in that article, I ranked 1-5 and 26-30, but 6-25 were merely grouped in the top half and bottom half, then listed alphabetically. We haven't released our rankings yet, but they are in -- you guessed it -- the Prospect Handbook, which actually ships next week. Thanks for study, though. We feel good about our process and we're proud of our track record.

Please email me from a regular email.
You were close to stumbling upon one of the most remarkable patterns in baseball. Email me back or start your range a new years earlier covering Davey Johnson's Mets

Ted

Sky:

Re: statistical significance. First of all, your Market variable ISN'T statistically significant. Second of all, this does not mean they have independent effects. It could mean that, but it also could mean that there's a high degree of collinearity between those variables. Did you test for this? Third of all, unless you have an understanding of why Market and Salary should have an independent effect on future wins, then you simply shouldn't include them in the model. Period. Do you have a theory on why they would? I'd love to hear it.

Re: Rankings vs. Z-scores. I doubt as well Nate would use rankings in an OLS model, but that wasn't my point. You're assuming that the value of each farm system is normally distributed, even though in most of the years you only have 28 data points. Assuming that any set of data points with n=28 is going to be normally distributed is a very questionable assumption, and you don't have much of a foundation to apply it.

JD,
Ok, I'll agree that market size is only marginally signficant. As a result it isn't doing a lot in the model. If you remove it, things will remain pretty much the same either way. I will add, however, that when forecasting futher into the future, market size has a stronger effect. Why? A team may overspend or underspend its means in a particular year, but market size is a more permanent measure of a team's ability to attract talent.

As for normality, you have to assume the rankings have some sort of distribution in order to use it in a parameterized model. It's certainly not linear. As defense for normality, take a look at the dollar valuations done at BtB:
http://www.beyondtheboxscore.com/2009/3/27/807059/farm-system-value-rankings
As it turns out, the distribution is.....approximately normal.

Regardless of whether or not the model fulfills the assumptions of the Central Limit Theorem, it doesn’t satisfy the other requirements of Gauss-Markov. Just looking at the output raises red flags. It is conceivable that last year's salary has a negative impact on future winning percentage. One can imagine that teams with high payrolls may be suffering from bloated contracts given to aging players who depress winning percentage. This seems unlikely though (and may be the result of the multicollinearity between this year’s salary and last). I would be shocked if the coefficient was still negative if model was run excluding this year’s salary.

Amongst other problems, in order for OLS to provide a best linear unbiased estimate the covariance of the error terms is equal to zero (no auto correlation). Yet that won’t be the case in this example because the error terms will surely be related from year to year as salaries and winning percentage over time are included. There will also be issues of multicolinearity since the right hand side variables are related (e.g. market and salary). The model is not likely to be homoscedastic where for example, the marginal return of wins declines as salary rises.

Most importantly, though, the endogenous variable is not independent of the exogenous variables. To see this think of a variable that was excluded from this model: revenue. Winning percentage is almost certainly a function of revenue (not exclusively of course, but it is a variable). Revenue is not perfect because clubs have different debt structure and philosophical constraints, but it is sufficient for this example. Salary captures revenue, but not in it entirety. In a simple model Salary (S) maybe a function of R- anticipated revenue as well as actual revenues as the acquisitions may be made through the season, the owner’s wealth (O) and debt (D) with an error term (u).
S= ß1+ ß2R+ ß3O+ß4D+u

If you plug this into the model (with Market (M)…) then Winning percentage (W):
W=∂1+∂2+∂3(ß1+ ß2R+ ß3O+ß4D+u)+∂4Wt-1+…

But revenue is certainly a function of winning percentage as more wins tend to increase attendance… Here then revenue could be expressed:
R=π1+π2M+ π3W+…

In this example, where winning percentage is a function of revenue and revenue is a function of winning percentage, there is a simultaneous equation.

There are fixes for many of the problems (autocorrelation, et al) and many software packages have quick tests and even fix them with little effort. However if there isn’t independence between left and right hand side variables then OLS will not provide an efficient and consistent estimate and another method should be used. Most relationships you would wish to examine suffer from similar problems and require methods other than OLS.

James,
Let me first say, that yes, the model is not airtight. There are some other methods that could be used to refine it. That said, it works pretty well overall.

In response to your first point, salary from this year has a positive effect on future WPCT, while salary from last year has a negative effect. This is exactly what we would expect. Teams that have increased their payroll from last year expect to improve. Which team will do better? A) team payroll of $100 million, finished .500 last year, spent $50 million last year. Or B) team payroll of $100 million, finished .500 last year, and spent $200 million last year. The answer is obviously A. That's why the variable is negative. You are correct that if you exclude this year's payroll, the variable will show the opposite effect.

I'll agree I ignored the autocorrelation. However, the main result of doing that is that my standard errors are probably artificially small. Estimates remain the same, and most of the variables have sufficient power to still be significant even after inflating them. Overall, the basic model works, even if there are minor adjustments that could be made.

Sky,
Absolutely, no equation is airtight otherwise it would be an identity and not a model. Furthermore, I wouldn’t be surprised if OLS produced a reasonably accurate model in this case (i.e. consistently high rankings by BA indicating a degree of the value in the farm system that translates into a higher winning percentages in the years to come).

Most of my comment concerned minor points. The predictive power of models with multicollinearity, heteroscedasticity and autocorrelation problems will be fine. However models are not only supposed to predict but explain, The failure of these assumptions affects the explanatory component of the model. The coefficient should tell us about the impact of each variable in isolation from the others. Autocorrelation, as between past and present salaries, inflates the coefficient of determination and leads to inefficient estimators. The negative coefficient does not fully explain the affect of the past year’s salary without considering the present year’s salary. In terms of prediction this is fine, but not for explanation. Though explanation was less the goal of this analysis than prediction, did you consider for example using percent increase/decrease from last year to the present as the second variable to the current payroll? The covariance of the error terms I’d guess would be zero, and though it might introduce multicollinearity, I suspect it would be a cleaner variable.

My greater concern would be with the dependence of the left and right hand side variables. Predictively, OLS may be incidentally correct (and I’d certainly concede that the probability is high here) even if the model fails to satisfy the assumptions of Gauss-Markov. Personally, I wouldn’t be comfortable interpreting this or similar models 9 years into the future. The Texas Rangers are a good example of the variability and impact of management that would make me uneasy stretching that far, though that really is nit-picking. But if I was trying to quantify the affect with a fine degree of precision with regard to the magnitude and not just direction I would use something other than OLS.

Excuse me, but I had a question I forgot. Did you consider whether or not you were rejecting variables that are in fact statistically significant?

... it doesn't seem to make sense that a BA ranking from one or four years ago would have much more predictive value that the BA ranking from two or three years ago.

I have a two hypotheses that might explain this.

1. Contending teams are looking to win now, and when they have a highly ranked system *now* (best expressed by last year's rankings), that implies they have highly regarded minor league prospects, some of whom can be traded for help in the current year. So I wonder whether last year's system ranking correlates better because teams in contention trade prospects for current year help.

2. As for 4 years as opposed to 3, 5, or 6, I'd guess arbitration and cost is a major factor. 4 years after a system is ranked #1, its top prospects should have 2-3 years of MLB experience, enough to be getting very good, but not enough yet to command much salary in arbitration. So for earlier years, your talent hasn't yet matured to the point to optimally help your big league club. But for later years, cost-conscious teams may have already had to trade players they no longer can afford.

Big picture, I think you're right that small sample size is likely the biggest source of the difference in p-values. But I also wonder whether the two effects I describe might well be true.

To test the first hypothesis, I suppose you could break teams up into two groups, based, say, on how close they are to contention at the All Star break. Obviously contending teams would have a better record than non-contenders, so you'd have to adjust for that, but my hypothesis of trading prospects for current year help would predict that in the contending group, there would still be a positive correlation between last year's farm system rankings and this year's winning percentages. It would expect non-contending teams would have much smaller correlation, or perhaps negative correlation (teams not in contention are, if anything, more likely to trade away good talent for even more prospects, further weakening their current team). Of course, that makes sample sizes even smaller, and it becomes even harder to draw meaningful conclusions.

Hey Sky,

I didn't see it pointed out in the comments, but the Rangers were #4 in 2008.

This is the closest to a source I can find: http://www.sportsworldny.com/index.php?showtopic=16764

Keep posting stuff like this i really like it