I was kind of baffled by some of the comments in the previous posts attacking you for your "tone". I didn't see anything that seemed egregious in the way you wrote that and it seems like some posters seemed personally affronted that you would question PECOTA in this way. I don't get it.

I suppose that it is statistically possible for a projection system like PECOTA to be so far under on a team's win total every year, but each year it gets harder to believe it's just chance. Is there something in the construction of the Angels roster in talent or playing time distribution that leads to the projections being so far under every year? Or, to put that question another way, is there something in PECOTA's M.O. that leads to it underestimating the Angels every year?

PECOTA is not an open source system so no one outside Baseball Prospectus really knows for sure all the inputs that go into it. That said, to the extent that PECOTA projects winning percentages based on a team's expected runs scored and runs allowed, it would not be surprising if the system routinely shortchanged the Angels as the club has outperformed its Pythagorean record by an average of five games per season over the past six years.

However, PECOTA's problems with the Angels have more to it than just Pythagorean as there is a yearly average of six additional wins that are unaccounted for since 2004. In fact, PECOTA has underestimated the Angels Pythagorean record every season during this period by one to ten wins.

I'll let Bill James explain the possible whys and wherefores (as excerpted from The Bill James Handbook 2010):

The most efficient team in baseball is usually the Los Angeles Angels—anyway it was in 2009, and it was in 2008, and it has been in other years. The Angels do little things so well that they are consistently able to grind five or ten more wins a year out of their team than what one would think was available. We don't really understand how they do this, to be frank, but since they do it every year, we know it's not luck. Saying that they "do the little things well" is just a way of covering for the fact that we don't actually know how they do it.

If it wasn't for the Angels, we might think it was all luck. There are a couple of parts of the Angels' success that we do understand. For one thing, they run the bases extremely well. They picked up about 96 bases last year, or about 20 runs, just by running the bases better than the average team. Twenty-two of those bases are "stolen base gain," but 74 of them are bases gained by things like going first-to-third on a single or tagging up and advancing. That helps a lot. The Angels in 2009 had 221 "Manufactured Runs," by far the most of any major league team. Second, they usually have a good bullpen, which means that they can put a good pitcher on the mound when the game is close. Even in 2009, when they didn't have a really good bullpen, they also didn't have a really bad bullpen. Those things help to make a team "efficient," as we are using the term.

Perhaps BP would be well served to fine tune PECOTA to account for these nuances and adjust its projections after spring training. Re the latter, Sky Andrecheck, in a study that he unveiled this week, found that a team's Cactus or Grapefruit League record is a "statistically significant predictor of its regular-season record, even when taking into account its pre-season projection." The Angels have finished with the best record in Arizona three of the past six springs and no worse than third with winning percentages ranging from .576 to .765. Sky's statistical model might add up to three wins per year based on a strong showing in the spring, which is something that has become commonplace for the Halos.

I'm no fan of PECOTA, but I would be a lot more convinced by this line of thinking if someone could explain why PECOTA under-rates the Angels. What is it about the Angels that PECOTA misses, and how do we know that the Angels have that this year? I'm pretty sure that BP doesn't run PECOTA and subtract 11 wins from the Angels tally.

In discounting the 76-win projection, rather than look for a long-running PECOTA anti-Angels bent, I'd be more willing to accept that PECOTA's accuracy has gone seriously downhill the last couple years. We know they were the worst of the major projection systems last year, and we know that they're kludging their results so far this year and still have a number of errors in their system.

However, even CHONE has the Angels projected for 81 wins. I don't think you can accuse him of anti-Angels bias. And CAIRO has the Angels projected for 80 wins.

So whatever the projection systems miss about the Angels, they all do it. It's not a PECOTA-specific failing.

Mike: See my comment directly above yours. The timing of our comments may have overlapped, causing you not to see how I had already responded to a similar question. I hope it sheds light on the situation.

Stick to your guns, Rich. As an Angels fan who generally enjoys statistical analysis, I have been mystified by the continual dismissal of the Angels' consistent overperformance vs. projections as 'luck'. BP is just the most prominent example, other systems haven't really done that much better. I'm gratified that you (and to my surprise, Bill James) are giving this some light. I can't say I'm surprised at many of the comments in the other thread, though, it mirrors what happens any time this subject gets brought up.

Are you forgetting the 2001-2002 Mariners? 116 wins down to 93 the following year. That is so embarrasing -- 3 times in the 2000's, the M's have dropped more than 20 from the previous season.

Btw, I'm a huge Kevin Jepsen fan. I don't think most people realize how good he was after adding the slider to his repertoire on July 3rd and consequently cutting back on his curveball usage. He was a different pitcher after that point.

Bill James has a good point about baserunning being something that usually isn't captured in the projections beyond the extent to which it is reflected in the stolen base game.

I thought someone, perhaps on this site even, had looked at bullpen leverage and determined that while it was a small contributor to the Angels success above the level their runs scored and allowed would suggest, that it did not explain the majority of the difference.

I don't believe the difference is all or even mostly luck, but I remain skeptical of anyone who claims they know either way. James is not correct that just because they've done it for several years straight that we know it's not luck. Over a period of years, some team would be the biggest outlier simply by chance. It's selection bias to assume, absent other evidence, that the fact that the Angels have outperformed for a number of years in a row proves anything.

Good catch, Nathaniel. I have corrected that oversight. Thank you.

On my site I have two listings of team projection methods. The first is the depth chart method, where I had the Angels at 81 wins. The second is the starting lineup method, and I have the Angels at 83 wins there.

The second is what I will consider my official projection, because I don't think my depth charts are any good. I don't have enough knowledge about all 30 teams to do that in a consistent manor, and in the future I will publish the starting lineup method only for team projections.

There is something there that is beyond PECOTA, or any other ranking system that the Angels have. I'm not exactly sure what it is, and neither is anyone else, but it's consistent. I'm guessing it's a combination of things. A great manager, a strong minor league system, consistent fan support, great owner and a sort of "anti-AL" style of play that is very effective. The Angels under Mike Scioscia have always had that "it" factor, and "it" is not luck, of this I am convinced.

Let's also take into consideration that 2009 was Kendry Morales' first year as a starter, if his numbers go down in the physical prime of his life I'd be shocked. He bashed in Cuba, he mashed in the minors and he slashed in the majors. Erick Aybar is another player that's only bound to get better, he still hasn't added SB to his game and he's the fastest player in the Angels system. He may be forced to swipe bags this season.

Let's also consider that Ervin Santana, Joe Saunders and Scott Kazmir all had down years last year. I would not expect all 3 to get worse in the prime of their pitching careers as all could potentially be front of the rotation starters. Then there's the bullpen, they are getting Scot Shields back and brought in Fernando Rodney, to go with an already strong front line of Bulger, Jepson and Fuentes. Games are going to be shortened.

Also, the Seattle Mariners scored 50 less runs than they gave up last year, suggesting their record over .500 was a fluke. THey lost the power than Branyon supplied, but added Figgins and Kotchman. The front of their staff is also going to be much better with addition of Lee. I expect the Mariners to have around the same record. As for the Rangers, they aren't any worse, but I do expect that young pitching staff of theirs to struggle as it did late in 2009. The A's have one of the best young rotations and bullpen in the game and no one knows it. They added some bats and should compete as well.

I have no statistical backing for this guess in the final win total of each team it's just what I call "gut feeling".

Angels 92 wins
Mariners 86 wins
Rangers 85 wins
Athletics 80 wins

I'd also just like to add, that I believe the AL West will be the most exciting division this year. Every team brings something to the table, and in the end, it may just come down to who makes the least amount of mistakes.

The Mariners are well coached and have a fantastic defense. It's hard to argue against that. The Angels have the most "pop" as well as experience and leadership. The Rangers are a young talented squad that will probably only get better over the next 5 years. The A's are jampacked with stars in the minors and young players in the majors, this could mean they may win a division crown in the next 5 years.

Best of luck to all teams and their fans and thanks to Mr. Lederer for posting this article. It brought to light certain traits that shouldn't be ignored in attempting to determine the standings of a team like the Angels.

I guess the closed minds of the stat orthodoxy approved of your tone in this post, Rich. Maybe you'll be spared their coming "Inquisition" of the heretics challenging their doctrine of the faith.

Perhaps the mysterious thing that the Angels have is named Mike Scioscia.

Yet more proof that baseball isn't played on paper (or in computer simulations). If it was, well, we wouldn't have as much fun trying to predict what happens next year, nor would "stat geeks" have so much fun trying to capture the genie in a bottle.

I find Rich's 7:22 comment much more interesting than the post itself, because it starts to confront the PECOTA prediction head on.

But it is just a start. Firstly, and I know this is Bill James talking, not Rich, but I wouldn't say - no one conversant with statistics would say - that "but since they do it every year, we know it's not luck." The problem is the word "know." I would say that the year to year overperformance is interesting, and may well not be luck (perhaps evenprobably not?), but "know" is too strong, and even if it isn't luck, we don't know that the .. whatever it is ... will continue.

But in any event that factor is about 5 wins. Rich seems to think that the real number should be ... well, he doesn't say, but apparently a lot higher than 81 wins (76+5).

And there's where I would like to see some more enagement of PECOTA's method. While some of their underlying numbers can be pretty opaque, the team win predictions are not. They basically take the sum total of the individual predictions. So Rich, which of the individual predictions do you disagree with?

Rev,

You haven't been reading long here, have you? Rich displays his obnoxious tone against anyone who disagrees with him, including and even especially the more "traditional" baseball analysts. His more appropriate tone in this post should be welcome to everyone, regardless of their feelings about analytical baseball statistics.

"Yet more proof that baseball isn't played on paper (or in computer simulations). If it was, well, we wouldn't have as much fun trying to predict what happens next year, nor would "stat geeks" have so much fun trying to capture the genie in a bottle."

As a simulation player, let me tell you that simulations are every bit as unpredictable as real life baseball. Well, if you do the simulations 10,000 times each and average them out you'll get the same result as the projections you put in.

But play it out as a season, one time only, and you'll get teams that surprise you, others that choke despite talent advantages, breakout players and sudden collapses. In other words, the fun stuff that makes baseball exciting.

If you're taking PECOTA seriously, you might want to look at this:
http://vegaswatch.net/2009/11/evaluating-april-mlb-predictions-2005.html

You could look at 2009 as an outlier, say they just had a bad year, but we know that Nate Silver, who created PECOTA, is no longer involved. Personally I can't take it seriously until the new custodians of the system prove that they can deliver accurate results.

Aha, this is more like it. By the way, sorry if my comments on the previous article seemed to lump with the other, more critical ones...I never meant to put what you were doing down, so much as constructively criticize what I felt was not a reflection on the "analysis" you typically provide.

I think your new article reconciles a lot of the issue. As you saw in one of my comments, I also anticipated this kind of result. To be fair to BP, and to you as well, I think you should stop looking at this as a prediction of "97 wins in 2009 to 76 wins in 2010".

Because BP consistently underprojects the record for the Angels, you should only question the drop they predict BEYOND the 11 win average. Since 11 of the 21 predicted fewer wins comes from their system, that means that they're projecting a 10 win loss from "baseball" factors (age, talent, etc)...that's still fairly large, so I think that's what you should focus on.

I suspect BP is more pessimistic about Angel pitching than you are, and that's why they're frowning more on team than you would expect. I don't think they're totally off base there...here are my thoughts on three key Angel SPs:

Saunders: Honestly, I think he's viewed as a better pitcher than he's been. The team needs him to be much more the '08 Saunders than the '09 version.

Kazmir: At the moment, you can't anticipate that he'll reflect the pitcher we've seen over the past five years. He's had various issues the past two years, health related and otherwise, and we may have already seen him at his best.

Pineiro: I'm of the opinion that, for many reasons, STL pitchers have greatly outpitched their ability while on the team. I think he needs to show that last year was real before we pencil him in for anything other than lots of crummy innings.

In other words, some people (and probably BP) are assuming things on the lower ends here: continued mediocrity for Saunders, injuries and ineffectiveness for Kazmir, Pineiro being no good, Santana not fully rebounding, and the bullpen struggling as a whole. I don't see any reason why these players CAN'T do well as well, but maybe BP just isn't bullish on any of these guys.

For the record, though, I'm with you on Santana. I was quite impressed with his second half and loved what he did in '08...he'll be on my fantasy baseball team for sure.

Chone, it's not like they changed the methodology when Nate left. That's kind of what bugs me about all of this analysis - say what you will about the system, but it's based upon objective metrics, which are little changed from year to year, it's not just a group of guys sitting in a room and making predictions. Now, that has it's advantages and disadvantages - bias and subjectivity is less likely to creep in, but losing the human factor is a problem sometimes (in this case, if it wasn't a purely mechanical system, you would think someone would step in and say "whoa, 21 games without huge personel changes is an awful lot.")

But one year is a pretty small sample size, and it's silly to dismiss their overall track record, which is stellar, despite a string of misses on the Angels in particular.

Myself, I certainly would take the over on 76 wins, but a 10 win decline would not surprise me at all. Looking at the position players/DH in particular, they're going to miss Figgins a lot, Hunter is likely to take a huge step back, Abreu likely a smaller one, whereas a couple of the younger players who took huge strides forward last year are more likely to have consolidation years where they maybe even go a little backwards a bit, as opposed to improve further. That may be outweighed a bit by some other players stepping it up, but I defy any objective person to take a close and careful look at that lineup and not see that a decline is likely. How much is of course another issue.

Chone wrote: "As a simulation player, let me tell you that simulations are every bit as unpredictable as real life baseball. Well, if you do the simulations 10,000 times each and average them out you'll get the same result as the projections you put in.

But play it out as a season, one time only, and you'll get teams that surprise you, others that choke despite talent advantages, breakout players and sudden collapses. In other words, the fun stuff that makes baseball exciting."

Certainly, and that is just the point: the season is "one time only", and thus it will always be made up of exceptions.

I tend to find computer simulated projections to be less trustworthy than arbitrarily assigned human-made ones (albeit informed by some kind of analysis and with some degree of intelligence). Computer simulations can see patterns of performance, but don't necessarily see WHY those patterns occur.

Take Joe Saunders, for example. He sucked for much of the year, was hurt (probably for longer than he let on), came back and was stellar. I'm not saying that the real Joe Saunders is the Glavine-esque one in the latter part of the season, but that his total year-end performance only tells part of the picture. We can look at Saunders' rate stats and say, "That's a mediocre pitcher." But we could do the same of Tom Glavine, or numerous very good pitchers. Saunders, in that sense, may be the quintessential Angels pitcher: He's better than he looks on paper.

Or let's look at Howie Kendrick. He slumped terribly during the first half, was sent down for a month or two and then came back and raked. Most projections will just look at his total season and factor that into their formula, without seeing the psychology involved.

So that's just it: computer projections do not, cannot, take into account human psychology. To put it another way, they can take into account the EFFECT but not the CAUSE.

As an Angels fan I am hardly unbiased. But I also know a lot more about Howie Kendrick than most non-Angels fans, and certainly more than a computer simulation. I know, for example, that he has a .360/.403/.569 in 399 minor league games, with even numbers across levels (if anything he improved the higher he went), and that these numbers are not only better than the minor league careers of many great major leaguers with similar skill-sets (e.g. Tony Gwynn), but that they are based upon tremendous hand-eye coordination and bat speed.

I know that Kendrick has had numerous stumbling blocks in his four year major league career, including persistent injuries, inconsistency, and an extended slump last year, all of which has never given him the experience of playing a full season through age 25. I know that when he came back from the minors last year, he hit close to that minor league line.

Will Kendrick hit .360/.403/.569 in the major leagues? Probably not. But we're talking about a guy who has put together a major league line of .302/.333/.434 while never playing consistently for more than a couple months at a time, never beeing fully healthy and fully at ease. Last year was his first fully healthy season, but he didn't settle in until the 2nd half, perhaps largely due to expectations (mainly his own) and a tendency to press. When he did come back from the minors he seemed to be a different hitter, and we know what happened then.

The point being, a statistical formula only sees the numbers. It doesn't see the story behind them, or the human being who puts up the numbers. PECOTA thinks Kendrick will hit .299/.345/.447 next year; I think he'll hit .320+ with an OPS around .850. Is it because I'm an Angels fan and unwilling to look at Kendrick in an unbiased fashion? Or is it because I have a sense of the story behind his numbers and even some sense of him as a human being? I'd say some of both, but it isn't either/or. I have even found myself over-compensating when it comes to the Angels because I want to account for my own bias. For example, last year I predicted that they would be much stronger offensively than analysts were saying, but I predicted 827 runs scored, not the 883 they actually did score, when at the time I remember wanting to predict 840-850 (this year I'm predicting 827 again).

"Chone, it's not like they changed the methodology when Nate left."

I'm not convinced. They tell us that they moved it from a collection of excel macros to a database - this is not a simple cut & paste, but a long process where errors can be made. My projections system is constantly being tweaked, it is similar to the one I rolled numbers out for 2007 in name only. I assume this is the case for other projectors, other than Marcel which does not change.

The Matt Wieters projection last year was a real red flag that had a lot of people shaking their heads. They projected that he'd treat MLB pitching like an inferior version of the minor league pitching he had been hammering. Then Wieters played like a rookie. When all was said and done, the system came in last place when it comes to measuring the accuracy of several systems.

My experience is that a projection system needs human oversight. You run the numbers, and then try to spot things that went wrong. Then you investigate your code, see why the results look strange. Sometimes you just figure it's doing things as intended even if the results are surprising. Sometimes you find something didn't work out right, and you fix it.

I go through this every year. I'm sure Nate Silver did as well. I don't know how well the current custodians of the system will do this year, but last year it is verifiable that they did a terrible job.

If you've been following their team projections those standings have been revised at least 4 times. I don't know if the revisions are making things better or worse, but that should put to rest the idea that they are taking a proven system from Nate Silver and running it without alterations.

"But one year is a pretty small sample size, and it's silly to dismiss their overall track record, which is stellar, despite a string of misses on the Angels in particular."

Here's an analogy: For 10 years, "Phillies 3rd baseman" has a .900 OPS or better. Next year, "Phillies 3rd baseman" OPS's .700.

What do you expect going forward? Does the one year make you throw away the track record? Well, yes it does if you know that the Phillies 3rd baseman is no longer Michael Jack Schmidt, but Rick Schu.

About the PECOTA prediction in general. Aside from the 76-86 record, PECOTA projects 780 runs scored vs. 835 runs allowed.

I can live with the 780 runs scored; I think it is a bit low (my prediction is 827), but it isn't ridiculous, just the usual stat projection conservatism.

But 835 runs allowed? Come now. First of all, that's 74 more runs allowed than last year (761), which was the most the Angels have allowed since 2000. And note that last year the Angels pitching staff was utterly deciminated by injury (and death, in one case).

OK, you say, but staff ace John Lackey is gone, replaced by Joel Pineiro. John Lackey started 27 games with a 3.83 ERA and is not being replaced by Joel Pineiro, but by Scott Kazmir, who happens to have a career ERA of 3.83 (to Lackey's career ERA of 3.81), although only that high because of his first half struggles with the Rays. John Lackey is not a great pitcher; he has had one great season (2007), when finished 3rd in AL Cy Young voting and could well have won the award, but that is the outlier. Here are Lackey's and Kazmir's ERAs the last five years:

Lackey: 3.44, 3.56, 3.01, 3.75, 3.83
Kazmir: 3.77, 3.24, 3.48, 3.49, 4.89

The point being, the pitcher that Lackey is being replaced by is, at the least, in the same ballpark. Joel Pineiro, as the article points out, is replacing starts by Matt Palmer and a bunch of rookies who weren't ready for the major leagues, so even if he churns out a league average ERA he'll solidify the bottom of the rotation.

As for the rest of the rotation, Joe Saunders will likely never be as good as his 2008 season, but he certainly was a lot better in the latter part of 2009 after coming back from the DL. If you split the difference between 2008 (3.41) and 2009 (4.60) you get an ERA around 4.00, which is what I think we can expect from Saunders in 2010.

As the article mentioned, Santana was injured for much of the year but came back strong in the second half. Of all the Angels starters, he has the ability to be the new "staff ace," although there is still some question about his health. As for Jered Weaver, he is the most reliable starter on the team and should be at least as good as last year.

Finally, the bullpen. For the first couple months the bullpen really struggled, but settled down by the second half, largely due to the emergence of Jason Bulger and Kevin Jepsen. If Bulger holds steady and Jepsen continues to improve, the addition of Fernando Rodney and the return of Scot Shields should improve the bullpen, if not to the peaks of a few years ago, at least better than last year.

PECOTA's predictions for starting pitcher ERAs:

Kazmir 4.32 (2nd lowest of career, at age 26)
Santana 4.60 (evidently 2008 was a fluke?)
Saunders 4.44 (ignoring his post-injury improvement?)
Weaver 4.06 (decline at age 27?)
Pineiro 4.29 (makes sense)

I think a more realistic expectation is something like:

Kazmir 3.70
Santana 3.50
Saunders 4.00
Weaver 3.70
Pineiro 4.30

I apologize for clogging up the comments, but one last one. So I'm predicting 827 runs scored and I'll say about 750 runs allowed, which would be a Pythagorean record of 85-77. Given that the Angels tend to out-perform their Pythagorean record, I'm predicting 90-93 wins, which should be good enough to win the AL West (I see the Mariners and Rangers both being in the upper 80s, and the Athletics around .500). How about this:

Angels 91-71
Mariners 88-74
Rangers 86-76
Athletics 80-82

Should be a good race.

I don't have BP2010 in front of me, but having read it over the weekend I'm pretty sure that it specifically mentioned how the Angels consistently outperform PECOTA by the same amounts that Mr. Lederer comes up with. While not pinpointed, the difference does seem to be related to the Angels' superior basrunning (1st to 3rd, etc.) as well as effective bullpen usage.

Obviously Scioscia has a lot to do with this - it's no coincidence that these do seem to be aspects of the game that the manager and coaches have more direct influence on - reliever selection, baserunning aggressiveness, etc.

Just my 2 cents.

Jonathan,

Something don't add up. 832 runs allowed is 5.15 per game, which suggests an ERA around 4.75.

Even PECOTA does not project any of our starting pitchers to be that bad, as you posted:

Kazmir 4.32 (2nd lowest of career, at age 26)
Santana 4.60 (evidently 2008 was a fluke?)
Saunders 4.44 (ignoring his post-injury improvement?)
Weaver 4.06 (decline at age 27?)
Pineiro 4.29 (makes sense)

That's a group average about 4.30-4.35? So how do you get a team figure of 4.75? Either the bullpen has to be horrible, or else a lot of injuries leading to Bell, O'Sullivan, etc. making starts.

Last year's bullpen had an ERA of 4.49, and that is as bad as it's been in the Scioscia era. The bullpen should be much better thanks to Jepsen having learned how to pitch and Shields coming back.

But even if I assume the 5 starters are 4.35 ERA, and the bullpen is 4.49 just like last year, then to get to 832 runs allowed I have to assume only 104 starts for our top 5, and 58 starts from the backup starters at an ERA of 6.00.

Chone,

All fair points. But looking at your own projections, it doesn't exactly look like you're that bullish on the Angels either - 81 to 83 wins. Not that far from PECOTA. Just eyeballing the individual projections quickly - let me know if I missed anything major - most of the "low" PECOTA projections that people like Jonathan are complaining about above aren't that far from your own projections.

Plainly put, statistical analysis of baseball is just plain inaccurate. I won't call it anti Angel, it's just wrong. That's all there is to it.

Just look at the facts, the Angels probably won't be as good offensively, will be the same defensively, and will be tremendously better in pitching and bullpen as well as health. You can't convince me that Matsui will miss 1/3 of the season like Vlad, Hunter will miss another 40 something games, Weaver the new staff ace will miss 1/3 of the year as will Santana, that Scot Shields will miss the entire year again, Kendrick will flounder and our top pitching prospect will tragically die. ALL OF THAT HAPPENED in 2009 and the Angels still won 97 games.

The Mariners were outscored by over 50 runs last year, making their over .500 mark luck in every sense of the term. Their offense is actually worse and their rotation better with the bullpen the same. If lady luck weren't in their corner in 2009 they wouldn't have won 80 games. It's foolish to believe this fortune will continue. The Mariners will be lucky to win 80 games this year.

Anyone who does their homework and keeps true to it will realize the Angels should be favored to win the AL West again, that the Mariners will more than likely win less games and our only serious competition will come from the Rangers, who we beat by 10 games last year and have slightly improved over the winter.

Are the Rangers 10 games better than they were last year? Probably not. Are the Angels going to be 5-10 games worse than last year? Given their hellacious string of injuries and death in 2009, probably not, in fact most indicators are pointing toward the Angels being better.

Larry,

Fair point. Though if the Angels win 90, then my projection of 83 (of the two team projection pages, that is the method I will stand behind, for all teams) will have an error that is half of PECOTA's 76.

I hope that my projections are dead wrong and the Angels beat them by 20 games or more. (Plus 11 in the playoffs).

But some of my PECOTA comments stand regardless of whether they update again and find the Angels at 87 wins. The system had a horrible year last season, and there are enough questions, turnover-related, to wonder if the system is capable of accuracy at this point. They might, but they need to prove themselves again.

halowood, I wouldn't say that "statistical analysis is anti-Angel" but "the Angels are anti-statistical analysis," at least to some degree, as they seem to consistently out-perform predicted performance, whether it is PECOTA, Pythagoras, and others.

And I agree, the Angels might actually be slightly better next year, at least insofar as they don't have any major weaknesses and should (crosses fingers) be relatively healthy. But they will likely win a few less games, if only by virtue of the fact that every other team in the division will have improved.

Chone, nice catch. I guess we can't answer that question unless "someone" is willing to do the legwork and add up the ERAs of every Angel PECOTA projection...

Agreed. Because the Rangers and A's are better teams, and because the Mariners added another good starter, chances are the Angels won't win 97 games. But then again, the Angels could have won 87 games last season and took the division. It's not looking like they'll need to win any more than 87 again this year to wrap this division up, I'm expecting between 90-95 wins. Not bad for a team who's time apparently came and went.

2002 was supposed to be a fluke. We weren't supposed to win in 2004. We weren't supposed to win or go to the ALCS in 2005. The Mariners were going to win in 2007, except they didn't, we did. The Rangers were supposed to win in 2009, we lost Tex and K-Rod and they had a 5 game lead in May and all the experts said the Angels were in disarray. We definitely weren't going to sweep the Boston Red Sox in ALDS let alone make it there. But we won the division by double digits and swept the chowds. 2010 the Angels lost Lackey, Figgins and Guerrero and the Mariners became big spenders.

Is anyone else here seeing a pattern?

"statistical analysis of baseball is just plain inaccurate"

What can one say about such a blanket statement?

Maybe I can redeem my snide comments against Rich by patiently explaining what's wrong with this statement. Statistical analysis can serve two purposes - retrospective and prospective. Retrospective analysis is pretty straight forward, and statistical analysis does it very well.

Prospective analysis is tougher - but not just for statistical analysis, it's tricky for traditional analysis as well. Baseball, like life, isn't always predictable.

In prospective analysis - predicting team and individual results - statistical analysis has a role, but isn't perfect and has its blind spots. Just like traditional analysis.

That said, there is plenty of evidence that pure statistical analysis can do a very good (if imperfect) job of predicting performance. Specifically with regard to team performance, the evidence as I understand it (and if there is evidence to the contrary, I'd love to hear it), is that the best predictions of team performance come from the statistically based systems.

Of course, those systems are always open to criticism, and we've seen some of that here. IMO there seems to be reasons - both in terms of perhaps slightly low performance predictions by PECOTA, combined with the Angel's fairly consistent out-performance of statistical predictions, and maybe some other “under the hood” flaw in PECOTA - to believe that the Angels will win in the neighborhood of 85 games or maybe a few more, rather than 76. I personally don't see any reason to believe they will be as good as they were last year, but maybe I'm missing something. That's not the point though, so much as that you can't just wave your hands and dismiss the statistical analysis. You need to come up with REASONS why the statistical analysis is wrong.

Pattern: The use of "we" as if the commenter was a member of the team.

Pattern: Assuming best case scenarios for every Angel player and ignoring the possibility of down years.

Actually one of the interesting 'patterns' I would claim about the Angels, is their ability (organizational approach stemming from the GM and Scioscia) to spread the risk and talent of the team (i.e. not having many weaknesses) to account for the possibility of injuries and down years.

It isn't necessary to paint a best case scenario (see Chone's post above about Angel starters ERA) to predict the Angels performing better than PECOTA and others might suggest.

Has anyone kept detailed stats about the accuracy of PECOTA?

Does PECOTA make accurate predictions more or less as often as the BBWA correctly awards gold gloves to the statistically best defenders?

Short answer is: NO. Long answer is that PECOTA apologists will widen the range of what is an acceptable outcome to the point that PECOTA is really not making a prediction.

Rev,

You know, people DO keep track of this stuff. Really. Really really. Prior to last year (see CHONE'S comments above) they were remarkably accurate overall regarding team projections. See CHONE's link for details.

As for individual player projections, I'm not personally aware of the data, but my understanding is that for the most part the player projections are quite accurate.

Now, accuracy is of course ALWAYS relative. Your un-evidenced claim that "apologists will widen the range of what is an acceptable outcome to the point that PECOTA is really not making a prediction" really demonstrates that you don’t understand what a “good” prediction is (at least in the baseball context). Because of the element of luck, the best prediction (computer or otherwise) will more often than not be at least a little off. Let’s say that an all knowing baseball fan was able to correctly integrate all relevant factors and predicted, “correctly,” just to throw out a number, that the Yankees would win 95 games this year. In fact, the Yankees most likely wouldn’t hit that 95 win prediction on the nose. Injuries (especially), but also just purely random performance fluctuations, would probably mean an actual win total somewhere between (say) 90 and 100 wins, but hitting the 95 win prediction on the nose would itself be partly a matter of luck, no matter how good the prediction was.

The bottom line question is simply whether PECOTA’s predictions are more accurate than subjective "eyeball" prediction. And they are. Or, at least, were until last year, with the jury perhaps still out as to whether the system is still working as well as it was (see CHONE's comments above).

To make the abstract a little more complete, see Jonathan’s 11:11 a.m. post. He compares PECOTA’s ERA predictions with his own “eyeball” predictions. I would bet a large sum of money that the real numbers will be far closer to the PECOTA predictions than to Jonathan’s “eyeball” projections.* But lets say that the PECOTA predictions are off by a little, but the real numbers are far closer PECOTA than Jonathan’s predictions. Is Rev gonna coma back here and say “Ha!” Didn’t hit the ERAs on the nose. What a lousy system. How can you defend a system that doesn’t make a perfect prediction.” That’s just an impossible and stupid standard.

As a side note, giving further credence to the PECOTA ERA predictions, CHONE’s system, and other statistical prediction systems, are close to the PECOTA predictions. CHONE, for example:

Kazmir 4.23
Santana 4.47
Saunders 4.57
Weaver 4.10
Pineiro 4.45

* That said, in a best case scenario, with everything breaking right for the Angels, Jonathan could be correct. But that chance doesn’t make his prediction a good one; it makes it a best case scenario.

I'm sorry, I got stuck at the part of the argument that was "if PECOTA is accurate, then the Angels win total will fall over twenty games from 2010".

The chart provided shows that just about every year there is some team, sometimes two (one in each league) that drops twenty games in the win column from the previous year or more. Now it would be interesting to see what all these teams had in common, and to predict which team in each league is the most likely this year to be the team that drops twenty games. Because apparently this seems to happen almost every year.

Now I'm persuaded that PECOTA gets the Angels wrong every year (and that their talent level is about the same as last year, though other teams in the division have gotten better), but I never put much stock in systems like PECOTA anyway. Once we can predict the performance of individual players one year out pretty consistently using statistical methods, I think we can start trying to the same thing with teams.

Look at the success of numbers based systems in predicting individual stocks and the stock market, where you have quite a bit more money riding on the outcome.

I think Rich is making the mistake of getting WAY too hung up on the PECOTA projections versus the final records, given that any team's final record itself may be distorted by irregular run distribution, winning a lot of one- or two-run games but losing the blowouts, thus distorting their run differential. Last year, the Angels went 27-18 (.600) in one-run games last year, the league's second best winning percentage, but they were 2-5 in games decided by 10 or more runs, so at the extremes they were were outscored by 26 runs despite going 29-23.

Thanks to that, the Angels overachieved relative to their third-order Pythagenpat record - their expected win total based upon the combination of events on the field (hits, walks, total bases, stolen bases, and outs of all kinds, as well as those of their opponents), all adjusted for park, league, and quality of competition - by at least 10 games for the second consecutive year, and by at least eight games for the third consecutive year (their 2008 club set a record by outdoing their 3rd-order by 16 games, the 2007 club did so by 8.1). As I wrote last fall (http://www.baseballprospectus.com/article.php?articleid=9529), both of those feats are unprecedented in the annals of baseball history.

So if PECOTA pegs the Angels at 81-81, and their 3rd-order Pythagorean record is 87-75, but they instead go 97-65 - all of which happened last year - are we supposed to hang our heads or blow up our blueprints simply because the system didn't see an historically unprecedented statistical outlier? I don't think so.

As I wrote in the BP 2010 annual (now available in bookstores and from the usual online outlets such as Amazon and Barnes and Noble), the Angels' continued overachievement - exceeding their first-order Pythag - for six years in a row goes hand in hand with their high league rankings in our reliever win expectancy stat, WXRL. High-leverage success - clutch hitting, clutch pitching, clutch baserunning - is exactly the kind of thing that moves the needle relative to a Pythagorean record. Since 1954, the correlation between a team's cumulative WXRL and their third-order discrepancy is .42, by far the highest of any of our offensive, defensive, or pitching metrics. As bullpen usage has become increasingly more codified via matchup specialists and one-inning closers, the correlation has increased; since 1996, the first full season since the strike, it's up to .48. By comparison, the latter-day correlations for our other win expectancy based stats are .20 for SNLVAR, our starting pitching stat, and .38 for WX, our hitter stat.

That the reliever stat would be so high relative to the others makes sense, given that the manager can more easily optimize his choice of players as the situation evolves inning to inning and batter to batter. The Angels' consistently strong performances in this department - ranking in the AL top three in team WXRL in five of the last eight years, and in the upper half of the league for each of the past 10 (Mike Scioscia's entire tenure) - shed plenty of light as to why they consistently outdo their Pythagorean records. That's not something that PECOTA is particularly trained to assess and account for, but it does go at least part of the way towards explaining the discrepancy between the Angels' preseason projections and their final records.

As an aside, I should remind you that at this point in time, the 2010 PECOTA projections should be considered preliminary. They're updated continuously as the spring progresses to take into account playing time decisions and roster moves, so to harp on them before a single spring training pitch has been thrown seems to me to be a bit premature. Don't get me wrong, we at BP are thrilled that people are so attuned to them, but I don't think you'll see anyone in our organization running to place bets on those numbers while it's still February.

Great to see Jay's response. I do find "the 2010 PECOTA projections should be considered preliminary" significant. I applaud BP for their openess about the ... issues ... that PECOTA is having this year, but I have to say that a combination of Chone's posts above and recent communications by the BP staff on their site have made it clear to me that there are still some real issues with this year's PECOTA projections.

But the bottom line is that just about any systematic analysis is going to show pythagorean wins in the low 80s this year for the Angels. Add perhaps 5 wins for sustainable performance above pythagorean wins for the Angels, based either on bullpen factors, baserunning, or both, and you get wins in the mid to upper 80s. Which still could be enough to win the division.

I haven't seen any convincing arguments here why we should think it should be higher.

To summarize, with respect to PECOTA and the Angels, I believe there are two issues at work.

1. The Angels have outperformed their Pythag record every year for the past six seasons.

2. PECOTA has not only underestimated the Angels’ win total every year during this period, but it has undershot the team’s Pythag record every year as well.

I believe the latter point suggests that there is something in the inputs that fails to account for the Angels’ (or Mike Scioscia’s) success. The Bill James quote I excerpted from the 2010 Handbook and Jay's comments in the BP Annual that he plugged above do a good job at explaining many of these factors.

By the way, I think it has been lost on many people that the purpose of the first article was based on my gut reaction to PECOTA’s latest win projection for the Angels. Am I the only one who believes PECOTA’s 76-86 forecast doesn’t pass the “smell test?” It was only at that point that I decided to go back and take a look at PECOTA’s record as it relates to the Angels. It is obviously not a good one. As such, my findings only added to the eyebrow-raising projection for this year. The fact that PECOTA has consistently and apparently systematically shortchanged the Angels is noteworthy, especially when you factor in the magnitude rather than just the direction of the misses.

"The Bill James quote I excerpted from the 2010 Handbook and Jay's comments in the BP Annual that he plugged above do a good job at explaining many of these factors."

But the Bill James quote this goes to issuer 1 rather than issue 2. And while Jay's comments are significant, I don't think they show specifically a sustained systemic problem with the Angel's pythagoean ranking, which is what you are getting at in point 2.

As for point two, is this true and if so by how much? If it is true, my initial assumption, absent some explanation to the contrary, or a LAGRE sustained underestimation, would be that it's random noise, and thus not likely to be repeated.

That said, in fairness to Rich, there does seem to be another factor here, call it issue number 3, which is that there is at least some reason to believe that the 2009 and 2010 PECOTA predictions are generally problematic, and that (as a starting point, at least) projections such as chone, which put them in the 81 to 83 win range, are more reliable.

As for point one, I think everyone here, and even Jay, would acknowledge some validity to this. The questions (it seems to me) are (1) how much of the gap is real (i.e., it is possible, maybe even likely, that a portion of the Angel's success in this reagard is luck), and (2) how much of the gap is sustainable (i.e., even to the extent it represents something real, is that something real sustainable). Five wins for this factor is IMO if anything overly generous.

So Rich, what's your projection?

I looked at the numbers. Going back to '03, it looks like the Angels out performed their pythagorean W/L by just under 4 wins per year, so my 5 win estimate may be generous (though most of their overperformance is in the last 3 years, for whatever that's worth).

Which means that PECOTA did underestimate pythagorean W/L by almost 5 wins per year, more that I would have thought. Again with the caveat that I don't have much faith in THIS year's PECOTA, I'd still need a lot of convincing that that isn't just random noise, absent a reason to believe otherwise.

All that said, if Rich is saying that he projects the Angels at 87 wins (implied but not stated in the original post), then I can't say I disagree.

"The bottom line question is simply whether PECOTA’s predictions are more accurate than subjective "eyeball" prediction. And they are. Or, at least, were until last year, with the jury perhaps still out as to whether the system is still working as well as it was (see CHONE's comments above)."

With all due respect, this is an inherently fallacious comparison as you are comparing a single prediction formula with "all" subjective predictions. It depends upon whose subjectivity we're talking about, and how informed it is by various factors: statistical analysis, player history and knowledge, understanding of trends, etc, not to mention pure and simple gut intuition.

"That said, in a best case scenario, with everything breaking right for the Angels, Jonathan could be correct. But that chance doesn’t make his prediction a good one; it makes it a best case scenario."

Actually, I wouldn't call my predictions best-case scenarios at all, but "mildly optimistic"--that is, if everyone is basically healthy and performs at a level commensurate with recent past performance, although with no major improvement either.

A best-case scenario would have both Kazmir and Santana returning to their pre-2009 injury form and building on it, with ERAs in the low 3.00s. It would have Saunders pitching liked he did for 2008 and most of 2009, with an ERA in the mid 3.00s; it would have Weaver continue to improve; and it would have Pineiro perform similarly to 2009.

My point being that the PECOTA projections are, if not absolute worst-case scenarios, than very pessimistic (the same with CHONE).

Here are PECOTA and CHONE:
Kazmir 4.32/4.23
Santana 4.60/4.47
Saunders 4.44/4.57
Weaver 4.06/4.10
Pineiro 4.29/4.45

Here are what I'd call a threefold eyeball projection, with high (reasonable best-case scenario), moderate (probable, given general health), and low (reasonable worst-case scenario).

Kazmir 3.30/3.70/4.80
Santana 3.00/3.50/5.00
Saunders 3.50/4.00/4.60
Weaver 3.40/3.70/4.10
Pineiro 3.70/4.20/.4.70

In all cases both PECOTA and CHONE are between my moderate and low projections, and also represent decline for each player. Should I say that again? PECOTA and CHONE represent a decline from peak level performance for every player; all pitchers are within their "sweet spot" range of peak performance and, when not injured, have performed significantly better than those predictions. Maybe some will decline or have bad years, but why would they ALL?

You might say that Kazmir's predicted ERA is lower than his 2009 ERA, but what about injury? Is his 4.89 ERA indicative of his true ability, especially when it is such an outlier from previous performance?

Or what about Joe Saunders? What the formulas see is a pitcher who has had ERAs of 4.71, 4.44, 3.41, and 4.60 the last four years, with an obvious outlier and strikeout and walk rates that match the higher ERAs. Before 2009 I was the first to say that his 2008 was a fluke, but then he continued where he left off until he got injured and tried to pitch through it; after he came back from the DL, he was better than ever.

What CHONE and PECOTA don't see, as far as I can tell, is that after Saunders took a big step forward in 2008 with a 3.41 ERA, he surprised skeptics (such as myself) with a 3.66 ERA through his first 15 starts in 2009, then went through a rough patch where he gave up 41 ER in his next 8 starts to see his ERA rise to 5.33, then went on the DL with an injury that he had been seemingly denying or even hiding. When he came back a few weeks later, rested and well, he pitched as well as he has ever done. Now look at these ERAs:

2008 31 starts: 3.41 ERA
First 15 starts: 3.66 ERA
Next 8 starts (injured?): 9.63 ERA
Final 8 starts: 2.55 ERA

So which is the outlier? Those 8 starts in the middle of 2009 or the 54 before and after? Is it "best-case scenario" to project that Saunders will post a 4.00 ERA in 2010? Based on those numbers, I might be being a bit conservative.

Again, my projections are based upon a fair amount of player knowledge, of their age and history and ability, as well as "intangibles." I know more than PECOTA or CHONE about these factors; what I can't do is create mathematical parabolas. But if those mathematical parabolas don't see, for example, the pattern in Joe Saunders' 2009 season, how accurate can they be?

Jonathan,

I dodn't have the time or the energy to go line by line through your post, but really this stands as an almost perfect example of why the better computer based systems do a bteer job of player projection than the typical fan who is just basing their projections upon "player knowledge, of their age and history and ability, as well as "intangibles." First of all, except possibly for intangibles, which most of the time don't amount to much, you don't necessarily know more about any of those things than PECOTA or CHONE. In fact, you probably know less. That's not a knock on you, but these systems take all that into account, and to the extent that those things can be quantified (most of them can be), they know EVERYTHING about those players, at least as much as the most zealous fan.

But even if you did, you're leaving a few things out of the equation. Mainly, this: what those systems know VERY well are career pattens, which matter a LOT in statistical projections.

Just to highlight one of several areas where you make a mistake in that regard, you state "PECOTA and CHONE represent a decline from peak level performance for every player." Which is true. It also represents reality - you seem to think that pitchers reach a peak and then either stay there or keep improving. And sometimes they do. But on average, there is a reversion to the mean. That is, a pitcher who, say, has an ERA of 4.00 one year, and 3.00 the next, is HIGHLY likely to decline in the year after that. Not all the way to 4.00, but part way there.

That's only one example of where you go wrong; there are others which I don't have time to address. The power of CHONE and PECOTA is this: many years of data regarding such patterns. As a result, they consistently outperform even well done subjective projections.

To be clear, I'm not saying that you do a bad job of subjective analysis. You put a lot of thought into it, and some of your points are good. But even the best such analysis will fall short, primarily because of lack of knowledge of performance patterns.

I think we're going to have to agree to disagree on this, LarryM, as my experience has been that eyeball projections can be just as accurate, if not moreso (by and large, mine for the Angels were much more accurate than PECOTA for 2009). Or to put it another way, bad eyeball projections (e.g. a fan's wishful fantasies) are worse than computer projections, but good eyeball projections (that take into account as many elements as possible) are better than computer projections.

I also disagree that PECOTA et al take into account the inner workings of a given year, as in my Saunders example. This is a major point that you didn't address.

nd I'm sorry, but saying I'm wrong in numerous areas doesn't really mean anything without further discussion. As the saying goes, "Show, don't tell." I understand time restraints, but I don't really get "You're wrong, but I don't have time to show you how." Why even bother?

And yes, I agree with you that analyzing career patterns is important, but only to an extent. I think in general the exception is the rule; you have to look at players on a case-by-case basis rather than relying too much on generalized patterns. This applies to "reversion to the mean" as well. Mathematical formulas will only get you so far; we have to remember that there are actual human beings creating those stats.