How Do Experts Make Their Predictions?
Yesterday, we at Baseball Analysts revealed our "expert" predictions. Now there's so much luck that goes into winning baseball games, that making predictions on Opening Day is basically a fool's game. Still, everyone loves to make them. It was my pleasure this season to be able to publish my picks for the first time, both here, and over at SI.com.
However, I'll have to admit that I was a little bit conflicted when asked to provide my pre-season selections. What, exactly, is the goal in making the picks? Do I merely choose the teams who I think have the best chance to win each division, league, World Series, etc? Or do I choose teams that I think are underrated by others, but might not necessarily be the teams I would stake my life upon? What does the average sportswriter do? Do they make selections just to make a statement or show how smart they are or do they stick to those that they really think are the best clubs?
Putting it into a more economic context, what is the payoff function which most sportswriters use? One possible method is a simple one, where the goal is simply to make as many correct picks as possible. If this is the goal, the best method is to simply pick teams which have the highest probability to win. The problem with this method is that there isn't a whole lot of room for creativity. Think the Marlins are an underrated ballclub, who has a good chance to contend? Perhaps you handicap the NL East as the following: PHI 33%, ATL 32%, FLA 30%. That's probably a lot more favorable towards the Marlins than most people would say, and a lot less favorable towards the Phillies. But, under this method of making predictions, you'd have to go with the conventional wisdom pick of the Philadelphia Phillies to win the East, just like the rest of the world.
Another drawback of this method is that when looking at a bunch of "expert" picks, there isn't much to choose from between the experts. Over at SI.com, all 13 experts, including yours truly, picked the Phillies to win the East. There's not much to choose from between the writers, and as a fan, it doesn't tell you much about what the Phillies chances actually are. Assuming writers use this method of making their selections, all it means is that each thinks that the Phillies have the best chance to win the East. Whether that chance is 30%, 50%, 80% or 99% is unknown.
However, there's some evidence that not every expert makes their picks this way. Jim Caple, at ESPN, is picking the Giants over the Twins in the 2010 World Series. SI's Tom Verducci is picking the Twins to win the AL Pennant. SI's Albert Chen has the Rays winning it all. Could these things happen? Sure, all of these teams have the potential to go all the way. However, I can't image that any of these experienced baseball writers would really stake their lives on these picks. More likely, these experts wanted to select teams which were underrated. On the off chance that the Giants do knock off Minnesota in the World Series, Jim Caple will look like a genius. If the Yankees defeat the Phillies, like I picked, I simply look like a purveyor of conventional wisdom. Thus, there may be incentive to make upset picks.
A payoff function that would reflect this thinking would give experts credit in inverse proportion to the number of other people making that same selection. Thus if a 60% of people were picking the Phillies to win the East, then you would get 1.67 points of "credit" (1/.6 = 1.67) if you also picked the Phillies and they actually won. If only 5% of people were picking the Marlins, you would get 20 points of credit (1/.05) for making that correct selection. This would give someone an incentive to pick the Marlins if they thought they were underrated, even if they didn't think they necessarily were the team most likely to win.
As a matter of fact, if all experts used this type of payoff function, the result would be an efficient marketplace, in which the percentage of writers making each pick would correspond to the probabilities that each team would actually win. This would occur, because an equilibrium would be reached where, according to the market, the expected payoff for selecting any of teams would be equal. To maximize points, you'd have to not pick the teams you thought would win, but instead pick the teams you thought were underrated by others. This would truly become efficient if experts could change their picks after viewing what the other experts had chosen, as it would reach an equilibrium where no expert could gain an advantage by changing his or her selection. A made-up example with 10 writers and the probabilities they assign to each team winning the AL East is below:
Writer G, though he thinks the Yankees are better than the Red Sox, thinks more highly of the Red Sox than most other writers, and hence he has incentive to pick the Boston over New York. The same goes with writer C and with writer E concerning the Rays. The result is that none of the writers have an incentive to change their picks, hence it reaches an equilibrium of six writers choosing the Yankees, three writers choosing the Red Sox, and one writer choosing the Rays. This roughly corresponds to the consensus probabilities of each team winning the AL East (it would match-up more exactly if there were more than just ten people making predictions). This is a more interesting outcome than if each writer simply chose the team they thought most likely to win. In that case, all but one would have chosen the Yankees - a far cry from the actual consensus handicapping of the division.
If this were actually done this way, this would actually yield the fan some cool information. Getting the collective probabilities of success from experts would be interesting and useful info. Unfortunately, it doesn't appear that writers make their picks this way either. None of the 49 writers for either ESPN or SI chose either the Indians or the Athletics to win their division. These teams certainly aren't the favorites, but they definitely have a better chance of winning than 1 in 49. If experts made their picks in the above manner, at least a few would have chosen these sleeper teams to win. Likewise, at SI, every writer picked the Phillies to win. However, the Phillies have far from a 100% chance to win the division. Were the payoffs doled out as above, a fair amount of writers (including me) would have changed their picks to the Braves or perhaps even the Marlins or Mets. Hence, it's pretty obvious that experts don't make their picks with this payoff in mind.
So, if experts don't go with either of these payoff functions, then how do they make their selections? Good question. For most, it's probably a combination of the two. I would say most probably pick the team that is most likely to win, however, if it's close they'll likely choose an upset, just to go against the conventional wisdom. At least, that's my hunch. It's pretty tough to tell what's actually going on inside the head of the average sportswriter. For me personally, the one area I went out on a limb was in picking the Tigers to win the AL Central. If my life depended on making the right selection, I might have gone in another direction, but seeing as I thought it was pretty much a 4-team crapshoot, I went with Detroit, a pretty good ballclub that no one else seemed to be picking. In all, the various ways people make selections, and of course, the uncertainty surrounding any Major League season, makes preseason picks fairly useless. Still, I'd love to see how writers would respond to participating in a system like the above, where the payoffs were inversely proportional to the number of experts making that same pick. At least then we could glean some interesting information out of them.