Is Canseco Robust?

Jose Canseco was a muscle bound slugging outfielder who currently is the self proclaimed poster boy of steroid use. His recent book Juiced describes how he helped several players access and use steroids throughout his career.

A popular belief is that steroids are the cause of the increase in home runs baseball saw during the late 1990s and early 2000s. If steroids increased home runs and Canseco helped players get steroids then we would expect players who played with Canseco to hit more home runs once he became their teammate. A study by two economists Eric Gould and Todd Kaplan in their analysis of Canseco's teammates found this very result. (see this slate article for a summary)

On reading the paper I felt the results seemed too good to be true. I'm not accusing the authors of faking the data, but I was curious how the results would hold up if they changed a few things that didn't seem typical in the way player performance is typically estimated. It looks like I wasn't alone JC Bradbury examined how the results would hold up if some things in the way the Canseco impact on his teammates' home runs were changed (link). In short Bradbury used only the years Canseco played 1985-2001 instead of all the years Canseco's teammates played 1970-2003. Bradbury made some other changes that I tend to agree with. When he makes these changes, Canseco no longer has a positive impact on his teammates home runs, if anything it's negative.

It's not clear if the Canseco effect via steroids exists or not, but Bradbury's analysis caste doubt on it. I think finding the Canseco effect doesn't exist could be almost as interesting.

The Kaplan and Gould paper was discussed in both slate and Freakonomics. On the one hand it shows economists can get a little too excited about a paper that has not been vetted. On the other hand within a few days or less, a new analysis was posted and our understanding was increased.