Trusting econometrics

One of my profs at Mason told the story of how he'd been offered a new boat if he could get the coefficient in a regression to be below two - which would have allowed a merger to proceed. He turned it down, but not everybody does. Unfortunately, in a whole pile of empirical work, you either have to really trust the guy doing the study, or make sure that his data's available for anybody to run robustness checks, or check that a bunch of people have found kinda the same thing. Degrees of freedom available in setting the specifications can sometimes let you pick your conclusion, like getting a coefficient that hits the right parameter value or the right t-stat.

David Levy and Susan Feigenbaum worried a lot about this in "The technological obsolescence of Scientific Fraud". Where investigators have preferences over outcomes, it's possible to achieve those outcomes through appropriate use of identifying restrictions or method - especially since there are lots of line calls in which techniques to use in different cases. They note that outright fraud makes results non-replicable while biased research winds up instead being fragile - the relationships break down when people change the set of covariates, or the time period, or the technique.

Note that none of this has to come through financial corruption either: simple publish-or-perish incentives are enough where journals are more interested in findings of significant than of insignificant results; DeLong and Lang jumped up and down about this twenty years ago. Ed Leamer made similar points even earlier (recent podcast). And then there's all the work by McCloskey.

Thomas Lumley today points to a nice piece in Psychological Science demonstrating the point.

In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings ( ≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

Degrees of freedom available to the researcher make it "unacceptably easy to publish "statistically significant" evidence consistent with any hypothesis." They demonstrate it by proving statistically that hearing "When I'm Sixty-Four" rather than a control song made people a year-and-a-half younger.

The lesson isn't radical skepticism of all statistical results, but rather a caution against overreliance on any one finding and an argument for discounting findings coming from folks whose work proves to be fragile.

Beauty in Economics

Dan Hamermesh writes:

While we don’t have studies of economists’ beauty and their salaries, we do know something about the impact of their looks on non-monetary outcomes. In a profession that pays well, but that does not offer immensely higher monetary rewards to the top people, the distinctions offered by various honors become important. One such measure of distinction is the esteem in which they are held by their colleagues.

In one study I examined how success in competitive elections to office in the American Economic Association, the leading professional organization in the field, is affected by the economists’ looks. Each voter (member of the association) receives pictures of the candidates along with the ballot so that the candidates’ looks confront you when you cast your vote.

Clearly, in such elections someone will win. So the relevant consideration is not the looks of the candidates alone, but instead how their looks compare to those of other candidates. The results show that moving from the 84th to the 16th percentile of looks lowers a candidate’s chance of winning the election — of obtaining this honor — from 56 percent to 44 percent. This effect adjusts for measures of the candidates’ scholarly productivity, their gender, and other characteristics. It suggests that even the choices of economists, many of whom like to think that they and their fellows are among the most rational people in the world, are affected by looks.

Good thing I'm not planning on running for any kind of office....

Previously...

Rationality and economists

Andrew Gelman takes a swipe at economists. I think he's got things wrong. Let's work through it.

First, Gelman argues economists are inconsistent in arguing for consumer rationality while arguing that people need economists to help them overcome their irrationalities, largely about government policies. But these arguments are hardly inconsistent. In environments where individuals face real costs of being wrong or irrational, they consume little irrationality. At the voting booth, their likelihood of decisiveness is sufficiently low that they can indulge biased but comforting beliefs about the true state of the world. That's Caplan's rational irrationality model; I find it rather convincing.

Here's how Gelman thinks we square the circle:

OK, now to return to the puzzle that got us started. How is it that economics-writers such as Levitt are so comfortable flipping back and forth between argument 1 (people are rational) and argument 2 (economists are rational, most people are not)?

The key, I believe, is that “rationality” is a good thing. We all like to associate with good things, right? Argument 1 has a populist feel (people are rational!) and argument 2 has an elitist feel (economists are special!). But both are ways of associating oneself with rationality. It’s almost like the important thing is to be in the same room with rationality; it hardly matters whether you yourself are the exemplar of rationality, or whether you’re celebrating the rationality of others.

I think it's rather that economists recognize that there can be a rather large disconnection between policies that are politically popular and ones that would maximize a reasonable conception of a social welfare function. You can get it through the combination of rational ignorance and logic of collective action or other public choice problems; you can also get it through Caplan's rational irrationality.

Now, I know Gelman rejects that the expected instrumental benefits of voting are low; he says that an altruist weighs the benefits to everybody else of his voting to make things better and consequently voting passes a rational instrumental cost benefit analysis. But surely if your vote is the decisive one making everybody better off as you see it, it's also the one that makes half the voting population worse off as they see it. And so Gelman's argument fails unless the voter can place himself in an epistemically privileged position: he has to know that he's making the voters who disagree with him better off. And I just can't see how that happens. That half the population disagrees with you at the ballot box ought to make you more uncertain about the benefits of your preferred policy unless you truly have expert knowledge.

Fortunately, we economists often do have expert knowledge about economic policy. Well, maybe not about macro beyond a short list of "don't do these twelve things lest you completely ruin everything". But in micro and applied price theory, we're decent.

I rather liked Gelman's PS:

P.S. Statisticians are special because, deep in our bones, we know about uncertainty. Economists know about incentives, physicists know about reality, movers can fit big things in the elevator on the first try, evolutionary psychologists know how to get their names in the newspaper, lawyers know you should never never never talk to the cops, and statisticians know about uncertainty. Of that, I’m sure.