The Freakonomics Fiasco in Perspective

by Steve Sailer

Part 1: I'm Shocked, Shocked to See This ... 

The most celebrated nonfiction book of the year is Freakonomics: A Rogue Economist Explores the Hidden Side of Everything by U. of Chicago superstar economist Steven D. Levitt and journalist Stephen J. Dubner. The most admired aspect of the book has been Levitt's theory that legalizing abortion cut the crime rate, which became Instant Conventional Wisdom. Now, it turns out, according to two economists at the Boston Fed who have checked Levitt's calculations in detail, that the abortion-cut-crime theory rested upon two mistakes Levitt made. So far, Levitt admits to making one error, saying it "is personally quite embarrassing."


Ever since my 1999 debate with Levitt in Slate.com, Levitt's fans have been telling me that my simpleminded little graphs and ratios of national-level crime trends showing, for example, that the teen homicide rate tripled in the first cohort born after Roe v. Wade couldn't possibly be right because Levitt's econometric state-level analysis was so much more gloriously, glamorously, incomprehensibly complicated than mine, and Occam's Butterknife says that the guy with the most convoluted argument wins.


This fiasco reveals much about what's wrong with public policy discourse in modern America. Fifteen minutes of Googling would have shown book reviewers of Freakonomics that the abortion-cut-crime theory hadn't come close to meeting the burden of proof, but, instead, much of America's intellectual elite fell head over heels for this theory. Being largely innumerate and unenterprising, the punditariat is unable or unwilling to apply simple reality checks to complex models. It's easier to simply engage in intellectual hero-worship and take a guru figure like Levitt on faith. A few book reviewers, like James Q. Wilson (America's leading expert on crime for several decades), expressed deep skepticism, but most were negligent.

Now, two economists have redone Levitt's work and found two fatal flaws in it. The Economist has a good summary here:

Oops-onomics
Dec 1st 2005
Did Steven Levitt, author of “Freakonomics”, get his most notorious paper wrong?

And the Wall Street Journal reports:

"'Freakonomics' Abortion Research Is Faulted by a Pair of Economists
By JON E. HILSENRATH
Staff Reporter of THE WALL STREET JOURNAL
November 28, 2005; Page A2

" Prepare to be second-guessed.

" That would have been useful advice for Steven Levitt, the University of Chicago economist and author of the smash-hit book "Freakonomics," which uses statistics to explore the hidden truths of everything from corruption in sumo wrestling to the dangers of owning a swimming pool.

" The book's neon-orange cover title advises readers to "prepare to be dazzled," and its sales have lived up to the hype. A million copies of the book are in print. The book, which was written with New York Times writer Stephen Dubner, has been on the New York Times best-seller list for 31 weeks and is atop The Wall Street Journal's list of bestsellers in the business category.

" But now economists at the Federal Reserve Bank of Boston are taking aim at the statistics behind one of Mr. Levitt's most controversial chapters. Mr. Levitt asserts there is a link between the legalization of abortion in the early 1970s and the drop in crime rates in the 1990s. Christopher Foote, a senior economist at the Boston Fed, and Christopher Goetz, a research assistant, say the research behind that conclusion is faulty.

" Long before he became a best-selling author, Mr. Levitt, 38 years old, had established a reputation among economists as a careful researcher who produced first-rate statistical studies on surprising subjects. In 2003, the American Economic Association named him the nation's best economist under 40, one of the most prestigious distinctions in the field. His abortion research was published in 2001 in the Quarterly Journal of Economics, an academic journal. (He was the subject of a page-one Wall Street Journal story1 in the same year.)

" The "Freakonomics" chapter on abortion grew out of statistical studies Mr. Levitt and a co-author, Yale Law School Prof. John Donohue, conducted on the subject. The theory: Unwanted children are more likely to become troubled adolescents, prone to crime and drug use, than are wanted children. When abortion was legalized in the 1970s, a whole generation of unwanted births were averted, leading to a drop in crime nearly two decades later when this phantom generation would have come of age.

"The Boston Fed's Mr. Foote says he spotted a missing formula in the programming of Mr. Levitt's original research. He argues the programming oversight made it difficult to pick up other factors that might have influenced crime rates during the 1980s and 1990s, like the crack wave that waxed and waned during that period. He also argues that in producing the research, Mr. Levitt should have counted arrests on a per-capita basis. Instead, he counted overall arrests. After he adjusted for both factors, Mr. Foote says, the abortion effect disappeared. [Emphasis mine.]

" "There are no statistical grounds for believing that the hypothetical youths who were aborted as fetuses would have been more likely to commit crimes had they reached maturity than the actual youths who developed from fetuses and carried to term," the authors assert in the report. Their work doesn't represent an official view of the Fed.

" Mr. Foote, 40, taught in Harvard's economics department between 1996 and 2002; served stints as an economist on the Council of Economic Advisers in 1994, 1995, 2002 and 2003; and served as an economic adviser to the Coalition Provisional Authority in Baghdad, Iraq, in 2003 and 2004.

" Mr. Levitt counters that Mr. Foote is looking only at a narrow subset of his overall work on abortion and crime, so his results are of limited value, and not grounds for dismissing the whole theory. He acknowledges the programming error, but says taken by itself, that error doesn't put much of a dent in his work. (Mr. Foote's result depends on changing that formula and on the adjustment for per-capita arrests.) Moreover, Mr. Levitt says the abortion theory has held up when examined in other countries, like Canada and Australia, and when applied to other subjects, like drug use.

" "Does this change my mind on the issue? Absolutely not," Mr. Levitt says. [More]

Levitt and John J. Donohue put together their abortion-cut-crime theory in a quick and dirty fashion in late 1998. There's nothing wrong with that -- that's an inevitable aspect of hypothesis-generation. Unfortunately, when their draft paper leaked to the Chicago Tribune in August 1999, they hadn't yet done the needed reality checks on their idea. 

For example, the peak years for serious violent crime by 12-17 year olds, as reported in the FBI's authoritative annual survey of crime victims were 1993 and 1994, or a couple of decades after Roe v. Wade made abortion legal nationally. 

Their key assumption about how humans behave was that legalizing abortion increased the "wantedness" of babies who were actually born, yet one obvious test was whether Roe v. Wade had driven down the illegitimacy rate. As it turns out, it definitely had not. Fathers, at least, were certainly not "wanting" babies more after Roe.

The most striking fact about legalized abortion, but also the least discussed, is its sizable pointlessness. Legalized abortion turned out to be a lot like Homer Simpson's toast: "To alcohol! The cause of, and solution for, all of life's problems." 

Legal abortion is a major cause of what it was supposed to cure -- unwanted pregnancies.  Levitt himself notes that following Roe, "Conceptions rose by nearly 30 percent, but births actually fell by 6 percent …" So for every six fetuses aborted in the 1970s, five would never have been conceived except for Roe! This ratio makes a sick joke out of Levitt’s assumption that legalization made a significant difference in how "wanted" children were. Indeed, perhaps the increase in the number of women who got pregnant figuring they would get an abortion, but then were too drunk or drugged or distracted to get to the clinic has meant that the average quality of the upbringing of surviving babies has declined.

My guess would be that the existence legal abortion made condoms unfashionable in the 1970s. Jonathan Klick's study suggested that legalized abortion caused a sizable increase in infections with sexually transmitted diseases, which supports that hypothesis.

In our August 1999 debate in Slate, I pointed out to Levitt that the national-level homicide data easily available on federal government websites showed that his theory had radically failed the test of history: the first cohort born after the legalization of abortion had a homicide rate as 14-17 year olds triple that of the last cohort born before legalization.

Rather than expressing doubts about his signature theory, however, Levitt dug in his heels in and relied on his extremely complicated state-level analyses to try to intimidate readers into ignoring my easy-to-understand national-level analyses about whether he'd come anywhere near meeting the burden of proof.

Hey, it worked. He's now rich and famous.

I told Levitt last month during the Bill Bennett Brouhaha, in which the former Education Secretary was widely denounced for making a reductio ad absurdum argument based on the racial aspect of Levitt's theory, that he should just walk away now from his most famous theory -- just admit that it's too hard to tell what actually happened. Levitt's now a celebrity so he hardly needs his trademark theory anymore to go on being a celebrity (i.e., famous for being famous). Otherwise, someday, some little-known economist was going to make his reputation by taking the Freakonomist down. Well, Levitt's nemesis has arrived.


A reader writes:

Will this really matter? I guess I have my doubts. We have moved into an era when facts matter less than myths.

Indeed. Virtually nobody will admit they were wrong about this. Way too many important people have too much invested in Levitt's celebrity. This is a fiasco for the economics profession -- the most famous young economist's most famous theory has been exposed after six years of adulation as based on incompetence. I wonder how many economics professors have book proposals in right now for that next bestseller "Berserkonomics"? (By the way, Levitt and Dubner are working on a sequel with a title that reflects their characteristic elegant taste: Superfreakonomics.)

In the general media as well, too many influential people publicly endorsed the theory when a small amount of due diligence with Google would have shown them it was deeply dubious.

And too many people want his abortion-cut-crime theory to be true for personal or political reasons. I've noticed, for example, that in online discussions, pro-lifers tend to want Levitt's theory to be true. They appear to want to be able to boast, "Even though legal abortion reduces the likelihood of me being a victim of crime, I'm still against it. That's how idealistic I am."

The notion that somebody would want to know what the truth is, rather than just to find a talking point for their pre-existing policy prejudice is alien to American thinking today.

Here's the abstract of Foote and Goetz's paper:

Testing Economic Hypotheses with State-Level Data: A Comment on Donohue and Levitt (2001) [PDF - full paper]

Working Paper 05-15
by Christopher L. Foote and Christopher F. Goetz

State-level data are often used in the empirical research of both macroeconomists and microeconomists. Using data that follows states over time allows economists to hold constant a host of potentially confounding factors that might contaminate an assignment of cause and effect. A good example is a fascinating paper by Donohue and Levitt (2001, henceforth DL), which purports to show that hypothetical individuals resulting from aborted fetuses, had they been born and developed into youths, would have been more likely to commit crimes than youths resulting from fetuses carried to term. We revisit that paper, showing that the actual implementation of DL’s statistical test in their paper differed from what was described. (Specifically, controls for state-year effects were left out of their regression model.) We show that when DL’s key test is run as described and augmented with state-level population data, evidence for higher per capita criminal propensities among the youths who would have developed, had they not been aborted as fetuses, vanishes. Two lessons for empirical researchers are, first, that controls may impact results in ways that are hard to predict, and second, that these controls are probably not powerful enough to compensate for the omission of a key variable in the regression model. (Data and programs to support this comment are available on the web site of the Federal Reserve Bank of Boston.)


Levitt's reply on his Freakonomics blog is here.


All my posting on this issue are at http://www.iSteve.com/abortion.htm

***Permalink***


Part 2: Levitt's response to the Freakonomics abortion-cut-crime theory fiasco

Levitt blogs:

Everything in Freakonomics is wrong!

Or at least that is the impression you might get if you read this article in today’s Wall Street Journal.

I will post a longer blog entry once I have had time to fully digest the working paper by Foote and Goetz which is the basis for the article.

For now, I will say just a few things:

1) It is not at all clear from the WSJ article is that Foote and Goetz are talking about only one of the five different pieces of evidence we put forth in our paper. They have no criticisms of the other four approaches, all of which point to the same conclusion.

2) There was a coding error that led the final table of my paper with John Donohue on legalized abortion to have specifications that did not match what we said we did in the text. (We’re still trying to figure out where we went wrong on this.) This is personally quite embarrassing because I pride myself on being careful with data. Still, that embarrassment aside, when you run the specifications we meant to run, you still find big, negative effects of abortion on arrests (although smaller in magnitude than what we report). The good news is that the story we put forth in the paper is not materially changed by the coding error.

3) Only when you make other changes to the specification that Foote and Goetz think are appropriate, do the results weaken further and in some cases disappear. The part of the paper that Foote and Goetz focus on is one that is incredibly demanding of the data. For those of you who are technically minded, our results survive if you include state*age interactions, year*age interactions, and state*year interactions. (We can include all these interactions because we have arrest data by state and single year of age.) Given how imperfect the abortion data are, I think most economists would be shocked that our results stand up to removing all of this variation, not that when you go even further in terms of demands on the data things get very weak.

Again, as I said, I will post again on this subject once I have had a chance to carefully study the details of what they have done, and after I have been able to go back to the raw data and understand why the results change when one does what Foote and Goetz do.

5 COMMENTS » Posted by Steven D. Levitt @ 2:46 pm on Monday, November 28, 2005 in General

In contrast, economist John R. Lott, a longtime critic of Levitt's theory who came in for a half page of ad hominem abuse in Freakonomics, is feeling better than Levitt is today. He blogged:

Christopher L. Foote and Christopher F. Goetz's paper can be found here. Personally, I think calling this a "programming oversight" is being much too nice. More importantly, everyone who works with panel data knows that you use fixed effects.

My own work concentrated on murder rates, but I also included fixed effects. Donohue and Levitt never provided us with all their data or their regressions and would never answer any questions that we had so I just assumed that they had included fixed effects from the beginning. It would have been nice if they had provided us with this same information years ago.

Financial economist and blogger Mahalanobis (Michael Stastny) makes first a technical point about Levitt's reliance on complexity of analysis a then a substantive point about Levitt's understanding of human behavior:

Levitt's response is on his website (see here) where he notes

The part of the paper that Foote and Goetz focus on is one that is incredibly demanding of the data. For those of you who are technically minded, our results survive if you include state*age interactions, year*age interactions, and state*year interactions.

3 interaction variables are necessary to get the right sign and significance? I think that is very technically demanding. In my experience, interaction variables are kitchen sink type regressors that induce severe multicollinearity and give spurious results. It's like an economist saying his results only appear after doing 3-stage least squares. I have to think something's not really there if you can't normalize the data somehow and show in a simple graph that the pattern is there (in this case, say, by showing the change in arrest rates for abortion and non-abortion states for the relevant age cohort).

I'm partial to the opposite theory, that abortion would, if anything, increase the proportion of evil-doers: abortion is more common among forward-thinking moms who would be good moms, less common among bad moms who view life as a series of random events that happen to them.

The reason that in Levitt's theory of American crime trends, Levitt cites only foreign studies claiming that women who have abortions would make less organized and effective mothers than the ones who went ahead and had their children is because the American studies of who gets an abortion came to the opposite conclusion.

This undermines Levitt's only argument these days about how abortion would cut crime (now that Levitt has hushed up his earlier racial eugenic/eucultural argument that because more blacks get abortions and more blacks commit murders, more abortions should mean fewer murders). These Americans studies were pointed out to Levitt by CCNY economist Ted Joyce in his response to Levitt & Donohue in the Journal of Human Resources, which was entitled "Did Legalized Abortion Lower Crime?" Joyce summed up two reason why Levitt's theory didn't work. The second was:

"Second, analysts, I being one, have tended to overestimate the selection effects associated with abortion. A careful examination of studies of pregnancy resolution reveals that women who abort are at lower risk of having children with criminal propensities than women of similar age, race and marital status who instead carried to term. For instance, in an early study of teens in Ventura County, California between 1972 and 1974, researchers demonstrated that pregnant teens with better grades, more completed schooling, and not on public assistance were much more likely to abort than their poorer, less academically oriented counterparts (Leibowitz, Eisen, and Chow 1986).

"Studies based on data from the National Health and Social Life Survey (NHSLS) and the National Longitudinal Survey of Youth (NLSY) make the same point (Michael 2000; Hotz, McElroy, and Sanders 1999). Indeed, Hotz, McElroy, and Sanders (1999) found that teens who abort are similar along observed characteristics to teens that were never pregnant, both of whom differ significantly from pregnant teens that spontaneously abort or carry to term.

"Nor is favorable selection limited to teens. Unmarried women that abort have more completed schooling and higher AFQT [the military's IQ test for applicants for enlistment] scores than their counterparts that carry the pregnancy to term (Powell-Griner and Trent 1987; Currie, Nixon, and Cole 1995).

"In sum, legalized abortion has improved the lives of many women by allowing them to avoid an unwanted birth. I found little evidence to suggest, however, that the legalization of abortion had an appreciable effect on the criminality of subsequent cohorts."


My earlier response to the latest Freakonomics fiasco is here.


All my blog postings on the controversy can be found at http://www.iSteve.com/abortion.htm

***Permalink***

Part 3: Abortion and crime: So, Levitt was wrong. But, what actually happened? 

Now that Freakonomics author Steven D. Levitt's mishandling of his abortion-crime data has been exposed by economist Christopher Foote, I'd like to review what actually happened in American over those decades.

As I tried to explain to Dr. Levitt when we debated in Slate in 1999, what happened, simplifying greatly, was that the vast youth crack crime wave first emerged in the later 1980s in the socially liberal states where legal abortion also had taken off first about 17 years earlier, most notably New York and California, which legalized abortion in 1970, three years before Roe v. Wade.

In other words, contrary to Levitt's theory, there was at the state level, a positive correlation (when appropriately weighted by population of state), between the legal abortion rate in the early 1970s and the teen homicide offending rate in the late 1980s and early 1990s among those youths born after legalization. Unfortunately, Dr. Levitt initially only looked at crime rates for the years 1985 and 1997 (and only looked at the uselessly crude age groups of over and under 25), so he completely missed how his theory had catastrophically failed its most obvious historical test.

Second, and also contrary to Levitt's theory, this vast youth murder wave took off first specifically in the demographic group that had the highest legal abortion rate: urban blacks. The non-white abortion rate peaked in 1977, well before the peak of the white abortion rate. The peak years for homicide among 14-17 year old black males were 1993 and 1994 -- i.e., the cohort born at the peak of the black usage of legal abortion in 1977. As Donohue and Levitt wrote in 2001, under their theory, the opposite was supposed to happen:

"Fertility declines [following the legalization of abortion] for black women are three times greater than for whites (12 percent compared to 4 percent). Given that homicide rates of black youths are roughly nine times higher than those of white youths, racial differences in the fertility effects of abortion are likely to translate into greater homicide reductions."

When William Bennett was denounced for mentioning on the radio the racial logic of Levitt's theory, Levitt tried to give the impression to journalists that race had never played an important role in his theory. Indeed, in Slate.com, William Saletan blamed me for giving Bennett the impression that Levitt was thinking about race. As Ross Douthat rebutted, the racial element in Levitt's theory was prominently played up in the national media even before my debate with Levitt.

But, as I tried to explain to Levitt in 1999, his racial eugenic /eucultural logic hadn't worked. Instead, among black males born in the late 1970s, their murder rate as 14-17 year olds was four times higher than among black males born in the late 1960s, before the legalization of abortion. The black to white teen murder rate ratio almost doubled after legalization. So, the Levitt-Donohue theory failed its first two historical tests in a disastrous fashion.

Then, two things happened historically that helped create the state-level negative correlation (presumably, assuming Foote's new technical critique doesn't completely eliminate it) between later 1970s abortion rates and later 1990s crime rates that Levitt and Donohue have emphasized so repeatedly, while trying to cover up the earlier negative correlation. (They imply that the longer the time lag between presumed cause and effect, the more trust we should put in it!)

1. From NY and CA, crack spread to more socially conservative states, where the abortion rate had also gone up later. So crime was higher in the mid to late 1990s in socially conservative states where abortion rates didn't go up until the late 1970s or early 1980s.

2. And, the crack wave burned out first in the places where it started first, most famously New York City.

We've all heard a million arguments about why crime fell in NYC in the 1990s, but an overlooked explanation was brought up by Knight-Ridder reporter Jonathan Tilove recently: there are today in NYC, 36% more black women alive than black men. Nationally, among all races, there are 8% more women than men alive.

Obviously, this gigantic black male shortage in NYC wasn't caused by abortion -- there was virtually no sex selective abortion at the time. No, it was mostly caused by an enormous increase in imprisonment and by the most dangerous black men murdering each other in large quantities in the late 1980s and early 1990s. (AIDS played a role too.) Levitt has never written, as far as I know, about the impact of these "selective post-natal abortions," as it were, on the crime rate, but it's clearly a substantial factor in a number of big cities that were hit hard by crack. (NYC is by no means unique in terms of the current black male shortage.)

Moreover, as I pointed out to Levitt in 1999, and as his deservedly famous chapter in "Freakonomics" on how dealing crack pays so badly confirmed, a lot of the next cohort of urban youths, those born more than a half decade after abortion was legalized in their state, figured out that dealing crack was a stupid career choice. Seeing how their older brothers and cousins were winding up in prisons, wheelchairs, and cemeteries, they became less likely to commit murder. Participating in the crack wars turned out to be, for the vast majority of the gangstas, extremely bad life choices, and it's hardly surprising that the later cohort born in the early 1980s did a better job of figuring this out.

But these anti-crime trends in the 1990s happened first where crack happened first, which tended to also be where legal abortion happened first, thus creating the most likely spurious correlation between legal abortion and the crime decline in the later 1990s that Freakonomics focuses upon.

So, for this controversy, the crucial issue is The Burden of Proof. Dr. Levitt has tried hard to hand the burden of proof off to his skeptics, claiming that he's looked at all other possible causes of the 1990s crime decline, and they aren't adequate to explain it, so abortion must be the cause of the remainder. That's a weak and irresponsible argument.

Of course, in reality, he hasn't looked at all the causes -- for example, I've never seen him take into account "selective post-natal abortions" of the most dangerous gangstas by other gangstas, nor the social learning impact on the next cohort of seeing their older brothers die or go to prison.

But, moreover, there's an old saying that large assertions require large evidence. And Levitt's abortion-cut-crime theory is one of the largest assertions in the social sciences in recent years. Clearly, the burden of proof rests on Dr. Levitt.

There's also an old idea in science called Occam's Razor, which more or less says that scientists should be biased toward simplicity in explanations. Throughout this six year controversy, Dr. Levitt has consistently gone for the most complicated, hard-to-understand, and (as we've seen this week, to Dr. Levitt's embarrassment) hard-to-check-up-on statistical models.

In contrast, he's combined statistical incomprehensibility with the most simple-minded behavioral models -- he has repeatedly assumed, despite all the evidence from American studies cited above, that ghetto women decide whether or not to engage in unprotected sex and whether or not get an abortion or have an illegitimate child for the same reasons that would appeal to highly educated women of his own class. While Levitt's style of thinking about how women respond to legalized abortion has proven highly persuasive to the nonfiction book-purchasing class, it doesn't explain much at all about the behavior of the classes in which potential criminals are typically raised. A reader of mine who was an inner city social worker wrote:

 

Middle class types see poor unwed teenage mothers as Scum of the Earth and a Terrible Social Problem. But poor women don’t see themselves that way. Instead, they think of themselves as human beings facing the age-old challenge of getting along in the world -- and, if they're lucky, passing their genes on to the next generation.

 

Maybe the technical opacity of Dr. Levitt's analysis was necessary -- social phenomena are terribly complicated. But the impact of his behavior on the public and on much of his profession has been to encourage among his numerous fans not a critical engagement with the historical and sociological record, but an attitude of faith, a warm feeling that this really smart guy has Figured It All Out using Really Complicated Statistics and we should just take his word for it.

As a marketing strategy, the oracular approach of "Freakonomics" has been mind-bogglingly successful, but perhaps I may be forgiven for wondering whether it advances the cause of good social science.

All the data cited above can be found documented at http://isteve.com/abortion.htm

***Permalink***

Steve Sailer (www.iSteve.com) is a columnist for VDARE.com and the film critic for The American Conservative.

Subscribe to The American Conservative

Steve Sailer's iSteve.com home page

Steve Sailer's film reviews

Steve Sailer's VDARE articles

email me

 

 

Steve Sailer's iSteve.com homepage 

iSteve film reviews

email me

 

 

 

Steve Sailer's iSteve.com homepage 

iSteve film reviews

email me