Editor's note: This contribution by the late journalist I.F. Stone first appeared in our January 1975 magazine.
By Jamie Bernstein
Yesterday, researchers from the National Bureau of Economic Research released a new paper looking at the effect of watching the MTV shows 16 and Pregnant and its spin-offs Teen Mom and Teen Mom 2 on teen birth rates. They claim they have found causal evidence to suggest that in areas in which the show is more popular, teen pregnancies are falling at a faster rate than areas in which the show is less popular.
The New York Times yesterday morning reported on this research with a very detailed article. This was followed by a ton of other reports on the study by news outlets, popular blogs and even sites on pop culture. Not soon after that, other outlets like Slate's XX Factor wrote articles refuting the conclusions of the study though they seemed to refute it almost entirely based on anecdotal evidence from watching the shows rather than on the strength or lack thereof of the research. As someone with academic training in doing this type of research, my skeptical alarm bells were going off while reading the NYT article. The type of research they're talking about, looking for the effect of a piece of pop culture on an extremely complex behavior like becoming pregnant, is notoriously difficult to study. There is a reason why there is so much debate over whether shows like 16 and Pregnant raise or reduce teen births, whether violent video games increase crime, or whether sexualized images in the media increase incidences of rape. These types of effects are extremely difficult to study because no pure control groups exist to compare users of different types of media. Of course the media we consume likely has an effect on who we become, but because we all choose what to consume and our choices reflect something about us, it is hard to tease out what differences caused us to choose to watch that tv show or listen to that piece of music and what differences were caused by it. This doesn't mean we can't study these things or that we shouldn't, as long as we understand how difficult it is and interpret results carefully.
Luckily for me, this particular paper is online and free to download so I was able to read through the original paper myself. The researchers Kearney and Levine first compared Nielsen ratings for 16 and Pregnant and its spin-offs Teen Mom and Teen Mom 2 in geographic areas with the decline in teen births in those areas. Although teen birth rates have been dropping all over the US for the last 20 years, the researchers found that teen birth rates fell much quicker in areas where 16 & Pregnant and its spin-offs had more young viewers. It's worth mentioning at this point that this is certainly interesting but not very convincing. It is quite possible that teens worried about pregnancy could be more likely to watch 16 and Pregnant. This would account for the correlations between the popularity of the show and the lower teen birth rates even if watching the show did not have any effect on birth rates. However, the researchers Kearney and Levine thought of this and had a trick up their sleeve to deal with it.
One of the tricks economists use when looking for a causal effects in areas in which we cannot run controlled experiments and therefore must use very uncontrolled real-world data, is to find a plausible "instrumental variable" or IV. We cannot know whether the correlation between watching 16 and Pregnant and lower teen pregnancies is caused by teens who are less likely to get pregnant choosing to watch 16 and Pregnant or by teens being scared away from pregnancy due to watching 16 and Pregnant. The causality could go either way. However, if we found another variable that affected whether teens watched 16 and Pregnant but did not affect teen birth rates directly, we could substitute that variable into our models. If we found it changed teen birth rates but there is no plausible mechanism whereby it could affect teen birth rates other than the fact that it encourages watching of 16 and Pregnant, we would have basically "tricked" our model into telling us how watching 16 and Pregnant affects teenagers.
For those whose heads are hurting a little bit after reading that, perhaps it will help to think of it in a more abstract manner. Let's say I find a correlation between people who do act A and people who do act B. I believe that act A causes act B but so far all I have is a correlation. However, variable C changes A but is completely unrelated to B. So, if C causes A and A really causes B, then I should find a correlation between C and B. However, if C causes A and B causes A, then the changes C makes on A shouldn't affect B and therefore I should not find any correlation between C & B. So, using C we can determine if A is causing B or B is causing A. We can tease out causality from correlation.
In this paper, Kearney and Levine use the differing popularity of MTV prior to the pilot of 16 and Pregnant as an instrumental variable for teen pregnancies. This only works as an IV if 1) the popularity of MTV in the years before 16 and Pregnant was on air does not and can not plausibly be related to teen birth rates and 2) the popularity of MTV in various geographic regions is highly correlated with the popularity of 16 and Pregnant and its spin-offs. If these two conditions are met, then they can check to see whether the popularity of MTV prior to 16 and Pregnant being on air is correlated with teen birth rates after 16 and Pregnant first aired. If it is, this correlation would be due to a causal relationship between watching 16 and Pregnant and teen birth rates. The researchers did just this and indeed found a strong correlation, implying that birth rates drop quicker in areas in which 16 and Pregnant is more popular because more teens are watching the show. Yay Science!
Er well, maybe "yay science?" would be more accurate. The thing is, this is a good paper but I'm still not totally convinced. The problem with instrumental variables is that your claims are only as robust as your IV. I personally am not convinced that MTV viewership prior to 16 and Pregnant is not plausibly correlated with teen birth rates. It makes sense to me that there may be external forces (such as economic class or liberal attitudes) that may both cause teens to be more likely to watch MTV and also make them more likely to use birth control.
Additionally, this study looked at the differences in the decreasing teen birth rates between regions. The paper mentions the South and Midwest as being geographic regions in which 16 and Pregnant, Teen Mom, Teen Mom 2, and MTV in general are most popular. If these areas already have higher teen pregnancy rates starting out, then they have more leeway to decline at a higher rate than areas that are starting out with lower teen birth rates. To put this in more concrete terms: if imaginary state Califoregon has a teen pregnancy rate of 2% in 2009 and a teen pregnancy rate of 1.5% in 2010, it will have decreased by 0.5 percentage points or 25%. If state Alabaxas has a teen pregnancy rate of 10% in 2009 and decreased to 7% in 2010, it will have had a decrease of 3 percentage points (compared to Califoregon's 0.5) and a 30% decrease (compared to Califoregon's 25%). Its rate of decrease was quicker than Califoregon's but it's also plausible that the lower the rate of teen births the more difficult it is to prevent the rest of the teen births. Califoregon already decreased teen births to a very low rate so getting it even lower is more difficult than it is for Alabaxas who is starting from a high rate. In the 16 and Pregnant study, if areas in which MTV is more popular (the South and Midwest) already have higher numbers of teen births, then it may already be decreasing at a faster rate than areas which are starting with a much lower level of pregnant teens. This would cause it to look as if 16 and Pregnant were causing teen pregnancy rates to drop at quicker rates in certain regions when in fact it would not be related.
The researchers also devote a huge portion of their paper to showing that tweets mentioning "birth control" and google searches for "birth control" peaked on the days in which 16 and Pregnant, Teen Mom, and Teen Mom 2 aired. In my opinion this doesn't tell us all that much. Tweets and Google Searches for the term "meth" would also plausibly be popular on the day the Breaking Bad finale aired, but that doesn't mean that people are having informative conversations about meth. Even so, it does create a possible line of evidence, however flimsy, between watching episodes in the 16 and Pregnant franchise and lowering teen birth rates.
Now that I've just hit you with all my personal skepticism about this study, I want to mention that I don't at all think it is a worthless research paper. These things are just really hard to study and there are always going to be flaws when looking at real-world data like this. In fact, even with these flaws, it's a really well-done piece of research and it does seem as if Kearney and Levine may have found some evidence that teens that watch the 16 and Pregnant franchise are less likely to get pregnant. They claim that the show 16 and Pregnant and its spin-offs can explain a full third of the decrease in teen pregnancies in the 18 months after the show first aired. My personal opinion after reading their study is that I'm not convinced that if there is an effect that is even close to that strong, though I think a small effect is possible.
I know some of you just read the word "opinion" and thought "but Jamie, this is science! Your opinion has nothing to do with it." However, when interpreting a model-based study such as this one, the model itself cannot be tested so in order to believe its results you must trust it. It is perfectly valid to have an opinion on the validity of the model and therefore the results. It is this lack of certainty and need for trust that makes a lot of people uncomfortable with social science research. However, social science methods of analysis may not be as concrete as those in the natural sciences where researchers can run controlled experiments, but that doesn't mean they are useless, only that we need to be a little more skeptical when interpreting its results.
My interpretation of this study is that there definitely seems to be some evidence to suggest that teens that watch 16 and Pregnant, Teen Mom, and Teen Mom 2 are moved to take more precautions against teen pregnancy. However, I believe that based on the weaknesses in the researchers instrumental variable that it probably doesn't account for the full 6% decrease in teen births that the researchers claim.