The value of Non-randomized policy experiments
A carefully designed and well-executed randomized control trial minimizes the selection problem and greatly helps researchers in their search of causality. But this does not mean that the use of a randomized experiment is the only way to run a successful policy experiment.
Sometimes researchers face the following trade-off. Running a randomized experiment makes it straightforward to evaluate the causal effect of interest, but researchers may want to forgo randomization and go for a non-randomized experiment in order to provide a larger “dose” of treatment in the experiment.
Boston had a serious problem with youth violence and homicide in the early 1990s, as the number of young homicide victims (age 24 or less) increased by 330 percent between 1987 and 1990 (from 22 to 73). Alarmed by this, numerous local, state, and federal law enforcement agencies got together and implemented a joint policy intervention called the Operation Ceasefire.
This intervention aimed to reduce youth violence and homicide by taking the following policy measures. First, the intervention disrupted the chain of illegal gun trafficking in Boston by strengthening monitoring and enforcement efforts on traffickers of high-risk guns (the types of guns used often by gang members) and crime gun traces. Second, members of the participating agencies, including police officers, probation and parole officers, and church and community leaders, directly reached out to local gang members and sent out the following message: A full-scale policy intervention aimed at reducing youth violence is in place and it would seek all possible legal measures to punish gang members who commit violence.
This threat was made credible thanks to the large number of participating agencies, including the Boston Police Department, the Massachusetts Department of Probation and Parole, the Suffolk County District Attorney, the U.S. District Attorney, the Bureau of Alcohol, Tobacco, and Firearms, the Massachusetts Department of Youth Services, Boston School Police, and gang outreach and case workers from the Boston Community Centers. Combined efforts of all these agencies could easily lead to intense police and prosecutorial attention on daily gang activities, and result in more severe bail, plea bargaining, probation, and parole terms on gang members who commit violence.
Importantly, the intervention also made it clear that, if the gangs committed other, non-violent crimes, their crimes would be processed as usual. Only those crimes that involve violence will turn on “the switch” and significantly disrupt daily activities of the gangs. This way, the intervention focused on a more modest but realistic goal of curbing the level of violence committed by local gangs, instead of trying to eliminate all gang activities and crimes.
The evaluation of the Operation Ceasefire by Anthony Braga, David Kennedy, Elin Waring, and Anne Piehl shows that the number of youth homicides, gun assaults, and shots fired all significantly fell after the introduction of the Operation in 1996. However, the Operation was not a randomized control trial, in which researchers randomly assigned some gangs to more severe punishment and others to the usual level of punishment. The Operation applied the same “treatment” on all local gangs in Boston instead. How can we tell then, how much of the observed decline in youth violence was caused by the Operation?
Had this been a random experiment, the evaluation of the causal effect of the intervention on crime would have been straightforward. We would simply have to compare the rates of violent crimes committed by the gang members who received the “threat” of intense monitoring, enforcement, and punishment and those who did not.
But running such a randomized experiment may not make much sense here. Imagine the agencies randomly choosing some gangs and telling them they would be subject to much more intense monitoring, enforcement, and punishment upon committing violence, while the other gangs would not face such consequences. Perhaps the “uniform” treatment in the Operation was what made the threat of severe punishment all the more credible, and made it clear that youth violence in Boston was no longer tolerated.
- Braga, Anthony A., David M. Kennedy, Elin J. Waring, and Anne Morrison Piehl. “Problem-oriented Policing, Deterrence, and Youth Violence: An Evaluation of Boston’s Operation Ceasefire.” Journal of Research in Crime and Delinquency 38 (2001): 195-225.
© Songman Kang, Hanyang University