Moving to Opportunity
Learning from one of the most comprehensive evaluations of all time
The Moving to Opportunity (MTO) program was an innovative housing mobility program initiated by the U.S. Department of Housing and Urban Development (HUD) back in 1994. The idea was to give low-income families living in highly disadvantaged urban neighborhoods a chance to relocate to areas with less poverty. Over 4,600 families participated. They were randomly assigned to one of three groups: one group received a housing voucher specifically for moving to low-poverty neighborhoods, another group was given a standard Section 8 voucher with no such restrictions, and the third group served as the control.
The first major evaluation of MTO took place in 2002, looking at how these families were doing 4 to 7 years after they were randomly assigned to different groups. Then, between 2008 and 2010, researchers conducted another follow-up evaluation, this time 10 to 15 years into the program. These studies focused on how the move affected both the adults and children in the families over time, examining things like mental and physical health, educational outcomes, and employment. In 2011, a comprehensive report was submitted to HUD, and since then, researchers have continued to publish findings from this enormous evaluation.
In this post, I want to discuss the MTO evaluation as an example of an excellent evaluation. In a follow-up post, I’ll propose some theoretical angles that might push the thinking about the evaluation further.
Dessert First: the Results
People in the two program groups were more likely to have more social ties with relatively more affluent people and to feel safer in their neighborhoods. The final evaluation also mentions that the families in the program tended to live in wealthier, less segregated neighborhoods with better housing and that they were less likely to be behind on rent. I take this to be more an implementation check than a “finding” because these results are direct results of the intervention - participants were given housing vouchers only redeemable in neighborhoods that met the program’s criteria.
The physical health of the people in the program improved. They had a lower prevalence of extreme obesity and diabetes, and reported fewer physical limitations than the people in the control group.
The mental health of adults in the program also improved, with lower prevalence of depression and anxiety. The girls and young women in the program saw lower rates of mood disorders, panic attacks, and other mental health issues. However, adults in the program were more likely to have dependence on drugs and alcohol and boys in the program were more likely to have PTSD.
Girls in the program were less likely to drink alcohol, but boys were more likely to smoke. Youth in general were more likely to be arrested for property crimes. Boys assigned to Section 8 housing were more likely to fall behind educationally and were less likely to attend college. However, boys were also less likely to be arrested for drug distribution.
The biggest bombshell result was dropped in 2015 by Raj Chetty, Nathaniel Hendren, and Lawrence Katz: as adults, participants who were under 13 at the time of random assignment saw average incomes 31% higher than participants in the control group. The program had an effect on incomes, but only on those of the children. The children were also more likely to attend college, to live in lower poverty neighborhoods - with women also less likely to become single parents.
Salad Course: Null Results
There isn’t a catchy name for the results we didn’t see, so let’s call them the null results. Technically, these are still “results”, but too often they get left out of the conversation, so I’m drawing special attention to them here, like the salad course draws your attention to the fact that you are in a nice restaurant.
These are some noticeable absences:
No health improvements for youth
No improvements in asthma, hypertension, or chronic pain for adults
No improvements in self-reported overall health for adults
No mental health improvements for boys
No improvements in income or employment rate for people who were adults or teens at the time of random assignment
No changes in risky behavior for youth
No changes in violent crime or drug crimes for youth
No changes in reading and math test scores
No changes in high school graduation rates
For many people who believe that neighborhood effects are key predictors of life outcomes this list may come as a shock. The MTO study is perhaps the most rigorous test of these hypotheses in the US context ever conducted. In a follow-up post, I’ll be asking some questions that can help us come to a deeper understanding of the complexity of null findings like these.
Main Course: What Makes this Evaluation Excellent
The MTO study had a lot going for it, including support from high-calibre research institutions and a mandate to evaluate direct from the US Congress. Let’s name some of the ingredients in this great evaluation.
Appropriate Research Design
The questions that were of key interest in the MTO program were causal questions, like: does changing neighborhoods increase the incomes of people living in poverty? Causal questions are best answered using causal research designs. This evaluation used random assignment and a baseline control condition, since these were the most appropriate methods to answer the core research questions. Often, requests for evaluations include causal questions but either 1) explicitly prohibit methods best suited for causal reasoning, or 2) fail to provide sufficient budget or time for the application of these methods. MTO made a hard commitment to an appropriate research design early in the process.
Mixed Methods
In addition to the quantitative methods I just mentioned, qualitative methods were also used in the study. A separate team of qualitative specialists conducted five years of ethnographic research with the participants to understand how being in the program had changed their lives. They wrote a whole book about what they discovered - more on this later. The combination of quantitative and qualitative methods allows the evaluation to be maximally useful in answering new questions posed by future researchers like us. Note that the ethnographic data collection in this study is totally consistent with the randomized controlled trial method. In fact, randomization and control increased the explanatory power of the qualitative methods.
Inclusion of Negative Results
The implications of the some of the negative results from this study, particularly the early waves, likely sent the program’s biggest supporters reeling. You can just hear them exclaiming: what do you mean ‘moving people out of poor neighborhoods doesn’t reduce crime rates’? There were a few positive headlines from the evaluation and it would have probably been easy to direct most attention towards those. Yet, these early negative results built credibility for the evaluation that would pay off in later years. When Chetty and colleagues broke the news in 2015 that the program did increase income, despite all the earlier negative findings, this result was treated with the gravity it deserved. In the bigger picture, negative findings from this study will help future planners use the right designs. For example, despite what you may have seen on the Fresh Prince of Bel-Air, moving people to different neighborhoods does not actually appear to be an effective way to reduce their risky criminal behavior.
Timeframe
The main economic results of the program took an entire generation to appear. Many programs want to speculate about the “long term” effects of their program, but only fund evaluation efforts for a handful of years. Obviously, longitudinal evaluations can be costly, but as the MTO studies show, they are sometimes necessary to actually measure change. If long-term outcomes are part of the evaluation but budgets are small, the program might consider writing a contract with a reliable evaluation team to spend considerable time on the evaluation for a couple of years, then set a much smaller maintenance budget for a decade or so, with a a couple of heavy years to wrap up at the end.
Goal-Free Approach
The initial congressional mandate to conduct an evaluation on Moving to Opportunity included the instruction to measure “the long-term housing, employment, and educational achievements of the families assisted under the demonstration program.” However, as the years went by, additional parties became interested in the evaluation and expanded its scope to include other potential outcomes, so that by the time of the final evaluation, physical health, mental health, and criminal behavior were also included. While the evaluation materials do not appear to pay homage to the goal-free approach, in practice the evaluation followed one due to its exploratory nature. Many of the important results of the evaluation were probably not contemplated by the 1992 Congress, but the evaluators included them anyway. Casting a wide net for outcomes allowed the evaluation to catch unintended positive and negative consequences.
See you next week for some reflections on how evaluation theory can help us to go beyond current practice, using Moving to Opportunity as a case study.
Thank you for dessert before salad.