2/20/17 A Program Reflection on the Evaluations of Models for Change and The National Campaign to Reform State Juvenile Justice Systems In a variety of ways and over two full decades, the MacArthur Foundation has supported efforts to promote and spread more rational, fair, effective, and developmentally appropriate responses to young people in conflict with the law. Prompted by a series of regressive and fear-driven juvenile justice policy and practice changes that swept the nation during the 1980s and 1990s, the Foundation initially entered the field in 1996, convening and supporting the MacArthur Research Network on Adolescent Development and Juvenile Justice. From 1997 through 2009, the Network oversaw a number of high-profile research studies establishing the profound developmental differences between adolescents and adults and exploring the justice implications of those differences. Models for Change was a long-term, multi-state effort to translate the Network s findings into policy and practice. Between 2004 and 2014, the Foundation invested more than $121 million in a large-scale effort to create replicable models of reform that would effectively hold young people accountable for their actions, provide for their rehabilitation, protect them from harm, increase their life chances, and manage the risk they pose to themselves and to public safety. Six years into Models for Change, in order to capitalize on the initiative s momentum and speed up the adoption of policies that would make juvenile justice systems fairer, the Foundation collaborated with a group of funders to launch a multi-state campaign aimed at changing juvenile justice policy in states across the country. From 2011-2015, the National Campaign to Reform State Juvenile Justice Systems operated in 37 states, with the goal of enacting meaningful juvenile justice reforms in 75% of them. Both Models for Change and the National Campaign have now been independently evaluated. This memo summarizes our reflections on the findings of these evaluations, the lessons we learned, and the implications for our current and future work in criminal justice. The Evaluations and Their Findings In 2013, the Foundation engaged Mathematica and the University of Maryland to conduct an independent summative evaluation of Models for Change. The evaluation was intended to shed light on the achievements and failings of the initiative, assess its strategies and tactics, and estimate its impact, both on the experiences of justice-involved youth in Models for Change jurisdictions and on the course of juvenile justice reform nationally. Evaluators used a mixed methods approach in exploring these issues, including extensive review of documents, interviews with key participants, and analysis of available outcome data from a variety of
sources. Some of the basic questions they explored included: To what extent did systems change occur as a result of Models for Change investments? What basic strategies and tactics were effective in generating systems change, and what factors influenced the degree of effectiveness? What were the important contexts at the outset of Models for Change, and how did these affect initiative progress and outcomes? What factors beyond Models for Change supported work have played a key role in the results achieved in Models for Change states? What can be learned from Models for Change that is transferable to other systems change efforts? In 2014, the Foundation engaged Grassroots Solutions to evaluate the National Campaign in collaboration with the public affairs firm of Berlin Rosen. The evaluation was primarily intended to determine how effective the National Campaign strategy was in achieving sustainable legislative and policy change at the state level, and to what extent it contributed to the next wave of reform in juvenile justice. Evaluators interviewed key participants, conducted surveys, and reviewed written materials and legislative databases in order to answer such questions as: Did the National Campaign accelerate progress in the states in which it worked? Which contextual factors or variables contributed to success or failure? What were the overall strengths and weaknesses of the National Campaign approach? Both evaluations concluded that Foundation-supported work had had meaningful impact, and both yielded useful learning capable of strengthening our future grantmaking. Evaluators concluded that all four of the core Models for Change states (where comprehensive reform efforts were supported over five-plus years) had made significant reform progress, and that the initiative had contributed to this progress primarily by bringing people together and supporting collaborative communities of reform. Without Models for Change, these states would not have been able to achieve the same level of stakeholder engagement and activation. Models for Change reform successes were associated with the development of credible evidence about problems and solutions; the forging of links between state and local juvenile justice systems; the use of multiple tactics to spur reform; and the devotion of resources to dissemination and replication. Multi-state, issue-focused action networks convened as part of Models for Change achieved positive results as well, and were associated with changes in policy and practice that were frequently replicated and sustained with funding from sources other than Models for Change. The national context analysis found that, over time, many jurisdictions outside the initiative came to focus on issues highlighted by Models for Change states, and often pursued reforms through strategies that were also being used in Models for Change. This alignment was in part due to successful communication and dissemination efforts supported by Models for Change. The evaluation of the National Campaign to Reform State Juvenile Justice Systems concluded that the effort had largely succeeded in meeting its ambitious goals, advancing juvenile justice reforms in 30 of the 37 states in which it worked. Of the 313 reforms pursued with strategic,
logistical, and financial support from the National Campaign, 208 were successfully adopted. A complementary national communications effort aimed at increasing the public s receptivity to juvenile justice reform generated high-quality media coverage and social media conversations that appeared to influence the broader narrative about juvenile justice, particularly among young people and parents. Lessons Learned We embrace the findings and lessons of both evaluation reports. In particular, five primary lessons emerging from these reports have been of significant benefit to us in strengthening the design of the Safety and Justice Challenge, our current effort to support and spread reform in the criminal justice system. Lesson 1. While flexibility is essential to supporting system reform work in a variety of states, consistent structure is needed to maximize impact. Though evaluators commended Models for Change for giving states room to set their own reform agendas, they also called attention to various features of the evolving initiative such as inconsistent approaches to the delivery of technical assistance, a tendency for rules to change over time, and a lack of clear and consistent expectations on such issues as progress measurement that point to the need for tighter structure and national oversight. Accordingly, the Safety and Justice Challenge is being overseen by a national intermediary organization, which among other duties is enforcing a coordinated team approach to providing technical assistance, a common timetable, and a set of consistent performance measurement and project management protocols across the initiative. Lesson 2. Allowing sites to compete and sometimes to fail can raise the overall quality and impact of a national reform effort. Evaluators found that Models for Change Action Networks, notwithstanding relatively modest investments from the Foundation, achieved surprising results, in part due to the spur of inter-site competition. The Safety and Justice Challenge seeks to harness the same benefits: the initial Challenge Network admission process was competitive, as was the process for selecting sites to receive deep implementation funding. Moreover, a clear message has been sent that underperforming sites will be terminated rather than funded indefinitely. Lesson 3. To maximize both impact and learning, partners in a broad system reform initiative should commit to a handful of simple quantitative measures of progress. Because of the complex state and local variations in the juvenile justice landscape, the Models for Change strategy was designed to generate multiple reform approaches suitable to a range of conditions. What resulted was a flexible, collaborative and opportunistic initiative, uniting many differing reform issues and approaches, both across states and over time. This made consistent progress measurement difficult at the initiative level. Moreover, the flexibility needed to achieve progress in any one place sometimes worked against the kind of rigor and standardization needed to establish effectiveness and promote replication elsewhere. For these reasons, in the Safety and Justice Challenge, the range of reform activities eligible for support
has been somewhat restricted, and all partners have committed to a narrow range of quantitative indicators as the primary measures of progress. Lesson 4. The time for enlisting allies is at the beginning of a system reform initiative, not the end. Models for Change initiative partners eventually came to include many national organizations representing professional stakeholders in the juvenile justice system. These strategic allies played a useful role in disseminating Models for Change innovations and learning to their membership, but in many cases were not engaged until after active work had ended. Because we recognize that this limited their utility as disseminators, national organizations representing criminal justice stakeholder groups have been involved from the start of the Safety and Justice Challenge, and in many cases participated in the launch of the initiative. Lesson 5. An evaluation and learning agenda must be built into initiative design. Likewise, we are mindful of the challenges of conducting an entirely retrospective evaluation of Models for Change, given the initiative s ambition, geographic scope, and duration. That is why a national evaluator was engaged for the Safety and Justice Challenge prior to its launch, a detailed evaluation plan was incorporated into the design of the initiative from the beginning, and an evaluation and learning partner plays an ongoing role in tracking an evolving landscape, challenging assumptions and helping to keep a rigorous focus on indicators of change. Conclusion We are proud of the Models for Change initiative, and we believe that the collective efforts of our large network of grantees and other partners have succeeded in contributing direction and momentum to a national juvenile justice reform movement. Their work, along with that of many others in government, philanthropy, and juvenile justice practice and advocacy, has led to remarkable progress in recent years, as the National Research Council has thoroughly documented in Reforming Juvenile Justice: A Developmental Approach. Landmarks of change include significant new limits on harsh sentencing of juveniles, expanded use of evidence-based interventions and services, and widespread and sustained reductions in detention, commitment and formal processing. These are all victories in the ongoing struggle to create a juvenile justice system that faithfully reflects both our values and our knowledge of adolescent development and the neurobiological bases of adolescent behavior. Like all of products of social reform, the achievements of the last wave of juvenile justice reform are not entirely secure. We fully expect the Developmental Model of juvenile justice to face challenges and threats. Mindful of this possibility, for the past several years we have made a series of legacy grants, intended to secure and sustain reform progress achieved by our Models for Change partners and to capitalize on that progress to contribute continued momentum to juvenile justice reform nationally. The extent to which these grants have been successful, and the future resilience of the Developmental Model of juvenile justice generally, remain to be seen. An independent evaluation of the legacy phase of Models for Change is currently in the field, and will be made publicly available when it is completed.