Evidence-Based Policymaking Speaking Truth to Power?

, vol. 72, no. 4, pp. 397 403 doi:10.1111/1467-8500.12037 RESEARCH AND EVALUATION Evidence-Based Policymaking Speaking Truth to Power? Brian W. Head University of Queensland brian.head@uq.edu.au Policy decision-makers, whether in the political or administrative sectors of government, have widely claimed to be in favour of evidencebased policy-making. After all, the alternative might be seen as policymaking based on ignorance, prejudice and opportunism, which are not readily admitted. A number of professionals have preferred the term evidence-informed policy-making, on the basis that decision-making is typically not derived from objective science but rather is based on reasoned argumentation, taking account of professional judgements, stakeholder interests and political contexts. Analytical evidence does not give rise to policy decisions in a straight-forward way. Moreover, on any policy issue there is potentially a vast and diverse range of relevant evidence and many sources of informed advice. Do policymakers really pay attention to evidence in making their decisions? Clearly they do, but they often choose to consider sources of expertise beyond the analytical reports produced by social scientists, program evaluators, and technical advisors. In some cases, the evidence that is actually utilised might be that which confirms the currently preferred position of decision-makers rather than alternative evidence which explores new options or questions business-as-usual. Policy bureaucrats have to weave together the diverse strands of knowledge and interest, and decide how courageous they wish to be in probing the weaknesses of a government s preferred position. Policy decisions do not wait for excellent information to become available; decisions will be taken even where evidence is fragmentary and uncertain. Policy-makers are pragmatic, mindful of community sentiment and stakeholder values. They are concerned with political feasibility as well as the quality of evidence (for reviews see Head 2010, 2013). Carol Weiss made a persuasive distinction many years ago between research evidence that is used directly to modify or adjust policy settings; that which is used selectively to legitimate a policy position; and research-based ideas which gradually influence the way in which problems and issues are understood and eventually addressed (Weiss 1979; Weiss & Bucuvalas 1980). These instrumental, political and conceptual forms of influence are fundamental to understanding the relationships between evidence and policy. In non-government sectors such as academia, business associations and community organisations professional experts often develop policy-relevant information and arguments. Evidence-informed policy ideas often lie on the shelf, and do not translate easily into policy influence. For example, in the academic sector, researchers are seldom successful in directly influencing policy decisions, even if they work hard to develop better relationships with policy staff. Think-tanks and industry lobbyists are actively involved in policy debates and political communication, but with variable success in influencing policy outcomes. In general, those who produce rigorous evidence and evidence-informed policy ideas do not control how their ideas are interpreted, modified and used by others.

398 Evidence-Based Policymaking December 2013 Since the 1980s, it has become important to recognise the diverse sources of expertise that have become available to decision-makers. This trend is sometimes described as the contestability of policy advice in a pluralistic world of public debate. Recent studies have attempted to understand how these various forms of knowledge are mobilised and integrated into policy and program areas (Nutley et al, 2007; Productivity Commission 2010). Scientific or rigorous knowledge is widely valued but has to contend with business, political and professional understandings about the nature of the problems, and about what kinds of solutions are feasible and practicable. The advice of public servants is generally crucial, but public service advice does not go unchallenged in modern policy debates. Ministerial minders, business lobbyists, media commentators, reports by thinktanks, leaders of community organisations, and many other actors may influence the manner in which policy judgements are made. Somewhat less clearly, program clients are sometimes also consulted, and their perspectives gathered and appraised as part of program reviews; their experience is relevant to understanding policy outcomes, and in this sense they can be seen as partners in policy improvement rather than just as objects of administration. The has published a number of articles relevant to these themes of evidence-based policy in recent years. To showcase the contribution the journal has made in relation to this topic, we have brought together a series of previously published papers in an online virtual issue available here: (http://onlinelibrary. wiley.com/journal/10.1111/(issn)1467-8500/ homepage/evidence-based_policy.htm). For this retrospective showcase, we have grouped twenty papers into four themes. 1. Foundational issues: values, politics and governance Head (2008) discussed recent trends to incorporate the results of systematic research (or evidence ) into policy development, program evaluation and program improvement. This process was linked to the emphasis in New Public Management (NPM) on efficiency, effectiveness, and what works? Secondly, some of the well-known challenges and limitations for evidence-based policy were outlined policy decisions emerge from politics, judgement and debate, rather than being deduced from empirical analysis. Policy debate and analysis involves an interplay between facts, norms and desired actions, in which evidence is diverse and contestable. Thirdly, the article outlined a distinction between technical issues where evidence can be deployed directly, and negotiated approaches to complex problemsolving. Finally, three lenses, or types of expert knowledge were outlined: systematic ( scientific ) research, program management experience ( practice ), and political judgement. These disparate bodies of knowledge become multiple sets of evidence that are weaved together to inform and influence policy. The divide between the domains of research and policy-making has troubled many commentators. Edwards (2005) elaborates on the gaps and mis-communications between the sectors, and explores ways in which the effective use of scholarly research in policy-making might be improved. The perspectives which each side holds about the other need to be mutually clarified and addressed. Edwards notes that there are serious doubts about whether the way social research is currently organised and presented can be effectively used by policy advisers and decision-makers. This requires adaptation on the part of social researchers. On the other hand, the organisation of policy advice processes within departments may pose obstacles to establishing better connections with researchers. This may require changes in public sector practices in order to make better use of the special skills, knowledge and perspectives of researchers. According to Kay (2011), evidence-based policy making has been criticised as a revival of the rationality project, in which democratic politics is discounted as the play of selfish interests. In response, the evidence-based policy movement has failed to articulate a defence in which the rationality animating the policy process is situational and contextual rather than

Head 399 technocratic and uniquely authoritative. Kay links the what works theme to the American pragmatist tradition which influenced the policy writings of Lasswell and later the New Labour government in the UK. Kay argues that the ambition for evidence-based policy-making should be seen in terms of the transition from a single, unique and universal rationality toward multiple rationalities that vary according to different policy making contexts. Understood in such terms, evidence-based policy-making can avoid several major criticisms, and offer strong potential to contribute to solving policy problems. The role of values has been a contentious theme in the discussion of efforts to make policy decisions more rational and evidencebased. Freiberg and Carson (2010) note that in policy areas like crime and corrections, rationalist approaches are often submerged by affective emotion-based approaches to law and order. Policy-making in the real world is linked to public opinion, so that debates about criminal justice are played out in broader arenas than the academy, the bureau, or the agency. Successful reform must take into account changes in public mood or emotions over time and be sensitive to different political and social cultures. Freiberg and Carson argue that criminal justice policies are more likely to be adopted if, in addition to the gathering and presentation of evidence, the proposals recognise and deal with the roles of emotions, symbols, faith, belief and religion in the criminal justice system. This requires extensive engagement and evidencebased dialogue with interested parties. The central importance of culture and civic engagement is nowhere more central than in regard to Indigenous policy issues. Maddison (2012) argues that there is a tension in the evidence-based policy paradigm regarding Aboriginal and Torres Strait Islander people, particularly with regard to their standing as providers of relevant evidence. Aboriginal people in Australia have primarily been seen as a problem to be solved. Racialised views of Aboriginal competence have allowed past policies, now recognised as harmful, to be justified as being for their own good. Maddison considers some of the complexities of the evidencebased policy paradigm as it applies to the Indigenous policy domain, arguing that in such a turbulent field the use of evidence is inevitably ideological and selective. The article concludes that, in light of persistent institutional inequalities, the full inclusion of Aboriginal and Torres Strait Islander voices and perspectives in genuine dialogue about policy is the only way to navigate this difficult terrain with any chance of success. 2. Knowledge and skills for policy-making According to Hess and Adams (2002), the changing nature of public administration entails a new emphasis on knowledge, whether linked to decision-making, performance monitoring or stakeholder engagement. However, they argue that debates in public administration have demonstrated a poor understanding of how knowledge is constructed and valued and of how public administration knowledge frames are changing in response to major structural shifts in political imperatives. In particular the retreat from economic rationalism and the embracing of social and human capital ideas with the search for third ways and triple bottom lines have brought more constructivist knowledge frames back into play. In this way, centralised rational/expert knowledge is being challenged by knowledge arising from cooperative, local inquiry and multiple knowledge frames are now being brought to bear in public administration. This article uses a historical comparison to show how changes in the knowledge requirements of public administration are demanding new skills of contemporary public administrators. Some of these issues were further explored in a symposium introduced by Bell (2004). Bell outlined some of the institutional and governance implications of the three symposium authors, emphasising the need to go beyond the narrowly rationalist approaches to policy and governance. Adams (2004) argued that the range of usable information for public policy is complex and distributed, but that policy debate has been overly dominated by instrumental and centralised information, constructed and controlled by functional and

400 Evidence-Based Policymaking December 2013 managerial experts the creed of expertise. He argues that other types of usable knowledge have begun to flow back into policy streams and in particular local knowledge (sometimes called community knowledge) is staging a major revival. This inductive and experiential knowledge is now being merged with the deductive paradigms of new public management. Parsons (2004) provided a strong critique of instrumental rationality as a central paradigm for how analysis and advice actually work in practice. He explores the theory and practice of building policy capacity and coherence, arguing that the UK New Labour concern with policy modernisation and the rational analysis of program choices has narrowly focused on instrumental rationality through a more systematic and strategic use of knowledge. However, according to Parsons, this instrumentalism has meant that policy design has worked within a very tightly constructed epistemological regime that neglects the crucially important non-instrumentalist approaches to policy knowledge and learning. The professional craft of weaving together diverse knowledge, in a constantly changing political and institutional context, offers a firmer basis for framing the problem of building policy capacity and coherence. Stewart (2004) develops an applied consultancy approach to some of these issues, with reference to public sector reform knowledge in the United States, New Zealand and Australia. Stewart argues that policy practitioners are justified in eschewing esoteric debates about positivist and alternative epistemologies for the validation of knowledge. Policy practitioners, he notes, are concerned with identifying and utilising diverse forms of knowledge to develop practical understandings of what works and does not work in delivering desired program outcomes. Such knowledge is imperfect; but its value does not depend on the seal of approval stemming from positivist validation. By contrast, Bhatta (2002) argues that policy units have tended to lack some of the strong analytical skills required by modern policy analysis. New Public Management (NPM) reforms since the 1990s have substantially altered the environment in which agencies of providing policy advice to ministers do their work. Yet, there are grounds for suggesting that systems in place in these policy shops in most jurisdictions do not reflect the rigour that is needed to analyse and render quality policy advice. As one possible remedy, Bhatta suggests that use of rigorous evidence-based analysis to inform the policy advice function be made an integral part of their learning and information protocol. Evidence-based analysis is seen as a useful tool in assisting the agencies to demonstrate their adherence to a quality assurance system. 3. Policy capacity and policy contestability The quality of policy decision-making in the real world depends partly on investment in good information and strong analytical skills, and partly on smart leadership to navigate the turbulent environment and the complex risks and challenges faced by decision-makers. Hardaker, Fleming and Lien (2009) argue that public policy-making does not follow the longestablished and well-recognised principles of rational decision analysis under risk. Public views of risk are often inconsistent and seemingly irrational, and a gulf exists between risk perceptions and attitudes of the public and those of experts. On the other hand, experts often claim unjustifiably high levels of confidence in their predictions of policy choice outcomes, creating a lack of public faith in their recommendations. While risky policy choices deserve systematic analysis, many challenges remain to effective implementation of such analyses. The authors suggest a need for more effective interaction between policy-makers, decision analysts and the public. The monopoly over policy advice by the public bureaucracy has long been broken. One of the emerging features of the policy-making landscape is the rise of think-tanks, building on the policy institutes that were established in the postwar decades. t Hart and Vromen (2008) survey recent developments in the world of think tanks as reported by the international literature, and then examine the implications for

Head 401 understanding the nature and role of Australian think tanks. These organisations are significant in a policy world where the authoritative claims of scientific knowledge are challenged by claims about the essential contestability of knowledge. They examine the role of thinktanks in this era of increasingly boundary-less, highly networked societies. The rise of new policy players, and the trend towards outsourcing many services, have led to reflections on the capacity of public agencies to provide coherent policy advice and to manage the programs that depend on partnerships and contracts with third parties. Tiernan (2011) notes that, internationally, there have been persistent complaints that the policy capacity of governments has declined. New public management (NPM) reforms focused on efficiency and emphasised management over policy competence. Political management reforms have led to greater ministerial control over policy processes, including promoting greater contestability in provision of policy advice. Ministers now have access to a diverse array of information, research and analysis from rapidly proliferating networks and alternative sources. Moreover, growing complexity, dynamism, technological change, globalisation and the rise of network governance have created demands for new policy skills and capacities. Among the case studies of policy capacity at national level, Wanna (2011) examines the role of Treasury as the major focus of economic policy advice and analysis. Treasury remains one of Australia s most intellectual and respected government departments, boasting a strong analytical approach to its policy work based on economic principles and modelling. It is able to recruit the brightest analytic minds. It is one of the main gatekeepers in the policy process and has a formidable reach in terms of its potential influence. Treasury may aspire to have the last word but it is far from the sole conduit of economic advice. There are many other credible players and sources of information, from the Productivity Commission to expert consultancy firms. While the market-place for ideas has changed, Treasury has taken seriously its need to constantly review and improve its policy analysis capability in response to changing circumstances. Lindquist and Tiernan (2011) summarise the case study findings in the symposium on the policy advising roles and capacities of the Australian Public Service. Putting recent experience in historical context, they consider the performance of the Commonwealth government s policy advisory system, the impact of prime ministers and centralisation, the link between advising and analytic capacities, and the system s resilience and readiness. They also consider whether recent dissatisfaction over APS advising reflects a lack of capacity or a culture clash, and they reflect on the responsibilities for ensuring high-quality policy advice. They recommend developing a more systematic approach to assessing policy advising capability, building on recent APS reforms. 4. Policy evaluation and accountability Neylan (2008) argues that the growing call for social policy to be evidence-based implies that evidence possesses an intrinsic authority. Much of the evidence used by governments to formulate or evaluate social policy is signified through statistics and the language of quantification. Evidence presented in this way has the appearance of certainty and a legitimacy that seems beyond challenge. Neylan argues that this perspective needs to be put in context. Having an appreciation of the history and sociology of the science of the state, as statistics was originally defined, helps demystify the authority of social statistics. This enables policymakers and program administrators to better discern the policy merit of numerical evidence, and to assess how other relevant sources may also be important. Evaluation of policies and programs is central to the evidence-based policy movement, but more emphasis has been placed on evaluating programs than on evaluating the quality of policy advice itself. Di Francesco (2000) argues that policy advice has generally remained outside the formal processes of performance evaluation. Evaluation, by its very nature, is designed to question both the

402 Evidence-Based Policymaking December 2013 effectiveness and relevance of government activities; applying it to policy advice opens up a traditionally confidential and politically sensitive arena. Di Francesco reports on an evaluation experiment in Australian government policy management reviews (PMRs) that sought to evaluate the quality of central agency policy advice. He traces the development of the PMR model around interdepartmental committee processes, the bureaucratic politics that diluted the focus on policy outcomes, and examines how central agencies steered evaluation away from questions of public accountability towards arrangements for achieving more effective control of the processes underpinning production of advice. The modern focus on program performance and managing for results has generated a strong interest in the development of key indicators to monitor progress. Hezri and Dovers (2009) examine how this approach works for ambiguous areas such as sustainability. The authors focus on how indicators function as an instrument within policies aimed at enhancing environmental objectives in Australia. By integrating information from the environmental, social and economic domains and then feeding knowledge into a wide range of policy sectors, sustainability indicator systems may facilitate cross-agency and -portfolio connectivity. Key characteristics of the various sustainability indicator systems developed to inform policy are described, noting the diverse approaches to indicator development in Australia. The structural relevance of indicator systems to the mechanisms of policy and institutions in Australia is discussed, and corresponding issues of institutional fit are analysed. The linkages between better evidence, better decision-making and greater accountability are important but are difficult to establish. Argyrous (2012) attempts to tackle some of these challenges, by providing practical guidance for improving the quality of evidencebased policy. Rather than adopting an approach that gives priority to particular types of research methodologies, Argyrous argues that evidence drawn from any methodology can be improved if appropriate standards of transparency and accountability are followed in the process of gathering, analysing, interpreting, and presenting evidence for policy. The article details what these standards of transparency and accountability mean in practice, how these standards can be achieved, and possible limits to their adoption. References Adams, D. 2004. Usable Knowledge in Public Policy. 63(1):29 42. Argyrous, G. 2012. Evidence Based Policy: Principles of Transparency and Accountability. 71(4):457 468. Bell, S. 2004. Appropriate Policy Knowledge, and Institutional and Governance Implications. 63(1):22 28. Bhatta, G. 2002. Evidence based Analysis and the Work of Policy Shops. Australian Journal of Public Administration 61(3):98 105. Di Francesco, M. 2000. An Evaluation Crucible: Evaluating Policy Advice in Australian Central Agencies. 59(1):36 48. Edwards, M. 2005. Social Science Research and Public Policy: Narrowing the Divide. Australian Journal of Public Administration 64(1):68 74. Freiberg, A. and Carson, W. G. 2010. The Limits to Evidence-Based Policy: Evidence, Emotion and Criminal Justice. Australian Journal of Public Administration 69(2):152 164. Hardaker, J. B., Fleming, E. and Lien, G. 2009. How Should Governments Make Risky Policy Decisions? 68(3):256 271. Hart, P. T. and Vromen, A. 2008. A New Era for Think Tanks in Public Policy? International Trends, Australian Realities. Australian Journal of Public Administration 67(2):135 148. Head, B. W. 2008. Three Lenses of Evidence-Based Policy. 67(1):1 11. Head, B. W. 2010. Reconsidering Evidence-based Policy: key issues and challenges. Policy and Society 29(2):77 94. Head, B. W. 2013. How do government agencies use evidence? Stockholm: Swedish National Board of Health and Welfare. http:// www.socialstyrelsen.se/lists/artikelkatalog/ Attachments/19163/2013-6-38.pdf

Head 403 Hess, M. and Adams, D. 2002. Knowing and Skilling in Contemporary Public Administration. 61(4):68 79. Hezri, A. A. and Dovers, S. R. 2009. Australia s Indicator-Based Sustainability Assessments and Public Policy. 68(3):303 318. Kay, A. 2011. Evidence-Based Policy-Making: The Elusive Search for Rational Public Administration. 70(3):236 245. Lindquist, E. and Tiernan, A. 2011. The Australian Public Service and Policy Advising: Meeting the Challenges of 21st Century Governance., 70(4):437 450. Maddison, S. 2012. Evidence and Contestation in the Indigenous Policy Domain: Voice, Ideology and Institutional Inequality. Australian Journal of Public Administration 71(3):269 277. Neylan, J. 2008. Social Policy and the Authority of Evidence. 67(1):12 19. Nutley, S., Walter, I. and Davies, H. 2007. Using Evidence: How Research Can Inform Public Services. Bristol: Policy Press. Parsons, W. 2004. Not Just Steering but Weaving: Relevant Knowledge and the Craft of Building Policy Capacity and Coherence. Australian Journal of Public Administration 63(1):43 57. Productivity Commission. 2010. Strengthening Evidence-based Policy in the Australian Federation, 2 volumes. Canberra: Productivity Commission. Stewart, R. G. 2004. Public Sector Reform Knowledge Production: Validation and Evidence as Agreement. 63(1):58 65. Tiernan, A. 2011. Advising Australian Federal Governments: Assessing the Evolving Capacity and Role of the Australian Public Service. Australian Journal of Public Administration, 70(4):335 346. Wanna, J. 2011. Treasury and Economic Policy Beyond the Dismal Science. Australian Journal of Public Administration, 70(4):347 364. Weiss, C. H. 1979. The Many Meanings of Research Utilization. Public Administration Review 39(5):426 431. Weiss, C. H. and Bucuvalas, M. J. 1980. Truth tests and utility tests: Decision-makers frames of reference for social science research. American Sociological Review 45(2):302 313.