Comparing policy work in bureaucracies in Belgium and Canada: Is decision-making rational or incremental? *** Work in progress ***

Transcription

1 Comparing policy work in bureaucracies in Belgium and Canada: Is decision-making rational or incremental? *** Work in progress *** Communication proposal Section 35 Open Section ECPR General Conference Montreal, August August 2015 David Aubin*, Marleen Brans 1, Ellen Fobé 2, and Stéphane Moyson 3 *Corresponding author: Institut de sciences politiques Louvain-Europe Université catholique de Louvain Place Montesquieu 1 box L , B-1348 Louvain-la-Neuve (Belgium) Tel Fax david.aubin@uclouvain.be Abstract Many actors conduct policy work both within and outside governments, but the literature is rather silent about the involvement of civil servants. How do civil servants conduct policy analysis? Do they use formalized analytical techniques (e.g. brainstorming) or more incremental methods based on experience and feedback? We conducted an original large-scale e-survey in the federal and regional Belgian ministries which replicates the questions of the Canadian surveys on policy analysis (Howlett et al. 2014). The survey was submitted to about 9,000 civil servants with a response rate of 30%. The analysis shows a balanced use of both sorts of techniques, despite a more intensive use of incremental techniques. These results are systematically compared with the results of the Canadian surveys. Keywords: policy work, policy analysis, analytical capacity, knowledge use, policy formulation tools 1 Katholieke Universiteit Leuven, Belgium. 2 Katholieke Universiteit Leuven, Belgium. 3 Erasmus University Rotterdam, The Netherlands 1

2 Introduction Many actors conduct policy work both within and outside governments. Policy work means the activity of collecting and processing knowledge and information, framing problems, developing answers and tools, and turning them into applicable political decisions, legal texts and administrative outputs. Policy work refers to individual or organizational contributions to the content of public policy. Literature addresses policy work by external actors (interest groups and lobbying), the interactions between these external actors and the public actors (policy networks and governance), the contribution of politicians to policy-making (party politics), but is much more silent about the involvement of civil servants (except e.g. Colebatch or Howlett). The empirical question remains: How do civil servants located at the centre of government actually conduct policy analysis? Two bodies of literature address the theoretical question of how civil servants work when they are involved in policy-making: Evidence-based policy-making and Lindblom s incrementalism. Evidence-based policy-making follow the great tradition of rationalism that promoted more formal analytical methods in public administration, in line with the mainstream heuristics developed in most public policy textbooks: the stages approach. On the other hand, Lindblom criticized this prescriptive approach of the policy process and proposed an alternative model that, according to him, was closer to actual practice. He claimed that policy-making is a trial-and-error process that is built out step-by-step and by small degrees. Then, two competing theories would describe policy work as a rational versus incremental activity. The aim of this paper is to empirically test these two competing theories about policy work in looking at the kinds of techniques that civil servants use in their daily work, either formalised techniques, used in rational analyses, or more empirical techniques, that would attest of a less rational and consequently more incremental activity. Do civil servants use formalized analytical techniques (e.g. brainstorming or cost-efficiency analysis) or more incremental techniques based on experience and feedback? The theoretical propositions are empirically tested through a large-scale survey in the Frenchspeaking federated entities in Belgium, namely the Walloon Region and the French Community. The survey is an electronic survey on policy work that was submitted in to 4,294 civil servants with a university diploma from the ministries and public agencies. It collected quantitative data about the profile and contributions of civil servants to the policy process. The results show a balanced use of both sorts of techniques. Many civil servants are trained and use formalized techniques in their policy work. However, among these techniques, the less formal, such as interviews and brainstorming, come first. In average, empirical techniques tend to be more intensively used than formalised ones, while the difference is not striking. In sum, civil servants combine the techniques, which mean that incrementalism is still a valid proposition for describing the policy process in the real world. 1 Policy work by civil servants Developing research about policy work, notably the involvement of civil servants in policy work has been invoked for long, without much impact until the years In 1959, Lindblom declared: There has been little analysis of the decision-making process now used by public administration (Lindblom, 1959). Despite some early accounts (e.g. Meltsner, 1976; Weiss, 1992; Weiss and Bucuvalas, 1980), only few systematic research about policy work was engaged to fill this gap: [Exactly] who does this policy work and how they do it 2

3 are important yet understudied questions for understanding the analysis generated and its impact on policy (Howlett, 2013: 4). Not much is known about policy work and policy workers, with even less analysis focusing on the subnational level: [A]dditional empirical and comparative studies are needed to bring more light to the subject of the nature of professional policy work in modern government (Howlett and Wellstead, 2011: 617). In sum, the accounts of policy work found in textbooks are somewhat disconnected from the actual practice (Radin, 2000: 183). More and better studies aimed at understanding the policy practice and what counts in policy work are solicited (Colebatch, 2006c). Policy work consists of providing analytical support to government for making intelligent choices for solving societal problems (Colebatch, 2006c). Traditionally, it was generated by bureaucratic officials located at the top of a public organization to guide policy-makers and support their decisions (Brans et al., 2012; Colebatch, 2006c). However, the monopoly of contemporary bureaucracies on expertise and policy advice was contested by several stakeholders who participate to the policy process. Nowadays, policy work is undertaken by any actor (individual or collective) that seeks to shape policy processes and outcomes (Tenbensel, 2006: 199): academics, think tank employees, consultants, politicians, staff of political parties, interest group representatives, citizen activists, etc. Current research on policy work aims at showing how the policy process and policy analysis actually operate, and how information and knowledge are provided in the decision-making process: The term policy work encapsulates the efforts to capture the present-day practice of policy analysis characterised by a move from a single textbook representation of policy analysis towards acknowledging the complex and multi-faceted nature of policy advisory systems in different countries [ ] (Kohoutek et al., 2013: 31). Three streams apply different methods for answering quite similar questions: policy bureaucracy, policy work, and policy capacity. The first approach belongs to the field of public administration and looks at the margins of manoeuvre civil servants have in the policy process. In Policy Bureaucracy (2005), Page and Jenkins ask two questions: What is the division of labour involved in making decisions within policy bureaucracies? And who does the work, with which political control? They show that middle-range officials are not simply concerned with subordinate embellishment and detail of issues settled at a higher level (p. 15), but rather work with considerable discretion. Their instructions are rarely specific enough to guide with any precision the work they do (p. 81). Ministers rarely give direct instructions, but tend to informally (and often orally) indicate the lines along which they would like their officials to work (p. 82 & 136). Even, civil servants are expected to determine when the minister should be involved (p. 140): Policy bureaucrats at middle-levels are able to take general indicators of ministerial intent and work them into fairly specific proposals or options by using a variety of cues to estimate what the minister is likely to want (p ). As a consequence, middle-range managers are involved in broader cross-cutting issues, not only details. Page s further research on secondary legislation (2012) confirms these results. Attention is raised on middle-range civil servants as important actors on the policy process whose precise roles are underestimated in public policy textbooks. The second approach of policy work was developed by Colebatch (2006c; 2010b), in line with Radin (2000, 2013a). Radin s aim is to observe the shift in the field of policy analysis given the political changes which occurred in the last 50 years, pointing globalisation, accountability, managerialism or the emergence of wicked problems (Radin, 2013b). Nowadays, policy analysts are confronted to many types of clients with competing perspectives and values (see also Majone, 1989). This obviously impacts the kind of analysis conducted and the methods used in providing evidence to the decision-makers. On his side, 3

4 Colebatch warns against a universal depiction of policy work (Colebatch, 2006b). He mobilises the interpretive approaches to policy analysis (Kohoutek et al., 2013: 45), and considers that the nature of policy work differs given the way public policy is framed, either as authoritative choice, structured interaction, or social construction (Colebatch, 2006b). Policy work can accordingly be perceived as a single (internal or external) advice to policy makers, as a construction or maintenance of relations among stakeholders, or as a common construction resulting from an interaction between stakeholders. Different forms of knowledge and different understandings of the process are produced and compete in different organizational locations. In such a context, the central skill in policy work is judgement: Policy workers are drawing on experience to determine which sort of knowledge will be most significant at any particular time, and need to be able to frame their argument in a way appropriate to the situation (Colebatch, 2006b: 317). Nevertheless, the question remains how civil servants actually behave in such an environment and how do they cope with limited information and ambiguity. The existence of multiple accounts of policy is a structural tension to be managed. Like Colebatch, Howlett aims at locating and clarifying the role of policy analysts in the policy process, but he uses a more instrumental approach which emphasises the policy analytical capacity of government in order to design good policies and relates his research to debates on policy design and evidence-based policy-making. The primary purpose of policy analysis is then to improve policy outcomes by the application of systematic analytical methodologies to public policy evaluations (Kohoutek et al., 2013: 46). Following the edited book Policy Analysis in Canada (Dobuzinskis et al., 2007), Howlett developed a research program on policy work (Howlett, 2009) and launched several surveys between 2006 and 2013 (Howlett et al., 2014, 2015). The account of policy work uses three concepts: policy advisory system, policy analytical styles and policy analytical capacity. Policy advisory system is the notion that the various advisors to government can be arrayed as actors within a complex system of internal and external sources of information who interact in various ways to provide the advice upon which decision-makers operate (Howlett, 2013: 5). Policy analytical styles refer to the kinds of analysis and techniques used in these systems. Among a plethora of techniques, only a small set is actually put into practice. Policy analytical capacity is the ability of organisations to carry out research and analysis of public policies. It refers to the amount of basic research a government can conduct or access, its ability to apply statistical methods, applied research methods, and advanced modelling techniques to this data and employ analytical techniques such as environmental scanning, trends analysis, and forecasting methods in order to gauge broad public opinion and attitudes, as well as those of interest groups and other major policy players, and to anticipate policy impacts (Howlett, 2009: 162). This research leads to two main empirical questions: what present-day policy analysts located in government bureaucracies or non-government organisations really do, and whether (and to what extent) education and professional training of policy analysts meets the requirements of evidence-based policy-making (Howlett, 2009: 168). No consensus emerged around a definition of policy work. The boundaries of policy work are certainly difficult to delineate, but even the nature of policy work itself depends on the epistemological position of the students: There will never be one, definitive account of policy work because policy is too ambiguous and contested to be defined in neutral ways, and because policy is an ongoing process, that evolves over time and eschews fixed and static demarcations (Colebatch et al., 2010a: 243). Nevertheless, a provisional definition can be proposed which relates on the kind of activities conducted in the policy process and the way information in collected and processed. First, Page and Jenkins connect policy work to the policy process (2005). They define policy work as developing, running or evaluating a 4

5 program or scheme: [Policy work] involves thinking about, or developing programs (Page and Jenkins, 2005: 56). They identify three types of policy jobs: Production, maintenance, and service. Production relates to the participation in writing a variety of sorts of documents : white papers, bills and other draft regulations, and settlement letters (p. 60). Maintenance focuses on policy implementation, but also includes running evaluation of programs (p. 70). Service policy jobs consist in offering knowledge and skills to others who produce policies. Policy work intervenes at different stages of the policy process. Second, Kohoutek also engages in the formulation of a definition related to the stages: Policy workers thus take part in defining social problems, clarifying public policy goals, identifying strategies for goal achievement, making recommendations of the most plausible solutions and possibly evaluating the effects of such solutions (Kohoutek et al., 2013: 34). This perspective is more constructivist than the former as it puts emphasis on the contribution of policy workers to the problem framing of public policies: policy work accounts for how actors identify and formulate problems, which meanings they attribute to different aspects of a problem (e.g. framing) and ultimately how they perceive the outside world (Kohoutek et al., 2013: 35). In the context of this paper, we intend to provisionally limit ourselves to this working definition which synthetises the various viewpoints noticed in the literature. We consider that policy work is the activity of collecting and processing knowledge and information, framing problems, developing answers and tools, and turning them into applicable political decisions, legal texts and administrative outputs. Far from denying the diversity of stakeholders committed in policy work, the focus of this paper remains policy work by civil servants within government, in other terms in-house policy work. In-house policy work has long been recognised as a core function of modern bureaucracies consisting of providing support to government for making intelligent choices for solving societal problems (Colebatch, 2006a). Policy work is not the ownership of pure analysts or in-house consultants, but a part of the day-to-day activity of middle-range officials. Of course, policy analysis may be concentrated in internal think tanks or consulting offices, but it can also be dispersed among the departments. The overarching concept of policy work thus covers not only policy analysts in the traditional sense but also other professions such as lawyers, doctors, architects, etc. that utilise their expertise one way of another in order to participate in the making of what we call policy (Kohoutek et al., 2013: 32). Are policy workers technicians? In the literature, civil servants are usually perceived as generalists or specialists (e.g. Lindquist and Desvaux, 2007; Mayer et al., 2004). On the one hand, specialised policy knowledge is related to expertise on the specific technical issues pertinent to [the unit s] domain of expertise (Lindquist and Desvaux, 2007: 123). For example, it refers to civil engineers who maintain the road infrastructure or biologists who monitor the dispersion of invasive species. Although these specialised persons have a job corresponding to their initial university training they are sometimes involved in policy analysis in their fields. On the other hand, generalist policy knowledge refers to capabilities to develop broader views on policy issues, to identify horizontal linkages across issues, and sometimes to develop more comprehensive as opposed to selective policy initiatives (Lindquist and Desvaux 2007: 123). Process skills and system knowledge can also be associated to generalist policy knowledge as far as they refer more to knowledge about the policy process than the substantive issues. According to Page and Jenkins, middle-range officials are not subject specialists (2005: 35 and 148), but this statement is only valid in Whitehall, and possibly confirmed in Germany too (Blum and Schubert, 2013; Fleischer, 2009). In these administrative systems, generalists move inside the administration regularly and take up new tasks in a short period of time. Although they are expected to work on 5

6 substantive issues, their expertise is not subject-based or technical. Other administrative systems (e.g. Belgium) give preference to specialists, which means to persons who have a reasonably deep understanding of a field or mastery of a set of technical skills (Lindquist and Desvaux 2007: 124). They are less mobile, and usually remain anchored to a single department. Policy analysis is a method for structuring information and providing opportunities for the development of alternative choices for the policy-maker (Gill and Saunders, 1992: 6-7), but it is more art than science (Bardach, 2000: xiv; cited in Prince, 2007: 165). Policy analysis is a craft, in which specialised knowledge and skills are put to use (Colebatch and Radin, 2006; Weimer, 1998). It draws on the mastery of policy analytical skills such as the ability to apply statistical methods and employ techniques such as environmental scanning, trends analysis, and foresight, and also the ability to engage in interactions with stakeholders, and to communicate effectively with political executives. Policy analysis provides many analytical techniques to produce sound analysis to inform the decision-making process (Colebatch et al., 2010b; Howlett, 2013; Mayer et al., 2004). They range from formal techniques (e.g. cost-benefit analysis) to less formal ones (e.g. public consultations) (Howlett, 2013: 4). Formal techniques are put forward in some textbooks which depict policy workers as experts who utilise a toolkit comprising microeconomic analysis, quantitative methods (cost-utility analyses), and analysis of organisations (e.g. Weimer and Vining, 1989). These formal techniques are usually associated with evidencebased policy-making (Davies et al., 2000; Nutley et al., 2007). Less formal techniques, that are called empirical techniques in this paper, are promoted in less instrumental accounts of policy analysis, which insist on the key activities of making, maintaining and coordinating the actors interactions: Instead of finding rational solutions to policy problems, policy work is used for getting support for certain ideas and interests (Kohoutek et al., 2013: 34). In empirical studies in Canada, Howlett confirms the importance of this relational aspect of policy work. All groups employ process-related tools more frequently than substantive content-related technical ones, reinforcing the procedural orientation in policy work identified in earlier studies (Howlett et al., 2015: 165). Brainstorming, consultation exercises, risk analysis, and checklists are the most used by policy analysts in Canada (>70% of respondents) (Howlett et al., 2015: 170). Cost-benefit analysis, scenario analysis, expert judgement and elicitation, cost-effectiveness analysis and focus groups come behind. The distinction is made between technical types of analysis and soft tools (Howlett et al., 2015: 171). Most policy workers in Canadian government primarily engage in process-related tasks and activities without a great deal of training or effort devoted to more formal policy appraisal techniques (Howlett et al., 2014: 274). As such, the use of sophisticated methods of policy appraisal and evaluation is exaggerated and analysts are engaged in more process-related activities. 2 Is in-house policy work based on evidence or still incremental? The new turn on evidence-based policy-making (Davies et al., 2000) revives a past debate on policy analysis between the rationalists and the incrementalists. In many sectors (notably health and the environment), the provision of (scientific) evidence takes the prominence over the expression of values in the political argumentation (e.g. the only legitimate cause of rejection of genetically-modified crops would be that they are at risk, not because most of the citizens reject the model of agriculture that lies behind). Practically, this spread of rationalism also involved a lot of planning and reporting activities which overwhelmed public 6

7 administrations and agencies in charge of implementation (e.g. the 2000 Water Framework Directive). The claim of evidence-based policy-making is a bit more sophisticated. Evidence-based policy-making is conducted on the basis of instrumental rationality and goals-means approach, with goals being clearly defined and reached with discrete policy alternatives (Brans and Vancoppenolle, 2005: 167). The basic idea is that policy programs should incorporate the results of systematic research. The evidence helps to answer two questions: What works? and what happens if we change these settings? (Head, 2008: 1). Although it works with analogy with evidence-based medicine, evidence-based policy-making takes a broader definition of evidence than clinical medicine (using experiments with randomized controlled trials). The idea is basically the same, politics should not be guided by emotions (as sometimes physicians are), but by a common knowledge that is broadly used and scientifically validated. This rationalist approach discards knowledge coming from intuition, individual experience, values and traditions as it introduces a hierarchy in the quality of arguments, the results of experiments officiating at the top (Head, 2010). The proponents of evidence-based policy-making do not get trapped in this caricature. They recognise that the nature of evidence can be multiple, and propose to enlarge evidence from scientific knowledge to new eligible sources: program management practice and political judgement (Head, 2008). Scientific knowledge assesses the causal effects of policy interventions using systematic approaches to gathering and analysing information. The range of techniques is large from the randomized controlled trials to action-research, belonging to the constructivist paradigm. Evidence in policy could also come out from the sphere of policy implementers (program delivery managers, contract managers, etc.) whose experience is able to assess the feasibility of a given policy design (Head, 2008: 6). Political knowledge is related to all the obstacles to scientific evidence in the policy process: partisanship, values and ideological positions, political commitments, mutual adjustments and political compromises. What constitutes evidence-based policy-making remains a contentious issue (Howlett, 2009; Tenbensel, 2004). This attempt to answer the critics to evidence-based policy-making enlarges the list of eligible techniques. Practically, the turn to evidence-based policy, reclaimed by some western governments (e.g. the Blair government in 1997 in the UK), implies much more formal analysis in the ministries. With evidence-based policy-making the formal analysis of policy alternatives is reclaimed, which reinvigorates the old debate between the rationalist and the incremental approaches put forward by Lindblom in the late 1950s. The politicians and civil servants face a renewed pressure for formalising policy decisions in a rational way. In his seminal article, Lindblom (1959) rejects the rationalist approach of decision-making. He considers that it lacks a capacity of description of the actual decision-making process. He sees the rational comprehensive method as follows: (1) Clarify the values or objectives; (2) Once the ends are isolated, find the different means to reach them; (3) Select the most appropriate mean to reach the ends through a comprehensive analysis studying every parameter and relying on theories (Lindblom, 1959). The rational comprehensive method relies on formal policy elaboration tools such as operations research, statistical decision theory or systems analysis. Given the limited rationality of the individuals (March, 1958; Simon, 1967), the applicability of this method is limited to very simple problems, whereas most policy decisions deal with complex problems. There are too much policy alternatives, conflicting values and limited theories for policy-makers to make a rational choice. To the rational comprehensive method (or root method), Lindblom opposes the method of successive limited comparisons (or branch method). Administrations and agencies operate within a small 7

8 range of politically and legally feasible alternatives. They restrict their choice to relatively few values and alternative policy proposals. The method of successive limited comparisons is thereof proposed as an alternative. It builds out from the current situation, step-by-step and by small degrees (Lindblom, 1959: 79). Policy-makers engage themselves in a small steps approach, based on the record of past experience and trial-and-error. First, policy-makers do not operate a distinction between facts and values, as well as between means and ends. [A]dministrators often are reduced to deciding policy without clarifying objective first (Lindblom, 1959: 82). Second, it is impossible to test good policy when administrators cannot agree on values or objectives. Often decision-makers agree on the means (or policy instruments) without sharing the same objectives: Agreement on policy thus becomes the only practicable test of the policy s correctness (Lindblom, 1959: 84). Third, given the limited capacity of individuals to be comprehensive, they operate simplifications. On the one hand, they consider only changes of small degree and limit the comparison of alternatives to the proposed solution and its consequences to the status quo situation. Fourth, policy change is the outcome of a succession of comparisons: Policy is not made once and for all; it is made and re-made endlessly. Past sequences of policy steps give knowledge about the probable consequences of further similar steps (Lindblom, 1959: 86). The administrator operates successive and modest incremental comparisons. Lindblom s incrementalism, taken as a theory and not a prescription for a better policy process (Lindblom, 1979), still belongs to the policy student tool kit (Rothmayr Allison and Saint-Martin, 2011). At the same time, rationalism is still much credible in public administration and a powerful argument for legitimizing political decisions or performance audits (Saint-Martin and Rothmayr Allison, 2011). The current promotion of evidence-based policy-making as a prescription confirms the importance of a balanced judgment when looking at policy work by civil servants. Do civil servants operate by successive limited comparisons or use systematic (scientific) research in they participation to the design and implementation of public policies? One point of entry in this question is to look at the techniques of data collection and analysis that civil servants use in their day-to-day practice. Many techniques of data collection and analysis are used in public administration. On the one hand, the systematic methods rely on the scientific method. Cost-benefit analysis and its many variants, e.g. efficiency analysis and multi-goal analysis, belong to the more formal ones (Vining and Boardman, 2007). Obviously experiments or quasi-experiments using microeconomic modelling fall in this category, but it should not be too limitative either. Even the proponents of evidence-based policy-making recognise that systematic enquiry encompasses a large set of techniques, including the qualitative methods (Head, 2008: 6). Thus, techniques such as interview, brainstorming, logic model and SWOT are considered as formal techniques too. In the survey, all the technics assimilated to scientific enquiry are put under the same banner of formalised techniques. The list reproduces the one used by Howlett and his colleagues for studying policy analysis in Canada (Howlett and Newman, 2010; Howlett et al., 2014; Howlett and Wellstead, 2011, 2012). On the other hand, civil servants use many other techniques to collect and analyse data that will be called empirical techniques. Empirical techniques are all the techniques that do not rely on scientific methods, either quantitative or qualitative. They encompass a wide range of practice from reading the newspapers to consultation of stakeholders. In fact, civil servants have access to many information sources. Among written sources, one finds various documents. Civil servants have access to newspapers, read scientific articles, and consult a wide range of reports from their civil service, other national administrations, international organisations and interest groups. These reports bring them precious information to design 8

9 and implement public policies. They also ask opinions of various kinds of experts, their colleagues, other public managers, in-house researchers, experts from international organisations, the social partners, other interest groups and the citizens. In sum, empirical techniques group together the non-scientific data collection and treatment and consists mainly in consulting documents and seeking the opinion of colleagues and stakeholders. The basic idea in this categorisation is that civil servants more turned towards rationalism use formalised techniques, whereas those engaged in incrementalism use the empirical techniques based on consultation. The aim is to see what kinds of techniques are mobilised in policy work in order to determine the nature of policy work, evidence-based or incremental. 3 Method The question of the nature of policy work, either rationalist or incremental is asked through the techniques used by civil servants in data collection and analysis. In this paper, the use of techniques is first described, and then correlated with policy work in order to exclude any use of techniques for any other activity (e.g. processing individual files, or maintaining the public infrastructure). The analysis relies on a survey conducted in 2014 and 2015 about policy analysis in Belgium, both at the federal level and in the two main regions of the country, that is Flanders and Wallonia. For a lack of resources, the Region of Brussels-Capital and the German Community were excluded from the population. This survey replicates most of the questionnaire used by Howlett and colleagues in Canada, but is partly adapted to national characteristics. The survey asks about 30 questions (with variations between the series) divided in four chapters: the nature of policy work, analytical techniques, advisory system in the sector, and training. The survey is a large-n electronic survey targeting a whole population of the middle-range and top managers both in the administrations and a selection of public agencies. The political advisors in the ministerial cabinets were the not included. Given the fact that policy analyst is not a recognised profession or position in the Belgian administrations, it was not possible to operate with samples. As such, the questionnaire was sent to all the civil servants holding a university degree. The survey was conducted with the authorisation of the heads of the administrations who provided the lists of the participants. These participants were invited to fulfil the questionnaire online. An in-house version of LimeSurvey was used, which means that it operates from the servers of the university. In this paper, the analysis is restricted to the French speaking Belgium, more precisely to the Walloon Region (current name being Wallonie) and the French Community (current name being Fédération Wallonie-Bruxelles). Belgium has been officially declared a federal country from 1993 and counts about 11mio inhabitants. The process began in the 1960s and came out to a double system of federalism. Two levels of authority were appointed, one on territorial grounds, the Regions, and the other on linguistic grounds, the Communities, the Regions receiving the economic competencies, and the Communities more personal competencies, mainly education and culture. This gave birth to three Regions: Wallonia, Flanders and Brussels-Capital (the last only in 1989), and three Communities, the French, the Flemish, and the German. The French Community is competent in two regions, Wallonia and Brussels, but not in Flanders. The Flemish Community and the Flemish Regions merged and share a single government and parliament, which is not the case for the French Belgian federated entities (for more information, see Brans et al., 2006; Deschouwer, 2009). As such, this survey does not cover the whole country, but only concerns the French Belgian subnational entities, Wallonia and the French Community. It encompasses the central 9

10 administrations and selected public agencies. The population is composed of all the civil servants with a university diploma working in these organisations. For both Service public de Wallonie, i.e. the central administration of Wallonia, and Ministère de la Fédération Wallonie-Bruxelles, i.e. the central administration of the French Community, the survey was sent to all the population, respectively 2,181 and 1,206 people. In the public agencies, a selection of agencies was contacted, and they provided the list. 907 questionnaires were sent, and the total reaches 4,294 questionnaires. For some agencies, they did not have an independent human resource management, then the personnel is part of the lists of the central administration and thus included in the survey (e.g. AWAC, Cellule de développement territorial and Easy-Wal for Wallonia). Some agencies refused to participate, but their absence does not impact the results as they involve few dozens of people and do not hold core policy competencies. Table 1. Population of the survey in the French-speaking Belgium Organisation Population Type of distribution Service public de Wallonie 2,181 Direct mailing FOREM 564 Internal mailing Agence wallonne pour l intégration des personnes 116 Direct mailing handicapées (AWIPH) Agence wallonne à l exportation (AWEX) 71 Direct mailing Société wallonne du logement (SWL) 50 Direct mailing Institut wallon d évaluation, de prospective et de 33 Internal mailing statistiques (IWEPS) CRAC 23 Ministère de la Fédération Wallonie-Bruxelles 1,206 Direct mailing Office national de l enfance (ONE) 27* Internal mailing Conseil supérieur de l audiovisuel (CSA) 23 Direct mailing Remark: * For ONE, number of people involved in the policy work according to the Director-General who established the list. The number of employees with a university degree is, in this case, much higher. The collection of addresses followed an official way. A formal letter was sent to the secretary-general of the ministries asking the agreement to launch a survey and to supply the list. With the agencies, the contacts were taken by , but the lists were also managed by the administrations themselves. Two methods were used to send the survey. The first, and most classic, was for the researchers to collect the addresses and send an individual invitation to each respondent. The second option consisted in providing a list of codes to the administrations which sent them to the respondents. This second method was developed with the organisations which were reluctant to communicate the lists of addresses of their employees. 4 Results of the survey The survey shows that civil servants use both formal and empirical techniques of information gathering and analysis. Except for a few individuals, this use is not intensive but regular. With further calculation, it appears that the use of formal and empirical techniques is both cumulative and correlated with policy work. 10

11 Overall, the response rate to the survey is around 40%. But among these respondents, not all the questionnaires were totally fulfilled. In addition, some persons in the last round used the opt out option if they did not feel involved in policy work. Some persons quitted without being implicated, some left the questionnaire empty probably because they did not identify themselves as policy workers. In sum, only the full and half full questionnaires were counted as responses which give a response rate of 21% (888 responses). The aim was to localise all the civil servants with a university degree. Within the population of ranks A or 1, only six persons declare not holding a university degree. In this analysis, we took a very secure position and only considered those 297 questionnaires that provided answers to all the items required for the analysis. A more thorough statistical treatment may increase this number and will be conducted later. Of course, this low response rate must be balanced with the fact that the survey was addressed to the whole population of civil servants with a university degree and that many of them do not feel involved in policy work. In fact, it was not possible to locate the policy workers before launching the survey. The presentation of the results first describes the two dependent variables, i.e. the use of formalized techniques, then the empirical techniques (both documents and opinions). On this basis, indexes are set up in order to relate the use of analytical techniques to policy work, and to compare the use of both series of techniques. 4.1 The use of formalised techniques (ANtec) The first lesson that the survey brings is that formalized techniques are used by the civil servants in French Belgium (see figure 1). In average, some techniques are used several times a year: interviews or focus groups (1.70), and brainstorming (1.58). Of course, the survey relies on the declarations and does not assess the degree of mastering of each practice. A second group of techniques appears to be used once a year or less: multicriteria analysis (1.00), stakeholders analysis (0.89), survey (0.88), cost-benefit analysis (0.87), and SWOT analysis (0.84). Close to this frequency but a bit below are benchmarking (0.74), and prospective analysis (0.70). Then, it appears that many formalized techniques are known and used in the public administration with a fair frequency. The first two are not the most formal ones, but they are closely followed by a series of techniques used for producing evidence on policies. The low average scores may be explained by the fact that French Belgian civil servants are specialists rather than generalists, and do not have an extensive training in policy analysis (i.e. internal consultants). We imagine that they manage their programs and are sometimes asked to work on its reform. When counting the frequency, 226 respondents (over 297) use interviews or focus groups, 96 use them several times a year, 78 once a month or more. About brainstorming, 221 use it, among which 109 use it several times a year and 63 once a month or more. The use of the other formalized techniques is less intensive. Multicriteria analysis is used by 146 respondents among which 63 use it several times a year and 21 more intensively. Stakeholders analysis comes very close to these figures. Survey are used frequently by 66 persons and intensively by 7 persons (once a month or more). Cost benefit analysis are similar to these figures. In sum, a few formalized techniques are more used than the others, but not very frequently. Only few respondents declare an intensive use of these techniques. 11

12 Figure 1. Average frequency of the use of each formalized technique Once a week or more Once a month Several times a month Once a year or less non used Interview or focus group Survey Brainstorming SWOT Regulatory impact assessment Environmental impact assessment Cost-benefit Analysis Multicriteria analysis Stakeholders analysis Management game Decision tree Prospective analysis Delphi method Benchmarking Logic model Next, the aim of the analysis is to relate the use of formalized techniques to policy work and individual characteristics of the civil servants. To allow this, the total use of formalized techniques was calculated per individual. The use of the 15 techniques was added up (a score of 0 to 4 for each which gives a minimum of 0 and a maximum of 60 divided again by 15 in order to get a mean score between 0 and 4). Over 297 observations, the mean of total use is 0.77, with a standard deviation of 0.57 (see table 2). Most civil servants use formalized techniques once a year or less in average. The percentiles indicate a more frequent use for 25% of them. A very few of them have an intensive use of these formalized techniques (use once a month or more) (see figure 2). Table 2. Total use of all formalized techniques per individual Use of formal techniques Percentiles Smallest 1% 0 0 5% % Obs % Sum of Wgt % Mean Largest Std. Dev % % Variance % Skewness % Kurtosis

13 0 Frequency ECPR Montreal 2015 Aubin et al. 26 August 2015 Figure 2. Total use of all formalized techniques Use of formal techniques The internal consistency of the index built was measured with a Cronbach s alpha test. This test means that when one item is higher for one person, do other items tend to be high too, for the same person? The score is high (0.86) and shows that either civil servants use many formalized techniques or they do not use them at all. In average, some use many techniques more often, without being pure analysts or experts in the use of one specific technique. This high consistency of the test also brings credibility to the later comparisons. In sum, formalized techniques are known and used by the civil servants in French Belgium. Most of them use these techniques from time to time and a few more intensively. While not being pure analysts or experts of one particular technique, those who declare a more intensive use tend to combine the use of several formalized techniques. 4.2 The use of empirical techniques documents (ANinf) The empirical techniques were identified with two questions, one about the use of documents and the other about the request of opinions. This part looks first at the kind of documents civil servants use in their policy tasks. The use of documents is frequent but not so much. The question was: To which frequency do you mobilise the following sources in order to collect useful knowledge and information in the conduct of your policy tasks? It focuses on in-house or public documents or reports from not-for-profit organizations, and then excludes reporting from interest groups (see figure 3). The answers emphasize an average use of current regulation or strategic plans (2.59) as the most frequent. Most of civil servants use them often (131 over 297) and only 21 never. Political documents from the minister are also widely used (1.93). 170 respondents use them sometimes or often. Surprisingly, the use of scientific articles comes fourth (1.71), just after newspapers (2.03). 84 persons use scientific articles often or always and 80 sometimes, that is more than half of the respondents, which shows some continuity with their initial training. In-house analysis is used quite well too: reports from internal research institutes (1.52), reports from federal institutes (1.25), and policy evaluations (1.07). They 13

14 also consult reports from international organisations (1.39) and parliamentary documents (1.34). In sum, civil servants rely rarely to often to official or public documents in order to collect information for policy work. Figure 3. Average frequency of the use of each empirical technique documents Always Often Sometimes Rarely Never Reports on policies Current regulation or strategic plans Regulatory impact assessments (RIA) Reports of policy evaluation Reports from national org. Reports from international org. Parliamentary documents Reports from NGO's Reports from think tanks Reports from other non-for profit org. Reports from regional research centers Scientific articles Newspapers Presented in an index form (see table 3), the mean of total use over 297 observations is 1.40, with a standard deviation of It appears superior to the use of formalized techniques. 25% of the civil servants use documents in policy sometimes or more often. Still, very few declare an intensive use of documents in policy work (about 10 persons between often and always, see figure 4). Once again, their answers are internally consistent (Cronbach s alpha of 0.90). When civil servants use documents, they use several kinds of them. Table 3. Total use of all empirical techniques documents per individual Use of incremental techniques - Documents Percentiles Smallest 1% 0 0 5% % Obs % Sum of Wgt % Mean Largest Std. Dev % % Variance % Skewness % Kurtosis

15 0 Frequency ECPR Montreal 2015 Aubin et al. 26 August 2015 Figure 4. Total use of all empirical techniques documents Use of incremental techniques - Documents In sum, the use of documents in policy work is regular but not intensive. Once again, it appears that some civil servants declare a more intensive use of documents than others. The gathering of opinions provides another indication about the use of incremental techniques in French Belgium s public administrations. 4.3 The use of empirical techniques opinions (ADactScale1) In their analytical tasks, civil servants do not only use documents, they request opinions from their colleagues or external experts. The question is: To what extent does you or your organisation request advice to the following actors from you own initiative. It appears that the most important requests are addressed to civil servants from the same sector (2.75) (see figure 5). 100 respondents consult their direct colleagues each week or more and 160 many times a year or once a month. Civil servants often ask advice from colleagues in the same sector. The other requests concern civil servants from other sectors (1.77), national public organisations (1.45) and experts from in-house research institutes (1.09). 197 civil servants consult their colleagues from other sectors several times a year (112) or more (75), but people from other public organizations that theirs only several times a year (94) or less (70). In-house research institutes are contacted several times a year by 69 and once a year or less by 73. The consultation of the stakeholders is less frequent and dispersed through a series of items. First, the most frequent requests to stakeholders are sent to the regional advisory bodies (0.97), followed by professional federations (0.82), the social partners (0.73) and NGOs (0.67). Individual companies are much less consulted than organized interests (0.58). Political parties and think tanks are the least frequently consulted organizations. They are never consulted by 254 respondents. 15

16 Figure 5. Average frequency of the use of each empirical technique opinions Always Often Sometimes Rarely Never Civil servants from same sector Civil servants from other sectors Regional consultative offices Other national organizations International organizations Employees' or employers' org. Professional federations NGO's Parties' research centers Think tanks Scientific research centers Regional research centers Consulting companies Citizen groups Individual companies Individual scientists Individual citizens The calculation of the index tells that about two third of the respondents request opinions about once a year in their policy work, in average. The mean is 0.84 with a standard deviation of 0.53 (see table 4). Only a third of the respondents gather opinions once a year or more, and very few several times a year. Once again the Cronbach s alpha is very high (0.87) which attest of an internal consistency in the answers. When civil servants request opinions, they tend to ask to many sources. This limited frequency may be explained by an absence of policy analysts and civil servants with a profile of specialist in their field (e.g. road maintenance or regulation of public libraries) who spend most of their time implementing public policies and are occasionally requested to participate to policy change. Table 4. Total use of all empirical techniques opinions per individual Use of incremental techniques - Opinions Percentiles Smallest 1% 0 0 5% % Obs % Sum of Wgt % Mean Largest Std. Dev % % Variance % Skewness % Kurtosis