Exploring the possibilities of Critical Frame Analysis for evaluating policies. A paper to be delivered at the IPSA World Congress, Fukuoka, 2006

Transcription

1 Exploring the possibilities of Critical Frame Analysis for evaluating policies Mis en forme : Interligne : Double A paper to be delivered at the IPSA World Congress, Fukuoka, 2006 María Bustelo (Complutense University, Madrid) and Mieke Verloo (Radboud University, Nijmegen) Abstract Some studies have pointed out that the type of policy evaluation generally less developed is the so-called design evaluation, that is based upon the content, and focuses on the design or conceptualization of a policy, rather than on their implementation or results (Bustelo, 2003). However, this kind of evaluation is a key one, not only for the assessment of the coherence and relevance of policies, but also for offering a rationale for interpreting their implementation and results. Studies on implementation problems of gender mainstreaming strategies suggest that they might be based on non-recognized and non-explored enough multiple interpretations and representations of the concept of gender (in)equality. Following this hypothesis, the MAGEEQ project (Mainstreaming Gender Equality in Europe) presents an approach at studying divergences in policy frames around gender equality as one of the elements connected to implementation problems (Verloo, 2004). Based on the empirical and comparative research in the MAGEEQ research, this paper will explore the possibilities of its methodology, Critical Frame Analysis, as an important contribution to design or conceptualization policy evaluation, and will discuss how it can be crucial for the evaluation of gender policies and gender mainstreaming. 1

2 The paper will further reflect upon the question if the use of Critical Frame Analysis can be extended to the evaluation and analysis of policies in general. 2

3 Exploring the possibilities of Critical Frame Analysis for evaluating policies A paper to be delivered at the IPSA World Congress, Fukuoka, 2006 María Bustelo (Complutense University, Madrid) and Mieke Verloo (Radboud University, Nijmegen) Introduction Until now, design evaluation has developed mostly in a positivistic paradigm, leading to the construction of cause-effect relationships as a primary way of understanding what is the design underlying policies. This means that design evaluation until now is based on assumptions that policy making is a rational process. As there is a wealth of empirical evidence and theoretical thinking that has challenged these assumptions, there is a need for concepts of design evaluation that avoid the rationalistic trap. This paper expands on criticisms of such assumptions, and presents a methodological framework to engage in design evaluation while avoiding assumptions of (absolute) rationality in policy making. Starting with an overview of design evaluation in theory and practice, it uses the case of gender equality policies to further develop ideas on the relationship between the quality of policies and the existence of implementation problems. It then discusses the idea of using a methodology - Critical Frame Analysis - that has proven its value in comparative gender equality policy analysis. This paper contends that this methodology, when used in design evaluation, could provide an alternative that does not fall into the rationalistic trap. In the last sections, the paper presents additional ideas for the use of this methodology outside the field of gender equality policies, and discusses both the potential and the challenges involved in used Critical Frame Analysis in design evaluation. 3

4 1. Public Policy Evaluation: the importance of program theory and design evaluation Some studies have pointed out that the type of policy evaluation generally less developed is the so-called design evaluation, that is based upon the content, and focuses on the design or conceptualization of a policy, rather than on their implementation or results (Bustelo, 2003). From another perspective, it is what other authors have identified as program theory (Weiss, 1998; Bickman, 1990; Wholey, 1987; Chen, 1990). The program theory is the set of beliefs that underlie action ( ). It is a set of hypotheses upon which people build their own programs. It is an explanation of the causal links that tie program inputs to expected program outputs (Weiss, 1998: 55). For these authors, whose perspective have been widely accepted among evaluation theorists, more than a type of evaluation, the identification or definition of the program theory is a necessary and prior step for getting to know the program or policy that needs to be evaluated. Indeed, this kind of evaluation, or exercise or step depending on how you conceive it - is a key one, not only for the assessment of the coherence and relevance of policies, but also for offering a rationale for interpreting their implementation and results, because it provides information on what it is that a policy attempts to achieve. However, as Carol Weiss recognizes, the notion of defining program theory has not typically been a component of evaluation (Weiss, 1998: 55). Many evaluations put more attention on the outcomes and not so much on the paths by which they were produced. But for having useful evaluations, which really contribute to the improvements of programs and public policies, a more holistic and complete view of public intervention is needed. Thus, any useful evaluation must answer not only the 4

5 question Did the program work?, but also What made it work? Why was it successful or unsuccessful? and How can we make it better? For Weiss, to make a respectable contribution to such discussion, it helps if the evaluator understands and investigates- the program s explicit or implicit theory (Weiss, 1998: 55). Thus, although formally recognized as an important type of evaluation or, even, as a necessary step for doing a complete and useful evaluation, it is an evaluation area not fully developed in practice. This fact might also be at the same time, the cause and effect of not having many references nor theoretical, nor practical - for doing this kind of evaluation. In fact, the literature available that addresses theories, models and methods for doing this design evaluation or program theory is much less prolific than for any other kind of evaluation. Moreover, all references available pertain mostly to a concrete scientific tradition, inserted mainly in a positivistic and hypo-deductive paradigm. Although the rationale for the importance and the need of defining program theories deals with ideas such as that programs and public interventions are complex and complicated phenomena, generally born out of experience and professional lore ( ) and they are not likely to be laid out in rational terms with clear-cut statements of why certain program activities have been selected and which actions are expected to lead to which desired ends (Weiss, 1998: 55), the theory behind program theory lies on a highly positivistic and rationalistic perspective. In fact, for the program theory authors, defining a program theory consists mainly on constructing a model of cause-effect relationships. This can be seen in the definitions some of these authors have given to the term of program theory: For Bickman (1987: 5) it is a plausible and sensible model of how a program is supposed to work, and for Wholey (1987: 78), program theory identifies program resources, program activities, and intended program outcomes, and specifies a chain of 5

6 causal assumptions linking program resources, activities, intermediate outcomes, and ultimate goals. Furthermore, approaches like the Theory-driven Evaluation, which stem from the program theory concept, have developed around the idea of defining or identifying the cause-effects relationships that compound the program theory in order to test them subsequently. These approaches are very positivistic and quantified-oriented. Other available references, not so linked to the program theory approach, but to the idea of doing a type of conceptualization or design evaluation, are also related to rationalistic approaches. Based primarily in the Logical Framework approach, some authors have developed guidelines for doing design or conceptual evaluation. As an example, Osuna and Márquez (2000) define this kind of evaluation as one that aims to review the program elements that justify its existence and the way in which (the program) is formulated and articulated. They identify two crucial evaluation criteria: rationality and coherence. The first one, rationality, which is conceived as the examination of the relationships between the observed needs and/or problems and the established objectives for the improvement and solution of those problems, implies two sub-criteria: relevance and pertinence. Relevance is defined by these authors as the evaluation of the quality and veracity of the diagnosis of needs previously done. Pertinence is the evaluation of the quality and adequacy of the program objectives and their definition. Coherence is the adequacy of the established objectives to the implementation program strategy. Internal coherence refers to the articulation of objectives, strategy instruments and their adequacy to problems, and external coherence is the analysis of the compatibility of the program objectives and strategy with other similar programs and policies in the same territory and at the same time. As it can be seen, these approaches are also within the tradition of considering that, although maybe programs and policies might not be completely rational, they basically are, they ought to be rational and must 6

7 seek for that rationality. That is, the norm (perfectly expressed through the evaluation criteria proposed for design or conceptual evaluation), what it is expected to be a good program or policy, is to be completely rational in logical terms. In contrast to this, public policy literature has repeatedly shown, already for quite some time, that public policies and programs are far less rational than the first rationalistic analysts had believed and maintained (as the incremental model by Lindblom; implementation studies by Wildavsky, Lipsky; literature on problem definition and agenda setting by Cobb, Elder, and so on and so forth). Especially, authors on problem and agenda setting have pointed out that public problems and issues are social and political constructions, and in dealing and negotiating with the different actors involved, policy makers are not very rational in addressing public issues, not even in defining or representing the public issues that should be addressed. The amount of actors and factors influencing the policy making process from the very beginning even from the very process of constructing the public issues to address - do not let public programs and policy be or work so logically rational. So to a certain extent, it is understandable that the approaches developed until now around design or conceptual evaluation and program theory have not been widely spread and used. Due to its logical and rational narrow framework, it is difficult to fit real programs or policies to the norm. At least, most often, evaluations are prone to be disappointing, because frequently program theories are going to be difficult to identify. There are thousands of not so-clear cause-effect relationships present and what is ideally looked for - logical rationality is almost impossible to find. Still, concept or design evaluation is an extremely important methodology to develop and a key piece in the puzzle for a better understanding of social intervention. What is much needed is to revisit the concept of program theory in conceptual evaluation as it is 7

8 has been developed so far. The idea of program theory as the set of beliefs or assumptions that underlie action, or the one of conceptual or design evaluation as the one which analyses those theories and how they work in motion and how they change over time, is a fully valid idea, and an excellent contribution to evaluation theory. More than developing a new concept, what is needed most is to develop different methodological approaches, based on different assumptions of the role of rationality, and which can present a different way of understanding possibilities for better programs and public policies. Critical Frame Analysis aims to contribute to this development. Summarizing, there is a double criticism of positivistic design evaluation: first that the assumptions underneath do not fit with reality, because policies or intervention programmes are the results of struggles as often as they result from rational considerations of how a problem can best be solved. Even in the latter case they are unavoidably trapped in historical logic, in what was seen as the problem before, what has been tried before, and which institutions have already been created to deal with the problems. Second, if policies are seen as constructed configurations of problems and (at best matching) solutions that are the result of struggles, this means that they most often will combine elements coming from different competing actors within the state and different mobilizing networks outside, that they are rooted in a history of success and failure of different actors with whom the current policymakers do or do not identify, that they are constructed in a context of existing and emerging dominant discourse frames, and linked to existing and emerging diffusion channels, and that they do not operate under conditions of complete information. All this results in policies that are better understood as assemblages than as a rational set of interventions for certain purposes. 8

9 How then to conceptualize a framework that can identify, describe and analyze such an assembled program theory? As a starting point, the following section will present a framework that has been developed for comparative research of policies that are understood as such assemblages. 2. Gender mainstreaming implementation problems and non-recognised multiple interpretations of the concept of gender (in)equality Following the rationale developed in the former section, design or conceptual evaluation is important, not only by itself, but also because the analysis of the theories underlying action and how they are articulated into solutions and actions, may also explain some implementation problems. In this paper, we built upon studies of gender equality policies that focus on implementation problems of gender equality poliices and especially gender mainstreaming. Gender mainstreaming, which is the strategy international and officially adopted by numerous nations in the UN World Conference on Women in Beijing in 1995, as a new and complementary way for achieving gender equality by giving attention to the overall policy making process. Gender mainstreaming implies the (re)organisation, improvement, development and evaluation of policy processes, so that a gender equality perspective is incorporated in all policies at all levels and at all stages, by the actors normally involved in policy-making (Group of Specialists, 1998: 19). The Group of Specialists on Mainstreaming of the Council of Europe (1998) pointed out from the very beginning that for the development of the mainstreaming strategy a broader concept of equality was needed. They recognised that gender equality is often misunderstood and that the concept implies much more than antidiscrimination and an exclusive focus on women s issues. However they did not recognise at that time that 9

10 gender equality is also a broad and deeply political concept, which can be interpreted in very different ways. The fact that gender mainstreaming in order to be effective requires the implication of very different actors aggravates this complexity of different meanings, understandings, and interpretations of what should be achieved with that strategy. In fact, although the variety of interpretations might enrich the policy making process, in the implementation process different ways of understanding gender inequality at the national, international and sub-national levels, and among different actors within each level are a continuous challenge to previously shared problems and objectives. The label gender inequality (or gender equality for that matter) does not have the same meaning for every actor. Thus, chances are high that each actor will implement different measures depending on his/her own interpretation of the concept, without previously make it explicit what he or she understands as equality because probably he/she would have taken for granted some common understanding of the concept. Therefore, gender inequality is a controversial policy problem and there is no consensus on what is the problem, who is responsible for it, or what could be the solution. Gender equality policies therefore are a good case for an attempt at design evaluation that is not based on assumptions of rationality. All over the world, there exist many different and/or competing interpretations of what is the gender inequality problem and a variety of strategies to solve it. The typical example is how differently framed the prostitution issue is in The Netherlands, where it is conceived as a job in need of better labour conditions, and in Sweden, where it is considered an expression of gender inequality and gender violence, therefore a crime that needs to be eliminated. Especially in the context of the European Union, which creates conditions of multi-level governance, and which has produced its own set of representations of the gender inequality problematic 10

11 (mostly related to labour market issues), such competing and diverging representations are connected to implementation problems. However, the recognition of this complexity of different meanings, understandings, and interpretations does not and could not mean that we need to search for a common interpretation in order to have neat, rational, and straight public interventions, which deal with the real solution to the gender inequality problem. On the contrary, multiple interpretations of every public issue not only gender equality - are at the very heart of democratic processes where different actors are supposed to actively participate. Precisely, the recognition, analysis and exploration of these different interpretations can contribute tremendously to the understanding of policies and the policy making process. To sum up, as it has already been pointed out (Verloo, 2005), implementation problems of gender mainstreaming strategies might be based on non-recognised and non-explored enough multiple interpretations and representations of the concept of gender (in)equality. These different interpretations are expressed in different policy frames. At the same time, the existence of different policy frames of the problem of inequality can explain the difference in gender equality policies across nations and contexts. A policy frame is an organised principle that transforms fragmentary or incidental information into a structured and meaningful policy problem, in which a solution is implicitly or explicitly enclosed, and frame analysis is the analysis of (policy) frames as used in the (re-)construction and negotiation of reality by social or political actors in their discourse (Verloo, 2004:11). 11

12 3. Critical Frame Analysis and the MAGEEQ Methodology With all these assumptions in mind, the MAGEEQ project (Mainstreaming Gender Equality in Europe), motivated by the objective to study divergences in policy frames around gender equality as one of the elements connected to implementation problems, has developed an approach - Critical Frame Analysis - that enables the comparative research of gender equality policies. The purpose of this methodology is mapping the different ways in which an issue is framed, through the analysis and comparison of policy frames and inconsistencies among and within frames in policy documents on gender (in)equality. In other words, with this methodology, we aim to make explicit the diverse interpretations of the gender (in)equality problem by the different political actors in the European context. For this, discursive products and processes are examined using frame analysis. As we have already seen in the definition of policy frame in the former section, policy frames are not simple descriptions of reality, but constructions or specific representations that give meaning to reality and structure its understanding. Therefore, policy frame analysis tries to identify dominant or/and conflicting frames built and used by the socio-political actors for rendering meaning to different situations and events, for attributing causality or blames, and for suggesting different courses of action. (Rein and Schön, 1993, 1994; Triandafyllidou and Fotiou, 1998; Snow and Benford 2000; McAdam, McCarthy and Zald, 1996) Besides social movement theory, which uses frame analysis for studying how social movements assign meaning and interpret social events for mobilising supporters and demobilising antagonists (McAdam, McCarthy and Zald, 1996; Ibarra and Tejerina, 1998), the concept of frame is also partly related to the agenda setting and the problem definition and representation literature (Cobb and Elder, 1983; Aguilar, 1996; 12

13 Kingdon,1995; Bachrach and Baratz, 1970). Bacchi s What s the problem? approach (1999) also reminds us that the definition of public issues that reach the agenda is not a mere description of objective facts, but a strategic representation about what the problem is, not only in what is represented as the problem, but also in what is represented as the solution proposed. Therefore, the MAGEEQ methodology focuses on the analysis of the elements of policy frames, on an assessment of the diversity in problem interpretation and in the solutions given to those problems, and on a critical reflection about (in)consistencies in gender policies. In this way, the analysis aims to identify the frame s (implicit or explicit) internal logic, and the exclusion processes by which some actors do not have voice in the interpretation of a problem and its solution. Typical questions guiding this analysis are: What is the problem and how it is represented? What is the solution offered to this problem? Who has the problem, that is who are the problem holders? Which is the normative group? Which is the target group? Who are responsible for creating or solving the problem? Who has voice in defining the problem and its solution? The focus of the analysis is on describing and comparing policy frames and on detecting inconsistencies between and within frames. Within this approach, it is assumed that the policy frames that are found in policy texts have a typical format of a (representation of a) diagnosis and a prognosis of the tackled problem. Hence, every policy includes a representation of a diagnosis (some social aspect or fact, which is considered problematic and needs changes), and a prognosis (some measures that are proposed as a solution to the problem, specifying what should be done), which are the two main elements in the frame creation process (Snow and Benford, 1988: 199). The MAGEEQ project, a European funded research project (5 th Framework Mis en forme : Interligne : Double, Ne pas ajuster l'espace entre le texte latin et asiatique Programme), which was developed from 2003 through 2006, covered a target period of 13

14 time for studying gender policy frames in six countries and the EU 1, from 1995 (date of the UN Women s Conference in Beijing) to Selected texts were chosen among policy texts such as plans, programs, reports, and legal texts such as norms, laws, etc.-, and parliamentary debates and political texts, such as political parties electoral programs. Other types of texts, such as mass media articles, expert texts and texts from the feminist movement were also used secondarily in the analysis when they were available and they expressed a clear position regarding the issue to be studied. For selecting the texts, a timeline of the chosen period was reconstructed where all the important moments and events regarding each issue were identified. These timelines were done for each country and case study and for each issue to be analysed. The different texts were added to the selection through those key moments and events until the analysis confirmed that other texts were not representing any additional substantial information to the already selected ones. For analysing the texts a set of Sensitising questions was developed (see annex 1). Mis en forme : Interligne : Double These sensitising questions, which are the base for describing and analysing frames, ascertain different dimensions for the diagnosis and prognosis. Besides, this list of questions facilitates a comparative analysis among frames, as for each text in each of the countries studied, answers to the same questions are given. In order to identify and analyse policy frames a qualitative approach is needed. Techniques based on Grounded Theory (Strauss and Corbin, 1997) have been used, such as word and phrase analysis repeatedly used in the texts, word analysis in relation to their context, implicit ideas in the text, and the organization of ideas in different positions within those dimensions. Three common issues related to gender policies were selected, based on their saliency in Mis en forme : Interligne : Double, Ne pas ajuster l'espace entre le texte latin et asiatique the EU and the six countries studied: family policy and, more concretely, reconciliation 1 The contexts of study were Austria, Greece, Hungary, Netherlands, Slovenia, Spain, and also the EU gender policies. 14

15 of working and family life, domestic violence and gender inequality in politics. The three of them are related to three structures organization of labour, organization of intimacy and organization of citizenship -, which reproduce and maintain gender inequality 2. A fourth issue was chosen in each country, so that also immigration, prostitution, anti-discrimination and homosexual rights were analysed, depending on the country case studied. Added to the saliency of the issues in each country, also the richness of material, both in terms of the quantity of documents available and the numbers of actors involved with different perspectives, was used as criterion for selecting these issues. The Critical Frame Analysis methodology used in the MAGEEQ project had two phases. In the first one, selected texts and documents were analysed using the sensitising questions, and the result of the analysis were summarised in English in what we have called a supertext 3 one supertext for each document-. Overall, taking all issues and cases into account, 414 supertexts were produced. Supertexts are a systematic and structured summary, which can be read and understood by someone who did not read the original document. In the second phase, all supertexts are scrutinised by a second analysis, which is comparative, and where the supertexts are the documents to analyse. Mis en forme : Interligne : Double 4. Critical Frame Analysis Methodology as an approach to study divergences in policy frames and to evaluate gender policies and mainstreaming strategies 2 On purpose, we did not want to choose topics related directly to the organization of labour, as the main focus in European gender policies is still on employment and work-related issues. However, issues on citizenship and around the private life of citizens can be expected to be of increasing importance within the European Union 3 A supertext is defined in contrast to the term sub-text. Meanwhile a sub-text is something to be analysed as an implicit meaning not explicitly said in the text, in a supertext the hidden meaning is made explicit. 15

16 Critical Frame Analysis as an approach to study divergences in policy frames can be used for evaluating both gender policies and mainstreaming strategies. This is because it provides a deep description of the content of these policies, that can incorporate inconsistencies and hence does not assume rationality to be always present. In this sense, the supertexts can be seen as an extraction of the program theory of a certain policy. This gives the possibility of doing design evaluation from a different epistemological approach than the traditional program theory and theory driven evaluation that have been linked to more positivistic approaches to evaluation. The frame analysis approach adopted in this study is both constructionist and deconstructionist (Lombardo 2006). It is rooted in an understanding of policy problems as being constructed, as based upon competing interpretations of what is the problem, and on the recognition that policy solutions are in-built in the representation of the problem (Bacchi 1999). In its methodology for mapping the different representations of gender in/equality as a policy problem, it adopts a deconstructionist approach. First of all, it treats gender equality and gender inequality as an empty signifier, studying it as an open concept that can be filled with a multitude of meanings. Also, its sensitising questions do not present an absolute norm against which policies are measured, but it allows for a relative norm (Stake, 1967) Although the questions allow normative assessment, with all their ideological and political implications, they are not absolute models of reference fixed once and for all, but are rather open to periodical revision and transformation in order to better adapt both to the variegated nature of policy texts and to the changes in the researchers theoretical perspectives (Lombardo 2006). Finally, it has open coding, granting more freedom and flexibility for interpreting the specific variations of a text, beyond the positions that are already known. Consequently, even if such an approach has its own difficulties for handling comparisons, this approach has 16

17 more chances to capture unexpected and inconsistent elements of frames that could be left out from the use of a pre-established coding template. In the evaluation of gender equality policies, Critical Frame Analysis allows for the evaluation of content, consistency and coherence in gender policies: How is gender (in)equality defined or represented under this policy? Is this policy really contributing to the goal it is supposed to? Is it really inclusive regarding the actors who are supposed to participate?) At present, its conceptual framework is developed to highlight not only the general elements of any progam theory (diagnosis, prognosis, roles in diagnosis and prognosis, voice), but also specific elements that enable the description of the program theory underneath gender equality policies (dimensions of gender, location, mechanisms, intersectionality, see Annex). Specifically for gender mainstreaming policies, Critical Frame Analysis is suited for both the expert-bureaucratic (where gender impact is to be assessed by specialist, gender experts and/or administrators) and the participatorydemocratic (a range of individuals and organizations are encouraged to contribute) approaches to gender mainstreaming (Beveridge et al., 2000), because it does not only analyze the gender content and frames but also integrates a voice analysis, an analysis of who is included or excluded in the texts or debates, and who is given standing (Marx Ferree 2002). In this sense, Critical Frame Analysis expands the way gender mainstreaming policies have been evaluated so far. Until now, all the scarce attempts to evaluate gender mainstreaming have been developed around the idea of evaluating impact, that is, the impact that this strategy 17

18 might have had in public policies (there is hardly any evaluation yet as to the output in society of the gender mainstreaming strategy). However, if we think of gender mainstreaming as a system in itself, besides evaluating impact, we need to evaluate also how this strategy is designed and implemented. In Carol Weiss terms we would need to make explicit not only the program theory (that refers to the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcomes of interest ( ) Program theory puts the emphasis on the responses of people to program activities ), but also the implementation theory (Does the program in this case the applied strategy - do all the things it intends to do? Implementation implicitly incorporates a theory about what is required to translate objectives into ongoing service delivery and program operation. ) (Weiss, 1998: 57-58) Critical Frame Analysis in its current form is suitable for evaluating gender policies, especially useful for design evaluation of gender public policies evaluating the specificity of these policies, such as the double strategy of tackling specific inequalities and of implementing a long-run gender mainstreaming strategy. But it can also be used to evaluate the gender component of other public policies searching and analysing the gender component of other policies, that is how they address the gender component, not only in the policy impact, but also in the design and implementation of the policy, which then allows to assess if they incorporate a gender equality perspective adequately, or if they rather contain a gender bias. As such, it can increase the awareness of the conceptual prejudices that may unintentionally shape policy discourses, and, consequently, it can reveal latent inconsistencies, or even gender bias, embedded in the design of public policies. Critical Frame Analysis might be applied not only to policy documents but also to other kinds of discourses such as interviews with policy makers or political activists. In a less 18

19 complex version, it might be a methodology for doing participatory evaluation of even very technical program theory or design. Our experience in the dissemination of the MAGEEQ methodology in Spain has been positive in this sense. People working in the every day life of gender programs have said that it is something very useful for them for reflecting on what they are doing. Apparently it can be somehow translated into everyday life reflection, which can be used for the conscious participation of professionals in program design they already participate unconsciously (Lipsky, 1980). If it would be used for design evaluation, to articulate the program theory of not gendered policies, other dimensions would need to be used for the detailing of the general outline on Diagnosis, Prognosis and Roles in Diagnosis and Prognosis. At this moment, in the absence of applications in other fields, it is not clear yet what these other dimensions might be. 5. Challenges and Opportunities for Critical Frame Analysis. There are certainly a number of challenges to the application and the further development of Critical Frame Analysis. As with all methods that use qualitative coding, there are problems of interpretation and comparison, as the result of subjective interpretation and coding. While open end coding can incorporate the wide set of meanings available in the texts, the other side of the coin is that this open-endedness of the coding template creates problems for comparisons, as there are no common codes agreed in advance and the subjective interpretation of codes favours the multiplication of synonymous ways of coding, thus generating confusions and delays in the comparative analysis. In the MAGEEQ project this has been handled by a collective process of revising codes. While this was satisfactory for our project, there is a need to 19

20 further tuning of coding if the method would be used for evaluation. What needs to be addressed is how to avoid that implicit frames of researchers have influence on the coding, Also, there are some limits concerning our issue and text selection, and concerning the possibilities for a dynamical analysis. In Critical Frame Analysis, when the policy frames (or the program theory) of a certain text are analytically articulated in their different components, the biased assumption of rationality is avoided. Rather, the supertext maps out both consistent and inconsistent, both rational and irrational elements. In this sense, CFA is a method that ranks low on normativity. It does not seem to take a position explicitly towards what is good and what is bad in policies This might seem contradictory to a method that aims to evaluate How can an evaluation method not be normative? The answer is twofold. While indeed, at the level of explicitness, CFA does not take a normative stance, and as such allows a judgment that seems separate from the Critical Frame Analysis, simultaneously, at the level of implicit understandings of the method, there are some normative understandings hidden. Asking which dimensions of gender are referred to, and which locations of structures of gender inequality, which mechanisms of (re)producing gender inequality, as well as asking who is given a voice in the policy text, only makes sense if the (normative) idea is that it matters how deep the gender analysis is, that it matters who is given voice. Underneath the sensitizing questions is a positioning of gender mainstreaming as rooted in normative concepts of displacement rather than inclusion or reversal (Squires, 1999) and empowerment (Verloo 2005). It assumes both gender mainstreaming and gender policies should be rooted in the transformative strategy of displacement. Its dimensions of gender and location - organisation of labour, organisation of intimacy and organisation of citizenship - are rooted in feminist literature. Its understanding of gender 20

21 mainstreaming is that this strategy should go beyond gender, displacing patriarchal gender differences and deconstructing discursive regimes that engender the subject (Verloo, 2005). It also assumes that both gender mainstreaming and gender policies should be linked to the concept of intersectionality and the recognition of diversity. The link to empowerment is present as it can help to identify how within the policy making process particular discursive strategies can modify the process itself by means of excluding some actors from the debate (Triandafyllidou and Fotiou 1998: 6.4). Therefore the methodology enables the researcher to give visibility to processes of exclusions which exist in policy discourses, and to assess who is given standing. 21

22 References: Aguilar Villanueva, L. F., (Ed.). (1996). Problemas Públicos y Agenda de Gobierno. Colección Antologías de Política Pública. México, Miguel Angel Porrúa. Grupo Editorial. Bacchi, C. L. (1999) Women, Policy and Politics. The Construction of Policy Problems. London: Sage. Bachrach and Baratz, Power and Poverty. Theory and Practice. Oxford: Oxford University Press. Benford, R. and D. Snow (2000) Framing Processes and Social Movements: An Overview and Assessment. Annual Review of Sociology 26: Bickman, L., (Ed.). (1990). Advances in Program Theory. New Directions for Program Evaluation, 47. San Francisco, Jossey-Bass. Bustelo, María "Gender Mainstreaming Evaluation: Ideas From a Meta- Evaluation Study of Eleven Evaluation Processes of Gender Equality Policies in Spain Evaluation, 19 (4). Pp Cobb, R. W. and C. D. Elder, Participation in American Politics: The Dynamics of Agenda-Building, 2nd ed (Boston: Allyn & Bacon). Chen, H. T Theory-Driven Evaluation: A Comprehensive Perspective. Newbury Park, Sage. Ferree, Myra Marx & Gamson, W., Gerhards, J. and Rucht, D Shaping Abortion Discourse. Democracy and Public Sphere in Germany and the United States. Cambridge: Cambridge University Press. Group of Specialists (1998) Gender mainstreaming; conceptual framework, methodology and presentation of good practices. Strasbourg, Council of Europe. 22

23 Ibarra, P. y Tejerina, B. (1998): Los movimientos sociales. Transformaciones políticas y cambio cultural. Trotta Ed. Madrid Kingdon, John W Agenda, Alternatives, and Public Policies. 2nd Edition. New York: HarperCollins College Publishers. Lipsky, M. (1980) Street Level Bureaucracy. New York: Russel Sage Lombardo 2006 McAdam, Doug, John D. McCarthy, and Mayer N. Zald (eds.), Comparative Perspectives on Social Movements: Political Opportunities, Mobilizing Structures, and Cultural Framings. New York: Cambridge University Press. Osuna, J.L., Márquez, C.: Guía para la evaluación de políticas públicas. Coords./Eds.: Osuna, J.L. y Márquez, C. 168 Páginas. Instituto de Desarrollo Regional. Fundación Universitaria Rein, Martin, and Donald Schön, Reframing Policy Discourse, in Frank Fischer and John Forester (eds.), The Argumentative Turn in Policy Analysis and Planning. Durham: Duke University Press, pp Seigart, D. & Brisolara, S. (2002) Feminist evaluation. explorations and experiences, in: New Directions for Evaluation, 96 (San Francisco: Jossey-Bass). Snow, D. A., & Benford, R. D. (1988). Ideology, frame resonance, and participant mobilization. International Social Movement Research, 1, Squires, Judith (1999) Gender and political theory, Cambridge: Polity Press. Stake, R. E. (1967). The Countenance of Educational Evaluation. Teachers College Record 68: Strauss, A. and J. Corbin (eds.), Grounded Theory in Practice. London: Sage Publications. 23

24 Triandafyllidou, A. & A. Fotiou, Sustainability and Modernity in the European Union: A Frame theory Approach to Policymaking, Sociological research online, vol. 3, no.1. < Verloo, Mieke Gender Mainstreaming Equality in Europe. A Frame Analysis Approach. Paper presented in the Conference of Europeanists in Chicago, March 11-13, Verloo, Mieke Displacement and Empowerment: Reflections on the Concept and Practice of the Council of Europe Approach to Gender Mainstreaming and Gender Equality, Social Politics 12, 3, pp Weiss, C. H. (1998). Evaluation. Methods for Studying Programs and Policies. Second Edition. Upper Saddle River (NJ), Prentice Hall. Wholey, J., (Ed.). (1987). Organizational Excellence: Stimulating quality and communicating value. Lexington, MA: Lexington Books. 24

25 ANNEX Bustelo and Verloo MAGEEQ Methodology of Critical Frame Analysis SUPER-TEXT TEMPLATE NUMBER/CODE/ TITLE (max 20 signs) Full title (In English and in original language) Country / Place Issue Date Type/status of document Actor(s) and gender of actor(s) if applicable Audience Event / reason / occasion of appearance Parts of text eliminated Voice SUMMARY Voice(s) speaking Perspective References: words/ concepts (and where they come from) References: actors References: documents Diagnosis SUMMARY What is represented as the problem? Why is it seen as a problem? Causality (what is seen as a cause of what?) 25

26 Dimensions of gender (social categories / identity / behaviour / norms & symbols / institutions) Intersectionality Mechanisms (resources /norms & interpretations / legitimisation of violence) Form (argumentation / style / conviction techniques, / dichotomies / metaphors / contrasts) Location (organisation of labour / organisation of intimacy / organisation of citizenship) Attribution of roles in diagnosis SUMMARY Causality (who is seen to have made the problem?) Responsibility (who is seen as responsible for the problem?) Problem holders (whose problem is it seen to be?) Normativity (what is a norm group if there is a problem group?) Active / passive roles (perpetrators / victims etc) Legitimisation of non-problem(s) Prognosis SUMMARY What to do? Hierarchy / priority in goals How to achieve goals (strategy / means / instruments)? Dimensions of gender (social categories / identity / behaviour / norms & symbols / institutions) Intersectionality Mechanisms (resources / norms & interpretations / violence) Form (argumentation/style /conviction techniques / dichotomies / metaphors) Location (organisation of labour / intimacy / citizenship) Attribution of roles in prognosis 26

27 SUMMARY Call for action and non-action (who should [not] do what?) Who has voice in suggesting suitable course of action? Who is acted upon? (target groups) Boundaries set to action Legitimisation of (non)action Normativity SUMMARY What is seen as good? What is seen as bad? Location of norms in the text (diagnosis / prognosis / elsewhere) Balance SUMMARY Emphasis on different dimensions / elements Frictions or contradictions within dimensions / elements Comments 27