社会科学的研究设计. (Research Design in the Social Sciences)

Transcription

1 社会科学的研究设计 (Research Design in the Social Sciences) 学年, 第二学期时间 : 周五下午 ; 地点 :H5( 教 )410 授课老师 : 唐世平 Office Hour: 周四下午, PM ( 文科楼 814) 本课程的核心设计思想 : 让学生从别人的博士论文的优点和缺点中学到如何写好的博士论 文 ( 或者说是, 一本像样的书稿 ) 本课程的前期基础要求 社会科学方法论基础 ( 定性 定量 : 必须有最基本的了解 ) 政治学原理 国际政治原理 比较政治 ( 初阶和进阶 ) 国际政治理论 ( 初阶和进阶 ) 一定的社会科学哲学 科学哲学 历史和其他知识 因此, 你最好是二年级的博士生或者是二年级的硕士生 ( 以后想读博士 ), 而你对你 ( 和他人 ) 的研究感到迷茫 课程要求 完成 80% 以上的阅读 (80% 为英文材料 ), 完成以下作业, 并且积极参与讨论小作业 A: 分两次对 4 本博士论文的研究设计的比较批评 ( 课堂 presentation) 20% 小作业 B: 一篇对两本博士论文的研究技术上的比较批评 ( 短文 5000 字左右 ) 20% 期终作业 : 提交一份研究计划 (20 页纸,1-1.2 万字左右 ) 应该包括 1) 研究问题 为什么研究这个问题?2) 文献综述, 你的 ( 预期 ) 贡献 ;3) 你准备怎么办 : 数据 ( 定性案例 定量数据 ), 为什么用某些数据或者某个案例, 准备如何处理 ;4) 逻辑安排 : 为何这么比较而不是那么比较 这个作业最好就是你的论文刚要 ( 修改后的 ) 50% 课堂讨论 考勤 :10% 特别要求 : 对于有额外阅读的各周, 学生必须在上课时提交读书笔记 最多允许 16 个学生选课 不允许旁听 ( 除非有我的特别许可 ), 因为涉及版权保护 ( 别人的 作品和我的思想即作品 ) 特别澄清: 这样的课程设计并不意味着我是 positivist, 而是我认为硕士和博士期间, 最好还是做点比较容易驾驭的 empirical research 玩更加大的或者 玄 的东西最好等到博士毕业后, 或者是我这种没读过社会科学博士, 因而可以瞎折腾的人 1

2 Text 1. Henry Brady & David Collier. 2004/2010. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham, MD: Rowman Littlefield. 2. Michael Coppedge, Democratization and Research Methods. Cambridge. 3. Shiping Tang Foundational Paradigms of Social Sciences, Philosophy of the Social Sciences. 4. Shadish, Cook, and Campbell Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin, pp Gary Goertz and James Mahoney A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences. Princeton. Strongly 1. David Freedman Statistical Models and Causal Inference: A Dialogue with the Social Sciences. Cambridge. 2. Daniel Steel Across Boundaries: Extrapolation in Biology and Social Sciences. Oxford. 3. Derek Beach and Rasmus Brun Pedersen Process-Tracing Methods. Michigan. 4. Eric Neumayer and Thomas Plumper Robustness Tests for Quantitative Research. Cambridge. 1. John Gerring Case Study Research: Principles and Practices. Cambridge. 2. Alexander George and Andrew Bennett, Case Studies and Theory Development. MIT. 3. Gary Goertz Social Science Concepts. Princeton. 4. Robert Keohane, Gary King, Sidney Verba Designing Social Inquiries: Scientific Inferences in Qualitative Research. Princeton. 5. Charles Ragin Fuzzy Set Social Sciences. University of Chicago Press. 6. Charles Ragin Redesign Social Inquiries: Fuzzy Sets and Beyond. Chicago. 7. Robert Bates et al Analytical Narratives. Princeton. Highly Relevant 1. James Coleman Foundations of Social Theory. Harvard. 2. John Gerring, Social Science Methodology: A Unified Framework. Cambridge. 3. Dani Rodrik ed., In Search of prosperity: Analytical Narratives on Economic Growth. Princeton. 4. Patrick Thaddeus Jackson The Conduct of Inquiry in International Relations: Philosophy of Sciences and its Implications for the Study of World Politics. London: Routledge. 5. Kenneth A. Waltz Theory of International Politics. Addison. 6. Alexander Wendt Social Theory of international Politics. Cambridge. 7. Theda Skocpol State and Revolution. Cambridge. 8. Andrew Abbott Time Matters: On Theory and Method. Chicago. 9. Andrew Abbott Methods of Discovery: Heuristics for the Social Sciences. Norton. 10. George Bennett and Jeffrey Checkel Process-tracing: From Metaphor to Analytical Tool. Cambridge. 2

3 课程安排 Part I: Introduction and Getting the Taste 第一周 : 介绍, 拷贝阅读材料 1. John Gerring Causation: A Unified Framework for the Social Sciences. Journal of Theoretical Politics 17(2): (A more detailed discussion is found in Gerring 2010, part III, Causation ) Reading 1. Michael Coppedge, Democratization and Research Methods. chap Hirschman, Albert O The Search for Paradigms as a Hindrance to Understanding. World Politics 22 (3): Phillip C. Huang "Theory and the Study of Modern Chinese History: Four Traps and a Question." Modern China 24 (2): 第二周 第三周 : 没课, 阅读材料 每本论文需要读 : 概论 理论框架 两章实证 结论 不需要全文读完除了阅读四本论文 ( 两本中文, 两本英文 ) 之外, 学生还要阅读以下材料 Readings for Weeks 2-3: 1. KKV, chap.3, Causality and Causal Inference, pp Ragin, 2000, chaps. 1 & 3, Diversity-oriented Research: Between Complexity and Generality, Studying Cases as Configurations, pp , Bunge, Mario Mechanism and Explanation. PoSS 27 (4): Henry Brady & David Collier Rethinking Social Inquiry, chaps. 3 &4. 批判性阅读指南 -I 1. What were their research questions? Why did they study the questions? 2. What were their causal explanations? Their format (in terms of: Y=a+b1A+b2B+e; or A+B=Outcome1; A-B=Outcome2; -A+B=Outcome3; -A-B=Outcome 4 etc). 3. Were they successful in supporting their causal explanations, empirically? Why a) How were the hypotheses developed? Are their hypotheses closely tied to their theory? b) How was the empirical inquiry designed? (Was the design consistent with the stated theoretical and empirical goals?) c) How was the empirical inquiry actually conducted? i. Was it proper, according to the stated hypotheses? ii. Was it faithful to the stated causal explanations, when presenting empirical evidences? d) Actual empirical Results e) How were the results interpreted (Were the results interpreted correctly)? 3

4 第四周 : 一对学生 PK 各自对两本中文论文的批评 : 赢家通吃第五周 : 一对学生 PK 各自对两本英文论文的批评 : 赢家通吃 Part II: Devils in the Details. 第六周 : 问题 ( 选择 ) 和技术选择 Use the literature on democratization as a case: after more than 40 years of much research, what have we learnt from this huge literature? 1. Shiping Tang. 2016a. Idea, Action, and Outcome: The Objects and the Tasks of Social Sciences, Shiping Tang. 2016b. The Reaches of the Social System: Six Channels, Shiping Tang, Outline of a New Attribution Theory in IR: Dimensions of Uncertainty and Their Cognitive Challenges. Chinese Journal of International Politics. 4. Renate Mayntz, Mechanisms in the Analysis of Social Macro-Phenomena, PoSS 34 (2): Dankwart A. Rustow Transitions to Democracy: Towards a Dynamic Model. Comparative Politics 2 (3): Bertram Malle How the Mind Explains Behavior. MIT, esp. chaps. 1&4 (how people explain others behaviors), chap. 5 (why people explain others behavior in particular ways). 2. Robert Jervis, System Effects (Princeton, 1997), chap. 2, System Effects. 3. C. Mantzavinos Explanations of Meaningful Actions. PoSS (2): Uebel, Thomas Narratives and Action Explanation. PoSS 42 (1): Samuel Hungtington 1991, The Third Wave, chap. 2, Why. 6. Giovanni Capoccia. 2007, Defending Democracy, Chaps. 1-3, 第七周 : 文献批评和你的贡献 ( 附带 : 章节设计 ) ( 如何做好的文献批评 ) 四位同学们比较两本论文的文献批评部分 ( 最好将其他人的做法和我的做法对比一下 ) 1. Shiping Tang The Onset of Ethnic War: a General Theory, pp Shiping Tang Understanding Ethnic Conflict: four waves and beyond, Oxford ERP 课堂讲座 : 我如何做文献批评 4

5 第八周 : 假说和案例选择 (Theory, Hypothesis, and Case selection) 通过阅读以下关于案例 ( 比较 ) 选择的讨论, 四位同学比较两本论文的案例选择部分 重点 : 案例的选择是否恰当, 相对于理论和假说来说? Key Questions: 1. Selection of Cases: Pathway Cases, Difficult cases (as more demanding tests), interesting cases (outliers, e.g., Moldova); positive cases vs. negative cases. 2. Why do you pick your variables (qualitative literature, QCA, quantitative stuff), easy cases (as probing)? Are the cases relevant for testing your variables and mechanisms? 1. John Gerring Case Study Research. Chap. 3, and Epilogue, 37-63, Henry Brady & David Collier Rethinking Social Inquiry, chap. 6, Claiming too much: warnings about selection Bias. 3. James Mahoney and Rachel Sweet Vanderpoel Set Diagrams and Qualitative Research, Comparative Political Studies 2015, Vol. 48(1) Derek Beach and Pedersen Selecting Appropriate Cases When Tracing Causal Mechanisms, Sociological Methods & Research, DOI: / James Mahoney and Gary Goertz The Possibility Principle: Choosing Negative Cases in Comparative Research. American Political Science Review 98, Barbara Geddes How the Cases You Choose Affect the Answers You Get. In Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics. University of Michigan Press, Rachel M. Gisselquist. 2014, Paired Comparison and Theory Development: Considerations for Case Selection, PS (April), Giovanni Sartori, Comparing and Miscomparing. J. of Theoretical Politics 3 (3): John Gerring Techniques for Choosing Cases. In Case Study Research: Principles and Practices. Cambridge, pp Henry Brady & David Collier Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham, MD: Rowman Littlefield. Chs Evan Lieberman Nested Analysis as a Mixed-Method Strategy for Comparative Research. American Political Science Review 99, pp Goertz and Mahoney. 2012, chap. 14, Case Selection and Hypothesis Testing. 8. Special Issue, Sociological Methods and Research, 2016, Case Study Research. 第九周 : 研究设计 III-2: 假说和案例比较研究 (structured comparison;process tracing; and Bayesian Logic) 通过阅读以下关于案例比较的讨论, 四位同学比较两本论文的案例比较部分 重点 : 是否有恰当的 过程追踪?( 相对于理论和假说来说 ) 5

6 1. George and Bennett Chap. 8, Controlled Comparison and Within-case Analysis, 2. David Collier Understanding Process Tracing. PS 44 (4): Beach and Pedersen Process-Tracing Methods. Michigan, chap. 5 & 6, Causal Inference and Process-tracing Methods, & Developing Empirical Tests of Causal Mechanisms, 4. Daniel Steel. 2008, chap. 9. Social Mechanisms and Process Tracing. 5. James Mahoney The Logic of Process Tracing Testing in the Social Sciences. Sociological Methods and Research 41 (4): Anna Grzymala-Busse Time will Tell? Temporality and the Analysis of Causal Mechanisms and Processes, Comparative Political Studies 44 (9): Bennett and Checkel, 2015, Process-tracing, chapters by Waldner, Lyall, Dunning, and Bennett (esp., Disciplining our conjectures: systematizing process tracing with Bayesian analysis, ) 3. Goertz and Mahoney. 2012, chap. 8, Causal Mechanism and Process Tracing 4. George and Bennett Chap. 10, Process Tracing and Historical Explanation, 5. John Gerring Internal Validity: Process Tracing. In Case Study Research: Principles and Practices. Cambridge, pp Dani Rodrik ed., In Search of prosperity. Princeton, chap. 4 Botswana, Chap. 6, India, Chap. 10, Poland and Romania 7. Robert Bates et al Analytical Narratives. Princeton. 8. James Mahoney Strategies of Causal Inference in Small-N Analysis, Sociological Methods and Research 28 (4): 第十周 : 研究讨论 : 研究设计 IV-1: 假说和定量分析 Theoretical, Conceptual and Measurement Issues/Innovations in Quantitative Studies 1. Marcus J. Kurtz and Andrew Schrank Growth and Governance: Models, Measurements, and Mechanisms, American Journal of Political Science 69 (2): Matthew Wilson A Discreet Critique of Discrete Regime Type data, Comparative Political Studies. 3. Ariel Ahram Concepts and Measurement in Multimethod Research. Political Research Quarterly. 4. Andreas Wimmer, Lars-Erik Cederman, and Brian Min Ethnic Politics and Armed Conflict: A Configurational Analysis of a New Global Data Set. American Sociological Review 74 (2): Hui Li and Shiping Tang Location, Location, and Location: The Ethno-geography of Oil and the Onset of Ethnic War, Chinese Political Science Review. 1. Colin Elman Explanatory Typologies. International Organization 59 (2): Schnelder, Andreas Judgment and Measurement in Political Science, Perspectives on Politics 3. Cederman, Wimmer, and Min Why Do Ethnic Groups Rebel? New Data and Analysis, World Politics 62 (1):

7 4. John Gerring et al., Democracy and Economic Growth: A Historical Perspective. World Politics 57 (2): Evan S. Lieberman and Prerna Singh Conceptualizing and Measuring Ethnic Politics: An Institutional Complement to Demographic, Behavioral, and Cognitive Approaches. Studies of Comparative and International Development 47: Kirk Bowman, and James Mahoney, Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America. Comparative Political Studies 38 (8): Seligson, Mitchell The Impact of Corruption on Regime Legitimacy: A Comparative Study of Four Latin American Countries. Journal of Politics 64 (2): Seligson, Mitchell The Measurement and Impact of Corruption Victimization: Survey Evidence from Latin America. World Development 34 (2): Goertz, Gary, Social Science Concepts: A User s Guide (Princeton: Princeton University Press, 2005), chaps Collier, LaPorte, and Jason Seawright Putting Typologies to Work: Concept Formation, Measurement, and Analytic Rigor. Political Research Quarterly. (within appendix) 第十一周 : 研究讨论 : 研究设计 IV-2: 假说和定量分析 Good Practices of Quantitative Research ( 四位同学比较两本论文的定量分析部分 ) 1. James Lee Ray Explaining Interstate Conflict and War: What should be controlled for? Conflict Management and Peace Science 20 (1): 接下来选择以下任何一组论文 政治体制 (political regime) 与经济增长 必读 1. Acemoglu et al Democracy does cause growth, NBER working paper. 2. Pozuelo, Slipowitz, and Vuletin Democracy does not cause growth: the importance of endogeneity argument, Inter-American Development Bank working paper. 3. Tang and Tang Democracy s Unique Advantage in Promoting Economic Growth: Quantitative evidence for a New Institutional Theory, Kyklos, forthcoming, Nov 政治体制 (political regime) 与经济增长 ( 推荐 ) 1. Shiping Tang Hierarchy, Liberty, and Innovation: Democracy s Unique Advantage in Promoting Growth Restated, working paper. 2. Michael Miller Electoral Authoritarianism and Human Development, CPS. 3. Carl Henrik Knutsen Democracy and Growth: A survey of arguments and results, International Area Studies Review. 4. Rory Truex The Myth of the Democratic Advantage, Studies in Comp. Int. Dev. 民主化 : 不平等 ( 内因 ), 外因? 必读 1. Stephan Haggard and Robert R. Kaufman Inequality and Regime Change: Democratic Transitions and the Stability of Democratic Rule, APSR 2. Triesman, Daniel Democracy by Mistake, NBER working paper,

8 3. Seva Gunitsky, From Shocks to Waves, International Organization 68 (2): 民主化 : 不平等 ( 内因 ), 外因?( 推荐 ) 1. Gimpelson and Treisman Misperceiving Inequality. NBER working paper. 2. McDonald, Patrick J Great Powers, Hierarchy, and Endogenous Regimes: Rethinking the Domestic Causes of Peace, International Organization, DOI: /S Strongly 1. Chris H. Achen Let s Put Garbage-can Regressions and Garbage-can Probits Where They Belong, Conflict Management and Peace Science 22: Schrodt, Philip Seven Deadly Sins of Contemporary Quantitative Political Analysis, Journal of Peace Research 3. Aaron Katz, Matthias vom Hau and James Mahoney Explaining the Great Reversal in Spanish America: Fuzzy-Set Analysis versus Regression Analysis. Sociological Methods & Research 33 (4): Pop-Eleches, Grigore Historical Legacies and Post-Communist Regime Change. Journal of Politics 69 (4): 唐睿 唐世平,2013. QCA vs 回归分析, 世界经济与政治 6. Travis Reynolds Institutional Determinants of Success among Forestry-Based Carbon Sequestration Projects in Sub-Saharan Africa, World Development 40 (3): Engerman, Stanley L., and Kenneth L. Sokoloff Factor Endowments, Inequality, and Paths of Development among New World Economies. Economia 3 (1): Acemoglu, Daron, Simon Johnson, James A. Robinson, and Pierre Yared Revisiting the Modernization Hypothesis, Journal of Monetary Economics 56: Charles Boix Development and Democratization. 10. Sven Wilson and David Butler A Lot More to Do: The Sensitivity of Time-Series Cross-Section Analyses to Simple Alternative Specifications. Political Analysis 15: Thomas Brambor, William Ro. Clark, and Matt Golder Understanding Interaction Models: Improving Empirical Analyses, Political Analysis 14 (1): Stephen Chaudion, Helen V. Milner, and Xun Pang International Systems and Domestic Politics: Linking Complex Interactions with Empirical Models, International Organization. 13. Anderson, Christopher J. and Matthew M. Singer The Sensitive Left and the Impervious Right: Multilevel Models and the Politics of Inequality, Ideology, and Legitimacy in Europe, Comparative Political Studies 41: 564. Additional Work: When you have time, read the debate on these two studies AJR (2001; 2002) and Their Critics 1. Acemoglu, Daron, Simon Johnson, and James A. Robinson The Colonial Origins of Comparative Development. American Economic Review, 91 (5): Acemoglu, Daron, Simon Johnson, and James A. Robinson Reversal of Fortune: Geography and Institutions in the Making of the Modern World Income Distribution. Quarterly Journal of Economics 117 (4):

9 3. Matthew Lange, James Mahoney, and Matthias vom Hau, Colonialism and Development: A Comparative Analysis of Spanish and British Colonies, American Journal of Sociology 111 (5): David Y. Albouy The Colonial Origins of Comparative Development: An Empirical Investigation: A Comment. American Economic Review 102 (6): AJRY (2008; 2009) and Their Critics 1. Acemoglu, Daron, Simon Johnson, James A. Robinson, and Pierre Yared Income and Democracy. American Economic Review, 91 (5): Tao et al The Impact of Income on Democracy Revisited. 3. Benhabib et al Reestablishing the Income-Democracy Nexus. 4. Moral-Benito and Bartolucci Income and Democracy: Revisiting the Evidences. 5. Fayad et al Income and Democracy: Lipset s law Revisited. 6. Sascha Claudius An Empirical Analysis of the Relationship between Income and Democracy. 第十二周 : 没课 ( 材料阅读与思考 ) Key Question: Are Qualitative and Quantitative Two Cultures (at War)? 1. Brady 2012; Elman 2012; Goertz and Mahoney, reply 2012, in CPS. 2. Evan Lieberman Nested Analysis as a Mixed-Method Strategy for Comparative Research. American Political Science Review 99, pp 唐世平,2015. 超越定性与定量之争, 公共行政评论,2015 年第 4 期 1. Henry Brady & David Collier Rethinking Social Inquiry, chaps. 4, Beck 2010; Causal Process Observation : Oxymoron or (Fine) Old Wine, Political Analysis. 3. Collier, Brady, and Seawright, Outdated Views of Qualitative Methods: Time to Move on, Political Analysis. 4. Sambanis, Nicholas Using Case Studies to Expand Economic Models of Civil War, Perspectives on Politics 2 (2): Ragin, Fuzzy-Set Social Sciences, Chap. 4, Causal Complexity, Michael Coppedge, Democratization and Research Methods. Cambridge, chaps. 4-6, Andrew Kydd, Trust and Mistrust in International Relations, part II, Fear and the Origins of the Cold War, chap. 3 and chap. 4 Princeton. 第十三周 : Factors, Mechanisms, and Generalization 1. James Mahoney Toward a Unified Theory of Causality, Comparative Political Studies 41 (4/5): Tulia G. Falleti and Julia F. Lynch, Context and Causal Mechanisms in Political Analysis. Comparative Political Studies. 42 (9): Attilia Ruzzene Drawing Lessons from Case Studies by Enhancing Comparability. Philosophy of the Social Sciences 42 (1): Daniel Steel A New Approach to Argument by Analogy: Extrapolation and Chain Graphs. 9

10 Philosophy of Science, 77 (6): Guala, F Extrapolation, Analogy, and Comparative Process Tracing. Philosophy of Science, 77 (6): Strongly 1. Daniel Steel. 2008, chap. 9. Social Mechanisms and Process Tracing. 2. Anna Grzymala-Busse Time will Tell? Temporality and the Analysis of Causal Mechanisms and Processes, Comparative Political Studies 44 (9): Jeffrey W. Lucas, Theory-testing, Generalization, and the Problem of External Validity. Sociological Theory. 22 (3): James Mahoney The Logic of Process Tracing Testing in the Social Sciences. Sociological Methods and Research 41 (4): Questions to be explored: 1) What is wrong with Mahoney s understanding of mechanisms and process-tracing, when considering generalization or extrapolation? 2) What may be the differences between generalizing mechanisms and generalizing factors? 批判性阅读指南 -II ( 秘密武器, 到这才公开 ) 第十四周 : 迈向 因素 + 机制 性的解释 ( 不同方法的互补 )-I: 以族群冲突为例课堂演讲 : 介绍我的族群冲突研究 ( 我是如何设计和开展这个雄心勃勃的研究的 ) 1. Roger Petersen Understanding Ethnic Conflict. Cambridge, chap. 2 and 3, pp Havard Hegre and Nicholas Sambanis, Sensitivity Analysis of Empirical Results on Civil War Onset, Journal of Conflict Resolution 50 (4): Shiping Tang, The Onset of Ethnic War: A General Theory, Sociological Theory. 4. Shiping Tang, Yihan Xiong, and Hui Li Does Oil Cause Ethnic War? Comparing Evidences from Process-tracing with Quantitative Results, Security Studies. 5. Hui Li and Shiping Tang Location, Location, and Location: The Ethno-geography of Oil and the Onset of Ethnic War, Chinese Political Science Review. Strongly 1. Cederman, Lars-Erik, and Luc Girardin Beyond Fractionalization, APSR 101 (1): James Fearon et al Ethnic Minority Rule and Civil War Onset, APSR 101 (1): Shiping Tang, The Security Dilemma and Ethnic Conflict: Toward a Dynamic and Integrative of Ethnic Conflict, Review of International Studies 37 (2): 王凯 唐世平,2013. 安全困境与族群冲突 5. 熊易寒 唐世平, 油田区位和族群冲突的升级, 世界经济与政治

11 1. Sambanis, Nicholas Do Ethnic and Nonethnic Civil Wars Have the Same Causes? A Theoretical and Empirical Inquiry (part 1), Journal of Conflict Resolution 45 (3): Stuart J. Kaufmann Modern Hatred. Cornell University Press. 3. Kaufman, Stuart J Symbolic Politics or Rational Choice? Testing Theories of Extreme Ethnic Violence. International Security 30: Collier, David, and Anke Hoeffler Greed and Grievance in Civil War. Oxford Economic Papers 56: Collier, Paul, Anke Hoeffler, and Dominic Rohner Beyond Greed and Grievance: feasibility and civil war, Oxford Economic Papers 61: Paul Collier et al On the Duration of Civil War, Journal of Peace Research 41 (3): Evan S. Lieberman and Prerna Singh Conceptualizing and Measuring Ethnic Politics: An Institutional Complement to Demographic, Behavioral, and Cognitive Approaches. Studies of Comparative and International Development 47: Part III: Show me what you have learned 第十五周 + 第十六周 : 介绍的期末作业的初步设想和计划 ( 类似于你的开题报告 ) 11