Approaches and Challenges to Studying Linguistic and Cultural Diversity with International Large-scale Assessments. Heike Wendt & Wilfried Bos

Transcription

1 Approaches and Challenges to Studying Linguistic and Cultural Diversity with International Large-scale Assessments Heike Wendt & Wilfried Bos

2 Agenda Part I: Overview of ILSAs Intents & purposes of ILSAs Sources and types of data Questions of Validity Part II: Exploring Linguistic and Cultural Diversity Operationalization Individual Level Comparative achievement of pupils with/without migrant background Institutional Level (Spec. focus on Germany) School composition and its relationship on student achievement in Germany Part III: Persistent Challenges to Studying Diversity with ILSAs Limited instrumentation and sampling Construct validity Cross-cultural validity Heike Wendt & Wilfried Bos

3 Part I Overview of International, Large-scale Assessments Heike Wendt & Wilfried Bos 3

4 Intents & Purposes of ILSAs Monitor and evaluate and analyze global educational performance across multiple periods and multiple countries (Postlethwaite, 006) to contrast and compare indicators on Core student outcomes, e.g. Reading, Math, & Science for TIMSS/PIRLS Key background factors outside of school system associated with student outcomes, e.g. socioeconomic class, migrant background, etc. Contribute to understanding of functioning, productivity, and equity of education systems individual level: differential educational experience among subpopulations, e.g. students with versus without migrant background school level: in-school practices & processes, e.g. composition effects system level: e.g. early childhood education, dual-language instruction Heike Wendt & Wilfried Bos 4

5 Validity in ISLAs Cross-cultural comparisons require equivalence in design and management of the study, the measured constructs themselves, the measurement instruments, sampling, data collection and processing, data analysis and reporting results. Requirements for validity of ILSAs Structural equivalence Measurement unit equivalence Full scale equivalence Heike Wendt & Wilfried Bos 5

6 Part II Exploring Cultural & Linguistic Diversity with ILSAs Heike Wendt & Wilfried Bos 6

7 Discerning & Collecting Important Data on Achievement with ILSAs: PIRLS Framework Educational Politics and Non-educational Conditions Social and Cultural Capital Socioeconomic Status Educational Level of Parents Migrant Background Internal School Conditions School leadership, faculty and staff, school programs, and school development Teachers Expertise, subject theories, training, interests, etc. Classes Composition, atmosphere, work habits Lessons Out-of-school Context Parental Support Students Individual learning ability Individual learning processes Achievement Outcomes Tarelli, Wendt, Bos & Zylowski (01). Heike Wendt & Wilfried Bos 7

8 Cultural & Linguistic Diversity Defined: Its Instrumentation & Operationalization PISA 01 Q In what country were you and your parents born? (Please tick one box in each column.) Q3 If you were NOT born in <country of test>, how old were you when you arrived in <country of test>? Q 4 What language do you speak at home most of the time? (Please tick only one box.) PI SA 01 PA1 In what country were the following people in the child s family born? (Please tick one answer per column.) PA If the child s father was NOT born in <country of test>, how old was he when he arrived in <country of test>? (If less than 1 months old, please write zero (0).) PA3 If the child s mother was NOT born in >country of test>, how old was she when she arrived in <country of test>? (If less than 1 months old, please write zero (0).) PA4 In what country are the father and mother of the child legal citizens? If they are dual citizens, you may tick more than one country. (Please tick as many as apply.) PA5 What language do the father and mother of the child speak at home most of the time? (Please tick one answer per column.) Items from the student questionnaires of PISA 01 and PIRLS 011 PIRLS 011 G 3 How often do you speak <language of test> at home? Items from the parent questionnaires of PISA 01 and PIRLS 011 PI RLS 011 What language did your child speak before he/she began school? When talking at home with your child, what language does the child s father (or stepfather or male guardian) use most often? What language does the child s mother (or stepmother or female guardian) use most often? (Ch k i l f h l ) Heike Wendt & Wilfried Bos 8

9 Linguistic Diversity In Focus: Percentages of Students by Familial Language Use in International Comparison from TIMSS/PIRLS 011 Test language is not spoken at home Sometimes test language is spoken at home In italics are the participants with restricted comparability. 1= The national target population not or not exclusively the fourth grade equivalent. = The take up rate and / or the exclusion of the national target population do not meet the international standards. Participants with high exclusion rates (> 0 %) are marked with *. 3= The participation rate at school- and/or pupil level do not meet the international specifications. A= In Slovenia the data on the test language was not collected. So there are no results. 1 1 Always or Participants A Never Sometimes almost always % (SE) % (SE) % (SE) Malta 4.8 (0.7) 58. (0.8) 17.1 (0.6) Singapore 6.0 (0.4) 61.6 (0.6) 3.4 (0.5) Taiwan 1.8 (0.) 48.1 (0.9) 50.1 (0.9) Hong Kong 3.9 (0.3) 7.8 (0.9) 68. (0.9) Spain 9.8 (0.6) 1.5 (0.9) 68.7 (1.1) 3 Belgium (Franz. Gem.).3 (0.3) 6.3 (1.3) 71.4 (1.4) International Mean 5.0 (0.1).0 (0.) 73.1 (0.) Canada. (0.) 4.1 (0.9) 73.7 (0.9) New Zealand 1.9 (0.) 3.7 (1.) 74.3 (1.3) Israel* 1.8 (0.) 3.3 (1.0) 74.9 (1.1) Bulgaria 6.9 (1.) 18.1 (1.7) 75.1 (.1) Austria 3.5 (0.3) 19.9 (1.0) 76.6 (1.1) France 1.1 (0.3) 1.3 (1.) 77.6 (1.3) 3 Netherlands 1.6 (0.3) 0.6 (0.7) 77.9 (0.8) England 1.3 (0.) 19.6 (1.6) 79. (1.7) 3 Italy.9 (0.4) 17.8 (0.8) 79.4 (1.0) VG EU 3.1 (0.1) 17.5 (0.) 79.4 (0.) Australia 1.0 (0.1) 19.5 (1.) 79.5 (1.3) Slovakia 3. (0.7) 17.1 (1.) 79.6 (1.6) Sweden 1.5 (0.3) 18.5 (0.9) 80.1 (1.0) Germany 0.8 (0.) 18.9 (1.1) 80.4 (1.1) VG OECD 1.9 (0.1) 16.8 (0.) 81.3 (0.) Norway 1.3 (0.) 17. (1.1) 81.5 (1.1) Denmark 1.1 (0.) 16.9 (0.9) 81.9 (0.9) Lithuania 0.8 (0.) 16.3 (1.1) 8.8 (1.) Ireland.3 (0.3) 13. (0.7) 84.4 (0.8) Russian Federation.5 (0.9) 1.9 (1.0) 84.6 (1.6) Croatia 0.9 (0.) 13.6 (0.9) 85.5 (1.0) USA 1.6 (0.1) 1.5 (0.6) 85.9 (0.6) Czech Republic 0.7 (0.) 1.5 (0.6) 86.8 (0.6) Finland 0.7 (0.) 10. (0.6) 89.1 (0.7) Romania 1.1 (0.3) 9.5 (1.3) 89.4 (1.5) Portugal 0.9 (0.) 9.4 (0.7) 89.7 (0.7) Poland 0.3 (0.1) 8.9 (0.5) 90.8 (0.5) Northern Ireland 1.0 (0.) 7.9 (0.6) 91. (0.6) Hungary 0.5 (0.1).1 (0.) 97.4 (0.3) Schwippert, Wendt & Tarelli (01)

10 Linguistic Diversity & Achievement: Reading Skills of Students Whose Families Always or Almost Always Speak the Test Language versus Sometimes or Never No statistically significant different from the difference in Germany (p >.05). Statistically significant differences (p <.05). In italics are the participants with restricted comparability. 1= The national target population not or not exclusively the fourth grade equivalent. = The usage rate and / or the exclusion of the national target population do not meet the international standards. Participants with high exclusion rates (> 0 %) are characterized with *. 3= The participation rate at school- and/or pupil level do not meet the international spezifications. A= In Slovenia the data on the test language was not collected. So there are no results. B= Inconsistencies in reported differences are due to the rounding process. Schwippert, Wendt & Tarelli (01) Sometimes Always or Participants A or Never Almost Always Differences B % M n (SE) % M i (SE) M i -M n (SE) Bulgaria (8.1) (.9) 6 (7.6) Hungary.6 48 (8.6) (.9) 59 (8.3) New Zealand (4.) (.) 43 (4.9) USA (.4) (1.6) 40 (.9) Romania (8.9) (4.) 40 (9.0) Austria (3.5) (1.9) 36 (3.7) Germany (3.5) (.1) 3 (3.) Sweden (4.1) (.1) 31 (3.8) Italy (3.5) (.3) 31 (3.7) Slovakia (5.7) (1.9) 30 (5.3) Denmark (3.5) (1.6) 30 (3.5) Singapore (3.5) (3.9) 30 (3.1) Finland (5.4) (1.7) 8 (5.) VG EU (1.1) (0.5) 7 (1.1) Malta (1.5) (4.1) 6 (4.5) Ireland (5.1) (.1) 5 (4.6) Spain (3.8) (.3) 4 (3.8) Northern Ireland (6.5) (.4) 19 (6.3) VG OECD (0.9) (0.5) 5 (0.9) 3 Netherlands (3.6) (1.7) 1 (3.1) Lithuania (3.0) (.) 0 (3.5) 3 Norway (3.9) (.1) 19 (4.1) Internat. arithmetic mean (0.8) (0.4) 18 (0.8) Australia (4.5) (.3) 18 (4.8) 3 England (4.5) (.8) 18 (4.6) France (4.0) (.6) 17 (3.6) Belgium (4.) (.7) 16 (3.7) Portugal (4.4) (.7) 1 (4.4) Russian Federation (8.5) (.3) 1 (8.0) Czech Republic (4.3) (.1) 1 (4.0) Canada (.) (1.7) 11 (1.8) Poland (3.9) (.1) 11 (3.9) Israel* (4.4) (.5) 10 (3.8) Hong Kong (3.3) (.3) 5 (.9) Croatia (3.8) (1.8) 5 (3.6) Taiwan (.3) (.) 4 (.5) 10 Average Achievement in reading

11 Diversity Factors Associated with Achievement Empirical evidence: differential educational achievement clear disadvantages for children with migrant background compared to German children Diefenbach (010) Features of children with migrant background or their family Features of the school as an institution Cultural or socioeconomic deficits Migratory situation/ Minority status Contextual conditions of school attendance Institutional discrimination Deficient culture of learning Lower class culture and culture of poverty Lack of human capital Intended repatriation Alternatives within ethnic community Counterculture Fear of stereotyping Effects of type of school Effects of class size and - composition Effects of teachers Enrollment in primary school Transition between primrary and scondary school Assignment to special needs school Language procificency Heike Wendt & Wilfried Bos 11

12 The life-situation of students with migrant backgrounds (associated factors) Percentages of Students with Employed Guardians, in Poverty Conditions, and with Kindergarten Experience by Migrant Background (PIRLS 011) Employed* Under Attended Female Guardian Male Guardian Poverty Line Kindergarten** M igration Background Both parents One parent No parents Familial Language is German Never Almost never Always or almost always * Full or part time ** At least 3 years Wendt, Stubbe & Schwippert (01) Heike Wendt & Wilfried Bos 1

13 Differential Achievement Explained with Social Background Social Background Traits Regressed on Reading Achievement (PIRLS 001, 011) Difference reduced from 4 points Economic and cultural capital can explain much of the achievement gap between students with and without migrant backgrounds Changes in context factors do not explain positive trend in achievement for students with migrant backgrounds 13 Wendt, Schurig, Ünlü & Bos (013)

14 Benchmark I II III IV V Migration Background % M (SE) SD (SE) Cultural Diversity & Achievement: Reading Skills of Students with Different Migration Backgrounds in Germany from PIRLS 001, 006, & 011 (Parents country of birth) PIRLS 001 no parent (1.8) 61 (0.8) one parent (4.8) 71 (.9) both parents (.8) 68 (.1) PIRLS 006 no parent (.3) 59 (1.3) one parent (4.3) 63 (3.3) both parents (4.1) 6 (.0) 011: Students without migrant background achieve on average 4 points more in reading (011: 55) : Students with migrant backgrounds score 15 points higher in 011 than in 001. PIRLS 011 no parent (1.9) 63 (1.5) one parent (4.4) 68 (3.0) both parents (4.0) 64 (.0) percentiles 5% 5% 75% 95% Schwippert, Wendt & Tarelli (01) 14 Confidence interval (+/- SE) around the arithmetic mean

15 Differential Achievement Explained with Social Background Social Background Traits Regressed on Reading Achievement (PIRLS 001, 011) Economic and cultural capital can explain much of the achievement gap between students with and without migrant backgrounds Changes in context factors do not explain positive trend in achievement for students with migrant backgrounds 15 Wendt, Schurig, Ünlü & Bos (013)

16 Differential Achievement Explained with Migration Background Parents with Immigration Background 001 ß (SE) 011 ß (SE) Difference Intercept 479. (4.8) *** (7.9) *** 10.8 *** Parental practise of culture A 18. (7.6) * 16.3 (8.1) * -1.8 *** Students practise of culture B 18.7 (6.0) *** 3. (6.9) ns *** Reading Motivation C 3.5 (6.0) *** 18. (7.8) * *** Reading Behavior D -7.1 (5.9) ns 13.7 (7.6) ns 0.8 *** Adequate possession of books within family E 0. (8.8) ns 4.9 (9.4) * Education level of parents F 34.5 (6.8) *** 19.4 (10.0) * *** R²=.180 R²=.156 ß = regression weights (unstandardised) Level of significance: ns= not significant; *= significant (p <.05); **= significant (p <.01); ***= significant (p <.001) A= Parental practise of culture (0= not high Index; 1= high index) B= Students practise of culture (0= not high Index; 1= high index) C= Reading Motivation (0= not high Index; 1= high Index) D= Reading Behavior (0= otherwise 1= Every day or almoust every day read for the own enjoyment outside the school) E= possession of books as reported of parents (0= maximum 100 books; 1= more than 100 books) F= Parents highest level of education (0= no degree in lower secondary education (Ise); 1= at least one parent with a hiher degree than Ise) Changes in context socio-economic and cultural factors and Home learning environment do not explain positive trend in achievement for students with migrant backgrounds 4.7 *** Schaufelberger, Goy, Tarelli, Wendt & Bos (013)

17 Diversity Factors Associated with Achievement Empirical evidence: differential educational achievement clear disadvantages for children with migrant background compared to German children Diefenbach (010) Features of children with migrant background or their family Features of the school as an institution Cultural or socioeconomic deficits Migratory situation/ Minority status Contextual conditions of school attendance Institutional discrimination Deficient culture of learning Lower class culture and culture of poverty Lack of human capital Intended repatriation Alternatives within ethnic community Counterculture Fear of stereotyping Effects of type of school Effects of class size and - composition Effects of teachers Enrollment in primary school Transition between primrary and scondary school Assignment to special needs school Language procificency Heike Wendt & Wilfried Bos

18 Cultural & Linguistic Diversity at the School Level Primary Schools In Germany By Composition Characteristics Both Parents with a Migration Background Maximum of 5 Books at Home Family Below the Poverty Line Both Parents with Lowest Secondary Certificate or No Certificate % 10% 0% 30% 40% 50% 60% 70% 80% 90% 100% Proportion of Students max. 10 Percent Proportion of Students 10-5 Percent Proportion of Students Percent Proportion of Students Percent Proportion of Students over 75 Percent Smith, Wendt, Kasper & Foy (014) Heike Wendt & Wilfried Bos 18

19 Cultural & Linguistic Diversity at the School Level Within- & Between-School Associations of Diversity with Achievement Intercept Economic Capital (...) Cultural Capital I nstitutional (...) I ncorporated (...) Cultural Familiarity At least one parent born in Germany (1) Language at home is German (1) REA MAT SCI REA MAT SCI REA MAT SCI Within (0.8) (19.3) 35.6 (18.7) -0.1 (3.6) 0.8 (3.3) 8.5* (4.0) 1.8* (3.7) 11.5* (.6) 18.* (3.6) Between 45.* (18.0) 48.1* (17.0) 49.1* (18.5) -8. (18.9) (18.0) -3.5 (16.8) 1 All level- predictors are modelled using the school medians, except internet at home, parents born in Germany and language used, which use school means Smith, Wendt, Kasper & Foy (014) Heike Wendt & Wilfried Bos 19

20 Part III Persistent Challenges to Studying Diversity with ILSAs Heike Wendt & Wilfried Bos 0

21 Persistent Challenges to International Educational Research using ILSAs Limited sample sizes; difficult to differentiate migrant groups due to economic restraints in some countries Over sampling necessary Limited items in questionnaires may in turn result in Construct bias Limited differentiation possibilities Assumption of cross-cultural validity should be empirically explored Heike Wendt & Wilfried Bos 1

22 Germany 0% 6 Quebec 7% 6 Challenge: Limited Cross-cultural Equivalence FIT Index Country Model fix free Difference AUT 0,1 0,1 0,00 RMS 1 SWE 0,11 0,11 0,00 CQU 0,13 0,13 0,00 AUT 0,88 0,87 0,01 TLI SWE 0,89 0,89 0,01 CQU 0,8 0,81 0,01 AUT 0,87 0,87 0,01 CFI 3 SWE 0,88 0,88 0,00 CQU 0,81 0,80 0,01 AUT 7,0 7,00 0,03 WRMS 4 SWE 6,38 6,34 0,04 CQU 7,66 7,57 0,09 AUT 1898, 1965, 67,00 Chi Square 5 SWE 1577,1 1619,85 4,73 CQU 103,10 186,63 83,54 1 : Root Mean Square Error of Approximation (lower value indicating better fit) : Tucker Lewis Index (higher values indicating better fit) 3 : Comparative fit index (higher values indicating better fitt) 4 : Weighted Root Mean Square Residual (lower values indicating better fitt) 5 : Chi Square Value (lower values off Chi Square Differnces indicating better fit of the fix model) Austria Sweden 3% 6 0% 6 6 : Percentage of students who never or only sometimes speak language of test at home In Germany, Austria, and Sweden show equal measurement characteristics for language variable In Quebec, language variable measures migration background differently

23 Persistent Challenges to International Educational Research using ILSAs Limited sample sizes; difficult to differentiate migrant groups due to economic restraints in some countries Over sampling necessary Limited items in questionnaires may in turn result in Construct bias Limited differentiation possibilities Assumption of cross-cultural validity should be empirically explored SEM provides opportunities: BIG effort to facilitate cooperation among theoretical and methodological experts as well as specialists from the field A new possibility to generate further research hypotheses could come out of exploration of the covariance structures in betweenand within-countries in comparative perspective Heike Wendt & Wilfried Bos 3

24 Thank your interest! Kontakt: Heike Wendt & Wilfried Bos 4

25 References Bos, W. & Wendt, H. (008). Bildungsungerechtigkeit in Deutschland: Zur Situation von Kindern und Jugendlichen mit Migrationshintergrund. In Bertelsmann Stiftung (Hrsg.), Integration braucht faire Bildungschancen (S ). Gütersloh: Verlag Bertelsmann Stiftung. Bos, W., Tarelli, I., Bremerich Vos, A. & Schwippert, K. (Hrsg.). (01). IGLU 011. Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich. Münster: Waxmann. Diefenbach, H. (010): Kinder und Jugendliche aus Migrantenfamilien im deutschen Bildungssystem. (Habilitationsschrift, 3. Auflage.) Erklärungen und empirische Befunde. Wiesbaden: VS Verlag für Sozialwissenschaften. Mullis, I.V.S., Martin, M.O., Foy, P., &Drucker, K.T. (01). PIRLS 011 International Results in Reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. PIRLS 011 Questionnaires. 013 International Association for the Evaluation of Educational Achievement (IEA). Publisher: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, Chestnut Hill, MA and International Association for the Evaluation of Educational Achievement (IEA), IEA Secretariat, Amsterdam, the Netherlands. PISA 01 Questionnaires. Download: Schaufelberger, R., Goy, M., Tarelli, I., Wendt, H. & Bos, W. (013, September). Lesemotivation, Leseverhalten und Leseleistungen von Viertklässlerinnen und Viertklässlern mit und ohne Migrationshintergrund im Trend. Erkenntnisse aus IGLU 001, 006 und 011. Poster presented at the Annual Conference of the Arbeitsgruppe für Empirische Pädagogische Forschung (AEPF), Dortmund. Schwippert, K., Wendt, H. & Tarelli, I. (01). Lesekompetenzen von Schülerinnen und Schülern mit Migrationshintergrund. In W. Bos, I. Tarelli, A. Bremerich Vos & K. Schwippert (Hrsg.), IGLU 011. Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich (S ). Münster: Waxmann. Smith, D.S., Wendt, H., Kasper, D. & Foy, P. (014). School composition and student achievement in Germany. Presentation on the occasion of the European Conference on Educational Research (ECER 014), September 0 05, Porto, Portugal. Tarelli, I., Wendt, H., Bos, W. & Zylowski, A. (01). Ziele, Anlage und Durchführung der Internationalen Grundschul Lese Untersuchung (IGLU 011). In W. Bos, I. Tarelli, A. Bremerich Vos & K. Schwippert (Hrsg.), IGLU 011. Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich (S. 7 67). Münster: Waxmann. Wendt, H., Schurig, M., Ünlü, A. & Bos, W. (013, September 10). Narrowed Achievement Gap or Changes in Demographics?: Interpreting Germany s PIRLS Trend Relative to Shifts in Background Characteristics. Presentation on the occasion of the European Conference on Educational Research (ECER 014), September 10 13, Bahçeşehir University, Istanbul, Turkey. Wendt, H., Stubbe, T. C. & Schwippert, K. (01). Soziale Herkunft und Lesekompetenzen von Schülerinnen und Schülern. In W. Bos, I. Tarelli, A. Bremerich Vos & K. Schwippert (Hrsg.), IGLU 011. Lesekompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich (S ). Münster: Waxmann. Heike Wendt & Wilfried Bos 5