Maciej Jakubowski Faculty of Economic Sciences, University of Warsaw, Poland email: mjakubowski@uw.edu.pl Test scores and income inequalities Preliminary version Not be cited without the author s permission Introduction International surveys in education became to be a popular source of information about student performance in economics. Increasing number of countries is participating in these surveys which extends opportunities for international studies. There are three surveys which cover more than 40 countries. These are PISA conducted by OECD, and TIMSS and PIRLS conducted by the consortium created by International Association for the Evaluation of Educational Achievement (IEA). This paper focuses mainly on PISA study because it covers all OECD countries which are of main interest here. The Programme for International Student Assessment was launched in 2000, followed by surveys in 2003 and 2006. Next waves are planned for 2009, 2013 and 2015. PISA is focused on measuring and comparing achievement of 15-year-olds in domains of reading, mathematics, and science. In PISA 2000 there were 43 countries participating and in 2006 the number of countries increased to 57. Testing framework for PISA is based on the former experiences of IEA with sophisticated procedures developed to satisfy comparability across countries. Obviously, there are many critics who claim that strict comparability cannot be achieved, because of discrepancies in goals of participating countries school systems and problems with test translations. While such criticism could be compelling we are assuming here that PISA measures some general abilities which are needed in any modern society and economy. Thus, we abstract from the divergence of what schools in different countries are teaching. We are focused on what general skills students have at the end of compulsory education, which is in most countries about the age PISA surveys. We are interested whether distribution of these skills is affecting income inequalities. At the same time we are fully aware of methodological problems with the comparability of PISA results when one wants to relate them to economic outcomes. These methodological problems are analyzed below and this is one of the contributions of this paper. The main goal of the paper is to relate score distribution observed in PISA to income distribution. More specifically, inequalities in skills are related to inequalities in income. Thus, we focus on methodological problems in comparing scores and income. The crucial difference between those two variables is that the later is real reflecting factual distribution in the population and its reliability depends solely on proper reporting and coding of relevant information, while the former is derived through IRT models, with imposed distribution usually normally shaped. In other words, test scores are reflecting latent traits and their distribution cannot be interpreted as representing factual distributions of skills or abilities in the population. Moreover, while income is measured on the interval scale, test scores are sometimes assumed
Jakubowski - Test scores and income inequalities 2 to be measured this way, however, many researchers doubt this. This have important consequences of measurement of inequality. As a consequence, typical measures of inequalities used in income studies could be invalid in educational studies despite the fact that many researchers are using them. We investigate this point indirectly by comparing different inequality indices of test scores with Gini coefficients of income distributions. The presumption is that if there are measures which strongly correlate at the country level, then that could mean that they are helpful, if not fully valid, as measures of educational inequalities.. The paper is organized as follows. The next section describes PISA survey in more details emphasizing the differences with income studies. Following section describes datasets used in the study. Section III discusses methodology, results, and possibilities for future studies. Section IV concludes. I. PISA and methodology of achievement surveys in education PISA (the Programme for International Student Assessment) was jointly developed by participating OECD countries. The goal of PISA is to assess student achievement based on the highly reliable and internationally comparable testing framework. PISA reliability is build on detailed description of assessment methodology, cautious procedures of translation and supervision of country specific implementations, and careful calibration of student scores using modern psychometric tools and rich set of collected background variables. The number of students participating in the survey depends on the country and wave with the median value about 5000-6000 students tested per country. The samples are believed to be representative to the population of 15-year-old students. Survey is conducted using 2-stage stratified sampling with primary sampling units at the school level and random draw of students within schools 1. Survey weights are provided by organizers to reflect different probability of schools and students to be sampled. Some countries were excluded from the official publications because they did not meet rate of response standards set by organizers, however, in most cases data for these countries are available in a publicly provided datasets and were analyzed in this research. Similar framework is used in two others well known international educational surveys: TIMSS and PIRLS. In fact, developers of these two surveys have the biggest experience in conducting internationally comparable assessments in education because TIMSS 1995 was the first large scale international survey in education. Similarly to PISA, these surveys benefit from international expertise in developing tests and setting survey standards. There is one crucial distinction between PISA and TIMSS. It is said that while PISA tries to assess the knowledge needed in future life, particularly on the labor market, TIMSS is assessing knowledge in relation to curriculum or what is declared as needed by school systems in participating countries. In practice this distinction is far from obvious because while it is hard to imagine any testable knowledge in mathematics which is not related to curriculum, it is also hardly discussable that there are mathematical skills taught in schools which are not correlated with knowledge, skills or abilities needed in adult life (even if not directly useful) (see Neidorf et al., 2006, for a discussion of similarities and discrepancies between achievement surveys). Nevertheless, if one believes this distinction then PISA is the survey she should use when relating to labor market and economic outcomes. PISA is conducted every 3 years. Data for 2000, 2003, and 2006 were already published. Each wave have different main domain. In 2000 that was reading literacy, in 2003 mathematics, and in 2006 science. This means that achievement in mathematics was measured in the most detailed way in 2003, for example, but scores for other domains were also produced. In this paper we assume that measurement of inequalities is 1 In fact, in some countries PISA have 3 stage sampling scheme but stage 1 indicators are not provided in the public datasets so we cannot used them. See Rabe-Hesketh, Skrondal, 2006, for the analysis of American PISA data where 1 st stage sampling is done over geographical regions.
Jakubowski - Test scores and income inequalities 3 so demanding that one could rely solely on scores for main domains. Thus, we measure inequalities in reading based on scores from 2000, in mathematics from 2003, and in science from 2006. Organizers claim that reading scores are comparable across all waves, while mathematics is comparable only between 2003 and 2006, and science scores from 2006 should not be related to earlier studies. Thus, even pooling the data from all waves to increase precision is not advisable in the case of mathematics and science. We decided then to estimate educational inequalities in each domain from distinct waves, but also to pooled the comparable scores from different waves. This means obviously that we are assuming that inequalities didn t change across waves. That is not true for many countries, including Poland, but still in most cases magnitude of change is relatively small comparing to sustaining differences across countries. Crucial to see the problem of comparability of test score and income distribution is to understand the methodology of obtaining achievement scores. In all educational surveys Item Response Theory (IRT) models are used to produce standardized score scales. The process is described in technical reports, however, it is still quite difficult or even impossible to repeat it using the information provided by organizers. While items data are available, specific models used in scale estimation are not presented. It should be emphasized that scaling process details affect outcomes and competing models could produce dissimilar results. It was shown that for non-oecd countries switching to other IRT model could dramatically change their position in rankings, especially when not the mean performance but measures of score dispersion are analyzed. Such statistics are sensitive to changes at the extremes of distribution and different IRT models could produce different distributions in this regard (see Brown et al., 2005). We don t know any research analyzing the robustness of more demanding estimation to choices in IRT scaling. While for OECD countries statistics from different surveys are quite agreeable, there are also important discrepancies which in some cases could be non negligible (see Micklewright, Schnepf, 2004, 2006; Jakubowski, 2007). Scales obtained from IRT models are then used to produce plausible values which are used in international comparisons of student literacy. Plausible values are draws from possible distributions of true scores. There are five plausible values in PISA datasets (similarly in TIMSS or PIRLS). If one wants to estimate any kind of population parameter then she has to estimate a model five times, separately with each plausible value. The average of five estimates should be used as a final estimate and variation among them should be taken into account when estimating standard errors. Taking simple average of plausible values, which is often done in economic studies, and then using it in the model could bias estimates of the parameters and surely underestimates standard errors. While this is not of importance when one is simply interested in a mean score for a country, this is more important when one wants to estimate percentiles or regression models and becomes crucial if score variance or inequality measures are to be obtained. Researchers often take the first plausible value and use that in the model. This approach is better than taking the average of plausible values but still not fully valid from a theoretical point of view and plausible values methodology (see technical reports for PISA, TIMSS or PIRLS for details of estimation and use of student achievement plausible values, and references given there). In this paper we repeated calculations for each of plausible values. This way percentiles and mean scores are identical to those presented in official reports. Summing up, PISA scores are estimated taking into account differences in items (test questions) difficulties and country-specific effects. Plausible values based on IRT estimates of latent traits are standardized to have mean 500 and standard deviation 100 among OECD countries. Using plausible values one can obtain unbiased estimates of the population parameters which are comparable across countries. Standardizing scores for OECD countries assures comparability between waves of PISA study despite the fact that the number of participating non-oecd countries is changing. However, there are still two points worth mentioning. Firstly, while PISA defines the population of interest as all 15-year-old students, countries differ in the way they sample students from the population. In some countries 15-year-old students in some grades or school types were not sampled. There are discrepancies in grade and age distributions across countries, and for the same countries across waves 2. Official comparisons and most of the 2 For example, in PISA 2000 sample there are only 9 th grade students in the Polish sample, while in 2003 and 2006 there are also students from the 7 th, 8 th, and 10 th grade. The number of students not in modal grade could be regarded as negligible (less than 5%)
Jakubowski - Test scores and income inequalities 4 researchers do not consider these discrepancies directly when mean score comparisons are made. However, they should be taken into account in international studies. In practice, we found the percentage of migrants affects country score distribution importantly. While international comparisons could take into account migrants, this is not fully valid in studies relating achievement to labor market outcomes. In this case migrants should be treated as separate group because as we argue below achievement scores in the test written in the official language of the country are not good predictors of their future labor market careers. II. Datasets PISA estimates are based on publicly available micro data from PISA 2000, 2003, and 2006. These datasets could be obtained from the official PISA OECD website. Datasets contain data even for countries which were excluded from the official reports because of not meeting response rate benchmarks. We used this data because exclusion was based on somehow arguable criteria and it was shown that at least in the case of UK additional weighting for non-response didn t change the results considerably (see XXX). Moreover, for some analysis (see below) we combined the data from all waves to limit the impact of wave-specific measurement and survey issues. Statistics used in further country-level analysis were calculated in three ways. Firstly, we estimated statistics of interest for reading based on the data from 2000, for mathematics based on 2003, and for science based on 2006. Thus, we used test scores in the main domain of each PISA study assuming that only the measurement with the highest precision could be used to judged countries score distributions 3. Secondly, we used all test scores in a particular domain from the waves with comparable scores. According to PISA organizers reading is comparable through all waves, but mathematics in 2006 could be compared only to 2003, while science in 2006 should not be compared to earlier results. Thus, in the second approach we pooled scores for reading from 2000, 2003, and 2006. For mathematics we used scores from 2003 and 2006. For science again only from 2006. Finally, we also used unweighted mean of student score in all domains to calculate statistics based on pooled data from all waves. This way all possible scores were used which clearly make results more robust to any discrepancies across measurement in scientific domains and PISA surveys 4. We called this mean scores all scores in the following sections. PISA data were merged with country-level statistics including and Gini coefficient of income inequality. data were taken from the World Bank, and Gini indices were taken from the United Nation s University World Income Inequality Database. variable was constructed as a mean of 2000-2006 yearly data. Gini of Income variable is a median of separate estimates from the most recent three-years of data after excluding observations not of the highest quality available for the particular country 5. Other ways of calculating single indicators of or Gini were also tried (e.g., average of all available values, Gini coefficients only based on Luxembourg Income Study etc.) but have no visible impact on final estimates. and the fact that grade distribution changes is usually overlooked. But reading literacy mean scores for the whole sample are 480, 497, and 508, in 2000, 2003, and 2006, respectively. However, for the sample of modal grade (9 th grade) similar mean scores are 480, 502, and 514. This is not negligible difference in some comparisons (see Jakubowski, 2008). 3 Obviously, one needs more items (test questions) to assess score distributions that simply calculating mean scores. IRT models are not that robust in measurement at the tails of latent distributions. So, to calculate percentiles at the extremes precise individual data are needed. 4 In 2003 and 2006 scores were imputed for all students regardless whether they solved the test booklet in a particular domain. In 2000 scores in some domains were missing for some students. For these observations only non-missing scores were used. 5 World Income Inequality Database indicates 4 levels of quality (1-4). If a country has observations of quality=1 then observations of lower quality were not considered at all. For the 57 countries present in PISA and in WIID database 39 have Gini from the highest quality surveys available. For 16 countries quality was marked as 2 and for two countries as 3 (Serbia and Tunisia). None of the countries analyzed here have the lowest quality of information according to the WIID database.
Jakubowski - Test scores and income inequalities 5 Final dataset used in the analysis contained observations for 57 countries which were present in one of the waves of PISA study and have Gini coefficients available in the WIID database. For some countries it was also not possible to indicate migrants in the PISA student datasets (e.g. Korea in 2000). Thus, statistics for this countries were not used if the sample without migrants was analyzed. The number of countries involved in analysis is presented in the result tables. III. Methodology, results, and ideas for future studies We use weighted student data to calculate percentiles and score means of achievement. All statistics were calculated five times, separately for each of plausible values, and then the average of them were taken as the final estimate. Thus, all statistics reflect survey design and plausible values imputation. Percentiles were then used to calculate differences between them which are believed to quantify educational inequalities in the best possible way if scores are assumed to be measured on the ordinal scale (see Micklewright, Schnepf, 2006). The P95-P5, P90-P10, P80-P20 differences were employed. Moreover, we also calculated the P50- P10 and P90-P50 differences to have some insides on whether the top or the bottom of score distribution is related to income inequalities. Finally, we also calculated Gini coefficients of achievement scores. While scores are clearly not measured in the interval scale, similar to the scale of Euro, it could be argued that standardization of scores across OECD countries make these statistics comparable across countries. One cannot evaluate their magnitude in relation to income inequality but could look at correlation or employ it in the regression. To make it more clear, we strongly agree that it is not valid to compare the magnitude of Gini for income with the magnitude of Gini for test scores (see Bedard, Ferrall, 2003, for analysis of this kind). However, we argue that one use as a relative measure of skills inequality between countries and to relate it to income inequality measures in a regression framework. As students in most of the OECD countries have similar mean scores this approach is reliable for these countries. Probably it is less valid to compare countries with dramatically different mean scores because IRT scaling squeezes scores at the tails. Thus, analysis was repeated for all countries and separately for OECD countries. We start with basic analysis of correlation between country-level Gini income coefficients and different measures of inequality in test scores, including Gini and percentile differences. Results are presented in the table below. There are four samples used for comparisons. First one includes all the countries with relevant income indicators and all students from these countries for which PISA plausible values in a main domain were available. Second sample uses scores for all students as before but only in OECD countries. Third sample uses data for all countries for which migrant status was available in PISA datasets. In this sample PISA based statistics were calculated only for non-migrant students. Fourth sample, considers again native students but only in OECD countries. For each sample there are two columns. In the first column correlation with Gini of income is presented and in the second correlation with the. Consider first the correlation between Gini of income and. Generally, correlations are stronger if only achievement of non-migrants is considered. This is in line with earlier findings about migrants role in the distribution of achievement and income (see Blau, Kahn, 2005). Intuitively, migrants can perform worse on the achievement tests because of language, and different educational or cultural background. However, they could be well paid in jobs which do not need language-related skills or if they work with other migrants. Thus, achievement tests are often weak predictors of their average future labor market outcomes. In our analysis including migrants could be very misleading because they often perform in the bottom of the distribution heavily affecting any statistics of inequality. Other important fact is that correlation for mathematics is stronger than for reading or science. While there are obvious explanations when reading and math are considered, including language problems but also bigger difficulties with translating reading test questions, it is hard to explain why inequalities in science
Jakubowski - Test scores and income inequalities 6 correlate less than in mathematics. That poses interesting questions about the comparability of scores in separate scientific domains in the context of comparisons with labor market outcomes. Table 1 contains also correlation of Gini indices with percentile differences. Percentile differences are visibly correlated only with the, while in most cases there is none or even negative correlation with the Gini of income. Relations for the sample of OECD countries are quite consistent, but for non-oecd countries there are apparent discrepancies among scientific domains with visibly weaker correlation for math. We found it very difficult to explain these differences because mathematical characteristics of different indices are mixed with problems related to achievement and income measurement. Maybe the key finding to understand how different indices work in this case is a much stronger positive correlation of P90-P50 with Gini coefficients, quite consistent across all samples and domains. Is inequality in a top part of the achievement distribution crucial for income inequalities or is that an artifact created by combining separate inequality measures which differently react to issues of scale incomparability in this case? At this moment we do not have a proper answer to this question. Nevertheless, this fact seems to be a good starting point for further, more in-depth analysis, which probably will have to based on individual observations for income and test scores. Such data were not available for this study. Thus, based on stronger correlations of Gini indices we decided to concentrate on them in what follows, having in mind that proper analysis should take into account possibly non interval scale of test scores. Table 1. Correlation between income inequalities and test score inequalities. All countries OECD countries Gini Income All students Non-migrant students All students Non-migrant students Gini of test scores Gini Income Gini of test scores Reading Gini Income Gini of test scores Gini Income 0.33 0.43 0.20 0.45 Gini of test scores P95-P5-0.10 0.56-0.03 0.48-0.02 0.82 0.12 0.66 P90-P10-0.10 0.59-0.03 0.52 0.01 0.85 0.16 0.72 P80-P20-0.11 0.63-0.03 0.58 0.04 0.86 0.21 0.77 P50-P10-0.29 0.36-0.28 0.23-0.15 0.72-0.09 0.51 P90-P50 0.17 0.79 0.30 0.76 0.24 0.87 0.47 0.77 N 42 42 41 41 28 28 27 27 Mathematics 0.71 0.76 0.61 0.70 P95-P5 0.02 0.31 0.16 0.33 0.04 0.59 0.17 0.58 P90-P10-0.04 0.24 0.10 0.25-0.02 0.52 0.11 0.51 P80-P20-0.12 0.16-0.01 0.14-0.11 0.42-0.01 0.37 P50-P10-0.25-0.05-0.17-0.11-0.24 0.23-0.19 0.10 P90-P50 0.32 0.61 0.42 0.64 0.30 0.74 0.40 0.75 N 39 39 39 39 30 30 30 30 Science 0.38 0.47 0.32 0.48 P95-P5-0.35 0.48-0.27 0.46-0.28 0.58-0.17 0.53 P90-P10-0.38 0.44-0.30 0.41-0.25 0.60-0.13 0.56 P80-P20-0.41 0.40-0.34 0.37-0.26 0.59-0.15 0.54 P50-P10-0.51 0.24-0.45 0.19-0.44 0.37-0.34 0.27 P90-P50-0.09 0.64-0.01 0.63 0.17 0.76 0.25 0.77 N 53 53 53 53 30 30 30 30
Gini index: income 20 30 40 50 Jakubowski - Test scores and income inequalities 7 Correlation coefficients could be misleading in such heterogeneous and small sample of countries. Scatter plots presented below could be more helpful. In this case we used Gini indices of test score inequality based on all student data comparable across PISA waves in a particular domain. Thus, for reading we pooled the data for 2000, 2003, and 2006, for math 2003 and 2006, and for science only 2006 results were utilized. Moreover, we used pooled data from all waves with student mean score in all domains as another way of summarizing PISA test scores. This way we tried to make analysis more robust. To make graphs clearer only data for OECD countries were presented. There are huge discrepancies in Gini coefficients, but obviously also in other characteristics, between OECD and non-oecd countries. Thus, simple two way analysis is not helpful in this case. On the scatter plots observations and fitted simple regression lines are presented for sample of all students (squares and solid line) and separately for non-migrant students (triangles and dash line). Excluding migrants shrinks toward the international mean. Moreover, fitted regression lines are steeper emphasizing the point that measurement of achievement for migrants do not reflect their future labor market prospects in a proper way. Scatter plots make also clear that there are strong outliers according to Gini coefficients of income but also of test scores. Turkey, USA and Mexico are OECD countries with visibly different income and test score distributions. Generally, while the positive relation between Gini indices is visible, there are also non negligible differences in this relation between countries and domains. For example, Gini coefficient in reading for US is slightly higher than the average, near the average in mathematics, but the highest in science. Germany have strong inequalities in reading, but only when migrants are included. If not, Gini coefficients are around OECD mean, and even below in science. There are also countries with very stable position, with exceptional Finland where inequalities in both scientific achievement and income are relatively low. Finland is so exceptional because this the top scorer in all PISA waves. Detailed data on Gini coefficients is given in the table A1 in the appendix (including non- OECD countries). Figure 1. Gini indices for test scores in reading and income. MEX MEX USA TUR USA KOR KOR FIN FIN PRT PRT POL POL NZL NZL GBR GBR ITA IRL IRL AUS ESP JPN AUS CHE CAN CHE ESP JPN CAN ITA GRC GRC FRAHUN FRA DEU BEL NOR NOR BEL NLD NLD LUX CZE CZESVK LUX AUT AUT ISL ISL DNK DNK SWE SWE DEU 8 9 10 11 12 13 Gini index: reading test scores Full sample: Linear fit Observed Without migrants: Linear fit Observed
Gini index: income 20 30 40 50 Gini index: income 20 30 40 50 Gini index: income 20 30 40 50 Jakubowski - Test scores and income inequalities 8 Figure 2. Gini indices for test scores in mathematics and income. MEX MEX USA USA TUR FIN FIN PRT POL NZL NZL KOR GBR GBR IRL IRL CHE JPN AUS AUS ESP ESP CHE CAN CAN FRA DEU NORFRA HUN BEL NLD LUX NLD SVK LUX CZE CZE AUT AUT ISL ISL DNK DNK SWE SWE ITAGRC DEU BEL 8 9 10 11 12 13 Gini index: mathematics test scores Full sample: Linear fit Observed Without migrants: Linear fit Observed Figure 3. Gini indices for test scores in science and income. MEX MEX TUR USA USA PRT PRT FIN FIN POL GRCNZL NZL KOR ITA ITA GBR GBR ESP IRL AUS AUSCHE CAN CHE ESPIRL JPN CAN HUN DEU BEL NORBEL CZE DEU NOR FRA FRA LUX NLD NLD SVK CZE LUX AUT AUT ISL ISL DNK DNK SWE SWE 8 9 10 11 12 Gini index: science test scores Full sample: Linear fit Observed Without migrants: Linear fit Observed Figure 4. Gini indices for test scores in all domains (unweighted) and income. MEX MEX USA USA TUR PRT POL NZL NZL KOR GBR GBR GRC ITA GRC ITA IRL IRL CHE ESP ESP AUS CHE JPN AUS CAN CAN HUN NOR DEU FRANOR FRABEL NLD LUX NLD CZE CZE LUX FIN FIN AUT AUT ISL ISL DNK DNK SWE SWE DEU BEL SVK 8 9 10 11 12 Gini index: all test scores Full sample: Linear fit Observed Without migrants: Linear fit Observed
Jakubowski - Test scores and income inequalities 9 Obviously simple bivariate analysis cannot be that convincing. Thus, we tried regression with additional predictors of inequalities in income. With cross-sectional data and low number of observations one cannot expect that multivariate analysis will be robust. Several simple specifications were tried with Gini of income as a dependent variable, and, PISA mean score, and per capita (in PPP $) as predictors 6. Mainly, we wanted to see which set of variables could be useful and what are the differences between scientific domains and samples considered. Regression results are presented in two tables. Table 2 presents results for all available countries, while Table 3 only for OECD countries. Analysis confirms that inequalities in test scores are better predictors of inequalities in income when migrant students are excluded. Moreover, these simple models are better predictors for OECD countries where for obvious reasons homogeneity in test score and income distribution is higher, but also measurement is more reliable. In the group of all countries are able to explain something between 20 to 30% of variance in income inequalities. In OECD countries models based on non-migrant student scores are able to predict from 20 to 47%. With and mean scores as additional controls models have equally good explanatory power for both groups of countries but in many cases estimated coefficients are no longer significant. If one agrees that the most robust information comes from the pooled data for all scientific domains then these simple models perform quite well, especially when migrant students are excluded. In all regressions with no-migrant students coefficient for remains significant and positive when per capita is included, but becomes insignificant when mean scores are included. Gini coefficient for test scores is strongly correlated with mean scores, but we do not know whether that indicates any causal relation or is simply a product of mathematical dependencies among this two statistic in the case of IRT scaled test scores. This an important issue which should be studied more carefully. Regressions are quite stable and robust to specifications, especially in the case of OECD countries and nonmigrant students. Even these simple models are able to explain almost half of the variance in income inequality across countries. This confirms the strong relation between inequalities in test scores (skills) and in income. Obviously, with so small sample there are issue of robustness of the results. One could argue that USA, Mexico, and Turkey, are visible outliers and should not be analyzed with other OECD countries. In fact, after excluding these countries correlation between Gini of income and becomes insignificant in many cases. However, similar argument could be made about Asian and European countries. Clearly, these are good arguments against simple analysis like the one presented here, but with this data any sophisticated analysis is not possible. One should treat these results as preliminary investigation into the relation between income and skills using international data. We believe that there are more crucial arguments against this approach. We related income inequalities measured in adult population to that observed in the 15-year-old student population. It was assumed that score and income distributions are stable across cohorts which is surely untrue. However, the question is how unstable are this relations across time and whether any changes within countries are stronger than long-term differences between countries. We know that income inequalities have own dynamic, but also know that there are quite stable between countries differences (see Atkinson, 2007). We also know that even across PISA waves achievement inequalities changed within countries. e.g. in Poland or Germany. However, despite these changes there are still evident differences in educational inequalities between for example Anglo-Saxon and Scandinavian countries. 6 We tried different specifications, e.g. with primary school participation rate, OECD dummy etc., but results were very unstable.
Jakubowski - Test scores and income inequalities 10 Table 2. Regression analysis for all countries. Full sample Dependent variable: Gini index of income inequality Sample without migrants (1) (2) (3) (4) (5) (6) (7) (8) 1.98*** (0.42) 11.81* (4.98) 0.28 (0.64) -0.08** 70.12*** (18.26) 1.17** (0.43) *** 27.55*** (6.02) Reading 0.52 (0.68) -0.04 * 53.19* (21.27) 2.23*** (0.45) 9.49 (5.16) 0.70 (0.70) -0.07* 60.11** (19.40) 1.41** (0.49) ** 24.25*** (6.83) Adj. R-squared 0.24 0.33 0.37 0.37 0.31 0.37 0.38 0.39 N 57 57 55 55 57 57 55 55 2.51* (1.14) 7.94 (12.99) -0.61 (1.19) -0.12*** 99.67*** (28.25) 1.52 (0.96) ** 26.14* (12.62) Mathematics -0.48 (1.13) -0.10* 91.62** (29.33) 2.74* (1.08) 5.97 (12.22) -0.31 (1.38) -0.11** 92.84** (32.35) 1.65 (1.05) * 24.43 (13.86) Adj. R-squared 0.22 0.46 0.42 0.50 0.28 0.47 0.43 0.50 N 54 54 52 52 54 54 52 52 2.59* (1.02) 7.12 (11.42) -0.05 (0.67) -0.12*** (0.02) 94.93*** (13.95) 2.02* (0.83) *** 21.67* (9.98) Science 0.44 (0.73) -0.09*** 77.74*** (16.36) 3.20** (0.97) 1.19 (10.72) 0.41 (0.73) -0.12*** (0.02) 86.62*** (15.25) 2.12* (0.89) *** 20.28 (10.76) Adj. R-squared 0.13 0.50 0.43 0.52 0.21 0.52 0.44 0.53 N 53 53 52 52 53 53 52 52 2.77** (1.01) 8.27 (10.34) 0.16 (1.06) -0.12*** 91.34*** (22.43) 1.81 (0.94) *** 24.67* (10.72) All scores 0.35 (1.07) -0.10** 79.98** (23.90) 3.10** (0.97) 5.53 (9.77) 0.47 (1.13) -0.11*** 85.06*** (23.84) 1.95 (1.00) ** 23.13* (11.41) Adj. R-squared 0.16 0.40 0.37 0.43 0.21 0.42 0.38 0.44 N 57 57 55 55 57 57 55 55 Robust standard errors in parentheses 0.76 (0.73) -0.04 49.62* (21.66) -0.30 (1.26) -0.10* 88.02** (31.57) 0.61 (0.76) -0.09*** 76.32*** (16.77) 0.51 (1.11) -0.09** 77.67** (24.24) * p<0.05, ** p<0.01, *** p<0.001
Jakubowski - Test scores and income inequalities 11 Table 3. Regression analysis for all countries. Dependent variable: Gini index of income inequality Full sample Sample without migrants (1) (2) (3) (4) (5) (6) (7) (8) Reading 2.12 (1.33) -0.88 (1.60) 1.96 (1.02) -0.05 (1.38) 3.42* (1.47) 0.95 (1.92) 2.73* (1.24) 1.00 (1.79) 7.90 (14.19) -0.15 (0.08) 113.99* (53.66) 16.63 (10.48) -0.10 (0.06) 86.93 (44.71) -5.22 (15.34) -0.10 (0.08) 70.66 (57.21) 7.11 (13.84) Adj. R-squared 0.08 0.21 0.21 0.24 0.20 0.24 0.24 0.25 N 30 30 30 30 30 30 30 30 Mathematics -0.07 (0.07) 62.02 (50.64) 3.99** (1.33) (0.94) 3.40** (1.16) -0.04 (0.89) 4.67*** (1.13) 0.96 (1.18) 4.24** (1.28) 0.73 (1.25) -10.69 (13.57) -0.14*** 102.52*** (23.38) 0.27 (12.93) -0.13** 100.01*** (23.46) -16.51 (11.22) -0.12** 81.67** (28.84) -9.96 (14.95) Adj. R-squared 0.28 0.47 0.32 0.47 0.40 0.50 0.39 0.48 N 30 30 30 30 30 30 30 30 Science -0.12** 84.21** (27.95) 3.05* (1.46) 0.45 (2.11) 4.27* (1.84) 1.87 (2.16) 4.57** (1.57) 1.76 (2.11) 4.16* (1.76) 2.00 (2.13) -1.51 (15.26) -0.14** 95.16* (37.51) * -5.33 (18.90) -0.10* * 66.62 (35.10) -16.44 (15.84) -0.12** 74.79 (37.72) -5.70 (18.85) Adj. R-squared 0.07 0.37 0.31 0.42 0.20 0.43 0.31 0.43 N 30 30 30 30 30 30 30 30 All domains (unweighted) -0.10* 65.78 (35.74) 4.94** (1.74) 2.13 (2.34) 4.39* (1.68) 2.36 (2.53) 5.93*** (1.58) 3.95 (2.42) 5.34** (1.73) 3.77 (2.72) -15.96 (16.18) -0.13 (0.07) 75.13 (52.60) -5.25 (16.49) -0.10 (0.07) 62.83 (53.44) -23.91 (14.35) -0.08 (0.07) 34.19 (54.39) -15.67 (17.22) Adj. R-squared 0.30 0.39 0.37 0.41 0.43 0.45 0.43 0.44 N 30 30 30 30 30 30 30 30 Robust standard errors in parentheses -0.07 (0.07) 33.31 (56.54) * p<0.05, ** p<0.01, *** p<0.001
Jakubowski - Test scores and income inequalities 12 While analysis presented in this paper should be seen as preliminary attempt to relate income and test score inequalities, the need for more detailed studies is clear. One should look for the opportunity to use micro data for income and achievement. With individual data it could be possible not only to construct more reliable student samples but also to relate them to more comparable adult samples. That was already done with IALS data (see Blau, Kahn, 2005; Devroye, Freeman, 2001). However, IALS provides data only for small number of the most advanced economies and cannot be compared in coverage and precision with studies like PISA, TIMSS or PIRLS. Moreover, with individual income data one could try to estimate inequalities for the cohorts which are the most similar to students tested in PISA, basically, for the youngest cohorts. In few years students tested in PISA 2000 will finish their education and will have already some labor market experience. Relating PISA student scores to labor market outcomes for the same cohorts should give much more reliable and interesting results. In future it will be even possible to track student cohorts across time within countries which should increase the possibility to observe how distribution of skills is affecting income inequalities. It is also needed to carefully analyze theoretical models of the relation between school and income inequalities. While results presented here suggest that there is link between inequalities in test scores and income, it is not that obvious how this relation looks like. On the empirical grounds, only individual panel data could provide reliable results in this case, but such datasets are available only in few countries and according to our knowledge no international study like this is planned. Thus, we have to deal with existing cross-sectional data and need to indicate school system and labor market institutions that could diminish or emphasize the effect of skill inequalities on income. As a final word we should mention that it is also not clear what is the direction of potential causal relationship. Surely, educational inequalities reflect persistent inequalities in adult population in ability, social and cultural background. These are strongly affecting student (child) performance. However, if schools have power to limit the impact of parental characteristics on human capital development then adult population inequalities should have smaller effect than in the system where inherited characteristics are emphasized. One could also argue that in the country where returns to education (or skills) are higher the motivation for parents and students is also higher. That could extend, but could also limit, educational inequalities depending on how motivation is distributed among students. Moreover, we studied only outcomes in the secondary education, while higher education systems could have profound impact on final skills distributions (see Cascio et al., 2008). Thus, beside the obvious direct effect of lower skills on personal income, there are indirect effects which could be non-negligible. It is dubious whether that could be studied with the country-level data. IV. Summary This paper made an attempt to analyze Gini indices of inequality for both income distributions and test scores distribution. Country-level data on Gini income coefficients available from the United Nations project were merged with statistics calculated from the student level PISA OECD study datasets. For test scores data not only Gini coefficients were estimated but also other measures of inequality based on percentile differences. While it was argued that percentile statistics are more valid in the case of test scores, which are probably measured on the ordinal scale, it was also demonstrated that Gini coefficients for income and test scores are visibly correlated. The test scores inequality measures based on percentiles were not significantly correlated with income inequalities except the P90-P50 difference. One possible interpretation is that skills inequalities in the top of test score distribution are of importance for income inequalities but we believe that one should be very careful with similar interpretations in this case. Because we have no individual income data, more detailed analysis was not possible and we left this observation for future studies.
Jakubowski - Test scores and income inequalities 13 It was also shown that simple regression models perform quite well, especially when migrant students are not considered in estimation of test score inequalities. That is in line with intuition given in earlier studies that achievement tests for migrants could be misleading if one takes them as an indicator of skills needed on the labor market, especially when language plays a role in assessment. Excluding migrants makes regression analysis more stable and simple models presented in the paper are able to explain almost half of the between-country variance in income inequalities. This study surely poses more questions than gives answers. While relation between test scores and income is of importance for educational and labor market policies, there are only few studies dealing with these issues. The attempts presented in the paper are preliminary. Future studies could employ micro level data for students and merge them with representative income studies for the same cohorts. That would make similar analysis more robust and could answer more detailed questions about the relation between skills acquired in schools and distribution of income using international evidence. References Atkinson, Anthony B., 2007, "The distribution of earnings in OECD countries", International Labour Review, vol. 146 Vol. 146 (2007), No. 1 2 Bedard Kelly, Ferrall Christopher, 2003. "Wage and test score dispersion: some international evidence," Economics of Education Review, Elsevier, vol. 22(1), pages 31-43, February. Blau Francine D., Kahn Lawrence M., 2005. "Do Cognitive Test Scores Explain Higher U.S. Wage Inequality?," The Review of Economics and Statistics, MIT Press, vol. 87(1), pages 184-193, December. Brown, Giorgina, Micklewright, John, Schnepf, Sylke V. and Waldmann, Robert, 2005."Cross-National Surveys of Learning Achievement: How Robust are the Findings?". IZA Discussion Paper No. 1652 Cascio Elizabeth, Clark Damon, Gordon Nora, 2008. "Education and the Age Profile of Literacy into Adulthood," NBER Working Papers 14073, National Bureau of Economic Research, Inc. Devroye Dan, Freeman Richard B., 2001. "Does Inequality in Skills Explain Inequality in Earnings Across Advanced Countries?," NBER Working Papers 8140, National Bureau of Economic Research, Inc. Jakubowski Maciej, 2007. Effects of tracking on achievement growth. Exploring difference-in-differences approach to PIRLS, TIMSS and PISA data. RSCAS-EUI discussion paper Micklewright, John & Schnepf, Sylke V., 2004. "Educational Achievement in English-Speaking Countries: Do Different Surveys Tell the Same Story?," IZA Discussion Papers 1186, Institute for the Study of Labor. Micklewright, John and Schnepf, Sylke V., 2006. "Inequality of Learning in Industrialised Countries". IZA Discussion Papers 2517, Institute for the Study of Labor (IZA). Neidorf, T.S., Binkley, M., Gattis, K. and Nohara, D, 2006. Comparing Mathematics Content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 Assessments. U.S. Department of Education. Washington, DC: National Center for Education Statistics. OECD, 2001. Knowledge and Skills for Life: First Results from PISA 2000. OECD, Paris. OECD, 2002. PISA 2000 Technical Report, OECD, Paris OECD, 2005, PISA 2003 Technical Report, OECD, Paris Rabe-Hesketh Sophia, Skrondal Anders, 2006. "Multilevel modelling of complex survey data," Journal Of The Royal Statistical Society Series A, Royal Statistical Society, vol. 127(4), pages 805-827.
Jakubowski - Test scores and income inequalities 14 Appendix Table A1. Gini indices of test scores and income for all countries Whole sample No migrants Gini of read math science all read math science all income ALB 16.2.. 8.6 16.0.. 8.5 4897.1 29.6 ARG 17.1 14.9 14.6 12.1 16.9 14.8 14.5 12.0 9182.9 50.5 AUS 10.6 10.0 10.8 9.3 10.4 9.8 10.6 9.1 30297.1 30.9 AUT 11.5 10.7 10.8 9.7 10.8 10.3 9.9 9.3 31470.0 26.0 AZE 11.1 5.5 8.1 6.7 11.0 5.5 8.1 6.7 3258.6 50.6 BEL 12.1 11.7 11.0 10.5 11.4 10.9 10.5 10.0 30005.7 28.0 BGR 15.1 13.7 14.0 11.2 14.6 13.7 13.9 11.1 8001.4 39.0 BRA 14.2 14.8 13.0 12.3 14.2 14.8 13.0 12.3 7562.9 57.1 CAN 9.9 9.2 9.9 8.3 9.6 9.0 9.6 8.2 31572.9 30.1 CHE 11.0 10.5 10.9 9.3 9.8 9.6 9.8 8.6 35767.1 31.0 CHL 13.0 12.0 11.8 9.7 13.0 12.1 11.9 9.7 10117.1 55.2 COL 15.8 13.4 12.4 12.0 15.8 13.4 12.4 12.0 5167.1 55.8 CZE 11.6 11.0 10.9 9.8 11.2 10.7 10.8 9.5 17070.0 26.8 DEU 12.7 11.4 11.0 10.5 11.2 10.3 10.1 9.5 28648.6 28.0 DNK 10.4 9.7 10.6 8.9 10.1 9.5 10.2 8.7 31132.9 24.0 ESP 10.6 10.3 10.5 9.1 10.5 10.2 10.2 9.0 24788.6 31.5 EST 9.5 8.8 8.9 8.6 9.4 8.8 8.9 8.6 12928.6 35.5 FIN 8.6 8.5 8.6 8.1 8.5 8.4 8.5 8.0 27830.0 26.0 FRA 11.1 10.6 11.7 9.8 10.8 10.2 11.4 9.5 28820.0 28.0 GBR 11.0 10.2 11.8 9.6 10.8 10.1 11.6 9.5 29210.0 33.0 GRC 12.2 11.7 11.1 9.8 12.1 11.7 10.9 9.7 25765.7 33.5 HKG 9.0 10.0 9.6 8.9 8.7 9.8 9.4 8.7 31431.4 42.2 HRV 10.6 10.1 9.9 9.6 10.5 10.1 9.8 9.6 11244.3 31.3 HUN 11.0 10.6 10.0 9.3 10.9 10.6 9.9 9.3 14060.0 27.7 IDN 11.2 12.3 10.0 10.4 11.0 12.1 9.9 10.4 2741.4 36.6 IRL 9.9 9.4 10.6 8.7 9.8 9.3 10.4 8.6 29038.6 32.0 ISL 11.0 9.9 11.2 9.2 10.8 9.8 11.0 9.1 30101.4 25.0 ISR 14.6 13.7 13.9 11.0 13.9 13.4 13.5 10.7 20425.7 38.1 ITA 11.8 11.7 11.4 9.9 11.5 11.5 11.2 9.7 26430.0 33.0 JOR 13.1 12.2 12.0 11.6 12.8 11.9 11.8 11.3 3937.1 36.4 JPN 11.0 10.2 10.6 9.5 10.8 10.2 10.5 9.5 28572.9 31.6 KGZ 20.3 15.6 14.5 14.8 20.1 15.4 14.4 14.7 1514.3 47.4 KOR 8.4 9.6 9.8 8.7 8.8 9.6 9.8 8.7 18900.0 33.0 LTU 11.5 10.4 10.5 10.2 11.5 10.4 10.4 10.1 10822.9 33.7 LUX 12.3 10.7 11.3 9.8 10.4 9.6 9.7 8.9 53494.3 27.0 LVA 11.3 9.9 9.7 8.9 11.1 9.9 9.7 8.8 10694.3 37.5 MEX 12.9 12.3 11.2 10.9 12.7 12.0 11.0 10.7 10401.4 51.0 MKD 14.3.. 8.1 12.7.. 7.8 6661.4 28.6 NLD 9.9 9.7 10.4 8.9 9.4 9.2 10.0 8.6 33020.0 27.0 NOR 11.8 10.6 11.3 9.7 11.4 10.4 10.8 9.4 43767.1 28.0 NZL 11.4 10.4 11.5 9.8 10.8 10.1 11.1 9.4 21508.6 33.7 PER 16.6.. 8.2 15.9.. 8.1 5494.3 49.8 POL 11.3 10.2 10.3 9.2 11.2 10.1 10.3 9.2 11790.0 35.2 PRT 11.5 10.9 10.6 9.4 11.4 10.8 10.5 9.4 18517.1 38.0 ROU 13.6 11.4 10.9 10.0 13.6 11.4 10.9 10.0 7857.1 36.3 RUS 11.7 10.9 10.6 9.3 11.7 10.8 10.5 9.3 9781.4 45.1 SRB 12.9 12.0 11.1 11.3 12.9 11.9 11.1 11.2 7365.7 36.8 SVK 11.9 10.6 10.8 10.4 11.8 10.6 10.8 10.4 13608.6 27.0 SVN 10.0 10.0 10.7 9.7 9.9 9.9 10.5 9.6 20000.0 24.0 SWE 10.4 10.3 10.6 9.1 9.9 9.9 10.1 8.8 29185.7 23.0 THA 10.5 11.1 10.4 9.1 10.6 11.0 10.3 9.1 6067.1 44.5 TUN 14.4 13.6 12.1 12.2 14.3 13.5 12.0 12.1 5418.6 40.6 TUR 12.0 13.0 11.0 11.5 12.0 13.0 11.0 11.5 6731.4 47.6 TWN 9.6 10.6 10.1 9.7 9.5 10.5 10.0 9.6. 33.9 URY 16.3 13.2 12.4 12.6 16.2 13.1 12.3 12.6 8038.6 45.2 USA 11.6 11.0 12.4 11.0 11.1 10.8 12.1 10.7 38605.7 46.4 YUG 11.1 11.0. 10.1 11.0 11.0. 10.1. 33.5 Mean 12.1 11.1 11.0 9.9 11.8 10.9 10.8 9.7 18740.4 35.7 Min 8.4 5.5 8.1 6.7 8.5 5.5 8.1 6.7 1514.3 23.0 Max 20.3 15.6 14.6 14.8 20.1 15.4 14.5 14.7 53494.3 57.1 Mean OECD 11.1 10.5 10.8 9.6 10.7 10.3 10.5 9.3 26670.4 31.4 Min OECD 8.4 8.5 8.6 8.1 8.5 8.4 8.5 8.0 6731.4 23.0 Max OECD 12.8 13.0 12.4 11.5 12.7 13.0 12.1 11.5 53494.3 51.0