Skills to occupational mismatch Session 3: Measuring skills demand and skills supply Composite Indicators Michaela Saisana michaela.saisana@jrc.ec.europa.eu European Commission Joint Research Centre Econometrics and Applied Statistics Unit 1
Statistics' best known face (to general public & media) THE 1.3m people of Mauritius love to prove famous people wrong. On independence from Britain in 1968, pundits such as a Nobel prize-winning economist, James Meade, and a novelist, V.S. Naipaul, did not give much of a chance to this tiny, isolated Indian Ocean island 1,800km (1,100 miles) off the coast of east Africa. Its people depended on a sugar economy and enjoyed a GDP per person of only $200. Yet the island now boasts a GDP per person of $7,000, and very few of its people live in absolute poverty. It once again ranks first in the latest annual Mo Ibrahim index, which measures governance in Africa. And it bagged 24th spot in the World Bank s global ranking for ease of doing business the only African country in the top 30, ahead of countries such as Germany and France. How does it pull it off? 16/10/2008 2
Statistics' best known face (to general public & media) 3
Definition: A composite indicator is formed when individual indicators are compiled into a single index, on the basis of an underlying model of the multi-dimensional concept that is being measured. (OECD Glossary of statistics) 4
(p.7): the role of statistical indicators has increased over the last two decades (i) more literacy, (ii) more complexity, (iii) more information society 5
Almost ten-fold increase since 2000 14,000 12,000 10,000 8,000 6,000 Scholar Google hits on "composite indicators" 7780 11800 4,000 3420 2,000 29 1550 0 1940 1960 1980 2000 2020 6
Quality of composite indicators Michaela Saisana, Andrea Saltelli, Stefano Tarantola, Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators J. R. Statistical Society A, 2005, 168(2):307-323. Paolo Paruolo, Michaela Saisana, Andrea Saltelli, Ratings and Rankings: Voodoo or Science? J. R. Statistical Society A, 2013, 176(3):609-634. Michaela Saisana, Beatrice d Hombres, Andrea Saltelli, Rickety Numbers: Volatility of university rankings and policy implications Research Policy, 2011, 40:165-177 7
Composite indicators: In house (most recent) 8
About 60 audit requests in the last 10 years (most recent) 9
Good practices on composite indicators 10
1/Place index in policy context (ex-ante) European Policy Context Definition: Civic competence (CC) is the ensemble of knowledge, skills, values and attitudes needed to be an active citizen EU countries agreed that CC is one of the 8 key competencies required from education and training for personal fulfilment, active citizenship, social cohesion and employability in a knowledge society. [European Parliament and Council 2006] 11
2/Link to theory Theories (models) for Civic Competence Models of citizenship in Europe Liberal Communitarian Civic republican Critical citizenship Hoskins, Saisana, Villalba, Measuring civic competence across Europe: ten years on, Comparative Education Review, 2013, forthcoming 12
Cognitive dimension Builds on all three models; higher level of knowledge, skills from the civic republican model; emphasis on critical thinking from the critical model that facilitate active involvement in decision making; may be influenced by liberal market emphasis on basic skills of reading literacy and ability to take autonomous decisions. 4 1 A model: Measuring Civic Competence at Youth 3 Social Justice Builds on the liberal, cosmopolitan and critical models; citizenship values of human rights and respecting equality and diversity. In addition, liberal values of respecting the democratic procedures. Citizenship values Builds on the civic republican model; traditions of civic responsibility, value placed upon engagement and notion of good participatory citizen. 2 Participatory attitudes Builds on the civic republican model; traditions of the individual disposition towards engagement. Broader than just national politics, includes critical citizenship aspects of protest and liberal activities such as volunteering. 13
2/Link to theory Data: ICCS 2009 Civic Competence Index (CCCI-2) Citizenship values Social Justice Participatory Attitudes Cognition about Democratic Institutions Democratic values (DEMVAL) Political and social issues (INTPOLS) Conventional citizenship (CITCON) Socialmovement related citizenship (CITSOC) Equal Rights for all ethnic/racial groups (ETHRGHT) Equal rights for immigrants (IMMRGHT) Gender Equality (GENEQL) Internal political efficacy (CITEFF) Legal protest (LEGPROT) Electoral participation (ELECPART) Political Participation (POLPART) Informal participation (INFPART) Knowledge and skills (PVCIV) Value of participation at school (VALPARTS) [Source: Hoskins, C. Villalba, M. Saisana, 2012] Self-efficacy (CITEFF) 14
3/Discuss what is not captured Not included in the CCCI-2 (because not yet covered in IEA studies): Civic republican qualities: solidarity, awareness of others, public spiritedness Critical citizenship qualities: empathy and care Cognitive qualities needed for critical reflection on social structures and power relations 15
4/Check for statistical coherence Statistical grouping of indicators into dimensions Civic Competence dimension Citizenship values Social justice Participatory attitudes Cognition about democratic institutions Civic competence indicator Principal Component 1 2 3 4 Conventional citizenship.303.124.760 -.137 Social-movement related citizenship.120.372.700 -.009 Democratic values.007.489.271.388 Equal Rights for all ethnic/racial groups.136.759.158.179 Equal rights for immigrants.102.816.060.020 Gender Equality -.043.440 -.096.642 Value of participation at school.169.444.402.185 Political and social issues.568 -.088.500.077 Internal political efficacy.653 -.104.385.166 Legal protest.714.277 -.018.017 Electoral participation.553.105.283.355 Political Participation.751.017.073 -.240 Informal participation.806.100.135 -.119 Self-efficacy.678.152.187.065 Knowledge and skills -.022.069 -.026.885 Note: Loadings after orthogonal rotation with Kaiser normalisation 38 countries 140,000 students (gr.8) 16
4/Check for statistical coherence Methodology CCCI-2 PCA & FA results: confirmed the conceptual grouping of indicators into four dimensions of civic competence; provided statistical justification for the use of a simple arithmetic average to combine the indicators within a dimension; and suggested that the two indicators related to equal rights for minorities (ETHRGHT, IMMRGHT) in the Social Justice dimension should count as one indicator. 17
4/Check for statistical coherence Correlations between dimensions are low dimensions capture distinct aspects of civic competence with practically little or no overlap of information between them. The Cognition dimension has low correlation to the Social justice dimension but no association to Citizenship. Hence young people with strong cognitive skills can have either strong (or low) citizenship and Participatory attitude and vice versa. Correlations:CCCI-2 and dimensions INDEX Citizenship Social Participatory Cognition INDEX 1.000 0.724 0.750 0.694 0.484 Citizenship 0.724 1.000 0.368 0.486-0.005 Social 0.750 0.368 1.000 0.260 0.395 Participatory 0.694 0.486 0.260 1.000 0.023 Cognition 0.484-0.005 0.395 0.023 1.000 Note: Pearson correlation coefficients (individual level, n=140,000). Spearman rank correlations are of the same order. Coefficients less than 0.1 are not statistically significant at the individual level. 18
What about the aggregation formula for the main dimensions of an index? In the CCCI-2 we went for a simple arithmetic average of the 4 pillars (several arguments in favour, yet ) 19
5/Carefully select aggregation formula Score Social 5 Participatory 9 Citizenship 2 How do we combine these dimensions into a single number? Arithmetic Average Geometric Average y n i 1 x i w i y n i 1 w x i i 20
5/Carefully select aggregation formula S P C Arithmetic Average Geometric Average Comparison with B A 7 7 7 7.0 7.0 B 5 9 2 5.3 4.5 B1 10 9 2 7.0 5.6 26.0% B2 5 9 7 7.0 6.8 51.8% Advantages of geometric versus arithmetic mean for the main dimensions of an index 1) implies only partial compensability, i.e. poor performance in one dimension cannot be fully compensated by good performance in another, 2) rewards balance by penalizing uneven performance between dimensions, 3) encourages improvements in the weak dimensions, i.e. the lower the performance in a particular dimension, the more urgent it becomes to improve in that dimension. 21
6/Do uncertainty analysis: which countries are most volatile? How coupled stairs are shaken in most of available literature How to shake coupled stairs 22
6/Do uncertainty analysis: which countries are most volatile? Conceptual framework and country-specific frameworks for civic competence Note: Color-codes indicate that the indicators belong to the same statistical group 23
Spain, Greece, Bulgaria (improve) Italy, Ireland, Norway (decline) Austria, Switzerland, Slovakia (unaffected) Most countries lose positions in the country-specific framework wrt the CCCI-2 24
7/Do sensitivity analysis: which assumptions have the highest impact? Use uncertainty and sensitivity analysis during the process of building an index, not only to criticize an existing one! Focus discussions on assumptions that matter the most. Source: The Innovation Indicator, European Commission 2013 25
8/Deconstruct an index At indicator level At country level 26
9/Link to policy (ex-post) Framework (WHO report) Sensitivity analysis Policy message Source: The Alcohol Policy Index (2007, PLoS Medicine, 4(4):752-759) 27
Conclusion In many contexts, e.g. when monitoring the mismatch between skills supply and demand, it is of added value to develop relevant and sound 28
(Composite) Indicators More reading at http://composite-indicators.jrc.ec.europa.eu 29