Rankings: Universities vs. National Higher Education Systems Benoit Millot
Outline 1. Background 2. Methodology 3. Results 4. Discussion 11/8/ 2
1. Background 11/8/ 3
Clear Shift Background: Leagues focus on a handful of U and ignore whole national Higher Education Systems (HES) U21 goes beyond U level (WCU): moves to country and system level U21 is closer to benchmarking initiatives U21 should contribute to increase respectability and usefulness of rankology 11/8/ 4
Questions If results are the same, why? What does it mean? If results are different, why? What does it mean? 11/8/ 5
2. Methodology 11/8/ 6
Problem # 1: Which leagues to select? The 4 leading leagues: ARWU QS THE Webometrics 11/8/ 7
Leagues: Common Pool of Countries Pool: Overlap of countries with a least one university amongst the top 400 in each league 11/8/ 8
Common Pool Australia Austria Belgium Brazil Canada China Czech Republic Denmark Finland France Germany Greece Hong Kong India Ireland Israel Italy Japan Mexico Netherlands New Zealand Norway Poland Portugal Russian Federation Singapore South Africa South Korea Spain Sweden Switzerland Taiwan United Kingdom United States
Problem # 2: How to compare results of analysis carried at 2 different levels? Aggregate the micro data(universities) to the macro data (countries) level Neutralize the effect of country size: Move from number of top U to supply or density of top U Number of top U per HE - aged population 11/8/ 10
United States United Kingdom Germany Canada Australia France Japan China Italy Netherlands Sweden South Korea Spain Switzerland Belgium Israel Brazil Hong Kong Taiwan Denmark Austria Finland Ireland Norway Saudi Arabia Greece New Zealand Poland Singapore South Africa Argentina Czech Republic Hungary India Iran, Islamic Mexico Portugal Russian Top Universities: Numbers vs. Density 40 35 30 China Brazil 25 Number of top 400 Universities 20 USA 15 10 Density of top 400 Universities 5 0 11/8/ 11
Density (QS 2012) 1 Finland 16.1 21 Italy 2.4 2 New Zealand 14.5 22 South K 2.3 3 Switzerland 13.4 23 Taiwan 1.9 4 Ireland 13.3 24 Japan 1.9 5 Denmark 11.5 25 Portuga 1.3 6 Australia 10.8 26 Greece 1.2 7 Hong Kong 10.3 27 Hungar 1.2 8 Norway 9.9 28 Czech R 1.2 9 Netherland 9.0 29 Chile 1.0 10 Sweden 8.6 30 Malays 0.9 11 Belgium 8.4 31 Argenti 0.7 12 UK 7.8 32 Russia 0.3 13 Austria 6.2 33 South A 0.3 14 Canada 5.6 34 Thailan 0.3 15 Germany 5.5 35 Poland 0.3 16 Israel 5.3 36 Mexico 0.1 17 Singapore 4.8 37 Brazil 0.1 18 France 3.8 38 Indones 0.1 19 USA 2.8 39 China 0.1 20 Spain 2.7 40 India 0.04 11/8/ 12
The leaders: Numbers and Density THE 2012 ARWU QS Webo Number of top 400 U USA USA USA USA UK UK UK Germany Germany Germany Germany UK Australia Canada Australia China Canada Australia France Canada Density of top 400 U New Zealand Sweden Finland Switzerland Switzerland Switzerland New Zealand Sweden Ireland Denmark Switzerland Denmark Sweden Netherlands Ireland Finland Denmark Hong Kong Denmark Netherlands 11/8/ 13
University Rankings: Convergence (1) Correlation between results of the 4 leagues: (1) number of top U in each country Nbr of Top 400 Universities: R 2 THE QS ARWU WEBO THE 0.98 0.98 0.96 QS 0.95 0.93 ARWU 0.98 WEBO IREG - Warsaw, 16-17 May 14
University Rankings: Convergence (2) Correlation between results of the 4 leagues: (2) Supply of top 400 U in each country Density of top 400 U: R 2 THE QS ARWU WEBO THE 0.96 0.87 0.78 QS 0.83 0.72 ARWU 0.86 WEBO IREG - Warsaw, 16-17 May 15
3. Results 11/8/ 16
Core Results: U21 and Leagues (2012) U21 and leagues overlap = inter-league overlap Same pool of 34 countries Countries in at least one league but not in U21 (2012) Countries in U21 but not in at least one league (2012) 11/8/ Colombia Argentina Malaysia Estonia Bulgaria Romania Iceland Chile Slovakia Lebanon Croatia Slovenia Oman Hungary Thailand Philippines Indonesia Turkey Saoudi Arabia Iran Ukraine UAE 17
The Winners: Countries Overlap (2012) THE 2012 ARWU QS Webo U21 2012 New Zealand Sweden Finland Switzerland USA Switzerland Switzerland New Zealand Sweden Sweden Ireland Denmark Switzerland Denmark Canada Sweden Netherlands Ireland Finland Finland Denmark Hong Kong Denmark Netherlands Denmark In all 5 rankings In all 4 leagues In 3 leagues and U21 In 2 leagues and U21 Denmark Switzerland Sweden Finland 11/8/ 18
U21 (Overall) & 4 Leagues (Density) - 2012 THE QS ARWU WEBO Overall Indicator 0.74 0.73 0.76 0.76 Country Ranks 0.85 0.88 0.85 0.81 11/8/ 19
U21 (Overall) & ARWU (Density) (1) 1/ U21 2012 & : Two additional countries (S. Arabia and Serbia) Only marginal changes in country rankings Lower overall scores in : 2012 Aver 61.4 58.6 STDEV 14.6 15.3 Max - min 65.5 64.7 <50 12 18 11/8/ 20
U21 (Overall) & ARWU (Density) (2) 2/ ARWU 2012 & : Same number of countries with top 400 Universities Saudi Arabia in (Iran out) Stability of the countries rankings (notwithstanding changes in universities rankings) 11/8/ 21
USA Sweden Switzerland Canada Denmark Finland Netherlands Australia Singapore UK Austria Norway Belgium New Zealand Germany France Hong Kong SAR Ireland Israel Spain Japan Portugal South Korea Czech Republic Taiwan Saudi Arabia Italy Poland Greece Serbia Russian Hungary Argentina Brazil China Mexico South Africa India U21 (Overall) & ARWU (Density) (3) 40 R 2 (country s rank) = 0.88 35 30 25 20 15 10 USA U21 ARWU 5 0 Hong Kong 11/8/ 22
U 21 (Modules) & Leagues (1) Correlation between U21 Indicators and Leagues ( 2012) U21 THE QS WEBO ARWU Overall 0.74 0.73 0.76 0.76 Ressources 0.69 0.69 0.77 0.78 Outputs 0.55 0.57 0.62 0.63 Environmment 0.50 0.49 0.38 0.45 Connectivity 0.73 0.65 0.58 0.63 11/8/ 23
U 21 (Modules) & Leagues (2) Correlation between U21 Indicators and ARWU ( 2012, ) U21 2012 Overall 0.88 0.76 Ressources 0.77 0.78 Outputs 0.81 0.63 Environmment 0.46 0.45 Connectivity 0.79 0.63 11/8/ 24
4. Discussion 11/8/ 25
Obvious Interpretation High Density of WCUs = Excellent HES (???) 11/8/ 26
Prudence!!! 1/ How are U21 countries selected? 2/How are the U21 modules estimated? AND 3/ What does an excellent HES mean? 11/8/ 27
U21: Mode of countries selection U21 selects countries (while leagues select universities) on the basis of: Members of G20 or (+) NSF research ranking 11/8/ 28
U21: Links between the 5 indicators Overall Ressources (25%) Environment (20%) Connectivity (15%) Output (40%) Overall 0.88 0.61 0.77 0.93 Ressources 0.39 0.59 0.73 Environment 0.46 0.51 Connectivity 0.58 Output 11/8/ 29
U21- Heavy Weight of Research U21 - Weights (reconstituted) Research 38% HES 29% Other 25% Governance 8% All 100% 11/8/ 30
Leagues as inputs to U21 U21 attributes include number of WCUs U21 attributes' measures use leagues results (ARWU & WEBO) 11/8/ 31
Conclusion No surprise to find (a) large overlap of countries and (b) strong similarities of results (despite different levels of analysis) Findings do not mean that high number of WCUs make an excellent system which should include quality of teaching, value added, student performance, relevance to the labor market, etc. But they show that a high density of WCUs does contribute to make an elite type of HES 11/8/ 32
Future (1) U21 Objectives = Creating huge expectations U21 Approach = Significant progress in the field of rankology (but ) U21 Results = Confirm Universities rankings U21 Use: Still more to be done to avoid misuse 11/8/ 33
Future (2) Increase in number of countries Lesser weight on research items Better coverage of teaching and other functions of universities Further focus on governance 11/8/ 34
Thank you benoitmillot@gmail.com 11/8/ 35