Automated Democracy Scores Thiago Marzagão *PhD candidate, Dept. of Political Science, Ohio State University,

Transcription

1 DRAFT. PLEASE ASK BEFORE QUOTING. Automated Democracy Scores Thiago Marzagão *PhD candidate, Dept. of Political Science, Ohio State University, Abstract In this paper I use automated text analysis to create the first machine-coded democracy index, which I call Automated Democracy Scores (ADS). I produce the ADS using the well-known Wordscores algorithm and 42 million news articles from 6,043 different sources. The ADS cover all independent countries in the period. Unlike the democracy indices we have today, the ADS are replicable, have standard errors small enough to actually distinguish between cases, and avoid contamination by human coders ideological biases; and a simple (though computationally demanding) extension of the method would yield daily data and real-time data. I create a website where anyone can replicate and tweak the data-generating process by changing the parameters of the underlying model (no coding required): Keywords Democracy. Measurement. Text-as-data. 1

2 In this paper I use automated text analysis to create the first machine-coded democracy index, which I call Automated Democracy Scores (ADS). 1 The basic idea behind the ADS is simple. News articles on, say, North Korea or Cuba contain words like censorship and repression more often than news articles on Belgium or Australia. Hence news articles contain regime-related information (even if we disregard word order and treat each article as a bag of words ). We can quantify that information to build a democracy index. I produce the ADS using the Wordscores algorithm, developed in Laver, Benoit, and Garry (2003), and 42 million news articles from 6,043 different sources. The ADS cover all independent countries in the period. Unlike the democracy indices we have today, the ADS are replicable, have standard errors small enough to actually distinguish between cases, and avoid contamination by human coders ideological biases; and a simple (though computationally demanding) extension of the method would yield daily data and real-time data. The next section explains why we need a new democracy index in the first place. The remaining sections explain the method in detail; show the results and how they compare to existing democracy data; and discuss some future extensions. Why do we need yet another democracy index? There are at least twelve democracy indices today (Pemstein, Meserve, and Melton 2010). They all draw to some extent from Dahl s (1972) conceptualization: democracy as a mixture of competition and participation. But they differ markedly in how they operationalize the concept - i.e., they differ in what empirical phenomena they pick as democracy manifestations; in how they aggregate these different empirical phenomena to produce a democracy scale; and in whether they model democracy as a categorical or continuous variable (Munck and Verkuilen 2002). In light of such diversity, do we really need yet another democracy index? I argue that we do, for three reasons. First, because the democracy indices we have 1 This work was funded by the Fulbright (grantee ID ); by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES (BEX 2821/09-5); and by the Ministério do Planejamento, Orçamento e Gestão - MPOG (proceed n / ). This work was supported in part by an allocation of computing time from the Ohio Supercomputer Center. I thank Irfan Nooruddin for reading and commenting several iterations of this paper and Sarah Brooks, Janet Box-Steffensmeier, Marcus Kurtz, Philipp Rehm, Shawn Treier, Paul DeBell, Carolyn Morgan, Margaret Hanson, Peter Tunkis, and Vittorio Merola for reading and commenting an early draft. All errors are mine. 2

3 today do not provide adequate measures of uncertainty. Without a good uncertainty measure we cannot know whether two countries are equally democratic or not, or whether a given country has become more (or less) democratic over time. That is, we cannot perform descriptive inference. Moreover, without a good uncertainty measure we cannot perform causal inference when democracy is one of the regressors. As Treier and Jackman (2008) warn, whenever democracy appears as an exploratory variable in empirical work, there is an (almost always ignored) errors-in-variables problem, potentially invalidating the substantive conclusions of these studies (203). Yet the two most popular indices - the Polity (Marshall, Gurr, and Jaggers 2013) and the Freedom House (Freedom House 2013) - only give us point estimates, without any measure of uncertainty. That prevents us from knowing, say, whether Uruguay (Polity score = 10) is really more democratic than Argentina (Polity score = 8) or whether the uncertainty of the measurement process is sufficient to make them statistically indistinguishable. Only two indices come with uncertainty measures: Treier and Jackman s (2008) and Pemstein, Meserve, and Melton s (2010). Treier and Jackman (2008) treat democracy as a latent variable and use an item-response model to extract it from Polity indicators. Treier and Jackman (2008) provide both the point estimates (the means of the marginal posterior distributions of the latent democracy variable) and confidence intervals (quantiles of the marginal posterior distributions). 2 Pemstein, Meserve, and Melton (2010) also treat democracy as a latent variable. They use a multirater ordinal probit model to extract that latent variable from twelve different measures (including the Polity and the Freedom House). And, as Treier and Jackman (2008), they also provide both point estimates (posterior means) and confidence intervals (posterior quantiles). They call their index Unified Democracy Scores (UDS). Both indices are big improvements over the Polity and Freedom House. It is hard to understand, in particular, why the UDS have not become the default democracy index in political science: the UDS summarize almost all pre-existing indices, have broad time 2 Armstrong (2011) does something similar, but using Freedom House indicators instead. His goal is different though - he extracts latent variables not to create a new democracy index but to investigate some properties of the Freedom House indicators. Also, he only reports the results for the 50 most populous countries and the full set of results is not available online. 3

4 and country coverage, and are freely available online. 3 Even if one is not interested in standard errors, why arbitrarily pick this or that individual democracy index when we can rely on the collective wisdom of all indices, condensed in the UDS? That political scientists continue to use the Polity and Freedom House is probably due to inertia. 4 If we do want standard errors though (and we should, for the sake of good descriptive and causal inference), we have a problem: both Treier and Jackman s (2008) index and the UDS offer standard errors that are too large to be useful. In Treier and Jackman s (2008) data 70 of the 153 countries are statistically indistinguishable from the United States (in the year the only year they report). In the UDS data 70% of the countries are all statistically indistinguishable from each other (in the year the last year in the UDS dataset); pairs as diverse (regime-wise) as Denmark and Suriname, Poland and Mali, or New Zealand and Mexico have overlapping confidence intervals. In short, existing democracy indices either have no standard errors or they have standard errors too large to be useful. That prevents us from doing descriptive inference and from knowing the effect of democracy on other variables. The second reason why we need a new democracy index is bias. Using structural equation modeling, Bollen and Paxton (2000) evaluate two popular sources of democracy data - the Freedom House and Arthur Banks Cross-National Time Series Data Archive (Banks [1971], updated through 1988) - and show that they are contaminated by ideological bias. Raters have policy preferences and boost the democracy scores of countries that adopt the correct policies. Hence we cannot reliably use existing democracy measures to estimate the impact of democracy on policy (or vice-versa). For instance, how can we assess the impact of democracy on welfare spending when our measure of democracy is partly based on welfare spending? The democracy measures we have today make our empirical tests circular. Bollen and Paxton (2000) only evaluate two democracy datasets, but all democracy data rely on country experts and for any given country there are only so many experts. As Munck and Verkuilen (2002) warn, for all the differences that go into the construction of these indices, they have relied, in some cases quite heavily, on the same sources and even the same precoded data. (29). Hence for all we know the Polity data are no less biased than the Freedom House data or the CNTS data. True, these democracy measures A less noble possibility is cherry-picking: perhaps researchers try different indices and pick the one that yields the correct results. 4

5 correlate highly, but that may merely indicate that they are all biased in similar ways (Munck and Verkuilen 2002). Unfortunately, whatever biases exist in those indices carry over to Treier and Jackman s (2008) and to the UDS. The latent-variable approach mitigates coders random errors, but not coders systematic errors. As Pemstein, Meserve, and Melton (2010) put it, the UDS rely on the assumption that raters perceive democracy levels in a noisy but unbiased fashion (10). The third reason why we need a new democracy index is replicability. Human-coded indices like the Polity and the Freedom House (and indices based on them, like Treier and Jackman s [2008] and the UDS) rely on country experts checking boxes on questionnaires. We cannot see what boxes they are checking, or why; all we observe are the final scores. The process is opaque and at odds with the increasingly demanding standards of openness and replicability of the field. Clearly, any of these three reasons alone justifies the creation of a new democracy index. News articles Our first task is to select the news articles to be used. Picking this or that news source - say, The New York Times or the The Wall Street Journal - would not do. The reason is that there would not be enough text. Countries like the United States and Russia are on the news all the time, but countries like Uruguay and Cambodia only get occasional coverage and countries like Tuvalu and Kiribati are almost never mentioned. A single newspaper or magazine, or even a handful thereof, would not provide the amount of text we need to produce reliable democracy scores for all 196 independent countries in the world. Hence I use a total of 6,043 news sources. These are all the news sources in English available on LexisNexis Academic, which is an online repository of journalistic content. The list includes American newspapers like The New York Times, USA Today, and The Washington Post; foreign newspapers like The Guardian and The Daily Telegraph; news agencies like Reuters, Agence France Presse (English edition), and Associated Press; and online sources like blogs and TV stations websites. I use LexisNexis s internal taxonomy to identify and select articles that contain regime- 5

6 related news. In particular, I choose all articles with one or more of the following tags: human rights violations (a subtag of crime, law enforcement and corrections ); elections and politics (a subtag of government and public administration ); human rights (a subtag of international relations and national security ); human rights and civil liberties law (a subtag of law and legal system ); and censorship (a subtag of society, social assistance and lifestyle ). 5 LexisNexis news database covers the period 1980-present, 6 so in theory the ADS could cover that period as well. LexisNexis provides search codes for all countries that exist today - e.g., #GC508# for Afghanistan and #GC342# for Mexico. That way we can search for news articles on a specific country and be sure that all results will turn up - even the ones that do not mention the name of country (for instance, many articles use the name of the country s capital when they mean the country s government - as in Moscow retaliated by canceling the summit ). Unfortunately, however, LexisNexis provides no search codes for countries that no longer exist. To search for articles on the Soviet Union, for instance, we would need to search for the name(s) of the country (Soviet Union, USSR), its derivatives (Soviet), the name of the capital (Moscow), etc - anything that might tell us that the article refers to the Soviet Union. Clearly that would not work. What if the article does not mention any of those terms? And what if the country has a name that is also a proper noun? We would have unreliable results. Thus we can only reliably search the period. Other than Yugoslavia, no country has ceased to exist since 1992, so we have search codes for basically everything. (Naturally, many countries were created in that period, but that is not a problem - the dataset will simply start in 2008 for Kosovo, in 2002 for East Timor, and so on). That selection - i.e., regime-related news, all countries that exist today, results in a total of about 42 million articles (around 4 billion words total), which I then 5 It would be interesting to know how the results change if we select different topic tags, but unfortunately that is no longer possible: on 12/23/2013 LexisNexis changed its user interface and the dozens of political tags and subtags that existed before are now collapsed into a single Government & Politics tag, which is too broad for our purposes here. 6 Actual coverage varies by news source. 6

7 organize by country-year. 7 To help reduce spurious associations I remove proper nouns 8 (in a probabilistic way) 9. For each country-year I merge all the corresponding news articles into a single document and transform the document into a term-frequency vector - i.e., a vector that contains the absolute frequency of each word. I then merge all term-frequency vectors into one big term-frequency matrix. Rows represent words and columns represent countryyears, so each cell gives us the absolute frequency of a given word in the news articles corresponding to a given country-year. 10 Algorithm There are several automated ways to extract data from text (see Grimmer and Stewart [2013] for an overview). The particular method I use is the Wordscores algorithm, created by Laver, Benoit, and Garry (2003) - henceforth LBG -, from which this section draws heavily. The next paragraphs explain the algorithm in detail, but here is the gist of it: we manually score some documents - called reference documents or training documents (Manning, Raghavan, and Schütze 2008); the algorithm learns from the reference docu- 7 A small proportion of the articles (about 0.05%) is left out. When a search produces more than 3,000 results LexisNexis only returns the first 1,000. Whenever possible I overcome that problem by searching for smaller periods of time. But in a few cases even searches for a single day produce more than 3,000 results. I could not figure out what criteria LexisNexis uses to select the 1,000 results it returns (I asked them by but they never replied). Thus to avoid any selection biases I just leave all results out in those cases. (The cases are: Pakistan 5/2/2011; Pakistan 5/3/2011; Afghanistan 10/8/2001; Afghanistan 10/9/2001; United Kingdom 7/8/2005; and United States, several dates between 2003 and 2012.) I realize that doing this may introduce selection biases of its own, but at least I am creating the selection biases myself, whereas I have no idea how LexisNexis selects those 1,000 results. In any case, it is doubtful that excluding 0.05% of the news articles will have any noticeable impact on the results. 8 As we will see later the scoring algorithm works by associating certain words with certain qualities. But we want those words to refer to general phenomena like torture, repression, and censorship, not to specific people or places. We do not want, for instance, Washington being associated with high levels of democracy just because the word appears frequently on news stories featuring a highly democratic country. Removing proper nouns helps avoid that. 9 I cannot possibly read 42 million articles. And we cannot simply remove all capitalized words, as that would eliminate the first word of every sentence, even if it is not a proper noun. Hence I apply the following rule: if all occurrences of the word are capitalized then that is probably a proper noun and therefore it is removed. (For each country-year I merge all the corresponding news articles into a single document and process each document in chunks of 10MB - to reduce memory usage -, so the check is restricted to the same 10MB chunk.) 10 The data are available under request. Eventually they will be available via Academic Torrents (http: //academictorrents.com/). 7

8 ments and uses that knowledge to score all other documents - called virgin documents. So far Wordscores has only been used to measure party ideology (from party manifestos and legislative speeches). To the best of my knowledge, this is the first time Wordscores - or any other method of automated text analysis - is used to measure democracy. The first step is to select the reference cases. In other words, we need to pick some of the 4,067 country-years we have here to serve as the baseline from which the machine will learn. Ideally the reference set must span the entire regime scale. If we only feed the algorithm, say, highly democratic cases, then the machine will not learn what words are associated with middle-of-the-road cases or authoritarian cases. To ensure that the reference set is broad enough I pick all country-years from 1992, the first year for which we have news articles (see previous section). Thus the ADS only cover the period even though we have news articles from 1992 as well. The vast majority of multivariate analyses that use some measure of democracy (Polity, Freedom House, etc) use pretty recent data, rarely going farther back in time than the 1970s, so the ADS should serve most applied research well. The second step is to give each reference case a score. For us that means assigning a democracy score to each country-year from I follow LBG and extract these reference scores from an existing index. 11 In particular, I choose Pemstein, Meserve, and Melton s (2010) UDS, which I mention before. The UDS have data on 184 countries for the year Hence we have 184 reference documents and 3,883 (4, ) virgin documents. The third step is to compute the word scores. Let F wr be the relative frequency of word w on reference document r. The probability that we are reading document r given that we see word w is then P (r w) = F wr / F wr. We let A r be the a priori position of r reference document r and compute each word score as S w = (P (r w) A r ). r The fourth step is to use the word scores to compute the scores of the remaining documents - the virgin documents. Let F wv be the relative frequency of word w on virgin document v. The score of virgin document v is then S v = (F wv S w ). w Intuitively, the algorithm uses the training documents to learn how word usage differs across the reference cases - for instance, it learns that the word censorship is more frequent the lower the democracy score of the document. The algorithm then uses that 11 Just to be clear, LBG were measuring party ideology, not democracy, so obviously the indices they use have nothing to do with the one I use here. 8

9 information to produce word scores (hence the name of the method), and later uses the word scores to score the virgin documents. A concrete example may help. Suppose that we choose North Korea 2012 and Belgium 2012 as our reference cases and assign them democracy scores of 0 and 10 respectively. We merge all news articles on North Korea in 2012 into a single document and merge all news articles on Belgium in 2012 into another document. Suppose now that the word censorship accounts for 15% of all the words in the North Korea document and for 1% of the words in the Belgium document. If we see the word censorship the probability that we are reading the North Korea document is 0.15/( ) = and the probability that we are reading the Belgium document is 0.01/( ) = The score of the word censorship is thus ( ) + ( ) = To score a virgin document we simply multiply each word score by its relative frequency and sum across. The fifth step is the computation of uncertainty measures for the point estimates. LBG propose the following measure of uncertainty: V v / N v, where V v = F wv (S w S v ) 2 w and N v is the total number of virgin words. The V v term captures the dispersion of the word scores around the score of the document. Its square root divided by the square root of N v gives us a standard error, which we can use to assess whether two cases are statistically different from each other. The sixth and final step is the re-scaling of the virgin scores. In any given text the most frequent words are the, of, and, etc, which usually are of not interest. Because these words have similar relative frequencies across all reference texts they will have centrist scores. For instance, if the accounts for 10% of our (hypothetical) North Korea document (whose manually assigned score is 0) and for 10% of the (also hypothetical) Belgium document (whose manually assigned score is 10), the score of the will be five, exactly in the middle of the scale. That makes the scores of the virgin documents bunch together around the middle of the scale; their dispersion is just not in the same metric as that of the reference texts. In LBG s estimations of party ideology in Britain, the scores of the reference documents range from 8.21 to 17.21, but the scores of the virgin documents range from to That is not a problem per se, as the scores of the virgin documents are perfectly comparable to each other. But they are not comparable to the scores of the reference documents, whose dispersion is higher, and that may be a problem depending on the 9

10 intended goals. 12 To correct for the bunching of virgin scores, LBG propose re-scaling these as follows: S v = (S v S v )(σ r /σ v ) + S v, where S v is the raw score of virgin document v, S v is the average raw score of all virgin texts, σ r is the standard deviation of the reference scores, and σ v is the standard deviation of the virgin scores. This transformation expands the raw virgin scores by making them have the same standard deviation as the reference scores. Martin and Vanberg (2008) propose an alternative re-scaling formula, but Benoit and Laver (2008) show that the original formula is more appropriate when there are many virgin cases and few reference cases, which is the case here. The final output is a dataset comprising all independent countries from 1993 to 2012, which makes for a total of 3,883 country-years. For each country-year three statistics are provided: the ADS point estimate, the ADS 95% lower bound, and the ADS 95% upper bound. Initially I considered having not only a democracy scale but also subcomponents, à la Polity. But of the 805 JSTOR-indexed articles that cite the Polity data over the last ten years, only a handful mention (and even fewer use) the Polity subcomponents (Pemstein, Meserve, and Melton [2010] also note this point). Hence there is simply not enough demand to justify breaking down the ADS into more specific items. 13 And, rich in regimerelated information as our news articles may be, they nonetheless become progressively less informative as we move from democracy down to, say, turnover percentage in the legislature. The more specific we get, the higher the noise-to-signal ratio. Wordscores is the best-known text-scaling method in political science. It has been subject to extensive scrutiny over the years and generally found to perform well, as long as the texts are not too short 14 and share enough vocabulary 15. Klemmensen, Hobolt, 12 We could of course remove irrelevant words, but identifying relevant and irrelevant words is not always so clear-cut. For instance, as I mention later, Monroe, Colaresi and Quinn (2008) find several nonobvious partisan words - like baby (Republican) and bankruptcy (Democrat) - in their analysis of legislative speeches in the US Senate. Thus if we exclude words a priori we risk throwing away important information. Moreover, removing any words would require knowledge of the language in which the text is written. That would defeat one of the biggest advantages of the method: the fact that it is language-blind (all we need to know are the positions of the reference documents). 13 Thus the ADS are bound to displease, for instance, Coppedge et al. (2011), who call for thicker measures of democracy. 14 If the texts are too short then there is simply not enough data to produce meaningful results. How short is too short is unclear though: all else equal more is better, but a 5,000-word text may contain more informative words than a 10,000-word text. 15 In the extreme case where the vocabulary of the reference texts and the vocabulary of the virgin texts 10

11 and Hansen (2007), for instance, use Wordscores to measure party ideology, with Danish manifestos and speeches, and find that the method yields scores that correlate highly with those produced independently by human coders. Beauchamp (2010) applies Wordscores to US Senate speeches and, as Klemmensen, Hobolt, and Hansen (2007), also finds that the estimates correlate highly with human-coded ones. Lowe (2008) notes that Wordscores lacks an explicit model for the data-generating process of the word frequencies. But he argues that, as long as the word frequencies follow an ideal point structure, 16 Wordscores should produce good estimates - and he notes that The empirical success of the method suggests that these assumptions may be reasonable. (370). Advantages over existing measures The ADS are intended to address the three issues discussed earlier: standard errors, ideological bias, and replicability. Small standard errors As shown above, with Wordscores the total number of virgin words goes in the denominator of the formula of the standard errors. Hence the more texts we have, the smaller the standard errors will be. Here we have 42 million news articles, so we should have standard errors small enough to distinguish even between very similar cases - say, between Sweden and Norway. As we will see later, that is indeed what happens. The ADS are the first democracy index whose uncertainty measure captures such fine-grained distinctions. Less ideological bias The ADS are not immune to contamination by ideological bias. First, the journalists and editors behind news articles have their own policy preferences. And second, at least in the case of supervised learning algorithms (like Wordscores), someone must choose and score the reference cases. are disjoint, we cannot even produce any estimates. 16 Lowe (2008) proposes that we interpret Wordscores as an approximation to correspondence analysis - which relies on the assumption of ideal point structure. 11

12 But the scope for manipulation is more restricted in the ADS. Journalists and editors have their policy preferences but there is a lot more ideological diversity among journalists (contrast The New York Times and The Wall Street Journal, for instance) than among political scientists, the vast majority of which are somewhere on the left of the ideology spectrum (Klein and Stern 2005; Maranto, Hess, and Redding 2009; Maranto and Woessner 2012). Combining 6,043 different news sources, as we do here, surely goes a long way toward mitigating ideological bias. The reference scores do offer a backdoor for manipulation but, unlike the anonymous raters who fill out the Polity and Freedom House questionnaires, the researcher who assigns reference scores does so in the open and thus bears reputational costs in case of mischief. The transparency of the process creates an incentive structure that rewards honesty. As Schedler (2012) puts it, The key to accountable expert measurement [...] is publicity. Rather than treating experts the same way as we treat survey subjects, whom we grant full anonymity, experts need to assume public responsibility for their measurement decisions. True, we are using the UDS for the reference scores, and the UDS themselves must be contaminated by ideological bias, as discussed before. But the ADS do not inherit that bias. We are using regime-related news, so the vast majority of policy and economic discussions is left out. Hence the biases of the UDS become, by and large, random noise in the ADS. Intuitively, imagine that the UDS are biased in favor of countries with generous welfare, like Sweden. The UDS of these countries will be boosted somewhat. But to the extent that the news articles we selected are focused on political regime and not on welfare policy, the algorithm will not associate those boosted scores with welfare-related words and hence the word scores will not be biased. They will be less efficient, as (ideally) no particular words will be associated with those boosted scores, but that is it. Replicability The process behind the ADS is fully transparent. All the choices (reference cases and scores) are visible to the public and every part of the process can be replicated exactly. Anyone with access to LexisNexis can download the same articles, apply the same algorithm, and verify the results. 12

13 There are practical obstacles though: downloading 42 million articles is time-consuming and the computations require powerful machines and non-trivial programming. 17 Therefore I created a website that facilitates the process: No coding is required: there is a table with empty cells corresponding to each country-year between 1992 and 2012 and you simply enter the scores for the reference cases you choose. The results are sent by . That way anyone can change the reference set and produce their own ADS, regardless of computational resources or programming skills. 18 Justifying some choices Why not use unsupervised learning instead? Wordscores is a type of supervised learning algorithm, by which I mean that the machine learns from an initial human input (the reference cases and their scores). But there are also unsupervised learning algorithms, which do not require an initial input. In these, the machine learns by itself not only how to measure but also what to measure. (See Manning, Raghavan, and Schütze [2008] for an introduction to both supervised and unsupervised learning in the context of text analysis.) In political science, a concrete example of unsupervised learning algorithm is the one developed by Slapin and Proksch (2008), popularly known as Wordfish. Like Wordscores, Wordfish is most commonly used to measure party ideology, using party manifestos or legislative speeches. The Wordfish method does not require the user to specify or score any reference texts. It will create a scale based on whatever underlying dimension has the most impact on word frequencies. 19 If we are talking about party manifestos, that 17 Wordscores has long been implemented in Stata and R, but these implementations load all the data into memory at once. That would not work here, as there are 200GB of data, so I had to write my own implementation of Wordscores (in Python). That implementation splits the data into chunks and processes each chunk individually, which reduces memory requirements (though not to the point where the script could be run on personal computers - there is a trade off between memory requirements and speed). 18 The operation uses Amazon Web Services and to keep costs down for now I need to pre-authorize the user s address. The goal is to secure funding and lift that restriction. 19 Slapin and Proksch explicitly model the data-generating process (DGP) behind word frequency. That DGP is assumed to follow a Poisson distribution (hence the name of method): y ijt = P oisson(λ ijt ), where y ijt is the frequency of word j in document i at time t. The parameter λ ijt is modeled as 13

14 dimension may be, say, the left-right dimension. But it may not. And if the scores turn out to be capturing something else there is no way to fix that; it may be hard to even know what is being captured. Supervised learning, on the other hand, allows us to calibrate the scale by explicitly showing the machine what a democratic country looks like or what a left-wing party looks like (depending on what we are trying to measure). That way we can have greater confidence in the construct validity of the resulting measure. 20 Why not use event data instead? An alternative approach would be to machine-code democracy based not on words but on events. Applications like Knowledge Manager 21 and TABARI (Schrodt 2001) can use dictionaries of actors and verbs to extract meaning from sentences. For instance, TABARI can correctly classify the sentence North Korean state media have called on the United States to forge ties of confidence with Pyongyang into the category Appeal for diplomatic cooperation (category #022 of the Conflict and Mediation Event Observations Codebook). There are voluminous event data available for free 22 and King and Lowe (2003) show that in some cases automated event coding can be as accurate as human coding. So why not use event data to produce the ADS? The reason is that although the coding itself is automated, it relies on dictionaries of actors and verbs that are produced manually, entry by entry. In other words, we must know the relevant actors and verbs a priori. With Wordscores, however, we let the data speak. As Hopkins and King (2007) note, automated text analysis allows us to discover relevant features a posteriori. For instance, Monroe, Colaresi and Quinn (2008) find several nonobvious partisan words, like baby and bankruptcy, which a hand-coded dictionary might have missed. As a consequence, event data can be of limited usefulness. Consider, for instance, the λ ijt = exp(α it + ψ j + β j ω it ), where α it is the fixed-effect of document i at time t, ψ j is the fixedeffect of word j, β j captures the relevance of word j in capturing the underlying concept (say, party ideology), and ω it is the estimated position of party i at time t. The model is estimated using an expectation-maximization (EM) algorithm (see McLachlan and Kirshnan [2007] for details on EM). 20 That said, Wordscores and Wordfish are not antithetical methods. Lowe (2008) and Benoit and Nulty (2013) argue that Wordscores is also model-based in a sense, only the model is implicit Most notably the Global Data on Events, Location, and Tone (GDELT), which contains over 200 million geolocated events from 1979 to See Leetaru and Schrodt (2013). 14

15 latest version of the World Handbook of Politics (WHP), machine-coded by the Knowledge Manager application. 23 It reports three recent coups in Canada (one in 1996, one in 1998, and one in 1999), 15 recent coups in the US (three of which taking place in 1994 alone), and none in 2002 Venezuela (even though there was one). 24 Similarly nonsensical statistics are reported for other political indicators, such as censorship measures, curfews, and political arrests. That is not a very promising output, especially given the time and effort put in the creation of event data dictionaries (around 4,000 hours each) 25. Hence I chose not to work with even data, at least for now. Overview of results The full dataset is available for download. 26 Figure 1 below gives an idea of the ADS distribution in The WHP can be downloaded from 24 I checked the WHP definition of coup, to make sure it is not peculiar, but that does not seem to explain the nonsensical results (the WHP defines a coup as an Irregular seizure of executive power, and rebellion by armed forces )

16 Figure 1. Automated Democracy Scores, 2012 Note: Range limits are Jenks natural breaks. As expected, democracy is highest in Western Europe and in the developed portion of the English-speaking world, and lowest in Africa and in the Middle East. Figure 2 below shows that the ADS follow a normal distribution. 16

17 Figure 2. Automated Democracy Scores, (with normal distribution) Table 1 below shows the ADS summary statistics by year. 17

18 Table 1. ADS summary statistics, by year N mean std. dev. min. max all As expected, the average ADS increases over time, from in 1993 to in That reflects the several democratization processes that happened over that period. We observe the same change in other democracy indices as well (between 1993 and 2012 the average Polity score 27 increased from 2.24 to 4.06 and the average Freedom House score 28 decreased from 7.46 to 6.63; 29 the average UDS score increased from 0.21 to 0.41 between 1993 and 2008, the last year in the UDS dataset). 27 polity2 28 civil liberties + political rights 29 Freedom House scores decrease with democracy. 18

19 Also as expected, the standard errors decrease with press coverage. The larger the document with the country-year s news articles, the narrower the corresponding confidence interval. As Figure 3 shows, that relationship is not linear though: after 500KB or so the confidence intervals shrink dramatically and do not change much afterwards, not even when the document has 15MB or more. 19

20 Figure 3. ADS range and press coverage Note: ADS range = 95% upper bound minus 95% lower bound. The ADS vs other indices - point estimates The ADS point estimates correlate with the UDS (posterior means), with the Polity s (polity2), and with the Freedom House s (civil liberties + political rights). 30 Table 2 below breaks down these correlations by year. 30 Pearson correlation. 20

21 Table 2. Correlation between ADS and other indices, by year UDS Polity a FH b UDS Polity a FH b n/a c n/a c n/a c n/a c a polity2 (see Marshall, Gurr, and Jaggers 2013, p. 17) b civil liberties + political rights (see Freedom House 2013) c The UDS do not cover the period. As we see, the correlations do not vary much over time. This is a good sign: it means that the ADS are not overly influenced by the idiosyncrasies of the year 1992, from which we extract the reference cases. Otherwise we would see the correlations decline sharply after The correlations do not vary much across indices either, other than being somewhat weaker for the Polity data. This is also a good sign: it means that the ADS are not overly influenced by the idiosyncrasies of the UDS, from which we extract the reference scores. 31 I also ran the algorithm using other years (rather than 1992) for the reference set, using UDS as well. I also re-ran the algorithm multiple years (up to all years but one) for the reference set, again using UDS. Finally, I also re-ran the algorithm using not the UDS but the Polity and Freedom House indices for the reference set. In all these scenarios the correlations remained in the vicinity of This corroborates Klemmensen, Hobolt, 31 Though we must remember that the UDS are partly based on the Polity and the Freedom House, so by extension the ADS also are. 32 The cases we use for the reference set cannot be used for the virgin set. For instance, in one scenario I used every other year for the reference, starting with In that scenario the reference set was thus [ ] and the virgin set was [ ]. To compute the correlations with other indices I only used the virgin 21

22 and Hansen s (2007) finding that Wordscores results are robust to the choice of reference texts. Country-wise, what are the most notable differences between the ADS and the UDS? Table 3 below shows the largest discrepancies. set. 22

23 Table 3. Largest discrepancies between ADS and UDS largest positive differences largest negative differences ADS UDS ADS UDS Swaziland Israel Liechtenstein Israel Liechtenstein Israel Ireland Israel Andorra Benin Luxembourg Israel Bhutan Yemen Ireland Israel Finland Tunisia China Oman The largest positive differences - i.e., the cases where the ADS are higher than the UDS - are mostly found in small countries with little press coverage. That is as expected: the less press attention, the fewer news articles we have to go by, and the harder it is to pinpoint the country s true democracy level. The largest negative differences, however, tell a different story. It seems as if either the ADS repeatedly underestimate Israel s democracy score or the UDS repeatedly overestimate it (and not only for the years shown in Table 3). We do not observe a country s true level of democracy, so we cannot know for sure whether the ADS or the UDS are biased, 33 but as discussed before the ADS should be unbiased to the extent that we managed to filter out news articles not related to political regime. As discussed before, whatever biases exist in the UDS should become, by and large, random noise in the ADS. The UDS, on the other hand, rely on the assumption that raters perceive democracy levels in a noisy but unbiased fashion (Pemstein, Meserve, and Melton 2010, 10), which as Bollen and Paxton (2000) have shown is simply not true. Hence whatever biases exist in the Polity, Freedom House, etc, wind up in the UDS as well. The data-generating process behind the UDS does not mitigate bias in any way. In other words, it seems more likely that the UDS are overestimating Israel s democracy scores than that the ADS are underestimating them. This pro-israel bias is interesting 33 Though of course these two possibilities are not mutually exclusive. 23

24 in itself, but it also raises the more general question of whether the UDS might have an overall conservative bias. To investigate that possibility I performed a difference-ofmeans test, splitting the data in two groups: country-years with left-wing governments and country-years with right-wing governments (I used Keefer s [2012] Dataset of Political Institutions for data on government ideological orientation.) 34 The test rejected the null hypothesis that the mean ADS-UDS difference is the same for the two groups: the mean ADS-UDS difference for left-wing country-years (-0.127, std. error = 0.024, n = 802) is statistically smaller than the mean ADS-UDS difference for right-wing country-years (-0.328, std. error = 0.025, n = 603), with p < As both means are negative, it seems that the UDS tend to reward right-wing governments. I also checked whether the UDS may be biased toward economic policy specifically. I split the country-years in the Index of Economic Freedom (Heritage Foundation 2014) dataset into two groups: statist (IEF score below the median) and non-statist (IEF score above the median). The difference-of-means test shows that the mean ADS-UDS difference for statists (-0.132, std. error = , n = 1057) is statistically lower than that of nonstatists (-0.215, std. error = 0.015, n = 1977), with p < Both means are negative here as well, so it seems that the UDS somehow reward free market policies. These findings are surprising. Political scientists are overwhelmingly on the left side of the political spectrum (Klein and Stern 2005; Maranto, Hess, and Redding 2009; Maranto and Woessner 2012), so if anything we would expect their democracy measures to be biased in favor of left-wing governments and policies, not against them. Perhaps the country experts who code the democracy indices behind the UDS are not political scientists for the most part. It is hard to know for sure, as the codebooks usually do not mention the coders backgrounds. 35 We cannot conclusively indict the UDS or its constituent indices though. Perhaps democracy and right-wing government are positively associated and the ADS are somehow less efficient at capturing that association. This is consistent with the Hayek-Friedman hypothesis that left-wing governments are detrimental to democracy because economic activism expands the state s coercive resources (Hayek 1944; Friedman 1962). As we do not observe a country s true level of democracy, it is hard to know for sure what is going on here. 34 I used the EXECLRC variable. 35 Marshall, Gurr, and Jaggers (2013), for instance, only say that at least four coders (6) coded each Polity case, without giving any further information. 24

25 At least until we know whether the UDS are biased or the ADS are inefficient, the ADS are the conservative choice. Say we regress economic policy on the UDS and find that more democratic countries tend to have less regulation. Is that relationship genuine or is it an artifact of the UDS being biased in favor of free market policies? With biased measures our tests become circular: we cannot know the effect of X on Y when our measure of X is partly based on Y. Inefficiency, on the other hand, merely makes our tests more conservative. The ADS vs other indices - standard errors As mentioned before the UDS data and also Treier and Jackman s (2008) data have not only point estimates but also standard errors. That is a big improvement over data like the Polity and the Freedom House, which provide point estimates only. But the UDS and Treier and Jackman s confidence intervals are too wide to be useful. Too many cases are statistically indistinguishable. The ADS, on the other hand, have smaller confidence intervals. These confidence intervals tend to be larger the less press coverage the country gets, but in all cases they are smaller than the corresponding UDS ones. Table 4 below shows, for each country in 2008 (the last year for which there are UDS data), how many other countries have overlapping confidence intervals in each dataset (UDS and ADS). 25

26 Table 4. Overlaps for the year 2008 UDS ADS UDS ADS Afghanistan Libya 51 2 Albania Liechtenstein Algeria Lithuania 88 3 Andorra Luxembourg Angola 76 4 Macedonia Antigua & Barbuda Madagascar Argentina Malawi Armenia Malaysia Australia 90 1 Maldives Austria 63 2 Mali Azerbaijan 70 1 Malta 68 1 Bahamas Mauritania 70 2 Bahrain 65 1 Mauritius Bangladesh Mexico Barbados Micronesia Belarus 65 3 Moldova Belgium 99 2 Mongolia Belize Montenegro Benin Morocco 76 4 Bhutan Mozambique Bolivia Myanmar 41 2 Bosnia-Herzegovina Namibia Botswana Nauru Brazil Nepal Brunei Netherlands 66 3 Bulgaria New Zealand 80 0 Burkina Faso Nicaragua Burundi Niger Cambodia Nigeria Cameroon 75 3 North Korea

27 Canada 90 2 Norway 66 8 Cape Verde Oman 59 0 Central African Rep Pakistan Chad 79 0 Palau Chile Panama China 58 5 Papua New Guinea Colombia Paraguay Comoros Peru Congo Brazzaville 71 5 Philippines Congo Kinshasa Poland 99 6 Costa Rica Portugal 90 4 Croatia Qatar 43 0 Cuba 58 2 Romania Cyprus 66 4 Russia Czech Rep Rwanda 85 5 Denmark 63 3 St. Kitts & Nevis Djibouti 98 1 St. Lucia Dominica St. Vin. & the Gren Dominican Rep Samoa East Timor San Marino Ecuador S. Tome & Principe Egypt 72 0 Saudi Arabia 29 5 El Salvador Senegal Equatorial Guinea 59 4 Serbia Eritrea 58 0 Seychelles Estonia Sierra Leone Ethiopia Singapore Fiji 76 5 Slovakia Finland 66 4 Slovenia 88 2 France Solomon Is Gabon 89 4 Somalia 62 1 Gambia 80 2 South Africa Georgia South Korea