Determining Automatic analysis of Populist Language Tweets in the Dutch Political spectrum

Transcription

1 Determining Automatic analysis of Populist Language Tweets in the Dutch Political spectrum T.R. van Brakel Universiteit Twente P.O. Box AE Enschede The Netherlands ABSTRACT The usage of populist language in politics has always been present. Since the rise of social media and it being the primary channel of quick communication to the populous, one can analyze populist language usage by looking at posted messages. However, this current analysis is very labor intensive. The main research question is to determine if it is possible to create a tool that will analyze populist language automatically according to a pre-programmed internal dictionary which is built on keywords and annotated data. Populist language and its related keywords are defined by a literature study. This annotated data is the outcome of a survey that consists of a random selection of tweets. The annotated data will be used for a LIWC dictionary analysis that determines the several word categories that have the highest relation with populism. The tool that is built on these word categories has its limitations but shows signs of steps towards automatic analysis. Keywords Populism, Social Media, Politics, Tool, Automatic analysis 1. INTRODUCTION Every politician, in some form, has resorted to the use of populist language. Whether on the radio, newspapers, tabloids or social media. This paper focuses on the usage of populist language on social media and more specifically, on Twitter, and tries to determine if it is possible to use a simple tool or a method that can automatically point out populist language usage. Since several previous studies have stated that the analysis of populist language is a demanding task, being either time consuming [10] or having problems in forming a clear definition and also being hard to quantify [3]. First off, a clear definition of populist language has to be determined beforehand. This ensures the analysis of the tweets from the Dutch political parties and their party leaders is not based on the researcher s subjective opinion. The definition of populist language will be based on a literature study. Meaning, analyzing definitions found in previous studies, comparing these and finding a best fit or combination Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 25 th Twente Student Conference on IT, July 1 st, 2016, Enschede, the Netherlands. Copyright 2016, University of Twente, Faculty of Electrical Engineering, Mathematics and Computer Science. of these definitions to continue the research with. The answer of this literature study is the answer to sub-question 1, What is populism/populist language?. It is important to have a clear and usable definition of populist language because this definition is used as a basis on which a survey is built where participants will rate tweets on being populist or not. In order to minimize participants rating the tweets based on their own perception of populist language this definition will be given before participants start the survey. The actual survey consists of tweets where participants give a rating depending on how populist they think the specific tweets are according to the definition. The tweets are from every Twitter account of every major Dutch political party and their respective leader. Major is the keyword here, since the focus isn t on local parties or parties that don t have a seat in the Dutch House of Representatives, in Dutch, the Tweede Kamer. This choice has been made because if every party would be included, there wouldn t be enough tweets of every Twitter account to analyze. Meaning, no conclusions can be made if an account uses populist language if there is only a handful of tweets being annotated. And also these small or local parties are less active on Twitter. Because only the major Dutch electoral parties are being analyzed something can be said about them using populist language on Twitter. More so, as previous studies suggest, that populist language is more prevalent in the extremist parts of the political spectrum [9]. This is also researched to a heavier extend than populist language in the broader political spectrum [7-9]. These facts lead into the second research question: Is populist language used only in the extremist s parts of the political spectrum?. This question is being answered by analyzing the survey results where participants will rate the tweets, but don t see which Twitter account posted these tweets. This is to make sure participants with a different political view to that specific party aren t biased when rating the tweet. The answer of this subquestion should result in a neat overview of which party is using populist language more often on Twitter and can hopefully add to the claims made in previous studies that this usage of populist language reflects on Twitter as well. The ultimate goal of this research is to determine if populist language can be analyzed automatically. This means determining which (key)words and word-categories are related to being populist. This follows out of previous studies [4] and the results of the survey. Analyzing word-categories will be done in a similar fashion of content analysis in a previous study by Wang et Al [16] using a tool named Linguistic Inquiry and Word Count(LIWC) [11]. LIWC is a tool that has seen widespread use in numerous studies where linguistic analysis is required and has proven to be very reliable. The results of LIWC coupled with the results of the survey which accounts use populist language more often and combined with findings 1

2 of previous studies should come up with keywords and wordcategories related to populism. This relates to the final subquestion: Which (key)words and word-categories are more related to populist language?. With the answer of this final sub-question a simple tool/method can be built to determine if populist language can be analyzed automatically. To test if the tool can analyze the tweets on populist language usage a part of the survey that is not used for LIWC analyzing but has been annotated by participants will be used as a test-set to determine if the tool can come up with satisfactory results. The next sections will follow the format in which the research questions were presented. Meaning, after a brief summary of the research questions the literature study containing the answer to sub-question one and findings about the appearance of populist language in the political spectrum and related keywords will be discussed first. After that the approach and the results of the survey will be presented and discussed. Followed by the LIWC analysis and method for automatic analysis. Finally, conclusions based on the main research question and recommendations for future work will be discussed. 1.1 Research questions As stated in the previous section this research aims to create a tool for automatic analysis of populist language. The question formulated for this research listens to: Can populist language be determined automatically via the use of a tool/method? The sub-questions that substantiate the research questions are: What is populism/populist language? Is populist language used only in the extremist s parts of the political spectrum? Which (key)words and word-categories are more related to populist language? 2. RELATED LITERATURE It is of the utmost importance to have a clear definition of populist language. Especially in the type of research conducted in this paper. Since we re going to ask participants of a survey to rate tweets on how populist they are. This means we cannot have people having different ideas or understanding of populism. This importance is reflected in the first sub-question, What is populism/populist language?. There have been multiple studies towards the definition of populist language. Populism can be seen as an ideology [5] or a political style [4, 6]. Some scholars tie populism to extreme right-winged parties and others do not [1]. The study by Hawkins even suggests that depending on your point of view or the topic being discussed the definition of populism slightly differs [3]. There is however common ground to be found on the definition of populist language. Shortsighted policies that appeal to the poor. The control of the government. The (corrupt) elite that doesn t care about the welfare of the general populous, in other words, a misrepresentation of democracy [8]. Another fact that is mentioned is the presence of a charismatic leader that intends to represent the people. For this research the definition that is given before the start of the survey will combine all these facts into a clear and understandable description of populist language. This definition can t be a very long sentence. This will only make participants more confused. We would like to have a sentence that is about two lines long preferable with all or most of the characteristics of populist language incorporated. The following part is a brief summary of different populist language definitions and characteristics found in previous studies. These findings will conclude with an answer to subquestion one and this definition will be used for the survey in a later stage of the research. Alvares and Dahlgren state that clearly the notion of the people is a key characteristic of populist language [1]. Also, as previously stated, the corrupt elite [8] is also a keyword which politicians tend to use if they communicate to the populous. As mentioned in Krämer: rural and peasant communities are cherished as a counterpart to industrial society that causes alienation from its complex institutions and division of labor [5]. This clearly means the working class that is being alienated by the policies of the elite. Hawkins outlines several factors of populism, namely different classes, elite and the people, shortsighted economic policies that appeal to the poor. Also the struggle of the people with the control from the government [3]. In Otjes & all populism is described as two claims. The first being, the government should reflect the will of the people. The second claims that The political elite has corrupted and distorted politics in order to deprive the people of power for the sake of their own [9]. Every other study has a bit of variation on the definition of populism or adds a very minor characteristic. To summarize, the key characteristics of populist language are, the people, (corrupt) elite, shortsighted claims/policies. With these three main keywords built in a sentence a working definition for the survey is created which also answers the question: What is populism/populist language?. Populism is a form of communication which focusses on the working class/poor people, presenting them a prosperous but infeasible claim that is actually beneficial for the elite. This is a definition that we think participants can understand and use if they are presented some example tweets combined with their rating before they start the survey. The next part of the literature study consists of conducting an analysis in which parts of the political spectrum populist language is used more or most often. The results of the survey could really substantiate or undermine claims made in previous studies that populist language is more apparent in the most extremist s parts of the political spectrum. For example, in the case of the appearance of populist language in politics there has been multiple studies that focus on populism in the extremist s parts of the political spectrum, especially since the rise of rightwinged populist parties [7-9]. The findings in these previous studies suggest that populist language is indeed more apparent on the extremist s parts of the political spectrum. A relevant fact stated by Otjes & Louwerse is that although left and rightwinged parties vote differently on subject in the Dutch house of parliament, the political rhetoric they generally use is very similar [9]. With the results of the survey we can see if this fact applies to the usage of populist language on Twitter as well. The final part of the literature study focusses on determining certain keywords and word-categories that are strongly connected to populism. One thing has to be taken into account though. This is the fact that in one context a message or a tweet can be interpreted as being populist, in another context it might not be populist. So only the keywords and categories that are 2

3 strongly related to populism are being used for building the tool. The keywords that come to mind are keywords used for the definition of populist language and are known to being strongly related with populism. Namely, the people and the (corrupt) elite. Jagers & Walgrave outlined synonyms that populists use on a regular basis to address the people. Examples are, the public, voters, citizens, taxpayers [4]. The same analysis can be made for the elite, namely, political elite, the state, the media, economic powers and intellectuals [4]. These two keywords are being named in almost every study or research regarding populism. Other keywords are being described vaguely but non besides these two have a strong connection with populism. These two keywords will form a dictionary category and their synonyms will form the data on which the tool can analyze their appearance in tweets. Rooduijn & Pauwels stated that the measurement of the people related words is nearly impossible and they focused on words that refer to anti-elitism [14]. It should be interesting to see if the LIWC analysis with added dictionaries comes up with the similar results. With the two keywords described above we created our own word categories that we can use for the tool. The LIWC tool however, which we use for analyzing texts, has an internal database of 68 word categories. Some of those categories are more related to populism and others are not. For example, Jagers & Walgrave state that the implicit motto of populists is I listen to you because I talk to you [4]. This motto substantiates notions made in Poblete about having a charismatic leader who will fulfill the will of the people even if this could result in breaking the law [12]. This means that talking about him/herself or heavily focusing on the word you is an indication of populism. Also relating to feelings is stated as being populist [4, 12]. But this category is influenced by context. In a relation with the above described keywords of populism the category feel could be perceived as being populist. If there s no notion of any of these keywords it is too hard to tell. 1 represents not populist and 5 means very populist. This is based on a definition given beforehand. This definition is a Dutch translation of the answer of sub-question one. Every participant will rate 40 tweets. Every tweet will be rated three times. This reduces the chance of biased answers and facilitates inter-rater agreement analysis. For analysis purposes all annotated tweets are split into a two sets. The dataset, which is going to be used for analysis of the different word categories of LIWC, consisting 200 tweets. The other set of annotated tweets are used to verify the outcome of the tool versus the rating of the participants. The test set consists of 80 annotated tweets. This means there is a total dataset of 280 annotated tweets. Since every participant rated 40 tweets, seven different versions of the survey had to be made to accommodate all the tweets. Every version of the survey had 3 participants so in total there were 21 participants. Before the participants annotate the tweets the age, gender, political affiliation, political activism and Twitter activity will be asked. These are for general purposes to see if there s any correlation with specific answers. There is no specific group that is the focus of this research. Meaning, all ages, political activism, affiliation and Twitter activity are accepted. Since a definition of populist language and three examples with explanation (as shown in figure 1.) are presented before the annotation part of the survey starts. This should ensure that every participant has the same idea/perception of populist language. The main point with the populist related LIWC categories is that they are all very sensitive to context. They all increase the likeliness of a text or a tweet being populist. But if there is no notion or a combination with the populist defined keywords then no conclusion can be made. The analysis of the different LIWC categories and their influence on the populist score is done in the LIWC and tool analysis part. This results into a still incomplete but work in progress answer to sub-question three, Which (key)words and word-categories are more related to populist language?. Which are the people and the (corrupt) elite. Where these two are keywords related to populist language and form their own respective word category for the automatic analysis tool. 3. SURVEY This part consists of the method(s) used for creation of the survey and the outcome with an explanation for clarification reasons. Discussion of these results are not done in this part. This is done in a combinatory fashion with the LIWC part. 3.1 Survey method In order to facilitate automatic analysis of a linguistic tool or method it should have an annotated dataset or corpus from which it can learn, or in this case, defining populist word categories. The dataset in this research consists of annotated tweets. Participants will score these tweets from 1 to 5. Where Figure 1. example tweets There is however, something to say about the biasedness of young people. young people tend to be less biased in their political views [13]. Also since we re using tweets as being our survey data coupled with the fact that young people tend to be more active on social media [15]. This makes young people a more favorable choice. But one could also say that due to the experience of older people they tend to have a better view on what populist language is. This could also work the other way around. They could be less receptive to the handed definition of populist language. All these facts combined result in an acceptance of every age group for the survey. The final part of the survey consists of participants writing down their definition of populist language. This has been done to compare the participant s views on populism with the given definition and see if there s extreme differences to be found. As a final remark a participant can comment on the way the survey is presented or if anything was unclear. 3

4 The tweets for the survey are random selected from a large archive of tweets. These tweets are from the accounts of every major Dutch political party (See appendix A). Random selection could mean that tweets of for example only one word could also come up as a result. But these are filtered out manually because these would make no sense to be annotated. Only tweets that are of relevant length are selected. Every Twitter account is represented with a minimum of 12 tweets. With the maximum amount of tweets being 14. This ensures that every account is almost equally represented and enables comparison based on the average rating of every Twitter account. After the tweets have been incorporated into the seven different versions of the survey and every participant filled in his/her respective survey the inter-rater agreement based on Fleiss Kappa was calculated [2]. The Fleiss Kappa scoring method is based on Cohen s Kappa, with a single change. It can determine the inter-rater agreement with more than two raters. Where Cohen s Kappa method allows a maximum of two raters. In the case of this research the n (the amount of raters) is three. The N is the amount of tweets. Also called, the amount to be rated instances, which is 40. These are to be distributed over 5 categories (k). In a later instance the k = 3. The next figure is the formula used to calculate the Kappa rating. Figure 2. Kappa formula Where P is the average of all the Pi of every tweet. Pi is the extent to which raters agree with the i-th subject, a tweet. Figure 3. P and Pi The P e consists of the average rating of all the Pj. Where Pj stands for the portion of total ratings that belong to a specific category. Figure 4. P e and Pj The final part of the survey analysis consisted of determining the average rating of every twitter account with regards to the tweets that have been annotated by the participants. This is done by calculating the average rating of the three participants for each individual tweet. This resulted in a score between 1 to 5 for every tweet. Since for each tweet it is known to which Twitter account it belongs to, this is not known for the participants. The rating of each tweet can be combined in a sheet and the average populist scoring of all tweets for every Twitter account can be calculated. The summarized result of these scores are presented in the result section. This average rating is also used in the LIWC analysis part. Where the average rating of every Twitter account is coupled to the different word categories analyzed by the LIWC tool. The method is on how this affects the chosen word categories is described in the LIWC analysis. 3.2 Results survey This part presents the results of the survey methods. Discussion and their relation to the current literature is done in the discussion part. In results shown in table 1 are the average rating outcome of a Twitter account based on the survey. The lowest score that an account can have is 1 and the highest score is a 5. It is important to note that a participant cannot see to which account a tweet belongs to. He or she can have an idea based on political knowledge but it is not explicitly mentioned or shown in the survey. The rating of all the tweets per account combined results in this average scoring list. Which shows promising correlation with the current literature. Table 1. Populist rating Twitter accounts Twitteraccount: Score: geertwilderspvv 3.18 jesseklaver 3.03 jan_nagel 3.00 PartijvdDieren 2.78 emileroemer 2.77 SPnl 2.72 PvdA 2.58 mariannethieme 2.50 APechtold 2.49 groenlinks 2.46 christenunie 2.38 keesvdstaaij 2.36 _PVV pluspartij 2.36 D sybrandbuma 2.22 diederiksamsom 2.21 gertjansegers 2.21 SGPnieuws 2.14 cdavandaag 2.13 VVD 2.00 MinPres 1.61 Table 2 shows the percentage of tweets that has been rated as being populist. Meaning, a tweet has that been rated by at least two participants as being populist. This has been done after the conversion of ratings to a 1 to 3 rating. Otherwise there was very low agreement-rate and resulting small dataset with regards to the tool. This is further explained in the Fleiss Kappa results. 4

5 The percentages for the learn and test set, as well the combined result is outlined. This is in advance and for clarification reasons to show which parts, the populist and nonpopulist parts, of the different sets are used in various other steps along the research timeline. The learn set consists of 200 tweets and the test set contains 80 tweets which makes for a total of 280 tweets. Table 2. Populist tweets percentage Populist tweets Percentage of set Learn set (200): % Test set (80): % Total (280): % The following table shows the inter-rater agreement based on Fleiss Kappa scoring method [2]. The first calculations showed very low kappa scores. These kappa scores were based on a 1 through 5 rating. So the decision was made calculate the kappa scores based on a 1 through 3 rating. This means, compared to the older rating, rating 1 and 2 were combined into one score, and rating 4 and 5 were combined into one score. So a rating of 1 meant low or no populist rating, a 2 meant modest populism and 3 meant high populist rating. The new kappa scores showed a higher agreement. Table 3. Fleiss Kappa scores Survey nr: Rating 1-5 Rating 1-3 Survey Survey Survey Survey Survey Survey Survey It is clearly noticeable that these kappa scores are not very high. Even when reducing the rating to three options the highest score does not surpass A possible explanation for this phenomena is given in the discussion. 4. LIWC ANALYSIS The second phase of the research consists of LIWC word category analysis and combined with the outcome of the survey, will result in the populist related word categories. 4.1 Method The analysis of the LIWC results is a build on the results of the survey. At first the dataset is split into a learn set and a test set. The test set will be used later for validation purposes of the tool. The next step was combining the results of the survey as presented in table 1 with the LIWC analysis. Via this way word categories that are related to populism can be analyzed. The problem here is; how can anyone determine if the one of the 68 different word categories of the Dutch LIWC dictionary is related to populism. Since every category is really context sensitive. This is where the results of the survey come in. Using the average rating that every Twitter account received out of the survey results we can reason which word categories mentioned in the literature are of influence on populist language. For example, the highest populist rated accounts have a tendency to use more negations. But are negations related to populism? This reasoning has been done for every category of the LIWC dictionary. 5 The next step was determining which words should fit into the two extra categories of the LIWC dictionary. These words are based on the previous studies by Jagers & Walgrave and Rooduijn and Pauwels [4, 14]. Although it is stated that the word category the people is very hard to measure without context [14] it is still included in the analysis. This means very specific words have to be chosen in order to not create an artificial inflated rating. The other category can have a bit more flexibility with the related words. Below a table with the words that relate to the word categories. Table 4. Word categories The people The elite volk bankiers bevolking bankier kiezer kapitalist belastingbetaler politici consument elite media waarheid oneerlijk leugen lieg bedrog propaganda ondemocratisch corrupt heersend These words are then programmed into the Dutch dictionary file that LIWC supplies with their tool. Meaning, the two new categories will be added to the LIWC dictionary as category 69 for the people and category 70 for the elite. This is a simple task since a LIWC dictionary file is a plain text file that can be easily altered. Figure 5. Extra categories LIWC The above figure shows the LIWC dictionary with the two extra categories. The Figure only shows a small portion of the entire dictionary file but it is clear that above the modulo operator the word categories are defined. The individual words and their respective word category are outlined below the modulo operator. The new words and categories are just added to the

6 dictionary file, nothing else is altered. If one was able to see more of figure five, the list of words would just continue on the normal alphabetical order of a regular LIWC dictionary. The next step is to analyze the tweets of every Twitter account and combine the outcome of the LIWC analysis with the rating outcome of the survey. The LIWC analysis is done based on word frequency. word counts against a dictionary Word freq. = 100% total word count These results combined with the outcome of the survey show which word categories are used most often by the highest rated populist Twitter accounts. For this, the rated Twitter accounts are split into two groups. One group consisting of the highest rated accounts and one group of the lowest rated accounts. After the split the mean of every word category is determined and compared with the percentages of presence of each word category. If the set of the highest rated populist accounts have a significant higher presence than the lowest rated accounts in a specific word category, then it is likely that this category is has a connection to populist language. WC = N i=1 i.rating if>0 amount of accounts with i.rating>0 This calculates the mean of every word category while also leaving out when a Twitter account(i) has no hits in that specific word category. The mean of the word category is used to calculate if the populist accounts have a higher presence in hits for every category. N H. R. A. with i. rating > WC i=1 N N ( H. R. A. with i. rating > WC ) + ( L. R. A. with i. rating > WC ) i=1 i=1 H.R.A stand for Highest Rated Accounts. L.R.A. stand for Lowest Rated Accounts. WC stands for the mean in each word category. Where i represents every Twitter account. This formula counts the number of accounts in each set (high and low populist rating) for every word category and compares if the H.R.A. has a significant higher presence than the L.R.A. Table 5. Example calculations word category Anger Anger Average (if i.rating> 0) nr. occurrences populist accounts 3 nr. occurrences non-populist accounts 1 % occurrence in populist accounts 75% % occurrence in non-populist accounts 25% The next step is to determine if the added word categories of LIWC have a relation with any of the other word categories. Meaning, does a Twitter account with a rating in one of the added word categories relate to a higher rating in one of the original word categories of LIWC. The final step in LIWC analysis is the comparison of populist and non-populist annotated tweets. For that the outcome of the learn set of the survey is split into populist rated Tweets, those with a majority vote of rating 3. The rest is assumed to be not populist or participants couldn t agree on a majority vote. The size of these two sets is outlined in table 2, namely the populist set is 42 in size and the non-populist set is 158 in size. Again this learn set is analyzed by LIWC to see if there are word categories that have significant more relation with populist language than non-populist language. Again this is based on frequency of occurrence. First off, the percentages of occurrence of every word category for both sets are calculated. Finally, the relative percentage of each word category and its occurrence in the both the populist and non-populist set is calculated. This is done by taking the summation of every LIWC word category. sumw. c. = i. rating i=i Where i represents each word category and N the amount of word categories, in this case 68. Next the frequency of occurrence for each word category for both individual sets is calculated. % occurrence sumw. c. The final calculation consists of the relative percentage of this specific word category where it was measured. Meaning, did it occur in the populist or non-populist set. % occurrence per set of each w. c. % occurrence combined sets of each w. c. Table 6. example calculations word category You N You populist set rating LIWC 1.55 non-populist set rating LIWC 0.6 relative LIWC populist set 1.69% relative LIWC non-populist set 0.68% sum 2.37% relative % of words in populist set 71.20% Relative % of words in non-populist set 28.80% All these different LIWC analysis methods combined results in a candidate set that should be considered to be implemented in the tool. 4.2 Results LIWC The results presented here are the results that show clear relations with populism via the survey results and or the literature. Presenting every LIWC category and its corresponding rating is not relevant for this research. In the analysis where Twitter accounts were separated an interesting relation was shown. Namely, the relation between the highest the elite category, which was added to the dictionary, and the original you category. These were by far the highest you rated Twitter accounts and had the highest rated the elite score. Since we were using a majority vote system to determine if a tweet should be considered populist we ll maintain the same majority vote philosophy for the word category consideration. This translates into a minimum of 66.7% occurrence in the populist set if a word category should be considered for integration into the tool. For the populist Twitter account analysis this resulted in the categories: I, Negations, Assent, Anger, Hearing, Friends, Humans, Future, Achieve, Leisure, Home, Religion and a few 6

7 punctuation categories where Quoting is the one that is most interesting. The categories that only had one or two Twitter account associated with them were left out because this could mean its party or politician specific choice of words, which does not necessarily mean it is related to populism. This resulted in a shortlist of, I, Negate, Anger, Hear, Humans, Future, Achieve and Leisure. This analysis where populist tweets and non-populist tweets have been split resulted in the following shortlist, where also a minimum of 1% total occurrence in both the sets (populist and non-populist) combined was taken into account. Considering the same reasons as described in the previous analysis. The word categories, using these different sets, that fit these criteria are You, Anger and Job. The results show some promising candidates for integration into the tool. However, some of these word categories don t really have a fit with the definition of populist language so these have to be omitted as well. Humans, Hearing and Leisure are categories that have no relation with the definition of populist language and for this reason, will not be implemented. This results into a final list of word categories. Namely, I, Negate, You, Anger, Job and Achieve. Each word category has its own relation to the definition of populism which was given earlier and these word categories relate to subjects which are spoken of most of the time when populism is perceived. 5. TOOL This section briefly describes the construction of the tool and presents the test results. 5.1 Method The automatic analysis tool is built in Python, a free of charge programming language which works from the command line. The database of the tool are the different word categories of LIWC that are found to be related to populism These are supplemented with the created categories the people and the elite. At first, these corresponding words to the word categories are extracted from the LIWC Dutch dictionary and are put in their own separate files, same as the supplementary categories. The tool then uses an input text file, called inputfile_tool.txt which holds the tweet. This tweet will be split into separate words and every word will be compared in relation to the implemented word categories. The rating will be based on a 1 to 5 rating, similar as the survey. This rating is based on number of word category hits, critical category hits ( the people and the elite ) and individual word hits (can have more words of the same category). A tweet starts with a rating of 1. At first, the algorithm checks if a minimum of 1 word of the tweet has a hit a word category. If this is the case, the rating is 2. The next step is determining if a critical category has a hit in the specific tweet. If this is the case the rating goes up to 4, if not, rating stays 2. The final step is calculating how many individual words were a hit, it requires a minimum of 3. If this is the case without a critical word category the rating is 3. If it is found that a critical word category has a keyword in the tweet as well, the rating reaches its maximum rating of Test results Every tweet that was rated populist by the participants was inserted in the tool and was compared with the results of the survey. For ease of viewing the numbers of the tweets analyzed are shown instead of the entire text. Tweet: Table 7. Results tool Rating tool: Rating survey: As it can be seen the results are not that satisfactory, this is explained in the following part. 6. DISCUSSION This section will discuss the results that arose on various parts of the research timeline. The order of discussion is the same as the order of how the research was conducted. At first, the literature study was conducted. This entire research is based on a strong conception of the definition of populist language. If this was a weak or vague definition the successive results could also be inconclusive. There was no real problem with determining a strong definition of populist language. Numerous studies have already tried to grasp this slippery definition. Although it took some time to find a good combination of the populist language definition there were plentiful examples to choose and combine from. The word categories that followed out of the LIWC analysis felt like being strongly connected to the definition. This was due to the fact that most of the studies trying to grasp the populist language definition had examples related to these word categories. A minor downside of the definition was the reception of the survey participants. Although not everyone filled in the comments section of the survey, since it was optional. Some described their concerns that they probably, unknowingly, used their own perception of populist language. As one clearly stated she found it hard to let go of her own definition. A mandatory question of the survey was, give your own definition of populist language. With this question it was possible to see some correlation in the answers but also some differences. Provocative, the public, the people even are named in these answers. But seeing this question is asked after the annotation of the tweets, and people still answer very differently explains the next part of the research. The inter-rater agreement based on Fleiss Kappa is very low. This can largely be explained by the fact that the definition of populist language can be really hard to explain, in words and in 7

8 text. This difference of answers coupled with the low inter-rater shows that still, even if you give a definition beforehand, people tend to follow their own perception. This lead to the decision to combine ratings 1 and 2 and 4 and 5 into a 1 to 3 rating system, as previously explained. This new rating system combined with a majority vote led to a still workable dataset on which the next parts of the research could continue. But it should be clear that compromises had to be made. A first clear, and linear with the literature, outcome was the rating of the Twitter accounts. These results are very similar with those of previous studies about populist language appearance in the more extreme spectrum of the political landscape. The survey results clearly reflect that this is apparent on Twitter as well. This continued into the LIWC analysis. Which also resulted, as previously stated, into similar results in comparison with the literature. Topics that are regularly being regarded as populism are also found as populist related word categories of LIWC. This can be seen in 2 ways. First the highest populist rated Twitter accounts have a higher rating in the word categories determined by literature as being related to populism. Second, when adding two additional word categories, namely the people and the elite the two core perceptions of populism. These two word categories show relation with populist accounts (those rated high by the survey) and having higher percentages in the related internal LIWC categories whilst also having a higher rating in these added categories. Another part of the LIWC analysis was outlining a set of populist rated tweets (majority vote rating 3) against a set of non-populist rated tweets. These populist tweets showed similar word categories scoring higher, percentagewise, than the non-populist tweets. Which further strengthened the position of these related word categories as being candidates for implementation. Ultimately, the goal was to see if automatic analysis of populist language is possible. For that, a very simple word counting tool was built. As the results show, very few ratings (survey vs tool outcome) are in the same range. This can largely be contributed by the fact that populism is still very context sensitive. Maybe the algorithm can be altered a bit but this will not significantly increase the agreement between the tool and the test set of the survey. Seeing this is a very raw way of counting words that are related to populism. When a word or multiple keywords are not in the input tweet, the tool cannot determine populism and thus, rates the tweet very low. There is however, some improvements that can be made. If the input of the tool is altered and the output of the tool is converted into something else, for example, a percentage rating. Whilst also increasing the input size. more keywords can be discovered. One must remember however that the algorithm has to be changed as well. Otherwise the risk exists of inflated results. The main point of improvement is the increase in size for discovery of words. Since several words related to populism are very specific, a simple word counting tool has difficulties coming up with the same result as a survey of people. 7. CONCLUSIONS The results in light of the research questions shows some promising results. What is populism/populist language? This question has been answered in the literature study and ripples through the entire research. Everything is built on this definition and we could say, done a decent job. Is populist language used only in the extremist s parts of the political spectrum? The answer to this question has been partially answered. The literature suggests, but hasn t intensively researched populism in its general appearance [9], that populism is indeed more present on the most extreme parts of the political spectrum. The results of the survey substantiate these claims. Seeing that these results show the more extreme left and right winged parties being rated more populist than others. The LIWC analysis however suggests that the occurrence of populist language is also apparent in the entire political spectrum. Since the two twitter accounts that have the highest the elite and you rating are not both rated as being populist by the survey. Which (key)words and word-categories are more related to populist language? This final sub-question builds on the first and is here to finalize the dictionary for the automatic tool. The three LIWC analysis showed the same word categories as being related to populism. With the bigger analysis coming up with some alternate categories. But due to relation issues with the literature or significance issues (being less than 1% of the total dictionary words) these couldn t be incorporated into the tool. This brings us to the research question: Can populist language be determined automatically via the use of a tool/method? The answer to this question can be a yes, with some remarks. Simple word counting cannot come up with satisfactory results. For that, populism is just to context sensitive. A few results of the tool are promising and with some refinement, a simple word counter could work better. Still, a word counter can do so much. Thus the final answer is that yes, it could eventually be done, some results have promise, but to facilitate automatic analysis of populist language a more intelligent tool has to be built. Most preferably a tool that uses machine learning or a form of artificial intelligence. 8. REFERENCES [1] C. Alvares, and P. Dahlgren, Populism, extremism and media: Mapping an uncertain terrain, European Journal of Communication, vol. 31, no. 1, pp , doi= / [2] J. L. Fleiss, Measuring nominal scale agreement among many raters, Psychological Bulletin, vol. 76, no. 5, pp , doi= /h [3] K. A. Hawkins, Is Chávez populist?: Measuring populist discourse in comparative perspective, Comparative Political Studies, vol. 42, no. 8, pp , doi= / [4] J. Jagers, and S. Walgrave, Populism as political communication style: An empirical study of political parties' discourse in Belgium, European Journal of Political Research, vol. 46, no. 3, pp , doi= /j x [5] B. Krämer, Media populism: A conceptual clarification and some theses on its effects, Communication Theory, vol. 24, no. 1, pp , doi= /comt [6] B. Moffitt, and S. Tormey, Rethinking populism: Politics, mediatisation and political style, Political Studies, vol. 62, no. 2, pp , doi= /

9 [7] C. Mudde, Populist radical right parties in Europe, [8] J. W. Müller, Parsing populism: Who is and who is not a populist these days?, Juncture, vol. 22, no. 2, pp , doi= /j x [9] S. Otjes, and T. Louwerse, Populists in Parliament: Comparing Left-Wing and Right-Wing Populism in the Netherlands, Political Studies, vol. 63, no. 1, pp , doi= / [10] T. Pauwels, Measuring populism: A quantitative text analysis of party literature in Belgium, Journal of Elections, Public Opinion and Parties, vol. 21, no. 1, pp , doi= / [11] J. W. Pennebaker, C. K. Chung, M. Ireland, A. Gonzales, and R. J. Booth. "The Development and Psychometric Properties of LIWC2007," LIWC.net; df. [12] M. E. Poblete, How to assess populist discourse through three current approaches, Journal of Political Ideologies, vol. 20, no. 2, pp , doi= / [13] G. Pollock, T. Brock, and M. Ellison, Populism, ideology and contradiction: mapping young people's political views, Sociological Review, vol. 63, no. S2, pp , doi= / X [14] M. Rooduijn, and T. Pauwels, Measuring populism: Comparing two methods of content analysis, West European Politics, vol. 34, no. 6, pp , doi= / [15] H. B. Vošner, S. Bobek, P. Kokol, and M. J. Krečič, Attitudes of active older Internet users towards online social networking, Computers in Human Behavior, vol. 55, pp , doi= /j.chb [16] W. Wang, I. Hernandez, D. A. Newman, J. He, and J. Bian, Twitter Analysis: Studying US Weekly Trends in Work Stress and Emotion, Applied Psychology, vol. 65, no. 2, pp , doi= /apps

10 APPENDIX A