An Unbiased Measure of Media Bias Using Latent Topic Models

Size: px

Start display at page:

Download "An Unbiased Measure of Media Bias Using Latent Topic Models"

Aron Curtis
5 years ago
Views:

1 An Unbiased Measure of Media Bias Using Latent Topic Models Lefteris Anastasopoulos 1 Aaron Kaufmann 2 Luke Miratrix 3 1 Harvard Kennedy School 2 Harvard University, Department of Government 3 Harvard University, Department of Statistics April 16, 2015

2 Goals 1 Use topic models to measure partisan selection bias and presentation bias in top 13 online media sources. 2 Rank sources by least to most absolute bias. 3 Automate calculation of bias scores for texts.

3 Goals 1 Use topic models to measure partisan selection bias and presentation bias in top 13 online media sources. 2 Rank sources by least to most absolute bias. 3 Automate calculation of bias scores for texts.

4 Goals 1 Use topic models to measure partisan selection bias and presentation bias in top 13 online media sources. 2 Rank sources by least to most absolute bias. 3 Automate calculation of bias scores for texts.

5 Is the Mainstream Media Biased? Most scholars say YES. Groseclose and Milyo (2005) - Liberal/Democrat bias. Gentzkow and Shapiro (2010) - Toward ideology of content consumers. Quinn and Ho (2008) - Liberal bias.

6 Is the Mainstream Media Biased? Most scholars say YES. Groseclose and Milyo (2005) - Liberal/Democrat bias. Gentzkow and Shapiro (2010) - Toward ideology of content consumers. Quinn and Ho (2008) - Liberal bias.

7 Is the Mainstream Media Biased? Most scholars say YES. Groseclose and Milyo (2005) - Liberal/Democrat bias. Gentzkow and Shapiro (2010) - Toward ideology of content consumers. Quinn and Ho (2008) - Liberal bias.

8 Is the Mainstream Media Biased? Most scholars say YES. Groseclose and Milyo (2005) - Liberal/Democrat bias. Gentzkow and Shapiro (2010) - Toward ideology of content consumers. Quinn and Ho (2008) - Liberal bias.

9 What is Media Bias? Presentation Bias - Spin or slant. How the same event/topic/issue is covered. Selection Bias - What is covered.

10 What is Media Bias? Presentation Bias - Spin or slant. How the same event/topic/issue is covered. Selection Bias - What is covered.

11 Presentation Bias: Example 1 Benghazi Anger Over a Film Fuels Anti-American Attacks in Libya and Egypt - New York Times, Sept. 12, 2012.

12 Presentation Bias: Example 1 Benghazi In glory ous bastard riot Muslims rip flag, kill over flick - New York Post, Sept. 12, 2012.

13 Presentation Bias: Example 1 Benghazi Protesters attack U.S. diplomatic compounds in Egypt, Libya - Washington Post, Sept. 12, 2012.

14 Presentation Bias: Example 2 Ferguson Ferguson: dailymail.co.uk, Nov. 24, 2014

15 Presentation Bias: Example 2 Ferguson Ferguson: Fox News, Nov. 24, 2014

16 Presentation Bias: Example 2 Ferguson Ferguson: Huffington Post, Nov. 24, 2014

17 Presentation Bias: Measurement Phrases Used by Republicans in Congress Source: Gentzkow and Shapiro (2010) Comparison of words and phrases in known partisan/ideological sources with articles. Groseclose and Milyo (2005) - Think tank citations. Quinn and Ho (2008) - Supreme Court opinions. Gentzkow and Shapiro (2010) - Congressional speeches.

18 Presentation Bias: Measurement Phrases Used by Republicans in Congress Source: Gentzkow and Shapiro (2010) Comparison of words and phrases in known partisan/ideological sources with articles. Groseclose and Milyo (2005) - Think tank citations. Quinn and Ho (2008) - Supreme Court opinions. Gentzkow and Shapiro (2010) - Congressional speeches.

19 Presentation Bias: Measurement Phrases Used by Republicans in Congress Source: Gentzkow and Shapiro (2010) Comparison of words and phrases in known partisan/ideological sources with articles. Groseclose and Milyo (2005) - Think tank citations. Quinn and Ho (2008) - Supreme Court opinions. Gentzkow and Shapiro (2010) - Congressional speeches.

Presentation Bias: Measurement Phrases Used by Republicans in Congress Source: Gentzkow and Shapiro (2010) Comparison of words and phrases in known partisan/ideological

20 Presentation Bias: Measurement Phrases Used by Republicans in Congress Source: Gentzkow and Shapiro (2010) Comparison of words and phrases in known partisan/ideological sources with articles. Groseclose and Milyo (2005) - Think tank citations. Quinn and Ho (2008) - Supreme Court opinions. Gentzkow and Shapiro (2010) - Congressional speeches.

21 Selection Bias: Example 1 Drudge Report

22 Selection Bias: Example 2 Huffington Post

23 Measuring Selection Bias: Framework S t ={s 1t, s 2t,..., s nt } At any time t there exists a universe of unobserved stories S t.

24 Measuring Selection Bias: Framework N dt S t News sources, N dt, conceptualized as sets of non-random draws from S t. A news source is a set of stories or that has a partisan/ideological valence.

25 Measuring Selection Bias: Framework N dt S t News sources, N dt, conceptualized as sets of non-random draws from S t. A news source is a set of stories or that has a partisan/ideological valence.

26 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

27 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

28 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

29 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

30 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

31 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

32 Measuring Selection Bias: Literature Goal was to restrict S t enough to measure selection bias. Barret and Peake (2007) 1 S t = Each location Bush traveled to in Baum and Groeling (2008) 1 S t = Associated Press and Reuters News Feeds. Larcinese, Puglisi and Snyder (2011) 1 S t = Government reports on the economy.

33 Measuring Selection Bias With topic models we are able to construct comprehensive measures of selection bias automatically. Very important in an era of internet news where spin-free narratives can be easily constructed.

34 Measuring Selection Bias With topic models we are able to construct comprehensive measures of selection bias automatically. Very important in an era of internet news where spin-free narratives can be easily constructed.

35 Measuring Selection Bias [I wanted to find a] single, emblematic college rape case [that would show] what it s like to be on campus now...where not only is rape so prevalent but also that there s this pervasive culture of sexual harassment/rape culture... - Sabrina Erdely, Rolling Stone Magazine

36 A Model of Selection Bias S tk ={s t1, s t2,..., s tk } S t Multinom(β) θ dtk ={θ dt1, θ dt2,..., θ dtk } θ dt Dir(α) I tk ={I t1, I t2,..., I tk } I t Multinom(γ) S tk : All news at time t is comprised of k topics. θ dtk : Each news source d has a distribution over these k topics. I tk : Each topic has a partisan valence - Democrat, Republican, Neutral.

37 A Model of Selection Bias S tk ={s t1, s t2,..., s tk } S t Multinom(β) θ dtk ={θ dt1, θ dt2,..., θ dtk } θ dt Dir(α) I tk ={I t1, I t2,..., I tk } I t Multinom(γ) S tk : All news at time t is comprised of k topics. θ dtk : Each news source d has a distribution over these k topics. I tk : Each topic has a partisan valence - Democrat, Republican, Neutral.

38 A Model of Selection Bias S tk ={s t1, s t2,..., s tk } S t Multinom(β) θ dtk ={θ dt1, θ dt2,..., θ dtk } θ dt Dir(α) I tk ={I t1, I t2,..., I tk } I t Multinom(γ) S tk : All news at time t is comprised of k topics. θ dtk : Each news source d has a distribution over these k topics. I tk : Each topic has a partisan valence - Democrat, Republican, Neutral.

39 A Model of Selection Bias SB dt =θdt T I t K = θ dtk I tk k=1 SB d - Selection bias for a news source d is thus an average of the partisan valences for each topic k weighted by

40 A Model of Selection Bias Bias Value Democrat-Skewed Content 1 < SB dt < 0 Neutral Content SB dt = 0 Republican-Skewed Content 0 < SB dt < 1 Values of SB dt and Direction of Bias Define Democrat = 1, Neutral = 0, Republican = 1.

41 A Model of Selection Bias: Examples θ NYT,2014 = (0.4, 0.2, 0.4) θ FoxNews,2014 = (0.4, 0.55, 0.05) Imagine all news organizations cover only three topics: Immigration, Terrorism and Global Warming. New York Times - 40% of coverage to Immigration, 20% to Terrorism, 40% to Global Warming Fox News - 40% of coverage to Immigration, 55% to Terrorism, 5% to Global Warming

42 A Model of Selection Bias: Examples θ NYT,2014 = (0.4, 0.2, 0.4) θ FoxNews,2014 = (0.4, 0.55, 0.05) Imagine all news organizations cover only three topics: Immigration, Terrorism and Global Warming. New York Times - 40% of coverage to Immigration, 20% to Terrorism, 40% to Global Warming Fox News - 40% of coverage to Immigration, 55% to Terrorism, 5% to Global Warming

43 A Model of Selection Bias: Examples θ NYT,2014 = (0.4, 0.2, 0.4) θ FoxNews,2014 = (0.4, 0.55, 0.05) Imagine all news organizations cover only three topics: Immigration, Terrorism and Global Warming. New York Times - 40% of coverage to Immigration, 20% to Terrorism, 40% to Global Warming Fox News - 40% of coverage to Immigration, 55% to Terrorism, 5% to Global Warming

44 A Model of Selection Bias: Examples Valences are: Immigration = 0 (Neutral), Terrorism = 0.5 (Republican), Global Warming= 0.5 (Democrat). Thus... I 2014 = (0, 0.5, 0.5) 3 SB NYT,2014 = θ knyt,2014 I k k=1 = (0.40 0) + ( ) + ( ) = SB FoxNews,2014 = θ kfox,2014 I k k=1 = (0.40 0) + ( ) + ( ) = 0.25

45 Modeling News Sources Using the Latent Dirichlet Allocation (LDA) LDA is a generative model of texts developed by Blei, Ng and Jordan (2003). LDA constructs a corpus using a documents and vocabularies. Used to automatically categorize large groups of documents.

46 Modeling News Sources Using the Latent Dirichlet Allocation (LDA) LDA is a generative model of texts developed by Blei, Ng and Jordan (2003). LDA constructs a corpus using a documents and vocabularies. Used to automatically categorize large groups of documents.

47 Modeling News Sources Using the Latent Dirichlet Allocation (LDA) LDA is a generative model of texts developed by Blei, Ng and Jordan (2003). LDA constructs a corpus using a documents and vocabularies. Used to automatically categorize large groups of documents.

48 Modeling News Sources Using the LDA: Overview TOPICS Topic1 Topic2.Topic30 CORPUS Internet News DOCUMENTS New York Times WalStreet Journal... LosAngeles Times TOPIC PROPORTIONS General Model of Internet News Sources Using the Latent Dirichlet Allocation

49 Modeling News Sources Using the LDA: Overview Corpus - 13 internet sources: CNN, NY Times, Fox News, LA Times, USA Today, Washington Post, Huffington Post, Chicago Sun Times, NY Daily News, ABC News, Wall Street Journal, NBC News. Documents - Each d = 1,..., 13 news sources. Topics - Arbitrarily set to K = 30.

50 Modeling News Sources Using the LDA: Overview Corpus - 13 internet sources: CNN, NY Times, Fox News, LA Times, USA Today, Washington Post, Huffington Post, Chicago Sun Times, NY Daily News, ABC News, Wall Street Journal, NBC News. Documents - Each d = 1,..., 13 news sources. Topics - Arbitrarily set to K = 30.

51 Modeling News Sources Using the LDA: Overview Corpus - 13 internet sources: CNN, NY Times, Fox News, LA Times, USA Today, Washington Post, Huffington Post, Chicago Sun Times, NY Daily News, ABC News, Wall Street Journal, NBC News. Documents - Each d = 1,..., 13 news sources. Topics - Arbitrarily set to K = 30.

52 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D 1 β k Dir(ψ), where k {1,..., 30} - the distribution over words that defines each of the K = 30 latent topics assumed to encompass the 12 news sources. 2 θ d Dir(α), where d {1,..., 13} - the distribution over topics for each news source. This, in combination with topic ideological valence I determines selection bias. 3 z d,n - topic assignment of the n th word in the d th news source. 4 w d,n - the n th word of the d th news source.

53 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D 1 β k Dir(ψ), where k {1,..., 30} - the distribution over words that defines each of the K = 30 latent topics assumed to encompass the 12 news sources. 2 θ d Dir(α), where d {1,..., 13} - the distribution over topics for each news source. This, in combination with topic ideological valence I determines selection bias. 3 z d,n - topic assignment of the n th word in the d th news source. 4 w d,n - the n th word of the d th news source.

54 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D 1 β k Dir(ψ), where k {1,..., 30} - the distribution over words that defines each of the K = 30 latent topics assumed to encompass the 12 news sources. 2 θ d Dir(α), where d {1,..., 13} - the distribution over topics for each news source. This, in combination with topic ideological valence I determines selection bias. 3 z d,n - topic assignment of the n th word in the d th news source. 4 w d,n - the n th word of the d th news source.

55 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D 1 β k Dir(ψ), where k {1,..., 30} - the distribution over words that defines each of the K = 30 latent topics assumed to encompass the 12 news sources. 2 θ d Dir(α), where d {1,..., 13} - the distribution over topics for each news source. This, in combination with topic ideological valence I determines selection bias. 3 z d,n - topic assignment of the n th word in the d th news source. 4 w d,n - the n th word of the d th news source.

56 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D 30 k=1 p(β k ψ) 12 d=1 ( p(θ d α) p(θ, z, w, β ψ, α) = N ) p(z d,n θ d )p(w d,n z d,n, β k ) n=1 Full model of news sources and topics.

57 Measuring Partisan Valence SB d = θ T d I Key parameters and distributions estimated using variational expectation maximization algorithm (VEM) (Wainwright and Jordan 2008). Measuring topic partisan valence. 1 Human coders. 2 Human coders + party platforms from 1980-Present.

58 Measuring Partisan Valence SB d = θ T d I Key parameters and distributions estimated using variational expectation maximization algorithm (VEM) (Wainwright and Jordan 2008). Measuring topic partisan valence. 1 Human coders. 2 Human coders + party platforms from 1980-Present.

59 Measuring Partisan Valence SB d = θ T d I Key parameters and distributions estimated using variational expectation maximization algorithm (VEM) (Wainwright and Jordan 2008). Measuring topic partisan valence. 1 Human coders. 2 Human coders + party platforms from 1980-Present.

60 Measuring Partisan Valence SB d = θ T d I Key parameters and distributions estimated using variational expectation maximization algorithm (VEM) (Wainwright and Jordan 2008). Measuring topic partisan valence. 1 Human coders. 2 Human coders + party platforms from 1980-Present.

61 Measuring Partisan Valence: Human Coders I k Multinom(Π) Π = {Π Democrat, Π Neutral, Π Republican } I = (E[I 1 ], E[I 2 ],..., E[I 30 ]) Topics labeled manually and automatically. Topics presented to Mechanical Turk workers. Do you think Republicans, Democrats or Neither talk more about [Topic k].

62 Measuring Partisan Valence: Human Coders I k Multinom(Π) Π = {Π Democrat, Π Neutral, Π Republican } I = (E[I 1 ], E[I 2 ],..., E[I 30 ]) Topics labeled manually and automatically. Topics presented to Mechanical Turk workers. Do you think Republicans, Democrats or Neither talk more about [Topic k].

63 Measuring Partisan Valence: Human Coders I k Multinom(Π) Π = {Π Democrat, Π Neutral, Π Republican } I = (E[I 1 ], E[I 2 ],..., E[I 30 ]) Topics labeled manually and automatically. Topics presented to Mechanical Turk workers. Do you think Republicans, Democrats or Neither talk more about [Topic k].

64 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

65 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

66 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

67 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

68 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

69 Measuring Partisan Valence: Using Human Coders and Party Platforms Data: Party platforms for Democratic and Republican parties from 1980-Present. Procedure: 1 Extract and label topics from news sources. 2 Extract topics from Democratic and Republican party platforms. 3 Mechanical Turk workers use labels from news sources to label platform topics. 4 For each topic k calculate proportion of topics labels Democrat and Republican.

70 Preliminary Results: Topics Topic 1 Topic 2 Topic 3 Topic 4 Topic 5 Topic 6 Police Shootings Robert Durst ISIS Academy Awards Crime Taxes & Budget 1 polic durst often show charg state 2 offic new polic year court tax 3 shot airport kill award case year 4 shoot mall man oscar prison pay 5 man say islamic best murder percent 6 fire told state new arrest million 7 depart polic attack will attorney budget 8 report man hostag one death feder 9 kill report syria time told plan 10 two york video actor prosecutor will

71 Preliminary Results: Topic Proportions by Source

72 Preliminary Results: Topic Proportions by Source

73 Preliminary Results: Topic Proportions by Source

74 Questions? Comments?

75 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D p(θ d α) = K i=1 Γ(α i) Γ ( K i=1 α ) i 30 i=1 θ α i 1 di Distribution over topic proportions.

76 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D p(β k ψ) = N i=1 Γ(ψ i) N Γ ( N i=1 ψ ) i i=1 β ψ i 1 ki Distribution over words for each topic..

77 Modeling News Sources Using the LDA: Nuts and Bolts K ψ βk α θd zd,n wd,n N D p(z d,n θ d );z d,n Multinom(θ d ) p(w d,n z d,n, β k ;w d,n Multinom(β k ) Distributions of topic assignments and word assignments.

FAIR AND BALANCED? QUANTIFYING MEDIA BIAS THROUGH CROWDSOURCED CONTENT ANALYSIS

Public Opinion Quarterly, Vol. 80, Special Issue, 2016, pp. 250 271 FAIR AND BALANCED? QUANTIFYING MEDIA BIAS THROUGH CROWDSOURCED CONTENT ANALYSIS CEREN BUDAK* SHARAD GOEL JUSTIN M. RAO Abstract It is