Citizens & Ideological Text April 19, PDF Free Download

Citizens & Ideological Text April 19, 2015 Brice D. L. Acree & Michael B. MacKuen University of North Carolina at Chapel Hill MPSA 2015 Acree & MacKuen Citizen Evaluations of Ideological Text 1/29

Outline Roadmap 1 Project Motivation 2 Data Collection 3 Results 4 Discussion Acree & MacKuen Citizen Evaluations of Ideological Text 2/29

Project Motivation Elites use language strategically Acree & MacKuen Citizen Evaluations of Ideological Text 3/29

Project Motivation Elites use language strategically Prop. of time spent in Left ideologies 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 Obama's Etch a Sketch 0.3 0.4 0.5 0.6 0.7 McCain's Etch a Sketch 0.3 0.4 0.5 0.6 0.7 Romney's Etch a Sketch 0.3 0.4 0.5 0.6 0.7 Prop. of time spent in Right ideologies Primary transitioning to General Acree & MacKuen Citizen Evaluations of Ideological Text 4/29

Project Motivation Elites use language strategically Acree & MacKuen Citizen Evaluations of Ideological Text 5/29

Project Motivation Elites use language strategically Acree & MacKuen Citizen Evaluations of Ideological Text 6/29

Project Motivation Do voters read rhetoric the way we do? (1) Can voters decipher the ideological content of language? Acree & MacKuen Citizen Evaluations of Ideological Text 7/29

Project Motivation Do voters read rhetoric the way we do? (1) Can voters decipher the ideological content of language? Can certain phrases cue voters to the ideology of a speaker? Can the effect exist in the absence of framing or topic cues? Acree & MacKuen Citizen Evaluations of Ideological Text 7/29

Project Motivation Do voters read rhetoric the way we do? (1) Can voters decipher the ideological content of language? Can certain phrases cue voters to the ideology of a speaker? Can the effect exist in the absence of framing or topic cues? How subtle can cues become for voters to see the difference? Acree & MacKuen Citizen Evaluations of Ideological Text 7/29

Project Motivation Do voters read rhetoric the way we do? (1) Can voters decipher the ideological content of language? Can certain phrases cue voters to the ideology of a speaker? Can the effect exist in the absence of framing or topic cues? How subtle can cues become for voters to see the difference? Does respondent ideology affect how s/he perceives the language? Acree & MacKuen Citizen Evaluations of Ideological Text 7/29

Project Motivation What is the process? (2) How do citizens read and understand political language? Acree & MacKuen Citizen Evaluations of Ideological Text 8/29

Project Motivation What is the process? (2) How do citizens read and understand political language? What signals are citizens looking for? Acree & MacKuen Citizen Evaluations of Ideological Text 8/29

Project Motivation What is the process? (2) How do citizens read and understand political language? What signals are citizens looking for? Which signals do respondents receive, and which go amiss? Does the process vary by respondent ideology, party or preference? Acree & MacKuen Citizen Evaluations of Ideological Text 8/29

Project Motivation What is the process? (2) How do citizens read and understand political language? What signals are citizens looking for? Which signals do respondents receive, and which go amiss? Does the process vary by respondent ideology, party or preference? Do citizens ever see signals that aren t actually in the text? Acree & MacKuen Citizen Evaluations of Ideological Text 8/29

Theory Citizen sophistication surely poses problems here... Why should citizens perceive changes in political rhetoric? Acree & MacKuen Citizen Evaluations of Ideological Text 9/29

Theory Citizen sophistication surely poses problems here... Why should citizens perceive changes in political rhetoric? Plenty of research shows that citizens are not terribly sophisticated. Framing and priming effects do exist. Acree & MacKuen Citizen Evaluations of Ideological Text 9/29

Theory Citizen sophistication surely poses problems here... Why should citizens perceive changes in political rhetoric? Plenty of research shows that citizens are not terribly sophisticated. Framing and priming effects do exist. The tone of rhetoric can serve a similar role. Acree & MacKuen Citizen Evaluations of Ideological Text 9/29

Theory Citizen sophistication surely poses problems here... Why should citizens perceive changes in political rhetoric? Plenty of research shows that citizens are not terribly sophisticated. Framing and priming effects do exist. The tone of rhetoric can serve a similar role. Language expresses sentiment in informative ways. Acree & MacKuen Citizen Evaluations of Ideological Text 9/29

Theory Motivating Example of Rhetorical Changes Consider the following example Acree & MacKuen Citizen Evaluations of Ideological Text 10/29

Theory Motivating Example of Rhetorical Changes Consider the following example Arizona residents were increasingly beleaguered by immigrants crossing their southern border. Arizona residents were increasingly besieged by illegals crossing their southern border. Acree & MacKuen Citizen Evaluations of Ideological Text 10/29

Theory Motivating Example of Rhetorical Changes Consider the following example Arizona residents were increasingly beleaguered by immigrants crossing their southern border. Arizona residents were increasingly besieged by illegals crossing their southern border. One word change has a sizable effect on the tone of the sentence. Acree & MacKuen Citizen Evaluations of Ideological Text 10/29

Project Design Experiments fielded using Mechanical Turk Three studies to examine citizen perception of ideological language Acree & MacKuen Citizen Evaluations of Ideological Text 11/29

Project Design Experiments fielded using Mechanical Turk Three studies to examine citizen perception of ideological language 1 Identifying phrases without context Acree & MacKuen Citizen Evaluations of Ideological Text 11/29

Project Design Experiments fielded using Mechanical Turk Three studies to examine citizen perception of ideological language 1 Identifying phrases without context 2 Identifying passages without policy statements Acree & MacKuen Citizen Evaluations of Ideological Text 11/29

Project Design Experiments fielded using Mechanical Turk Three studies to examine citizen perception of ideological language 1 Identifying phrases without context 2 Identifying passages without policy statements 3 Experimentally changing language in a short statement Acree & MacKuen Citizen Evaluations of Ideological Text 11/29

Project Design Experiments fielded using Mechanical Turk Three studies to examine citizen perception of ideological language 1 Identifying phrases without context 2 Identifying passages without policy statements 3 Experimentally changing language in a short statement Acree & MacKuen Citizen Evaluations of Ideological Text 12/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Example: Top libertarian terms include personal property, Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Example: Top libertarian terms include personal property, founding fathers, Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Example: Top libertarian terms include personal property, founding fathers, and Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Example: Top libertarian terms include personal property, founding fathers, and... Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

studies Design: Commonalities Source of texual data used in our We identify ideological language using machine learning We use data from Sim, Acree, Smith & Gross (2013) Uses a corpus of nearly 200 books and magazines Modeling yields words/phrases associated with 7 ideological groups Example: Top libertarian terms include personal property, founding fathers, and... raw milk Acree & MacKuen Citizen Evaluations of Ideological Text 13/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Between 50 and 150 respondents in each design Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Between 50 and 150 respondents in each design Standard battery of political and demographic questions Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Between 50 and 150 respondents in each design Standard battery of political and demographic questions Skews slightly Democratic, female, young Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Between 50 and 150 respondents in each design Standard battery of political and demographic questions Skews slightly Democratic, female, young Set of political knowledge questions Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Commonalities Subject recruitment and instruments We recruit respondents on Amazon s Mechanical Turk Between 50 and 150 respondents in each design Standard battery of political and demographic questions Skews slightly Democratic, female, young Set of political knowledge questions Skews knowledgeable Acree & MacKuen Citizen Evaluations of Ideological Text 14/29

Design: Main Experiment Manipulating language Subjects see an excerpt from a Question-and-Answer with a blogger Acree & MacKuen Citizen Evaluations of Ideological Text 15/29

Design: Main Experiment Manipulating language Subjects see an excerpt from a Question-and-Answer with a blogger Q: Immigration has received a lot of attention in the past few years. Do you consider immigration to be an important political debate for today? Acree & MacKuen Citizen Evaluations of Ideological Text 15/29

Design: Main Experiment Manipulating language Subjects receive either this response: Acree & MacKuen Citizen Evaluations of Ideological Text 16/29

Design: Main Experiment Manipulating language Subjects receive either this response: Yes, I think immigration is an important issue. There are more than 12 million undocumented workers in this country. These immigrants tend to work for less than an average wage, and live in poverty, which puts incredible stress on their families as well as straining federal programs. Plus, the fact that these undocumented workers accept lower wages undercuts the labor market, pushing down wages for everyone. So yes, immigration is an important debate to have. It s a question of social justice and economic opportunity. Acree & MacKuen Citizen Evaluations of Ideological Text 16/29

Design: Main Experiment Manipulating language Or subjects receive this response: Yes, I think immigration is an important issue. There are more than 12 million illegal aliens in this country. These illegals tend to work for less than market wages, and live in poverty, which puts incredible stress on their families as well as straining the welfare state. Plus, the fact that these illegal workers accept lower wages undercuts the free market, pushing down wages for hard working Americans. So yes, illegal immigration is an important debate to have. It s a question of justice and economic liberty. Acree & MacKuen Citizen Evaluations of Ideological Text 17/29

Design: Commonalities Subjects & instruments We ask participants to place the blogger s likely ideological perspective Acree & MacKuen Citizen Evaluations of Ideological Text 18/29

Design: Commonalities Subjects & instruments We ask participants to place the blogger s likely ideological perspective Place the blogger on a scale from 1 to 7 Acree & MacKuen Citizen Evaluations of Ideological Text 18/29

Design: Commonalities Subjects & instruments We ask participants to place the blogger s likely ideological perspective Place the blogger on a scale from 1 to 7 Is it likely that the blogger has extreme views? Acree & MacKuen Citizen Evaluations of Ideological Text 18/29

Design: Commonalities Subjects & instruments We ask participants to place the blogger s likely ideological perspective Place the blogger on a scale from 1 to 7 Is it likely that the blogger has extreme views? Can you explain how you arrived at your decision? Acree & MacKuen Citizen Evaluations of Ideological Text 18/29

Design: Commonalities Subjects & instruments We ask participants to place the blogger s likely ideological perspective Place the blogger on a scale from 1 to 7 Is it likely that the blogger has extreme views? Can you explain how you arrived at your decision? Do you agree with the blogger s statement? Acree & MacKuen Citizen Evaluations of Ideological Text 18/29

Results Participants respond strongly to the language Respondent Ratings of Blogger in Neutral/Liberal Condition 15 10 5 0 1 2 3 4 5 6 7 R's Ideology Assessment of Blogger Figure: Ratings of blogger in neutral condition Acree & MacKuen Citizen Evaluations of Ideological Text 19/29

Results Participants respond strongly to the language Respondent Ratings of Blogger in Conservative Condition 30 20 10 0 1 2 3 4 5 6 7 R's Ideology Assessment of Blogger Figure: Ratings of blogger in conservative condition Acree & MacKuen Citizen Evaluations of Ideological Text 20/29

Results How do respondents vary in their response? Some key results: Acree & MacKuen Citizen Evaluations of Ideological Text 21/29

Results How do respondents vary in their response? Some key results: Respondents rate the passages in the expected direction Acree & MacKuen Citizen Evaluations of Ideological Text 21/29

Results How do respondents vary in their response? Some key results: Respondents rate the passages in the expected direction Ratings of the conservative text are 1.1 points higher than neutral text Ratings of neutral text not discernible from scale midpoint Agreement correlates with seeing the message as moderate Acree & MacKuen Citizen Evaluations of Ideological Text 21/29

Results How do respondents vary in their response? Some key results: Respondents rate the passages in the expected direction Ratings of the conservative text are 1.1 points higher than neutral text Ratings of neutral text not discernible from scale midpoint Agreement correlates with seeing the message as moderate 68% rate the blogger as moderate; 55% agree with statement Acree & MacKuen Citizen Evaluations of Ideological Text 21/29

Results How do respondents vary in their response? Some key results: Respondents rate the passages in the expected direction Ratings of the conservative text are 1.1 points higher than neutral text Ratings of neutral text not discernible from scale midpoint Agreement correlates with seeing the message as moderate 68% rate the blogger as moderate; 55% agree with statement Agreement increases Pr(Rate Moderate) by about 80% Acree & MacKuen Citizen Evaluations of Ideological Text 21/29

Results How do respondents vary in their response? Some key results broken up by various groups: Acree & MacKuen Citizen Evaluations of Ideological Text 22/29

Results How do respondents vary in their response? Some key results broken up by various groups: Conservatives see a bigger difference than liberals Acree & MacKuen Citizen Evaluations of Ideological Text 22/29

Results How do respondents vary in their response? Some key results broken up by various groups: Conservatives see a bigger difference than liberals Difference for conservatives (1.75) larger than for liberals (0.96) Conservatives see the neutral message as 0.8 points more liberal No difference between liberals/conservatives in rating the treatment text Acree & MacKuen Citizen Evaluations of Ideological Text 22/29

Results How do respondents vary in their response? Some key results broken up by various groups: Conservatives see a bigger difference than liberals Difference for conservatives (1.75) larger than for liberals (0.96) Conservatives see the neutral message as 0.8 points more liberal No difference between liberals/conservatives in rating the treatment text Same pattern substituting respondent party for ideology Acree & MacKuen Citizen Evaluations of Ideological Text 22/29

Results How do respondents vary in their response? Some key results broken up by various groups: Conservatives see a bigger difference than liberals Difference for conservatives (1.75) larger than for liberals (0.96) Conservatives see the neutral message as 0.8 points more liberal No difference between liberals/conservatives in rating the treatment text Same pattern substituting respondent party for ideology No differences by sophistication or attention Sophisticates no more likely to give the right categorization Acree & MacKuen Citizen Evaluations of Ideological Text 22/29

Results Participants respond strongly to the language Respondent Ratings of Blogger 5 Average Blogger Rating 4 3 Neutral::Repub Neutral::Dem Con::Rep Con::Dem 1:4 Figure: Mean ratings in each condition, by respondent party, with 90% intervals. Acree & MacKuen Citizen Evaluations of Ideological Text 23/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic 37% cite language as a primary cue informing their choice Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic 37% cite language as a primary cue informing their choice Ex: He uses the phrase illegals, which seems like something someone who is conservative would say. Another thing that makes him appear very conservative is his note at the end about economic liberty... which conservative people focus a lot of attention on. Ex: I chose that number because I think liberals are concerned about social justice and trying to make the world more equitable, which is not possible to do. Liberals probably want the government to intervene and solve the problem of immigration, which just will not work. Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic 37% cite language as a primary cue informing their choice Ex: He uses the phrase illegals, which seems like something someone who is conservative would say. Another thing that makes him appear very conservative is his note at the end about economic liberty... which conservative people focus a lot of attention on. Ex: I chose that number because I think liberals are concerned about social justice and trying to make the world more equitable, which is not possible to do. Liberals probably want the government to intervene and solve the problem of immigration, which just will not work. Many ( 30%) use the ( fixed) frame as an indicator Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic 37% cite language as a primary cue informing their choice Ex: He uses the phrase illegals, which seems like something someone who is conservative would say. Another thing that makes him appear very conservative is his note at the end about economic liberty... which conservative people focus a lot of attention on. Ex: I chose that number because I think liberals are concerned about social justice and trying to make the world more equitable, which is not possible to do. Liberals probably want the government to intervene and solve the problem of immigration, which just will not work. Many (30%) use the (fixed) frame as an indicator but the treatment effect isn t attenuated Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What is the reasoning process? We code respondent open-ended explanations of their logic 37% cite language as a primary cue informing their choice Ex: He uses the phrase illegals, which seems like something someone who is conservative would say. Another thing that makes him appear very conservative is his note at the end about economic liberty... which conservative people focus a lot of attention on. Ex: I chose that number because I think liberals are concerned about social justice and trying to make the world more equitable, which is not possible to do. Liberals probably want the government to intervene and solve the problem of immigration, which just will not work. Many (30%) use the (fixed) frame as an indicator but the treatment effect isn t attenuated Some attempt to infer policy positions from the text, or else draw inferences from what the blogger does not write. Acree & MacKuen Citizen Evaluations of Ideological Text 24/29

Results What if the frame is not neutral? We push this harder with a more extreme frame: Acree & MacKuen Citizen Evaluations of Ideological Text 25/29

Results What if the frame is not neutral? We push this harder with a more extreme frame: We use the following passage: We must address immigration immediately. There are 12 million [illegal aliens/undocumented workers] in this country, and we re going to have another 12 million if we don t address the underlying problems. We have only passed incremental changes in the past, and that is not enough. We need to increase border [patrols/security] to address our [open] border, and we need to find a way to handle the millions of [aliens/foreign born workers] already in this country. American [working men/workers] demand action, and I stand with them. I support comprehensive immigration reform. Treatment effect is smaller, in the expected direction, but not statistically distinguishable Much smaller sample (50); boundary effect Acree & MacKuen Citizen Evaluations of Ideological Text 25/29

Thoughts What have we learned? Rhetoric can send important signals Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thoughts What have we learned? Rhetoric can send important signals Citizens perceive ideological content in rhetoric Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thoughts What have we learned? Rhetoric can send important signals Citizens perceive ideological content in rhetoric The effect exists even when the frame and policy are fixed Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thoughts What have we learned? Rhetoric can send important signals Citizens perceive ideological content in rhetoric The effect exists even when the frame and policy are fixed Some people consciously try to analyze the rhetoric But even those using other heuristics arrive at similar ends The tone of language plays some subconscious role in how we make inferences about political messages Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thoughts What have we learned? Rhetoric can send important signals Citizens perceive ideological content in rhetoric The effect exists even when the frame and policy are fixed Some people consciously try to analyze the rhetoric But even those using other heuristics arrive at similar ends The tone of language plays some subconscious role in how we make inferences about political messages Next steps: Other issue areas, conflicting policy/frame/rhetoric, and... Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thoughts What have we learned? Rhetoric can send important signals Citizens perceive ideological content in rhetoric The effect exists even when the frame and policy are fixed Some people consciously try to analyze the rhetoric But even those using other heuristics arrive at similar ends The tone of language plays some subconscious role in how we make inferences about political messages Next steps: Other issue areas, conflicting policy/frame/rhetoric, and...? Acree & MacKuen Citizen Evaluations of Ideological Text 26/29

Thank you We look forward to your comments and questions. Acree & MacKuen Citizen Evaluations of Ideological Text 27/29

Using Appendices 1 Figure??: Deviation from Ideological Balance Acree & MacKuen Citizen Evaluations of Ideological Text 28/29

Some Figure in Appendix Back to Appendix Acree & MacKuen Citizen Evaluations of Ideological Text 29/29