Biases in Message Credibility and Voter Expectations EGAP Preregisration GATED until June 28, 2017 Summary. Election polls in horserace coverage characterize a competitive information environment with inconsistent results being published within short intervals of time. And these small or large differences in results do not always reflect real electoral shift. In response to variability in the results and methodological quality of polls, over the last decade, we have seen the increasing use and media coverage of polling aggregations/averages from sites like RealClearPolitics, FiveThirtyEight, CNN poll of polls, and Huffpost Pollster. Polls vary in direction, in terms of the gap between candidates, and poll of polls either replicate or stand in contrast to individual poll findings. Whereas a large literature documents the effects of polls on citizens perceptions, attitudes, and political behavior in various contexts, these dynamic aspects of the competitive information environment as well as the perceptions of polling averages have not been investigated. Such an information environment provides a perfect scenario to test motivational biases and objective updating of beliefs (Bayesian updating) in an experimental set up where we present, consecutively, various combinations of multiple messages. We set-up an online survey experiment in order to leverage these dynamics of horserace coverage in the context of 2016 U.S. presidential elections. We test through a large national survey experiment (N=1200, Qualtrics), the extent to which people engage in motivational resistance (motivated reasoning) or Bayesian updating (objective assessment of poll finding) while they encounter consecutively presented consistent (or inconsistent) singular poll reports (or polling averages of five recent polls) that represent small (or large gaps) between the presumptive nominees (Clinton and Trump). As outcome measures, we focus on perceived credibility of messages and voter expectations of support for candidates. 1
Table 1. Manipulations Set up Condition Time 1 Trump-Clinton Time 1 Time 2 Trump-Clinton Time 2 c1 summative T+2 45-43 singular T+6 47-41 c2 summative C+2 43-45 singular C+6 41-47 c3 summative T+2 45-43 singular C+2 44-46 c4 summative C+2 43-45 singular T+2 46-44 c5 singular T+2 44-42 summative T+2 45-43 c6 singular C+2 42-44 summative C+2 43-45 c7 singular T+6 47-41 summative T+2 45-43 c8 singular C+6 41-47 summative C+2 43-45 c9 singular C+2 44-46 summative T+2 45-43 c10 singular T+2 46-44 summative C+2 43-45 c11a singular T+2 45-43 singular C+2 44-46 c11b Singular C+2 44-46 Singular T+2 45-43 c12a singular C+2 44-46 singular C+2 43-45 c12b singular T+2 44-46 singular T+2 43-45 Table Notes. 1. T is Trump, C is Clinton. 2. Singular is single poll result and Summative is a polling average result 3. Plus (+) sign mean that candidate s % is leading the other candidate by that amount. 4. Conditions 1 through 10 have also 2 versions (a and b) where we either include or exclude a basic explanation of the logic behind polling averages. It is technically another manipulation although we treat it as a dosage (strenght of message) and will test its impact as well as running analyses with a and bs combined for each condition. Methodology. 1. Online survey experiment on a national sample of N=1200 respondents, Qualtrics panel where we present hypothetical news stories 2. Within subjects cross-sectional study with 2 sets of manipulation screening and outcome question answering. 3. Outcome variables: Poll Credibility (asked both at t1 and t2), Voter Prediction (asked only at t2). 4. We plan to conduct OLS and logistic regression analyses with interaction tests to predict the outcome variables. Manipulations. Please interpret below manipulations togethr with the Table 1 above and example manipulation stories at the end of this document. 1. Poll result: Clinton lead vs Trump lead 2. Poll gap: 2% vs 6% (only for singular polls) 3. Poll type: Singular poll vs Aggregate poll 4. Poll consistency: Consistent vs Inconsistent 5. Poll order: Singular+Summative vs Summative+Singular vs Singular+Singular 6. Theoretical explanation: Absence of presence of 3 rd paragraph in polling average stories (only for polling averages, see Note 4 in Table Notes above) 2
Hypotheses. For simplicity, we refer only to poll credibility below, but these hypotheses apply to the direction and magnitude of shifts in voter predictions as well. Bias is operationalized as discounting of unfavorable message, both in credibility and election prediction assessments. 1. Respondents will discredit unfavorable polls (those showing their candidate as losing) more than those favorable polls. (c1 through c12b by party identification at t1) 2. Summative polls will have greater effect size than singular polls (at t1) and they will do so more if the logic of polling average is included in the news story. (c1 through c4 vs c5 through c12b by party identification at t1) 3. The consonance between summative and singular polls will facilitate biased perceptions, the dissonance will mitigate it. (c1 through c10 by party identification at t2) 4. When summative is presented first, it will lead to less biased evaluation of secondarilypresented individual polls (at t2). (c1 through c4 vs c5 through c10 by party identification at t2) 5. When summative comes post hoc, it will facilitate bias (if both singular and summative poll is favorable), or it serves as a corrective attempt by mitigating bias (if singular poll is favorable and the summative poll is unfavorable). (c5 through c10 by party identification at t2) 6. The effect sizes of singular polls in which there is wider gap between candidates will be stronger than those that have smaller gaps (at t1). (c7 through 8 vs c5,6,9,10 by party identification at t1) 7. When two consecutively presented singular polls contradict each other, there will be less bias, than when they align with each other. (c11 vs c12a and b by party identification at t2) 8. The cumulative effects of singular+summative polls will be greater (both in facilitating and reducing bias) than singular+singular polls (at t2). (c1 through c10 vs c11 through c12b by party identification at t2) 9. All these relationships will be moderated by political knowledge, methodological knowledge/numeracy. According to motivated reasoning, those with greater ability should have greater bias (at t1 and t2). (c1 through c12b by party identification and ability measures at both t1 and t2) 3
Example manipulation story 1: A polling average result showing Clinton lead with the 3 rd explanatory (corrective) paragraph included. 4
Example manipulation story 2: A polling average result showing Trump lead with the 3 rd explanatory (corrective) paragraph excluded. 5
Example manipulation story 3: A poll result showing Clinton lead with a 6% gap Example manipulation story 4: A poll result showing Trump lead with a 2% gap 6