Hierarchical Item Response Models for Analyzing Public Opinion

Hierarchical Item Response Models for Analyzing Public Opinion Xiang Zhou Harvard University July 16, 2017 Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 1

Features of Public Opinion Data Responses in dierent formats Dichotomous: e.g., yes/no; favor/oppose; agree/disagree; Interval: e.g., feeling thermometer Ordinal (Likert scale): e.g., strongly agree /agree /disagree /strongly disagree; Multiple Items are often used to gauge attitudes in the same domain or toward the same issue. Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 2

For example, the ANES uses the following items to tap racial attitudes. 1. Generations of slavery and discrimination have created conditions that make it dicult for blacks to work their way out of the lower class 2. Irish, Italians, Jewish and many other minorities overcame prejudice and worked their way up. Blacks should to the same without any special favors 3. It's really a matter of some people not trying hard enough; if blacks would only try harder they could be just as well o as whites 4. Over the past few years blacks have gotten less than they deserve. For each item, respondents may (1) agree strongly, (2) agree somewhat, (3) neither agree nor disagree, (4) disagree somewhat, or (5) disagree strongly. Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 3

Analytical Strategies A major goal of public opinion research is to investigate how preferences dier among individuals, vary across space, or change over time. Two conventional approaches 1. Run generalized linear models (binary/ordinal logit), one for each item, and check if results for dierent items align with each other. 2. Combine multiple dichotomous/ordinal variables into a composite measure and then run conventional regressions. simple average (e.g., DiMaggio, Evans and Bryson 1996) PCA/factor analysis (e.g., Layman and Carsey 2002; Ansolabehere, Rodden and Snyder 2008) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 4

Limitations of Existing Approaches analytic strategy equidistant response categories equal item weights researcher discretion item by item no no large simple average PCA/factor analysis yes yes small yes no small Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 5

Item Response Theory (IRT) Models Modeling survey responses as parametric functions of (unknown) item and respondent characteristics. That is, for respondent i and item j, the probability of choosing response h is Pr(Y ij = h) = P jh (θ i ), h = 0, 1, 2,..., H j 1. Binary IRT widely adopted by political scientists to estimate the ideological positions of political elites. Seldom used to analyze mass opinion. less ideological constraint among the mass public (Converse 1964) in each dimension, number of items often too few to precisely estimate θ i tension between (a) dimension of the ideological space and (b) the precision with which "ideal points" are estimated Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 6

A Natural "Compromise": Hierarchical IRT Models Specify a model of latent preferences θ i at level II, e.g., θ i = γ T xi + ɛ i i = 1, 2,..., N, Compared with the item-by-item approach pooling information from multiple items, increasing power reducing the number of models from J to 1, leaving little room for researcher discretion Compared with the two-step approach (simple average, PCA, factor analysis, conventional IRT) integrating measurement and analysis more eciency estimating via MLE, producing correct asymptotic inference Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 7

Model Details: Level I Binary response items (binary logit) P jh (θ i ) = exp [ h(α j + β j θ i ) ], h = 0, 1, 1 + exp(α j + β j θ i ) Ordinal response items (proportional odds) P jh (θ i ) = Pr(y ij h) Pr(y ij h + 1) = exp(α jh + β j θ i ) 1 + exp(α jh + β j θ i ) exp(α j h+1 + β j θ i ) 1 + exp(α j h+1 + β j θ i ), where = α j 0 > α j 1... > α j Hj 1 > α j Hj =. Interval response items (normal linear model) y ij = α j + β j θ i + ɛ ij Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 8

Model Details: Level II Level II: A heteroscedastic regression model indep θ i N(µ i, σi 2 ), µ i = γ T xi log σ 2 i = λ T zi Relation with existing models in the literature A generalization of random-eects ideal point models (Mislevy1987; Londregan2000; Bailey2001; Lewis2001) A variant of the Multiple Indicators and Multiple Causes (MIMIC) model (Joreskog and Goldberger 1975; Jackson 1983; Muthen 1984) Level II is akin to a standard heteroscedastic regression (Cook and Weisberg 1983; Aitkin 1987; Verbyla 1993) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 9

Identication Constraints Consider the binary logit case. We can rewrite the model as logit Pr(Y ij = 1) = α j + β j {γ T x i + ɛ i exp(λ T z i /2)}. location constraint: set i γt xi = 0 so that the arithmetic mean of the prior means of the latent preferences equals zero. scale constraint: set i λt zi = 0 so that the geometric mean of the prior variances of the latent preferences equals one. direction constraint: restrict the sign of one discrimination parameter, say β 1, to be positive (or negative) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 10

Estimation and Inference Maximizing the marginal likelihood p(y α, β, γ, λ, x, z) = p(y θ, α, β) p(θ γ, λ, x, z)dθ Treating θ as missing data, apply the EM algorithm θ Expectation step: using quadrature methods to evaluate the Q function Maximization step: tting J separate GLMs, one for each item, to update α j and β j (to grouped data dened by quadrature points) conditional maximization to solve for γ and λ (usually 2-3 steps to converge) Using observed information Î (α, β, γ, λ) to estimate standard errors EAP estimates of θ i (as a byproduct) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 11

Illustration with ANES Data, 1972-2012 Four issue domains: economics, civil rights, morality, foreign policy (Baldassarri and Gelman 2008) Economics (15 items), e.g., Support for government or private health insurance (7 categories) Support for government guarantee of jobs and income(7 categories) Federal spending on Social Security(3 categories) Federal spending on public schools (3 categories) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 12

Civil rights (17 items), e.g., Should the government help blacks? (7 categories) We have gone too far in pushing equal rights in this country (5 categories) We would have fewer problems if people were treated more equally (5 categories) Opinion on armative action (4 categories) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 13

Morality (10 items), e.g., Should women have equal role in business, industry, and government? (7 categories) Fewer problems if there were more emphasis on traditional family ties (5 categories) Favor or oppose laws to protect homosexuals against job discrimination (4 categories) When should abortion be permitted? (4 categories) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 14

Foreign policy (4 items) Should we try hard to get along with Russia? (7 categories) Should we spend more or less on defense? (7 categories) Federal spending on foreign aid (3 categories) Federal spending on space/science/technology (3 categories) Note: Since many (if not most) questions were not asked consistently over the years, data are highly unbalanced for all of the four domains. Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 15

A Comparison with MCMCpack::MCMCirtHier1d MCMCpack oers a function MCMCirtHier1d() that can t hierarchical binary IRT model with prior means depending on covariates (Martin, Quinn, and Park 2011) Focus on ANES 2012, economic issues (N = 5820, J = 10) Covariates for the mean equation (x i ): Party ID (Democrat, independent [including leaners], Republican) Education (high school or less, some college or above) Full interactions of party ID and education Construct binary data by dichotomizing ordinal responses at mean scores of the integer scales Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 16

Table 1: Computation Time and Coecient Estimates under Dierent Models, ANES 2012, Economic Domain. MCMC Dichotomized MMLE-EM Dichotomized MMLE-EM Ordinal Computation Time 471 seconds 7 seconds 7 seconds Independent 0.535*** (0.059) Republican 1.542*** (0.072) College 0.060 (0.053) Independent*College 0.520*** (0.075) Republican*College 0.457*** (0.088) 0.516*** (0.061) 1.506*** (0.075) 0.067 (0.056) 0.508*** (0.076) 0.456*** (0.089) Note: p<.1, *p<.05, **p<.01, ***p<.001 (two-tailed tests). 0.541*** (0.056) 1.510*** (0.069) 0.118* (0.053) 0.469*** (0.071) 0.331*** (0.084) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 17

Table 1: Computation Time and Coecient Estimates under Dierent Models, ANES 2012, Economic Domain. MCMC Dichotomized MMLE-EM Dichotomized MMLE-EM Ordinal Computation Time 471 seconds 7 seconds 7 seconds Independent 0.535*** (0.059) 0.516*** (0.061) 0.541*** (0.056) Republican 1.542*** (0.072) 1.506*** (0.075) 1.510*** (0.069) College 0.060 (0.053) 0.067 (0.056) 0.118* (0.053) Independent*College 0.520*** (0.075) 0.508*** (0.076) 0.469*** (0.071) Republican*College 0.457*** (0.088) 0.456*** (0.089) 0.331*** (0.084) Note: p<.1, *p<.05, **p<.01, ***p<.001 (two-tailed tests). Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 18

Table 1: Computation Time and Coecient Estimates under Dierent Models, ANES 2012, Economic Domain. MCMC Dichotomized MMLE-EM Dichotomized MMLE-EM Ordinal Computation Time 471 seconds 7 seconds 7 seconds Independent 0.535*** (0.059) Republican 1.542*** (0.072) 0.516*** (0.061) 1.506*** (0.075) 0.541*** (0.056) 1.510*** (0.069) College 0.060 0.067 0.118* (0.053) (0.056) Independent*College 0.520*** (0.075) 0.508*** (0.076) Republican*College 0.457*** 0.456*** (0.088) (0.089) Note: p<.1, *p<.05, **p<.01, ***p<.001 (two-tailed tests). (0.053) 0.469*** (0.071) 0.331*** (0.084) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 21

Latent Preference Estimates Figure 1: Latent Preference Estimates under Dierent Models, ANES 2012, Economic Domain. Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 22

Application I: Party Polarization Research has shown increased party polarization in the US, not only among party elites but also in the electorate (Layman et al. 2006). Yet few studies have precisely tracked the patterns and timing of party polarization in the long run. For each issue domain, model the mean with following covariates Party ID (Democrat, independent [including leaners], Republican) Education (high school or less, some college or above) Year splines (quadratic, three degrees of freedom) Full interactions of Party ID, education, and year splines Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 23

Trends in Party Polarization Figure 2: Trends in Policy Conservatism in Dierent Domains, by Party ID and Education (with 90% CIs) Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 24

Application II: Mass Polarization Party polarization, even among the electorate, does not necessarily reect polarization in the broader society (Carsey and Layman 2006; Hill and Tausanovitch 2015). To assess trends in mass polarization, specify both the mean and the variance as functions of year splines (with no other predictors) Track trends in the variance component Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 25

Trends in Mass Polarization Figure 3: Trends in Means and Variances of Policy Conservatism by Issue Domain (with 90% CIs). Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 26

Application III: Ideological Constraint With escalating party polarization, especially at the top, have ordinary citizens become more ideologically coherent (across issue domains)? For each issue domain, model both the mean and the variance with Year splines (quadratic, three degrees of freedom) Extract EAP estimates of latent preferences and track their correlations between domains Note: EAP estimates of latent preferences are extremely close under dierent models Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 27

Trends in Ideological Constraint Figure 4: Trends in Ideological Constraints Between Issue Domains, by Education Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 28

Conclusion A parsimonious, principled, and ecient way to analyze public opinion data integrates measurement and analysis in a single step improves statistical power and produces correct inference reveals how preferences dier among individuals, vary across space, or change over time helps identify patterns and trends in opinion polarization and ideological constraint Software in development Xiang Zhou (Harvard University) Hierarchical IRT for Public Opinion July 16, 2017 Page 29