Weighting for Nonresponse on Round Two of the New Immigrant Survey. Douglas S. Massey Princeton University. Guillermina Jasso New York University

Transcription

1 Weighting for Nonresponse on Round Two of the New Immigrant Survey Douglas S. Massey Princeton University Guillermina Jasso New York University Monica Espinoza Princeton University January 17, 2017 Abstract The New Immigrant Survey, a longitudinal survey of persons who became legal permanent residents in 2003 and were re-interviewed approximately five years later, experienced a marked decline in response rates between the baseline (R1) and follow-up (R2) interviews. Using R1 data we develop a statistical model to identify the determinants of response on R2. We then use that model to derive weights that correct for nonresponse and evaluate their efficacy through a counterfactual analysis. We then examine the effect of weighting for nonresponse on estimated trends. Although few variables were associated with the probability of response, the likelihood of inclusion in the follow-up survey was by no means random. However, a counterfactual analysis we undertook to test the weighting scheme suggested that they were effective in correcting for nonresponse bias and important to use in assessing trends between the two survey rounds. We recommend the use of nonresponse weights that are now available to users on the NIS website. Although the precision of estimates based on R2 data is somewhat reduced because of the smaller sample size, point estimates computed using nonresponse weights should yield valid and unbiased inferences about the progress of new immigrants in the United States. 1

2 The New Immigrant Survey (NIS) is a longitudinal survey of adults who became legal permanent residents of the United States during May through November of Follow-up interviews with the same respondents were conducted from June 2007 through December The goal of the survey project was to generate a representative, reliable, and accurate public use data file on the characteristics of new legal immigrants and their progress in the United States. Data from the first round of the survey, officially labeled NIS but here referred to simply as Round 1 (R1), were released in Data from the follow-up survey, labeled NIS but here referred to as Round 2 (R2), were released in April Data from both rounds of the survey are available for download from the project website ( Whereas the R1 survey achieved a robust response rate of 68.6% the R2 response rate came in at a more anemic 46.1%. In this paper we undertake a methodological analysis to consider the implications of the one-third reduction in response rate between R1 and R2. Although the NIS included samples of children and spouses, in this paper we address nonresponse only among sampled adults. We begin with a description of the design and implementation of the survey s two rounds and discuss the likely reasons for the loss of respondents between R1 and R2. Drawing on R1 data we then specify and estimate an equation to predict the likelihood of a successful reinterview on R2. After interpreting the model s coefficients to identify the determinants of selection into R2, we use the estimated equation to generate predicted probabilities of response and use them to develop a set of case weights to correct for nonresponse bias. We evaluate the efficacy of the weighting scheme by undertaking a counterfactual analysis for Round 1. This exercise compares parameter estimates for R1 derived under three conditions: from the entire set of R1 respondents, from R1 respondents without R2 non-respondents, and from R1 respondents 2

3 without R2 non-respondents but weighted to correct for nonresponse. Finally, we examine changes over time for variables included on both survey rounds, comparing trends observed with and without using weights to correct for nonresponse. We conclude with an appraisal of the reliability and validity of the R2 data for making inferences about the progress of new legal immigrants to the United States. TWO ROUNDS OF THE NEW IMMIGRANT SURVEY As noted above, the baseline NIS survey is a representative sample of adults who became legal permanent residents (LPRs) in the United States during May through November of 2003, with a midpoint roughly in August of Respondents were randomly selected from a list of new permanent residents who received their Green Cards during this period. The list was obtained from the U.S. Citizenship and Immigration Services (USCIS), a successor agency to the earlier Immigration and Naturalization Service (INS). The average time between admission to permanent residence and interview was 17 weeks. A total of 12,488 immigrants were sampled and 8,573 completed the survey for a response rate of 68.6%. In addition to the main sampled immigrant, interviewers also surveyed spouses if they lived in the household (n=4,334) and interviewed up to two co-resident children aged 8-12 (n=1,072). The R1 survey employed a stratified sampling design that under-sampled immigrants with spouse-of-u.s.-citizen visas and over-sampled persons admitted as principals on employment and diversity visas. In order to derive representative, unbiased point estimates of population parameters design weights must be used, where the weights are the inverse of the probability of selection into the sample. These are applied to individual cases when estimating parameters such as means and proportions and their associated standard errors. Since the selection probabilities are fractions under 1.0, weighting by their inverse increases the 3

4 contribution of the case as the selection probability falls, thus giving more weight to respondents who were under-sampled and less weight to those who were over-sampled. Follow-up interviews with these same respondents were conducted from June 2007 through December 2009, yielding a rough midpoint in September 2008, just over five years from the midpoint of the baseline survey. Between the time of the R1 and R2 surveys, however, the context for interviewing immigrants had changed quite dramatically. Between August 2003 and September 2008, GDP growth dropped from 3.74% to 1.88% per year and from September 2008 to the end of the interview period in December 2009 GDP shrank by a remarkable 9%. Over the same period, consumer confidence fell by 52% and business bankruptcies rose by 24%, a wave of economic turmoil that coincided with an upsurge in anti-immigrant sentiment (Massey and Sanchez 2010). Indeed, between 2000 and 2006 the percentage of Americans who labeled immigrants as a burden to society rose from 38% to 52% and by the latter date 53% had come to believe that illegal immigrants should be required to go home (Pew Research Center 2006). Not coincidentally immigrant deportations rose by 50% from 2000 to 2006 and by the end of fieldwork in 2009 the annual total reached 392,000. Under these conditions it is hardly surprising that in 2007 some 72% of foreign born Hispanics said that the immigration policy debate had made life in the U.S. more difficult for Hispanics and 67% agreed that they worried some or a lot about deportation (Pew Research Center 2007). The context for interviewing immigrants thus became much more hostile between the baseline and follow-up survey, a shift that logically can be expected to have made locating R1 respondents and securing their cooperation more difficult. The tasks of finding and reinterviewing R1 respondents were further complicated by authorities at USCIS, who reneged on 4

5 an agreement signed in January 2007 to provide address updates from the Change-of-Address (AR-11) files for NIS investigators to use in locating respondents for the follow-up. LPRs are required to notify USCIS of changes of address within ten days of the change, thus yielding an up-to-date address file for all legal resident aliens, at least in theory. Unfortunately access to this resource was unexpectedly denied without explanation by authorities at USCIS and investigators were forced to rely on their own data and other methods to track down respondents from the baseline survey (which is ironic since USCIS was an original funder of the study). NIS interviewers naturally recorded the respondent s address at the time of the R1 survey, and in addition asked about future travel plans while also obtaining the name of a friend or relative who does not live with you at this address but who resides in the United States and who will know how to get in touch with your if you move. Immigrants are by definition a mobile population, and with an average gap of around five years between R1 and R2, many of the original respondents and their relatives had indeed moved. Moreover, in the hostile anti-immigrant context that had emerged by the time of the follow-up interviews, friends and relatives were often reluctant to provide contact information to outsiders they did not know. As a result, apart from the difficulty of securing respondents cooperation it proved challenging to locate respondents in the first place and roughly half of the R1 respondents were never found. In the end, the downturn in the economy, the rise of anti-immigrant sentiment, and the sharp increase in deportations between the R1 and R2 surveys, when combined with the lack of cooperation from the USCIS, augured for a lower response rate than achieved on the baseline survey; and as already noted this outcome did, in fact, come to pass. The number of completed interviews with adult immigrants was 3,902, for a response rate of 46.1%. In addition, follow-up 5

6 interviews were completed with just 1,771 spouses (40.9% of the R1 cohort) and 392 children (36.6% of the R1 cohort). Although we cannot truly know the reason for the marked decline in response rates, it is clear that something happened to trigger the drop. The more important question, however, is what effect the decline in response rates has on the reliability and validity of the NIS as a longitudinal survey. NONRESPONSE, RELIABILITY, AND VALIDITY At a minimum, the drop in sample size from 8,573 to 3,902 will reduce the precision of parameter estimates, increasing standard errors and thus decreasing the reliability with which R2 variables and R1-R2 trends can be measured. Little can be done to offset this decline in precision. In practical terms, the reduction in sample size makes it more difficult to find significant effects and thus increases the likelihood of Type II errors. In making inferences from R2 data, therefore, researchers are statistically more likely to reject a true hypothesis than to confirm a false one. A reduced sample size by itself does not necessarily introduce bias, however. Whether estimates are biased because of a loss to follow-up depends on the degree to which R2 respondents differ from those in R1 and the extent to which these differences are associated with variables under study. Mathematically there is no fixed relationship between response rate and bias (Bethlehem 2002) and a meta-analysis by Peytchev (2013) indeed found no empirical correlation between response rates and degree of bias across a broad sample of surveys. According to an equation derived by Bethlehem (2002), the degree of bias introduced by nonresponse is inversely related to the response rate but directly related to the correlation between the probability of response and survey variables of interest (Olson 2013). If variables of interest are uncorrelated with the factors that produced the nonresponse, no bias will be 6

7 introduced into estimates no matter how low the response rate is (Lessler and Kalsbeek 1992; Massey and Tourangeau 2013). In simple cross-sectional surveys it is difficult to know the degree to which factors that produced a high degree of nonresponse are correlated with survey variables and thus likely to produce biased results. At best, paradata can be used to estimate a model predicting whether or not an interview was completed. Paradata are auxiliary data outside the survey that are available for both respondents and nonrespondents (Olson 2013). Examples include information such as the time of attempted contacts, the number of call-backs, interviewer observations about recruitment encounters, and any administrative or census data that can be linked to sampling elements (Smith and Kim 2013). If paradata are available, models to predict the likelihood of response can be estimated and used to generate predicted probabilities of response, the inverse of which yield weights that in theory correct for nonresponse bias (Czajka 2013). As with design weights, cases with a response probability near 1.0 will yield weights that confer less influence in computing the parameter estimate whereas those with low response probabilities carry more influence (Czajka 2013; Massey and Tourangeau 2013). In general, the greater the amount of reliable paradata available to the researcher, the more accurate the predicted probability of response and the more effective the correction for nonresponse bias (Smith and Kim 2013). In the case of longitudinal surveys, of course, considerable data are available from the baseline sample to predict the likelihood of response on the follow-up (Schoeni et al. 2013). To the extent that variables measured on R1 of the NIS are related to the likelihood of response on R2, therefore, we are in a position to generate weights that correct for nonresponse bias. To the extent that unobserved variables are correlated with those included on the R1 survey, weighted 7

8 parameter estimates will also correct for bias introduced by unobserved heterogeneity, though how well the correction works to eliminate bias from unobserved factors cannot be known in practice. MODELING RESPONSE PROBABILITIES Our goal in selecting independent variables for this analysis was to be as encompassing as possible and to use all available R1 data to maximize the fit of the model and to identify which of the many possible predictors determined selection into the R2 sample. The set of predictors we compiled includes demographic characteristics such as age, gender, number of children in the household, number of children living outside the United States, marital status, and race/ethnicity. Indicators of human capital used in the model include years of education, English ability, foreign language skills, and ratings of current and prior health. Geographic effects were assessed using dummy variables to indicate country or region of birth and place of interview. The model also included a battery of labor market indicators, such as current employment, hours worked, hourly wage, union membership, occupation, whether the job was obtained before receiving permanent residence, total household income; and as a control for potential racial discrimination we added an interviewer-assessed skin color rating. Wealth was measured using dummy variables to indicate categories of net worth and home ownership. Immigration-related variables include immigrant visa, months of U.S. experience prior to permanent residence, whether the respondent reported prior undocumented experience, and stated intention to live permanently in the United States. Finally, the model included religious affiliation and frequency of religious service attendance. We began by estimating a logistic regression model that used all available R1 data to predict whether respondents successfully completed an R2 interview. Using the full set of 8

9 independent variables, however, we discovered that the sample size dropped from 8,573 to 6,435 owing to the list-wise deletion of cases with missing values. The drop in sample size obviously creates a problem if our goal is to generate nonresponse weights for all cases, so we inspected the distribution of missing values across variables and found unusually high frequencies for seven variables: whether the interview was in English, whether the respondent had ever spoken another language, whether another language was spoken at home, whether the respondent belonged to a labor union, hours worked per week on the current job, hourly wage, and months of prior U.S. experience. We then examined inspected the logistic regression coefficients for these variables (the full model is presented in Appendix A) and determined than none was significant in predicting the likelihood of response. We then eliminated these variables from the model and reestimated the logistic regression model to generate the equation estimates for the complete R1 sample, shown in Figure 1. Much of the missing was not due to item non response but rather aspects of the study design, as when certain questions were only asked of a randomly selected portion of the respondents or when the interview was done by phone, thus precluding the interviewer assessment of skin tone. TABLE 1 ABOUT HERE Rarely do researchers have such a wealth of data to estimate a model predicting the likelihood of survey response. Even after we eliminated variables characterized by high rates of missing data on R1 the breadth of information is impressive. Given the large number and diversity of predictors in the model what is perhaps most surprising is how few were actually significant in determining the likelihood of response on R2. In addition to the variables already eliminated, the estimates in Table 1 indicate that inclusion in R2 was not significantly related to marital status, current health status, occupation, whether the current job was obtained before or 9

10 after achieving permanent residence, skin color, net worth, prior undocumented experience, or frequency of religious attendance. The strongest predictors pertained to demographic background, years of education, and intentions for future U.S. residence. Among demographic characteristics, females were significantly more likely than males to respond to the R2 survey, with the odds being 22% greater for women [determined by taking the exponent of the logistic regression coefficient to derive the associated odds ratio: exp(0.201)=1.22]. The likelihood of response varied in curvilinear fashion with respect to age. With each additional year the odds of response rose by 3% but declined by 0.03% with respect to age squared, yielding a curve that rises from age 18 to 50 and then declines into older age. The odds of response also increase by 4.8% for each additional child present in the household, and by 10% for each child living abroad. With respect to education, the odds of response rise steadily as years of schooling go beyond six. Compared to those with a primary education of less, the odds of inclusion are 23% greater for those with 6-11 years of education, 33% greater for high school graduates, 42% greater for those with some college, and 53% greater for college graduates. Finally with respect to settlement intentions, the odds of response were paradoxically 23% lower for those intending to remain in the U.S. for the rest of their lives (and 40% lower for those who didn t answer this question). Although not as strongly or systematically related to the likelihood of response as the foregoing factors, race/ethnicity, English ability, health compared to a year ago, and health before coming to the United States were also significant in predicting the likelihood of R2 response. Only one racial/ethnic category was associated with the likelihood of response. Hispanics were 32% more likely than all other groups to complete the R2 survey. This result, 10

11 combined with the fact that prior undocumented status and months of prior U.S. experience (see Appendix A) had no significant effect on response probabilities, suggests that people who reported prior undocumented experience did not self-select out of the R2 sample, despite the rise in anti-immigrant sentiment from 2003 to Perhaps surprisingly, the likelihood of an R2 response decreased as English ability increased, culminating in a significant coefficient for those who understood English very well, who displayed 25% lower odds of response than those reporting lower levels of English comprehension. Although current health had no effect on the likelihood of an R2 response, the odds of inclusion were 19% lower for those who reported worse health compared to a year ago but 13% higher for those who reported better health than they experienced before coming to the United States. The two benchmarks are not necessarily the same because many new permanent residents were already living in the United States and were simply adjusting status to become legal permanent residents. In any event, the results are consistent in suggesting that better prior health yields a higher likelihood of response. Of the 10 categories for country or region of birth, only three proved to be statistically significant in predicting the likelihood of response, all negative in their effect. Persons from the Middle East and North Africa were 37% less likely to respond to the R2 survey, which is perhaps not surprising given the tenor of the social climate in the wake of 9/11 and the rise of anti- Muslim sentiment associated with the War on Terror. The odds of response were likewise 31% lower among immigrants from South Asia and the Pacific, a region that also includes many Muslims. Although East Asia contains very few Muslims, immigrants from that region nonetheless displayed 60% lower odds of responding to the R2 survey. The fact that immigrants from Mexico were neither more nor less likely to be included in the follow-up than those in 11

12 English speaking nations again suggests that persons with prior undocumented experienced were not systematically selected out of the R2 sample. With respect to place of interview only four of the 15 geographic categories were significant in predicting the likelihood of response. Whereas the odds of inclusion were 21% lower for respondents interviewed in New York, they were 29% higher among those from New England (i.e. Connecticut, Massachusetts, Maine, New Hampshire, Rhode Island, or Vermont), 28% higher among those from the South Atlantic (i.e. Georgia, North Carolina, South Carolina, Virginia, or West Virginia), and 34% higher among those from the West South Central region (i.e. Iowa, Minnesota, Missouri, North Dakota, South Dakota, Nebraska, or Kansas). Again, the fact that California, Florida, Illinois, New Jersey, and Texas did not display significantly lower probabilities response suggests that immigrants with prior undocumented experience did not selfselect out of R2, as these states house a disproportionate share of America s undocumented population (see Warren and Warren 2013). This proposition is buttressed by the fact that the South Atlantic and West North Central themselves displayed higher likelihoods of response, and these are regions containing a large share of new immigrant destinations (Massey and Capoferro 2008). Completion of the R2 survey was also not very selective with respect to labor force indicators. As already noted the likelihood of response was not affected by occupation, wages, the timing of job acquisition (before or after permanent residence), wages, or income; and with respect to current employment out of six categories only those temporarily laid off displayed a statistically significant departure from the other groups, with 52% lower odds of being included in the follow-up. 12

13 Turning to immigrant class of admission, we see that immigrants entering with numerically-limited relative-of-u.s. citizen visas, diversity visas, and other visas were more likely to be in R2 than those holding other kinds of visas. Thus the odds of inclusion were 24% greater for those holding a numerically-limited citizen-family visa, 18% greater for those on a diversity visa, and 28% greater for those in the residual other visa category. It is not immediately clear why those on numerically-limited citizen-family visas, diversity visas, or other visas were more likely to respond to R2 sample. These effects certainly cannot be attributed to differences in the intent to stay in the United States, since that variable was separately controlled in the equation. The fact that those admitted with a legalization visa were neither more or less likely than others to complete the R2 survey once again suggests that there was little selfselection out of the panel by persons with prior undocumented experience. Finally religious affiliation does not systematically predict inclusion in R2, except that the odds of response were 25% greater for those who professed no religion at all. Likewise, the categories for frequency of religious service attendance displayed no significant effects. Thus neither religion no religiosity seems to have affected response probabilities on Round 2. The fact the coefficient for Muslims here is not significant suggests that if anti-muslim sentiment had an effect on response rates, it was expressed regionally rather on the basis of religion belief per se, with Muslim immigrants from the Middle East, North Africa, and South Asia bearing more visible and bearing the brunt of the effect and those from the Balkans or Caucuses largely escaping the effect. 13

14 THE EFFICACY OF NONRESPONSE WEIGHTING Using the results shown in Table 1 we inserted the R1 characteristics of each respondent observed into the estimated equation to generate a predicted probability of inclusion in R2. We then took the inverse of this probability to derive weights to correct for nonresponse bias. In order to assess the efficacy of the correction we undertook a counterfactual analysis using the data from R1. Fist we applied design weights to the full R1 sample to derive unbiased estimates of the population parameters. Then we eliminated R2 non-respondents from the R1 data and derived parameter estimates from this reduced sample using design weights alone, thereby creating point estimates uncorrected for the process of non-response observed on R2. We then re-estimated the parameters after applying the nonresponse weights in addition to the design weights to derive corrected estimates. Finally we subtracted the uncorrected and corrected estimates from the unbiased estimates and compared them to the original unbiased estimates to assess the efficacy of our correction procedure. Table 2 presents the results of this exercise. The first column shows variable values estimated using the full R1 sample with design weights (the true values). The second column shows values estimated using design weights for the R1 sample after R2 non-respondents were removed (i.e. the biased values: those that would be obtained if the nonresponse process observed on R2 had occurred on R1). The third column takes the estimates of column two and applies the nonresponse weights (yielding corrected values: those that would be achieved by applying the proposed weighting scheme). The final columns show the error values for computations based on the biased and corrected values, in column (4) subtracting column (2) from column (1) (i.e. biased minus true) and in column (5) subtracting column (3) from column (1) (i.e. corrected minus true). 14

15 TABLE 2 ABOUT HERE Consider the first panel, which displays the gender distribution achieved when using the true, biased, and corrected values. From the model in Table 1 we know that females were more likely to respond than males, with the odds of response being 23% greater than those of men. Hence, when the design weights are applied to the R1 sample reduced by the R2 nonresponse process we observe an over-representation of women and an under-representation of men. Whereas the true distribution derived when design weights are applied to the full sample consists of 56.4% women and 43.6% men, the selection-reduced sample yields estimates of 58.8% women and 41.2% men, clearly overstating the presence of women. When the nonresponse weights are applied, however, the distribution moves much closer to the true distribution, yielding an estimate of 56.9% women and 43.1% men. Although still not equal to the true value, the overestimate of women has been reduced from 2.4 to 0.5 points and the underestimate of men of necessity simultaneously shrank from -2.4 to -0.5 points (see columns (4) and (5), a clear improvement in accuracy. Whereas before the correction for nonresponse the estimated percentage of women in the reduced R1 sample was significantly different from that computed from the full R1 sample (p<0.001) once the weights were applied the difference was no longer close to statistical significance. In Columns (4) and (5) those errors that simple t-tests reveal to be significant departures from true values (p<0.05) are marked with asterisks. For any variable (e.g. education), the t-tests are not independent of one another, since errors in one category will affect values in the other categories. In the prior example, for example, a 2.4 overestimate of women necessarily implies a 2.4 percent under estimate of men. This effect becomes less obvious as the number categories 15

16 increases, but the principle is the same. Nonetheless each asterisk indicates a significant error in the estimate of that one single parameter. As can be seen, Column (4) is riddled with asterisks, 75 to be exact, indicating numerous significant differences from true values in point estimates based on the selection-reduced sample uncorrected for nonresponse. In column (5), however, we see that the number of asterisks has been dramatically reduced by applying the nonresponse weights, falling to just ten; and four of these are for point estimates of the interviewer-assigned skin color rating, an error-prone subjective judgment to begin with. Even here, however, the mean skin color rating is identical across all estimates. At the bottom of Columns (4) and (5) we compute total error by summing the absolute values of all departures from the true value across all variables, yielding figures of and 49.6 for the biased and corrected estimates, respectively. In other words, application of the nonresponse weights has reduced the total error by 64% and left very few significant departures from true parameters, suggesting the weights are indeed effective in countering biases in the data introduced by nonresponse. The foregoing constituted a counterfactual analysis that examined the effect that nonresponse would have on R1 estimates if the baseline survey had been subject to the same process of nonresponse as observed in the follow-up. We cannot perform a comparable analysis on R2 data since we cannot derive a benchmark of true R2 values. We can, however, assess what effect weighting or not weighting for nonresponse might have on the measurement of trends between R1 and R2. Table 3 thus shows values of variables included on both survey rounds estimated with and without nonresponse weights to discern how different trends would be in the absence of correcting for nonresponse. Column (1) presents values estimated circa 2003 using the full R1 survey; Column (2) presents values of the same variables estimated circa

17 without using nonresponse weights; Column (3) repeats the foregoing estimation with nonresponse weights; and columns (4) and (5) show the trends that result from using and not using nonresponse weights for the R2 estimates. Statistically significant differences (p<0.05) between the latter two columns are indicated with an asterisk. TABLE 3 ABOUT HERE A number of trends in variable values do not seem to be significantly affected by application or non-application of nonresponse weights. Trends over time are not statistically different, for example, when using weighted or unweighted R2 estimates for occupational status, overall health insurance coverage, private health insurance coverage, source of private health insurance coverage, coverage by non-u.s. health insurance, coverage by Medicaid, or total household income. Although the differences are generally small, we nonetheless observe significant differences across many other variables. Thus in the absence of nonresponse weighting we would underestimate the increase in the percentage of respondents reporting poor health, as well as the increase the percentage registered for Medicare. In contrast, we would overestimate the increase in the percentage married as well as the percentage aged 35-44, 45-54, and 65+, the percentage Hispanic, and the percentage of homeowners. Likewise we would overestimate the decrease in the percentage Buddhist, the percentage aged <25 and as well as the percentage Asian. More seriously, in the absence of weighting for nonresponse we would mistakenly report a decrease in the percentage of respondents living in New York, an increase in percentage of Catholics, a decrease in the percentage of Muslims, an increase the percentage with no religious affiliation, and a decrease in the percentage retired. It is thus clear that reliance on unweighted estimates would in many cases lead to incorrect conclusions. We have therefore made 17

18 nonresponse weights available on the NIS website and recommend their use in deriving parameter estimates using R2 data. The use of weights necessarily decreases the efficiency of estimation by introducing an additional source of variation beyond sampling error. The loss of effectiveness associated with the use of weights is indicated by computing the design effect, which is the ratio of the variance of a weighted estimate to that which would have been achieved using simple random sampling (and hence no weights). Table 4 presents design effects associated with the three weighting schemes employed in Table 3: the true R1 parameter estimates achieved using design weights alone; the biased R2 estimates achieved by using design weights but no correction for nonresponse; and the corrected R2 estimates achieved by applying weights for both design and nonresponse. TABLE 4 ABOUT HERE Comparing the design effects in the third column with those I the first and second columns we see that applying nonresponse weights has a very modest effect. Well-designed surveys generally have design effects in the range of 1.0 to 3.0. When weights to correct for the stratified sampling design of the NIS are applied, the average design effect across all variables in the table is 1.33, and when weights for nonresponse are added to the weighting scheme the design effect rises to just 1.43, a relatively small effect. In other words, weighting the data to correct for nonresponse entails little loss of efficiency in estimation, again underscoring the efficacy of the proposed correction. 18

19 CONCLUSION Between the baseline (R1) and follow-up (R2) samples of the New Immigrant Survey the response rate dropped from 69% to 46%, resulting in the loss of 54% of respondents from the longitudinal database. Here we conducted a detailed analysis to assess the implications of this loss to follow-up for the reliability and validity of estimates derived from the R2 data. Very clearly the reduction of sample size means that parameters estimated using R2 data will be less precise and reliable, thus increasing the likelihood of Type II errors---failing to confirm hypotheses that are, in fact, true but undetectable because of a lack of statistical power. The decline in sample size normally will not increase the likelihood of making a Type I error, however mistakenly concluding that a hypothesis is true when it is not and is in this sense the effect of nonresponse is conservative. An elevated rate of nonresponse, however, also increases the potential for bias in estimates based on R2 data. Prior work has shown that the degree of bias is not a given, however. The size and direction of the bias introduced by nonresponse inversely related to the response rate and directly related to the size of the correlation between the response probability and variables of interest. As a consequence, there is no universal level of bias that can be assigned to a dataset owing to nonresponse. In practice, the degree of bias will vary from topic to topic depending on the variables under analysis and the degree of their correlation with the likelihood of response. In our analysis, we took advantage of the wealth of data available from the R1 survey to identify which variables observed in the baseline sample were, indeed, associated with the probability of successfully completing an R2 interview. Our estimated model predicting response is reassuring in that many variables likely to be of interest to immigration researchers were 19

20 unrelated to the likelihood of response, including age, marital status, race/ethnicity, English ability, foreign language skills, current health status, health status before coming to the United States, current employment status, hours worked, hourly wages, union membership, whether a job was obtained before achieving permanent residence, skin color, months of prior U.S. experience, prior undocumented experience, or frequency of religious attendance. The likelihood of completing an R2 interview was not random, however. According to our logistic regression estimates, the odds of inclusion proved to be greater for women, professionals, homeowners, Catholics, persons holding numerically-limited relative-of-u.s.- citizen visas, diversity visas, legalization, and other visas, those professing no religious affiliation, persons interviewed in the South Atlantic region, and those from households reporting a negative net worth and incomes between $53,000 to $95,000. The odds were lower for persons reporting their health to be worse than a year ago, born in the Middle East and North Africa or Southeast Asia and the Pacific, those interviewed in New York state, and respondents intending to live in the U.S. for the rest of their lives. Using the logistic regression model, we inserted variable values observed for each R1 respondent to generate predicted probabilities of inclusion in the R2 survey and then computed nonresponse weights by taking the inverse of the estimated response probability. We then undertook a counterfactual analysis to test the efficacy of our weighting scheme by removing R2 non-respondents from the R1 data and applying weights to the remaining R1 data to observe how close weighted estimates of variable values came to the actual values computed from the entire R1 sample. We found that unweighted parameter estimates based on the reduced R1 sample displayed numerous statistically significant discrepancies from the true values computed from 20

21 the full R1 sample. In other words, if the same selective pattern of nonresponse observed on R2 were to have affected the R1 data, many biased estimates would result. We also found, however, that when nonresponse weights were applied, total error was reduced by 64% and that statistically significant bias was eliminated from the vast majority of point estimates. Moreover, in the few cases where significant differences persisted the absolute value of the discrepancy was generally small and unlikely to affect overall conclusions. Finally when we turned to the R2 data and inspected trends in variables measured on both rounds of the survey using weighted and unweighted estimates we found that the estimated trends were often statistically different from one another, underestimating increases in variable values in two cases, overestimating increases in six cases, overestimating decreases in four cases, and mistakenly detecting nonexistent increases or decreases in five cases. Although the size of the bias in measuring trends was in most instances small, we nonetheless recommend using nonresponse weights for computing point estimates from R2 data and to this end have made the weights available on the NIS website. Although the precision of estimates based on R2 data may be lower because of the smaller sample size, the increase in the design effect attributable to the application of nonresponse weights is quite small. In the end, we conclude that parameter estimates computed using both design and nonresponse weights should produce valid and unbiased inferences about the progress of new immigrants in the United States. 21

22 REFERENCES Bethlehem, Jelke Weighting Nonresponse Adjustments Based on Auxiliary Information. Pp in Survey Nonresponse, eds. R. M. Groves, D. A. Dillman, J. L. Eltinge and R. J. A. Little. New York: John Wiley & Sons. Czajka, John L Can Administrative Records Be Used to Reduce Nonresponse Bias? Annals of the American Academy of Political and Social Science 645: Lessler, Judith T., and William D. Kalsbeek Nonsampling Error in Surveys. New York: John Wiley & Sons. Massey, Douglas S., and Chiara Capoferro The Geographic Diversification of U.S. Immigration. Pp in Douglas S. Massey, ed., New Faces in New Places: The Changing Geography of American Immigration. New York: Russell Sage. Massey, Douglas S., and Magaly Sánchez Brokered Boundaries: Creating Immigrant Identity in Anti-Immigrant Times. New York: Russell Sage Foundation. Massey, Douglas S., and Roger Tourangeau Where Do We Go from Here? Nonresponse and Social Measurement. Annals of the American Academy of Political and Social Science 645: Olson, Kristen Paradata for Nonresponse Adjustment. Annals of the American Academy of Political and Social Science 645: , Pew Research Center America's Immigration Quandary: No Consensus on Immigration Problem or Proposed Fixes. Washington, DC: Pew Research Center. Pew Hispanic Center The 2007 National Survey of Latinos: As Illegal Immigration Issue Heats Up, Hispanics Feel a Chill. Washington, DC: Pew Research Center. 22

23 Peytchev, Andy Consequences of Survey Nonresponse. Annals of the American Academy of Political and Social Science 645: Schoeni, Robert F., Frank Stafford, Katherine A. Mcgonagle, and Patricia Andreski Response Rates in National Panel Surveys. The Annals of the American Academy of Political and Social Science 645: Smith, Tom W., and Jibum Kim An Assessment of the Multi-level Integrated Database Approach. Annals of the American Academy of Political and Social Science 645: Warren, Robert, and John Robert Warren Unauthorized Immigration to the United States: Annual Estimates and Components of Change, International Migration Review 47(3):

24 Table 1. Logistic regression model used to generate nonresponse weights for full sample of 8,573 respondents. Standard Independent Variables Coefficient Error. Demographic Background Age 0.030*** Age squared ** Female 0.201*** No. Children in Household 0.047** No. Children living outside of US 0.095* Marital Status Never Married-Not in Union Separated-Divorced-Widowed Married or in Union Race/Ethnicity Non-Hispanic White Non-Hispanic Asian Non-Hispanic Black Non-Hispanic Other Hispanic 0.276** Years of Education <6 years years 0.214** Years 0.283*** years 0.353*** Year 0.426*** English Ability Understand Not at All Understand Not Well Understand Well Understand Very Well *** Current Health Status Poor Fair Good Very good Excellent Continued 24

25 Table 1. Continued. Standard Independent Variables Coefficient Error. Health Compared to Year Ago About the Same Better Worse * Health Before Coming to U.S. About the Same Better 0.123* Worse Country/Region of Birth English Speaking Nations Western Europe Eastern Europe Central Asia Middle East and North Africa ** Sub-Saharan Africa South Asia Southeast Asia and Pacific * East Asia ** Mexico Other Latin America/Caribbean Place of Interview California Florida Illinois New Jersey New York *** Texas New England 0.256** Middle Atlantic South Atlantic 0.250** East South Central East North Central West North Central 0.292* West South Central Mountain Pacific Non-Continental US territories Continued 25

26 Table 1. Continued Standard Independent Variables Coefficient Error. Current Employment Working Unemployed and looking Temporarily laid off * Disabled Retired Homemaker Other Occupation Laborers and Helpers Not Working Service Workers Operatives Craft Workers Administrative Support Workers Sales Workers Technicians Managerial Professionals Other When Job Obtained Not Working Job before LPR Job after LPR Total Household Income Zero to < to < to <23784) to < to < to < >= Missing cases *** Darkness of Skin Color Skin Color Rating Skin Color Missing Continued 26

27 Table 1. Continued Standard Independent Variables Coefficient Error. Net Worth Negative Zero to <10, ,000 to <50, ,000 to 200, >=200, Missing Property Home Owner 0.143** Immigrant Class of Admission Rel. of Citizen-Unlimited Rel. of Citizen-Limited 0.215** Relative of LPR Employment Diversity 0.165* Refugee/Asylee/Parolee Legalization Other 0.250*** Prior Immigrant Experience Formerly Undocumented Future Intentions Intends to Live in US Rest of Life 0.267** Intends Missing 0.200* Religious Affiliation Protestant Catholic Orthodox Muslim Jewish Buddhist Hindu No Religion 0.223** Other Religion Continued 27

28 Table 1. Continued Standard Independent Variables Coefficient Error. Frequency of Religious Attendance Never Sporadically Regularly Frequently Very Frequently Constant *** LR chi2(115) *** Log likelihood Pseudo R Observations 8,573 28

29 Table 2. Estimated values of selected variables from Round 1 of the New Immigrant Survey under three conditions: Full R1 sample with design weights, R1 sample without missing R2 cases and design weights, and R1 sample without R2 missing cases and weights for design and nonresponse. (1) (2) (3) (4) (5) True Biased Corrected R1 Sample R1 Sample Full R1 without without Missing Error with Sample Missing R2 R2 Cases and and without with Cases and Weights for Weighting for Design Design for Design & Nonresponse Variable Weights Weights Nonresponse Before After Gender Female * 0.5 Male * -0.5 Age at Interview < * to to * to * to * -0.5 >= * -0.1 Education < 6 years * years years years years * 0.4 Children in Household No Children * Child Children * Children * -0.6 Children Outside US No Children * Child Children * Children Current Health Excellent Very good Good Fair Poor Continued 29

30 Table 2. Continued. (1) (2) (3) (4) (5) R1 Sample R1 Sample Full R1 without without Missing Error with Sample Missing R2 R2 Cases and and without with Cases and Weights for Weighting for Design Design for Design & Nonresponse Variable Weights Weights Nonresponse Before After Health Compared to Last U.S. Trip Better About the Same Worse Health Compared to Year Ago Better About the Same * 0.0 Worse Religion at Interview Catholic * 0.0 Orthodox Protestant Muslim * 0.3 Jewish Buddhist * -0.4 Hindu No Religion Other * -0.1 Frequency of Service Attendance Never Sporadically Regularly Frequency * 0.0 Very Frequently * 0.1 Immigrant Class of Admission Relative of Citizen-Unlimited Relative of Citizen-Limited Relative of LPR Employment Diversity Refugee/Asylee/Parolee Legalization * 0.1 Other Continued 30

31 Table 2. Continued. (1) (2) (3) (4) (5) True Biased Corrected R1 Sample R1 Sample Full R1 without without Missing Error with Sample Missing R2 R2 Cases and and without with Cases and Weights for Weighting for Design Design for Design & Nonresponse Variable Weights Weights Nonresponse Before After Marital Status Never Married-Not in Union Separated-Divorced-Widowed * 0.0 Married or In Union * -0.3 Prior US Experience Mean Months Zero to 3 Months Months to 1 Year * 1 to 1.5 Years to 2 Years to 3 Years to 4 Years to 5 Years to 10 Years to 15 Years * 0.2 >15 Years Undocumented Experience Formerly Undocumented * 0.3 Documented-No Prior Experience * -0.3 Intends to Live in U.S. Rest of Life Yes * -0.4 No * 0.4 Current Employment Working Now Unemployed and Looking * Temporarily Laid Off Disabled Retired * -0.2 Homemaker Other Continued 31