Response to the Evaluation Panel s Critique of Poverty Mapping Peter Lanjouw and Martin Ravallion 1 World Bank, October 2006 The Evaluation of World Bank Research (hereafter the Report) focuses some of its more critical comments on the Poverty Mapping (PM) methods devised by Bank researchers. 2 The basic idea of PM is summarized well by the Report: Most countries have a recent census which contains data on variables correlated with poverty, such as education, landholdings, occupation, and demographic structure. Household income or expenditure data from surveys can be linked to the census poverty correlates to produce a set of numbers that reflect the relationship between those variables (education, land ) and income. Those numbers can be used to impute poverty estimates from census variables.. (Report, pp.25-26) While the Report finds this research to be innovative and to be addressing an important need in our client countries, it claims that the quality of the statistical work is weak and unconvincing. The Report gives the impression that World Bank researchers have been more interested in producing attractive maps of poverty than tools that can provide reliable information to policy makers in developing countries. We welcome constructive critical comment, but this Report seriously misrepresents the Bank s methodological work on PM, the way that outside researchers and statisticians view this work, and the manner in which the PM team has worked with partner countries. This note takes issue with the three main claims made by the Report in its critique of the Bank s poverty mapping work. Claim 1: The Bank overstates the statistical precision of its poverty maps At the heart of the Panel s doubts about the Bank s poverty mapping work is a belief that the statements made about the precision of local level poverty estimates are seriously misleading. To quote them: the panel doubts that the methods used by the Bank to assess margins of error generally are adequate, nor can they rule out the possibility that the true margins of error are many multiples of those that are presented. (Report, p.26). The reason given for this negative assessment is that the Panel believes that the Bank s poverty mapping methodology assumes that 1 Both authors are staff of the World Bank s Development Research Group (DECRG). Peter Lanjouw leads the team of researchers in DECRG that developed the methodology used by the Bank for producing poverty maps. Martin Ravallion is the manager in DECRG with overall responsibility for research on poverty and inequality. 2 The Report we refer to is An Evaluation of World Bank Research, 1998-2005, September 2006. The Report was written by a panel comprising Abhijit Banerjee, Angus Deaton (Chair), Nora Lustig and Ken Rogoff. On the evaluation of Bank research on poverty, the Panel was assisted by Esther Duflo, Murray Leibbrandt and Martin Wittenberg. The report can be found at: http://econ.worldbank.org/wbsite/external/extdec/extresearch/0,,contentmdk:21165468~ menupk:598503~pagepk:64165401~pipk:64165026~thesitepk:469382,00.html.
prediction errors in each enumeration area (EA) are independent of each other. The Report argues that this assumption need not hold, in which case the Bank s methodology will underestimate the standard errors of the local-level estimates of poverty measures. It should first be noted that the methods developed by the Bank s PM team have been subjected to a great deal of professional (internal and external) scrutiny, including by reviewers for the leading academic journal in econometrics, which published the key paper outlining the methods. 3 The Panel fails to note that the Bank s researchers explicitly addressed the issues they raise four years ago in a key reference document that was supplied for the evaluation. 4 This paper makes clear that, contrary to what was supposed by the Panel, the Bank s methodology does not require that the prediction errors in each EA are independent of one another. It is straightforward to allow for the correlation of the deviation from predictions observed to occur within an EA (in the survey data) to actually apply at, say, a city-wide level, or some other level well above the EA, when carrying out the simulations with the census data. Thus the entire observed correlation of the deviation in predictions occurring between households in a survey EA can be assumed to apply across all households in the city. This would be quite a conservative assumption as in all likelihood only a fraction of the EA-level inter-household correlation would apply at this higher level, if at all. The 2002 paper produced by the PM team, shows that even with such a conservative assumption standard errors are not inevitably higher. The impact on standard errors depends crucially on the basic model of consumption or poverty estimated with the household survey data. The PM team has shown that it is possible to include amongst the correlates of poverty in the survey-based model a wide variety of EA-level variables that have been calculated using census data (or obtained from some other tertiary data set, such as geo-coded rainfall or infrastructure data). Higher level aggregates can also be merged into the household survey. Including such variables in the model offers the analyst the prospect of largely removing the EA-level correlation in the deviation of local poverty from its prediction. If the remaining location effect is sufficiently small, the degree to which standard errors are understated when EA s are assumed to be independent becomes negligible. But conversely, standard errors can be more significantly understated if location effects have not been captured. The issue is ultimately an empirical one, requiring careful attention during implementation of the approach. The PM team has devoted a great deal of attention to the question of how this strategy for minimizing the EA-level correlation can be implemented, and has pointed to diagnostic statistics that are available to check whether a given specification of the survey-based model is successful in this respect. Experience accumulated in many countries has revealed that the strategy proposed by the Bank s researchers generally works quite well. This methodological point could have been verified quite easily from the technical papers provided to the Panel. Since the evaluation period, the research team has also done new work that has validated their methods in a situation in which the "true" poverty map is known drawing on a data set for Mexico that combines a census-like data structure with direct measures of 3 Elbers, Chris, Lanjouw, Peter and Lanjouw, Jean, 2003, Micro-Level Estimation of Poverty and Inequality, Econometrica, 71(1): 355-64. 4 Elbers, C., Lanjouw, J.O., and Lanjouw, P. (2002) Micro-Level Estimation of Welfare, Policy Research Working Paper No. 2911, the World Bank. This paper is the basic reference document that the PM team provides to any enquiries about the Bank s methodology. 2
household expenditure. 5 In this setting the team is able to compare the standard PM estimates to the "truth" and is also able to see whether the standard errors produced are excessively optimistic. The evidence indicates that the standard errors are in fact quite reasonable. While this paper had not been produced in time for the evaluation, the draft could have been made available to the Panel if there had been any consultation with the PM team. In summary, the PM team does not accept the suggestion that its methods for gauging precision are inadequate. The team believes that there are clear signposts in the procedure that can help determine whether or not standard errors, based on the independence of EA s assumption, are likely to be severe underestimates of the true standard errors. Moreover, the possibility exists, within the basic framework, to gauge how much larger standard errors could be if the assumption of independence was dropped. In practical applications, close attention to this issue is built explicitly into the way that analysts are expected to work with the software that has been developed by the team. Furthermore, all technical documentation, including manuals aimed at providing guidance to practitioners, stress the importance of focusing on these issues. Finally, it should be noted that during the extensive consultations that have occurred between the team and numerous experts in the field of Small Area Statistics, the PM team s approach to dealing with the concerns raised by the Panel has never been singled out as problematic. Claim 2: The US statistical service has rejected the Bank s methods as being unreliable Alongside its central methodological concern the Report claims that the US Census Bureau has rejected the World Bank s poverty mapping method because the statistical methods were judged to be inadequate. To quote the Panel: If the method cannot reliably deliver what it claims, the Bank should not provide statistical methods to its clients that have been rejected by the US statistical service. (Report, p.26). We would submit that the US Census Bureau does not use the Bank s method because the US government has the resources needed for measuring poverty directly at local level. The situation is very different in the low and middle-income countries in which the Bank works. In that light, the fact that the US statistical service does not use the Bank s methods is of little or no relevance. To elaborate, it should first be noted that, up to the year 2000, the Census Bureau has been able to directly measure poverty at the local level based on data on household incomes collected directly in the Census long form questionnaire. In 2010 the Census will no longer include a long form questionnaire. Instead, the Census Bureau has launched the American Community Survey (ACS) which is to be fielded annually and which has a sample size of some 3 million households (roughly 1 out of every 40 or so households in the country). With this massive household survey the Census Bureau will be in a position to publish local level estimates of poverty on an annual basis at the very local level. The World Bank s methodology could never offer the wealth of detailed information that will be available from the ACS, because even in the US census data are not available more frequently than once every ten years. However, in the developing world in which the Bank team works, the idea of an annual survey of the scale of the ACS remains unimaginable. Consider a stylized developing country with a population of (say) 20 million people in 4 million households. The country does a LSMStype survey of 5,000 households every five years at a cost that is proportional to sample size. To 5 Demombynes, G., Elbers, C., Lanjouw J.O., and Lanjouw, P. (2006) How Good a Map: Putting Small Area Estimation to the Test, mimeo, DECRG, the World Bank. 3
achieve the 1/40 sampling rate of the ACS the government of this country would need 20 times the sample size, and to do it annually (as in the ACS) it would end up incurring 100 times the cost of its present survey! If anything, this is likely to be an underestimate, since surveying costs will probably rise as the logistical and human resource constraints found in a developing countrysetting start to bite. Of course the extra cost would bring a benefit, allowing the country to construct a poverty map built up from directly measured incomes or expenditures. And a sample 20 times larger than the original LSMS survey would reduce the original survey-based standard errors by a factor of 4-5, quite possibly achieving a level of precision that is greater than what might be feasible with a poverty map based on the Bank s methodology. These simple calculations are only meant to be indicative of the type of trade off facing Statistics Offices throughout the developing world. In some countries the trade-off will be greater, and in some it will lower. But there is a trade off. The choice made will depend on many factors, including the country s wealth. The US government is willing to incur a large increase in its public outlays on data collection to achieve greater precision; understandably, most developing countries are not. Indeed, we would expect that even fielding a household survey like the ACS once every 10 years, would be beyond the financial, and possibly logistical, means of Statistical Offices in most developing countries. This is precisely why the Bank team set out to explore options for squeezing more out of existing data sources via its small-area estimation methodology. Claim 3: The Bank s researchers have not been client focused The Bank s experience of poverty mapping in South Africa is cited and is used to draw broad inferences about how the World Bank researchers involved in the PM project have been working with their counterparts. With reference to the South Africa poverty maps, the Report claims that the poverty maps were not locally owned, with the technical work being done only in Washington, and members of the Bank team were not responsive to (informed) technical questions from South Africans working on poverty measurement, including South African academics working with the Ministry of Finance. (Report, p.26). This discussion is quite misleading both because it is unwise to extrapolate on the basis of a single example and, more importantly, because the Panel s understanding of the poverty mapping process in South Africa, and the Bank s involvement in this process, is seriously misinformed. To start, it should be made clear that the only poverty map for South Africa in which the PM team had any role to play involved the 1996 population census and the 1995 IES/OHS household survey. The World Bank team was an active and very hands-on partner with Statistics South Africa in producing that poverty map. It is very easy to document the lengths to which the Bank team went to engage with Statistics South Africa in producing it. There is absolutely no basis to the suggestion that the technical work was done in Washington D.C., or that there was no willingness by the Bank team to respond to technical questions and concerns. Bank staff spent many person months in South Africa working alongside, and in close collaboration with, staff in Statistics South Africa. The Agency has also very explicitly acknowledged the collaborative nature of the work on the 1996 poverty map: the Report can be downloaded off the Statistics South Africa website and it is clearly denoted as the output of a joint research project with the World Bank. 4
It has come to our attention that Statistics South Africa has experimented with a second poverty map, involving 2001 Census data. The World Bank team had no involvement whatsoever in producing this second poverty map and, indeed, has never seen the results from this exercise nor been invited to assess the work that was done. While the team cannot provide any comment on the quality of the 2001 poverty map or on the process that led to it, we would like to point out that the very existence of such an effort testifies to the lengths that the team went, in its original collaboration with Statistics South Africa, to transfer knowledge and to build capacity. As the team was never invited to provide any assistance for this second map, there is certainly no basis for blaming the team for concerns that might stem from that effort. Moreover, even though the PM team had no involvement in poverty mapping in South Africa beyond the original 1996 map, the team has never neglected its responsibilities with regards to advice and guidance. Indeed, the PM team has electronic records of very detailed correspondence in mid- 2003 between the team and the relevant member of the evaluation team concerning details of the poverty mapping methodology. The picture painted by the Report of an unengaged Bank team, refusing to respond to queries from South Africans is demonstrably inaccurate. Looking forward, the research group will continue with its PM work, including pursuing opportunities to validate the methodology and the team will certainly be addressing, as it has in the past, the concerns raised by the Report. In addition the team will continue its efforts to refine and upgrade the software that it has been developing for poverty mapping purposes. This software not only reduces the computational burden of producing poverty maps, but provides a convenient and efficient platform upon which to probe robustness and to explore opportunities for methodological refinement. 5