The story so far: The Government of India recently appointed a panel under the chairmanship of Pronab Sen, former Chief Statistician of India to review the methodology of the National Statistical Organisation (NSO).
Why is there a need for review?
This happened in the backdrop of articles by Shamika Ravi and Bibek Debroy arguing that the usage of outdated survey methodology by national surveys such as the National Sample Survey (NSS), National Family Health Survey (NFHS) and Periodic Labour Force Survey (PLFS), have systematically underestimated India’s development. According to them, this archaic methodology has failed to capture reality in the recent past as the “Indian economy has been incredibly dynamic in the last 30 years”. On the other hand, P. C. Mohanan and Amitabh Kundu have reasoned that there is no systematic underestimation of development by these national surveys. Therefore, there is no need for restructuring and overhauling the survey methodology just because it doesn’t suit certain narratives of development. However, they accept that there may be errors, which should be minimised. They also advocate the usage of appropriate sample weights to make the national sample adequately representative.
Why is it important?
National level data is a key resource for research, policymaking and development planning, so it is of utmost importance to understand and analyse both claims in the light of existing evidence. For this purpose, we will be taking a closer look at NFHS data, which is being conducted by the Ministry of Health and Family Welfare for the last 30 years with the International Institute of Population Sciences (IIPS) as the nodal agency.
Does the NFHS have a rural bias?
Ms. Ravi and Mr. Debroy in separate articles have argued that one of the ways in which national surveys like the NFHS are underestimating Indian development is through “rural bias in terms of representation”. In other words, the survey methodology, which depends heavily on the last Census data, systematically overestimates the rural population. However, the examination of five rounds of NFHS data doesn’t show any such systematic bias towards rural population. If we match the estimates of the urban population by the NFHS with the corresponding World Bank estimates and urban percentage projections from the Census figures of 1991, 2001 and 2011, it is clear that there is no evidence of any systematic rural bias (Table).
On the other hand, there is evidence of rural population underestimation by NFHS-3. Overestimation of rural population seems to have taken place by NFHS-2 and NFHS-5. Only NFHS-1 and NFHS-4 estimates seem to be really close to World Bank estimates and projections based on Census data. However, these errors seem random rather than systematic.
How can we minimise such errors?
Generally, there are higher percentages of no-response in urban areas compared to rural areas. However, this also does not seem to have any systematic relation with either rural or urban bias in estimation. For example, NFHS-1 with its near correct estimation of urban population had the lowest urban response rate at 94.5%, while NFHS-5 with its underestimation of urban population had the second lowest urban response rate of 95.6%.
Nonetheless, no response or not, there seems to be room for improvement in minimising the errors and the way sample weights are assigned. A cursory glance at the percentages of the urban sample in the unweighted sample indicates that assigning appropriate weights may help to correct the errors to a great extent. In NFHS 1, 2, 3, 4 and 5 the unweighted percentages of urban sample were 31.0%, 31.3%, 44.2%, 28.0% and 24.2% respectively (Table). If the sample weights are appropriately assigned, after taking into account all possible sources of error, then underrepresentation of either rural or urban seems to get corrected to a large extent.
What next?
The Pronab Sen Committee needs to address these concerns to make the sample adequately representative rather than go for a complete overhaul of the survey methodology. In the name of removing systematic rural bias in survey methodology, which does not seem to exist in the first place, let us not import systematic urban bias rampant in policymaking, planning and financing to national level surveys. Let us remove the bias where it actually exists.
The writers are researchers in Development Economics.
