In a write-up in Sankhyā: The Indian Journal of Statistics, in 1935, Tagore referred to the magic of mathematics as “the dance steps of numbers in the arena of time and space, which weave the maya of appearance, the incessant flow of changes that ever is and is not”. As today’s data-obsessed world aspires to make data-driven decisions on every bit of life and lifestyle, one has to determine how the “maya” is woven and what data might genuinely offer.
Analyses and findings
An example first. The findings of a study that provided independent analysts with nearly four million words from roughly 8,000 comments in an online academic forum were described in a paper in Organisational Behaviour and Human Decision Processes (2021) by a large collaborative group of scientists. The study posed two straightforward hypotheses about how gender and academic status influence contributions to the forum’s discussions. Within 29 studies, radically varied analyses and dispersed empirical findings were reported; several cases had significant effects in opposite directions for the same research question. Had it been only a single analyst using a single method, one would have stopped at that result.
What is mentioned above is but one instance of conflicting conclusions drawn from the same data by several experts. Such situations may, of course, cause a great deal of controversy if the data and/or topic are of public interest. Such a situation may seem weird at first. As a result, the common person is perplexed; they are unsure about the causes of these discrepancies and, more crucially, whose analysis to trust. Even experts struggle to comprehend the merits of various contradictory conclusions.
Well, statistical analyses are not unique, especially when different methodologies are used. Interestingly, depending on the models and associated margins of error, all findings could be correct, even if they are contradictory in nature. However, certain analyses may contain significant errors. But it is never easy to identify them, even for experts. The history of civilisation shows that the conclusions drawn from the simple observation that the sun crosses the sky every day could even be varied and contradictory. It took centuries to settle the issue. Thus, domain expertise is necessary; generic statistical aptitude alone is insufficient, so far as data analyses are concerned.
Anchored to a ‘model’
How is it possible that the same data could lead to contradictory conclusions? This is normal, though, as statistical analyses rely on an underlying “model” to explain the pattern of the data. The decision-making processes depend heavily on these models, the relevant metrics, and the associated margin of error for drawing conclusions. While any statistical decision is subject to some margin of error, a common person is frequently unaware of this. Additionally, models are created using the expertise and judgement of the statisticians concerned. Interestingly, George E.P. Box, a British statistician, is credited with the famous quote: “All models are wrong, but some are useful”. And a “suitable” model is frequently not statistically chosen before doing an analysis.
Thus, understanding the causality underlying a phenomenon should serve as the foundation for any data analysis. Let us consider a well-known example: the gender bias data for admissions for the autumn of 1973 at the University of California, Berkeley. According to the data, just 35% of women applicants were admitted, compared to 44% of men who applied — a significant enough difference to be explained by chance. A comprehensive examination of the data, however, revealed that different departments had varying degrees of difficulty admitting students; and that men tended to apply to less competitive departments with higher acceptance rates, while women tended to apply to more competitive departments with lower admission rates. The department-wise data actually revealed a “small but statistically significant bias in favor of women”.
In Delhi, in 1947
Sensing the heartbeats of data is a tricky task in itself. Another example. A few members of a minority community took refuge in the Red Fort during the communal riots in Delhi in 1947. The number was unknown to the government, and contractors hired to feed them charged high amounts. A team from the Indian Statistical Institute (ISI) obtained three estimates of the number by dividing the quantities of rice, pulses and salt used per day, as quoted by the contractors by the respective per capita requirements of these commodities. The estimate obtained by rice was the largest, and the salt estimate was the least. Since rice was the most expensive, its quantity was presumably inflated. And because salt was inexpensive, the contractors did not use it that way. The ISI team suggested using the quantity obtained from salt as an estimate of the number of refugees.
It demonstrates how statistical expertise, wisdom, and a solid understanding of reality are all essential for fully grasping “the dance steps of numbers”. But extracting the best out of data is never easy. Only top statistical experts, along with experts with domain knowledge, may, at most, aspire to perform that job.
Atanu Biswas is Professor of Statistics, Indian Statistical Institute, Kolkata
