As a business anthropologist working as a mixed-methods researcher, I support the use of qualitative and quantitive methods, including big data. But, I do not support the use of only big data.
Research and the insights that are generated are best when produced from a multitude of methods that can reinforce and validate each other.
But in industry, there appears to be a mythology enveloping big data even though big data practices are often failing to generate the desired ROI.
To understand this phenomenon, let’s take a look at big data from an anthropological perspective.
What Is Big Data
If we are to discuss big data, we should first state what data is. Data, for the sake of this conversation, is being defined as digital data collected and stored in binary form for analysis by custom or commercial algorithms.
Typically the data is part of a large dataset, is incredibly diverse in nature, and is collected at a rapid rate which would have been unfathomable even a decade ago. These attributes are summed up as the 3Vs (volume, variety, and velocity).
But big in this sense, does not equate to a depth of understanding, despite the fact that many adoptees believe it will.
The Big Data Problem: ROI
In February of 2017, Gartner estimated the worldwide business intelligence and analytics market would reach $18.3 Billion, however, according to research by NewVantage Partners, only 37% of companies who are trying to be data-driven have been successful.
The Big Data Landscape
Despite the findings by NewVantage Partners, worldwide big data market revenues for software and services are projected to increase from $42B in 2018 to $103B in 2027, attaining a Compound Annual Growth Rate (CAGR) of 10.48% according to Wikibon.
According to an Accenture study, 79% of enterprise executives agree that companies that do not embrace big data will lose their competitive position and could face extinction. As a result, 83% have pursued big data projects to seize a competitive edge.
Big Data Success
Proponents of big data have described the discipline and the methods for collecting, analyzing, and applying the insights as nothing short of a revolution in information.
This has led to big data initiatives being implemented across a wide range of private and public sectors such as business process optimization, demand-oriented energy supply, market- and trend forecasting, uncovering illegal financial transactions, predictive policing, enhanced health research by analyzing population diseases, cancer research, and software-supported medical diagnosis.
In these applications, big data has been demonstrated to be incredibly adept at surfacing correlations within data. This has already proven to be very useful in the health sector by finding interrelations between symptoms of different diseases and for exploring the side effects of drugs.
Big Data Mythology
Detractors, however, point out that big data is not the solution to all of our problems as some passionate adoptees believe. Like any other technology, big data is a cultural innovation with all of its own trappings, and likewise, it must be viewed as such.
As with any technology, our culture shapes it, as much as it shapes our culture. Likewise, when discussing big data, we need to be mindful of our beliefs and motivations because they are shaped by society, for better or worse.
To that end, we need to respect that big data is a “cultural, technological, and scholarly phenomenon that rests on the interplay of technology, analysis, and mythology” as Boyd and Crawford have pointed out.
That is not to say that big data has no place in society. Of course, it does. But like any cultural construct, big data has its own mythology, and we need to be aware of how that influences us.
We must recognize that by adopting the big data mythology we shape how we see and understand the problem and solutions space. This applies to individual practitioners as well as society at large.
As Bruno Latour said: “Change the instruments, and you will change the entire social theory that goes with them.”
What is the Future of Big Data?
The future of big data is certainly increased adoption simply because it involves quantification, even if it does not produce the desired ROI.
As an applied anthropologist, I am painfully aware of the quantification bias in our society, especially in industry. And though I am absolutely not against the adoption of big data for that purpose, what I would like to see is a future where big data is combined with qualitative methods to produce richer, more nuanced insights.
Big data ought to be combined with the concept of thick data (an extension of Geertz Thick Description) popularized by Tricia Wang in her article Why Big Data Needs Thick Data.
Let’s Combine Big + Thick Data
Data scientists and social scientists should be paired with each other in the process of asking questions, listening, defining, reframing, collecting, analyzing, and presenting the data.
People and teams are limited by their individual and shared cultural understanding. By introducing other perspectives, including the perspective of those who have had their (consumers/users) data collected, we can enrich the collective knowledge for the benefit of all.
Social scientists should not resist the introduction of big data. They should partner with data scientists as close as possible to find trends in the noise that should be explored in more depth.
Data scientists, on the other hand, should not view the findings and insights of qualitative methods as too small. They should leverage the rich understanding qualitative research offers to give meaning to the trends they are seeing in the data.
If we do those things, we will be much closer to understanding the underlying causes of the messy problems we face, rather than just identifying irrelevant trends in the noise.
