Please give me a detailed introduction, including history, of Big Data
Hello! Thanks for your question about the history of Big Data. The short version is that the Big Data has been a long time in coming and was predicated by inherent necessities of higher levels of organization. However, the modern term came about in the mid-aughts and reached prominence in the early tens. While many critics and barriers existed and exist for big data, more and more fields are utilizing and being transformed by it. Below you will find a deep dive of my findings.
As this article entitled "A Short History of Big Data" points out, the conceptual origin of Big Data can theoretically be traced back millennia ago to Mesopotamian accounting primarily dealing with agriculture. Throughout the preceding time and civilizations, higher levels of organization in business, government, and culture have continued to necessitate larger scale gathering storage and analysis of data in various mediums The article highlights John Gaunt as the founder of modern statistics, Herman Hollerith as the inventor the first means of digital data, and US Government projects including Social Security and the NSA as further key events in modern data practices leading to larger and larger scales of data collection and use.
As for "Big Data" an etymological look at the term published by the New York Times comes to inconclusive results, unearthing sources from a musing about junk mail published in 1989, a more relevant use of the term in a 2003 published paper on macroeconomic modeling and the term's use by an early Silicon Valley icon in presentation slides before it started to filter into informal technical material online. However, as "A Short History..." points out the modern development of the term and concept arose from Roger Mougalas in 2005. Progressing from there, Big Data, that is the utilization of exponentially larger data stores, progressed from there.
The New York Times tells us, in 2010 The Library of Congress and Twitter struck deals to use big data stores from twitter for cultural and linguistic research, and likewise declared 2012 to have been the "breakout year for Big Data". "A Short History..." points out that data sizes and Big Data experiences exponentially growth leading McKinsey to project large shortages of human infrastructure to deal with the data by 2018. Big Data is being used to analyze social history even predating the digital age and is shaking up education, politics and even law enforcement among many other fields as it is applied in new contexts.
SKEPTICISM AND RESISTANCE
As with many new technological systems, Big Data was not and is not without its fair share of critics and resistors. Written by a data scientist, this article tells us that one barrier in implementing Big Data projects successfully is the traditional and often times politics laden structure of any organization. It explains that Big Data inherently needs unified data and artificial divisions within companies to preserve barriers or horde data as capital will undermine any project. This also relates to other issues which the article points out causes projects to fail and which are rooted in traditional business and organizational systems, namely lack of agility and lack of prioritizing data. A history of US law enforcement's adoption of Big Data particularly highlights this element of practical barriers and resistance within an organization tied very much to a traditional and political framework. This history cites lack of funding, lack of centralization to even have such data, political concerns and even personal attachment to traditional methods as major barriers.
An article written by a former Big Data skeptic, details that in many fields and education in particular ethical concerns about data collection, use, and privacy still underpin some amount of criticism of its use such as a 2014 Forbes article by Neil M Richards. Ethical concerns also have arisen around Big Data implementation in political campaigning. As cited in this AdWeek article, Clinton herself has counted her campaign and the DNC's relatively mediocre data utilization as a factor in her 2016 upset loss. The use of data analysis generally and Big Data procedures can have effects on the campaign and candidate which Vox points out is the reason why it has been associated with increased political polarization by candidates who now specifically target solely their base.
A recent article entitled "Benefits of Skepticism: Big Data" brings up another central criticism of Big Data, is the formulation, filtering and assessment of questions and how it is applied may be subject to errors such as correlation fallacies, recency and confirmation bias, and have inherent flaws in data quality. Not asking the right or loosely defined questions is likewise listed here as one of the primary reasons which Big Data projects fail, because "Flawed Hypothesis are built, leading to numerous hidden assumptions that result in crap business use cases." Similarly, AdWeek addresses the 2016 election upset similarly in which Big Data processes used on specific indicators had been proven wrong in their conclusions. Likewise, many skeptics address the fact that many questions and concerns may not be addressed in this manner at all, for example this article extolling the large gains in historical research Big Data processes made by analyzing 150 years of newspapers also tells us that certain questions about history and society "will always be the realm of the humanities and social sciences, and never that of machines."
NEW MARKETS AND CAREERS
Big Data has applications in an ever-growing list of markets and with it come new specialized careers as well. For almost any business or oganization, Big Data can provide big benefits that are addressed in this Harvard Business Review article which lists firms using it for cutting costs and adding revenue by adding greater precision and speed to their processes. As addressed in this article by a former critic of Big Data, she now applauds Big Data's usage to help assess and aid at-risk students to streamline the educational process. And Big Data is providing utility for very unexpected social science fields like history as illustrated here. Tech Crunch tells us that start-ups enabled by the use of Big Data are aiming at revolutionizing waste management, agriculture, and energy use among other fields. And finally, while the process has been slower and messier, governance such as law enforcement and national level politics have been adopting Big Data processes for their own ends as well.
Careers in Big Data in any of these industries have certain commonalities despite working with vastly different data as pointed out by this article listing data jobs. These job description which generally include the collection, storage, organization and utilization of any type of data are said to be in high demand, driven by the shortages projected by McKinsey as mentioned here.
To wrap it up, the perception of Big Data is varied, with a fair share of resistors and critics even today as it continues to enter new fields and create new careers. Thanks for using Wonder! Please let us know if we can help with anything else!