“Big Data”

epitomizes a buzz word. Tech blogs began following the emerging field of analyzing very, very, very large amounts of data (e.g.  the nearly 300,000 tweets, 220,000 Instagram photos, 72 hours of YouTube video content and the 2.5 million pieces of content shared by Facebook users every single minute) that humanity is generating at an unprecedented pace about 10 years ago. What was once an interesting exercise has become much more as startups have built tools that allow businesses to gather and analyze big data in real time. Tools like Hadoop and MongoDB are few among many of the entrants vying to lead the big data revolution that transcends mainstay relational SQL databases in favor of NoSQL databases that hoard every last datapoint that they can, even if those data provide no perceivable analytical value at present.

While many organizations are waking up to the potential of big data, most are slow to understand it. Companies are quick to advertise that they “leverage big data analytics” to “provide value” and discover “new synergies”, but more often than not they are putting a new dress on the same dog. While tools to harness big data are developing rapidly, people can be the limiting factor in catalyzing a shift to big data culture. This was recently espoused by Mark Gazit in an op-ed on TechCrunch. He cites research that catalogues the talent drought for data scientists, noting that there simply isn’t enough human horsepower available for companies attempting to integrate big data into their core business models. Rather than calling for more training programs, Gazit champions the efficacy of machine learning and artificial intelligence in bringing big data to the masses:

Ultimately, by solving the issues that prevent full optimization of big data analytics — especially the human factor and its disproportionate impact on the current-day process — organizations will be able to detect and address all types of threats and opportunities much more rapidly. This capability is becoming increasingly crucial in an era when data is being generated by both humans and machines, and is sure to become a pivotal way for businesses to create situational awareness, detect issues and optimize operations to achieve their business objectives. – Mark Gazit, CEO of ThetaRay

It remains to be seen how machines and humans will complement and subordinate one another as the data economy continues to evolve. Greater efficiencies, lower costs, and more predictability are inevitable – the question is, who will we have to thank?