Without any doubt there is a massive amount of data out there; according to one study, the world used over 2.8 trillion gigabytes in 2012. The total volume isn’t the point. We might even use the old cliché (from another context), “Size doesn’t matter.”
The point is not to be confused by the volume of data, but rather to better understand how to analyze it and to convert it into insights, innovations, and business value. Conventional analytics and big data analytics are often very fuzzy to differentiate. What is actually different about big data and how does this relate to traditional data management and analytics? E.g. when we have structured data sets the traditional approach is to use a mechanistic form of analysis – what we in IT tend to use. Whereas with unstructured data heuristic patterns of analysis are required to help identify trends. Selecting a data set for analysis from structured data will be easy by comparison to selecting and analyzing a set from a true “Big Data”-base. These databases usually contain both structured and unstructured data/information and cannot be queried using SQL or other traditional data manipulation tools. People with the skill sets required to process and analyze that type of data are and will likely remain in very short supply for years to come.
The idea of analyzing data to make sense of what’s happening in businesses has been with us for a long time. For example, an analytics group was started by UPS as far back as 1954.
A recent study that I found in the Harvard business magazine suggests that only 0.5 percent of the worldwide 2.8 zettabytes of data is analyzed in any way. The greatest barrier to analysis is that we first have to impose structure on big data; not all of it will be useful—the study estimates about 25 percent has potential value.
So for me it is important to ask what new techniques or technical architectures can be applied to an enterprise to make big data work e.g. by making us more efficient in decision making.