Big Data demystified

Real-time streaming analytics, machine learning and the Internet of Things (IoT) are just some of the buzzwords in the world of Big Data. Understanding what they mean and how they relate to your organisation will help deliver better business outcomes.

As an emerging field, Big Data lacks authoritative definitions. The term is generally accepted to describe the processes, methods and technologies for a data analytics platform that encompasses the three V’s: volume, velocity and variety.

1. Volume refers to the large amounts of data now being generated; every click on a website creates data.
2. Velocity covers the speed with which data is generated; consider the speed with which wearable fitness devices create data.
3. Variety deals with the different types of data available; beyond traditional structured data in databases there’s now unstructured data such as tweets and multimedia such as videos.

Critically, Big Data from a corporate perspective generally incorporates both internally available data and relevant external data, such as customer interactions on social media.

Just like a traditional Data Warehouse (DW) environment, a Big Data environment still needs to acquire, store, transform, analyse and present information. Presently, these tasks are generally done using programming languages and in batch processes. As there are few definitive software leaders and even fewer best-practices, most organisations are making exploratory steps in the Big Data domain rather than making solid commitments.

While many consider real-time analytics–where data is acquired and processed in real-time to be part of the Big Data domain–the procedure uses different software, and processing is done in real-time rather than overnight or in batches. In addition to providing insights as data is processed, real-time analytics can be taken one step further by embedding it into operational processes; consider a call centre where an operator’s call-script is modified as the call progresses, in response to the conversation and the statistical probabilities of particular outcomes.

The implementation of real-time analytics in your business should be driven by use-cases where real-time intelligence, operational insights or decision processes are required or would drive improvements.

The world of Big Data offers many exciting opportunities to improve business performance by harnessing large data sets. Starting with the business use-cases within your organisation is the best way to determine if a Big Data solution is appropriate.

Do you have any of the V’s in your organisation? If so, a Big Data solution may be pertinent, if not then perhaps a DW solution is a better fit.