Skip main navigation

What is big data?

In this article, Sebastian Binnewies provides an introduction to big data, considering a possible definition and the V’s that characterise big data.
A question mark in front of an abstract data graph
© Griffith University
No doubt you’ve heard the terms ‘big data’ and ‘analytics’ being thrown around in the media. Let’s look at what these concepts really mean.
Watch this short video for a quick introduction to what big data is and the possibilities that it holds.

This is an additional video, hosted on YouTube.

Defining big data

‘Big data’ refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.1
If you look closely at this definition, you can see that it is framed in terms of time. It uses the word ‘typical’, and thus refers to current state-of-the-art technology. So, what we called big data 10 years ago, may not be big data now because the ‘typical’ tools and technologies have changed. And what we call big data now, may not be big data in 5 years.2 In the future, we may still use traditional data collection, storage, and processing systems, however, most likely in conjunction with newer systems.

The V’s that characterise big data

To determine whether data is big data, we can also consider the V’s that characterise big data. The four most commonly defined V dimensions are volume, variety, velocity, and veracity.3

Volume

Volume refers to the quantity of data to be stored. For example, Walmart deals with big data. They handle more than 1 million customer transactions every hour, importing more than 2.5 petabytes of data into their database. This is about 167 times the amount of information contained in all the books in the US Library of Congress.
The following table lists the different storage capacity units. To put these in context, there are 8,000,000,000,000,000,000,000,000 bits (that’s an eight followed by 24 zeros) in one yottabyte.
TermCapacityAbbreviation
Bit0 or 1 valueb
Byte8 bitsB
Kilobyte1024* bytesKB
Megabyte1024 KBMB
Gigabyte1024 MBGB
Terabyte1024 GBTB
Petabyte1024 TBPB
Exabyte1024 PBEB
Zettabyte1024 EBZB
Yottabyte1024 ZBYB
* Note that because bits are binary in nature and are the basis on which all other storage values are based, all values for data storage units are defined in terms of powers of 2. For example, the prefix kilo typically means 1000; however, in data storage, a kilobyte = 210 = 1024 bytes.2 (Table 14.1, Storage Capacity Units; p. 651)
To manage big volumes of data, we have two options for handling additional load.2
  • Scale up, meaning we keep the same number of systems to store and process data, but migrate each system to a larger system.
  • Scale out, meaning we increase the number of systems, but do not migrate to larger systems.

Velocity

Velocity refers to the speed at which data is entered into a system and must be processed. For example, Amazon captures every click of the mouse while shoppers are browsing on its website.2 This happens rapidly.
Velocity is important in stream processing. Think of all the data from radio-frequency identification (RFID), global positioning system (GPS), near-field communication (NFC), and Bluetooth sensors flooding in to a system. Stream processing aims to aggregate single data points from high-velocity data, in order to trigger a high-level event when a certain pattern is detected. It also focuses on deciding which data to keep from a stream, since it is unfeasible to retain all the data that is rushing in.

Variety

Variety refers to the complexity of data formats. Big data consists of different forms of data. For example, when a telecommunications company like Telstra records data on calls to its call centre, this data includes both:
  • structured data, which conforms to a predefined data model (e.g., your customer ID, the timestamp of your call, your service type), and
  • unstructured data (e.g., the recording of the call, notes that the call centre operator makes during the call, the problem history related to your call).

Veracity

Veracity refers to the trustworthiness of data. The more data is collected and analysed automatically but not captured in its entirety (due to the high volume and velocity), the higher the uncertainty about the accuracy of data. For example, it is particularly challenging to verify the truthfulness of posts on social media platforms, as we do not always know the posters’ backgrounds and their intentions. In fact, detecting fake reviews, fake news, and fake friends is currently an active research area.

The four V’s as an infographic

The IBM Big Data & Analytics Hub provides an infographic which explains and gives examples of each of the four V’s.
To expand the infographic, click on the image. You will also find a downloadable PDF text version of this infographic in the downloads section at the end of the step.
Volume, Velocity, Variety and Veracity

Other V’s

Further V’s that are often mentioned as key characteristics of big data are:
  • value: how meaningful the data is
  • visualisation: graphical representations to assist humans in understanding big data.
Hopefully, you now have an idea of what big data is. In the next step we will discuss where all the data is coming from.

Your task

How would you define big data?
Share your thoughts in the comments.

References

  1. Manyika J, Chui M, Brown B, Bughin J, Dobbs R, Roxburgh C, Byers AH. Big data: The next frontier for innovation, competition, and productivity [Internet]. McKinsey Global Institute; 2011[cited 2018 Oct 24]. 143 p. Available from: https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-next-frontier-for-innovation 
  2. Coronel C, Morris S. Database systems: Design, implementation, and management. 12th ed. Boston (MA): Cengage Learning; 2016.  2 3 4
  3. Elmasri R, Navathe SB. Fundamentals of database systems. 7th ed. Pearson; 2017. 
© Griffith University
This article is from the free online

Big Data Analytics: Opportunities, Challenges and the Future

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education