What is big data?

This is an additional video, hosted on YouTube.
Defining big data
If you look closely at this definition, you can see that it is framed in terms of time. It uses the word ‘typical’, and thus refers to current state-of-the-art technology. So, what we called big data 10 years ago, may not be big data now because the ‘typical’ tools and technologies have changed. And what we call big data now, may not be big data in 5 years.2 In the future, we may still use traditional data collection, storage, and processing systems, however, most likely in conjunction with newer systems.‘Big data’ refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.1
The V’s that characterise big data
To determine whether data is big data, we can also consider the V’s that characterise big data. The four most commonly defined V dimensions are volume, variety, velocity, and veracity.3Volume
Volume refers to the quantity of data to be stored. For example, Walmart deals with big data. They handle more than 1 million customer transactions every hour, importing more than 2.5 petabytes of data into their database. This is about 167 times the amount of information contained in all the books in the US Library of Congress.The following table lists the different storage capacity units. To put these in context, there are 8,000,000,000,000,000,000,000,000 bits (that’s an eight followed by 24 zeros) in one yottabyte.Term | Capacity | Abbreviation |
---|---|---|
Bit | 0 or 1 value | b |
Byte | 8 bits | B |
Kilobyte | 1024* bytes | KB |
Megabyte | 1024 KB | MB |
Gigabyte | 1024 MB | GB |
Terabyte | 1024 GB | TB |
Petabyte | 1024 TB | PB |
Exabyte | 1024 PB | EB |
Zettabyte | 1024 EB | ZB |
Yottabyte | 1024 ZB | YB |
- Scale up, meaning we keep the same number of systems to store and process data, but migrate each system to a larger system.
- Scale out, meaning we increase the number of systems, but do not migrate to larger systems.
Velocity
Velocity refers to the speed at which data is entered into a system and must be processed. For example, Amazon captures every click of the mouse while shoppers are browsing on its website.2 This happens rapidly.Velocity is important in stream processing. Think of all the data from radio-frequency identification (RFID), global positioning system (GPS), near-field communication (NFC), and Bluetooth sensors flooding in to a system. Stream processing aims to aggregate single data points from high-velocity data, in order to trigger a high-level event when a certain pattern is detected. It also focuses on deciding which data to keep from a stream, since it is unfeasible to retain all the data that is rushing in.Variety
Variety refers to the complexity of data formats. Big data consists of different forms of data. For example, when a telecommunications company like Telstra records data on calls to its call centre, this data includes both:- structured data, which conforms to a predefined data model (e.g., your customer ID, the timestamp of your call, your service type), and
- unstructured data (e.g., the recording of the call, notes that the call centre operator makes during the call, the problem history related to your call).
Veracity
Veracity refers to the trustworthiness of data. The more data is collected and analysed automatically but not captured in its entirety (due to the high volume and velocity), the higher the uncertainty about the accuracy of data. For example, it is particularly challenging to verify the truthfulness of posts on social media platforms, as we do not always know the posters’ backgrounds and their intentions. In fact, detecting fake reviews, fake news, and fake friends is currently an active research area.The four V’s as an infographic
The IBM Big Data & Analytics Hub provides an infographic which explains and gives examples of each of the four V’s.To expand the infographic, click on the image. You will also find a downloadable PDF text version of this infographic in the downloads section at the end of the step.
Other V’s
Further V’s that are often mentioned as key characteristics of big data are:- value: how meaningful the data is
- visualisation: graphical representations to assist humans in understanding big data.
Your task
How would you define big data?Share your thoughts in the comments.References
- Manyika J, Chui M, Brown B, Bughin J, Dobbs R, Roxburgh C, Byers AH. Big data: The next frontier for innovation, competition, and productivity [Internet]. McKinsey Global Institute; 2011[cited 2018 Oct 24]. 143 p. Available from: https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-next-frontier-for-innovation ↩
- Coronel C, Morris S. Database systems: Design, implementation, and management. 12th ed. Boston (MA): Cengage Learning; 2016. ↩ ↩2 ↩3 ↩4
- Elmasri R, Navathe SB. Fundamentals of database systems. 7th ed. Pearson; 2017. ↩
Big Data Analytics: Opportunities, Challenges and the Future

Our purpose is to transform access to education.
We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.
We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.
Learn more about how FutureLearn is transforming access to education