What Are the Common Data Sources?

We know that data-driven decision-making is important, but where does all this data come from?

Traditionally, business management systems such as Enterprise Resource Planning (ERP) or Customer Relationship Management (CRM) software were the main source of data for making decisions. These days, the variety of data sources used by organisations to make better decisions is growing.

Let’s look at some of these common data sources.

Internal data

Internal data is captured by your organisational processes. It can be transactional data such as customer purchases, or payments for staff and contractors. Your organisation might have machine-generated data from sensors or devices used to manufacture a product, or recorded by the product itself (e.g. smartphones or IoT devices). Finally, you might capture engagement data from your customers or clients.

Examples include:

• email marketing metrics (email opens, click rates)
• information in customer profiles
• records of customer interactions (email queries, support calls etc.)
• online activity (e.g. placing items in an online shopping cart).

You’ll typically find this data stored in databases, operational systems (e.g. CRM, ERP) or in system log files.

Using internal data has advantages.

• It’s already there – you can get started right away.
• You might not need to apply for use (i.e. no restrictions for use in your organisation).
• You can talk to the people who gathered the data.
• It might already be in the right format for you (i.e. same code, software, using the same standards and style guides), which can save a lot of time.

Third-party analytics

Third-party analytics are useful if you don’t have the capacity to capture data. These are available for a number of business functions. Web analytics are the most well known.

Third-party web analytics services can provide cost-effective collection and analysis and evaluate how your website performs over time, or against averages across the provider’s customer base. Google Analytics is a popular tool that provides businesses with the ability to analyse and understand how users find and use their websites and applications.

External data

External data can include almost anything – from historical demographic data to market prices, from weather conditions to social-media trends. Organisations use external data to analyse and model economic, political, social, or environmental factors that influence their business.

External data can be derived from a variety of sources. There are open initiatives (e.g. data.gov.uk), social-media services (e.g. Twitter or Linkedin), and paid services (e.g. Thompson Reuters or Westlaw).

Open data

Open data is accessible to everyone and is free to use. However, if it’s high-level data, or it’s heavily summarised and aggregated, it might not be very relevant to you. It might also not be in the format you need, or it might be very difficult for you to make sense of. All of these challenges can require a lot of time to make the data usable.

Open data sources include:

• government data – data.gov (US), data.gov.uk (UK), data.gov.au (AUS)
• health and scientific data – World Health Organisation (WHO), Nature.com scientific data, Open Science Data Cloud (OSCDC), Center for Open Science
• social media – Google trends (look at national trends on search terms), Yahoo finance (great for stock market information), Twitter (allows you to search by tags and users, which can be downloaded by using Twitter APIs).

Accessing data through APIs

Application programming interfaces (APIs) aren’t a source of data but they do provide a way to share data – you can take it from one app to another and route, translate, and prepare it for use. APIs can be used for a range of applications, from social (e.g. Twitter, Facebook), to utilities (e.g. Dropbox) and commerce (e.g. Mailchimp, Slack), to name a view.

Data is everywhere. An analyst’s intuition and a subject matter expert’s input will guide you to the right data from the right sources to address and solve your business problem.