'Big Data' is a term that's been bandied about in ever increasing frequency over the last few years - usually by people selling incredibly expensive technology solutions. The claim is that getting a handle on your big data can lead to big gains in productivity, efficiency and competitive edge. But are these claims justified? And if they are, what constitutes big data and how do you know when you've passed a threshold that requires a different approach to data management?
What is it?
Big Data is an elastic term - there's no numerical definition of what constitutes a big data set. For some organisations a database that expands into the tens of gigabytes starts to become unwieldy. For others, that cut-off lies in the tens of terabytes. As technology advances and computing power grows, in accordance with Moore's Law, the lines keep moving ever upwards. (Almost to illustrate the point, the version of MS Word I'm using to type this post recognises the word 'gigabytes' but marks up 'terabytes' as a spelling error.)
Big Data therefore refers more to an approach to data handling than it does to the actual amount of data that needs to be handled. When your current set-up no longer allows you to use data in an efficient way or doesn't allow you to pull the information you need from your data easily then you need to start thinking in terms of big data.
Examples of Big Data
Big data approaches originated in the spheres of scientific research and government. Scientists regularly need to mine and visualise massive datasets of e.g. weather readings, to draw conclusions and recognise trends. Governments need to handle incredible amounts of data on tax, criminal records, vehicle registration etc. Techniques of manipulating big data were developed to make these processes as efficient and useful as possible.
Then came the internet revolution and the amount of data flying around the world exploded. Website interactions, online transactions and social networking meant that all of a sudden masses of data on customer activity was available and companies rapidly started working out how they could use it to their advantage.
Companies like Google, Facebook and Amazon led the way.
- Amazon process millions of transactions every day - both through direct sales and via third-party sellers. In 2005 they had the world's three largest Linux databases.
- Facebook needs to hold billions of pictures and make them available for instant recall. They also need to keep tabs on the myriad interconnections between their users and all their updates, not to mention advertising data. Their latest data centre in Lulea, Sweden, is the size of 11 football pitches.
- Google's data makes Facebook's look like an amateur stamp collection. They are incredibly secretive about their huge data centres but they admit to at least a dozen of comparable size.
- Walmart - the US supermarket chain and owners of ASDA - process about a million transactions an hour. The database which records them all is estimated to be in the range of 250 petabytes large (2560 terbaytes).
What Can You Do with Big Data?
From a business perspective, the more data you can collect on your customers' habits, preferences and behaviour - whether related to purchases they've made from you or not - the better. You can use this data to build a model of different types of customer, what marketing, selling and customer services approaches they best respond to and to help predict what their demands are likely to be in the future.
Types of beneficial data include:
- Transactional data can be used to find price points, determine product line-up, identify barriers to purchase and provide re-marketing information.
- Web analytics will show you how customers are finding you, which marketing methods work best, how well visitors to your site are converting and what can be done to improve that rate.
- Customer services data can help you identify weaknesses in your service offering and help you retain more customers that would otherwise find the waiting arms of your competitors.
- Social media data can help you build profiles of your customers to better target them with marketing material. It can also help to identify gaps in the marketplace and provide information on how you and your competitors are generally perceived.
- Purchasing data can help you identify trends in contract prices, which suppliers are better value over the long term and where there is room for renegotiation.
- Employee data provides insight on the performance of individual staff members, whole teams and managers. It can also help you determine the productivity benefits of changes to processes or technology.
The big data approach means putting in place technology that allows you to not only record and store the above types of data but also be able to use it in a beneficial way. Being able to put the data in a format from which you can draw meaningful, actionable conclusions that actually benefit your business is crucial.
If you can't easily use your data for visualisations, comparisons, trend-spotting, model building and all the other kinds of analysis that lead to efficiency, productivity or revenue gains then you're just collecting data for the sake of it.
Do You Have Big Data?
Perhaps a more sensible question is: do you have the potential to collect data? To which the answer is undoubtedly 'yes'. Would that data allow you to improve your business? Again, the answer is probably 'yes'. Which leads to the question: could you be doing better?
Depending on the size of your business you might find that your current set-up involving spreadsheets and free software like Google Analytics lets you do everything you need to. On the other hand, your current relational databases might be creaking under the demands of your sales and marketing teams and you simply can't keep up with the social media activity surrounding your brand online. In this case, it's time to review your approach to data and start investigating how you can make the data available to you start working for you.