The future of understanding complex issues and markets
By Greg Stewart
A lot of hype has been generated about the impact of Big Data in business today but few businesses are actually harnessing its full potential, with many companies lacking the knowledge, skills or technology to unlock the real value that a well-managed Big Data process and system has to offer.
There is no question as to the potential of Big Data within business and for business decision shaping, and this can be achieved with a well-planned approach.
So let’s first unpack what traditional data is compared to Big Data and exactly what big data is.
Traditional data, is data kept by most businesses either in a central server, or if a business has a central enterprise system it is stored here for access across the business. This includes data such as client / customer information, transactional data (sales & revenue data), procurement data, production & distribution data and market data (market research or intelligence). This data often holds important trends or insights and is used to make informed decisions or to determine strategy and future growth developments, or to improve performance and efficiencies within the organisation.
As an example, a digital publisher might measure the engagement level of different topics from its newsletter, and over time can personalise the newsletter content to specific readers and increase the overall readership and engagement levels of its articles. This in turn assists in driving advertising or sponsorship revenue.
What is Big Data then?
Big Data deals with large volumes of both structured, semi structured and unstructured data, typically sourced from outside the business on multiple platforms – social media, e-commerce and transactional data, Earth learning (satellite images), security camera data, mobile phone app data, location data and IOT (Internet of Things) data are examples, the size of which is too large and complex to be processed, managed or analysed by traditional data processing systems and software. Big Data is also characterised by the frequency that the data is generated and updated (mostly this happens daily) and is quite dynamic in its detail.
Digital platforms and communication across multiple channels has developed a plethora of activity driven data and the era of “personalisation” has driven the accumulation of Big Data to the point where hardly any human activity today goes unmeasured or unrecorded. The scale of this data is vast and almost incomprehensible. To store, analyse and utilise this data in a meaningful way in business, requires a whole new level of understanding and various computer and data science and analytical skills along with new digital tools such as machine learning and AI.
A small example of this type of Big Data application happened right now as I write this article. I had a gym session planned in my Google calendar for today and 30 minutes beforehand I received a message from Spotify to tell me it had created a special gym / work-out playlist for me to try out. Coincidence? Not likely, it is more likely some very sophisticated algorithm, driven by various data accumulated around my activities, triggered a timed message to me at a time when I should have been heading to the gym and when I might consider consuming some music. The music selected was carefully curated to be a combination of my known likes on Spotify and with an energised beat to match my pending exercise session. Intrusive? – Yes it is, but also very effective, and that is why doing Big Data well requires a number of things to be in place before it can hit the mark successfully
- Volume and frequency– without sufficient data volume, sourced regularly, valuable insights into customer behaviour, market trends and other critical information influencing development opportunities and potential in the data are potentially lost
- A clear strategy and expanded goals for the use of your Big Data should be explored before the data is accumulated, organised and analysed as to ensure that the correct data is targeted and acquired and meaningful insights are generated via your analysis processes.
- Software and analytical tools plus skilled analytical expertise is a must for any real success to be achieved with Big Data. Machine learning and AI are part of essential tools to be able to achieve any scope and proper value, and this may include integration of various unrelated data sets as well as layering of traditional data sets over Big Data to provide product or market specific insights. Outsourcing your Big Data development or system management to professional consulting specialists should be a serious consideration as they have the expertise and experience to advise and structure a winning big data operation that adds value to your business.
- A business integrated approach is recommended, as often the implementation of a Big Data strategy requires the input and co-operation of various disparate structures and departments within an organisation and there should be a broadly understood strategy within the organisation with complimenting departments preparing internal data in a way that makes it compatible with your big data strategy.
- Ethical considerations are today critical to take into account when planning to use Big Data, and more pressure is likely to be exerted by consumer rights groups in the future, on the use of personal activity data. Businesses are well advised to take this into consideration in their data sourcing, and also in their planning & execution of strategies. Having said that there are good approaches and modern big data sources that work in collecting big data ethically and this should be included in any Big Data procurement decisions.