Big Data represents both a challenge and an opportunity for business and government. However, whilst it has been one of the main ‘hype topics’ in the last twelve months, its potential for driving change in many aspects of life should be of interest to all of us.

Big Data, Big Deal?

Rising data velocity, volume and variety has led to the coining of the term ‘big data’. It is estimated the ‘digital universe’ will grow from 0.92 Zettabytes to 35.2 Zettabytes in the next decade – a 44-fold increase, and 90% of which will be unstructured. As people start to realise the vast potential of this information it will open opportunities but also potentially a number of problems for organisations seeking to unleash the power within.

Not least of these problems will be how to manage it. The relatively small size of stored data in the past has meant that the analysis of that data has been quite straightforward and in the domain of the business teams, with IT providing tools, storage and access. However the growth in scale and complexity of the datasets causes a problem of how to manage them, a problem that has led to demand for a new role in the organisation, the Data Scientist, drawing upon disciplines from both computer science and vertical industries to process, analyse and store Big Data.

So where does this data come from? A variety of sources, including positioning information, mobile sensors, utilities, gene sequencing, video surveillance, and a wealth of other sources, and separating the digital wheat from the chaff will be the real challenge for data scientists, as a significant proportion of this data is useless.

Although it is unlikely that Big Data will take organisations by surprise – and not all organisations need a strategy for handling Big Data, it is something which all organisations should consider and prepare for. After all, organisational structure and change is often the largest factor hindering organisations taking advantage of Big Data, with existing silos and skill sets often being inadequate to readily take on the challenge.

One could argue that Big Data and its analysis should be relevant to every part of the organisation. Clearly there are business opportunities which can arise as a result of the information held as Big Data, and understanding this information can lead to innovation and improvement of business models. Similarly, failing to act on this information can lead to missed opportunities and lost profits. Consequently whether you are in the legal, financial, operations or in any other part of the organisational tree, Big Data is extremely relevant to your role in the future of the organisation.

Even in the public sector, making the most of Big Data could be critical to delivering more agile and cost effective public services, including improving education, innovating in R&D, driving better healthcare and maintaining competitiveness in an era where globalisation can have substantial influence on local affairs.

The government and public sector is also likely to be a major source of the data and the correct use of this data will drive public sector change and, by using open and anonymous data sets, could allow all sorts of commercial industries to benefit from broader and deeper sample sets than have ever been available before.

Big Data can come in many forms. For example, one team at the Sanger Institute / Wellcome Trust generates a terabyte of data each day carrying out gene sequencing. This wealth of data renders Moore’s Law (which states that the complexity of silicon circuits will double every 18 months) insufficient to process the sheer volume of information – traditional ways of handling this kind of data are simply inadequate.

This leads to evolved ‘data triage’ policies, with organisations having to be firm with which data needs to be processed and stored, and which does not. Clearly the effective use of IT will play a key role in enabling the benefits of Big Data to be realised.

The Role of Cloud

Solving this IT problem of managing rapidly growing data sets and supporting the compute power to process it all should sit in the realm of cloud computing with infrastructure being rapidly provisioned upon demand.

However, there are a number of barriers. Many organisations automatically believe that cloud computing can help them to process or store this data – but this may not always be the case. Primarily, transferring large volumes of data to the cloud frequently makes this undertaking unfeasible because it simply takes too long to move it via the internet.

At present, it may make more sense to retain and analyse data in-house, but in future, cloud could reduce costs and increase speed of data handling. Cloud today does offer rapid deployment and for back up and archiving can offer an agile and relatively low cost solution.

The key point is to have an ICT strategy that takes a holistic view of Big Data and takes in cloud computing as a part of that. The use of cloud computing as part of an overall programme of IT to underpin this need is critical, probably combining private and public clouds, different types of compute and storage and possibly new cloud applications for processing, analysing and unleashing the potential of Big Data.

Big Data touches many different sectors and represents both an opportunity and a threat. Once properly processed, it has the potential to help change organisations and business, for example helping identify new customer trends and tackle hereditary diseases.

However, making the most of it may not be a straightforward process and organisations should focus on nurturing the next generation of ‘data scientists’ to help deal with this sea change as well as seeking out partners and likeminded companies which can support and offer advice on how best to tackle the many faces of big data.

Underneath this, organisations should ensure that the ICT strategy fully caters for the greater complexity of information management, utilising skilled partners to augment internal skills and ensure that the accelerating growth of Big Data can be fully supported.