When a technology trend is the star of its own hour-long BBC TV programme (Horizon, The Age of Big Data, 4 April 2013), it’s safe to assume that it has entered mainstream thinking. The programme showed how using big data was enabling LA cops to predict crime hotspots, the London money markets to map trading patterns and advertisers to find hints of what people might want to buy even before they realise it themselves.

The message was that analysing big data is currently revolutionising the way we make decisions and view the universe. In business it is transforming the way we buy and sell. So if the Horizon programme is right, why do many businesses still see the analysis and use of big data as an IT issue rather than an integral part of strategic thinking in the boardroom?

Talend recently conducted a survey on big data adoption, talking to IT data professionals in 231 businesses about their business objectives, adoption challenges and technologies used. The results indicate that many more businesses are now doing more than just capturing and managing the vast amounts of information about their customers, suppliers and operations.

The survey revealed that 41 per cent are not only already using big data but also have a strategy surrounding it – and 62 per cent of these are enjoying benefits including business process optimisation and improvements to their marketing and sales approach.

As might be expected, 39 per cent said that big data initiatives that are driven by IT departments with a bottom-up approach, tended to be more efficient in collecting and analysing large data sets. However, 48 per cent said that their big data strategies are driven by lines of business or executives, suggesting that senior management teams are now recognising that using big data wisely can mean increased revenue, improved customer service and faster time-to-market.

A survey question about the top three business drivers throws more light on why big data is becoming integral to broader business planning and tactics. A decisive 68 per cent thought that the most important business driver of big data adoption was increasing the accuracy and depth of predictive analysis; in other words, improving the ability of businesses to analyse current and historical data to make future predictions. Some 51 per cent indicated that revenue optimisation was a key driver and 48% cited new revenue generation.

The ability to identify new ways to generate revenue and to optimise revenue streams are probably the most obvious reasons for working with big data. Many companies of all sizes are awash with customer information which can be used to develop laser-sharp targeted marketing, detect fraud to help stem losses or fine-tune inventories based on historical trends.

But perhaps it is the language of analytics which appeals the most to the C-suite – its potential to drive productivity and sharpen business edge is certainly a major selling point to the board.

A different survey, this time carried out by technology research firm the Economist Intelligence Unit, found that top business executives want big data access for more of their employees in the hope that more people using analytics tools will lead to a more productive organisation. In this research, 77 per cent of global executives agreed that more employees should be able to access big data, while 66 per cent said that big data is ‘instrumental in seizing market opportunities.’

Yet all these ‘wants’ can come at a price – processing massive amounts of data can be complex, costly and cumbersome, especially as data volumes and formats continue to grow. To return to the Talend survey, despite the current ‘buzz’ surrounding big data as a business tool, 47 per cent of respondents had no expectations of investing in big data over the next three years. Some 37 per cent admitted that financial restraints were a barrier.

Perhaps it’s not surprising that many boardrooms choose to give a higher priority to other projects. With the prohibitive costs of many big data warehousing and analytics systems making them hard to justify, IT departments are often left to work autonomously without comprehensive strategic direction.

So is there any way that these businesses can avoid getting left behind? From the onset, open source technology has been at the forefront of massive data management. The leading big data platform, Hadoop is first and foremost an open source project managed by the Apache Software Foundation. Yet many wealthy and large corporates, despite their ability to pay the high price tag of more conventional technologies, have led the field in big data advancements, and proven the value of these open source solution

Maybe because, open source provides the most effective way to address the large scale challenges of big data such as scalability and can be deployed and get the job done faster and more accurately, at a fraction of the price of alternative solutions. In this way, open source solutions unlock the potential of big data for a wider range of businesses large and small.

Even beyond Hadoop, many of the latest frameworks and databases have their roots in the open source world, where developers routinely create new approaches to problems that haven’t hit the mainstream. Today, the biggest providers of online communication and data transactions use and contribute to innovative, open development initiatives that our helping to drive the open source industry forward.

CEOs and other senior directors need to know that big data adoption can be simplified and made affordable in this way – and it will might well earn IT departments much needed kudos if they introduce them to this knowledge. Only then will they be able to make big data analysis a central business tool.