“Our BI system can already do all of this”, is a typical refrain from IT management when they first encounter in-memory analytics—that is until they see it in action. Legacy BI systems cannot do ‘all’ that in-memory analytics can, but more important than this is the how, where, and by whom, of in-memory analytics. Its revolutionary impact is the competitive advantage it gives business users through their empowerment.
In-memory analytics is leading a ‘fractionalisation’ revolution. After the revolution, successful organisations will have both in-memory analytics capabilities and the systems perceived as ‘traditional BI’. Each will be essential and valuable in its place and there are examples in history of ‘fractionalisation’ revolutions that give us insight into how this process will unfold.
Have you ever noticed that old industrial plants are multi-story buildings? Replacing human or animal power with mechanical power began with water-wheels. Mills were built to stack as much of the power consuming activities close to the power source as possible. Low friction, load-carrying bearings were an expensive ‘high tech’ component at that time.
One large drive shaft was attached to the power source on a vertical axis, extending the full height of the building. Only one load-carrying bearing was required at the base of the shaft. Power was distributed throughout the plant with belts and pulleys on each floor. When mechanical power sources (steam, internal combustion or electric) were introduced, they replaced the water-wheel but the rest of the factory remained unchanged.
The power source was connected at the base of shaft and the industrial revolution took a step forward. Customers and suppliers could now be located more conveniently to each other rather than being constrained by the availability of water power. Seasons no longer affected plant operations, leading to improvements in efficiency and productivity.
Workflows were still constrained by the vertical layout dictated by the central power shaft. In the 1920s fractional horsepower electric motors were developed. Power could be delivered precisely where and when needed and controlled by an expert user at the point of use. Machines did not need to be connected to a central power source—the need for the vertical factory layout disappeared.
Factories could be organised to maximise the efficiency of the workflow. Work in progress moved horizontally across the floor rather than vertically. The second wave of the industrial revolution began. Power was democratised, mechanical power become available and cost effective for tasks and organisations that previously could not justify or afford it. The power could be applied precisely where needed under the control of the user.
Did factory managers of the time debate the merits of installing all these little motors when they already had power throughout the factory? Probably, history has proven that this was the correct decision.
Fractional horsepower motors enabled many productivity, efficiency, and quality-of-life enhancing products and services, and obsolesced numerous products and services. Hand crank washers and washboards are a thing of the past. The iceman no longer delivers blocks of ice to cool the ‘icebox’. Refrigerators driven by fractional horsepower motors provide ice and cooling. Computing followed a similar journey from big, central computers to highly distributed computing power.
The process of ‘fractionalising’ computing shows no signs of stopping. Computers are now embedded in devices that we don’t even think about. We no longer cart punch cards to readers. We don’t use terminals that are connected to mainframes. Does anyone remember the now humorous idea of a ‘Wang terminal’ for delivering word processing? There were IT managers that stated ‘why do we need to deploy word processing software when we already have a network of Wang terminals?’
We are on the cusp of the next technology-led ‘fractionalisation’ revolution—Business Intelligence/Analytics can now be fractionalised. The CTO/CIO from the opening paragraph wasn’t pondering the right issue. This issue isn’t about whether legacy BI systems can do what in-memory analytics can, the issue is how, where, and by whom it is accomplished.
Big BI is like the huge electric motor connected to the bottom of the drive shaft at a vertically structured factory. It is a significant improvement over the water wheel, but it relies on a complex, inefficient, rigid network of belts and pulleys to distribute the power and capability throughout the business.
BI systems are critical for the foundational tasks of centralising data and creating fundamental reports. These are ‘big motor’ tasks that precede expert analysis. The challenging issues solved by those analysts require fractional power they can control.
Optimum results are achieved when expert users can independently apply in-memory analytics to solve problems and make decisions. The intellectual capital of organisations is more effectively deployed when individuals have greater control over analytic tools to meet challenges and solve problems. The proliferation of ‘spreadmarts’ in most organisations is proof of the need for more control and flexibility than that provided by big BI.
The process of reshaping analysis provided by BI systems is slow and prone to losing experts’ unique insight and skills. Users need BI as a source of data that they can then use to craft precise analysis that navigates the shifting landscape they face, using a system they control without the delays and translation errors that can occur in the IT development processes. The BI big motor can deliver the information, the in-memory analytics fractional motor enables the user to create the analysis. It is not an either/or decision, both systems are needed.
Organisations need to decide if they want to be in the ice business, delivering blocks of ice, forcing users that need a few cubes of ice to break out the ice-pick and chisel away to get the cup of cubes they need, or do they want to provide refrigeration, enabling user to get the ice they need when they need it, do they want to run a network of Wang terminals or do they want to provide word processing capabilities to the users.
There is still and always will be a need for big electric motors and big BI. They are irreplaceable in some situations. The ‘fractional’ power of in-memory visual analytics is the next wave. The fractional revolution has started, surf the wave or be drowned.