BI, Analytics and Big Data

Used correctly, ‘big data’ refers essentially to the huge amounts of unstructured data that keep arising from various social media or any other source and not only to large volumes of data per se. Very large volumes of data have been dealt with for decades with the help of databases designed to manage them. The term ‘big data’ is a much later description while the earlier generation (just a few years) were brought up on ‘unstructured data’, which is a better description. Another dimension that we need to keep in mind is the fact we are now dealing with data that comes in multiple formats. In many fields, the challenge is to find intelligence (that will guide action or policy) that lies ‘scattered’ across multiple formats of data. It so happens that government agencies have had to face this challenge more than others, looking at videos, texts of all kinds, voice data, just pictures or some visual data and putting them all together to produce the intelligence lying within. Inherently, such activities become a collaborative effort, of which we will have more to say in later posts.

To come back. The challenge is to find how to connect the two to produce the intelligence that serves business goals. Although this aspect hasn’t featured in any of the discussions, this debate actually raises what is a fundamental question: How do we define an information system? Often, even if not expressed, what is meant by an information system is the data, generated principally by transactions and operations, captured in a defined manner. Some data external to the entity’s transactions may also be captured and be part of the system. There is no doubt that some software can detect patterns or the lack of it in such data and that itself could be of great use. The doubt however is over whether such business intelligence can answer the ‘Why’ for any phenomenon that is observed from analyses of data. And this is true irrespective of the nature of the business. The need to explain observed phenomena can be satisfied only by a recourse to data that lies outside any such system, certainly outside data generated by transactions.

Logic suggests that the horizontal cannot completely overcome what is essentially vertical. Understanding the limits of the horizontal and bringing in the vertical will differentiate the really good analytics companies from others. However, it is equally important to pursue what horizontal analyses can throw up, which demands some creative skills. Success will accrue only to those who can help customers understand and analyse their data, internal and external, to produce outcomes that take them forward.

Is Big Data redefining BI? This question was asked about a decade ago, when Analytics, as distinct from Business Intelligence, was emerging into the mainstream. The idea of bringing this into the discussion is to let some history into the field of analysis & analytics, to focus on the goal and the enablers, in a sense asking the enablers whether they are enabling. In the first decade of 21^st century, many established software companies had established distinct leadership positions in the BI segment. The segment grew and all leading IT services companies were vying for a larger share of the implementation business. The tech media went into hyperbole in hailing BI as the key to producing intelligence, just the way Analytics is being marketed today. However, within a short time, Analytics was being introduced as the key to find the competitive edge. Thomas Davenport’s book ‘Competing on Analytics’ perhaps signaled the shift. And analytics began occupying greater mid space and budgets as well. It is important to pause here because it says something about the business of software. BI or Analytics are not just functions but businesses and hence will also obey other aspects or at least be driven by other considerations, which, sometimes, could detract from the principal goal.

Old BI & analytics

This was the charge brought against BI firms, who, many argued, tended to be consumed by technology per se at the expense of users. There were critics openly skeptical of BI firms despite their ‘innovations’ which promised to speed up access to data considerably, but at significant cost – it might at best enable a faster trip to nowhere. Big, old, traditional BI companies are good at producing technologies that enhance the infrastructure of BI —more and faster—but not the actual use of data in ways that lead to greater intelligence. Being big, focused primarily on technology from an engineering perspective, and devoutly sales driven made it difficult for the then major BI companies to develop useful tools for activities that support decision making: data exploration, sense-making, and communication. To meet this challenge, they would have had to shift their focus from technology to the humans who use it and expand their perspective to embrace design, and commit their efforts to what actually works, rather than silly, shiny features that fill their existing products with smoke and mirrors. The question then asked was this: Will they do it in time or will analytics become the exclusive realm of smaller and more agile vendors, leaving traditional BI companies in the back room to maintain the infrastructure (data collection, transformation, cleansing, warehousing, and production reporting)? Some of them tried to incorporate the user angle by ushering in people who focused on user experience. To me, this is the unvarying story of the growth of software – there is a breakthrough only to be consumed by itself leading to a focus on technology per se and ‘forgetting’ the users. Unfortunately, there was the additional aspect of rising cost of using BI software. In a licensing environment, the cost per person did turn out to be very high, forcing companies to start looking for alternatives. This is true of any business where the eyes move away from the goal of helping users, which became compounded with the way IT was organized in enterprises. BI companies were primarily aligned with IT departments. As a result, there were more funds for appropriate for products that handle the back-end BI infrastructure, but not for data exploration, sense-making, and presentation products. This is similar to an operating system which uses a great deal of its memory leaving the user to keep investing in more and more hardware.

Many BI users shifted to tools & techniques rather than full-blown software to get what they wanted: the ability to analyse data to see what it ‘contained’. Until they got swamped by the next ‘big thing’ – analytics. It is very puzzling to note that besides operating systems, databases, application software, none of the other software products or product groups seemed to fit into the analysis of product life cycle, which is a separate investigation.

Big data

If we went by the pattern consistently thrown up by the business of software, the fascination with big data together with analytics will likely go the same way: technology subsuming users and their needs. There is a growing body of tools and techniques that claim to unearth all that is hidden in big data. Much the same is likely happening in the manner in which in several machine learning algorithms are being vigorously pursued for unravelling diverse range of problems. Or the use of natural language processing or programming. We will examine the world and business of ML algorithms and NLP later but for the moment let me leave you with this final thought. For the moment, let me leave you with this.

To grasp the potential impact of Big Data, look to the microscope, says Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management. The microscope, invented four centuries ago, allowed people to see and measure things as never before — at the cellular level. It was a revolution in measurement. Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. The microscope is unwavering in its focus. We just hope that so does big data.

Takeaways

Neglect of user needs and an exclusive obsession with technology leads to deep dissatisfaction

The shift in the business of intelligence is indicative of user dissatisfaction

Much the same could repeat in analytics and big data

Look to the microscope to gauge the potential of big data