Enterprise Search & Analytics

I started writing this post with the question as to whether we need enterprise search once we have analytics as the preferred discipline to generate intelligence. And if we did, what would it do? Where does it become useful, assuming that there are areas that analytics has still not penetrated enough? Unless, we just went along with the trend of proliferation of software, which has been happening since the beginning of software as a business. That’s a different topic and we will address the anomaly of ERPs functioning with a multitude of other software. Let us stick to the stated topic now.

If you looked at the definitions of enterprise search, it seems no different from what has been known for long as Knowledge management, although many leading firms in the IT consulting and services business do make a distinction. I haven’t been able to find any convincing reason why one shouldn’t merge into the other. As recently as October 20, 2020, an article explains the difference between the two and also argues why it is important now (https://research.aimultiple.com/enterprise-search/). And the reasons mentioned here are (not surprisingly) the same as those used to be mentioned in support of KM – the time taken to locate a document, relevant information, cost of not finding the right information (quoting an IDC study). An August 2020 article says much the same in support of ES as it “can help organizations to assist in locating vital information within no time”. It does overstate the potential of ES software engines when it states that ES “is a large-scale search system that provides the means to search both unstructured and structured data sources with a single query”. And, given the current fashion of artificial intelligence, now we have AI based enterprise search!

In the 1980s and 90s, KM was a flourishing discipline and almost all consulting firms had some ‘product’ or the other to address the challenges of finding the right information when needed, including employees’ knowledge. One part of KM was linked to enterprise documents (in more than one format) and the other part was linked to employees’ tacit knowledge. There was this famous article titled ‘If only we knew we knew’, referring to a situation when companies didn’t even know they had the requisite knowledge within themselves. KM was originally confined to harnessing employees’ knowledge by capturing it in meaningful and accessible formats. Later it expanded to include information relevant to an enterprise but not available within. In my view, what is being touted as ES is about finding the right information within an organization in an easy manner.

Enterprise search as distinct from KM

Let us explore the ‘search’ part a little more to see why perhaps ES did not take off as ‘search’ and became KM. More than 15 years ago, I maintained (in an internal note) that the enterprise search market was virgin territory and that Google will not succeed since this was a completely different terrain altogether. I am not going into the details of search algorithms but we need to recognize a fundamental difference. Any good search engine is a function of the wisdom of crowds, responding to the ways in which people search, layers and layers of links and reverse links. You must have observed that search engines offer options when you enter a search clue or clues and you must have also observed instances when it fails to offer. Therein lies the difference. If there is an absolutely new search clue (even on an ‘old’ topic), the engine will be unable to offer alternatives, but as many more people search in a similar fashion, the engine will begin to recognize and respond. As an experiment, try to think of some really abstruse topics, unlikely to have been thought of by any but a very few people. You will see what I am saying. I have encountered this as an active user of search engines for over two decades. Just like markets, search engines also need volumes. Familiarity does not breed contempt but greater degree of accuracy.

Among the earliest to grasp this was Bill Gates, who used to address global CEOs in the then annual Microsoft Annual CEO Summit. A tech media covering the 2005 Summit remarked that “Among the primary concerns in developing internal-search software is that not every worker should be permitted access to sensitive documents, such as financial data or personnel information. Also, Web searches typically answer queries by ranking pages based on the number of links they have, a system that “doesn’t work in a corporate environment,” he noted. “There just aren’t that many links done.” (https://www.cnet.com/news/gates-information-overload-is-overblown/). The last observation is decisive – will there be ‘sufficient’ links (and reverse links) within an organization to feed the engine? What about large organizations spread over the world? Can they succeed in creating adequate number of links? Intuitively, it does seem possible and I will take it up later.

The success of Google as a search engine (in its early stages) created an opening for a different search with a different purpose – enterprise search, itself preceded by specific domain-focused search engines. If a search engines feeds on what is being searched, it stands a better chance of improving if the language became more and more familiar as would be the case with a domain-focused search engine.

However, information and analysis on specific aspects of a business was already available. Those who belong to an earlier generation in IT will recall that any good accounting software came with vendor and procurement management. A little later, CRM was born as an independent software to be either taken over or supplemented by business intelligence (BI). Strictly speaking, the advent of BI as a separate software should have spelt the death of enterprise search but clearly it hasn’t. Even KM for that matter, which logically should have yielded space to any of the umpteen document management systems. To be sure, such systems would have designed keeping in mind the principles of KM. Or KM should have been assigned a narrower albeit very important focus of continuous capture of employees’ knowledge. I am not sure if that has indeed taken place. It is extremely unlikely. There are products that offer Content Analytics with Enterprise search with natural language capabilities. That means marketing how this helps deal with big data since a large chunk of such data is unlikely to be textual. I can understand incorporating NLP into KM but to merge it with ES is stretching it. More like a refusal to let go. The language used to promote ES is the same as used to promote Analytics, which was earlier Business Intelligence – overcoming information or data in silos.

Meanwhile, let us accept that creating intelligence within an organization is not a matter of enterprise search alone. “Everyone inside an organization – from the CEO to the newly hired information worker – has to be able to find the information they need and then use it to drive smart decisions and take action,” said Kevin Johnson, co-president of the Platforms and Services Division at Microsoft. “Search is just one piece of the solution. Organizations that enable employees to create, access, use and share information efficiently will be more successful at building customer relationships and better at getting great results from their people.” Need for a single point of entry – is this possible? (https://news.microsoft.com/2006/05/17/gates-advises-ceos-software-puts-information-to-work-for-people/)

Search may be just one piece of the enterprise solution but is a market of significant size, part of the unchanging fundamental tendency of the IT/software industry to proliferate. In 2015, Grand View Research estimated that the ES market would reach $5.02 by 2020 with key players being Coveo Corp, IBM, SAP, Oracle, Microsoft, Dassault Systems – see the jump in estimate in one year!

(https://www.researchgate.net/publication/275772285_Enterprise_Search_Market_Share_Growth_To_2020/link/5547015e0cf234bdb21daf56/download). According to another report, “Organizations that hold an authoritative status in the Enterprise Search Software market are AddSearch, Elasticsearch, FishEye, Amazon CloudSearch, Swiftype, SLI Systems, Apache Solr, Algolia, Coveo, Inbenta and SearchSpring” (https://www.aeresearch.net/enterprise-search-software-market-315545). An August 2016 report estimated that “The global enterprise search market is expected to reach USD 8.90 billion by 2024, according to a new study by Grand View Research, Inc. The increasing demand for solutions offering time-saving data search capabilities is expected to be a key factor driving the market growth over the next eight years. The growing need to efficiently supervise large volumes of data in an organization in order to improve the operational efficiency is propelling the adoption of enterprise search solutions”. It added that “Google (Google Search Appliance), HP Autonomy (Verity), SharePoint Search (Acquired by Microsoft), and IBM corporation are the leading players dominating the market. However, Microsoft Corporation has recently stopped the commercialization of standalone products, namely ‘Fast’ and ‘Transfer’. The large numbers of enterprises are using enterprise search platforms ‘Fast’ and ‘Transfer’ in the current scenario”.

(https://www.grandviewresearch.com/press-release/global-enterprise-search-market)

My quest started off trying to understand whether there is any place for enterprise search when the field of analytics is growing in use among all kinds of organizations. It seemed very intuitive that Analytics should displace all else that came before. But that is a separate topic. Topics, like software, also proliferate!

Takeaways

ES has grown different from the way it began

A huge market, its links to analytics haven’t been explored

The limited role of ES per se in enterprise analytics needs to be understood