Shree Pragada, Founder & CEO, ExeCue, Inc.

Despite what you may have thought, keyword search engines are not going extinct. They will continue to serve a good share of the search market into the foreseeable future. However, next-generation search engines will have keyword engines coexisting with new technologies that are widely expected to be semantic search technologies driven by natural language processing (NLP) or more broadly by knowledge base (KB).

For the average reader, the workings of keyword search engines are fairly obvious. NLP and KB engines are similar in that they both are semantic search techniques and they understand the meaning of a search query. However, they differ in their approaches. NLP engines use natural language processing to understand meaning, while KB engines apply a broader suite of knowledge with NLP being a potential part of it. It is most similar to how we as people understand the meaning of a sentence. We use a variety of knowledge bases, some vertical or domain specific and others general. We might use our knowledge of banking services, alternate fuels, basic math, or conversion rules (like three feet is one yard), and so on. 

Before we start analyzing each of the techniques let us understand what really matters in the business of search. Three things that top the list are (1) the quality or relevance of search results, (2) the learning ability of the search engine over time, and (3) the scope or how broadly it can be applied across multiple domains. Let’s compare the keyword, NLP and KB search techniques across these key metrics.

Keyword Search Engines

A basic keyword search produces numerous results that may not be very relevant or meaningful. Keyword search engines use page popularity as a substitute for meaning, which is critical in ranking search results.

As for applicability, a keyword search platform can be applied almost universally whether it is as meaningful language content or programming code, or simply a bunch of keywords.

With regard to learning, major search engines have over time come to use a variety of sophisticated techniques like searching for word clusters, making spelling suggestions, looking for phrases, and so on to produce very good search results compared with a basic keyword search. Such enhancements can continue to improve the quality of search results, but not significantly more so than the current poor-quality results that are widely agreed to be under 5% relevant.

NLP Engines

NLP-based search engines can certainly improve the quality of search results by using computational linguistics to understand the meaning of the search query and match it to content. It is realistic to expect that the quality of results can go into the 20-25% range.

Although NLP search engines cannot be universally applied like keyword engines, they can cover as much as 50-60% of current search content.

As for learning, NLP search engines will not offer any better search results over time using the same linguistics processors. It’s not like our knowledge of language will be so much better next year and the year after.

Overall, given the leap in quality of results and the broad applicability across language-based content, NLP engines have the capability to produce the next big leap in general Internet search.

Knowledge Base Engines

KB engines can produce the highest-quality and most relevant results within well-defined domains – reaching up to 80-90% in relevance by using a variety of vertical and general knowledge bases, even including NLP.

KB engines will most closely emulate human capabilities and can continually improve on search results as the knowledge base grows through community input.

KB engines are constrained by the lack of readily available general and vertical-focused knowledge bases and the time it takes to develop them.  Lastly, like NLP engines, KB engines do not have any advantage over keyword engines when the search content or user queries are just keywords or unconnected words.

The Future

Look for NLP engines to offer the next big leap in general Internet search given their broad applicability and improved quality of results over keyword search engines. Also, in the near future, KB search engines will offer the highest-quality search results across verticals. With knowledge bases built and shared across more verticals, KB search engines can become the best of the new generation of search tools.

We can be certain that future search engines will be driven not just by keyword search techniques but by a combination of keyword, NLP and KB search techniques. KB engines will be used when knowledge bases are available. NLP engines will kick in when content is language-based but no knowledge base is available. Finally, keyword engines will be used for all other content and serve as the search engines of last resort.

View comments / leave a comment

Posted under Portfolio company guest article