Semantic search experts discuss the future of online searching

The 2009 Semantic Technology Conference (SemTech) opened in San Jose CA on June 14. According to a conference representative, attendance was up by 25% this year compared to last. 

On Wednesday this week, the opening keynote at SemTech was an executive panel moderated by Guidewire group senior analyst Carla Thompson, on the topic of semantic searching. The panel included representatives from Google, Ask.com, Yahoo, Microsoft, and from startups Hakia and True Knowledge.

These companies have either recently launched or are actively creating intelligent search engines, which try to defy the limitations of keyword searching. Tomasz Imielinski, EVP of Global search and answers for Ask.com, described the benefit of semantic searching as a reduction in the distance between a question and an answer. In doing so, the semantic search bypasses the need for endless scrolling through dozens of results pages.

Semantic technologies, unlike keyword based searches, try to understand the real meaning behind a search phrase. Unlike Google as we know it today, semantic search engines do not search for keywords. The concept of semantic searching is not new; artificial intelligence labs all over the world have been attacking the issue of machine-learning for decades. With the proliferation of information available online, semantic searching and its enabling technologies are now becoming mainstream.

Scott Prevost, GM and director of product at Bing (owned by Microsoft), described semantic searching (or natural language searching) as the effort to “understand unstructured text”. But unlike keyword searching, which is largely based on how frequently a keyword is located in a document, and which will return such documents in ordered sequence, different semantic engines will bring up different results to the exact same online query. The reason is free-flowing language is extremely difficult for a computer to understand and process.

Peter Norvig, Director of research at Google, pointed out that the accuracy and relevance of an answer depends to a large degree on the nature of the question. The answer to the question “what is the population of France” is fairly straightforward for a computer to process. But answering the question “what are the benefits of meditation on human life” will never be that easy; this type of query may result to a variety of articles from experts, opposing opinions, and conflicting statements across search engines.

Who, then, decides which results are the most relevant or the most accurate? This is a topic of active conversation among experts. What seems obvious, however, is that the more popular results should not necessarily be considered the most accurate. Andrew Tomkins, Chief Scientist at Yahoo, believes that the accuracy or relevance of the results should be judged based on their impact on specific task metrics.

Such metrics don’t exist today, however. And creating them is no easy task, either. Regardless, the evolution of online searching will continue; and so will the efforts of established and of newer companies to capture this evolving market. In the near future perhaps all searches will be semantic, and they won’t be limited to the web.

Some experts anticipate virtual assistants, like the one launched this week by Siri, to become experts at understanding natural language. When that happens, finding the best answers might not require keypad typing; we might soon be able to get quick answers to our questions right on our phones.