“Search is a solved problem” and other annoying fallacies

Abstract

Since Google became a celebrity in the early noughties, many people with the power to control and direct research resources have taken the view that there is no more research to be done on the problem of information retrieval. In reality, there are so many variants of “the search problem” that not all have been catalogued, and few have been solved to the point where we can rely absolutely on the quality of results. Apparently no-one told the Web search companies that the problem was solved as, since that time, they have researched and developed a range of new search facilities and invested heavily in improving their basic products. Google, Yahoo! and Microsoft all maintain search research and development teams much larger than the biggest University computer science departments! Through my involvement with the Funnelback internet and enterprise search company I have worked on many twists on the information retrieval problem which are not modelled in well-known test collections, and not encountered in basic Web search. In my talk I will try to outline some of the issues in trying to apply information retrieval and string processing theory into commercial practice.