Using Vector Databases for the Selection of Related Occupations: An Empirical Evaluation Using O*NET

Abstract

Career planning agencies and other organizations can help workers if they are able to effectively identify related occupations that are relevant to the task at hand. Occupational knowledge bases such as O*NET and ESCO represent mature attempts to categorize occupations and describe them in detail so that they can be used to search for related occupations. Vector databases offer an opportunity to find related occupations based on large pre-trained word and sentence embeddings and their associated retrieval algorithms for similarity search. This paper reports a systematic empirical evaluation of the possibilities of using vector databases for related occupation retrieval using different document structures, embeddings, and retrieval configurations for two popular open source vector databases, and using the O*NET curated database. The objective was to understand the extent to which curated relations capture all the meaningful relations in a context of retrieval. The results show that, independent of the database used, distance metrics, sentence embeddings, and the selection of text fragments are all significant in the overall retrieval performance when comparing with curated relations, but they also retrieve other relevant occupations based on text similarity. Further, the precision is high for smaller cutoffs in the results list, which is especially important for settings in which vector database retrieval is set up as part of a Retrieval Augmented Generation (RAG) pattern. The inspection of highly ranked retrieved related occupations not explicit in the curated database reveals that text similarity captures the taxonomical grouping of some occupations in some cases, but also other cross-cuts different aspects that are distinct from the hierarchical organization of the database in most of the cases. This suggests that text retrieval should be combined with querying explicit relations in practical applications.