Knoll: Creating a Knowledge Ecosystem for Large Language Models

Abstract

Large language models are designed to encode general purpose knowledge about the world from Internet data. Yet, a wealth of information falls outside this scope - ranging from personal preferences to organizational policies, from community-specific advice to up-to-date news - that users want models to access but remains unavailable. In this paper, we propose a knowledge ecosystem in which end-users can create, curate, and configure custom knowledge modules that are utilized by language models, such as ChatGPT and Claude. To support this vision, we introduce Knoll, a software infrastructure that allows users to make modules by clipping content from the web or authoring shared documents on Google Docs and GitHub, add modules that others have made, and rely on the system to insert relevant knowledge when interacting with an LLM. We conduct a public deployment of Knoll reaching over 200 users who employed the system for a diverse set of tasks including personalized recommendations, advice-seeking, and writing assistance. Knoll improves the quality of generated responses with participants preferring responses generated with Knoll over baseline GPT-4o responses for 81.5% of the queries when external knowledge is needed.