An Evaluation of Six Artificial Intelligence Tools for Formulary Management

Abstract

Background: Artificial intelligence (AI) has the potential to significantly transform health care, with substantial implications for pharmacy practice. The integration of AI technologies into formulary management and pharmacy and therapeutics (P&T) committee workflows offers opportunities to enhance evidence-based, safe, and timely clinical decision-making by processing large datasets and delivering actionable insights. While professional pharmacy organizations emphasize responsible AI integration, limited evidence exists on AI performance for formulary management tasks. Methods: This pilot study qualitatively evaluated six freely available AI tools (Perplexity, Google Gemini, ChatGPT, Microsoft Copilot, Llama, and Claude) for drug monograph development using three previously completed monographs as reference standards. An independent reviewer assessed each AI tool's performance for content accuracy, completeness, source credibility, and clinical relevance. Results: Initial broad prompts requesting complete monographs yielded responses that were vague, incomplete, or inaccurate across all tools. Refined prompt engineering strategies using targeted, section-specific requests substantially improved output quality, with all AI tools producing more detailed, accurate, and scholarly responses. The tools demonstrated marked variability in source quality and transparency, with some consistently providing citations to peer-reviewed literature while others offered minimal sourcing or relied on less credible references. Literature review capabilities improved substantially when full-text articles were uploaded compared with abstract-only inputs; however, human oversight remains essential for critical evaluation and synthesis of conclusions. Discussion: The findings from this pilot evaluation provided the foundation for the development of institutional implementation considerations at West Virginia University (WVU) Medicine, establishing five core principles for AI use in formulary management, privacy and compliance requirements, and emphasis on human oversight, with clear delineation of permitted and prohibited applications. Conclusion: This pilot evaluation provides health care organizations with qualitative performance insights for responsible AI implementation in formulary management workflows.