Task-driven framework using large models for digital pathology

Abstract

Microscopy is an indispensable tool for collecting biomedical information in pathological diagnosis, but manual annotation, measurement and interpretation are labor-intensive and costly. Here, we propose a task-driven framework powered by large models that excel in visual analysis and real-time control, paving the way for the next generation of microscopes. We achieve proof-of-concept success on clinical tasks, specifically in adaptive analysis of H&E-stained liver tissue slides. This work demonstrates the advanced capabilities for future digital pathology, setting a new standard for intelligent, efficient, and real-time analysis in clinical applications.