In this paper, we present a pipeline for image extraction from historical documents using foundation models, and evaluate text-image prompts and their effectiveness on humanities datasets of varying levels of complexity. The motivation for this approach stems from the high interest of historians in visual elements printed alongside historical texts on the one hand, and from the relative lack of well-annotated datasets within the humanities when compared to other domains. We propose a sequential approach that relies on GroundDINO and Meta’s Segment-Anything-Model (SAM) to retrieve a significant portion of visual data from historical documents that can then be used for downstream development tasks and dataset creation, as well as evaluate the effect of different linguistic prompts on the resulting detections.
CITATION STYLE
El-Hajj, H., & Valleriani, M. (2024). Prompt Me a Dataset: An Investigation of Text-Image Prompting for Historical Image Dataset Creation Using Foundation Models. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 14366, pp. 247–257). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-51026-7_22
Mendeley helps you to discover research relevant for your work.