A multimodal deep learning framework for nutritional estimation and health-oriented recipe analysis

Abstract

Unhealthy diets and poor nutrition are major risk factors for various diseases. Traditional AI-based methods for nutritional estimation often rely solely on image analysis, limiting their accuracy. This paper introduces a multimodal AI approach that combines visual and textual information from recipes to improve the estimation of calories, macronutrients, and ingredient details in support of healthier food choices. Our method integrates image embeddings from convolutional neural networks with text representations from sentence transformers. We evaluate our approach through comparative experiments and validate it using World Health Organisation (WHO) nutritional guidelines, expert assessments, and a user survey involving individuals with varying levels of nutrition knowledge. Results show that our model achieves 88.12% accuracy in predicting recipe healthiness and outperforms state-of-the-art methods in aligning with individual nutrition goals. The best performance was achieved by freezing both CNN and sentence-BERT layers while training only the multimodal fusion component. Expanding the nutritional targets further improved estimation quality, with over 80% agreement between human evaluations and expert judgments. These findings highlight the potential of our approach to bridge the gap between recipes and personalised nutritional insights.