Adaptive Multimodal Artificial Intelligence with Liquid Neural Network for Edge Computing-Based Augmented Reality

Abstract

Multimodal artificial intelligence (AI) integrated with augmented reality (AR) can advance healthcare education by accommodating diverse input modalities and complex data. This work enhances the PGxKnow platform for pharmacogenomics instruction by combining voice-based navigation, eye-tracking analytics, gesture recognition, and adaptive spatial mapping on the Microsoft HoloLens 2. A domain-adapted Bidirectional Long Short-Term Memory (BiLSTM) model processes specialized pharmacogenomics voice commands, while gaze data are interpreted by an attention-based CNN and refined via a Spatial Transformer Network to highlight contextually relevant visuals. Gesture inputs are recognized by a CNN optimized for pinch, rotate, and zoom actions, allowing direct manipulation of molecular models. Liquid Neural Networks dynamically adjust hologram placement in response to environmental changes, ensuring stability and coherence. A transformer-based fusion model integrates all modalities, preserving temporal context and leveraging self-attention for adaptive, context-aware responses. This multimodal AI-AR integration demonstrates how advanced computing techniques can deliver cohesive, real-time, and domain-specific educational experiences in pharmacogenomics.