Talk, Listen, Connect: How Humans and AI Evaluate Empathy in Responses to Emotionally Charged Narratives

Abstract

Social interactions promote well-being, yet barriers like geographic distance, time limitations, and mental health conditions can limit face-to-face interactions. Emotionally responsive AI systems, such as chatbots, offer new opportunities for social and emotional support, but raise critical questions about how empathy is perceived and experienced in human-AI interactions. This study examines how empathy is evaluated in AI-generated versus human responses and what factors evoke empathic expression in both. Using personal narratives, we explored how persona attributes (e.g., gender, empathic traits, shared experiences) and story qualities affect empathy ratings. We compared responses from standard and fine-tuned AI models with human judgments. Results show that while humans are highly sensitive to emotional vividness and shared experience, AI-generated responses are less influenced by these cues and often lack nuance in empathic expression. These findings highlight key challenges in designing emotionally intelligent systems that respond meaningfully across diverse users and contexts. Our work advances the evaluation of AI-mediated empathy and informs the design of ethically aware, emotionally responsive systems for use in social connection and mental health support.