Exploring How Task Complexity and User Self-Efficacy Shape AI Agent Design for AI-assisted Translation

Abstract

AI Agents promise to reduce prompt engineering burden through autonomous workflows, yet their user experience impact remains under-explored. We conducted a within-subjects study (N=20) comparing a single LLM chat interface with a structured agent pipeline for AI-assisted translation. Task complexity (three levels) and user self-efficacy (AI and translation) were examined as moderating factors. Under the evaluated translation tasks, the single LLM was often preferred to the structured pipeline in satisfaction, workload (NASA-TLX), and efficiency (Inter-Message Interval). Usability differences widened in higher-complexity tasks, while medium complexity suggested potential outcome-level trade-offs. Higher self-efficacy users were also more sensitive to reduced perceived control in the pipeline condition. These findings reflect one structured agent pipeline under specific translation tasks rather than agentic systems broadly. We derive design implications for agent interfaces, including separating user intent from content generation and supporting adaptive levels of automation to preserve user agency.