Residual Feature Enhancement for Large Language Models: Methodology and Applications

Abstract

Large language models (LLMs) have achieved remarkable progress in natural language processing, yet their ability to perform complex logical reasoning remains limited. Existing approaches such as prompting, retrieval-augmented generation, and parameter-efficient fine-tuning (PEFT) provide partial improvements but often suffer from prompt sensitivity, semantic compression, or additional computational cost. In this work, we propose a Residual Feature Enhancement (RFE) module, a lightweight architectural component designed to strengthen reasoning ability while maintaining computational efficiency. RFE integrates a dimension-preserving linear transformation, SwiGLU nonlinear activation, and residual connections to enrich attention outputs without altering the backbone structure. We conducted comprehensive experiments across six reasoning and comprehension benchmarks—LogiQA, ReClor, LogiQA2.0, GSM8K, HellaSwag, and MBPP—covering deductive reasoning, standardized test comprehension, commonsense inference, and program synthesis. Results demonstrate that ChatGLM4-9B augmented with RFE consistently achieves superior performance compared with both adapter-based methods and larger-scale baselines. Specifically, ChatGLM4-9B+ RFE attains 68.20% on LogiQA, 82.00% on ReClor, 79.74% on LogiQA 2.0, 95.68% on GSM8K, 72.42% on HellaSwag, and 56.82% on MBPP, all of which surpass the Adapter mechanism (67.68%, 81.15%, 78.52%, 94.47%, 66.85%, 55.02%) and show clear advantages over open-source baselines such as Qwen1.5-MoE-A2.7B, Llama3.1-8B, and DeepSeek distilled models. Ablation studies further confirm that removing RFE leads to performance degradation of up to 3.84 percentage points, and convergence analysis shows improved stability and faster training.