Transformers and State-Space Models: Fine-Tuning Techniques for Solving Differential Equations

Abstract

Large language models (LLMs) have recently demonstrated remarkable capabilities in natural language processing, mathematical reasoning, and code generation. However, their potential for solving differential equations—fundamental to applied mathematics, physics, and engineering—remains insufficiently explored. For the first time, we applied LLMs as translators from the textual form of an equation into the textual representation of its analytical solution for a broad class of equations. More precisely, we introduced a benchmark and fine-tuning protocol for differential equation solving with pre-trained LLMs. We curated a dataset of 300,000 differential equations and corresponding solutions to fine-tune T5-small, Phi-4-mini, DeepSeek-R1-Distill-Qwen, and two Mamba variants (130M and 2.8B parameters). Performance was evaluated using BLEU and TeXBLEU metrics. Phi-4-mini achieved the best results, with average BLEU > 0.9 and TeXBLEU > 0.78 across all considered equation classes, which shows the strong generalization abilities of the model. Therefore, this model should be further investigated on a broader class of differential equations and potentially can be used as a part of mathematical agents for solving more complex particular tasks, for example, from physics or engineering. Based on our results, DeepSeek-R1-Distill-Qwen consistently underperformed, while T5 showed strong results for the most frequent equation type but degraded on less common ones. Mamba models achieved the highest TeXBLEU scores despite relatively low BLEU, attributable to their production of lengthy outputs mixing correct expressions with irrelevant ones.