Parameter-Efficient Fine-Tuning of LLMs for Low-Power Sentiment Analysis Applications

Abstract

This paper quantifies how parameter-efficient fine-tuning narrows the performance gap between compact transformers and massive Large Language Models (LLMs) on sentiment analysis while reducing memory, energy, and financial budgets. Experiments use the publicly released BERT-base-uncased checkpoint (12 layers, 110 M parameters), the Stanford Sentiment Treebank-2 benchmark (67,349 movie phrases with binary polarity labels), PyTorch and Hugging Face Transformers 4.41, Weights & Biases for experiment tracking, and an NVIDIA A100 GPU configured for mixed-precision computation. Low-Rank Adaptation (LoRA) adapters (rank 16) are injected into the query/key/value projections, so only 0.54 % of weights are updated. The model was trained for five epochs with AdamW, cosine scheduling, batch size 64, and optional 4-bit post-training quantization. Accuracy and macro-averaged F1 are logged every 25 steps. The LoRA-tuned model achieves 91% accuracy and 0.91 F1 on the SST-2 test set, an improvement of 40 percentage points over the off-the-shelf checkpoint and comparable to GPT-4o-mini (93%), while using fewer than 1∕1500 of its parameters. Validation loss plateaued without overfitting, and 4-bit quantization compressed the model to 27 MB with <0.5 point accuracy loss. Energy profiling shows a 73% reduction in GPU consumption compared with full-parameter fine-tuning. Purpose-built adapters and quantization can unlock high-quality NLP on edge devices. Future work should extend the protocol to multilingual corpora, streaming inference, federated learning, and on-device continual adaptation to preserve accuracy under concept drift and safeguard user privacy.