Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values

About

Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.

Chengwei Sun, Jiwei Wei, Yujia Wu, Yiming Shi, Shiyuan He, Zeyu Ma, Ning Xie, Yang Yang• 2024

Related benchmarks

TaskDatasetResultRank
Image ClassificationCIFAR-10
Accuracy99.3
875
Image ClassificationDTD
Accuracy80.5
599
Image ClassificationRESISC45
Accuracy95.2
472
Image ClassificationStanfordCars
Accuracy83.3
384
Image ClassificationCIFAR-100
Accuracy93.9
357
Image ClassificationOxfordPets
Accuracy93.4
298
Image ClassificationFGVC Aircraft--
112
Natural Language UnderstandingGLUE
SST-295.4
40
Showing 8 of 8 rows

Other info

Follow for update