Parameter-Efficient Fine-Tuning (LoRA & DoRa & QLoRA)
Collection
A collection of parameter-efficient fine-tuning experiments for sentiment classification using chat-based instruction tuning • 4 items • Updated
Model Card: llm-course-hw3-dora
This model was fine-tuned as part of Homework 3 in the HSE LLM Course.
It applies a custom DoRA implementation for sentiment classification, using standard causal language modeling training.
The model predicts a sentiment label (negative, neutral, or positive) by generating a short textual output conditioned on the input text.
Model Sources
- Base model: https://huggingface.co/OuteAI/Lite-Oute-1-300M-Instruct
- Dataset: https://huggingface.co/datasets/cardiffnlp/tweet_eval
Training Method
Training follows a standard causal LM setup with frozen backbone weights and trainable DoRA adapters inserted into attention projection layers.
Training Hyperparameters
- PEFT method: DoRA (custom implementation)
- Rank: 8
- DoRA alpha: 16
- Target modules: attention projections (
k_proj,v_proj) - Batch size: 32
- Learning rate: 5e-4
- Optimizer: AdamW
- Precision: FP16 (mixed precision)
- Epochs: 3
Trainable parameters: ~0.14% of total model parameters.
Results
- Macro F1: ~0.5 on the test set
- Downloads last month
- 1
Model tree for sodeniZz/llm-course-hw3-dora
Base model
OuteAI/Lite-Oute-1-300M-Instruct