CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction
  Tuning for BabyLMs
This work investigates whether small-scale LMs can benefit from instruction
tuning. We compare conversational and question-answering instruction tuning
datasets, applied either in a merged or sequential curriculum, using
decoder-only models with 100M and 140M parameters. Evaluation spans both
fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and
psycholinguistic correlation) settings. Results show that instruction tuning
yields small but consistent gains in fine-tuning scenarios, with sequential
curricula outperforming merged data; however, improvements do not consistently
transfer to zero-shot tasks, suggesting a trade-off between interaction-focused
adaptation and broad linguistic generalization. These results highlight both
the potential and the constraints of adapting human-inspired learning
strategies to low-resource LMs, and point toward hybrid, curriculum-based
approaches for enhancing generalization under ecological training limits.
690332aa72739622ee92a995
Luca Capone
claimed_verified
2025-11-03T20:57:01.916Z
6568b9ff2f7ea4b5acfbd4a5
/avatars/1b2a0745ab079dd1cfd2abb732be2096.svg
luca capone
lucacpncoding
user
lucacpncoding
690332aa72739622ee92a996
Alessandro Bondielli
claimed_verified
2025-10-31T14:27:52.295Z
623d834d19b08016c234a211
https://cdn-avatars.huggingface.co/v1/production/uploads/1668524353147-623d834d19b08016c234a211.jpeg
Alessandro Bondielli
alessandrobondielli
user
alessandrobondielli
690332aa72739622ee92a997
Alessandro Lenci
CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction
  Tuning for BabyLMs
This work investigates whether small-scale LMs can benefit from instruction
tuning. We compare conversational and question-answering instruction tuning
datasets, applied either in a merged or sequential curriculum, using
decoder-only models with 100M and 140M parameters. Evaluation spans both
fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and
psycholinguistic correlation) settings. Results show that instruction tuning
yields small but consistent gains in fine-tuning scenarios, with sequential
curricula outperforming merged data; however, improvements do not consistently
transfer to zero-shot tasks, suggesting a trade-off between interaction-focused
adaptation and broad linguistic generalization. These results highlight both
the potential and the constraints of adapting human-inspired learning
strategies to low-resource LMs, and point toward hybrid, curriculum-based
approaches for enhancing generalization under ecological training limits.
Instruction tuning improves fine-tuning performance for small-scale language models but does not consistently enhance zero-shot capabilities, indicating a trade-off between adaptation and generalization.
instruction tuning conversational question-answering decoder-only models fine-tuning zero-shot SuperGLUE BLiMP EWoK WUGs entity tracking psycholinguistic correlation sequential curricula merged data hybrid curriculum-based approaches