COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available https://github.com/mts-ai/COMPOT{here}.
699555668d17d1ee8c10ecdd
Denis Makhov
claimed_verified
2026-02-18T09:08:06.033Z
67bc9a9ed9a970de04255711
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_WzRhjLJnkf_ALkP4BuM_.png
Denis Makhov
dennismak1994
user
dennismak1994
699555668d17d1ee8c10ecde
Dmitriy Shopkhoev
claimed_verified
2026-02-18T09:08:09.251Z
66465dfa508db0bde50d95f2
/avatars/8b4a583dc0f3cab0f1cd9a1be3daa01b.svg
Dmitry Shopkhoev
dimitriish
user
dimitriish
699555668d17d1ee8c10ecdf
Magauiya Zhussip
699555668d17d1ee8c10ece0
Ammar Ali
claimed_verified
2026-02-18T09:08:07.754Z
6166db59f78a267701a78c2a
/avatars/8784efc36f67719e9455b1f081340ed9.svg
Ammar Ali
ammarali32
user
ammarali32
699555668d17d1ee8c10ece1
Baher Mohammad
699555668d17d1ee8c10ece2
Stamatios Lefkimmiatis
COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available https://github.com/mts-ai/COMPOT{here}.
COMPOT is a training-free compression framework for Transformer models that uses sparse dictionary learning with orthogonal dictionaries and closed-form updates to achieve better quality-compression trade-offs than traditional low-rank methods.
Transformer models truncated singular value decomposition sparse dictionary learning matrix Procrustes orthogonalization weight factorization orthogonal dictionaries sparse coding layer-wise compression dynamic allocation strategy post-training quantization