Grounding and Enhancing Informativeness and Utility in Dataset Distillation
Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.
6985591b4ad556f294b7eb09
Shaobo Wang
claimed_verified
2026-02-06T18:52:05.358Z
66968099c952e09a4cb29f78
https://cdn-avatars.huggingface.co/v1/production/uploads/66968099c952e09a4cb29f78/n90NI2R3E9_RqCyMjDCQF.webp
Wang
Steven-Shaobo
user
Steven-Shaobo
6985591b4ad556f294b7eb0a
Yantai Yang
6985591b4ad556f294b7eb0b
Guo Chen
6985591b4ad556f294b7eb0c
Peiru Li
6985591b4ad556f294b7eb0d
Kaixin Li
claimed_verified
2026-02-06T18:52:02.746Z
6346be8f7fb9f11870c63998
https://cdn-avatars.huggingface.co/v1/production/uploads/6346be8f7fb9f11870c63998/tFWawSkXL6bv1zgvzFWQd.png
Kaixin Li
likaixin
user
likaixin
6985591b4ad556f294b7eb0e
Yufa Zhou
6985591b4ad556f294b7eb0f
Zhaorun Chen
6985591b4ad556f294b7eb10
Linfeng Zhang
Grounding and Enhancing Informativeness and Utility in Dataset Distillation
Dataset Distillation (DD) seeks to create a compact dataset from a large, real-world dataset. While recent methods often rely on heuristic approaches to balance efficiency and quality, the fundamental relationship between original and synthetic data remains underexplored. This paper revisits knowledge distillation-based dataset distillation within a solid theoretical framework. We introduce the concepts of Informativeness and Utility, capturing crucial information within a sample and essential samples in the training set, respectively. Building on these principles, we define optimal dataset distillation mathematically. We then present InfoUtil, a framework that balances informativeness and utility in synthesizing the distilled dataset. InfoUtil incorporates two key components: (1) game-theoretic informativeness maximization using Shapley Value attribution to extract key information from samples, and (2) principled utility maximization by selecting globally influential samples based on Gradient Norm. These components ensure that the distilled dataset is both informative and utility-optimized. Experiments demonstrate that our method achieves a 6.1\% performance improvement over the previous state-of-the-art approach on ImageNet-1K dataset using ResNet-18.
Dataset distillation method that balances informativeness and utility through game-theoretic and gradient-based optimization techniques, achieving improved performance on ImageNet-1K.
dataset distillation knowledge distillation informativeness utility Shapley Value attribution Gradient Norm game-theoretic optimization