MIST: Mutual Information Via Supervised Training
We propose a fully data-driven approach to designing mutual information (MI) estimators. Since any MI estimator is a function of the observed sample from two random variables, we parameterize this function with a neural network (MIST) and train it end-to-end to predict MI values. Training is performed on a large meta-dataset of 625,000 synthetic joint distributions with known ground-truth MI. To handle variable sample sizes and dimensions, we employ a two-dimensional attention scheme ensuring permutation invariance across input samples. To quantify uncertainty, we optimize a quantile regression loss, enabling the estimator to approximate the sampling distribution of MI rather than return a single point estimate. This research program departs from prior work by taking a fully empirical route, trading universal theoretical guarantees for flexibility and efficiency. Empirically, the learned estimators largely outperform classical baselines across sample sizes and dimensions, including on joint distributions unseen during training. The resulting quantile-based intervals are well-calibrated and more reliable than bootstrap-based confidence intervals, while inference is orders of magnitude faster than existing neural baselines. Beyond immediate empirical gains, this framework yields trainable, fully differentiable estimators that can be embedded into larger learning pipelines. Moreover, exploiting MI's invariance to invertible transformations, meta-datasets can be adapted to arbitrary data modalities via normalizing flows, enabling flexible training for diverse target meta-distributions.
692573ed16eb3a9f13103b05
German Gritsai
claimed_verified
2025-11-25T09:34:27.825Z
63e5017caf814a2149ec772d
https://cdn-avatars.huggingface.co/v1/production/uploads/63e5017caf814a2149ec772d/W1Vfd4Na5kTmxeHoChSIB.jpeg
German G
grgera
user
grgera
692573ed16eb3a9f13103b06
Megan Richards
692573ed16eb3a9f13103b07
Maxime Méloux
claimed_verified
2025-11-25T15:53:13.298Z
641c5ae9dad248407395e3ab
/avatars/27c0bcac855c8683017443d6c713f3c8.svg
Maxime M
pie3636
user
pie3636
692573ed16eb3a9f13103b08
Kyunghyun Cho
692573ed16eb3a9f13103b09
Maxime Peyrard
claimed_verified
2025-11-25T12:10:03.931Z
6369394dd322a76e1ea4bdf6
/avatars/a4e5ab0167025fbbfc970d54630ce754.svg
Maxime Peyrard
peyrardm
user
peyrardm
MIST: Mutual Information Via Supervised Training
We propose a fully data-driven approach to designing mutual information (MI) estimators. Since any MI estimator is a function of the observed sample from two random variables, we parameterize this function with a neural network (MIST) and train it end-to-end to predict MI values. Training is performed on a large meta-dataset of 625,000 synthetic joint distributions with known ground-truth MI. To handle variable sample sizes and dimensions, we employ a two-dimensional attention scheme ensuring permutation invariance across input samples. To quantify uncertainty, we optimize a quantile regression loss, enabling the estimator to approximate the sampling distribution of MI rather than return a single point estimate. This research program departs from prior work by taking a fully empirical route, trading universal theoretical guarantees for flexibility and efficiency. Empirically, the learned estimators largely outperform classical baselines across sample sizes and dimensions, including on joint distributions unseen during training. The resulting quantile-based intervals are well-calibrated and more reliable than bootstrap-based confidence intervals, while inference is orders of magnitude faster than existing neural baselines. Beyond immediate empirical gains, this framework yields trainable, fully differentiable estimators that can be embedded into larger learning pipelines. Moreover, exploiting MI's invariance to invertible transformations, meta-datasets can be adapted to arbitrary data modalities via normalizing flows, enabling flexible training for diverse target meta-distributions.
A data-driven neural network approach estimates mutual information using a meta-dataset of synthetic distributions, offering flexibility, efficiency, and uncertainty quantification.
mutual information MI estimator neural network MIST meta-dataset two-dimensional attention permutation invariance quantile regression quantile-based intervals bootstrap-based confidence intervals normalizing flows