Representation collapse
Representation collapse is a phenomenon in machine learning and representation learning where a model maps different inputs to the same or very similar embeddings, which means it loses important information about how the data is spread out.[1][2] It is frequently encountered in self-supervised learning, especially within contrastive and non-contrastive frameworks, when training objectives or model architectures do not maintain variance across representations. Collapse results in degenerate solutions characterized by uninformative learned features, significantly impairing downstream task performance. Various techniques have been proposed to mitigate representation collapse, including the use of negative samples, architectural asymmetry, stop-gradient operations, variance regularization, and redundancy reduction objectives, as seen in methods such as SimCLR, BYOL, and VICReg. Comprehending and averting representation collapse is regarded as a fundamental challenge in the advancement of stable and efficient self-supervised learning systems.[3][4][5][6][7][8]
See also
- Representation learning
- Self-supervised learning
- Contrastive learning
- Dimensionality reduction
- Overfitting
- Feature extraction
References
- ↑ Zhao, Wenhao; Zou, Qiran; Shah, Rushi; Liu, Dianbo (2024-11-25), Representation Collapsing Problems in Vector Quantization, arXiv, doi:10.48550/arXiv.2411.16550, arXiv:2411.16550, http://arxiv.org/abs/2411.16550, retrieved 2026-04-27
- ↑ Jing, Li; Vincent, Pascal; LeCun, Yann; Tian, Yuandong (2022-04-23), Understanding Dimensional Collapse in Contrastive Self-supervised Learning, arXiv, doi:10.48550/arXiv.2110.09348, arXiv:2110.09348, http://arxiv.org/abs/2110.09348, retrieved 2026-04-27
- ↑ Chen, Ting; Kornblith, Simon; Norouzi, Mohammad; Hinton, Geoffrey (2020). "A Simple Framework for Contrastive Learning of Visual Representations". Proceedings of the 37th International Conference on Machine Learning.
- ↑ Grill, Jean-Bastien; Strub, Florian; Altché, Florent; Tallec, Corentin (2020). "Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning". Advances in Neural Information Processing Systems.
- ↑ Bardes, Adrien; Ponce, Jean; LeCun, Yann (2022). "VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning". International Conference on Learning Representations.
- ↑ Zbontar, Jure; Jing, Li; Misra, Ishan; LeCun, Yann; Denis, Stéphane (2021). "Barlow Twins: Self-Supervised Learning via Redundancy Reduction". International Conference on Machine Learning.
- ↑ Jing, Li; Tian, Yingli (2021). "Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence.
- ↑ Wang, Xiaolong; Isola, Phillip (2020). "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere". International Conference on Machine Learning.
 |  |
