Stabilizing Transformer Training by Preventing Attention Entropy Collapse
Paper
•
2303.06296
•
Published
•
1
Dan Busbridge
dbusbridge
AI & ML interests
Deep learning, optimization, self-supervised learning, representation learning, large language modeling, equivariance, geometric deep learning, attention mechanisms, transformers
Organizations
authored
6
papers
about 1 year ago
The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning
Paper
•
2307.10907
•
Published
•
8
Poly-View Contrastive Learning
Paper
•
2403.05490
•
Published
Position Prediction as an Effective Pretraining Strategy
Paper
•
2207.07611
•
Published
•
1
Distillation Scaling Laws
Paper
•
2502.08606
•
Published
•
47
Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
Paper
•
2501.12370
•
Published
•
11
authored
a
paper
over 1 year ago
authored
a
paper
over 2 years ago