Applied Maths Seminar: Ying -- Learning theory for contrastive representation learning: Old is new

Yiming Ying will give a seminar at Wednesday 11th March at 12pm in Carslaw 451.
Afterwards we will go to lunch (all welcome, free for students).  

Title: Learning theory for contrastive representation learning: Old is new 

Abstract: Contrastive representation learning (CRL) has emerged as a core training
paradigm for modern foundation models such as open AI’s CLIP model and other large
vision–language systems.  By learning representations that pull positive pairs
together while pushing negative pairs apart, CRL produces feature embeddings that
transfer effectively to a wide range of downstream tasks.  Despite its empirical
success, the theoretical reasons behind why contrastive learning works remain largely
unexplored.  

In this talk, we show that key theoretical questions about CRL can be addressed using
the classical framework of Statistical Learning Theory (SLT)—illustrating the theme
that old ideas/framework remain powerful for understanding new learning paradigms in
modern machine learning.  We establish both the statistical consistency and
generalisation properties of contrastive objectives.  We show that minimising the
population contrastive risk leads to optimal retrieval performance in the downstream
task and derive calibration-type inequalities that connect upstream contrastive learning
with downstream task performance.  We further analyse generalisation for contrastive
learning, revealing an explicit trade-off between the number of anchor samples and the
number of negative examples.  These results provide a theoretical explanation for the
empirical practices of contrastive learning and highlight how classical
learning-theoretic tools can illuminate the foundations of contemporary representation
learning.