Demystifying Noise Contrastive Estimation
Summary
The article explains Noise Contrastive Estimation (NCE) and its variants Local NCE, Global NCE, and InfoNCE, including their mathematical foundations and practical considerations like partition function estimation and negative sampling. It highlights how these methods reformulate learning p(x|c) as binary or cross-entropy objectives and situates them within contrastive learning, with references to applications in NLP, computer vision, and multimodal models like CLIP.