Knowledge Distillation of Black-Box Large Language Models
Summary
The arXiv paper introduces Proxy-KD, a knowledge distillation approach that uses a proxy model to transfer knowledge from black-box LLMs to smaller models. Experiments show Proxy-KD improves KD from black-box teachers and can exceed traditional white-box KD methods, offering a new pathway for improving compact LLMs.