Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Summary
The paper investigates how RL adaptation in post-training LLMs distributes across transformer layers. It finds that training a single transformer layer can recover most, and sometimes more than, the gains of full-parameter RL training, with middle layers contributing the most. This layer-contribution pattern is stable across models, RL algorithms, and tasks.