Retrofitting language models to operate over bytes
Summary
Nature's article introduces a method called byteification to retrofit existing subword LLMs to operate over UTF-8 bytes, using a two-stage training procedure to build byte-level models. It details a latent-token LTLM architecture with a local encoder, non-causal boundary predictor using one-byte lookahead for prefill, a global model, and a local decoder, and reports that byteified Bolmo/Blama/Bwen models achieve strong performance close to subword baselines with improved efficiency and open-source tooling.