Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Summary
The paper investigates tokenizer-free byte language models using a flat Transformer, showing that longer byte sequences can provide useful extra computation when combined with token-superposition training and hashed embeddings. It demonstrates that Byte models scale effectively under fixed compute, reveal emergent local abstractions, and explore inference-time strategies like speculative decoding to improve efficiency.