From Tokens to Bytes I: Scaling Hierarchical Networks
Summary
The post introduces hierarchical networks (H-Nets) that learn tokenization end-to-end on raw bytes, arguing this approach can significantly improve efficiency and generalize across domains. It presents initial results showing compute and data efficiency gains versus BPE-based transformers, and outlines plans for larger-scale training and cross-domain applications (biology, vision) in subsequent posts.