What thousands of conversations look like inside a language model
Summary
This field note maps thousands of conversations inside Qwen3-1.7B to visualize how token states evolve in the model's residual stream. It compares English, Chinese, German, Croatian, and Python code, revealing language-dependent geometry and showing that most variation in token states lies in a shared, multilingual structure, with different languages occupying nearby regions in a Global PCA frame. The piece discusses methods (Global PCA vs Conversation PCA), key metrics (e.g., 99.3% plane fit for English; 80.1% variation captured by Conversation PCA), and the limits of what a 3D projection can reveal.