The information geometry of large language models is shared, learned, and controllable

What structure do language models with different architectures share, and how can we change one behaviour while preserving others? This work connects these questions through the Fisher–Rao geometry of next-token predictions. A model’s predictions define this geometry, whereas activation geometry depends on the choice of internal coordinates.

Across transformer, state-space and recurrent models, output geometry shows stronger agreement than activation geometry and supports semantic-category transfer. Agreement with human word choices improves with predictive accuracy, scale and training, then further with model-only calibration. Token probabilities and read-out geometry predict the spectrum (how strongly different directions affect predictions) and the effective number of directions resolved by damping at a given scale.

Controlled experiments show that geometry follows the language distribution across architectures. Pretraining-corpus statistics predict held-out fact acquisition without recalibration. Randomised experiments show that deeper evidence (patterns requiring a wider context or combinations of cues) delays acquisition across all tested architectures and evidence constructions.

The geometry also specifies local interventions with minimal disturbance, predicts their relative cost and enables reusable control. Updates learned on one set of prompts transfer to unseen prompts while preserving behaviour on reference prompts better than Euclidean control. The same correction improves steering, editing, attribution, dictionary learning and fine-tuning.