Commit History
-
B3a-pivot: real Kimi K2.6 weights demo - eviction policy validated on actual MLA attention distribution
GENOMA LABS / researchLoaded all 7 attention weights (q_a_proj, q_b_proj, q_a_layernorm, kv_a_proj_with_mqa, kv_b_proj, kv_a_layernorm, o_proj) of Kimi K2.6 layer 0 from the published checkpoint, instantiated a transformers DeepseekV3Attention module with the canonical Kimi K2.6 config (61L 64H qk_dim=192 v_dim=128 kv_lora_rank=512), ran one full-prefix forward over 256 synthetic tokens on TITAN RTX in 0.07s, captured the real attention distribution, and applied the H2O heavy-hitter eviction policy. Result: H2O policy keeps heavy-hitters scoring 3.66x higher on average than the tokens it evicts (kept-mean 141.08 vs evicted-mean 38.58). Score range on real Kimi attention: 0.336 to 350.870 (1000x spread, std ~= mean = 64). This is concrete validation that the eviction policy makes sensible decisions on real frontier-MoE MLA attention distributions, not just on synthetic / random data as in the multi-step validation notebook. Adds: - scripts/kimi_layer_eviction_demo.py single-forward Kimi demo - noteboo…aac6c2e