eyes-ml/Muse-Glimmer-30B_jacobian-lens

J-lens for Muse-Glimmer-30B

A standard Jacobian lens (J-lens) for meta-models/Muse-Glimmer-30B — Meta's 52-layer dense multimodal model with its frozen ~1.8B PE-Core-G vision encoder. The lens reads out, at every layer and token position, the vocabulary tokens an activation is poised to be verbalized as (the J-space / global-workspace method — Verbalizable Representations Form a Global Workspace in Language Models, 2026; reference implementation anthropics/jacobian-lens).

Working artifact, published as fitted.

Fit

fit_lens.py eyes-ml/Muse-Glimmer-30B --out_dir out/ --keep_hf_cache \
    --max_seq_len 128 --n_prompts 900 --stop_at_delta 1e-3

Fitter	neuronpedia_utils/jlens/fit_lens.py @ 7724688 — the Neuronpedia adaptation of the official jlens demo/fit_lens.py
Checkpoint	eyes-ml/Muse-Glimmer-30B @ 97e6fe0 — clean duplicate of meta-models/Muse-Glimmer-30B
Corpus	WikiText-103 (Salesforce/wikitext · wikitext-103-raw-v1 · train), streamed and re-chunked to ~2000-char prompts (fitter default)
Prompts	900 · max_seq_len 128 → 111 valid positions per prompt
Target layer	final layer (fitter default)
Other args	fitter defaults at that commit: dim_batch 8, bfloat16, single-GPU device_map=cuda, …
Stop rule	--stop_at_delta 1e-3 (10-prompt smoothed mean, ≥100 prompts) — not reached: the run ended at the --n_prompts cap with smoothed Δmean ≈ 1.5e-3
Final identity_distance (‖J−I‖_F/√d)	≈ 0.2375, flat over the tail
Compute	≈109 s/prompt fit time; ≈27.5 h wall-clock on 1× H100

Toolchain deltas

Two minimal changes for Muse/VLM compatibility, on top of the pinned fitter:

1. transformers pinned to a61d9a5 (5.16.0.dev0) for Muse Glimmer support.
2. fit_lens.py L356: transformers.AutoModelForCausalLM → transformers.AutoModelForImageTextToText, so the fitter accepts transformers-shaped VLMs (jlens.from_hf then auto-detects the text decoder).

Loading

Muse-Glimmer-30B_jacobian_lens.pt is the jlens save format — a plain torch-loadable dict, read exactly as JacobianLens.load does:

import torch
lens = torch.load("Muse-Glimmer-30B_jacobian_lens.pt", map_location="cpu", weights_only=True)
lens.keys()             # dict_keys(['J', 'n_prompts', 'source_layers', 'd_model'])
lens["d_model"]         # 6656
lens["source_layers"]   # [0, 1, ..., 50]

J maps each of the 51 source layers (0–50) to its 6656 × 6656 float16 Jacobian into the final-layer target basis. Glimmer's head applies an output multiplier and a logit softcap, so calibrated probabilities are softmax(20 · tanh(c · W_U · norm(J_ℓ · h_ℓ) / 20)) with c = 0.19611613513818404 (= 1/√26), per the pinned modeling_muse_glimmer.py; the softcap is monotone, so token rankings match plain W_U · norm(J_ℓ · h_ℓ). There is no embedded provenance field in this save format; the Fit table above is the provenance.

Notes

• Anchor. This lens anchors at the final layer (fitter default); anchors differ across published J-lens artifacts (some use the penultimate layer), so mind the anchor before comparing lenses head-to-head.
• Distillation provenance. Glimmer is not a from-scratch pretrain: Meta reports it was "trained … on Muse Spark's outputs using logit distillation", with a data mix similar to the teacher's (announcement). The lens is therefore reading a model whose next-token posterior was inherited from Muse Spark rather than formed directly on organic web text — and there is no base/instruct pair to diff against.
• Text-side fit. The corpus is text-only, so the Jacobians are estimated on language-decoder activations at text positions. Reads at image-token positions are untested.
• Convergence log. The full convergence curve did not survive; the tail (last 14 prompts) ships as convergence_tail.csv.