Utkarsh736/qwen3.5-0.8b-hindi-jacobian-lens

Qwen3.5-0.8B Hindi Jacobian Lens

A Jacobian lens fitted for Qwen/Qwen3.5-0.8B on Hindi Wikipedia text.

⚠️ Status: Experimental / Unvalidated. This is a work-in-progress fit. A sanity check produced degenerate top-token readouts. See Sanity Check Results below. This lens should be treated as a pipeline test, not a validated interpretability tool.

What is a Jacobian Lens?

A Jacobian lens is a linear map that transports intermediate-layer residual stream representations into the final layer's basis. Unlike the Logit Lens, which assumes intermediate representations already live in the final layer's geometric space, the Jacobian lens explicitly corrects for the representational shift across layers. It is used to inspect what a model "intends" to output at each layer of its forward pass.

For more background, see the Transformer Circuits article and the reference implementation:

• Transformer Circuits: Workspace (2026)
• anthropics/jacobian-lens

Model Details

Property	Value
Base model	Qwen/Qwen3.5-0.8B
Architecture	Causal decoder-only transformer
Layers	24
d_model	1024
Fitting corpus	HuggingFaceFW/finewiki, hi config
Fitting language	Hindi
Prompt count	150
Max sequence length	128
dim_batch	64
Precision	float32 during fit; saved artifact ~48 MB
Fitting hardware	RTX Pro 6000 Blackwell 96 GB (molab)
Fitting library	transformer_lens (TransformerBridge + JacobianLens)

Files

File	Description
qwen3.5_0.8b_hindi_lens.pt	The fitted Jacobian lens artifact.
hindi_prompts.jsonl	The exact prompt manifest used for fitting (150 Hindi Wikipedia passages, ≥600 characters each). SHA256 recorded in the lens metadata.

Intended Use

This lens is intended for mechanistic interpretability research on Qwen/Qwen3.5-0.8B, specifically for probing how the model's internal representations evolve across layers when processing Hindi text.

It was fitted as part of an experiment on multilingual bias — the eventual goal is to apply this lens to Hindi prompts from the BharatBBQ benchmark and observe whether stereotype-congruent tokens surface in the model's internal workspace at a higher rate than counter-stereotype tokens.

Limitations

• Model-bound: This lens is fitted for the exact checkpoint Qwen/Qwen3.5-0.8B. It is mathematically invalid for any other model, even if the architecture is identical.
• Language-bound: The lens was fitted on Hindi Wikipedia text. Applying it to English, Bengali, or any other language will produce noisy or misleading readouts.
• Linear approximation: The Jacobian lens is a linear approximation of a nonlinear system. It cannot capture sharp, discontinuous decisions. Readouts should be treated as a noisy but informative signal, not ground truth.
• Small fit: 150 prompts is at the low end of what is recommended. Published lenses typically use up to 1,000 prompts. Quality may improve with a larger fit.
• Unvalidated: See sanity check below.

Sanity Check Results

After fitting, the lens was applied to the Hindi prompt:

भारत की राजधानी है text

("The capital of India is")

Expected: Later layers should converge toward tokens related to Delhi / capital / "नई" (New) / "दिल्ली" (Delhi).

Observed: Top tokens at layers 12, 18, and 23 were punctuation and fragments:

Layer	Top tokens
12	?, ?, ै, ??, !
18	?, ?, ?!, ??, ??
23	?, \n, ?, ल, र

This is degenerate output. The final layer (layer 23) is always read with the identity transport, so its lens readout should exactly match the model's actual output distribution. Seeing punctuation there strongly suggests either a model boot/tokenization issue or that the fit itself is unstable.

Next steps: Refit with a more conservative dim_batch (16 or 32), a larger prompt set (200–300), and verify the model boot configuration against the TransformerBridge requirements.

Usage

from transformer_lens import TransformerBridge
from transformer_lens.tools.analysis import JacobianLens
import torch

model = TransformerBridge.boot_transformers(
    "Qwen/Qwen3.5-0.8B",
    device="cuda",
    dtype=torch.float32,
)

lens = JacobianLens.load("qwen3.5_0.8b_hindi_lens.pt")

result = lens.readout(
    model,
    "भारत की राजधानी है",
    layers=[12, 18, 23],
    positions=[-1],
    top_k=5,
)
top = result.top_tokens(model.tokenizer, k=5)
print(top)

Citation

If you use this lens, please cite the Jacobian lens methodology:

@article{transformer-circuits-workspace-2026,
  title={Workspace},
  author={Transformer Circuits},
  year={2026},
  url={https://transformer-circuits.pub/2026/workspace/index.html}
}

Provenance

• Fitting corpus: HuggingFaceFW/finewiki, config hi, split train, streamed.

• Prompt manifest SHA256: Recorded in the lens metadata under prompt_manifest_sha256.

• Fitting date: 2026-09-18

• Fitting notebook: molab (marimo), RTX Pro 6000 Blackwell.

---

This model card was written as part of an ongoing multilingual bias interpretability experiment. The lens will be superseded by a refit with corrected parameters.