---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
datasets:
  - HuggingFaceFW/fineweb-edu
language:
  - en
tags:
  - qwen3
  - positional-encoding
  - interpretability
  - control-model
  - research
library_name: transformers
---

# Qwen3-0.6B — RoPE retained, recalibrated on 1B tokens (control model)

**A research artifact, not a general-purpose model.** This is Qwen3-0.6B recalibrated on 1B tokens
of FineWeb-Edu with RoPE left **active** throughout. It is the matched control for
[`vhallac/qwen3-0.6b-nope-recal-1b`](https://huggingface.co/vhallac/qwen3-0.6b-nope-recal-1b),
which received **identical** training — same corpus, same cached token stream, same step count,
same schedule, same seed — with RoPE removed.

The only difference between the two models is RoPE. That is the entire point of this checkpoint.

Loading is unremarkable — this is a standard Qwen3 model:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "vhallac/qwen3-0.6b-rope-recal-1b"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_id)
```

(Its sibling **does** require a runtime patch. Read that model's card before using it.)

## Why a control model exists

Part of the [`rope-as-scaffold`](https://github.com/vhallac/crockpot-experiments/tree/main/experiments/rope-as-scaffold)
research program, which tested whether **RoPE is a training scaffold that can be discarded after
pretraining**.

The problem this checkpoint solves: earlier comparisons pitted a RoPE-removed model that had
received 1B extra tokens against an untouched base model that had received none. Any difference
confounds *"RoPE was removed"* with *"received extra in-domain training."* Since the recalibration
corpus is also the evaluation domain, that confound is large and it runs in a direction that
flatters the RoPE-removed model.

This model holds the training constant so the comparison isolates RoPE. It changed the program's
conclusions: one prior result reversed sign, another was halved in magnitude, and a persistent
perplexity penalty became visible that had been invisible against the un-adapted baseline.

## Results

Held-out FineWeb-Edu perplexity, fp32, 5M-token frozen eval slice, identical harness for all three:

| model | CE | PPL |
|---|---:|---:|
| `Qwen/Qwen3-0.6B` (base, no extra training) | 3.0819 | 21.80 |
| **this model** (control, RoPE on) | 2.6569 | **14.25** |
| `qwen3-0.6b-nope-recal-1b` (RoPE removed) | 2.8260 | **16.88** |

Reading these correctly matters. Recalibration is worth **0.425 nats** (3.0819 → 2.6569). Removing
RoPE gives **0.169 nats** of that back (2.6569 → 2.8260). The RoPE-removed model's apparent
improvement over base (21.80 → 16.88) is the net of those two, and reporting it against base alone
makes a real regression look like a gain.

## Known limitations

- **Effective context ≈ 2048 tokens** for the *recalibrated* behaviour, despite
  `max_position_embeddings: 40960` inherited from base. Unlike its NoPE sibling this model does not
  collapse beyond 2048 (it inherits base's 32k-context pretraining), but its recalibration only
  covered 2048.
- English-only recalibration corpus (FineWeb-Edu), while Qwen3 is multilingual + code + math.
  Expect domain skew relative to base outside the eval domain.
- Not instruction-tuned beyond whatever base carried; recalibration was plain LM training.
- Single seed, single recipe, 0.6B scale.

## Training

| | |
|---|---|
| base | `Qwen/Qwen3-0.6B` @ `c1899de289a04d12100db370d81485cdf75e47ca` |
| rotary | **active throughout** (no patch applied) |
| corpus | `HuggingFaceFW/fineweb-edu`, `sample-10BT`, streamed in provider order |
| tokens | 1B (1907 steps × 524,288 tokens), context 2048 |
| optimizer | AdamW β=(0.9, 0.95), wd 0.1, grad clip 1.0 |
| LR | 1e-3 peak, 2% warmup, cosine → 10% of peak |
| precision | bf16 |
| seed | 0 |
| hardware | 1× H100 SXM, ~7h |

Trained from the **same cached token file** as its NoPE sibling, not merely the same corpus — the
two models saw the identical token sequence in the identical order. `training_metrics.csv` and
`training_manifest.json` are included for full provenance.

> Note: `training_manifest.json` in this repo carries `"artifact": "qwen3-droped"` — a known
> mislabel from a hardcoded string in the training script. Every functional field is correct for
> *this* model, including `rotary_patch_applied: false` and `output_dir: /workspace/qwen3-rope-recal`.

## Reproduction and full analysis

- Code, specs, and lab notebook: [vhallac/crockpot-experiments](https://github.com/vhallac/crockpot-experiments/tree/main/experiments/rope-as-scaffold)
- The controlled comparison this model enabled: `NOTEBOOK.md`, entry `2026-07-28 — RS-amendment-2-3`
- Pre-registered plan: `RS-amendment-2-3.md`; recipe of record `RS1-spec.md` §11

## License

Apache 2.0, inherited from Qwen3-0.6B.