Checkpoint 14K which removes 44% of the prediction error a plain 4-step run has against the 8-step teacher

README.md

@@ -28,23 +28,24 @@ Right: 4 steps with this LoRA. Same seed throughout; click for full size. More c
 | file                                                     | use it when                                          |
 | -------------------------------------------------------- | ---------------------------------------------------- |
 | `krea2_turbo_4step_rank_64_lora_latest.safetensors`      | **normally** — always the newest accepted checkpoint |
-| `krea2_turbo_4step_rank_64_lora_chk00010000.safetensors` | pin this exact checkpoint                            |
+| `krea2_turbo_4step_rank_64_lora_chk00014000.safetensors` | pin this exact checkpoint                            |
 
 and, beside them, the same files with a `_comfyui` suffix for ComfyUI. Earlier checkpoints
-(`chk00004000`, `chk00005000`, `chk00006000`) are kept in
+(`chk00004000`, `chk00005000`, `chk00006000`, `chk00010000`) are kept in
 [`older_checkpoints/`](older_checkpoints), and their resolution sweeps stay in place, so the
 progression remains visible and comparable.
 
-The numbered files are points on **one continuous run**, not separate experiments — `chk00010000`
-resumed from `chk00006000` rather than restarting.
+The numbered files are points on **one continuous run**, not separate experiments — `chk00014000`
+resumed from `chk00010000`, which resumed from `chk00006000`, rather than restarting.
 
-`chk00010000` removes **30%** of the prediction error a plain 4-step run has against the 8-step
-teacher, where `chk00006000` removed 26% and `chk00005000` 18%. Measured against each other rather
-than against the no-LoRA run, its remaining error is **5% smaller than `chk00006000`'s** and **15%
-smaller than `chk00005000`'s**. What that number is — and what it is not — is in
-[chk00010000 vs chk00006000](#chk00010000-vs-chk00006000).
+`chk00014000` removes **44%** of the prediction error a plain 4-step run has against the 8-step
+teacher, where `chk00010000` removed 40% and `chk00006000` 27% — all measured on the same enlarged
+held-out set (100 prompts across every trained resolution). Measured against each other rather than
+against the no-LoRA run, its remaining error is **6% smaller than `chk00010000`'s** and **23%
+smaller than `chk00006000`'s**. What that number is — and what it is not — is in
+[chk00014000 vs chk00010000](#chk00014000-vs-chk00010000).
 
-> **`_latest` currently points to `chk00010000`** — 10,000 training samples. This line is updated
+> **`_latest` currently points to `chk00014000`** — 14,000 training samples. This line is updated
 > whenever a newer checkpoint is published.
 
 **This is work in progress and better checkpoints may follow.** Training is ongoing, so
@@ -74,7 +75,19 @@ was published. The cause was traced to the optimiser: a constant learning rate w
 lets the adapter keep drifting after it has converged, so its magnitude grows and it over-applies
 its own correction. The same span was retrained from `chk00006000` with a cosine learning-rate decay
 and weight decay, and every checkpoint of that second run improved on the one before it.
-`chk00010000` is its end point — the current end of the process, not simply the longest run so far.
+`chk00010000` was its end point.
+
+`chk00014000` is the next example, and it shows the other half of the same lesson. The run was
+continued from `chk00010000` over the whole pool of teacher trajectories, with two changes: the
+final, texture-deciding call of the schedule was weighted more heavily in the loss, and a running
+**average of the weights** was kept beside the live ones and scored at every evaluation (a
+single checkpoint is one sample of a weight vector that moves from step to step; the average is
+its mean). At 14,000 samples the averaged weights measured a smaller gap to the teacher than any
+checkpoint before them, and a smaller gap than the live weights at the same point — so the averaged
+weights are what `chk00014000` is. The run was allowed to continue well past that point, and the
+adapter's magnitude began to grow again: later checkpoints measured worse on the held-out set and
+further from the teacher in fixed-seed renders, and none of them was published. The number is
+chosen by measurement, not by how far the run went.
 
 ### Timeline of training process
 
@@ -102,11 +115,14 @@ shards is produced. A quiet stretch is shard generation, not abandonment; `_late
 the newest checkpoint that passed review.
 
 The current checkpoint and the one before it mark the last two revisions of the recipe:
-`chk00006000` was the first trained on **int8** teacher trajectories and an int8 base (see
-[Hardware](#hardware)), and `chk00010000` continued it with **cosine learning-rate decay and weight
-decay**. The checkpoints before them were earlier points on the same run under an NF4 teacher; each
-published step has measured a smaller gap to the teacher than the one before it, and the earlier
-files are kept in [`older_checkpoints/`](older_checkpoints) for comparison only.
+`chk00010000` continued `chk00006000` with **cosine learning-rate decay and weight decay**, and
+`chk00014000` continued `chk00010000` over the full pool of teacher trajectories with the final
+call of the schedule **weighted more heavily** and the shipped weights taken as a **running
+average** of the trained ones. The checkpoints before them were earlier points on the same run
+(`chk00006000` was the first trained on **int8** teacher trajectories and an int8 base, see
+[Hardware](#hardware); the ones before it used an NF4 teacher). Each published checkpoint has
+measured a smaller gap to the teacher than the one it replaced, and the earlier files are kept in
+[`older_checkpoints/`](older_checkpoints) for comparison only.
 
 Every file records which checkpoint it actually is in its safetensors metadata (`checkpoint`,
 `training_samples`, and `rolling_pointer` on the `_latest` copies), so a downloaded file can always
@@ -115,54 +131,72 @@ be identified even if renamed.
 ### Checkpoint numbering
 
 ```
-krea2_turbo_4step_rank_64_lora_chk00010000.safetensors
+krea2_turbo_4step_rank_64_lora_chk00014000.safetensors
                                    ^^^^^^^^
 ```
 
-The number is the count of **training samples consumed** — `chk00010000` has seen 10,000. It is not
+The number is the count of **training samples consumed** — `chk00014000` has seen 14,000. It is not
 an epoch count, a step count, or a quality score. Higher-numbered checkpoints are further along the
 same continuous run, resuming from the previous checkpoint rather than restarting.
 
-## chk00010000 vs chk00006000
+## chk00014000 vs chk00010000
 
-**The measure.** Every checkpoint is scored the same way. On 40 held-out prompts that never receive
-a gradient step, the 4-step student's predicted velocity is compared with the 8-step teacher's
-recorded two-step chord at each of the four sigmas the student actually deploys on (see
+**The measure.** Every checkpoint is scored the same way. On a fixed set of held-out prompts that
+never receive a gradient step, the 4-step student's predicted velocity is compared with the 8-step
+teacher's recorded two-step chord at each of the four sigmas the student actually deploys on (see
 [Method](#method)). The result is a relative error — the **gap to the teacher** — and it is
-deterministic: the same weights always score the same number. Krea 2 Turbo run at 4 steps with no
-LoRA scores 4.85e-02; that is the deficit this LoRA exists to close.
+deterministic: the same weights always score the same number. The set behind the figures below is
+the enlarged one: **one hundred held-out prompts across all eleven trained resolutions** (800
+teacher states). Every earlier checkpoint has been re-scored on it, so all rows are comparable;
+the earlier, smaller set (eight prompts at four resolutions) gave different absolute values and is
+no longer used. Krea 2 Turbo run at 4 steps with no LoRA scores 4.69e-02 on this set; that is the
+deficit this LoRA exists to close.
 
 |                                  | gap to the 8-step teacher | of the 4-step deficit closed |
 | -------------------------------- | ------------------------- | ---------------------------- |
-| Krea 2 Turbo — 4 steps, no LoRA | 4.85e-02                  | 0%                           |
-| `chk00005000`                    | 3.98e-02                  | 18%                          |
-| `chk00006000`                    | 3.57e-02                  | 26%                          |
-| **`chk00010000`**                | **3.38e-02**              | **30%**                      |
+| Krea 2 Turbo — 4 steps, no LoRA | 4.69e-02                  | 0%                           |
+| `chk00005000`                    | 3.26e-02                  | 30%                          |
+| `chk00006000`                    | 3.45e-02                  | 27%                          |
+| `chk00010000`                    | 2.81e-02                  | 40%                          |
+| **`chk00014000`**                | **2.64e-02**              | **44%**                      |
 
 Two ways to read the same numbers, with different denominators — they are not meant to be added:
 
-- **Against the no-LoRA run** (the right-hand column): `chk00010000` has removed 30% of the
-  4-step deficit, 4 percentage points more than `chk00006000`'s 26%.
-- **Against each other** (the gap column): `chk00010000`'s remaining error is **5.4% smaller than
-  `chk00006000`'s** (3.38 vs 3.57) and **15% smaller than `chk00005000`'s** (3.38 vs 3.98). The
+- **Against the no-LoRA run** (the right-hand column): `chk00014000` has removed 44% of the
+  4-step deficit, about 4 percentage points more than `chk00010000`'s 40%.
+- **Against each other** (the gap column): `chk00014000`'s remaining error is **6.1% smaller than
+  `chk00010000`'s** (2.64 vs 2.81) and **23% smaller than `chk00006000`'s** (2.64 vs 3.45). The
   same 4 points of deficit are a larger share of a gap that has already shrunk, which is why the
   checkpoint-to-checkpoint figure is the bigger number.
 
+One thing the larger set changed: on it, `chk00005000` scores slightly better than `chk00006000`
+(3.26 vs 3.45), the opposite of the order the small set gave. Eight prompts were not enough to
+separate two neighbouring checkpoints reliably; one hundred across every resolution is what the
+set was enlarged for, and the rows above are the numbers that count from here on.
+
 **Read these as error figures, not as quality fractions.** A plain 4-step run already produces a
 complete, coherent image; what it loses against 8 steps is fine detail and texture, and that loss
-shows up as its predictions sitting about 4.9% away from the teacher's. The LoRA brings that to
-about 3.4%. "30% of the deficit closed" therefore does not mean "30% of the teacher's quality" — it
-means 30% of that 4.9% error is gone. A gap of zero would be the student reproducing the teacher's
+shows up as its predictions sitting about 4.7% away from the teacher's. The LoRA brings that to
+about 2.6%. "44% of the deficit closed" therefore does not mean "44% of the teacher's quality" — it
+means 44% of that 4.7% error is gone. A gap of zero would be the student reproducing the teacher's
 two-step chords exactly, i.e. perfect 8-step equivalence in 4 steps, which no low-rank adapter on
 a frozen base reaches and which the images do not need in order to match.
 
-**Why it is better, not just later.** `chk00006000` → `chk00010000` was trained with a cosine
-learning-rate decay and weight decay (see [How checkpoints get chosen](#how-checkpoints-get-chosen)).
-Under that recipe the gap improved at every checkpoint — 3.56e-02 at 7,000 samples, 3.44e-02 at
-8,000, 3.41e-02 at 9,000, 3.38e-02 at 10,000 — and the adapter's magnitude stopped growing from
-8,000 samples on, which is what the weight decay is there to do. The earlier continuation over the
-same span, without those two changes, finished at 4.06e-02: worse than the checkpoint it started
-from. That run was discarded.
+**Why it is better, not just later.** `chk00010000` → `chk00014000` continued the same run over
+the whole pool of teacher trajectories, with the last call of the 4-step schedule — the one that
+decides fine texture — weighted more heavily in the loss, and with a running average of the
+adapter weights scored beside the live ones at every evaluation (see
+[How checkpoints get chosen](#how-checkpoints-get-chosen)). The averaged weights at 14,000 samples
+are what ships: 2.64e-02, against 2.77e-02 for the live weights at the same point and 2.81e-02 for
+`chk00010000`. The gain is on every one of the four calls, not just the weighted one: against
+`chk00010000`, `chk00014000`'s error is 5.9% lower at the first sigma (composition), 4.2% and 7.0%
+lower at the two middle sigmas, and 6.2% lower at the final, texture-deciding sigma; a full 4-call
+rollout from the teacher's noise also ends 6% nearer the teacher's final latent. In fixed-seed
+renders of the fifteen published prompts `chk00014000` lands nearer the
+8-step images than `chk00010000` does, and its texture energy is closer
+to the teacher's (0.90 of the teacher's on a scale where 1.0 is teacher-like, against 0.86 for
+`chk00010000`) without crossing it. The continuation past 14,000 samples measured worse at every
+subsequent evaluation as the adapter's magnitude grew again; none of it was published.
 
 **What the number does not say.** It is a training-space measure and does not rank images pixel
 for pixel. With a fixed seed the student's trajectory drifts away from the teacher's, so
@@ -342,11 +376,11 @@ ships bf16 simply because it is the one build guaranteed to run everywhere.
 > graph executes fine right up to the sampler. The fp8 _text encoder_ does run on MPS, but only
 > because ComfyUI places it on CPU; the published workflow does not rely on that.
 
-> **`_latest_comfyui` currently points to `chk00010000`** — 10,000 training samples.
+> **`_latest_comfyui` currently points to `chk00014000`** — 14,000 training samples.
 
 The workflow references the **`_latest`** LoRA filename deliberately: as training continues and
 newer checkpoints are published, replacing that one file is enough — the workflow itself never
-needs editing. A numbered copy (`..._chk00010000_comfyui.safetensors`) sits beside it if you would
+needs editing. A numbered copy (`..._chk00014000_comfyui.safetensors`) sits beside it if you would
 rather pin a specific checkpoint in the node.
 
 **Why a separate file.** ComfyUI addresses the transformer by its own layer names, so the adapter
@@ -389,8 +423,8 @@ cherry-picked; a sweep is the complete set or it is not published.
 
 ```
 checkpoint_resolution_sweeps/
-├── chk4000/  chk5000/  chk6000/
-└── chk10000/               one folder per checkpoint, named for the checkpoint it renders
+├── chk4000/  chk5000/  chk6000/  chk10000/
+└── chk14000/               one folder per checkpoint, named for the checkpoint it renders
     ├── 512x512/            one folder per resolution
     │   ├── portrait.jpg
     │   ├── kingfisher.jpg
@@ -496,7 +530,7 @@ targets are permanent, so the teacher's error is baked into every trajectory for
 `chk00006000` and every checkpoint after it are trained this way. Everything up to and including
 `chk00005000` used an NF4 teacher and an NF4 base. On the same held-out set the cleaner targets were
 worth a **13% smaller gap to the teacher** at `chk00006000`; the optimiser changes that followed
-took it further (see [chk00010000 vs chk00006000](#chk00010000-vs-chk00006000)).
+took it further (see [chk00014000 vs chk00010000](#chk00014000-vs-chk00010000)).
 
 There is a second, subtler effect worth stating plainly. Re-scoring the _older_ `chk00005000`
 against int8 targets instead of NF4 ones drops its measured gap-recovery from 28.9% to 17.9% — the