HinoMoto-1B v1 Continued-PT (Continued Pretraining with Bench Mix)

HinoMoto-1B v1 Phase 2 Full に 継続事前学習 (corpus 95% + bench 5%) を 1500 step 適用した派生.

「LoRA Cultural SFT」 では echo collapse (= 同 token 連発) が起きてしまった失敗を回避するために考案. mass corpus を anchor として bench 表現を 自然に内化 させる別アプローチ.

なぜ continued pretraining か

SFT v1 (LoRA r=32, lr=2e-4, 5 epoch) で 1B base + bench v0.6 train (551 items) を学習させたところ、 全 3 軸で base より大幅に悪化:

metric base full SFT v1 (失敗)
family score /12 5.08 3.68 (-27%)
family degenerate 9.1% 31.8% (+250%)
keigo pass 11.4% 7.1% (-38%)
keigo degenerate 5.7% 34.3% (+500%)
silence degenerate 6.0% 46.0%

→ mode collapse: model が prompt template (= 等) を echo loop. 1B base が WSD final で stable point に深く着地 + 小データで mode crush.

代替案: continued pretraining

Training summary

Item Value
Base HinoMoto-1B Phase 2 Full (50,000 step)
Steps 1,500 (continued)
Optimizer AdamW (lr 1e-5, cosine schedule, warmup 100)
Batch 1 × grad_accum 8
Seq len 512
dtype bf16 mixed
Corpus / bench ratio 95% / 5%
Hardware RTX 3090 24GB
Wall-clock ~21 hr (= GPU 共有による slowdown)

Bench results (HinoMoto-Bench-ja v0.6)

metric base full Continued-PT v1 Δ
family score /12 5.08 5.45 +0.37 ✅
family degenerate 9.1% 5.5% -3.6% ✅
keigo pass 11.4% 11.4% 0
keigo degenerate 5.7% 5.7% 0
silence pass 6.0% 2.0% -4.0%
silence degenerate 6.0% 6.0% 0

→ family 軸 改善 (+ score, -degenerate). keigo / silence は base 同等. → SFT v1/v2 と異なり mode collapse 発生せず.

Usage

import torch from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained( "FiShota/hinomoto-1b-continued-pt-v1", dtype=torch.bfloat16, ).to("cuda") tok = AutoTokenizer.from_pretrained("FiShota/hinomoto-1b-continued-pt-v1")

License

CC BY 4.0. Use for any purpose with attribution.

Limitations

  • 1 seed (= seed 0) のみ. variance 未測定
  • bench 軸 で base 同等 (劇的改善ではない). 構造的問題解決 (= echo collapse 回避) が本質
  • 200MB corpus 由来の知識制限 + no safety alignment

Related

Citation

@misc{hinomoto-1b-continued-pt-v1-2026, author = {ryu (FIshota)}, title = {HinoMoto-1B v1 Continued-PT (Continued Pretraining with Bench Mix, mode collapse-free)}, year = {2026}, publisher = {Hugging Face}, }


Generated: 2026-05-20 (HinoMoto/ryu + Claude)

Downloads last month
19
Safetensors
Model size
1.0B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for FiShota/hinomoto-1b-continued-pt-v1

Finetuned
(2)
this model