1pp-1.7b-asst-rawanneal-seed1234-base

One Persona Pretraining (1PP) experiment model: 1.66B parameters, pretraining condition original documents, then rewritten conversations, assistant-turn loss in the cooldown.

Mid-training variant: the seed-1234 replicate of the production raw run 1pp-1.7b-raw-seed1234-base up to step 28,600 (the last step at peak learning rate), then its cooldown (the last 3,177 of 31,777 steps, the original linear decay) on the 1PP conversations instead of the documents. The corpora are step-aligned, so each cooldown step holds the conversations rewritten from exactly the documents the raw run trained on at that step. It answers how much of the 1PP effect a conversation-only cooldown recovers; compare with the full 1PP run 1pp-1.7b-asst-seed1234-base.

Seed replicate of 1pp-1.7b-asst-rawanneal-base: the same recipe, with --seed 1234 instead of 28 for the pretraining parameter initialization of the raw run it branches from. The original packed document corpus was lost before that raw run, so its first 28,600 steps train on a rebuild of the original documents: 31,717 of 31,777 steps reproduce the original document and token counts exactly; 60 steps are each one document short, 139,055 fewer tokens of 66.19B in total; 36 documents were accepted on a long exact prefix match whose remaining difference has no known cause. This model therefore differs from the seed-28 mid-training variant in the seed and in these documents; the cooldown conversations are identical. Because of the rebuild, 7 of the 3,177 cooldown steps (29,033, 29,926, 30,347, 30,543, 30,683, 30,980, 31,557) hold one conversation whose source document is missing from the same step of the seed-42 raw run.

Part of a 3 × 3 study: three sizes (0.5B, 1B, 1.7B) × three pretraining conditions on the same 47.8M source documents in the same order (original documents; rewritten conversations with loss on assistant turns; rewritten conversations with loss on user and assistant turns). Every run saw the identical batch sequence, so the conditions differ only in the document text and the loss mask (this model's rebuilt documents are the exception described above). Models are grouped in the 1pp collection.

Architecture

Llama-style decoder, 24 layers, hidden 2,048, FFN 8,192 (SwiGLU), attention heads / KV heads 16 / 4 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus <|pad|>; <|endoftext|> is the end-of-document token.

Pretraining

Data: the original DCLM-edu documents for the first 28,600 steps (loss on all document tokens and on <|endoftext|>), then the 1PP conversations rewritten from those documents; loss only on assistant turns (no loss on user turns or <|endoftext|>) for the last 3,177 steps (the cooldown). One pass over 47.8M documents (66.2B tokens of original documents; 63.0B tokens as conversations), 31,777 steps at global batch 512 × 4,096 tokens, cross-document attention masking, best-fit packing with step-aligned document assignment. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for embeddings and norms, warmup 2,000 steps, constant, linear decay over the last 10% to 1/100, weight decay 0.1, bf16.

Validation loss (per token, 2,433 held-out documents, final checkpoint):

assistant text	user text	document text
1.513	4.916	2.978

Chat format

ChatML without a system turn (the models never saw one):

<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n

The bundled chat_template renders exactly this. Generation stops at <|im_end|> (id 2) or <|endoftext|> (id 0); both are listed in eos_token_id. This is a base model; the conversation conditions produce chat-formatted text, the raw baseline plain text.

Verification

The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:

set	HF loss	Megatron reference	abs. diff
val50m segments [3]	1.5112	1.5126	0.0014
raw_val50m segments [8]	2.9791	2.9779	0.0012

Links

• Training logs: wandb projects 1pp-training and 1pp-sft
• Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.