GPT-2 Reward Model — Anthropic HH-RLHF

A GPT-2-based reward model trained on the Anthropic HH-RLHF preference dataset using the Bradley–Terry preference-learning objective.

The model takes a response and produces a scalar reward. Given two responses to the same prompt, the reward model is trained to assign a higher reward to the human-preferred response.

Hugging Face repository: philomath-1209/gpt2-reward_model_hh-rlhf

Model Summary

This project implements a reward model from first principles rather than using a pre-built reward-model training framework.

The architecture consists of:

Input response
      ↓
GPT-2 tokenizer
      ↓
GPT-2 Transformer backbone
      ↓
Final non-padding token representation
      ↓
Linear reward head
      ↓
Scalar reward

For a pair of responses:

chosen response   ──→ reward_w
rejected response ──→ reward_l

                 reward_w - reward_l
                         ↓
                  Bradley–Terry
                     objective

The model does not apply a sigmoid to the final scalar reward. The Bradley–Terry objective operates directly on the reward difference.

Training Dataset

The model was trained using:

Dataset: Anthropic/hh-rlhf

The original training split was divided deterministically into training and validation subsets using a fixed seed.

The original test split was kept separate and was not used for model selection.

Split	Examples
Training	144,720
Validation	16,080
Test	8,552

The test set was used only for final evaluation.

Preference Learning

Each training example contains two responses:

chosen — the preferred response
rejected — the non-preferred response

The reward model produces:

$r_{w} = r (x, y_{w})$

for the chosen response and:

$r_{l} = r (x, y_{l})$

for the rejected response.

The Bradley–Terry model defines the probability that the chosen response is preferred as:

$P(y_w \succ y_l) = \sigma(r_w-r_l)$

where:

$\sigma(z)=\frac{1}{1+e^{-z}}$

The training objective is:

$\mathcal{L} = -\log\sigma(r_w-r_l)$

The implementation uses logsigmoid for numerical stability.

Model Architecture

GPT-2 Backbone

The model uses the pretrained GPT-2 Transformer architecture as the representation backbone.

The Transformer produces a contextual representation for every token:

$H = (h_1,h_2,\ldots,h_T)$

where (T) is the sequence length.

Final Non-Padding Token

Because the model needs one representation for the entire response, the hidden state corresponding to the final non-padding token is selected.

For an attention mask (m):

$t_{\text{final}} = \sum_i m_i - 1$

and:

$h_{\text{final}} = H_{t_{\text{final}}}$

Reward Head

The selected representation is passed through a linear layer:

$r = Wh_{\text{final}} + b$

producing one scalar reward.

No sigmoid is applied to this scalar.

Training Configuration

The final full-dataset experiment used:

Parameter	Value
Backbone	GPT-2
Maximum sequence length	512
Batch size	32
Learning rate	(1\times10^{-5})
Weight decay	0.01
Gradient clipping	1.0
Epochs configured	5
Mixed precision	BF16
Scheduler	Linear warmup + decay
Warmup ratio	10%
Random seed	42

AdamW was used with separate parameter groups so that bias and normalization parameters were excluded from weight decay.

Checkpoint Selection

A checkpoint was saved after each completed epoch.

Two checkpoint types were maintained:

latest_model.pt — most recent completed epoch
best_model.pt — checkpoint with the best validation accuracy

The final model used for test evaluation and export was:

Epoch 4 — best validation checkpoint

Epoch 5 did not improve validation accuracy.

Results

Best Validation Checkpoint

Metric	Result
Epoch	4
Validation Loss	0.6311
Validation Accuracy	62.48%

Final Test Evaluation

The untouched test split contained 8,552 examples.

Metric	Result
Test Loss	0.6362
Test Accuracy	62.61%

The validation and test accuracies differ by only 0.13 percentage points.

Training Metrics at Epoch 5

For reference, the final configured epoch produced:

Metric	Result
Train Loss	0.5704
Train Accuracy	68.78%
Validation Loss	0.6311
Validation Accuracy	62.48%

The Epoch-4 checkpoint remained the best checkpoint.

Important Training Note

During the final experiment, an implementation issue caused the scheduler's configured number of training steps and the actual global step count to become inconsistent.

The scheduler was configured for:

4,523 batches × 5 epochs = 22,615 steps

while the recorded final global step reached 27,138.

Consequently, the learning rate reached zero during the final epoch.

This did not change the selected checkpoint: the best model was Epoch 4, before this issue affected the final epoch.

The scheduler/resume implementation has subsequently been identified for correction in the training repository so that future runs explicitly verify that expected and actual optimizer steps agree.

The published model itself is the validated Epoch-4 checkpoint.

Export Verification

The final checkpoint was exported to a Hugging Face Transformers-compatible model.

The exported model contains:

config.json
custom configuration_reward.py
custom modeling_reward.py
model.safetensors
GPT-2 tokenizer files

The exported model was loaded independently and compared against the original PyTorch checkpoint.

Round-trip verification

Examples tested: 20
Maximum reward difference: 0.0000000000

PASS: Exported model matches the original checkpoint.

This verifies that the exported model reproduces the original checkpoint's reward outputs for the tested examples.

Loading the Model

The model can be loaded through the Hugging Face Transformers AutoModel interface.

from transformers import AutoTokenizer, AutoModel

model_id = "philomath-1209/gpt2-reward_model_hh-rlhf"

tokenizer = AutoTokenizer.from_pretrained(
    model_id
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Because this repository contains a custom Transformers model implementation, trust_remote_code=True is required when loading the custom architecture.

Only enable trust_remote_code=True for repositories whose custom code you trust and have inspected.

Scoring a Response

import torch

text = "Your response goes here."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

with torch.no_grad():
    reward = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
    )

print("Reward:", reward.item())

The returned scalar is the model's learned reward for the response.

Comparing Two Responses

import torch

chosen = "Preferred response."
rejected = "Less preferred response."

chosen_inputs = tokenizer(
    chosen,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

rejected_inputs = tokenizer(
    rejected,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

with torch.no_grad():

    chosen_reward = model(
        input_ids=chosen_inputs["input_ids"],
        attention_mask=chosen_inputs["attention_mask"],
    )

    rejected_reward = model(
        input_ids=rejected_inputs["input_ids"],
        attention_mask=rejected_inputs["attention_mask"],
    )

print("Chosen reward:", chosen_reward.item())
print("Rejected reward:", rejected_reward.item())

A higher reward indicates that the model assigns greater preference to that response.

Interpreting the Reward

The absolute magnitude of the reward should not be interpreted as a universal quality score.

The model is trained primarily through relative comparisons:

$r (y_{w}) > r (y_{l})$

The difference between two rewards is therefore more meaningful than the absolute value of either reward.

For example:

Response A → 2.3
Response B → 1.1

indicates that the model prefers A over B according to its learned reward function.

It does not mean that A has some universally defined "quality score" of 2.3.

Limitations

This model should be interpreted as a learned preference model for the training distribution and task on which it was trained.

The reported:

62.61%

is pairwise preference accuracy on the held-out HH-RLHF test set.

It should not be interpreted as:

universal human-preference accuracy
general-purpose response quality
factual correctness
safety correctness
alignment across arbitrary domains
a calibrated absolute quality score

The model can prefer an individual rejected response over its corresponding chosen response. Pairwise accuracy below 100% is expected for a learned reward model.

The model was trained on a specific preference dataset, and its learned reward function may inherit the biases, limitations, and distribution characteristics of that dataset.

Intended Use

This model is primarily intended for:

studying reward modeling
preference-learning experiments
understanding Bradley–Terry reward objectives
experimenting with RLHF pipelines
ranking candidate responses
educational and research purposes

It is not intended to be treated as a universal evaluator of language-model outputs.

Training Repository

The complete implementation, including:

dataset processing
reward-model architecture
Bradley–Terry loss
training loop
evaluation
checkpointing
resume logic
export utilities
verification scripts
configuration

is available in the corresponding GitHub repository.

Reproducibility

The project uses a fixed random seed and explicit configuration files.

The full workflow is:

Dataset
   ↓
Train / Validation split
   ↓
Reward model
   ↓
Bradley–Terry training
   ↓
Checkpoint selection
   ↓
Test evaluation
   ↓
Hugging Face export
   ↓
Export verification

The training repository contains the exact configuration and commands used during development and final evaluation.

License and Dataset Attribution

This model is built using the Anthropic/hh-rlhf dataset and the GPT-2 architecture.

Users should review the original dataset and GPT-2 licensing and usage terms before redistributing or deploying derivative systems.

Citation

If you use this implementation or model in your work, please refer to the associated GitHub repository and Hugging Face model repository.

Hugging Face: philomath-1209/gpt2-reward_model_hh-rlhf

Downloads last month: 24

Safetensors

Model size

0.1B params

Tensor type

F32

Video Preview

Reinforcement Learning

Model tree for philomath-1209/gpt2-reward_model_hh-rlhf

Base model

openai-community/gpt2

Finetuned

(2,354)

this model

philomath-1209
/

gpt2-reward_model_hh-rlhf