GAMMAai Jamba 3B Merge v1

This repository contains a complete Jamba-family research checkpoint from the first GAMMAai fusion batch. It combines the compact instruction profile of ai21labs/AI21-Jamba2-3B with the reasoning-oriented Jamba 3B branch while retaining a single deployable JambaForCausalLM checkpoint.

Checkpoint

  • Architecture: hybrid Mamba/attention JambaForCausalLM
  • Hidden size: 2,560
  • Layers: 28
  • Attention cadence: one attention layer every 14 layers
  • Precision: FP16
  • Declared context window: 262,144 tokens
  • Weight package: one complete model.safetensors file
  • Tokenizer, chat template and configuration: included

The two parent checkpoints share the same Jamba 3B topology and Apache-2.0 license family. The public artifact is structurally complete. The exact coefficient history from the early fusion run was not preserved in this repository, so this release is presented as a research checkpoint rather than as a benchmark-certified replacement for either parent.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "GammaAGI/ai21-jamba-3b-merge-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain why staged verification reduces model-training risk."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))

Intended Use

The checkpoint is intended for research, comparative inference, chromosome extraction and controlled downstream adaptation. Validate task quality, long-context behavior and safety on your own workload before production use.

Limitations

  • No public benchmark superiority claim is made.
  • The 262K configuration value is not a substitute for measured long-context quality at every length.
  • Fusion can inherit limitations and biases from both parent checkpoints.
  • Production deployment requires an independent generation and safety evaluation.

Project

GAMMAai develops verifier-first, staged model evolution through EvoStream, SafeTrain and measurable no-regress gates. This checkpoint is retained as a complete historical model product; temporary adapters, failed experiments and partial weight artifacts are maintained outside the public model catalog.

Downloads last month
23
Safetensors
Model size
3B params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for GammaAGI/ai21-jamba-3b-merge-v1

Finetuned
(7)
this model