drex-max-v1 / README.md
NischayDnk's picture
Duplicate from NischayDnk/Drex-Max
753f55d
|
Raw History Blame
5.3 kB
---
base_model: Qwen/Qwen3.8-27B
base_model_relation: finetune
license: other
license_name: nace-ai-open-rail-m
license_link: LICENSE
library_name: pytorch
pipeline_tag: zero-shot-classification
tags:
- decision-model
- system-one
- multimodal
- drex
- qwen3.8
private: true
---
# Drex-Max
A **multimodal decision model** on a fine-tuned [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) hybrid backbone. It reads a state (text, JSON, and optional images) plus typed questions (`choice`, `noul`, ordinal `score`) and returns a probability for every option in one forward pass per question. Nothing is generated.
The 16 full-attention layers are **non-causal**: every token can attend to the whole sequence. The other 48 layers are **Gated DeltaNet linear-attention** blocks and stay left-to-right. A dedicated 255-way readout sits on the last hidden state. Images go through the native vision tower (up to 1.6 MP each) and are visible to every question on the request.
## Architecture
| | |
|---|---|
| Base | Qwen3.8-27B, then decision fine-tune |
| Layers | 64: 48 Gated DeltaNet (linear attention) + 16 full attention (every 4th) |
| Hidden / MLP | 5120 / 17408 |
| Full attention | 24 query heads, 4 KV heads, head dim 256 — **non-causal** |
| Linear attention | causal Gated DeltaNet scan |
| Vision | native tower, several images per request |
| Decision head | `readout.safetensors` `[255 × 5120]`, last-token pooling |
| Options per pass | up to 255 |
| Context | 262,144 tokens |
| Size | 49 GB (bf16) |
Default causal chat inference does not reproduce these scores. Use the bundled loader.
## Inference
```bash
pip install -r requirements.txt
python inference.py --model NischayDnk/Drex-Max
python inference.py --model NischayDnk/Drex-Max --request req.json
```
```python
from pathlib import Path
import sys
from huggingface_hub import snapshot_download
ckpt = Path(snapshot_download("NischayDnk/Drex-Max"))
sys.path.insert(0, str(ckpt / "code"))
from drex_max.model import DecisionModel, answer
model = DecisionModel(ckpt, device="cuda")
row = {
"state": {"ticket": "I was charged twice for order A-104."},
"question": {
"type": "choice",
"instructions": "Which team handles this?",
"criteria": {
"billing": "Charges and refunds",
"shipping": "Delivery",
"other": None,
},
},
# "images": ["receipt.jpg"],
}
print(answer(row["question"], model.predict([row])[0]))
```
Request shape (System One):
```json
{"state": {"ticket": "I was charged twice for order A-104."},
"images": ["receipt.jpg"],
"questions": {"team": {"type": "choice", "instructions": "Which team handles this?",
"criteria": {"billing": "Charges and refunds", "shipping": "Delivery", "other": null}},
"refund": {"type": "noul", "instructions": "Does the customer ask for a refund?"}}}
```
Needs a CUDA GPU with room for ~49 GiB of weights plus activations. Python 3.12+, `transformers==5.17.0`.
## Benchmarks
### Decision Index 0.3 (official kit, public suite)
140,178 scored requests. Area columns are chance-corrected skill × 100. Unanswered requests count as wrong.
| Model | Index | Raw | Knowledge | Language | Retrieval | Tools | Arts |
|---|---|---|---|---|---|---|---|
| **Drex-Max** | **67.59** | **75.48** | **57.2** | **72.2** | **68.8** | **82.3** | **53.3** |
| Drex v2.0 (0.2.1) | 60.78 | 69.70 | 47.7 | 65.9 | 62.9 | 76.0 | 49.1 |
| Jev 1.13.0 (0.2.1) | 57.91 | 68.09 | 51.4 | 62.0 | 55.4 | 75.1 | 37.7 |
Drex v2.0 and Jev are the published 0.2.1 public index. Drex-Max is the official 0.3 public index (ForecastBench retired, GSM8K added). Same scoring rule: chance-corrected skill, area-balanced.
<p><img src="decision_index.png" alt="Decision Index" width="720"></p>
<p><img src="areas.png" alt="Areas" width="720"></p>
### JevBench (231 public items)
Choice = top option, yes/no = noul ≥ 0.5, ordinal = top level.
| Model | Accuracy | Easy | Standard | Hard |
|---|---|---|---|---|
| **Drex-Max** | **92.2%** (213/231) | 100% | 100% | **83.8%** |
| Drex v2.0 | 85.7% (198/231) | 100% | 95.8% | 73.0% |
| Jev 1.13.0 | 86.6% (200/231) | 100% | 98.6% | 73.0% |
### Decision Bench v4
| Model | Accuracy |
|---|---|
| **Drex-Max** | **94.30%** (1010/1071) |
| Drex v2.0 | — |
### Vision (public hold-out)
400 held-out items from the Decision Index vision-board public sets (CV-Bench, BLINK, RealWorldQA, R-Bench-M, MMMU-Pro vision). Skill is chance-corrected accuracy.
| | Accuracy | Skill |
|---|---|---|
| **Drex-Max** | **71.5%** (286/400) | **61.9** |
| Bench | n | Acc | Skill |
|---|---|---|---|
| CV-Bench | 80 | 93.8 | 88.9 |
| RealWorldQA | 80 | 82.5 | 73.2 |
| BLINK | 80 | 67.5 | 48.3 |
| MMMU-Pro vision | 80 | 57.5 | 51.7 |
| R-Bench-M | 80 | 56.3 | 47.5 |
<p><img src="vision.png" alt="Vision" width="720"></p>
## Files
| | |
|---|---|
| `model-*.safetensors`, `config.json` | Qwen3.8-27B backbone after decision fine-tune (bf16) |
| `readout.safetensors` | 255-way decision head |
| `decision_config.json` | option codes, temperature, non-causal full attention |
| `code/drex_max/` | loader: hybrid mask + readout + images |
| `inference.py` | System One scorer |
| `decision_index.png`, `areas.png`, `vision.png` | card plots |