Instructions to use shreyansh12183/vigyan-3b-1.58bit-qat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use shreyansh12183/vigyan-3b-1.58bit-qat with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="shreyansh12183/vigyan-3b-1.58bit-qat") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("shreyansh12183/vigyan-3b-1.58bit-qat") model = AutoModelForCausalLM.from_pretrained("shreyansh12183/vigyan-3b-1.58bit-qat", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use shreyansh12183/vigyan-3b-1.58bit-qat with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "shreyansh12183/vigyan-3b-1.58bit-qat" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shreyansh12183/vigyan-3b-1.58bit-qat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/shreyansh12183/vigyan-3b-1.58bit-qat
- SGLang
How to use shreyansh12183/vigyan-3b-1.58bit-qat with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "shreyansh12183/vigyan-3b-1.58bit-qat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shreyansh12183/vigyan-3b-1.58bit-qat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "shreyansh12183/vigyan-3b-1.58bit-qat" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "shreyansh12183/vigyan-3b-1.58bit-qat", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use shreyansh12183/vigyan-3b-1.58bit-qat with Docker Model Runner:
docker model run hf.co/shreyansh12183/vigyan-3b-1.58bit-qat
- ๐งฌ Vigyan-3B 1.58-Bit QAT (Experimental Research Preview)
๐งฌ Vigyan-3B 1.58-Bit QAT (Experimental Research Preview)
Vigyan-3B 1.58-Bit QAT is an open-science Experimental Research Preview investigating ultra-low-power, multiplication-free ternary foundation models at the 3-Billion parameter scale.
Trained via a 3,000-step Quantization-Aware Training (QAT) Knowledge Distillation loop using the Straight-Through Estimator (STE), all 252 linear projection matrices are constrained strictly to ternary weights ${-1, 0, +1}$, eliminating heavy floating-point matrix multiplications (GEMM) in favor of pure integer addition and subtraction.
This repository is published openly for non-commercial academic research to provide the global open-weights community with an unredacted, empirical benchmark study of ternary quantization dynamics on dense STEM foundation models.
๐๏ธ Empirical Evaluation: The 500-Question Sovereign Benchmark
To evaluate the mathematical and syntactic preservation of 1.58-bit ternary quantization, the model was tested against a 500-question unseen benchmark evaluated deterministically by an Algorithmic Chief Judge (SymPy CAS + PyVerilog IEEE 1364 AST + Physical 5% Constant Tolerance Band) across two distinct regimes (1,000 total evaluations):
1. Executive Shot-Regime Performance
| Evaluation Regime | Questions Evaluated | Questions Passed | Accuracy (%) | Demonstration Lift |
|---|---|---|---|---|
| 0-Shot Chain-of-Thought (Intrinsic) | 500 | 54 | 10.8% | Baseline |
| 4-Shot In-Context CoT (Exemplars) | 500 | 84 | 16.8% | +6.0% ๐ |
2. 5-Vector Capability Breakdown
| Capability Vector | Total Qs | 0-Shot Accuracy | 4-Shot Accuracy | In-Context Delta (ฮ) | Key Finding |
|---|---|---|---|---|---|
| Standardized Reasoning & STEM | 100 | 40.0% | 50.0% | +10.0% | Strong retention: Solved 50/100 multi-step GSM8k & physics proofs. |
| PhD Pure Mathematics | 100 | 11.0% | 22.0% | +11.0% | Verifiable derivation: Passed 22/100 SymPy symbolic equivalence proofs. |
| Silicon RTL & EDA | 100 | 2.0% | 9.0% | +7.0% | Exhibits syntax degradation on long Verilog AST structures. |
| Astrophysics & Orbital Mechanics | 100 | 1.0% | 2.0% | +1.0% | Equation derivation degradation on high-dimensional multi-body problems. |
| Biomedical & Quantum Chemistry | 100 | 0.0% | 1.0% | +1.0% | Biochemical syntax breakdown under ternary constraints. |
๐ฌ Scientific Findings: The 1.58-Bit "Entropy Choke"
This release contributes a vital empirical finding to the low-bit quantization literature:
- Numerical Reasoning vs. Syntactic Entropy:
- Constraining weights to 3 discrete states ($\log_2(3) \approx 1.585$ bits) successfully preserves underlying numerical relationships and algebraic reasoning (50% on Standardized STEM, 22% on PhD Math).
- However, language generation heads exhibit entropy collapse when generating long, strictly structured code syntax (e.g. repetition loops like
let (i = 0; i++) { ... }or falling back to generic Python templates instead of synthesizable Verilog modules).
- The 3-Bit Sweet Spot:
- This empirical data demonstrates that while 1.58-bit ternary is viable for classification and short reasoning, 3-bit quantization (
IQ3_M, 8 discrete centroids) represents the true Pareto frontier for complete multi-domain code generation and complex syntax retention.
- This empirical data demonstrates that while 1.58-bit ternary is viable for classification and short reasoning, 3-bit quantization (
โ๏ธ Architecture & Precision
| Layer Type | Parameters Quantized | Precision | Weight States |
|---|---|---|---|
| Token Embeddings | 151,936 vocab | FP16 (Frozen) | Continuous FP16 |
Attention Projections (q, k, v, o) |
144 Linear Layers | BitNet b1.58 Ternary | ${-1\cdot\gamma, 0, +1\cdot\gamma}$ |
MLP Projections (gate, up, down) |
108 Linear Layers | BitNet b1.58 Ternary | ${-1\cdot\gamma, 0, +1\cdot\gamma}$ |
Normalization Layers (RMSNorm) |
All Layers | FP16 | Continuous FP16 |
| LM Head | Output Head | FP16 | Continuous FP16 |
- $\gamma$ is the tensor-wise absolute mean scale factor: $\gamma = \frac{1}{nm} \sum_{i,j} |W_{ij}|$.
- During inference, projection weights strictly evaluate to ternary states: $W_{\text{ternary}} = \text{clamp}(\text{round}(W / \gamma), -1, 1) \times \gamma$.
๐ป Quickstart: Loading in Python (Transformers)
Because standard transformers loads weights as continuous floats, you can discretize the model to true BitNet ternary precision upon loading using the snippet below:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_ID = "shreyansh12183/vigyan-3b-1.58bit-qat"
print("Loading tokenizer and model...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
torch_dtype=torch.float16,
device_map="auto"
)
# Explicitly discretize projection weights to pure ternary states {-gamma, 0, +gamma}
print("Discretizing weights to strict BitNet b1.58 ternary precision...")
with torch.no_grad():
for name, module in model.named_modules():
for child_name, child in module.named_children():
# Bypass embeddings, norms, and output head
if any(b in child_name.lower() or b in name.lower() for b in ["embed", "norm", "lm_head"]):
continue
if isinstance(child, torch.nn.Linear):
gamma = child.weight.data.abs().mean().clamp(min=1e-5)
child.weight.data = torch.clamp(torch.round(child.weight.data / gamma), -1.0, 1.0) * gamma
print("โ Model is running in pure BitNet b1.58 ternary mode!")
# Test inference
prompt = "<๏ฝSystem๏ฝ>You are Vigyan AI. Derive the answer step-by-step.<๏ฝUser๏ฝ>Calculate the kinetic energy of a 2 kg object moving at 3 m/s.<๏ฝAssistant๏ฝ><think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.2)
print("\nModel Output:")
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
๐ฆ Hardware Footprint & GGUF Format
- GGUF Binary:
vigyan-3b-quad-master-1.58bit-i2_s.gguf - Size on Disk: ~650 MB
- RAM Required: < 1.0 GB RAM
- Target Hardware: Runs on commodity CPUs (Intel, AMD Ryzen, Apple Silicon M-series, Raspberry Pi 5) without requiring any dedicated GPU.
- Execution Engines: Compatible with
bitnet.cppandllama.cppusing 2-bit integer packed block representation (i2_s/IQ2_XXS).
๐ Intended Use & Non-Commercial License
This model is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
Permitted:
- Academic & Scientific Research: Analyzing low-bit representations, loss landscapes, and knowledge distillation dynamics.
- Benchmarking & Evaluation: Testing against emerging low-bit quantization harnesses.
- Educational Exploration: Learning how Straight-Through Estimators function in ternary neural networks.
Prohibited:
- Commercial Deployment: Any commercial deployment, monetization, or fee-charging API integration is strictly prohibited under CC BY-NC 4.0.
๐ Citation
If you use or reference Vigyan-3B 1.58-Bit QAT in your research, please cite:
@misc{vigyan_ai_2026_3b_158bit,
author = {Shreyansh Singh and The Vigyan AI Research Initiative},
title = {Vigyan-3B 1.58-Bit QAT: An Empirical Study of Ternary Quantization-Aware Knowledge Distillation on Dense STEM Foundation Models},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/shreyansh12183/vigyan-3b-1.58bit-qat}},
note = {Experimental Research Preview licensed under CC BY-NC 4.0}
}
Developed with sovereign AI independence by Vigyan AI.
- Downloads last month
- 56
Model tree for shreyansh12183/vigyan-3b-1.58bit-qat
Base model
Qwen/Qwen2.5-3B