--- license: apache-2.0 base_model: ornith-ai/Ornith-1.5-35B-A3B-MLX pipeline_tag: text-generation library_name: mlx tags: - mlx - basalt - apple-silicon - quantization - macaq - hadamard - qwen3_5_moe ---

Ornith-1.5-35B-A3B-MACaQ-3bit-MLX

Calibrated MACaQ Low-Precision Checkpoint for Apple Silicon (MLX & Basalt)

bas-alt.vercel.app GitHub Run with Basalt

## Overview `goodmai/Ornith-1.5-35B-A3B-MACaQ-3bit-MLX` is a **MACaQ** (*Model Agnostic Calibrated Quantization*, [arXiv:2611.10649](https://arxiv.org/abs/2611.10649)) checkpoint quantized from [`ornith-ai/Ornith-1.5-35B-A3B-MLX`](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-MLX) for Apple Silicon Metal / MLX via **[Basalt](https://github.com/goodmai/basalt)** ([bas-alt.vercel.app](https://bas-alt.vercel.app)). - **Base Model**: [`ornith-ai/Ornith-1.5-35B-A3B-MLX`](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-MLX) - **Architecture / Family**: `qwen3_5_moe` (`text`) - **Quantization**: `MACaQ 3-bit MoE (group_size=64, 256 experts)` (`macaq_quantization` v1, channel-balance $\alpha = 0.75$, Sylvester block-Hadamard $B = 64$, active-set Hessian coordinate-descent code refinement) - **Total Checkpoint Size**: `15.36 GB` - **Runtime**: Native single-simdgroup barrier-free Metal Walsh-Hadamard (`macaq_single_simd_wht`) + Apple Silicon `quantizedMM` in [Basalt](https://github.com/goodmai/basalt) ## Measured Benchmarks (Apple M4 Pro 24 GB) | Metric | Official BF16 (`ornith-ai/Ornith-1.5-35B-A3B-MLX`) | **`Ornith-1.5-35B-A3B-MACaQ-3bit-MLX`** | |---|---:|---:| | Checkpoint Size | `69.32 GB` | **`15.36 GB` (`4.51×` compression — fits on 24 GB Macs!)** | | Architecture | `qwen3_5_moe` (`40` layers, `256` experts, `A3B` active) | **MACaQ 3-bit (`B=64, α=0.75`) + 4-bit embeddings/head** | | Quantized Modules | — | **`250` dense + `120` stacked 256-expert `MACaQSwitchLinear` (`773,928,363` flips)** | | Integrity (`basalt models inspect`) | `intact=true` | **`intact=true`, `quantizationMode=macaq` (`5356.6s` on M4 Pro)** |
Raw Basalt PerformanceBenchmark & Live Generation Output (M4 Pro 24 GB) ```text Benchmark — model: /private/tmp/Ornith-1.5-35B-A3B-MACaQ-3bit-MLX iterations: 3 tokens: 80 warmup: true Loading model… done Warmup… done # TPS TTFT (s) Time (s) ────────────────────────────────────────── 1 77.35 0.128 1.17 2 78.23 0.135 1.16 3 78.12 0.116 1.14 TPS avg=77.899 min=77.349 max=78.228 σ=0.391 tok/s TTFT avg=0.126 min=0.116 max=0.135 σ=0.008 s GenTime avg=1.157 min=1.143 max=1.166 σ=0.010 s Finish length 3 --- OUTPUT --- Open source models have become increasingly important across software, AI, and various technical domains. Here's an overview of why they matter: ## Access and Affordability Open source models are typically free or low-cost to access, which democratizes technology. This lowers barriers to entry for individuals, startups, and organizations that might not afford expensive proprietary alternatives. It allows wider participation in innovation regardless of budget --- END OUTPUT --- Memory (active): 14623 MB Hardware profile updated: /Users/alexeiboklag/.gem/hardware_profile.json (measured ✓) ```
## Quick Start with Basalt ```bash # Verify 14 shards and MACaQ MoE quantization manifest basalt models inspect goodmai/Ornith-1.5-35B-A3B-MACaQ-3bit-MLX # Run 35B-A3B MoE natively on 24 GB+ Apple Silicon Macs! basalt serve --model goodmai/Ornith-1.5-35B-A3B-MACaQ-3bit-MLX ``` ## Method Reference Quantized using Basalt's native Swift/Metal implementation of Huawei CSL's **MACaQ** algorithm: 1. **Channel Balance**: $p_j = \mathrm{std}(W_{:,j})^{0.75} / \mathrm{rms}(X_{:,j})^{0.25}$ equalizes weight and activation outlier dynamic ranges prior to block rotation. 2. **Sylvester Walsh-Hadamard Frame**: $W' = (W \,\mathrm{diag}(p)^{-1}) H_{64}$, $H' = H_{64}^\top (\mathrm{diag}(p)\, H\, \mathrm{diag}(p)) H_{64}$. 3. **Active-Set Hessian Flip Refinement**: Monotonic greedy $\pm 1$ discrete grid coordinate descent minimizing $\mathrm{tr}(\Delta W\, H'\, \Delta W^\top)$ (with Hessian-weighted E8M0 exponent offset screening $\{+1, 0, -1\}$ for `mxfp4`).