--- license: apache-2.0 base_model: - trohrbaugh/Qwen3.8-27B-heretic-ara tags: - qwen - qwen3.8 - qwen3.8-27b - gguf - rocm - rocmfpx - activefpx - promptforge - amd - strix-halo - gfx1151 - mtp - heretic - uncensored --- # Qwen3.8-27B-Heretic-ROCmFPX-STRIX ROCmFPX / ActiveFPX build of [`trohrbaugh/Qwen3.8-27B-heretic-ara`](https://huggingface.co/trohrbaugh/Qwen3.8-27B-heretic-ara), targeting AMD Strix Halo / Radeon 8060S (`gfx1151`) and the specialized [CIRU ROCmFPX](https://github.com/ciru-ai/ROCmFPX) + PromptForge inference stack. This repository contains: * a compact ROCmFPX / ActiveFPX GGUF using the CIRU Qwen3.8 v3-style precision allocation; * a model-specific full-W8 PromptForge FFN sidecar; * a model-specific full-W8 PromptForge Gated DeltaNet sidecar. The GGUF can be used on its own with a compatible ROCmFPX-capable llama.cpp runtime. The `.pfs` files are optional serving-time compute views for CIRU PromptForge and trade additional memory for substantially faster supported prompt-processing paths. ## Model provenance ### Original model The architecture and original model weights originate from: [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) Qwen3.8-27B is a dense 27B model with: * 64 language-model layers; * hidden dimension 5120; * FFN intermediate dimension 17,408; * a repeating hybrid Gated DeltaNet / gated-attention layout; * native MTP support; * a 248,320-token padded vocabulary; * native 262,144-token context; * native multimodal architecture. ### Heretic / ARA source The full-precision source used for this repository is: [`trohrbaugh/Qwen3.8-27B-heretic-ara`](https://huggingface.co/trohrbaugh/Qwen3.8-27B-heretic-ara) That model is a behavior-modified Qwen3.8-27B produced using the [Heretic](https://github.com/p-e-w/heretic) project and its Arbitrary-Rank Ablation (ARA) approach. This repository does **not** perform an additional fine-tune. The ROCmFPX GGUF and PromptForge sidecars were generated from the Heretic-Ara BF16 weights. ## Quantization and inference provenance The quantization format and specialized inference implementation come from: * [CIRU ROCmFPX](https://github.com/ciru-ai/ROCmFPX) * [CIRU Qwen3.8 ActiveFPX + PromptForge reference release](https://huggingface.co/jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge) * [llama.cpp](https://github.com/ggml-org/llama.cpp) * [AMD Composable Kernel](https://github.com/ROCm/composable_kernel) CIRU's Qwen3.8 work adds model-specific `gfx1151` paths including: * fused FFN gate/up processing; * accelerated SwiGLU/down projection; * merged QKV/Z projection for Gated DeltaNet layers; * prepacked W8 PromptForge compute views; * small-row W8 execution; * ROCmFPX low-bit decode paths; * native Qwen3.8 MTP support. ## About this v3-style quant The compact GGUF in this repository was generated from the Heretic-Ara BF16 source using the **per-tensor precision allocation from CIRU's Qwen3.8 v3 release** as the reference. That includes the important v3 storage decisions such as: * Q8 output projection; * selected LateQ6 precision islands; * the remaining mixed ROCmFPX tensor allocation. ### FFNEQ note CIRU's official v3 model is named: `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` The `FFNEQ` part refers to an FFN-equalization treatment in CIRU's model-production process. This Heretic build reproduces the **v3 tensor-type / precision allocation**, but does **not claim to reproduce CIRU's unpublished FFNEQ weight transformation** unless explicitly stated otherwise. In other words: ```text Heretic-Ara BF16 | +--> CIRU v3-style Q8 / LateQ6 / ROCmFPX allocation | | | +--> compact GGUF | +--> BF16-derived PromptForge W8 views | +--> FFN.pfs +--> GDN.pfs ``` ## PromptForge sidecars The `.pfs` files are **not independent models** and should not be mixed with another Qwen3.8 checkpoint. They are prepacked W8 compute views generated from this model's BF16 weights. ### FFN sidecar Contains W8 prompt-time views of: ```text ffn_gate ffn_up ffn_down ``` for all 64 language-model layers. The fixed Qwen3.8 PromptForge FFN ABI uses a file size of: ```text 17,123,004,416 bytes ``` ### GDN sidecar Contains the merged W8: ```text attn_qkv + attn_gate/Z ``` views used by the 48 Gated DeltaNet layers. The fixed GDN sidecar size is: ```text 4,029,685,760 bytes ``` ## Output-K8 status CIRU's official v3 release also includes a third sidecar: ```text PromptForge-Output-K8.pfs ``` That file provides a draft-only top-8 output shortlist used by the v3 MTP path before authoritative Q8 reranking. **This repository does not currently include a Heretic-specific Output-K8 proxy.** Do not use CIRU's Output-K8 sidecar with this model: it was generated from CIRU's output weights, not the Heretic-Ara weights. The FFN and GDN PromptForge paths work independently of Output-K8. --- # Running with the CIRU PromptForge runtime The optimized path is intended primarily for AMD Strix Halo / Radeon 8060S (`gfx1151`). CIRU developed and validated the Qwen3.8 PromptForge work with a pinned TheRock ROCm 7.15 environment and a pinned Composable Kernel revision. A sufficiently compatible ROCm toolchain with working `gfx1151` HIP support is required. ## 1. Download this repository Install the Hugging Face CLI if necessary: ```bash python3 -m pip install -U huggingface_hub ``` Download the repository: ```bash export MODEL_DIR="$HOME/models/Qwen3.8-27B-Heretic-ROCmFPX-STRIX" mkdir -p "$MODEL_DIR" hf download \ tonyrishwain/Qwen3.8-27B-Heretic-ROCmFPX-STRIX \ --local-dir "$MODEL_DIR" ``` Find the artifacts: ```bash export MODEL="$(find "$MODEL_DIR" -maxdepth 1 -type f -name '*.gguf' | head -n1)" export FFN="$(find "$MODEL_DIR" -maxdepth 1 -type f -iname '*ffn*.pfs' | head -n1)" export GDN="$(find "$MODEL_DIR" -maxdepth 1 -type f -iname '*gdn*.pfs' | head -n1)" printf 'MODEL=%s\nFFN=%s\nGDN=%s\n' \ "$MODEL" "$FFN" "$GDN" ``` Verify the PFS sizes: ```bash stat -c '%n %s' "$FFN" "$GDN" ``` Expected: ```text FFN: 17123004416 GDN: 4029685760 ``` ## 2. Install build dependencies ### Fedora ```bash sudo dnf install -y \ @development-tools \ cmake \ ninja-build \ git \ python3 \ libcurl-devel ``` You also need a ROCm/HIP installation capable of compiling and running `gfx1151`. CIRU's published v3 measurements use ROCm 7.15. Distribution-provided ROCm versions may differ. Verify your GPU: ```bash rocminfo | grep -m1 gfx1151 ``` You should see `gfx1151`. ## 3. Download CIRU's v3 runtime patch The v3 runtime is built from CIRU's public v2.3 tag plus the v3 Output-K8 patch. Even though this model does not currently use Output-K8, using the current patched runtime keeps the execution stack aligned with CIRU v3. ```bash export CIRU_PATCH="$HOME/ciru-qwen38-v3" mkdir -p "$CIRU_PATCH" hf download \ jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge \ runtime/qwen38-v3-output-k8-runtime.patch \ --local-dir "$CIRU_PATCH" ``` ## 4. Clone ROCmFPX ```bash mkdir -p "$HOME/src" cd "$HOME/src" git clone https://github.com/ciru-ai/ROCmFPX.git cd ROCmFPX git checkout qwen3.8-activefpx-promptforge-v2.3 git am \ "$CIRU_PATCH/runtime/qwen38-v3-output-k8-runtime.patch" ``` ## 5. Clone the pinned Composable Kernel ```bash cd "$HOME/src" git clone https://github.com/ROCm/composable_kernel.git git -C composable_kernel checkout \ fdf4bb7fcc984811cef48ce817d89aac064b984a ``` Set: ```bash export ROCMFPX="$HOME/src/ROCmFPX" export CK="$HOME/src/composable_kernel" ``` ## 6. Configure ROCm If your ROCm installation is under `/opt/rocm`: ```bash export ROCM_PATH=/opt/rocm export PATH="$ROCM_PATH/bin:$ROCM_PATH/llvm/bin:$PATH" export LD_LIBRARY_PATH="$ROCM_PATH/lib:$ROCM_PATH/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" ``` Check: ```bash "$ROCM_PATH/bin/rocminfo" | grep -m1 gfx1151 ``` On Fedora or a custom TheRock installation, `ROCM_PATH` may be somewhere else. Adjust the path accordingly. ## 7. Build the PromptForge server ```bash cd "$ROCMFPX" cmake -S . -B build-promptforge -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER="$ROCM_PATH/llvm/bin/clang" \ -DCMAKE_CXX_COMPILER="$ROCM_PATH/llvm/bin/clang++" \ -DCMAKE_HIP_COMPILER="$ROCM_PATH/llvm/bin/clang++" \ -DCMAKE_PREFIX_PATH="$ROCM_PATH" \ -DGGML_HIP=ON \ -DGGML_CUDA=OFF \ -DGGML_VULKAN=OFF \ -DGGML_HIP_FORCE_MMQ=ON \ -DGGML_HIP_GRAPHS=ON \ -DGGML_HIP_MMQ_MFMA=ON \ -DGGML_HIP_NO_VMM=ON \ -DGGML_HIP_ROCWMMA_FATTN=OFF \ -DGGML_NATIVE=ON \ -DAMDGPU_TARGETS=gfx1151 \ -DGPU_BUILD_TARGETS=gfx1151 \ -DPROMPTFORGE_CK_ROOT="$CK" \ -DGGML_BUILD_TESTS=OFF \ -DLLAMA_BUILD_TESTS=OFF \ -DLLAMA_BUILD_SERVER=ON ``` Build: ```bash cmake --build build-promptforge \ --target llama-server \ -j"$(nproc)" ``` ## 8. Configure PromptForge for this model Enable the two model-specific sidecars: ```bash export PROMPTFORGE_SIDECAR="$FFN" export PROMPTFORGE_GDN_SIDECAR="$GDN" export PROMPTFORGE_MODE=m2048_fused_tail1476 export PROMPTFORGE_ENABLE_SMALLM_W8=1 export GGML_CUDA_GRAPH_OPT=0 export HIP_VISIBLE_DEVICES=0 export HSA_OVERRIDE_GFX_VERSION=11.5.1 ``` Because this repository does **not** contain a Heretic-specific Output-K8 proxy, explicitly disable that path: ```bash unset PROMPTFORGE_MTP_OUTPUT_K8 unset PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE unset PROMPTFORGE_MTP_OUTPUT_K8_PROXY ``` Also keep the experimental IU4/ngram paths disabled: ```bash unset PROMPTFORGE_IU4_SIDECAR unset PROMPTFORGE_GDN_IU4_SIDECAR unset PROMPTFORGE_ENABLE_NGRAM_MOD unset PROMPTFORGE_ENABLE_NGRAM_M65_IU4 ``` For the pre-Output-K8 MTP selection path: ```bash export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1 unset LLAMA_MTP_BACKEND_GREEDY ``` Set runtime library paths: ```bash export LD_LIBRARY_PATH="$ROCMFPX/build-promptforge/bin:$ROCM_PATH/lib:$ROCM_PATH/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" ``` ## 9. Start the server ```bash cd "$ROCMFPX" ./build-promptforge/bin/llama-server \ -m "$MODEL" \ --alias heretic \ --host 127.0.0.1 \ --port 8080 \ --jinja \ -fit off \ -dev ROCm0 \ -ngl 999 \ -c 262144 \ -b 2048 \ -ub 2048 \ -fa on \ -ctk f16 \ -ctv f16 \ -t 16 \ -tb 16 \ -np 1 \ -ctxcp 0 \ --cache-ram 0 \ --no-cache-prompt \ --no-cache-idle-slots \ --timeout 3600 \ --metrics \ --spec-type draft-mtp \ --spec-draft-device ROCm0 \ --spec-draft-ngl 999 \ --spec-draft-type-k f16 \ --spec-draft-type-v f16 \ --spec-draft-n-max 4 \ --spec-draft-n-min 0 \ --spec-draft-p-min 0.0 \ --spec-draft-p-split 0.10 \ --spec-draft-backend-sampling \ --temp 0.7 \ --top-p 0.95 \ --reasoning off \ --reasoning-format none \ --reasoning-budget -1 ``` The OpenAI-compatible API will be available at: ```text http://127.0.0.1:8080/v1 ``` ## 10. Test the server ```bash curl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "heretic", "messages": [ { "role": "user", "content": "Explain why the sky appears blue." } ], "max_tokens": 256, "temperature": 0.7 }' ``` ## Smaller context configurations The CIRU v3 reference profile uses: ```text -c 262144 ``` which is appropriate for a high-memory Strix Halo configuration but consumes substantial unified memory. If you do not need the full context window, you can reduce it, for example: ```text -c 65536 ``` or: ```text -c 131072 ``` This does not change the GGUF or PFS files; it only changes runtime memory allocation. --- # GGUF-only operation The `.pfs` files are optional. To run the compact model without PromptForge: ```bash unset PROMPTFORGE_SIDECAR unset PROMPTFORGE_GDN_SIDECAR unset PROMPTFORGE_MODE unset PROMPTFORGE_ENABLE_SMALLM_W8 unset PROMPTFORGE_MTP_OUTPUT_K8 unset PROMPTFORGE_MTP_OUTPUT_K8_PROXY ``` Then start a compatible llama.cpp / ROCmFPX server with the GGUF normally. The GGUF remains the authoritative model representation. PromptForge sidecars are alternate compute views used only on qualified execution paths. --- # Vision support Qwen3.8-27B is natively multimodal, but this repository currently focuses on the language-model GGUF and PromptForge language-model sidecars. No vision projector is assumed by the launch command above. If adding multimodal support, use a compatible Qwen3.8 projector whose provenance has been verified against the source checkpoint rather than blindly mixing unrelated model artifacts. --- # Why the PFS files are model-specific PromptForge does not simply contain generic kernels. The sidecars contain quantized/prepacked views of the actual model weights. Therefore: ```text Heretic GGUF + Heretic PFS correct CIRU GGUF + CIRU PFS correct Heretic GGUF + CIRU PFS incorrect ``` Mixing sidecars between different fine-tunes or ablations may silently cause PromptForge execution to use weights from a different model. Always keep the GGUF and `.pfs` files from the same repository/build together. --- # Hardware target Primary target: ```text AMD Ryzen AI Max+ 395 Radeon 8060S gfx1151 Strix Halo ``` The ROCmFPX formats themselves may work on other AMD hardware, but PromptForge contains specialized model- and kernel-level paths and should not be assumed portable without testing. CIRU's published Qwen3.8 performance work was developed specifically around `gfx1151`. --- # Notes on "uncensored" This repository inherits the behavioral modification of the Heretic-Ara source model. "Uncensored" or "decensored" describes the model-building goal; it is not a guarantee that every prompt will produce a particular response, nor that the model is more accurate than the original Qwen checkpoint. Quantization can also introduce small numerical differences relative to BF16. Users should evaluate the model for their own workloads. --- # Credits Model: * [Qwen / Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) * [trohrbaugh / Qwen3.8-27B-heretic-ara](https://huggingface.co/trohrbaugh/Qwen3.8-27B-heretic-ara) * [p-e-w / Heretic](https://github.com/p-e-w/heretic) Quantization and inference: * [CIRU ROCmFPX](https://github.com/ciru-ai/ROCmFPX) * [CIRU Qwen3.8 ActiveFPX + PromptForge](https://huggingface.co/jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge) * [llama.cpp](https://github.com/ggml-org/llama.cpp) * [AMD Composable Kernel](https://github.com/ROCm/composable_kernel) Thanks to the Qwen team, the Heretic contributors, CIRU Inference Lab, llama.cpp contributors, and AMD ROCm/Composable Kernel contributors. ## License The upstream Qwen3.8 model is released under Apache-2.0. This derived model repository follows the applicable upstream model license. Runtime projects and third-party components retain their own respective licenses.