alpha2beta/AliceAI-T5-35B-A0.6B-GGUF

AliceAI-T5-35B-A0.6B GGUF (IQ4_NL)

GGUF quantization of yandex/AliceAI-T5-35B-A0.6B, a Yandex encoder-decoder language model with sparse mixture-of-experts layers (~34.35B unique parameters, 512 experts per MoE layer with top-8 routing, 1,536-wide hidden state, 16 encoder layers, 12 decoder layers, shared encoder/decoder embeddings, RoPE with YaRN).

Provenance: the quantization file in this repo was sourced from the community release at ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF and is published here unchanged for convenient access. Quantization work credit belongs to that release. See the official upstream model card for the original model description and benchmark claims.

Compatibility warning: AliceAI-T5 is a base encoder-decoder model, not an instruction/chat model. It requires an AliceAI-aware llama.cpp runtime with encoder, cross-attention, and decoder support (custom aliceai_t5_moe architecture, not decoder-only). Stock llama-cli/llama-server, Ollama, vLLM, and LM Studio runtimes have not been validated for this architecture. A working runtime is derived from the XHToken/llama.cpp fork — see "Build the AliceAI-aware runtime" below.

Files

File	Size	SHA256
AliceAI-T5-35B-A0.6B-IQ4_NL.gguf	19,579,469,728 bytes (~19.6 GB)	ff2083c820493054b7a3b662cae6732cffa654f8150721eff225d00f47732abd
aliceai-chat.jinja	single-turn QA chat template (see below)	—

Download and verify

hf download alpha2beta/AliceAI-T5-35B-A0.6B-GGUF \
  AliceAI-T5-35B-A0.6B-IQ4_NL.gguf aliceai-chat.jinja \
  --local-dir ./aliceai-t5

sha256sum ./aliceai-t5/AliceAI-T5-35B-A0.6B-IQ4_NL.gguf
# expected: ff2083c820493054b7a3b662cae6732cffa654f8150721eff225d00f47732abd

Build the AliceAI-aware runtime

Stock llama.cpp binaries cannot load this model (unknown model architecture: 'aliceai_t5_moe'). Build the patched runtime:

git clone https://github.com/XHToken/llama.cpp.git aliceai-llama.cpp
cd aliceai-llama.cpp
git checkout 4a3635c32fc9f044c2bde9ebeabf50c7e1ec5991

mkdir -p /tmp/aliceai-runtime
hf download ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF \
  reproducibility/aliceai-runtime.patch --local-dir /tmp/aliceai-runtime
git apply /tmp/aliceai-runtime/reproducibility/aliceai-runtime.patch

# CUDA hosts:
cmake -B build -DGGML_CUDA=ON
# Vulkan hosts (e.g. Intel Arc iGPU):
# cmake -B build -DGGML_VULKAN=ON
# CPU-only:
# cmake -B build -DGGML_CUDA=OFF
cmake --build build --config Release -j --target llama-server llama-simple

Quick start

Smoke test with the llama-simple encoder-decoder harness from the patched runtime (ordinary llama-cli is not sufficient for this architecture):

./build/bin/llama-simple \
  -m ./aliceai-t5/AliceAI-T5-35B-A0.6B-IQ4_NL.gguf \
  -ngl 0 -n 16 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'
# expected: Paris

The tested protocol prepends the upstream [_S_] mode token, appends <SPAN#0> to the encoder input, and seeds the decoder with [decoder_bos, <SPAN#0>]; the harness handles these encoder-decoder details.

Serve an OpenAI-compatible endpoint with the same runtime:

./build/bin/llama-server \
  -m ./aliceai-t5/AliceAI-T5-35B-A0.6B-IQ4_NL.gguf \
  --alias aliceai-t5 \
  --host 127.0.0.1 --port 8001 \
  --n-gpu-layers 99 \
  --ctx-size 8192 --parallel 1 \
  --batch-size 8192 --ubatch-size 2048 \
  --flash-attn on --jinja \
  --chat-template-file ./aliceai-t5/aliceai-chat.jinja \
  --temp 0

curl http://127.0.0.1:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"aliceai-t5","messages":[{"role":"user","content":"What is the capital of France?"}],"temperature":0,"max_tokens":64}'
# {"choices":[{"message":{"content":" Paris."},"finish_reason":"stop", ...}], ...}

CPU-only example:

./build/bin/llama-server \
  -m ./aliceai-t5/AliceAI-T5-35B-A0.6B-IQ4_NL.gguf \
  --host 127.0.0.1 --port 8001 \
  --jinja --chat-template-file ./aliceai-t5/aliceai-chat.jinja \
  --n-gpu-layers 0 \
  --ctx-size 2048 --parallel 1 \
  --batch-size 2048 --ubatch-size 2048 \
  --threads 6 --flash-attn off \
  --temp 0

Prompt format and template notes

The expected text-to-text prompt format is:

Question: What is the capital of France?
Answer:

• Do not add a ChatML system prompt and do not expect multi-turn assistant behavior from this base model.
• aliceai-chat.jinja is a single-turn QA adapter: it uses the latest user message only and does not preserve chat history.
• Use temperature=0 for factual QA; default sampling temperatures can produce unstable or empty answers (e.g. a bare "hello" typically returns empty even at temperature=0 — phrase inputs as questions/tasks).
• Do not omit --chat-template-file: without it, llama.cpp may select a decoder-only ChatML template and expose tokens such as <|im_start|>.
• An encoder input of N tokens requires --ubatch-size greater than N; reducing it cannot make a failing encoder request succeed. If the log still reports an encoder batch size of 512 after setting --ubatch-size 2048, the binary is not the patched runtime — rebuild as above.
• KV cache defaults to f16; pass --cache-type-k q8_0 --cache-type-v q8_0 to reduce memory at some quality cost.

Known limitations

• Not instruction-tuned or chat-tuned; task behavior can differ substantially from an assistant model.
• The upstream 128K context claim has not been validated for this GGUF file.
• GGUF file size is not a minimum VRAM requirement; leave headroom for runtime buffers, context, and the selected backend.
• Some runtimes may display UTF-8 byte-fallback pieces as <0xHH> tokens until their output detokenizer reconstructs the original bytes.

License and attribution

Upstream model: Apache License 2.0 (see yandex/AliceAI-T5-35B-A0.6B). This is an unofficial community quantization mirror, not an official Yandex release or endorsement.