Model gets stuck in recursive loop
When the context windows exceeds about 22,000 tokens the model will start recursive generation, outputting the same tokens over and over again in a loop until you hard interrupt. Running on Llama.cpp with the BF16 quant. This is basically useless.
I have same behaviour on Q4 quantization with llama cpp 😞
Using custom jinja template from here doesn't help.
I'm having success on 27B MTP variant Qwopus using the original chat template: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/chat_template.jinja. Perhaps 35B one is similar? No hallucinations caused by quantization when k-cache is kept at bf16 and Q5_K_M following the recommended sampling parameters and using preserve_thinking option.
EDIT: nevermind, my template was for 3.6, while this is likely 3.5 based model.
When the context windows exceeds about 22,000 tokens the model will start recursive generation, outputting the same tokens over and over again in a loop until you hard interrupt. Running on Llama.cpp with the BF16 quant. This is basically useless.
Привет. У меня linux, rx6800. Модель работает хорошо только с драйвером ROCm .
Если у тебя тоже AMD - надо разбираться с драйверами и методом запуска.
Вот как я запускаю:
/run/media/... /llama-ROCm/llama-server
-m "/run/media/... /Ornith-1.0-35B/ornith-1.0-35b-Q4_K_M.gguf"
--host 0.0.0.0 --port 1338 --api-key sk-1234
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --frequency-penalty 0.0 --repeat-penalty 1.0
--threads 8 --threads-batch 16 --no-mmap -fa on
--parallel 1 -cb --kv_unified
--ubatch-size 1024
--batch-size 1024
--jinja
--chat-template-file "/run/media/... /Qwen3.6/chat_template_V20.jinja"
--chat-template-kwargs '{"enable_thinking":true, "preserve_thinking":true, "auto_disable_thinking_with_tools":false}'
--fit on --fit-ctx 131072 --fit-target 2560
--cache-type-k q8_0 --cache-type-v q8_0 \
Got same error in Q4_K_M (using https://github.com/TheTom/llama-cpp-turboquant with). Find out k-cache and v-cache quantization cause this behavior, surely not template alone.
When the context windows exceeds about 22,000 tokens the model will start recursive generation, outputting the same tokens over and over again in a loop until you hard interrupt. Running on Llama.cpp with the BF16 quant. This is basically useless.
Hola, agrega una restricción en el "System prompt" de LM Studio, esto puede mejorar mucho la respuesta y evitar el loop: por ejemplo, agrega lo siguiente:
expert role, uses CoT
En LM Studio funcionó muy bien para mí, cuéntame si te sirvió.
I have same behaviour on Q4 quantization with llama cpp 😞
Using custom jinja template from here doesn't help.
Hola, prueba lo siguiente, agrega una restricción en el "System prompt" de LM Studio, esto puede mejorar mucho la respuesta y evitar el loop: por ejemplo, agrega lo siguiente:
expert role, uses CoT
En LM Studio funcionó muy bien para mí, cuéntame si te sirvió.
Hice muchas pruebas para llegar a esa mejora.
This model has been trained on broken Qwen3.5-35B-A3B official weight release that Alibaba Group shipped to people. It's not issue with training or learning data. It's issue with model itself. Same problem exists in Qwen3.6-35B-A3B base weights. It's main reason why this model gets stuck in a recursive loop even after full finetune in BF16 safetensors - expert tensors in block 0 has 40% of noisy zero blocks that are visible only on Q8_0 quantization.
Three ssm_con1vd tensors have enormous big signal scale and accumulate cascade error distributing during training on entire neural network.