Update README_zh.md

README_zh.md

@@ -1,96 +1,268 @@
-base_model:
-- Qwen/Qwen3-Coder-Next
-pipeline_tag: text-generation
----
-<p align="center">
-  <img src="https://cdn-uploads.huggingface.co/production/uploads/685e122d50df66f41587d406/kzCYol3JJ0nC3x3RvBZWd.png" alt="语调">
-</p>
-
-Language 中文|[English](https://huggingface.co/YCWTG/Qwen3-Coder-Next-int2-AutoRound-best)
-## Model Details
-
-This model is an int2 model with group_size 512 and symmetric quantization of [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) generated by [intel/auto-round](https://github.com/intel/auto-round). Please follow the license of the original model.
-
-### MMLU-Pro
-
-| Model | Accuracy | Delta |
-|-------|----------|-------|
-| BF16 | 52.90% | - |
-| **W2A16** | **51.27%** | **-1.63%** |
-
-## How to Use
-
-### HF Usage
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_name = "YCWTG/Qwen3-Coder-Next-int2-AutoRound-best"
-
-# load the tokenizer and the model
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
-
-# prepare the model input
-prompt = "Write a quick sort algorithm."
-messages = [{"role": "user", "content": prompt}]
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True,
-)
-model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
-
-# conduct text completion
-generated_ids = model.generate(**model_inputs, max_new_tokens=65536)
-output_ids = generated_ids[0][len(model_inputs.input_ids[0]) :].tolist()
-
-content = tokenizer.decode(output_ids, skip_special_tokens=True)
-
-print("content:", content)
-```
-
-### VLLM Usage
-
-```bash
-vllm serve YCWTG/Qwen3-Coder-Next-int2-AutoRound-best \
-    --trust-remote-code \
-    --dtype bfloat16 \
-    --tensor_parallel_size 1
-```
-
-## Generate the Model
-
-```bash
-auto-round-best --model Qwen/Qwen3-Coder-Next  --scheme "W2A16G32"  --format "auto_round"
-```
-
-## Ethical Considerations and Limitations
-
-The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
-Therefore, before deploying any applications of the model, developers should perform safety testing.
-
-## Caveats and Recommendations
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
-Here are a couple of useful links to learn more about Intel's AI software:
-
-- [Intel Neural Compressor](https://github.com/intel/neural-compressor)
-- [AutoRound](https://github.com/intel/auto-round)
-
-## Disclaimer
-
-The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
-
-## Cite
-
-```
-@article{cheng2023optimize,
-  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
-  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
-  journal={arXiv preprint arXiv:2309.05516},
-  year={2023}
-}
-```
-
-[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)## Benchmark Results
+<p align="center">
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/685e122d50df66f41587d406/kzCYol3JJ0nC3x3RvBZWd.png" alt="语调">
+</p>
+
+Language 中文|[English](https://huggingface.co/YCWTG/Qwen3-Coder-Next-int2-AutoRound-best)
+## 模型详情
+
+该模型是一个 **mixed-bits INT2 量化** 模型，group_size 为 512，并对 [Qwen/Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) 采用对称量化，由 [intel/auto-round](https://github.com/intel/auto-round) 生成。请遵循原始模型的 license。
+
+### 量化策略（Intel MoE Recipe）
+
+| 层类型                                | Bits   | 说明                  |
+| ---------------------------------- | ------ | ------------------- |
+| Expert layers（512 个 experts）       | 2-bit  | MoE expert MLPs     |
+| Non-expert layers（attention, gate） | 16-bit | 为保证质量使用更高精度         |
+| shared_expert_gate                 | 16-bit | 跳过（shape 不能被 32 整除） |
+| lm_head                            | 原始精度   | 被 AutoRound 排除      |
+
+### 模型大小
+
+* **原始 BF16**：约 160GB
+* **mixed INT2**：约 25GB（**减少 84%↓↓**）
+
+
+## 快速开始
+
+### HF 使用方法
+
+```python
+import math
+import os
+
+os.environ.setdefault(
+    "PYTORCH_ALLOC_CONF",
+    # 保持更安全的 allocator 默认设置，并透明迁移已弃用的环境变量
+    os.environ.pop("PYTORCH_CUDA_ALLOC_CONF", None) or "expandable_segments:True",
+)
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_NAME = "YCWTG/Qwen3-Coder-Next-int2-mixed-AutoRound"
+AUTO_MAX_TOKENS = True
+MANUAL_MAX_NEW_TOKENS = 128
+AUTO_MAX_TOKENS_RATIO = 1.5
+
+HAS_CUDA = torch.cuda.is_available()
+# 读取一次 VRAM 总量，并据此选择默认加载模式
+GPU_TOTAL_MIB = torch.cuda.get_device_properties(0).total_memory // (1024 ** 2) if HAS_CUDA else 0
+# 32GB 级别 GPU 默认 False；更小显存的 GPU 默认 True
+ENABLE_CPU_OFFLOAD = HAS_CUDA and GPU_TOTAL_MIB < 32000
+MAX_MEMORY = {0: "18GiB", "cpu": "64GiB"} if ENABLE_CPU_OFFLOAD else {0: "22GiB", "cpu": "16GiB"}
+
+
+def get_input_device(model):
+    # 使用 device_map="auto" 时，第一个可用设备可能不是 model.device
+    device_map = getattr(model, "hf_device_map", None)
+    cpu_device = None
+    if isinstance(device_map, dict):
+        for loc in device_map.values():
+            if isinstance(loc, int):
+                return torch.device(f"cuda:{loc}")
+            if isinstance(loc, str):
+                if loc.startswith("cuda"):
+                    return torch.device(loc)
+                if loc.startswith("cpu"):
+                    cpu_device = torch.device("cpu")
+    return cpu_device or next(model.parameters()).device
+
+
+def load_model():
+    print("正在加载模型...")
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True, use_fast=True)
+    # 使用 EOS 作为 PAD，避免在没有 pad token 的模型上进行 chat generation 时出现警告
+    tokenizer.pad_token = tokenizer.pad_token or tokenizer.eos_token
+    tokenizer.padding_side = "left"
+
+    model_kwargs = {
+        "pretrained_model_name_or_path": MODEL_NAME,
+        "dtype": torch.bfloat16,
+        "trust_remote_code": True,
+        "low_cpu_mem_usage": True,
+        "device_map": "auto" if HAS_CUDA else "cpu",
+    }
+    if HAS_CUDA:
+        print(f"GPU 总显存: {GPU_TOTAL_MIB} MiB")
+        model_kwargs["max_memory"] = MAX_MEMORY
+        if ENABLE_CPU_OFFLOAD:
+            model_kwargs["offload_buffers"] = True
+            print("CPU offload: 开启")
+        else:
+            print("CPU offload: 关闭（优先使用 GPU，允许少量 CPU 溢出）")
+    else:
+        print("CUDA 不可用，在 CPU 上运行")
+
+    try:
+        model = AutoModelForCausalLM.from_pretrained(**model_kwargs)
+    except RuntimeError as e:
+        # 提供对新手更友好的提示，而不是只显示原始报错堆栈
+        if "out of memory" in str(e).lower():
+            print("\n加载模型时发生 CUDA OOM")
+            print("请关闭其他 GPU 程序，或设置 ENABLE_CPU_OFFLOAD = True 后重试")
+        raise
+    model.eval()
+    return model, tokenizer
+
+
+def multiline_input():
+    print('用户（单独一行输入 "END" 发送，输入 "exit" 退出）：')
+    lines = []
+    while True:
+        line = input()
+        text = line.strip()
+        if text.lower() in {"exit", "quit"}:
+            return None
+        if text == "END":
+            break
+        lines.append(line)
+    return "\n".join(lines)
+
+
+def build_input_ids(tokenizer, messages, device):
+    if getattr(tokenizer, "chat_template", None):
+        # chat 模型的优先路径：由 tokenizer 构建 prompt 格式
+        prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    else:
+        # 没有内置 chat template 时的通用回退方案
+        prompt = "\n".join(
+            [f"{'User' if m['role'] == 'user' else 'Assistant'}: {m['content']}" for m in messages]
+            + ["Assistant:"]
+        )
+    return tokenizer(prompt, return_tensors="pt")["input_ids"].to(device)
+
+
+def chat_loop(model, tokenizer):
+    print("\n===== 对话已开始 =====\n")
+    print(f"自动 max_tokens: {'开启' if AUTO_MAX_TOKENS else '关闭'}")
+    if not AUTO_MAX_TOKENS:
+        print(f"手动 max_new_tokens: {MANUAL_MAX_NEW_TOKENS}")
+    print(
+        "提示：设置 ENABLE_CPU_OFFLOAD = False 可尝试更快的全 GPU 模式"
+        if ENABLE_CPU_OFFLOAD
+        else "提示：若 max_tokens 过大导致 CUDA OOM，可设置 ENABLE_CPU_OFFLOAD = True"
+    )
+
+    messages = []
+    device = get_input_device(model)
+    print(f"输入设备: {device}")
+
+    while True:
+        user_text = multiline_input()
+        if user_text is None:
+            break
+
+        messages.append({"role": "user", "content": user_text})
+        input_ids = build_input_ids(tokenizer, messages, device)
+        prompt_tokens = int(input_ids.shape[-1])
+        # 自动模式：输出长度随 prompt 长度缩放（默认 1.5 倍）
+        max_new_tokens = max(1, math.ceil(prompt_tokens * AUTO_MAX_TOKENS_RATIO)) if AUTO_MAX_TOKENS else int(MANUAL_MAX_NEW_TOKENS)
+
+        print(f"Prompt tokens: {prompt_tokens}")
+        print(f"max_new_tokens: {max_new_tokens}")
+
+        try:
+            with torch.inference_mode():
+                output_ids = model.generate(
+                    input_ids=input_ids,
+                    max_new_tokens=max_new_tokens,
+                    do_sample=True,
+                    temperature=0.9,
+                    top_p=0.9,
+                    top_k=40,
+                    use_cache=False,
+                    pad_token_id=tokenizer.pad_token_id,
+                    eos_token_id=tokenizer.eos_token_id,
+                )
+        except RuntimeError as e:
+            error_text = str(e).lower()
+            if HAS_CUDA and ("cublas_status_alloc_failed" in error_text or "out of memory" in error_text):
+                # 清理 CUDA 缓存块，使下一次尝试在更干净的 CUDA 状态下开始
+                torch.cuda.empty_cache()
+                print("\n生成过程中发生 CUDA OOM")
+                print("请设置 ENABLE_CPU_OFFLOAD = True，或关闭 AUTO_MAX_TOKENS 并降低 MANUAL_MAX_NEW_TOKENS")
+                messages.pop()
+                continue
+            raise
+
+        reply_text = tokenizer.decode(output_ids[0, input_ids.shape[-1]:], skip_special_tokens=True)
+        print(f"\nAssistant:\n{reply_text}\n")
+        messages.append({"role": "assistant", "content": reply_text})
+
+
+if __name__ == "__main__":
+    model, tokenizer = load_model()
+    chat_loop(model, tokenizer)
+```
+
+## 生成模型
+
+```python
+from auto_round import AutoRound
+
+model_name = "Qwen/Qwen3-Coder-Next"
+
+# 为 mixed-bits 构建 layer config（Intel recipe）
+layer_config = {}
+for i in range(48):  # 48 层
+    prefix = f"model.layers.{i}"
+
+    # Attention 层 -> 16-bit
+    if i in [3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47]:  # self_attn 层
+        for proj in ["q_proj", "k_proj", "v_proj", "o_proj"]:
+            layer_config[f"{prefix}.self_attn.{proj}"] = {"bits": 16}
+    else:  # linear_attn 层
+        for proj in ["in_proj_qkvz", "in_proj_ba", "out_proj"]:
+            layer_config[f"{prefix}.linear_attn.{proj}"] = {"bits": 16}
+
+    # MLP gate -> 16-bit
+    layer_config[f"{prefix}.mlp.gate"] = {"bits": 16}
+
+    # shared_expert_gate -> 16-bit（跳过）
+    layer_config[f"{prefix}.mlp.shared_expert_gate"] = {"bits": 16}
+
+autoround = AutoRound(
+    model_name,
+    bits=2,  # experts 默认 2-bit
+    group_size=128,
+    sym=True,
+    iters=1000,
+    nsamples=512,
+    lr=2e-3,
+    layer_config=layer_config,
+    low_gpu_mem_usage=True,
+    enable_alg_ext=True
+)
+output_dir="~/.cache/model/Qwen3-Coder-Next-int2-mixed-AutoRound"
+autoround.quantize_and_save(output_dir, format="auto_round")
+```
+
+## 伦理考量与局限性
+
+该模型可能会生成事实不准确的输出，因此不应被依赖用于提供绝对准确的信息。由于 pretrained model 以及 finetuning datasets 的局限性，模型可能生成带有低俗（lewd）、偏见（biased）或其他具有冒犯性的内容。
+因此，在部署任何基于该模型的应用之前，开发者应进行安全测试。
+
+
+
+## 注意事项与建议
+
+用户（包括直接用户和下游用户）应充分了解模型的风险、偏见和局限性。
+以下是一些用于进一步了解 Intel AI software 的有用链接：
+
+* [Intel Neural Compressor](https://github.com/intel/neural-compressor)
+* [AutoRound](https://github.com/intel/auto-round)
+
+
+
+## 免责声明
+
+本模型的 license 不构成法律建议。对于第三方使用本模型所产生的行为，我们不承担责任。在将该模型用于商业用途之前，请咨询律师。
+
+
+
+## 引用
+
+（引用代码保持不变）
+
+[arxiv](https://arxiv.org/abs/2309.05516)
+[github](https://github.com/intel/auto-round)