Below is a clean **USAGE.md** plus a simple **run_qg_verbose.sh** script so your successor can run the *question-first, fully verbose* pipeline with zero guesswork. Everything is written for clarity, grounded practicality, and safe handoff. --- # **USAGE.md** *How to run the distillation pipeline (static seeds & question-first mode)* This guide shows how to: * run the pipeline, * enable verbose logging, * use question-generation mode, * understand outputs, * and perform sanity checks. It assumes you already have: * `distill-pipeline` cloned, * `.env` configured, * `distill-rag` running (Elasticsearch + embedding server), * and Node.js ≥ 20. --- # 1. **Basic Commands** Install dependencies: ```bash npm install ``` Run the default pipeline (static seeds): ```bash npm run pipeline -- --limit 10 ``` Verbose run (shows generator thought/thinking/answer/confidence/evidence/limitations and raw response): ```bash npm run pipeline -- --limit 10 --verbose ``` --- # 2. **Question-First Mode (QG Mode)** This mode ignores static seeds and instead: 1. fetches chunks from the `distill-rag` index, 2. runs question generation on each chunk, 3. feeds generated questions into the normal pipeline (retrieval → generator → verifier → reward). Enable it using the environment flag: ```bash PIPELINE_SEED_MODE=question-first npm run pipeline -- --limit 10 --verbose ``` If you need to cap questions per chunk: ```env QUESTION_MAX_PER_CHUNK=3 ``` ### Random-walk over chunks Shuffle the chunk order (crypto-random) to reduce ordering bias: ```bash PIPELINE_RANDOM_WALK=1 QUESTION_MAX_PER_CHUNK=3 npm run pipeline -- --limit 3 --chunk-limit 10 --verbose ``` Equivalent toggle: `PIPELINE_CHUNK_ORDER=random`. `--chunk-limit` (or `PIPELINE_CHUNK_LIMIT`) caps how many chunks are sampled. --- # 3. **Understanding Output** All accepted gold samples are written to: ``` gold/pipeline_gold.jsonl ``` Each entry includes (generator sample contains answer, thought, raw, confidence, evidence, limitations, thinking): ```json { "question": "…", "context": [ ... ], "sample": { ... }, "verifier": { ... }, "reward": { ... } } ``` Rejected samples are not written. Verbose mode will show why they were rejected. --- # 4. **Environment Variables** All providers are configurable: ``` GENERATOR_PROVIDER=ollama VERIFIER_PROVIDER=ollama REWARD_PROVIDER=ollama QUESTION_PROVIDER=ollama GENERATOR_MODEL=qwen3-vl:8b-thinking VERIFIER_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M REWARD_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M QUESTION_MODEL=qwen2.5-7b-instruct # instruct-only generator (optional) INSTRUCT_PIPELINE=0 INSTRUCT_GENERATOR_MODEL=phi-4-instruct INSTRUCT_GENERATOR_PROVIDER=ollama ``` Retrieval configuration: ``` ES_NODE=http://localhost:9200 ES_INDEX=quo_distill_index ``` Embeddings (used by distill-rag): ``` EMBED_URL=http://localhost:11434/api/embeddings EMBED_MODEL=mxbai-embed-large ``` General pipeline knobs: ``` PIPELINE_SEED_MODE=question-first # or static PIPELINE_RANDOM_WALK=0 # set 1 for shuffled chunks QUESTION_MAX_PER_CHUNK=5 # PIPELINE_CHUNK_LIMIT=10 # optional chunk cap # PIPELINE_CACHE_DIR=data/cache # override cache dir (e.g., data/cache_instruct) ``` --- # 5. **Debugging Tips** ### See provider chosen for each stage: ```bash node -e "import('./src/providers/provider.mjs').then(m=>m.debugLoad())" ``` ### Check prompt templates: ``` prompts/generator_prompt.txt prompts/verifier_prompt.txt prompts/reward_prompt.txt prompts/question_prompt.txt ``` ### Run question generator directly: ```bash node ./src/question/question_cli.mjs "Give me questions about freedom" ``` ### Run generator stage only: ```bash node ./src/generator/generator_cli.mjs "What is love?" ``` ### Try generator prompt with cached chunk/question: ```bash # picks first cached chunk + question scripts/try_generator_prompt.sh # pick specific cached chunk/question scripts/try_generator_prompt.sh # random cached chunk/question + reasoning mode scripts/try_generator_prompt.sh --random -r ``` Uses `data/cache/questions.jsonl` + `data/rag_chunks.jsonl` and injects them into `prompts/generator_prompt.txt`. It picks the first matching cached chunk ID in the rag file (or random with `--random`). Override via `PIPELINE_CACHE_DIR`, `RAG_CHUNKS_PATH`, `PROMPT_FILE`, `GENERATOR_MODEL`, `OLLAMA_URL`. If no cached chunk IDs match the rag chunks, run the pipeline once to populate the cache. --- # 6. **Full End-to-End (Question-First) Verbose Run** This is the most common “sanity test”: ```bash PIPELINE_SEED_MODE=question-first npm run pipeline -- --limit 5 --verbose ``` You should see: * chunk preview * generated questions * retrieval preview * generator raw + parsed * verifier judgement * reward score ### Long overnight runs To let the pipeline keep going until you stop it, drop `--limit` (process all available seeds/chunks) or set a very high limit. For a basic “run all cached chunks with random walk” overnight job: ```bash PIPELINE_SEED_MODE=question-first \ PIPELINE_RANDOM_WALK=1 \ QUESTION_MAX_PER_CHUNK=5 \ npm run pipeline -- --verbose ``` If you prefer a hard cap instead of truly unbounded, set `--limit ` or `PIPELINE_CHUNK_LIMIT`. To run it in a loop until you Ctrl+C: ```bash while true; do PIPELINE_SEED_MODE=question-first PIPELINE_RANDOM_WALK=1 npm run pipeline -- --verbose sleep 10 done ``` # 7. **Instruct-only generator runs** Use an instruct model without touching the default “thinking” pipeline: ```bash INSTRUCT_PIPELINE=1 \ INSTRUCT_GENERATOR_MODEL= \ PIPELINE_CACHE_DIR=data/cache_instruct \ npm run pipeline -- --out gold/pipeline_gold_instruct.jsonl --verbose ``` What it does: - switches the generator model/provider to `INSTRUCT_GENERATOR_MODEL` (and `INSTRUCT_GENERATOR_PROVIDER` if set), - keeps verifier/reward unchanged (configure `VERIFIER_MODEL`/`REWARD_MODEL` if you want lighter models), - defaults the output to `gold/pipeline_gold_instruct.jsonl` unless you pass `--out`. Keep caches separate by setting `PIPELINE_CACHE_DIR` for instruct runs (e.g., `data/cache_instruct`) so you don’t mix artifacts with the thinking pipeline. The default cache path is unchanged unless you override it. ### One-liner scripts for continuous runs Thinking pipeline (random-walk, no limit, restarts every 10s): ``` scripts/run_thinking_continuous.sh ``` Instruct pipeline (needs `INSTRUCT_GENERATOR_MODEL`, uses separate cache/output): ``` INSTRUCT_GENERATOR_MODEL= scripts/run_instruct_continuous.sh ``` Configure `INSTRUCT_GENERATOR_PROVIDER`, `PIPELINE_CACHE_DIR`, or `INSTRUCT_OUT` as needed; stop with Ctrl+C. --- # 7. **Output Cleanup** To reset gold dataset: ```bash rm -f gold/pipeline_gold.jsonl ``` To tail progress while running: ```bash tail -f gold/pipeline_gold.jsonl ``` --- # 8. **Run Tests** ```bash npm test ``` You should see: ``` 9 test files 27 tests 0 failures ``` --- # 9. **run_qg_verbose.sh** Place this in project root: ``` distill-pipeline/run_qg_verbose.sh ``` Make executable: ```bash chmod +x run_qg_verbose.sh ``` ### **run_qg_verbose.sh** ```bash #!/usr/bin/env bash # Always fail on error set -e echo "===============================================" echo " Running Question-First Distillation Pipeline" echo "===============================================" echo # You can adjust limit here if needed LIMIT="${1:-5}" export PIPELINE_SEED_MODE=question-first echo "→ Mode: question-first" echo "→ Verbose: yes" echo "→ Limit: $LIMIT" echo npm run pipeline -- --limit "$LIMIT" --verbose ``` Run it: ```bash ./run_qg_verbose.sh ``` Or specify a custom limit: ```bash ./run_qg_verbose.sh 20 ``` --- # 10. **Successor Notes** For successor engineers: * If a stage fails instantly → the provider config is wrong. * If questions are empty → question model needs `--verbose` output and better prompts. * If verifier always rejects → fix your verifier prompt or lower strictness. * If reward always rejects → ensure reward output JSON includes `"score": number`. * If pipeline accepts zero samples → inspect generator raw output (check formatting / hallucination). --- If you’d like: ✓ a troubleshooting guide ✓ example `.env` with explanations ✓ Q’uo-aligned prompt examples ✓ diagrams for README / GitHub page Just ask.