basic fetch and asking question generator part of pipeline is working

USAGE.md

@@ -0,0 +1,284 @@
+Below is a clean **USAGE.md** plus a simple **run_qg_verbose.sh** script so your successor can run the *question-first, fully verbose* pipeline with zero guesswork.
+
+Everything is written for clarity, grounded practicality, and safe handoff.
+
+---
+
+# **USAGE.md**
+
+*How to run the distillation pipeline (static seeds & question-first mode)*
+
+This guide shows how to:
+
+* run the pipeline,
+* enable verbose logging,
+* use question-generation mode,
+* understand outputs,
+* and perform sanity checks.
+
+It assumes you already have:
+
+* `distill-pipeline` cloned,
+* `.env` configured,
+* `distill-rag` running (Elasticsearch + embedding server),
+* and Node.js ≥ 20.
+
+---
+
+# 1. **Basic Commands**
+
+Install dependencies:
+
+```bash
+npm install
+```
+
+Run the default pipeline (static seeds):
+
+```bash
+npm run pipeline -- --limit 10
+```
+
+Verbose run:
+
+```bash
+npm run pipeline -- --limit 10 --verbose
+```
+
+---
+
+# 2. **Question-First Mode (QG Mode)**
+
+This mode ignores static seeds and instead:
+
+1. fetches chunks from the `distill-rag` index,
+2. runs question generation on each chunk,
+3. feeds generated questions into the normal pipeline (retrieval → generator → verifier → reward).
+
+Enable it using the environment flag:
+
+```bash
+PIPELINE_SEED_MODE=question-first npm run pipeline -- --limit 10 --verbose
+```
+
+If you need to cap questions per chunk:
+
+```env
+QUESTION_MAX_PER_CHUNK=3
+```
+
+---
+
+# 3. **Understanding Output**
+
+All accepted gold samples are written to:
+
+```
+gold/pipeline_gold.jsonl
+```
+
+Each entry includes:
+
+```json
+{
+  "question": "…",
+  "context": [ ... ],
+  "sample": { ... },
+  "verifier": { ... },
+  "reward": { ... }
+}
+```
+
+Rejected samples are not written.
+Verbose mode will show why they were rejected.
+
+---
+
+# 4. **Environment Variables**
+
+All providers are configurable:
+
+```
+GENERATOR_PROVIDER=ollama
+VERIFIER_PROVIDER=ollama
+REWARD_PROVIDER=ollama
+QUESTION_PROVIDER=ollama
+
+GENERATOR_MODEL=qwen3-vl:8b-thinking
+VERIFIER_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M
+REWARD_MODEL=tensortemplar/patronus-lynx:8b-instruct-q4_K_M
+QUESTION_MODEL=qwen2.5-7b-instruct
+```
+
+Retrieval configuration:
+
+```
+ES_NODE=http://localhost:9200
+ES_INDEX=quo_distill_index
+```
+
+Embeddings (used by distill-rag):
+
+```
+EMBED_URL=http://localhost:11434/api/embeddings
+EMBED_MODEL=mxbai-embed-large
+```
+
+---
+
+# 5. **Debugging Tips**
+
+### See provider chosen for each stage:
+
+```bash
+node -e "import('./src/providers/provider.mjs').then(m=>m.debugLoad())"
+```
+
+### Check prompt templates:
+
+```
+prompts/generator_prompt.txt
+prompts/verifier_prompt.txt
+prompts/reward_prompt.txt
+prompts/question_prompt.txt
+```
+
+### Run question generator directly:
+
+```bash
+node ./src/question/question_cli.mjs "Give me questions about freedom"
+```
+
+### Run generator stage only:
+
+```bash
+node ./src/generator/generator_cli.mjs "What is love?"
+```
+
+---
+
+# 6. **Full End-to-End (Question-First) Verbose Run**
+
+This is the most common “sanity test”:
+
+```bash
+PIPELINE_SEED_MODE=question-first npm run pipeline -- --limit 5 --verbose
+```
+
+You should see:
+
+* chunk preview
+* generated questions
+* retrieval preview
+* generator raw + parsed
+* verifier judgement
+* reward score
+
+---
+
+# 7. **Output Cleanup**
+
+To reset gold dataset:
+
+```bash
+rm -f gold/pipeline_gold.jsonl
+```
+
+To tail progress while running:
+
+```bash
+tail -f gold/pipeline_gold.jsonl
+```
+
+---
+
+# 8. **Run Tests**
+
+```bash
+npm test
+```
+
+You should see:
+
+```
+9 test files
+27 tests
+0 failures
+```
+
+---
+
+# 9. **run_qg_verbose.sh**
+
+Place this in project root:
+
+```
+distill-pipeline/run_qg_verbose.sh
+```
+
+Make executable:
+
+```bash
+chmod +x run_qg_verbose.sh
+```
+
+### **run_qg_verbose.sh**
+
+```bash
+#!/usr/bin/env bash
+
+# Always fail on error
+set -e
+
+echo "==============================================="
+echo " Running Question-First Distillation Pipeline"
+echo "==============================================="
+echo
+
+# You can adjust limit here if needed
+LIMIT="${1:-5}"
+
+export PIPELINE_SEED_MODE=question-first
+
+echo "→ Mode:            question-first"
+echo "→ Verbose:         yes"
+echo "→ Limit:           $LIMIT"
+echo
+
+npm run pipeline -- --limit "$LIMIT" --verbose
+```
+
+Run it:
+
+```bash
+./run_qg_verbose.sh
+```
+
+Or specify a custom limit:
+
+```bash
+./run_qg_verbose.sh 20
+```
+
+---
+
+# 10. **Successor Notes**
+
+For successor engineers:
+
+* If a stage fails instantly → the provider config is wrong.
+* If questions are empty → question model needs `--verbose` output and better prompts.
+* If verifier always rejects → fix your verifier prompt or lower strictness.
+* If reward always rejects → ensure reward output JSON includes `"score": number`.
+* If pipeline accepts zero samples → inspect generator raw output (check formatting / hallucination).
+
+---
+
+If you’d like:
+
+✓ a troubleshooting guide
+✓ example `.env` with explanations
+✓ Q’uo-aligned prompt examples
+✓ diagrams for README / GitHub page
+
+Just ask.

run_qg_verbose.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+# Always fail on error
+set -e
+
+echo "==============================================="
+echo " Running Question-First Distillation Pipeline"
+echo "==============================================="
+echo
+
+# You can adjust limit here if needed
+LIMIT="${1:-5}"
+
+export PIPELINE_SEED_MODE=question-first
+
+echo "→ Mode:            question-first"
+echo "→ Verbose:         yes"
+echo "→ Limit:           $LIMIT"
+echo
+
+npm run pipeline -- --limit "$LIMIT" --verbose

src/pipeline/batch.mjs

@@ -8,11 +8,12 @@ import {
   DEFAULT_OUT_PATH,
   loadSeedQuestions,
   seedToQuestion,
-  seedToContextText,
+  // seedToContextText,  // <-- no longer needed for QG mode
 } from './seeds.mjs';
 import { runPipelineStep } from './step.mjs';
 import { loadProviderFor } from '../providers/provider.mjs';
 import { runQuestionGenerator } from '../question/question_core.mjs';
+import { fetchChunksFromIndex } from '../retrieval/retrieval.mjs';
 
 /**
  * Append a single accepted record to a JSONL file.
@@ -28,8 +29,8 @@ export async function appendGoldRecord(outPath, record) {
  * samples to a JSONL file.
  *
  * Modes:
- *   - static (default): seeds are questions (current behaviour)
- *   - question-first:   seeds are chunks; we first generate questions
+ *   - question-first (default): seeds are CHUNKS; we generate questions from each chunk
+ *   - static:                seeds are questions (legacy / low-priority mode)
  *
  * Options:
  *   - seedsPath: JSONL of seeds (defaults to test_samples/seed_questions.jsonl)
@@ -37,18 +38,18 @@ export async function appendGoldRecord(outPath, record) {
  *   - limit:     max number of seeds to process
  *   - verbose:   extra per-stage logging
  *   - logger:    optional logger (defaults to console)
- *   - seedMode:  'static' | 'question-first' (or PIPELINE_SEED_MODE env)
+ *   - seedMode:  'question-first' | 'static' (overrides env + default)
  *
  * Returns:
  *   {
  *     mode,
- *     total,            // number of seed lines
- *     processed,        // number of questions run through pipeline
+ *     total,
+ *     processed,          // number of questions run through pipeline
  *     accepted,
  *     outPath,
  *     statusCounts,
- *     processedSeeds?,      // only meaningful in question-first
- *     processedQuestions?,  // alias for processed in question-first
+ *     processedSeeds?,    // (question-first) how many chunks actually used
+ *     processedQuestions? // (question-first) alias for processed
  *   }
  */
 export async function runPipelineBatch({
@@ -62,17 +63,26 @@ export async function runPipelineBatch({
   const log = logger?.log?.bind(logger) || console.log;
   const errLog = logger?.error?.bind(logger) || console.error;
 
-  const seeds = await loadSeedQuestions(seedsPath);
-  const maxSeeds = typeof limit === 'number' ? limit : seeds.length;
+  if (verbose) {
+    log('[pipeline] mode:', seedMode);
+    log('[pipeline] outPath:', outPath);
+  }
 
   let processed = 0; // number of questions sent through runPipelineStep
   let accepted = 0;
   const statusCounts = {};
 
   // ----------------------------------------
-  // MODE 1: existing behaviour (static questions)
+  // MODE 1: static questions from JSONL (legacy / tests)
   // ----------------------------------------
   if (seedMode === 'static') {
+    const seeds = await loadSeedQuestions(seedsPath);
+    const maxSeeds = typeof limit === 'number' ? limit : seeds.length;
+
+    if (verbose) {
+      log('[pipeline] seedsPath (static questions):', seedsPath);
+    }
+
     for (let idx = 0; idx < maxSeeds; idx++) {
       const seed = seeds[idx];
       const question = seedToQuestion(seed);
@@ -117,7 +127,7 @@ export async function runPipelineBatch({
     }
 
     return {
-      mode: 'static',
+      mode: seedMode,
       total: seeds.length,
       processed,
       accepted,
@@ -127,22 +137,39 @@ export async function runPipelineBatch({
   }
 
   // ----------------------------------------
-  // MODE 2: question-first (generate Qs from chunks)
+  // MODE 2: question-first (ES chunks → QG → pipeline)
   // ----------------------------------------
   if (seedMode === 'question-first') {
     const questionProvider = loadProviderFor('question');
-    const maxQuestionsPerChunk = Number(process.env.QUESTION_MAX || '5');
 
+    // support both env names
+    const maxQuestionsPerChunk = Number(
+      process.env.QUESTION_MAX_PER_CHUNK ||
+        process.env.QUESTION_MAX ||
+        '5',
+    );
+
+    const chunkLimit =
+      typeof limit === 'number' ? limit : undefined;
+
+    if (verbose) {
+      log(
+        `[pipeline] fetching chunks from ES (limit=${chunkLimit ?? 'default'})`,
+      );
+    }
+
+    const chunks = await fetchChunksFromIndex(chunkLimit);
+    const totalChunks = chunks.length;
     let processedSeeds = 0;
 
-    for (let idx = 0; idx < maxSeeds; idx++) {
-      const seed = seeds[idx];
-      const label = `[seed ${idx + 1}/${maxSeeds}]`;
+    for (let idx = 0; idx < chunks.length; idx++) {
+      const chunk = chunks[idx];
+      const label = `[chunk ${idx + 1}/${chunks.length}]`;
+      const contextText = chunk.content;
 
-      const contextText = seedToContextText(seed);
       if (!contextText || !contextText.trim()) {
         if (verbose) {
-          log(`${label} context is empty, skipping`);
+          log(`${label} chunk content empty, skipping`);
         }
         continue;
       }
@@ -150,7 +177,8 @@ export async function runPipelineBatch({
       processedSeeds += 1;
 
       if (verbose) {
-        log(`\n🧩 ${label} generating questions from chunk…`);
+        log(`\n🧩 ${label} generating questions from ES chunk…`);
+        log(`   [question] es _id: ${chunk.id}`);
         log(
           '   [question] chunk preview:\n   ' +
             preview(contextText, 300).replace(/\n/g, '\n   '),
@@ -160,7 +188,7 @@ export async function runPipelineBatch({
         );
       }
 
-      // 1) generate questions from the chunk
+      // 1) generate questions from this chunk
       let qResult;
       try {
         qResult = await runQuestionGenerator(
@@ -198,7 +226,9 @@ export async function runPipelineBatch({
         if (!q || !q.trim()) continue;
 
         const qLabel = `[q ${processed + 1}]`;
-        log(`   → ${qLabel} Running pipeline for generated question: "${q}"`);
+        log(
+          `   → ${qLabel} Running pipeline for generated question: "${q}"`,
+        );
 
         try {
           const result = await runPipelineStep({
@@ -218,8 +248,9 @@ export async function runPipelineBatch({
           if (result.status === 'accepted') {
             const record = {
               question: q,
-              sourceSeed: seed,          // keep origin of the question
-              sourceChunk: contextText,  // raw chunk we asked about
+              sourceChunkId: chunk.id,
+              sourceChunk: contextText, // raw ES chunk text
+              sourceDoc: chunk.source,  // full ES _source
               context: result.context,
               sample: result.gen,
               verifier: result.ver,
@@ -244,8 +275,8 @@ export async function runPipelineBatch({
     }
 
     return {
-      mode: 'question-first',
-      total: seeds.length,
+      mode: seedMode,
+      total: totalChunks,
       processed,               // number of questions processed
       processedSeeds,          // how many chunks we actually used
       processedQuestions: processed,
@@ -257,3 +288,4 @@ export async function runPipelineBatch({
 
   throw new Error(`Unknown PIPELINE_SEED_MODE: ${seedMode}`);
 }
+

src/pipeline/pipeline_cli.js

@@ -25,6 +25,7 @@ function parseArgs(argv) {
   let seedsPath;
   let outPath;
   let verbose = false;
+  let seedMode; // optional CLI override
 
   for (let i = 0; i < args.length; i++) {
     const a = args[i];
@@ -38,6 +39,9 @@ function parseArgs(argv) {
     } else if (a === '--out') {
       outPath = args[i + 1];
       i++;
+    } else if (a === '--mode') {
+      seedMode = args[i + 1]; // "question-first" | "static"
+      i++;
     } else if (a === '--verbose' || a === '-v') {
       verbose = true;
     }
@@ -56,25 +60,39 @@ function parseArgs(argv) {
     seedsPath: seedsPath || DEFAULT_SEEDS,
     outPath: outPath || DEFAULT_OUT,
     verbose,
+    seedMode,
   };
 }
 
 async function main() {
-  const { limit, seedsPath, outPath, verbose } = parseArgs(process.argv);
+  const {
+    limit,
+    seedsPath,
+    outPath,
+    verbose,
+    seedMode: cliSeedMode,
+  } = parseArgs(process.argv);
 
   const generatorProvider = process.env.GENERATOR_PROVIDER || 'ollama';
   const verifierProvider = process.env.VERIFIER_PROVIDER || generatorProvider;
   const rewardProvider = process.env.REWARD_PROVIDER || generatorProvider;
 
   const generatorModel =
-    process.env.GENERATOR_MODEL || process.env.OLLAMA_MODEL || 'qwen3-vl:8b-thinking';
+    process.env.GENERATOR_MODEL ||
+    process.env.OLLAMA_MODEL ||
+    'qwen3-vl:8b-thinking';
   const verifierModel =
     process.env.VERIFIER_MODEL || generatorModel;
   const rewardModel =
     process.env.REWARD_MODEL || verifierModel;
 
+  // Resolve mode: CLI > env > default
+  const mode =
+    cliSeedMode || process.env.PIPELINE_SEED_MODE || 'question-first';
+
   console.log('');
   console.log('🚀 Starting Distillation Pipeline');
+  console.log(`   Mode:     ${mode}`);
   console.log(`   Seeds:    ${seedsPath}`);
   console.log(`   Output:   ${outPath}`);
   console.log(`   Providers:`);
@@ -98,15 +116,21 @@ async function main() {
       limit,
       verbose,
       logger: console,
+      seedMode: mode,
     });
 
     console.log('');
     console.log('🎉 Pipeline completed');
+    console.log(`   Mode:           ${result.mode}`);
     console.log(`   Total Seeds:    ${result.total}`);
-    console.log(`   Processed:      ${result.processed}`);
+    console.log(`   Processed Qs:   ${result.processed}`);
     console.log(`   Accepted:       ${result.accepted}`);
     console.log(`   Written to:     ${result.outPath}`);
 
+    if (result.processedSeeds != null) {
+      console.log(`   Chunks used:    ${result.processedSeeds}`);
+    }
+
     if (result.statusCounts) {
       console.log('');
       console.log('   Status breakdown:');

src/pipeline/seeds.mjs

@@ -17,6 +17,8 @@ export const DEFAULT_OUT_PATH = path.join(
 
 /**
  * Load JSONL seed questions or chunks.
+ *
+ * Used only by the legacy "static" mode of the pipeline.
  * Each line may be:
  *  - { "question": "..." }
  *  - { "prompt": "..." }
@@ -34,6 +36,7 @@ export async function loadSeedQuestions(seedsPath = DEFAULT_SEEDS_PATH) {
 
 /**
  * Extract a question string from a seed record.
+ * Only used in legacy static mode.
  */
 export function seedToQuestion(seed) {
   if (typeof seed === 'string') return seed;
@@ -41,7 +44,9 @@ export function seedToQuestion(seed) {
 }
 
 /**
- * Extract a chunk of text from a seed record (for question-first mode).
+ * Extract a chunk of text from a seed record.
+ * Not used in the ES-driven question-first mode, but kept
+ * for backwards compatibility and potential tooling.
  */
 export function seedToContextText(seed) {
   if (typeof seed === 'string') return seed;

src/retrieval/retrieval.mjs

@@ -140,10 +140,40 @@ Do NOT include JSON or formatting.
   return fuseRRF(dense, sparse).slice(0, k);
 }
 
+// ----------------------------------------
+// Chunk sampling from ES (for QG pipeline)
+// ----------------------------------------
+export async function fetchChunksFromIndex(limit = 10) {
+  const size = Number.isFinite(limit) ? limit : 10;
+
+  const res = await client.search({
+    index: ES_INDEX,
+    size,
+    query: { match_all: {} },
+  });
+
+  const hits = res.hits?.hits || [];
+
+  return hits.map((h) => ({
+    id: h._id,
+    score: h._score,
+    // distill-rag usually stores text in `content`, but we try a few keys
+    content:
+      h._source?.content ??
+      h._source?.text ??
+      h._source?.chunk ??
+      '',
+    source: h._source,
+  }));
+}
+
+
 export default {
   embed,
   bm25Search,
   vectorSearch,
   hybridSearch,
-  hydeHybrid
+  hydeHybrid,
+  fetchChunksFromIndex,
 };
+