My Observations during actual "work" use cases

#59
by Omnomynous - opened

Running on M2 Studio, M2 Ultra 192GB. Using llama.cpp backend, custom harness.

First test case: Q4 variant. Use case: Contextualize on existing and functioning codebase for an MCP tool + server backend that provide the harness with image generation capabilities using qwen-image-2.1. Capture the gaps in the pipeline to allow image editing. Current featureset only supports generation. Identify gaps, document in a new issue on gitea (local github like server), use internet, coding, and git skills.

Test result: Did research, found gaps. Didn't use online resources, preferred using local documentation in locally cloned repos (perfectly acceptable). Got hung up on adding labels to the git issue! I had to tell it to RE-READ the skill file which had a workflow for adding labels. It had already read the file, but just... "forgot" it had the info in context. The re-read worked. Chalking this up to the heavy Q4 quanting and moving up to Q6 for my next experiment.

Q6 - Gave it a task to refactor a triplet of small repos into a single monorepo. It did it. Definitely fewer tokens than vanilla Q6.

Sign up or log in to comment