For me, this model consistently reasons only during the first turn of a conversation. On subsequent turns, the reasoning is broken in one of the following ways:
No reasoning at all.
Reasoning is present but not formatted as reasoning; it is simply part of the response.
Reasoning is correctly parsed but does not appear at the beginning (see example below).
I am not sure whether the problem lies with the training data or the chat template. I am running it in LM Studio and tried several chat templates (Google's original one for Gemma 26B, as well as Unsloth's templates) to test whether the issue was template-related, but to no avail. The vanilla Gemma 26B does not have this issue for me.
I tested it a bit more and found that, in Unsloth Studio, even the first turn fails to initiate reasoning. However, I noticed that, in some cases, the model tries to output the <think> token instead of the <|channel>thought\n token (the standard for Gemma 4). So the problem may lie in the training data having an incorrect <think> token, as well as an incorrect chat template with <|channel>thought\n injection.
You're probably not wrong. This is my first time training a model (thanks for checking it out btw!) and I basically didn't know what I was doing. Still mostly don't, but I ran the experiment, and... I have a thing, even if it's only half-baked. I'll have to be more careful next time I build a model, which might be a while.
Again, thank you for checking this out, and hopefully the next version will turn out better!
