Instructions to use sophosympatheia/Glistening-Gem-31B-v2.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sophosympatheia/Glistening-Gem-31B-v2.1 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="sophosympatheia/Glistening-Gem-31B-v2.1") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("sophosympatheia/Glistening-Gem-31B-v2.1") model = AutoModelForMultimodalLM.from_pretrained("sophosympatheia/Glistening-Gem-31B-v2.1", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sophosympatheia/Glistening-Gem-31B-v2.1 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sophosympatheia/Glistening-Gem-31B-v2.1" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sophosympatheia/Glistening-Gem-31B-v2.1", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/sophosympatheia/Glistening-Gem-31B-v2.1
- SGLang
How to use sophosympatheia/Glistening-Gem-31B-v2.1 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sophosympatheia/Glistening-Gem-31B-v2.1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sophosympatheia/Glistening-Gem-31B-v2.1", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sophosympatheia/Glistening-Gem-31B-v2.1" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sophosympatheia/Glistening-Gem-31B-v2.1", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use sophosympatheia/Glistening-Gem-31B-v2.1 with Docker Model Runner:
docker model run hf.co/sophosympatheia/Glistening-Gem-31B-v2.1
Best Gemma 4 merge for creative writing
This is without a doubt the best merge for creative writing. One small request: as Artemis v1.1 has come out, if you even make another version would you please consider using that latest version for the merge instead of v1? Thank you again for this merge, it is awesome!
I'm glad you're enjoying it. I ran a few experiments with Artemis v1.1, but then I settled on Artemis v1 to maximize creativity. From the author's notes comparing the two versions:
The v1 release was overdue. It was a good early attempt that pulled out the best writing potential in Gemma 4. However, v1 might require some handholding, token banning, etc., for stability and to avoid dash spiraling. With that said, many fans still vouched for [what is now] v1.
The v1.1 release addresses the issues of v1 while retaining as much of the charm that v1 had.
I wanted max creativity from that ingredient, and I figured that the other merge ingredients would smooth out the rough edges of v1. Artemis v1 yielded better results than v1.1 as I was testing the merge recipe, so I kept it in the final blend.
I have sometimes found that the latest version of a model isn't always the best ingredient for a merge recipe where other models can compensate for issues.
Understood, it is very true that latest is not always the greatest. Thanks again for your work on this merge. Truly astounding!
Hi @megaray99 . For creative writing, do you mean like storytelling? If so, can you share your system prompt and sampler settings? Thanks.
The model is very good, mate.
Hi @megaray99 . For creative writing, do you mean like storytelling? If so, can you share your system prompt and sampler settings? Thanks.
Yes, I do mean storytelling. I use the default oobabooga system prompt in chat-instruct mode. But I always end my story prompts with the text "Please make this story as long as possible: elaborate - basically take this prompt as just a seed, but you add whatever you need to make it into a very detailed novel length story based on the prompt specifications. As you are aiming for a long form novel, You can build the story over multiple responses, it doesn't have to be all in one response." This keeps the story open ended (otherwise normally it tries to rush to finish the story in one response). To continue, I just click "send" button on an empty prompt, and it will pick up and continue the story in the next response. I can get a story to continue indefinitely this way. The sampler settings are very important, I went through tons of iterations and tuning to arrive at this, but I found these settings unleash the full creativity of this merge while maintaining near perfect coherence:
temperature: 1.1
top_p: 0.8
min_p: 0.2
top_n_sigma: 0.8
xtc_probability: 0.5
xtc_threshold: 0.1
Keep in mind I'm using imatrix q6 quant, so if you have lower quant you may need to lower the temp and tighten the top_p,min_p and top_n_sigma parameters