Instructions to use MuXodious/gpt-oss-20b-RichardErkhov-heresy with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MuXodious/gpt-oss-20b-RichardErkhov-heresy with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MuXodious/gpt-oss-20b-RichardErkhov-heresy")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("MuXodious/gpt-oss-20b-RichardErkhov-heresy")
model = AutoModelForCausalLM.from_pretrained("MuXodious/gpt-oss-20b-RichardErkhov-heresy", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MuXodious/gpt-oss-20b-RichardErkhov-heresy with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MuXodious/gpt-oss-20b-RichardErkhov-heresy"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/gpt-oss-20b-RichardErkhov-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/MuXodious/gpt-oss-20b-RichardErkhov-heresy

SGLang

How to use MuXodious/gpt-oss-20b-RichardErkhov-heresy with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MuXodious/gpt-oss-20b-RichardErkhov-heresy" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/gpt-oss-20b-RichardErkhov-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MuXodious/gpt-oss-20b-RichardErkhov-heresy" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MuXodious/gpt-oss-20b-RichardErkhov-heresy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use MuXodious/gpt-oss-20b-RichardErkhov-heresy with Docker Model Runner:
```
docker model run hf.co/MuXodious/gpt-oss-20b-RichardErkhov-heresy
```

Writing Model Requests

#25

by redaihf - opened Sep 23

Discussion

redaihf

Sep 23

•

edited 2 days ago

There is a new finetune series built and benchmarked for creative writing tasks. Please consider Hereticising:

MuXodious

Owner Sep 23

•

edited Sep 23

I have skipped about 5 cities in three weeks now and don't have neither remote nor on-site access to my workstation; hence, my absence. Have you tested LoRA-enabled defensorship for these models? I believe I haven't released the LoRA for Q3.5 9B SOMPOA model, but 27B should fare just fine.

redaihf

Sep 23

•

edited Sep 23

I won't be able to run Hemmingway on my little NPU unless @Phil2Sat makes a suitable magic quant. It may also be relevant that @Naphula has also experienced problems merging the Gemma SOMPOA LoRA into a finetune/merge. Keep on enjoying your whistlestop tour so to speak.

Naphula

Sep 23

•

edited Sep 23

600 likes in 3 days, must be a good writer. Downloading it now. @redaihf What are your hardware limits? Maybe I can modify FrankenQuant for "user templates". For instance my PC is 64GB RAM and 8GB VRAM which is the current profile setup. You could reconfigure the settings and it would automatically quantize Hemmingway for maximum quality and speed on your system (a few variants with extra headroom can be uploaded too in case you OOM), imatrix support optional.

MuXodious

Owner Sep 23

•

edited Sep 23

I should look into that LoRA issue, if I would be allowed back into my shack anytime sooner. Also, yeah, what do you mean by "my little NPU" (sounds awfully like that pony cartoon).Don't tell you have been persevering on an integrated accelerator, which damningly can be a low-power Snapdragon package. You should scalp 5060 Tis before I or those pesky AI startups do, now that they wiped clean the 3090 off the entire second-hand market.

Anyways, I have a ton of notifications from you guys. You must be hard at work all this time. @Naphula I hope you recovered (or at least are on a better path to recovery) from your dicey financial situation without resorting to selling your prized RAM.

redaihf

Sep 24

@redaihf What are your hardware limits?

I have 16GB that can be addressed from the NPU. The previous Qwen quant that worked well for me was an Apex Micro of some kind.

Naphula

Sep 24

You should scalp 5060 Tis before I or those pesky AI startups do, now that they wiped clean the 3090 off the entire second-hand market.

@Naphula I hope you recovered (or at least are on a better path to recovery) from your dicey financial situation without resorting to selling your prized RAM.

Thanks, yeah I was trying to save up for a 3090 and 128GB RAM last year but didn't make enough. Now the price has risen too much. I still have the 64GB and 4TB SSD card (for now). A 5060 Ti would be nice but its unlikely I'll be able to buy one of those in time either. I'd have to avoid putting any more money into runpod finetunes/ablations for a while, which I'm considering because I don't want to be stuck with 8GB GPU caveman tech once alt man scoops all the used cards.

I should look into that LoRA issue, if I would be allowed back into my shack anytime sooner.

Looking forward to seeing your input on this once you return. I have not attempted to run LoRA extraction of a merge yet.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment