{ "schema_version": 1, "module": "_ggml_attn_metal_f6752d0", "functions": [ { "name": "flash_attn", "qualname": "flash_attn", "module": "_ggml_attn_metal_f6752d0", "kind": "function", "doc": "Flash attention through ggml's vector kernel — the path upstream picks for decode.\n\nArgs:\n q: `(n_seqs, n_heads, n_q, head_dim)`.\n k, v: `(n_seqs, n_heads_kv, n_kv, head_dim)`. Grouped-query attention is native, so do **not**\n expand them to `n_heads` first — that copy is exactly what this avoids. An f16 cache is read\n as f16, as llama.cpp reads its default cache; other dtypes are read as f32.\n mask: `(n_seqs, 1, n_q, n_kv)` additive mask, or None. Cast to f16 internally, as the kernel\n requires. **A None mask means attend to everything**: ggml has no `is_causal` argument, so\n causality has to arrive as a mask. `flash_attn_forward` builds one when it must.\n scale: softmax scale; defaults to `head_dim ** -0.5`.\n sinks: attention sinks, one logit per query head that joins each row's softmax and takes no\n value (gpt-oss's `s_aux`); upstream's `ggml_flash_attn_ext_add_sinks`.\n softcap: logit softcapping, `softcap * tanh(scores / softcap)` before the softmax (Gemma 2's\n `attn_logit_softcapping`); 0 for none.\n\nReturns:\n `(n_seqs, n_q, n_heads, head_dim)` f32 — tokens before heads, which is what SDPA gives after its\n own `.transpose(1, 2)`, so a caller usually wants precisely this and no further permute.\n\nAsk `supports_flash_attn` first: it answers for whichever of upstream's two paths the shape selects.", "signature": { "parameters": [ { "name": "q", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "k", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "v", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "mask", "kind": "POSITIONAL_OR_KEYWORD", "default": "None", "annotation": "torch.Tensor | None" }, { "name": "scale", "kind": "POSITIONAL_OR_KEYWORD", "default": "None", "annotation": "float | None" }, { "name": "sinks", "kind": "POSITIONAL_OR_KEYWORD", "default": "None", "annotation": "torch.Tensor | None" }, { "name": "softcap", "kind": "POSITIONAL_OR_KEYWORD", "default": "0.0", "annotation": "float" } ], "return_annotation": "torch.Tensor" } }, { "name": "supports_flash_attn", "qualname": "supports_flash_attn", "module": "_ggml_attn_metal_f6752d0", "kind": "function", "doc": "Whether this build has a flash-attention kernel for these shapes.\n\nUpstream picks its vector kernel for `n_q < 20` and its tiled one above that; both are ported, so\nwhat is declined is head-dim pairs neither has a template for, and `head_dim_k < head_dim_v`.", "signature": { "parameters": [ { "name": "n_q", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "int" }, { "name": "head_dim_k", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "int" }, { "name": "head_dim_v", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "int" } ], "return_annotation": "bool" } }, { "name": "flash_attn_forward", "qualname": "flash_attn_forward", "module": "_ggml_attn_metal_f6752d0", "kind": "function", "doc": "Attention entry point shaped for `transformers`' attention interface.\n\nSelected with `attn_implementation=\":flash_attn_forward\"`, which is how a kernel provides an\nattention implementation without the caller reaching into `ALL_ATTENTION_FUNCTIONS`.\n\nBoth of upstream's paths are ported -- the vector kernel for `n_q < 20`, the tiled one above it -- so\ndecode and prefill both run on ggml's kernels. A head-dim pair neither has a template for raises, the\nway `flash_attention_2` and the block-sparse kernels do, rather than quietly running torch's attention\nunder a name that promises ggml's: what upstream instantiates does not depend on the input, so a model\nwhose head dims are unsupported is unsupported for its whole run, and saying so at the first forward\nbeats reporting a speedup that never happened. Grouped-query attention is native to the kernel, so\n`key`/`value` keep their own head count.", "signature": { "parameters": [ { "name": "module", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.nn.modules.module.Module" }, { "name": "query", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "key", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "value", "kind": "POSITIONAL_OR_KEYWORD", "default": null, "annotation": "torch.Tensor" }, { "name": "attention_mask", "kind": "POSITIONAL_OR_KEYWORD", "default": "None", "annotation": "torch.Tensor | None" }, { "name": "scaling", "kind": "POSITIONAL_OR_KEYWORD", "default": "None", "annotation": "float | None" }, { "name": "dropout", "kind": "POSITIONAL_OR_KEYWORD", "default": "0.0", "annotation": "float" }, { "name": "kwargs", "kind": "VAR_KEYWORD", "default": null, "annotation": null } ], "return_annotation": "tuple[torch.Tensor, None]" } } ], "layers": [] }