--- language: - pt license: gemma library_name: sentence-transformers pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - embeddings - text-embeddings - text-embeddings-inference - retrieval - information-retrieval - semantic-search - rag - reranking - clustering - sts - portuguese - pt-br - brazilian-portuguese - mteb - vocabulary-trimming - knowledge-distillation - model-soup base_model: - google/embeddinggemma-300m - tardellirs/embeddinggemma-pt-br datasets: - HuggingFaceFW/fineweb-2 - celsowm/jurisprudencias_stj - celsowm/leis_ordinarias_1988_2024 - celsowm/codigo_tributario_lei_5172_1966 - bancodobrasil/bbrc_brazilian_banking_regulation_corpora - rodrigedilson/bacen2024 - carolina-c4ai/corpus-carolina ---
# 🐦 Colibri β€” Brazilian-Portuguese Embeddings **A tiny embedding model that punches far above its weight.** *Named after the colibri (hummingbird) β€” the smallest bird, yet it out-flies much larger ones.* `~157M params` Β· `768-dim` Β· `plain SentenceTransformer (no adapters)` Β· `runs on a cheap CPU VPS`
--- Colibri is a compact **Brazilian-Portuguese text-embedding model** derived from [`google/embeddinggemma-300m`](https://huggingface.co/google/embeddinggemma-300m). Despite being **half the size**, it **matches or beats much larger multilingual embedders on MTEB-BR** β€” including 7B and 27B models β€” and is **designed small on purpose so it runs comfortably on a simple CPU VPS**, with no GPU required. **Use cases:** Portuguese semantic search, retrieval-augmented generation (**RAG**), retrieval, reranking, clustering, semantic textual similarity (STS) and classification β€” for Brazilian-Portuguese text across legal, tax, financial, medical, academic and general domains. ## ✨ Highlights - πŸ† **Beats bigger models** on MTEB-BR β€” including `embeddinggemma-300m`, and 7B / 27B multilingual embedders. - πŸͺΆ **Half the footprint** of `embeddinggemma-300m`: ~607 MB vs ~1.2 GB, and less RAM. - πŸ’» **Runs on a $-few/month CPU VPS** β€” benchmarked on 4 vCPU / 16 GB (no GPU). - πŸ”Œ **Plain `SentenceTransformer`** β€” no adapters, no LoRA at inference. Drop-in. - πŸ“ **Matryoshka dimensions** (768 / 512 / 256 / 128) + an **fp16** branch + a **fast ONNX** path. --- ## πŸ₯‡ Quality β€” MTEB-BR Evaluated on **[MTEB-BR](https://huggingface.co/spaces/MTEB-BR/leaderboard)** (22 native Brazilian-Portuguese tasks: retrieval, reranking, STS, classification, clustering, pair-classification). Score = mean over the 22 tasks. | Model | Params | MTEB-BR | |---|---:|:---:| | 🐦 **Colibri** (this model) | **~157M** | **0.6501** | | google/embeddinggemma-300m | 300M | 0.6490 | | Linq-AI-Research/Linq-Embed-Mistral | 7B | 0.6473 | | openai/text-embedding-3-large | – | 0.6449 | | intfloat/multilingual-e5-large-instruct | 560M | 0.6409 | | Salesforce/SFR-Embedding-2_R | 7B | 0.6397 | | Alibaba-NLP/gte-Qwen2-7B-instruct | 7B | 0.6392 | | microsoft/harrier-oss-v1-27b | 27B | 0.6390 | | BAAI/bge-m3 | 568M | 0.6157 | | intfloat/multilingual-e5-large | 560M | 0.5909 | > A ~157M-parameter model outperforming 7B–27B multilingual embedders on Brazilian Portuguese. **Size ↔ quality frontier.** Colibri sits on the open-model Pareto frontier for MTEB-BR β€” it **dominates its own base** `embeddinggemma-300m` (half the size, higher score) and matches models up to ~10Γ— larger: ![MTEB-BR open-model size vs quality frontier β€” Colibri anchors the knee](https://huggingface.co/tardellirs/colibri-embed-ptbr/resolve/main/pareto.png) --- ## πŸ’» Built for a simple CPU VPS Colibri was designed to be **small enough for cheap, GPU-less hosting**. Benchmarked head-to-head against `google/embeddinggemma-300m` on a **[Verda](https://verda.com/?utm_content=mteb-pt) `CPU.4V.16G` (4 vCPU, 16 GB RAM)** β€” a typical small VPS β€” encoding Brazilian-Portuguese sentences: | Precision | Model | Model size | Peak RAM | Latency (p50) | Throughput | |---|---|---:|---:|---:|---:| | **fp32** | 🐦 **Colibri** | **607 MB** | **969 MB** | 76 ms | 37 sent/s | | fp32 | embeddinggemma-300m | 1211 MB | 1253 MB | 78 ms | 36 sent/s | | **ONNX** | 🐦 **Colibri** | ~600 MB | 2.9 GB | **33 ms** | **49 sent/s** | | ONNX | embeddinggemma-300m | ~1200 MB | 5.3 GB | 38 ms | 41 sent/s | **Takeaways** - **Half the size and less RAM** than `embeddinggemma-300m` β€” fits a smaller, cheaper VPS. - **Same encode latency** at equal precision (the vocabulary trim shrinks the token-embedding matrix, not the transformer compute) β€” you lose nothing in speed. - The **ONNX** backend roughly **doubles throughput** (33 ms vs 76 ms) β€” best when latency matters; fp32 is the most RAM-frugal. *Benchmark hardware: [Verda](https://verda.com/?utm_content=mteb-pt) `CPU.4V.16G`. Numbers are indicative single-node CPU measurements.* --- ## πŸš€ Usage ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("tardellirs/colibri-embed-ptbr") query = "Como declarar imposto de renda de aluguel?" docs = [ "Rendimentos de aluguel devem ser informados na ficha de Rendimentos TributΓ‘veis...", "O Pix Γ© um meio de pagamento instantΓ’neo criado pelo Banco Central...", ] q = model.encode(query, prompt_name="query") # "task: search result | query: " d = model.encode(docs, prompt_name="document") # "title: none | text: " print(model.similarity(q, d)) ``` **Faster on CPU β€” ONNX backend** (β‰ˆ2Γ— throughput, no code changes to your pipeline): ```python model = SentenceTransformer("tardellirs/colibri-embed-ptbr", backend="onnx") ``` Prompts follow EmbeddingGemma conventions (query / document / classification / clustering / similarity). --- ## πŸŽ›οΈ Variants: precision & dimension **Precision** β€” a half-precision branch (same quality, half the size; best for GPU/storage β€” on CPU it's not faster): | Variant | Load | Size | |---|---|---| | fp32 (default) | `SentenceTransformer("tardellirs/colibri-embed-ptbr")` | ~607 MB | | fp16 | `SentenceTransformer("tardellirs/colibri-embed-ptbr", revision="fp16")` | ~304 MB | **Matryoshka dimensions** β€” truncate the 768-dim output to **512 / 256 / 128** for smaller vectors (cheaper index, faster search) with a graceful quality trade-off: ```python model = SentenceTransformer("tardellirs/colibri-embed-ptbr", truncate_dim=256) ``` *(The MTEB-BR score above is for the full 768-dim embeddings.)* --- ## πŸ”§ How it was built A three-stage pipeline: **vocabulary trimming β†’ knowledge distillation β†’ model soup**. **1. Vocabulary trimming.** [`tardellirs/embeddinggemma-pt-br`](https://huggingface.co/tardellirs/embeddinggemma-pt-br) trims `embeddinggemma-300m`'s 262k multilingual vocabulary down to a **~64k Brazilian-Portuguese vocabulary**, cutting the model from ~300M to **~157M effective parameters** with negligible quality loss. Code: **[github.com/tardellirs/embedding-vocab-trimmer](https://github.com/tardellirs/embedding-vocab-trimmer)**. **2. Knowledge distillation.** Relational (similarity-preserving) KD from **two complementary teachers** β€” [`Qwen/Qwen3-Embedding-4B`](https://huggingface.co/Qwen/Qwen3-Embedding-4B) (strong at clustering) and [`Qwen/Qwen3-Embedding-8B`](https://huggingface.co/Qwen/Qwen3-Embedding-8B) (strong at retrieval / reranking). The student learns to reproduce the **average of the two teachers' pairwise-similarity matrices** (dimension-agnostic; preserves STS). Trained on a diverse **~100k-passage corpus of native Brazilian Portuguese** spanning legal, tax, financial, medical, academic, technical, encyclopedic and general-web domains: | Domain | Sources | |---|---| | General web | [HuggingFaceFW/fineweb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) (pt) | | Legal | [jurisprudencias_stj](https://huggingface.co/datasets/celsowm/jurisprudencias_stj), [leis_ordinarias_1988_2024](https://huggingface.co/datasets/celsowm/leis_ordinarias_1988_2024) | | Tax | [codigo_tributario](https://huggingface.co/datasets/celsowm/codigo_tributario_lei_5172_1966) | | Financial regulation | [bbrc](https://huggingface.co/datasets/bancodobrasil/bbrc_brazilian_banking_regulation_corpora), [bacen2024](https://huggingface.co/datasets/rodrigedilson/bacen2024) | | Academic | Brazilian scientific abstracts, [corpus-carolina](https://huggingface.co/datasets/carolina-c4ai/corpus-carolina) | | Medical | Brazilian clinical / health texts | | Technical | Stack Overflow em PortuguΓͺs | | Encyclopedic | Wikipedia-PT | > **Evaluation integrity:** the model is trained only on training / non-evaluation splits β€” **every MTEB-BR > evaluation example is held out**, so the scores reflect generalization, not memorization. **3. Model soup + merge.** Distillation checkpoints were combined (model soup / weight averaging) and linearly **merged with the trimmed base**, with the mixing weight chosen on held-out validation. Both operations yield ordinary weights, so the published model is a single standalone encoder. --- ## πŸ”— Links - Base: [google/embeddinggemma-300m](https://huggingface.co/google/embeddinggemma-300m) Β· Trimmed base: [tardellirs/embeddinggemma-pt-br](https://huggingface.co/tardellirs/embeddinggemma-pt-br) - Trimming code: [github.com/tardellirs/embedding-vocab-trimmer](https://github.com/tardellirs/embedding-vocab-trimmer) - Distillation teachers: [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) Β· [Qwen/Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) - Benchmark: [MTEB-BR](https://huggingface.co/spaces/MTEB-BR/leaderboard) Β· CPU benchmark hardware: [Verda](https://verda.com/?utm_content=mteb-pt) --- ## πŸ™ Acknowledgments We gratefully acknowledge **[Verda](https://verda.com/?utm_content=mteb-pt)** for the GPU compute credits that supported this work.
Built with vocabulary trimming Β· relational knowledge distillation Β· model soup.