Post
415
multi-agents orchestration is old news by now. Try multi-teams of agents ! that is a whole different nightmare ... This is the real signal about the sparks of AGI.
ml-fw-prerelease
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
alielfilali01ย
posted an update 10 days ago
omarkamaliย
posted an update about 1 month ago
Post
185
Apparently grammar can travel without a passport ๐
We trained a new neural grammar model *Tarakib* across 166 languages on top of Sawtone [1] - itself trained on nearly 400 langs. We then tested it on 28 languages that contributed no training updates to the grammar task.
The surprising result: it still learned useful parts-of-speech structure based on the foundational linguistic representation offered by Sawtone.
Performance varied considerably, and the transfer was not limited to close, same-script relatives but crossed writing systems. We found connections across language families, and about a quarter of the stable transfer links crossed writing systems.
This does not mean universal parsing is solved. But it suggests that Sawtone's representations contain universal grammatical clues that a shared model can reuse in languages it was never directly taught to parse.
A small step toward "spaCy for every language" โ๏ธ
1. https://www.omneitylabs.com/models/sawtone
We trained a new neural grammar model *Tarakib* across 166 languages on top of Sawtone [1] - itself trained on nearly 400 langs. We then tested it on 28 languages that contributed no training updates to the grammar task.
The surprising result: it still learned useful parts-of-speech structure based on the foundational linguistic representation offered by Sawtone.
Performance varied considerably, and the transfer was not limited to close, same-script relatives but crossed writing systems. We found connections across language families, and about a quarter of the stable transfer links crossed writing systems.
This does not mean universal parsing is solved. But it suggests that Sawtone's representations contain universal grammatical clues that a shared model can reuse in languages it was never directly taught to parse.
A small step toward "spaCy for every language" โ๏ธ
1. https://www.omneitylabs.com/models/sawtone
alielfilali01ย
authored a
paper about 2 months ago
BramVanroyย
posted an update 2 months ago
Post
2805
**I benchmarked HF buckets against https access for Common Crawl.**
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
Took me a while to get round to do this but I benchmarked access to Common Crawl via https vs hf buckets. Both experiments were run at night in Europe. I do not think other hardware problems were impacting the speeds since CPU processing time of the non-download pipeline components were highly similar (within 2% identical) and below only the WarcReader speeds of datatrove are used.
Experiment: selected 5 disjoint samples of 64 files each (randomly from the latest crawl; 20,499 docs/file). Those five batches were then processed by 32 single-core tasks with 4GB/core (five batches to calculate CIs). Paired experiment between using https and hf bucket.
- https: 40.0 [39.3-40.6] (seconds per WARC file)
- hf bucket: 172.0 [122.7-221.2]
That is a difference of about 4x in streaming speed. You'll see that https is also more stable (smaller CI).
I also ran raw throughput tests to the endpoints to measure rate limiting (64MiB transfer at 8/32/128/256 concurrent readers) and rate limiting seems not an issue for either: at any of those parallel reader numbers, their respective speeds stay about the same.
Note that, given CC scale, this is still a small test. Rate limiting may become more obvious when processing a full crawl. I do not know whether the https endpoint vs HF bucket will shut you out earlier with which limits.
alibayramย
authored a
paper 4 months ago
alibayramย
submitted a
paper to Daily Papers 4 months ago
Post
1428
New blog post!
An introduction to a little-known but highly effective model reduction method: ๐ง๐ฟ๐ถ๐บ๐บ๐ถ๐ป๐ดโ๏ธ
We show how to reduce model size (we went up to 87.24% reduction) while preserving its performance.
We applied this technique to 16 different model families across several modalities to illustrate that it works on any architecture (as long as the embedding layer is the last one of the model) and on any modality involving text.
From these 16 families, we generated over ๐ฑ,๐ฑ๐ฌ๐ฌ ๐บ๐ผ๐ป๐ผ๐น๐ถ๐ป๐ด๐๐ฎ๐น ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ถ๐ป ๐ญ๐ฎ๐ฐ ๐ฑ๐ถ๐ณ๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ๐ ๐
Key takeaways from our experiments:
1๏ธโฃ Trimming does not require a GPU. Our models were obtained on a CPU.
2๏ธโฃ This method scales up to at least 4B parameters (we did not test beyond that).
3๏ธโฃ Trimmed model is smaller than the original while preserving its performance. If you observe a slight performance drop, just fine-tuned to recover or even surpass the original performance.
4๏ธโฃ For an equivalent compute budget, it is better to trim then fine-tune rather than fine-tuning the original model. Since the model is smaller, you can run more epochs/show more data and get in fine a better model than the original.
5๏ธโฃ Trimming is a competitive alternative to distillation and quantization. E.g. we obtained our alternative to DistilBERT in 9 minutes on CPU vs. 90 hours of GPU for the latter.
6๏ธโฃ Trimming could generate reasoning traces in the language of the trimmed model. This could be an alternative to generating traces in English and then translating them into the desired language.
And many other things (such as how much data are needed, the impact of the database used, the order in which it should be done, etc.) are available in the blogpost!
Blogpost: https://huggingface.co/blog/lbourdois/introduction-to-trimming
Models: alphaedge-ai/Trimming_models_search
An introduction to a little-known but highly effective model reduction method: ๐ง๐ฟ๐ถ๐บ๐บ๐ถ๐ป๐ดโ๏ธ
We show how to reduce model size (we went up to 87.24% reduction) while preserving its performance.
We applied this technique to 16 different model families across several modalities to illustrate that it works on any architecture (as long as the embedding layer is the last one of the model) and on any modality involving text.
From these 16 families, we generated over ๐ฑ,๐ฑ๐ฌ๐ฌ ๐บ๐ผ๐ป๐ผ๐น๐ถ๐ป๐ด๐๐ฎ๐น ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ถ๐ป ๐ญ๐ฎ๐ฐ ๐ฑ๐ถ๐ณ๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ๐ ๐
Key takeaways from our experiments:
1๏ธโฃ Trimming does not require a GPU. Our models were obtained on a CPU.
2๏ธโฃ This method scales up to at least 4B parameters (we did not test beyond that).
3๏ธโฃ Trimmed model is smaller than the original while preserving its performance. If you observe a slight performance drop, just fine-tuned to recover or even surpass the original performance.
4๏ธโฃ For an equivalent compute budget, it is better to trim then fine-tune rather than fine-tuning the original model. Since the model is smaller, you can run more epochs/show more data and get in fine a better model than the original.
5๏ธโฃ Trimming is a competitive alternative to distillation and quantization. E.g. we obtained our alternative to DistilBERT in 9 minutes on CPU vs. 90 hours of GPU for the latter.
6๏ธโฃ Trimming could generate reasoning traces in the language of the trimmed model. This could be an alternative to generating traces in English and then translating them into the desired language.
And many other things (such as how much data are needed, the impact of the database used, the order in which it should be done, etc.) are available in the blogpost!
Blogpost: https://huggingface.co/blog/lbourdois/introduction-to-trimming
Models: alphaedge-ai/Trimming_models_search
- 4 replies
alielfilali01ย
posted an update 4 months ago
Post
856
Plans in HTML > Plans in Markdown
omarkamaliย
posted an update 6 months ago
Post
1090
Just sharing a little breakthrough with Gherbal LID where we managed to distinguish the 15 variants of Arabic with 6 variants above 90%, 10 variants above 85% accuracy, practically distinguishing Moroccan and Algerian (which overlap massively).
It also embraces the duality of MSA and arabic variants pioneered in ALDi by @AMR-KELEG et al.
Now we're only bottlenecked by the availability of high quality data for the low scoring variants such as Iraqi, Libyan, Sudanese, Adeni ...
More on Gherbal at:
https://omneitylabs.com/models/gherbal
It also embraces the duality of MSA and arabic variants pioneered in ALDi by @AMR-KELEG et al.
Now we're only bottlenecked by the availability of high quality data for the low scoring variants such as Iraqi, Libyan, Sudanese, Adeni ...
More on Gherbal at:
https://omneitylabs.com/models/gherbal
- 1 reply
omarkamaliย
posted an update 6 months ago
Post
4642
We got Qwen 3.5 to count Rs in Strawberry correctly! ๐จ
Building on Sawtone, weโve been testing a different way to feed language into an LLM to build the next generation of multilingual AI.
The usual setup gives the model tokenized text and asks it to perform various linguistic tasks. That works surprisingly well, until it doesnโt. Accents disappear. Words get mangled. Internal structure gets blurred away. And the cost of that gets higher once you move into multilingual and lower-resource settings.
So we tried adding a second path.
In addition to the normal text input, the model also receives Sawtone: a byte-level word representation that preserves how a word is written, how it sounds, and how it is structured.
Same LLM. Better interface.
In this proof of concept with Qwen 3.5 0.8B, that pushed our eval from 64% to 88%. The gains showed up exactly where tokenized models usually get shaky: diacritics, character order, exact spelling, and other form-sensitive behavior.
Sawtone itself is tokenizer-free, byte-level, and pre-trained across 507 languages.
Still early, but promising!
Building on Sawtone, weโve been testing a different way to feed language into an LLM to build the next generation of multilingual AI.
The usual setup gives the model tokenized text and asks it to perform various linguistic tasks. That works surprisingly well, until it doesnโt. Accents disappear. Words get mangled. Internal structure gets blurred away. And the cost of that gets higher once you move into multilingual and lower-resource settings.
So we tried adding a second path.
In addition to the normal text input, the model also receives Sawtone: a byte-level word representation that preserves how a word is written, how it sounds, and how it is structured.
Same LLM. Better interface.
In this proof of concept with Qwen 3.5 0.8B, that pushed our eval from 64% to 88%. The gains showed up exactly where tokenized models usually get shaky: diacritics, character order, exact spelling, and other form-sensitive behavior.
Sawtone itself is tokenizer-free, byte-level, and pre-trained across 507 languages.
Still early, but promising!
- 5 replies
omarkamaliย
posted an update 6 months ago
Post
247
๐ LID Benchmark update:
โข 10 Regional Leaderboards
โข 17 LID models (+7 new, incl. non-fastText based)
โข 449 languages in total (200+ additional)
โข Fixed: F1 macro reporting error
โข Normalized language codes for more accurate results
The dataset is also updated, now with individual model predictions to reproduce and validate our findings.
omneity-labs/lid-benchmark
โข 10 Regional Leaderboards
โข 17 LID models (+7 new, incl. non-fastText based)
โข 449 languages in total (200+ additional)
โข Fixed: F1 macro reporting error
โข Normalized language codes for more accurate results
The dataset is also updated, now with individual model predictions to reproduce and validate our findings.
omneity-labs/lid-benchmark
omarkamaliย
posted an update 7 months ago
Post
235
Omneity Labs LID Benchmark is live ๐ฅ
- 8 Evals
- 10 Models (GlotLID, OpenLID, our own Gherbal and others)
- 200+ Languages
- One Leaderboard To Rule Them All!
Come find your language and which LID model supports it best in this space ๐
omneity-labs/lid-benchmark
- 8 Evals
- 10 Models (GlotLID, OpenLID, our own Gherbal and others)
- 200+ Languages
- One Leaderboard To Rule Them All!
Come find your language and which LID model supports it best in this space ๐
omneity-labs/lid-benchmark
omarkamaliย
posted an update 7 months ago
Post
1919
I just might have cracked tokenizer-free LLMs. No vocab, no softmax.
I'm training a 22M params LLM rn to test this "thing" and it's able to formulate coherent sentences ๐คฏ
Bear in mind, this is a completely new, tokenizer-free LLM architecture with built-in language universality.
Check the explainer video to understand what's happening. Feedback welcome on this approach!
I'm training a 22M params LLM rn to test this "thing" and it's able to formulate coherent sentences ๐คฏ
Bear in mind, this is a completely new, tokenizer-free LLM architecture with built-in language universality.
Check the explainer video to understand what's happening. Feedback welcome on this approach!
- 14 replies
omarkamaliย
posted an update 7 months ago
Post
351
You're probably training on outdated Wikipedia data right now and don't know it. ๐ก
In June last year, a friend from the Moroccan Wikipedia community slid into my DMs: "Are you using the current version? The official dataset is severely outdated. We added so many articles nowhere to be found on HuggingFace."
He was right. I was running a 2023 snapshot. In 2025. The official Wikipedia dataset, the one hundreds of labs and researchers grab by default without a second thought, was frozen in time.
โข For English, that's 700,000 missing articles.
โข For Moroccan Arabic, 30% of the language's entire Wikipedia.
โข For 31 other languages, there was literally no text corpus at all until recently.
I could've shrugged and moved on. Instead I spent the next months building a monthly automated pipeline for 340+ languages, on my personal laptop, nearly killing it several times in the process (100% disk, frozen screen, the works).
Nous Research trained Hermes 4 on it. INRIA cited it. It's now three years ahead of what most people are training on.
Here's the full story of how I built Wikipedia Monthly ๐
https://omarkamali.com/blog/wikipedia-monthly-pipeline
In June last year, a friend from the Moroccan Wikipedia community slid into my DMs: "Are you using the current version? The official dataset is severely outdated. We added so many articles nowhere to be found on HuggingFace."
He was right. I was running a 2023 snapshot. In 2025. The official Wikipedia dataset, the one hundreds of labs and researchers grab by default without a second thought, was frozen in time.
โข For English, that's 700,000 missing articles.
โข For Moroccan Arabic, 30% of the language's entire Wikipedia.
โข For 31 other languages, there was literally no text corpus at all until recently.
I could've shrugged and moved on. Instead I spent the next months building a monthly automated pipeline for 340+ languages, on my personal laptop, nearly killing it several times in the process (100% disk, frozen screen, the works).
Nous Research trained Hermes 4 on it. INRIA cited it. It's now three years ahead of what most people are training on.
Here's the full story of how I built Wikipedia Monthly ๐
https://omarkamali.com/blog/wikipedia-monthly-pipeline
omarkamaliย
posted an update 9 months ago
Post
1715
New year, new dataset ๐
I just released omarkamali/wikipedia-labels, with all the structural labels and namespace from wikipedia in 300+ languages. A gift for the data preprocessors and cleaners among us.
Happy new year 2026 everyone! ๐
I just released omarkamali/wikipedia-labels, with all the structural labels and namespace from wikipedia in 300+ languages. A gift for the data preprocessors and cleaners among us.
Happy new year 2026 everyone! ๐
omarkamaliย
posted an update 10 months ago
Post
307
Picomon v0.2.0 released! ๐ซ
- Supports all of AMD, Nvidia and Apple Silicon ๐งโ๐งโ๐งโ๐ง
- Beautiful TUI with themes (who said monitoring should be boring?) ๐
- Shareable Rig Cards! Boast to friends, family and foes alike ๐ซจ
Get it now!
- Supports all of AMD, Nvidia and Apple Silicon ๐งโ๐งโ๐งโ๐ง
- Beautiful TUI with themes (who said monitoring should be boring?) ๐
- Shareable Rig Cards! Boast to friends, family and foes alike ๐ซจ
Get it now!
uvx picomon or pip install picomon then picomon 
- 3 replies
omarkamaliย
posted an update 10 months ago
Post
3509
Hello
Just run
Repo at https://github.com/omarkamali/picomon
Or pypi at https://pypi.org/project/picomon
picomon! AMD GPU Monitoring made easyJust run
uvx picomon and behold:โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ GPU 0 GFX 42% UMC 21% โ โ GPU 1 GFX 78% UMC 66% โ
โ PWR 135/250W (54%) VRAM 10.0/16.0GB 62% โ โ PWR 210/250W (84%) VRAM 14.5/16.0GB 90% โ
โ โ โ โ
โ GFX โโโโโโโ
โโโโโโโ
โโโโ โ โ GFX โโโโ
โโโโโโโ
โโโโโ
โ โ
โ PWR โโโโโโโโ
โโโโโโโ
โโโ โ โ PWR โโโโโ
โโโโโโโ
โโโโโ โ
โ VRM โโโโโโโโ
โโโโโโโโ
โโ โ โ VRM โโโโ
โโโโโโโโโ
โโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโRepo at https://github.com/omarkamali/picomon
Or pypi at https://pypi.org/project/picomon
omarkamaliย
posted an update 10 months ago
Post
5242
Exciting updates to the Wikipedia Monthly dataset for November! ๐
ใป Fixed a bug to remove infobox leftovers and other wiki markers such as
ใป New python package https://pypi.org/project/wikisets: a dataset builder with efficient sampling so you can combine the languages you want seamlessly for any date (ideal for pretraining data but works for any purpose)
ใป Moved the pipeline to a large server. Much higher costs but with better reliability and predictability (let me know if you'd like to sponsor this!).
ใป Dataset sizes are unfortunately missing for this month due to shenanigans with the migration, but should be back in December's update.
Check out the dataset:
omarkamali/wikipedia-monthly
ใป Fixed a bug to remove infobox leftovers and other wiki markers such as
__TOC__ใป New python package https://pypi.org/project/wikisets: a dataset builder with efficient sampling so you can combine the languages you want seamlessly for any date (ideal for pretraining data but works for any purpose)
ใป Moved the pipeline to a large server. Much higher costs but with better reliability and predictability (let me know if you'd like to sponsor this!).
ใป Dataset sizes are unfortunately missing for this month due to shenanigans with the migration, but should be back in December's update.
Check out the dataset:
omarkamali/wikipedia-monthly
SivilTaramย
authored a
paper 11 months ago
nouamanetaziย
posted an update 12 months ago
Post
5256
After training ๐๐ฆ๐จ๐ฅ๐๐๐ on ๐๐๐ ๐๐๐๐๐ฌ for nearly a month, I've come to realize something most people overlook: ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ฆ๐๐ค๐-๐จ๐ซ-๐๐ซ๐๐๐ค ๐๐๐๐ญ๐จ๐ซ ๐ข๐ง ๐๐๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ . ๐ฅ
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious ๐๐๐๐ ๐๐ซ๐ซ๐จ๐ซ๐ฌ, or when your expensive GPU cluster is running at ๐๐% ๐๐๐๐ข๐๐ข๐๐ง๐๐ฒ, the problem isn't your model. It's most probably a ๐ฆ๐ข๐ฌ๐ฎ๐ฌ๐ ๐จ๐ ๐ญ๐ก๐ ๐ก๐๐ซ๐๐ฐ๐๐ซ๐. ๐ ๏ธ
Questions that seemed simple but had no clear answers: Why is ๐๐จ๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฅ๐จ๐ฐ๐๐ซ ๐ญ๐ก๐๐ง ๐๐๐ง๐ฌ๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ? Which ๐๐๐๐ ๐๐ฅ๐๐ ๐ฌ should we actually set? How often should we checkpoint without killing throughput?
That's why we built ๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค ๐: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ฅ๐๐ฒ๐๐ซ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: ๐๐๐๐ ๐ก๐ข๐ญ๐ญ๐ข๐ง๐ ๐ ๐๐/๐ฌ, ๐๐๐๐ข๐ง๐ค ๐.๐ ๐ซ๐๐๐๐ก๐ข๐ง๐ ๐๐๐ ๐๐/๐ฌ, ๐๐๐๐ ๐๐๐ง๐ ๐๐ญ ๐๐.๐ ๐๐/๐ฌ. Then we ran collective operations across ๐๐๐ ๐๐๐๐ฌ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ๐๐๐ ๐๐/๐ฌ on a single node to ๐๐๐-๐๐๐ ๐๐/๐ฌ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค: https://lnkd.in/e5MKXUHS
Shared with โค๏ธ by the HuggingFace team
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious ๐๐๐๐ ๐๐ซ๐ซ๐จ๐ซ๐ฌ, or when your expensive GPU cluster is running at ๐๐% ๐๐๐๐ข๐๐ข๐๐ง๐๐ฒ, the problem isn't your model. It's most probably a ๐ฆ๐ข๐ฌ๐ฎ๐ฌ๐ ๐จ๐ ๐ญ๐ก๐ ๐ก๐๐ซ๐๐ฐ๐๐ซ๐. ๐ ๏ธ
Questions that seemed simple but had no clear answers: Why is ๐๐จ๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฌ๐ฅ๐จ๐ฐ๐๐ซ ๐ญ๐ก๐๐ง ๐๐๐ง๐ฌ๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ? Which ๐๐๐๐ ๐๐ฅ๐๐ ๐ฌ should we actually set? How often should we checkpoint without killing throughput?
That's why we built ๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค ๐: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the ๐ข๐ง๐๐ซ๐๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ฎ๐ซ๐ ๐ฅ๐๐ฒ๐๐ซ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: ๐๐๐๐ ๐ก๐ข๐ญ๐ญ๐ข๐ง๐ ๐ ๐๐/๐ฌ, ๐๐๐๐ข๐ง๐ค ๐.๐ ๐ซ๐๐๐๐ก๐ข๐ง๐ ๐๐๐ ๐๐/๐ฌ, ๐๐๐๐ ๐๐๐ง๐ ๐๐ญ ๐๐.๐ ๐๐/๐ฌ. Then we ran collective operations across ๐๐๐ ๐๐๐๐ฌ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from ๐๐๐ ๐๐/๐ฌ on a single node to ๐๐๐-๐๐๐ ๐๐/๐ฌ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
๐๐ก๐ ๐๐ฆ๐จ๐ฅ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ฅ๐๐ฒ๐๐จ๐จ๐ค: https://lnkd.in/e5MKXUHS
Shared with โค๏ธ by the HuggingFace team