Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,367 @@
+---
+language:
+- bn
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- text-generation-inference
+datasets:
+- Polygl0t/bengali-corpus
+- allenai/big-reasoning-traces
+- HuggingFaceTB/smollm-corpus
+- HuggingFaceTB/finemath
+- HuggingFaceFW/fineweb-edu
+- allenai/math-meta-reasoning-filtered
+- nvidia/OpenScience
+metrics:
+- perplexity
+pipeline_tag: text-generation
+widget:
+- text: "বাংলাদেশের রাজধানী হলো"
+  example_title: উদাহরণ
+- text: "বাংলাদেশের প্রধান নদী হলো"
+  example_title: উদাহরণ
+inference:
+  parameters:
+    repetition_penalty: 1.2
+    temperature: 0.1
+    top_k: 50
+    top_p: 1.0
+    max_new_tokens: 150
+co2_eq_emissions:
+  emissions: 333000
+  source: CodeCarbon
+  training_type: pre-training
+  geographical_location: Germany
+  hardware_used: NVIDIA A100-SXM4-80GB
+model-index:
+- name: LilTii-v0.1
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: Bangla MMLU
+      type: hishab/bangla-mmlu
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 26.08
+      name: accuracy
+    source:
+      url: https://github.com/hishab-nlp/lm-evaluation-harness
+      name: Language Model Evaluation Harness (Bengali)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: BoolQ-BN
+      type: hishab/boolq_bn
+      split: train
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 60.64
+      name: accuracy
+    source:
+      url: https://github.com/hishab-nlp/lm-evaluation-harness
+      name: Language Model Evaluation Harness (Bengali)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: CommonsenseQA-BN
+      type: hishab/commonsenseqa-bn
+      split: train
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 32.43
+      name: accuracy
+    source:
+      url: https://github.com/hishab-nlp/lm-evaluation-harness
+      name: Language Model Evaluation Harness (Bengali)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: OpenBookQA-BN
+      type: hishab/openbookqa-bn
+      split: train
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 32.19
+      name: accuracy
+    source:
+      url: https://github.com/hishab-nlp/lm-evaluation-harness
+      name: Language Model Evaluation Harness (Bengali)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: PIQA-BN
+      type: hishab/piqa-bn
+      split: train
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 60.50
+      name: accuracy
+    source:
+      url: https://github.com/hishab-nlp/lm-evaluation-harness
+      name: Language Model Evaluation Harness (Bengali) 
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: ARC-Challenge (BN)
+      type: arc_bn
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc_norm
+      value: 26.17
+      name: normalized accuracy
+    source:
+      url: https://github.com/Nkluge-correa/lm-evaluation-harness-multilingual
+      name: Language Model Evaluation Harness (Multilingual)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HellaSwag (BN)
+      type: hellaswag_bn
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: acc_norm
+      value: 32.20
+      name: normalized accuracy
+    source:
+      url: https://github.com/Nkluge-correa/lm-evaluation-harness-multilingual
+      name: Language Model Evaluation Harness (Multilingual)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU (BN)
+      type: mmlu_bn
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc_norm
+      value: 27.06
+      name: normalized accuracy
+    source:
+      url: https://github.com/Nkluge-correa/lm-evaluation-harness-multilingual
+      name: Language Model Evaluation Harness (Multilingual)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: TruthfulQA (BN)
+      type: truthfulqa_bn
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: mc1
+      value: 25.48
+      name: bleurt
+    source:
+      url: https://github.com/Nkluge-correa/lm-evaluation-harness-multilingual
+      name: Language Model Evaluation Harness (Multilingual)
+---
+# LilTii-v0.2
+
+<img src="./logo.png" alt="A round logo of a cartoon tiger in a hoodie, cap, and gold chain with the text 'LilTii' below." height="200">
+
+## Model Summary
+
+**[LilTii-v0.2](https://huggingface.co/Polygl0t/LilTii-v0.2)** is a decoder-transformer natively pretrained in Bengali and English. LilTii is part of the [Polygl0t](https://huggingface.co/Polygl0t) initiative to advance language models for low-resource languages.
+
+## Details
+
+- **Architecture:** a Transformer-based model pre-trained via causal language modeling
+- **Size:** 670,127,616 parameters
+- **Context length:** 4096 tokens
+- **Dataset(s):** 
+  - [Polygl0t/bengali-corpus](https://huggingface.co/datasets/Polygl0t/bengali-corpus)
+  - [allenai/big-reasoning-traces](https://huggingface.co/datasets/allenai/big-reasoning-traces)
+  - [HuggingFaceTB/smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)
+  - [HuggingFaceTB/finemath](https://huggingface.co/datasets/HuggingFaceTB/finemath)
+  - [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
+  - [allenai/math-meta-reasoning-filtered](https://huggingface.co/datasets/allenai/math-meta-reasoning-filtered)
+  - [nvidia/OpenScience](https://huggingface.co/datasets/nvidia/OpenScience)
+- **Language(s):** Bengali, English
+- **Batch size:** 2,097,152 tokens
+- **Number of steps:** 110,000
+- **GPU:** 8 NVIDIA A100-SXM4-80GB
+- **Training time**: ~ 215 hours
+- **Emissions:** 333 KgCO2 (Germany)
+- **Total energy consumption:** 875 kWh
+
+This repository has the [source code](https://github.com/Nkluge-correa/polyglot) used to train this model.
+
+### Learning Rate
+
+This model was trained with a linear learning rate (i.e., warmup-stable-decay). Warmup was done over the first 2,000 steps to a peak learning rate of 7e-4, followed by a stable phase until step 98,000 (this revision is named `end-of-stable-ckpt`), and then a linear decay to 0 for the last 12,000 steps. Checkpoints were saved every 2,500 steps, which equates to approximately 5 billion tokens.
+
+The main branch of this repository contains the final checkpoint saved at step 110,000. All other checkpoints are available as separate branches. To load a specific checkpoint, you can use the following code snippet:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Polygl0t/LilTii-v0.2"
+revision = "step-2500"  # Change this to the desired checkpoint branch
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, revision=revision)
+```
+
+## Intended Uses
+
+The primary intended use LilTii is to serve as foundations for research and development involving native Bengali language modeling. Checkpoints saved during training are designed to provide a controlled setting for performing comparative experiments, specifically regarding the effects of active pretraining on the performance of currently available benchmarks. You may also fine-tune and adapt LilTii for deployment if your use follows the Apache 2.0 license. If you decide to use LilTii as a basis for your fine-tuned model, please conduct your own risk and bias assessment.
+
+## Out-of-scope Use
+
+- LilTii is **not intended for deployment**. It is not an out-of-the-box product and should not be used for human-facing interactions.
+
+- LilTii is for **the Bengali language only** and is unsuitable for text generation tasks in other languages.
+
+- LilTii has **not been fine-tuned** for downstream tasks.
+
+## Basic usage
+
+### Via HuggingFace Transformers
+
+Using the `pipeline`:
+
+```python
+from transformers import pipeline
+
+generator = pipeline("text-generation", model="Polygl0t/LilTii-v0.2")
+
+completions  = generator("বাংলাদেশের রাজধানী হলো", num_return_sequences=2, max_new_tokens=150)
+
+for comp in completions:
+  print(f"🤖 {comp['generated_text']}")
+```
+
+Using the `AutoTokenizer` and `AutoModelForCausalLM`:
+
+```python
+from transformers import GenerationConfig, TextGenerationPipeline, AutoTokenizer, AutoModelForCausalLM
+import torch
+
+# Specify the model and tokenizer
+model_id = "Polygl0t/LilTii-v0.2"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+
+# Specify the generation parameters as you like
+generation_config = GenerationConfig(
+    **{
+    "do_sample": True,
+    "max_new_tokens": 150,
+    "renormalize_logits": True,
+    "repetition_penalty": 1.2,
+    "temperature": 0.1,
+    "top_k": 50,
+    "top_p": 1.0,
+    "use_cache": True, 
+  }
+)
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+generator = TextGenerationPipeline(model=model, task="text-generation", tokenizer=tokenizer, device=device)
+
+# Generate text
+prompt = "বাংলাদেশের রাজধানী হলো"
+completion = generator(prompt, generation_config=generation_config)
+print(completion[0]['generated_text'])
+```
+
+### Via vLLM Inference
+
+```python
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
+import torch
+
+model_id = "Polygl0t/LilTii-v0.2"
+prompt = "বাংলাদেশের রাজধানী হলো"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    model_id, 
+    use_fast=True,
+)
+
+model = LLM(
+    model=model_id,
+)
+
+sampling_params = SamplingParams(
+    max_tokens=150,
+    stop=[tokenizer.eos_token],
+    stop_token_ids=[tokenizer.eos_token_id],
+    n=2,
+    temperature=0.1,
+    repetition_penalty=1.2,
+    top_k=50,
+    top_p=1.0
+)
+
+outputs = model.generate([prompt], sampling_params)
+
+for output in outputs:
+    print(f"🤖 {output.outputs[0].text}")
+```
+
+## Limitations
+
+Like almost all other language models trained on large text datasets scraped from the web, the LilTii shows behavior that does not make it an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. LilTii is subject to the following:
+
+- **Hallucinations:** LilTii can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.
+
+- **Biases and Toxicity:** LilTii inherits the social and historical stereotypes from the data used to train it. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.
+
+- **Language Limitations:** LilTii is primarily designed to interact with Bengali. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.
+
+- **Repetition and Verbosity:** LilTii may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.
+
+Hence, even though LilTii is released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications.
+
+## Evaluations
+
+The table below compares our two versions of LilTii against other base models of similar size. The NPM (Normalized Performance Metric) is a metric designed to provide a balanced view of model performance across various tasks, accounting for the inherent difficulty of each task (Pires et al., [2023](https://arxiv.org/abs/2304.07880)). It normalizes the performance of each model on a given task by comparing it to a baseline performance, which represents a the performance of a random model.
+
+|                 | NPM (mean) | ARC Challenge | HellaSwag | MMLU     | TruthfulQA MC1 | Bangla MMLU | BoolQ-BN | CommonsenseQA-BN | OpenBookQA-BN | PIQA-BN  |
+|-----------------|------------|---------------|-----------|----------|----------------|-------------|----------|------------------|---------------|----------|
+| LilTii-v0.2     | 9.62724    | 0.261762      | 0.322008  | 0.270631 | 0.254802       | 0.260881    | 0.606481 | 0.324324         | 0.321932      | 0.605005 |
+| Qwen3-0.6B-Base | 8.0721     | 0.2284        | 0.286951  | 0.297865 | 0.271447       | 0.325695    | 0.657407 | 0.230139         | 0.305835      | 0.527203 |
+| Qwen2.5-0.5B    | 6.03669    | 0.229256      | 0.284246  | 0.283144 | 0.28169        | 0.308407    | 0.578704 | 0.22113          | 0.305835      | 0.535909 |
+| LilTii-v0.1     | 5.81863    | 0.235244      | 0.303073  | 0.264088 | 0.240717       | 0.239525    | 0.520833 | 0.302211         | 0.327968      | 0.587051 |
+
+
+## Cite as 🤗
+
+```latex
+
+```
+
+## Aknowlegments
+
+Polyglot is a project funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the State of North Rhine-Westphalia (MWK) as part of TRA Sustainable Futures (University of Bonn) and the Excellence Strategy of the federal and state governments.
+
+We also gratefully acknowledge the granted access to the [Marvin cluster](https://www.hpc.uni-bonn.de/en/systems/marvin) hosted by [University of Bonn](https://www.uni-bonn.de/en) along with the support provided by its High Performance Computing & Analytics Lab.
+
+## License
+
+LilTii is licensed under the Apache License, Version 2.0. For more details, see the [LICENSE](LICENSE) file.