Instructions to use ai4bharat/indictrans2-en-indic-1B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ai4bharat/indictrans2-en-indic-1B with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="ai4bharat/indictrans2-en-indic-1B", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("ai4bharat/indictrans2-en-indic-1B", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
ImportError: cannot import name 'PreTrainedTokenizerBase' from 'transformers.tokenization_utils' (unknown location)
recreation:
!pip uninstall -q transformers
!pip install -q nltk sacremoses pandas regex mock mosestokenizer
!pip install -q transformers==4.26.0
!python -m nltk.downloader punkt
!pip install -q IndicTransToolkit@git+https://github.com/VarunGumma/IndicTransToolkit.git
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from IndicTransToolkit.processor import IndicProcessor
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
Error:
ImportError Traceback (most recent call last)
/tmp/ipykernel_9188/3720895640.py in <cell line: 0>()
1 import torch
2 from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
----> 3 from IndicTransToolkit.processor import IndicProcessor
4
5 DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
2 frames
/usr/local/lib/python3.13/dist-packages/IndicTransToolkit/collator.py in IndicDataCollator()
7 class IndicDataCollator:
8 from transformers.utils import PaddingStrategy
----> 9 from transformers.tokenization_utils import PreTrainedTokenizerBase
10 from transformers.data.data_collator import pad_without_fast_tokenizer_warning
11
ImportError: cannot import name 'PreTrainedTokenizerBase' from 'transformers.tokenization_utils' (unknown location)