Support transformers 4.48–5.x: fix ONNX import, KV cache and tokenizer errors; SDPA by default

#19
by VarunGumma - opened
AI4Bharat org

This PR updates the remote code so the model works with current transformers releases, without changing its outputs.

Errors fixed

  • ModuleNotFoundError: No module named 'transformers.onnx' (transformers v5): the ONNX export config is only defined when transformers.onnx exists.
  • AttributeError: ... _special_tokens_map when loading the tokenizer (transformers v5).
  • AttributeError: 'NoneType' object has no attribute 'shape' in generate (transformers >= 4.54): KV caching now uses EncoderDecoderCache.
  • from_pretrained(..., attn_implementation=...) was ignored, config.hidden_size raised, tokenizer save_pretrained β†’ from_pretrained failed, get_tgt_vocab() raised.

Changes

  • modeling_indictrans.py: KV cache through Cache objects; attention through ALL_ATTENTION_FUNCTIONS (eager, sdpa, flash_attention_2); weight tying and buffer initialisation compatible with transformers v5; beam search no longer reorders the cross-attention cache (it is identical across the beams of a sentence).
  • configuration_indictrans.py, tokenization_indictrans.py: the fixes above.
  • config.json: drop "attn_implementation": "eager", so SDPA is used by default. Pass attn_implementation="eager" or "flash_attention_2" to choose explicitly.

Compatibility

  • Same outputs: with eager attention the forward pass is bit-identical to the previous code in fp32 and fp16 (encoder, teacher forcing, KV-cache decoding, training with dropout, gradients), and greedy, beam and sampled generations are token-identical. SDPA only differs by floating-point rounding.
  • Tested with transformers 4.53.2, 4.57.6 and 5.21.0 on CPU and Apple MPS (smoke-tested with 4.48.3); flash_attention_2 was not tested (no CUDA GPU). Requires transformers >= 4.48; to keep an older version, load with revision= set to the current main commit.
  • Throughput (dist-200M, MPS fp16, 64 sentences, batch 32): greedy +32%, 5-beam search 11Γ— faster on transformers 5.21 than on 4.53.2. Beam search in transformers 4.53.2 also appended runaway tokens to some batched outputs; 4.57 and later do not.

Related: https://github.com/VarunGumma/IndicTransToolkit/issues/28, https://github.com/VarunGumma/IndicTransToolkit/issues/29, https://github.com/AI4Bharat/IndicTrans2/issues/128, toolkit fix https://github.com/VarunGumma/IndicTransToolkit/pull/31

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment