---
license: mit
tags:
  - visual-question-answering
  - VQA
  - vilt
  - transformer
  - vision-language
  - inclusive-ai
datasets:
  - vizwiz
language:
  - en
library_name: transformers
pipeline_tag: visual-question-answering
model-index:
  - name: vilt-vqa-vizwiz
    results:
      - task:
          type: visual-question-answering
          name: Visual Question Answering
        dataset:
          name: VizWiz
          type: vizwiz
        metrics:
          - name: Accuracy
            type: accuracy
            value: 29.01%
          - name: BLEU-1
            type: bleu
            value: 0.3017
---

# ViLT VQA (Fine-tuned on VizWiz)

This model is a fine-tuned version of [ViLT](https://arxiv.org/abs/2102.03334) (Vision-and-Language Transformer) on the **VizWiz** dataset—a collection of real-world visual questions submitted by blind and visually impaired users.

ViLT is a **lightweight and efficient VLM** that aligns text and image embeddings via a transformer encoder without using an explicit visual feature extractor (e.g. CNN or ViT), resulting in faster inference and reduced computational cost.

## Model Details

- **Base Model**: `dandelin/vilt-b32-finetuned-vqa`
- **Fine-tuned on**: Sample of VizWiz VQA dataset
- **Framework**: Hugging Face Transformers (PyTorch)
- **Use Case**: Assistive VQA systems for accessibility and inclusion

## Intended Use

Designed for Visual Question Answering in practical, assistive settings. Suitable for low-latency deployments where model speed is critical.

### Example Usage

```python
from transformers import ViltProcessor, ViltForQuestionAnswering
from PIL import Image
import requests

processor = ViltProcessor.from_pretrained("Zagarsuren/vilt-finetuned-vizwiz")
model = ViltForQuestionAnswering.from_pretrained("Zagarsuren/vilt-finetuned-vizwiz")

image = Image.open(requests.get("https://example.com/image.jpg", stream=True).raw)
question = "What colour is the jacket?"

encoding = processor(image, question, return_tensors="pt")
outputs = model(**encoding)
predicted_answer = model.config.id2label[outputs.logits.argmax(-1).item()]
print(predicted_answer)
````

## Evaluation Results

| Metric              | Score |
| ------------------- | ----- |
| Accuracy            | 29.01% |
| BLEU-1              | 0.3017  |
| Response Time (avg) | 13.93ms   |

ViLT offers faster inference than larger VLMs (e.g. Florence-2), making it ideal for edge deployment and resource-constrained environments. However, its performance is comparatively lower on unanswerable and complex reasoning tasks.

## Limitations

* Weaker performance on **complex and compositional reasoning**
* Struggles with **low-quality or cluttered images** typical in VizWiz
* May produce uncertain answers for ambiguous or unanswerable questions

## Citation

If you use this model, please cite:

```
@{
  title={VisionAid-VQA: Inclusive Visual Question Answering Using Deep Learning and Multimodal Attention Mechanisms},
  author={Zagarsuren Sukhbaatar},
  year={2025},
  url={https://huggingface.co/Zagarsuren/vilt-finetuned-vizwiz}
}
```

## License

MIT License