README.md

---
language:
- en
license: apache-2.0
library_name: pytorch
tags:
- depth-estimation
- monocular-depth
- metric-depth
- close-range
- depth-anything-v2
- fine-tuned
- intel-realsense
- d405
- hand-tracking
- virtual-keyboard
- human-computer-interaction
- cvpr2026
pipeline_tag: depth-estimation
model-index:
- name: depth-anything-v2-vits-d405-finetuned
  results:
  - task:
      type: depth-estimation
      name: Monocular Depth Estimation
    dataset:
      name: D405 Hand-Surface Depth
      type: muxiddin19/d405-hand-surface-depth
    metrics:
    - name: MAE (mm)
      type: mae
      value: 3.84
    - name: RMSE (mm)
      type: rmse
      value: 4.8
    - name: delta1 (%)
      type: delta1
      value: 95.96
    - name: abs_rel
      type: abs_rel
      value: 0.008
    - name: SiLog
      type: silog
      value: 0.057
---

<div align="center">

# Depth Anything V2 ViT-S — D405 Close-Range Fine-tuned

### Real-Time Multimodal Fingertip Contact Detection via Depth and Motion Fusion for Vision-Based Human-Computer Interaction

**CVPR 2026**

[Mukhiddin Toshpulatov](https://orcid.org/0000-0002-8819-852X)<sup>1,2,4,5</sup> &middot; Wookey Lee<sup>2</sup> &middot; Suan Lee<sup>3</sup> &middot; Geehyuk Lee<sup>1</sup>

<sup>1</sup>SpaceTop, SoC, KAIST &nbsp;&nbsp; <sup>2</sup>VoiceAI, BMSE, Inha University &nbsp;&nbsp; <sup>3</sup>SoCS, Semyung University &nbsp;&nbsp; <sup>4</sup>Dep. of CE, Gachon University, South Korea &nbsp;&nbsp; <sup>5</sup>Jizzakh branch of the National University of Uzbekistan

[![Paper](https://img.shields.io/badge/CVPR-2026-blue.svg)](#)
[![GitHub](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/muxiddin19/Multimodal-Fingertip-Contact-Detection-via-Depth-and-Motion-Fusion)
[![Dataset](https://img.shields.io/badge/🤗-Dataset-orange.svg)](https://huggingface.co/datasets/muxiddin19/d405-hand-surface-depth)
[![Project Page](https://img.shields.io/badge/Project-Page-green.svg)](https://muxiddin19.github.io/Multimodal-Fingertip-Contact-Detection-via-Depth-and-Motion-Fusion/)

</div>

## Model Description

This is a **fine-tuned** [Depth Anything V2 ViT-S](https://github.com/DepthAnything/Depth-Anything-V2) model, adapted for **metric depth estimation at close range (7–50 cm)** using data from an Intel RealSense D405 stereo depth camera. The model reduces depth estimation error by **68% (12.3 mm → 3.84 mm MAE)** compared to the pre-trained baseline in the operating range relevant to hand-surface interaction.

The fine-tuned model serves as the depth backbone for a real-time vision-based virtual keyboard system that achieves **94.2% contact detection accuracy** at **30 FPS** on consumer hardware.

### Key Improvements over Base Model

| Metric | Pre-trained DA2-ViTS | **This Model (fine-tuned)** | Improvement |
|:-------|:---------------------|:----------------------------|:------------|
| MAE (mm) | 12.3 | **3.84** | 68% reduction |
| δ1 (%) | 87.2 | **95.96** | +8.8 pp |
| RMSE (mm) | 18.4 | **4.8** | 74% reduction |
| abs_rel | 0.042 | **0.008** | 5.3× better |
| SiLog | 0.312 | **0.057** | 5.5× better |

## Usage

### Installation

```bash
pip install torch torchvision timm opencv-python numpy
git clone https://github.com/DepthAnything/Depth-Anything-V2.git
```

### Inference

> **Critical**: `max_depth` must be set to `0.5` at inference. The DPT head computes `depth = sigmoid(features) × max_depth`. Using the default `max_depth=20.0` will produce predictions **40× too large**.

```python
import torch
import cv2
import numpy as np
from depth_anything_v2.dpt import DepthAnythingV2

# Initialize model with correct max_depth
model = DepthAnythingV2(
    encoder='vits',
    features=64,
    out_channels=[48, 96, 192, 384],
    max_depth=0.5  # MUST match training config
)

# Load fine-tuned weights
state_dict = torch.load(
    'depth_anything_v2_vits_d405_finetuned.pth',
    map_location='cpu'
)['model']

# Strip DDP prefix if present
cleaned = {k.replace('module.', ''): v for k, v in state_dict.items()}
model.load_state_dict(cleaned)
model.eval().cuda()

# Predict depth from RGB image
image = cv2.imread('your_image.png')
with torch.no_grad():
    depth_m = model.infer_image(image)  # Returns metric depth in meters

# Convert to millimeters
depth_mm = depth_m * 1000.0
print(f"Depth range: {depth_mm.min():.1f} - {depth_mm.max():.1f} mm")
```

### Batch Inference

```python
from torchvision.transforms import Compose, Normalize, Resize, ToTensor

transform = Compose([
    Resize((518, 518)),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

# Process a batch
images = torch.stack([transform(img) for img in image_list]).cuda()
with torch.no_grad():
    predictions = model(images)  # [B, 1, H, W] metric depth in meters
```

## Training Details

### Architecture

| Component | Specification |
|:----------|:-------------|
| Encoder | ViT-S/14 (DINOv2 backbone) |
| Decoder | DPT head (features=64) |
| Output channels | [48, 96, 192, 384] |
| Depth activation | sigmoid × max_depth (0.5 m) |
| Parameters | ~24.8M |

### Training Configuration

| Parameter | Value | Reference |
|:----------|:------|:----------|
| Base checkpoint | `depth_anything_v2_vits.pth` (relative depth) | — |
| Loss function | SiLog (Scale-Invariant Logarithmic) | Paper §4.1 |
| Optimizer | AdamW (β₁=0.9, β₂=0.999, wd=0.01) | Paper §4.1 |
| Encoder learning rate | 5 × 10⁻⁶ | Supp. §6 |
| Decoder learning rate | 5 × 10⁻⁵ (10× encoder) | Supp. §6 |
| LR schedule | Cosine annealing | Paper §4.1 |
| Effective batch size | 16 (bs=4 × grad_accum=4) | Paper §4.1 |
| Epochs | 80 | Paper §4.1 |
| Total optimizer steps | ~200K | Paper §4.1 |
| Input resolution | 518 × 518 | DA2 default |
| Depth range | 0.001 m – 0.5 m | Paper §3.1 |
| Augmentation | Random horizontal flip, color jitter | Paper §4.2 |
| Normalization | ImageNet mean/std | DA2 standard |
| Hardware | 1× NVIDIA RTX 3090 (24 GB) | — |
| Training time | ~8.2 hours | — |

### Why Relative Pretrained Weights (Not Metric)?

The base `depth_anything_v2_vits.pth` (relative depth) was chosen over `depth_anything_v2_metric_vits.pth` (Hypersim-pretrained metric) because:

- The metric checkpoint is trained on indoor scenes at 0.5–20 m range, which introduces strong bias away from our 7–50 cm operating range
- The relative checkpoint provides a better feature foundation for domain adaptation to close-range hand-surface geometry
- Empirically, fine-tuning from relative weights converged faster and achieved lower validation loss

### Training Dataset

[D405 Hand-Surface Depth Dataset](https://huggingface.co/datasets/muxiddin19/d405-hand-surface-depth) — 53,300 RGB-depth pairs from 15 participants at four camera angles (30°, 45°, 60°, 90°), captured with Intel RealSense D405.

| Split | Frames | Participants |
|:------|-------:|:-------------|
| Train | 42,640 (80%) | P01, P03, P04, P06, P07, P09–P13, P18 + supplementary |
| Validation | 5,330 (10%) | P05, P16 + supplementary |
| Test | 5,330 (10%) | P08, P17 + supplementary |

Splits are **participant-stratified** (no identity leakage).

## Evaluation

### Depth Metrics on Test Set

Evaluated on 9,021 held-out frames from 2 participants (P05, P08) across all 4 camera angles.

| Metric | Pre-trained | **Fine-tuned (this model)** |
|:-------|:------------|:----------------------------|
| MAE (mm) | 12.3 | **3.84** |
| δ1 (%) | 87.2 | **95.96** |
| RMSE (mm) | 18.4 | **4.8** |
| abs_rel | 0.042 | **0.008** |
| SiLog | 0.312 | **0.057** |

### Downstream Task: Contact Detection

When used as the depth backbone for fingertip contact detection with a velocity-gated hysteresis state machine:

| Metric | Value |
|:-------|:------|
| Contact detection accuracy | 94.2% |
| F1-score | 94.4% |
| False positive rate | 4.2% |
| Typing speed (WPM) | 45.6 |
| Character error rate | 3.1% |
| Inference speed | 30 FPS (RTX 3060 Ti) |

## Model Files

| File | Size | Description |
|:-----|:-----|:------------|
| `depth_anything_v2_vits_d405_finetuned.pth` | ~284 MB | Fine-tuned checkpoint (contains `model`, `optimizer`, `epoch`, `previous_best`) |
| `config.json` | — | Model configuration |

### Checkpoint Contents

```python
checkpoint = torch.load('depth_anything_v2_vits_d405_finetuned.pth')
checkpoint.keys()  # ['model', 'optimizer', 'epoch', 'previous_best']

# To extract model weights only:
model_weights = checkpoint['model']
```

## Limitations

- **Depth range**: Optimized for 7–50 cm. Performance degrades significantly outside this range.
- **Scene type**: Trained on hand-surface typing scenes. May not generalize well to arbitrary indoor/outdoor scenes.
- **max_depth dependency**: Inference MUST use `max_depth=0.5`. Mismatched values produce incorrect depth scales.
- **Camera specificity**: Depth supervision comes from Intel RealSense D405. Performance may vary when applied to depth data from other sensors.
- **Lighting**: Trained primarily under controlled LED lighting (~5000K, 800+ lux). Performance in extreme lighting conditions is not validated.

## Intended Use

### Primary Use Cases

- Real-time fingertip contact detection for vision-based virtual keyboards
- Close-range metric depth estimation for hand-object interaction
- Monocular depth estimation in tabletop AR/VR scenarios
- Research on depth model adaptation for specialized domains

### Out-of-Scope Use

- General-purpose depth estimation at arbitrary ranges
- Outdoor or long-range depth estimation
- Safety-critical applications without additional validation

## Citation

```bibtex
@inproceedings{toshpulatov2026realtime,
  title={Real-Time Multimodal Fingertip Contact Detection via Depth and Motion
         Fusion for Vision-Based Human-Computer Interaction},
  author={Toshpulatov, Mukhiddin and Lee, Wookey and Lee, Suan and Lee, Geehyuk},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision
             and Pattern Recognition (CVPR)},
  year={2026}
}
```

## Acknowledgements

- [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) — base model architecture and pre-trained weights
- [DINOv2](https://github.com/facebookresearch/dinov2) — vision transformer backbone
- [Intel RealSense](https://github.com/IntelRealSense/librealsense) — D405 depth camera for ground truth supervision

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).