README.md

metadata

title: Sanjaya Engine
emoji: 🛰️
colorFrom: gray
colorTo: yellow
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
pinned: true
license: apache-2.0
models:
  - robbyant/lingbot-map
  - depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf
  - depth-anything/Depth-Anything-V2-Metric-Outdoor-Small-hf
  - google/owlv2-base-patch16-ensemble
tags:
  - 3d-reconstruction
  - slam
  - scene-graph
  - open-vocabulary-detection
  - computer-vision
short_description: One ordinary camera, a 3D mental map. Open research engine.

Sanjaya engine

One ordinary camera, a 3D mental map. Upload a short walk-through video or a few photos. Sanjaya rebuilds the space in 3D at real-world scale, finds doors, stairs, people and other objects, keeps a compact mental map, and shows where the map ends.

The open research engine of Sanjaya: turning any camera into a shared 3D mental map for teams entering unknown spaces (disaster response, inspection, situational awareness; never targeting).

Pipeline

Stage	Model / method	Adds
1. Geometry	LingBot-Map streaming reconstruction	3D points + camera path
2. Scale	Depth Anything V2 Metric + pixel-vote alignment (ours)	metres
3. Objects	OWLv2 open-vocabulary detection + 3D lifting and multi-view fusion (ours)	objects in 3D with evidence
4. Mental map	places graph, Hydra-inspired (ours)	KB-sized map for people and robots
5. Coverage	frontier grid (ours)	floor seen, unexplored edges

Code: sanjaya/ (one module per stage) and app.py (GPU stage + interface).

Mission bundle (sanjaya_mission.zip)

All 3D data in one frame: origin = first camera, +Y up, -Z = first forward direction, metres.

• scene.glb point cloud, camera path, object markers
• mental_map.json places, objects, edges (path, shortcut, near)
• trajectory.json camera centres and poses
• manifest.json stats, objects (position, size, views, evidence box), frontiers, settings
• crops/ best view of each object; keyframes/ RGB + per-pixel 3D map (*_xyz.f32, float32 [h,w,3])
• floorplan.png, mentalmap.png

API

import { Client, handle_file } from "@gradio/client";
const app = await Client.connect("netha01/sanjaya-engine", { hf_token: process.env.HF_TOKEN });
const r = await app.predict("/map", {
  image_files: [], video_file: { video: handle_file(videoUrl) },
  scene: "Indoor", things: "door, staircase, person, fire extinguisher",
  fps: 6, max_frames: 96, keyframes: 12, conf_pct: 50, min_score: 0.3,
});
// r.data: [scene.glb, floorplan, mentalmap, objects gallery, status, bundle.zip, stats]

Limits

Scale from one camera is estimated (see the reported spread). Up to 128 frames per run. Glass, mirrors, blank walls and darkness reduce quality. Rooms and live streaming are next.

Credits

LingBot-Map (Robbyant, Apache-2.0) · Depth Anything V2 (Yang et al. 2024) · OWLv2 (Minderer et al. 2023, Apache-2.0) · ideas from Hydra (MIT SPARK), ConceptGraphs, frontier exploration (Yamauchi 1997) · torchvision-free preprocessing and ZeroGPU pattern following the community LingBot-Map Space by WilliamQM (Apache-2.0). Sanjaya Engine: Apache-2.0.