X-Lens
Real-Time Metric Depth Estimation
with Heterogeneous Cameras

¹D-Robotics ²The University of Tokyo ³Soochow University

^*Equal Contribution ^†Project Lead

Tech Report arXiv Live Demo Code 🤗 Dataset

TL;DR A compact 0.04B feed-forward model that takes a calibrated mix of fisheye and pinhole views and predicts dense metric depth plus a global scale in one pass, running in real time on six-view heterogeneous rigs.

Abstract

One compact model, fisheye and pinhole, one metric space.

We present X-Lens, a compact feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. To support real-time downstream perception, X-Lens is built around a geometry-aware heterogeneous camera formulation with two key components. Learnable calibration tokens provide a coarse alignment between fisheye and pinhole projective spaces, while a Jacobian-parameterized distortion bias injected into cross-attention models local projection changes and promotes cross-camera consistency, enabling robust generalization with only 0.04B parameters and up to 41 FPS. The model predicts dense depth together with a global metric scale, avoiding auxiliary reconstruction targets that increase computation and optimization complexity.

To learn such cross-camera generalization at scale and depth, X-Lens is trained on multiple public datasets and OmniScene, our newly released large-scale synthetic dataset containing approximately 266K synchronized six-view frames, 1.7M individual images, and 103 indoor and outdoor scenes. Extensive experiments on both real-world and synthetic indoor and outdoor datasets demonstrate superior heterogeneous-camera metric depth accuracy, reducing AbsRel by 25.4% on OmniScene-Full over the strongest baseline while using 88.9% fewer parameters, with competitive performance on conventional fisheye-only and pinhole-only settings.

X-Lens teaser: metric point clouds from six-view heterogeneous rigs for indoor and outdoor scenes, with RGB inputs and predicted depth maps — Metric point clouds projected from calibrated six-view heterogeneous cameras (top), with RGB inputs and predicted depth maps from pinhole and fisheye views (bottom).

Quantitative Results

A balanced view of accuracy, efficiency, and real-time performance across three benchmarks.

Single-view

KITTI360

1 view

Multi-view

ETH3D

2 views

Heterogeneous

OmniScene-Full

6 views

How to read Each axis is normalized relative to the best result in its table; farther from the center is better. Hover a model line or tap its legend to isolate it.

Qualitative Comparison

State of the art across the board. Qualitative comparisons on two real captures: a surround rig with four fisheye cameras, and a rig with six pinhole cameras.

Fisheye · Real 4-camera rig

RGB Input

DAC

UniDAC

Ours

Pinhole · Real 6-camera rig

RGB Input

DepthAnything3-Small

VGGT-Omega

Ours

Method	Params ↓	Scale AbsRel ↓	AbsRel ↓	RMSE ↓	δ₁ ↑	τ₁.₀₃ ↑	FPS ↑
MapAnything	1.23B	0.3701	0.1746	2.1834	0.7357	0.1647	5
DepthAnyCamera	0.06B	0.2571	0.2066	2.3981	0.7653	0.2411	1
UniDAC	0.36B	0.2506	0.1368	2.6125	0.8156	0.2511	1
X-Lens Ours	0.04B	0.1181	0.1021	1.5993	0.8982	0.3724	22

Heterogeneous-camera evaluation on OmniScene-Full with six input views (4 fisheye + 2 pinhole). FPS measured at 504×798. Best in gradient.

Interactive Demo

Drag to orbit fused metric point clouds predicted by X-Lens, from real 4-fisheye captures and six-camera heterogeneous scenes. Use the Measure button to click any two points and read out their true metric distance in meters, and Crop to box-select and remove regions such as the ceiling. The strip below shows the RGB inputs behind each cloud.

Point Cloud Viewer

Point Cloud 1

Loading point cloud…

OmniScene Dataset

A large-scale synthetic dataset for multi-view metric depth with heterogeneous cameras, released openly on Hugging Face.

103complex scenes

564motion sequences

266Ksix-view frames

1.7M+calibrated images

OmniScene scene categories rendered from the six-camera rig: apartment, bazaar, gym, museum, office, palace, parking, railway — Eight of the scene categories, each rendered from the calibrated six-camera rig: four 180° fisheye and two pinhole views, fully synchronized.

Scenes span residential, commercial, industrial, urban, and sci-fi environments built from professionally authored Kujiale and Unreal Engine assets. Every frame carries dense, noise-free metric ground truth: pixel-accurate z-depth, validity masks, sky indicators, and per-frame OpenCV-convention intrinsics and extrinsics. Validation is drawn from the training scenes, while the test set is held out on fully disjoint scenes for evaluation on unseen environments.

🤗 Download on Hugging Face

Application in Vision-Action Model

X-Lens as a geometry-aware visual encoder for multi-view robotic manipulation.

Speed x1

SLAM and loco-manipulation applications: coming soon.

Citation

If X-Lens is useful in your research, please cite our paper.

@article{zhou2026xlens,
  title   = {X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras},
  author  = {Zhou, Heng and Liu, Shuhong and He, Yonghao and Zhang, Bohao and Fu, Fa and
             Hou, Chenhui and Hou, Xianbao and Han, Lijun and Yang, Cong and Sui, Wei},
  journal = {arXiv preprint arXiv:2607.12993},
  year    = {2026},
}

Correspondence

wei.sui@d-robotics.cc

X-Lens Real-Time Metric Depth Estimation with Heterogeneous Cameras