qhjqhj00/research-skills-pool/aria-nerf-eval

aria-nerf-eval

Aria-NeRF: Multimodal Egocentric View Synthesis — Sun et al. (2023) (arXiv:2311.06455, 2023)

What this evaluates

Evaluates NeRF-based models on their ability to synthesize novel views from egocentric, multimodal sensor data captured in dynamic real-world environments. It probes how well current neural rendering methods handle temporal dynamics, lens distortion, and non-visual cues like IMU and gaze.

Datasets

• Aria-NeRF Dataset — total ?; splits: test (-1)

Metrics

• PSNR (primary) — range: other
• Peak Signal-to-Noise Ratio computed in decibels between rendered and ground-truth images. Higher values indicate better reconstruction fidelity.
• SSIM — range: [0, 1]
• Structural Similarity Index measuring perceived structural changes between rendered and ground-truth images. Values range from -1 to 1, with 1 indicating identical structure.
• LPIPS — range: [0, 1]
• Learned Perceptual Image Patch Similarity using deep features to measure perceptual distance between images. Lower values indicate higher perceptual similarity.

Input / output format

Input: Multimodal sensor streams from Aria Glasses including RGB frames, IMU, eye-tracking, audio, GPS, Wi-Fi, Bluetooth, and environmental data.

Output: Rendered novel-view images (either de-distorted or fisheye depending on the baseline model) generated by the NeRF representation.

Scoring recipe

def evaluate(rendered_img, gt_img):
    mse = np.mean((rendered_img - gt_img) ** 2)
    psnr = 10 * np.log10(255**2 / mse)
    ssim = compute_ssim(rendered_img, gt_img)
    lpips = compute_lpips(rendered_img, gt_img)
    return {'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips}

Common pitfalls

• Nerfacto outputs de-distorted images while NeuralDiff outputs fisheye images, requiring careful alignment or distortion-aware comparison for fair metric calculation.
• Static image metrics (PSNR/SSIM) may not capture temporal consistency or background/foreground disentanglement quality in dynamic egocentric scenes.

Evidence (verbatim from paper)

In terms of PSNR, SSIM, and LPIPS metrics, NeuralDiff generally surpasses Nerfacto across various scenarios.

Citation

@misc{sun2023arianerf,
  title={Aria-NeRF: Multimodal Egocentric View Synthesis},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2311.06455}
}

• arXiv: 2311.06455