Neural Rendering in VR: A Technical Overview

VR rendering is arguably the hardest real-time graphics problem. You need to render two viewpoints simultaneously — left and right eye — at a combined resolution exceeding 4K, while maintaining at least 90 FPS to avoid motion sickness. Traditional approaches like Multi-View Rendering (MVR) help, but they can't overcome the fundamental bandwidth problem: there are simply too many pixels to shade.

At 91FPS, we've developed a neural rendering pipeline that renders at 1080p internally and upscales to 4K-per-eye using a lightweight transformer model. The result: 120 FPS VR on consumer GPUs with visual quality indistinguishable from native 4K rendering.

The Foveated Neural Upscaler

Our approach combines two techniques — foveated rendering and neural super-resolution — into a single model we call the Foveated Neural Upscaler (FNU).

Traditional foveated rendering renders the center of the display (where the user is looking) at full resolution and the periphery at reduced resolution. Our FNU takes this further: it renders the entire scene at 1080p and uses a vision transformer to reconstruct 4K output, with reconstruction quality weighted by eye-tracking data.

Architecture

Eye Tracking Data (200Hz)
        │
        ▼
┌──────────────────────┐
│   Foveation Map      │
│ (Gaussian falloff)   │
└──────────┬───────────┘
           │
1080p ────►├────────────────┐
Frame      │                │
           │     ┌──────────▼──────────┐
           │     │  Vision Transformer  │
           └────►│  (6 layers, 8 heads) │
                 │  Patch size: 8×8     │
                 └──────────┬──────────┘
                            │
                     ┌──────▼──────┐
                     │ 4K Output   │
                     │ (per eye)   │
                     └─────────────┘

The vision transformer processes 8×8 pixel patches from the 1080p input. The foveation map — derived from 200Hz eye tracking — modulates the attention weights, so patches in the user's focal area receive more transformer capacity than peripheral patches. This is dynamic compute allocation — the model "spends" its FLOP budget where it matters most.

Performance Benchmarks

Measured on an RTX 4080 with a Quest Pro (eye tracking enabled):

Native 4K rendering: 72 FPS per eye, GPU at 98% utilization, 280W power draw
Foveated Neural Upscaler (1080p → 4K): 120 FPS per eye, GPU at 72% utilization, 195W power draw

The FNU model itself runs in 1.8ms on the GPU's tensor cores — faster than the time saved by rendering at 1080p instead of 4K. The net gain is roughly 30% lower frame time with no perceptible quality difference in double-blind A/B testing.

Training Pipeline

We trained the FNU model on a dataset of 2.4 million paired frames — 1080p input, 4K ground truth — captured across 200 VR titles. The dataset includes diverse environments: indoor tactical shooters, open-world exploration games, flight simulators, and medical VR training apps.

Key training detail: we use a perceptually-weighted loss function that combines L1 pixel loss with a VGG-based perceptual loss term. The perceptual loss is itself weighted by the foveation map, so the model learns to prioritize reconstruction quality in the foveal region. Training takes ~72 hours on 8× A100 GPUs.

What's Next

We're exploring temporal transformer models that process sequences of frames rather than individual frames. Early results show a further 15% reduction in inference time by exploiting inter-frame coherence. We're also working on a mobile-optimized FNU targeting standalone VR headsets like the Quest 4, where the power budget is even tighter.

Excited about our VR rendering SDK? Stay tuned for updates.