Pattern Recognition · Volume 183 · 2027
Deflickering Vision-Based Occupancy Networks through Lightweight Spatio-Temporal Correlation
Abstract
Vision-based occupancy networks (VONs) provide an end-to-end solution for reconstructing 3D environments in autonomous driving. However, existing methods often suffer from temporal inconsistencies, manifesting as flickering effects that compromise visual experience and adversely affect decision-making. While recent approaches have incorporated historical data to mitigate the issue, they often incur high computational costs and may introduce noisy information that interferes with object detection.
We propose OccLinker, a novel plugin framework designed to seamlessly integrate with existing VONs for boosting performance. Our method efficiently consolidates historical static and motion cues, learns sparse latent correlations with current features through a dual cross-attention mechanism, and produces correction occupancy components to refine the base network’s predictions. We propose a new temporal consistency metric to quantitatively identify flickering effects. Extensive experiments on two benchmark datasets demonstrate that our method delivers superior performance with negligible computational overhead, while effectively eliminating flickering artifacts.
Method
A plug-in beside a frozen occupancy network
Sparse latent correlation
Instead of fusing dense historical feature volumes, OccLinker compresses historical static cues and inter-frame motion cues into a small set of latent tokens with a lightweight tokenization encoder.
A dual cross-attention module then correlates them with the current static feature: one path links historical static features to the present scene, the other links motion features to it. Static and dynamic correlations stay separate, so moving objects are tracked without blurring the static background.
The base network is never retrained. OccLinker only adds a correction to its output, so it plugs into single-frame and history-aware VONs alike.
Measuring flicker
A temporal consistency metric
Accuracy metrics score each frame in isolation and do not see flicker. We compare consecutive predictions directly: among voxels occupied in both frames, we count how many change their semantic label. S̄m scores voxels of moving object classes and S̄s scores static classes. Both are averaged over scenes; higher means steadier predictions.
Results
Gains on every host network, at almost no cost
| Benchmark | Host VON | IoU | mIoU | S̄m | S̄s |
|---|---|---|---|---|---|
| SurroundOcc | MonoScene | 10.04 → 13.10+3.06 | 1.15 → 1.69+0.54 | 46.53 → 54.21+7.68 | 81.77 → 83.84+2.07 |
| SurroundOcc | SurroundOcc | 31.49 → 33.12+1.63 | 20.30 → 20.67+0.37 | 58.33 → 60.64+2.31 | 91.71 → 92.54+0.83 |
| SurroundOcc | BEVFormer | 30.50 → 30.91+0.41 | 16.75 → 17.09+0.34 | 52.84 → 55.21+2.37 | 84.21 → 85.99+1.78 |
| Occ3D | SurroundOcc | 51.89 → 52.13+0.24 | 7.24 → 10.33+3.09 | 65.35 → 69.60+4.25 | 89.54 → 90.91+1.37 |
| Occ3D | ViewFormer | 70.39 → 70.63+0.24 | 40.46 → 41.30+0.84 | 67.26 → 70.13+2.87 | 86.06 → 87.10+1.04 |
Each row shows the host network alone → with OccLinker. Numbers are from Tables 1 and 2 of the paper; full comparisons with other methods are in the paper.
Negligible overhead
On ViewFormer, OccLinker adds 105 MB of GPU memory (4,582 → 4,687 MB) and about 0.01 s per frame (0.11 → 0.12 s on an NVIDIA L20, 0.13 → 0.14 s on an A100), while mIoU rises from 40.46 to 41.30.
Among the history-aware VONs we compare, it has the highest mIoU, and only the base ViewFormer uses less memory.
More experiments
What else the paper tests
Against another plug-in
CVT-Occ, a cost-volume plug-in, is added to the same host networks. On SurroundOcc it reaches 32.18 IoU (OccLinker: 33.12). On ViewFormer it slightly lowers mIoU (40.35 vs. 40.46), while OccLinker raises it to 41.30.
Image backbones
Gains hold on CNN and Transformer backbones: +1.63 IoU / +0.37 mIoU with R101-DCN, +0.33 / +0.23 with Swin-T, and +0.25 / +0.16 with ViT.
Training-free temporal filtering
Smoothing the base predictions over time raises S̄m to 67.22 but drops mIoU to 17.67; with ego-motion compensation mIoU only recovers to 20.13. OccLinker improves accuracy and consistency together (33.12 IoU, 20.67 mIoU).
Which correlations matter
Removing the historical-static, current-static or motion correction lowers mIoU to 19.98–20.10 (full model: 20.67). Swapping the roles of queries and keys/values drops IoU from 33.12 to 31.97.
Motion cue
Frame differences beat optical flow and cost less: 33.12 vs. 32.80 IoU, 20.67 vs. 20.27 mIoU, and 6.18 ms vs. 9.22 ms per frame. Raw images as motion input reach only 19.45 mIoU.
Number of previous keyframes
One previous keyframe (L = 1) gives the best accuracy; L = 2 is slightly steadier (S̄m 61.00) but less accurate. Training memory grows by about 15 GB per extra frame, and L = 4 runs out of memory.
Where to aggregate in ViewFormer
Late aggregation beats early aggregation: 41.30 vs. 40.58 mIoU, with less memory (4,687 vs. 4,780 MB) and lower latency (121 vs. 159 ms).
Theoretical analysis
With the base network frozen, lowering the training loss raises a lower bound on how much information the correction ΔOt carries about the base network’s residual error (Theorem 1).
Qualitative results
Stable predictions under occlusion and hard conditions
BibTeX
@article{yu2027occlinker,
title = {Deflickering Vision-Based Occupancy Networks through Lightweight
Spatio-Temporal Correlation},
author = {Yu, Fengcheng and Xu, Haoran and Xia, Canming
and Zong, Ziyang and Tan, Guang},
journal = {Pattern Recognition},
volume = {183},
pages = {114988},
year = {2027},
doi = {10.1016/j.patcog.2026.114988}
}