RealWeather Realistic and Scene-Faithful Weather Translation with Driving World Models

Yuwei Ning1 Liangzhi Wang1 Yi Xiao1 Zhenhua Wu1 Yun Pang1 Mingkun Chang1 Jichang LI1 Guanbin Li1,2✉
1Sun Yat-sen University 2Shenzhen Loop Area Institute

arXiv Code

TL;DR A driving world model for realistic and scene-faithful weather translation in driving videos

Single Weather Translation

RealWeather supports translating clear-weather driving videos into rainy, snowy, and foggy conditions while faithfully preserving the original scene structure and dynamics.

Weather Strength Control

RealWeather enables control over the intensity of each weather effect, ranging from light to extreme conditions.

🌫 Fog Strength Control



🌧 Rain Strength Control



❄️ Snowflake Strength Control

Mixed Weather Conditions

RealWeather can combine multiple weather effects simultaneously, enabling complex real-world scenarios such as rain with fog, or snow with rain.

☀️ Sunny (input) → 🌧 + ❄️ Rain & Snowflake



☀️ Sunny (input) → ❄️ + 🌫 Snowflake & Fog



☀️ Sunny (input) → 🌧 + 🌫 Rain & Fog

Weather Removal

RealWeather supports bidirectional translation — it can also remove adverse weather effects from driving videos, restoring clear-weather visibility.

🌫 → ☀️ Fog Removal



🌧 → ☀️ Rain Removal



❄️ → ☀️ Snow Removal

Out-of-Distribution (OOD) Scenarios

RealWeather generalizes to out-of-distribution scenarios never seen during training, including simulation inputs, day-to-night transitions, and fisheye camera views.

🖥 Simulation Inputs



🌤 → 🌙 Day-to-Night Translation


📷 Fisheye Camera

Method

Overview of the RealWeather method

Overview of RealWeather. (A) Progressive Realism Bootstrapping enables the model to learn weather effects directly from real-world videos, progressively adapt to real-style inputs, and support bidirectional weather editing. (B) Scene-Fidelity RL Optimization further improves structural fidelity with segmentation-based rewards, reducing hallucinations while preserving the learned weather translation capability.


Data-Pipeline

RealWeather data construction pipeline

Overview of the Pseudo-Clear Generation pipeline of RealWeather. Adverse-Weather Preprocessing removes transient precipitation artifacts using established adverse-weather removal models. Clear-Control Extraction constructs a clear-weather anchor frame and weather-agnostic geometric controls. Finally, Conditional Generation leverages a state-of-the-art driving world model to propagate the clear-weather appearance from the anchor frame throughout the video, producing a scene-consistent pseudo-clear counterpart.