Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Feed-forward Motion In-betweening for Any 4D

About

4D dynamics (3D geometry evolving over time) is a fundamental representation of the physical world and plays a crucial role in world modeling (e.g., animation and games). Owing to the scarcity of large-scale, long-horizon 4D mesh data with arbitrary shapes, early text-to-4D methods rely on distillation or test-time optimization from video diffusion priors, making inference prohibitively slow. Recent feed-forward generators greatly reduce inference cost but offer limited spatiotemporal controllability, and short-horizon generation often leads to error accumulation in long-horizon sequences. We propose a novel feed-forward in-betweening framework for arbitrary 4D meshes with keyframe conditioning. Building on universal mesh-animation latents, we introduce a frame-wise mesh VAE that encodes each frame into topology-agnostic latent tokens anchored by a reference mesh for keyframe conditioning. We further introduce a keyframe-conditioned rectified flow model with an MMDiT backbone that synthesizes non-keyframe frames conditioned on sparse keyframes. Experiments show strong performance and improved controllability on both DyMesh16 and DyMesh32 benchmarks.

Hiroki Nishizawa, Hubert P. H. Shum, Yoshihiro Fukuhara, Hirokatsu Kataoka, Shigeo Morishima• 2026

Related benchmarks

TaskDatasetResultRank
Video-to-4D Motion Generation4D Generation Evaluation Set--
12
4D motion in-betweeningDyMesh16 short-horizon
RMSE1.852
9
Text-and-Mesh-Driven Motion In-betweening4D Generation Evaluation Set
Speed4
4
3D + Video-to-4D Motion Generation4D Generation Evaluation Set--
4
Long-horizon 4D motion in-betweeningDyMesh32 (100 samples)
RMSE2.132
3
Motion In-betweeningDyMesh32 long-horizon (test)
Naturalness3.85
3
Motion In-betweeningDyMesh16 short-horizon (test)
Naturalness Score3.74
3
Showing 7 of 7 rows

Other info

Follow for update