Feed-forward Motion In-betweening for Any 4D
About
4D dynamics (3D geometry evolving over time) is a fundamental representation of the physical world and plays a crucial role in world modeling (e.g., animation and games). Owing to the scarcity of large-scale, long-horizon 4D mesh data with arbitrary shapes, early text-to-4D methods rely on distillation or test-time optimization from video diffusion priors, making inference prohibitively slow. Recent feed-forward generators greatly reduce inference cost but offer limited spatiotemporal controllability, and short-horizon generation often leads to error accumulation in long-horizon sequences. We propose a novel feed-forward in-betweening framework for arbitrary 4D meshes with keyframe conditioning. Building on universal mesh-animation latents, we introduce a frame-wise mesh VAE that encodes each frame into topology-agnostic latent tokens anchored by a reference mesh for keyframe conditioning. We further introduce a keyframe-conditioned rectified flow model with an MMDiT backbone that synthesizes non-keyframe frames conditioned on sparse keyframes. Experiments show strong performance and improved controllability on both DyMesh16 and DyMesh32 benchmarks.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Video-to-4D Motion Generation | 4D Generation Evaluation Set | -- | 12 | |
| 4D motion in-betweening | DyMesh16 short-horizon | RMSE1.852 | 9 | |
| Text-and-Mesh-Driven Motion In-betweening | 4D Generation Evaluation Set | Speed4 | 4 | |
| 3D + Video-to-4D Motion Generation | 4D Generation Evaluation Set | -- | 4 | |
| Long-horizon 4D motion in-betweening | DyMesh32 (100 samples) | RMSE2.132 | 3 | |
| Motion In-betweening | DyMesh32 long-horizon (test) | Naturalness3.85 | 3 | |
| Motion In-betweening | DyMesh16 short-horizon (test) | Naturalness Score3.74 | 3 |