Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations

About

Generative models transform random noise into images; their inversion aims to transform images back to structured noise for recovery and editing. This paper addresses two key tasks: (i) inversion and (ii) editing of a real image using stochastic equivalents of rectified flow models (such as Flux). Although Diffusion Models (DMs) have recently dominated the field of generative modeling for images, their inversion presents faithfulness and editability challenges due to nonlinearities in drift and diffusion. Existing state-of-the-art DM inversion approaches rely on training of additional parameters or test-time optimization of latent variables; both are expensive in practice. Rectified Flows (RFs) offer a promising alternative to diffusion models, yet their inversion has been underexplored. We propose RF inversion using dynamic optimal control derived via a linear quadratic regulator. We prove that the resulting vector field is equivalent to a rectified stochastic differential equation. Additionally, we extend our framework to design a stochastic sampler for Flux. Our inversion method allows for state-of-the-art performance in zero-shot inversion and editing, outperforming prior works in stroke-to-image synthesis and semantic image editing, with large-scale human evaluations confirming user preference.

Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, Wen-Sheng Chu• 2024

Related benchmarks

TaskDatasetResultRank
Image EditingPIE-Bench
PSNR22.03
116
Image EditingPIE-Bench (test)
PSNR20.2
46
Image Semantic EditingPIE-Bench (test)
PSNR20.82
18
Image Editing1024 x 1024 resolution
Runtime (4090, s)117.2
14
Image EditingEditEval v2
LPIPS0.5659
14
Image EditingSNR-Bench 1.0 (test)
Reward Model Structural Score3.36
12
Text-Guided Image EditingPIE-Bench (test)
Structure Distance4.17e+4
8
Text-based Image EditingComplex-PIE-Bench
CLIP-T26.59
7
Text-based Image EditingPIE-Bench++
CLIP-T24.59
7
Image Inversion and ReconstructionDCI (Densely Captioned Images) (first 1K images)
LPIPS0.5044
7
Showing 10 of 15 rows

Other info

Follow for update