Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

StyleMaster: Stylize Your Video with Artistic Generation and Translation

About

Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at https://zixuan-ye.github.io/stylemaster

Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, Wenhan Luo• 2024

Related benchmarks

TaskDatasetResultRank
Image Style TransferCurated image style transfer (test)
CSD-Score0.45
4
Stylized Video GenerationStyleCrafter 192 style-prompt pairs (test)
CLIP-Text0.305
3
Video Style TransferStylized Video Generation Prompts Table 7
CSD-Score0.434
3
Style TransferCurated Style Transfer (test)
VCSD0.29
3
Style-image-guided Video Stylization90-video stylization 1.0 (test)
CSD Score0.198
3
Video StylizationUser Study Style Image Condition (test)
Style Consistency1.17
2
Showing 6 of 6 rows

Other info

Code

Follow for update