Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

Who Wrote this Code? Watermarking for Code Generation

About

Since the remarkable generation performance of large language models raised ethical and legal concerns, approaches to detect machine-generated text by embedding watermarks are being developed. However, we discover that the existing works fail to function appropriately in code generation tasks due to the task's nature of having low entropy. Extending a logit-modifying watermark method, we propose Selective WatErmarking via Entropy Thresholding (SWEET), which enhances detection ability and mitigates code quality degeneration by removing low-entropy segments at generating and detecting watermarks. Our experiments show that SWEET significantly improves code quality preservation while outperforming all baselines, including post-hoc detection methods, in detecting machine-generated code text. Our code is available in https://github.com/hongcheki/sweet-watermark.

Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, Gunhee Kim• 2023

Related benchmarks

TaskDatasetResultRank
Code GenerationHumanEval
Pass@142.01
850
Code GenerationHumanEval (test)
Pass@118.9
444
Code GenerationHumanEval+
Pass@137.82
189
Code GenerationMBPP+
Pass@142.79
122
Text WatermarkingC4
PPL10.0633
27
Paraphrase Attack RobustnessBookSum
AUC98.49
20
Paraphrase Attack RobustnessC4 RealNewsLike
AUC0.9731
20
Spoofing Attack RobustnessC4 RealNewsLike
AUC0.573
20
Spoofing Attack RobustnessBookSum
AUC0.5136
20
Spoofing attack traceabilityRTP-LX (test)
AUC58.74
20
Showing 10 of 35 rows

Other info

Code

Follow for update