Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

About

High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language models (LLMs). While recent data augmentation techniques have facilitated the creation of large-scale benchmarks, existing LLM-generated datasets often suffer from limited reliability, diversity, and scalability. To address these challenges, we introduce PuzzleClone, a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. Our approach features three key innovations: (1) encoding seed puzzles into structured logical specifications, (2) generating scalable variants through systematic variable and constraint randomization, and (3) ensuring validity via a reproduction mechanism. Applying PuzzleClone, we construct PC-83K, a benchmark comprising over 83K diverse and programmatically validated puzzles. The generated puzzles span a wide spectrum of difficulty and formats, posing significant challenges to current state-of-the-art models. Experimental results show that post training (SFT and RL) on PC-83K yields substantial improvements not only on the testset but also on various logic and mathematical benchmarks. Post training raises average performance on PC-83K from 14.5 to 66.0 and delivers consistent improvements across 7 logic and mathematical benchmarks up to 18.4 absolute percentage points (SATBench from 51.6 to 70.0). Our code and data are available at https://github.com/HiThink-Research/PuzzleClone.

Kai Xiong, Yanwei Huang, Rongjunchen Zhang, Kun Chen, Haipang Wu, Yingcai Wu• 2025

Related benchmarks

TaskDatasetResultRank
Mathematical ReasoningAMC 2023
Accuracy67.5
104
Logical reasoningPC-83K 1.0 (test)--
23
Logical reasoningBBEH Mini
Accuracy17
6
Logical reasoningPC-83K (Normal)
Accuracy71
5
Logical reasoningPC-83K Hard
Accuracy61
5
Logical reasoningPC-SL-35K
Accuracy55.3
5
Logical reasoningSatBench
Accuracy70
5
Showing 7 of 7 rows

Other info

Follow for update