Share your thoughts, 1 month free Claude Pro on usSee more

Plan Generation on 60-task evaluation suite 10 domains

89Code PQS

PII Redaction

Updated 1mo ago

Evaluation Results

Method	Links
PII Redaction 2026.03		89	40	71	95	89	91	100	90	83	95	86.1	84.8	83.5	16.5
PII Redaction 2026.03		85	80	75	88	93	93	89	88	88	100	88.7	88.6	83.5	16.5
Raw Context 2026.03		81	40	67	90	95	88	87	78	91	100	83.2	83.3	0	100
PlanTwin 2026.03		81	75	64	72	98	80	87	67	93	83	81	76.5	100	61.5
PlanTwin 2026.03		79	80	68	72	100	74	79	67	90	91	80.8	77.6	100	61.7
Raw Context 2026.03		79	65	67	76	83	77	87	78	95	89	80.2	76.1	0	100
Raw Context 2026.03		77	77	67	81	85	80	93	84	93	98	84.3	83.1	0	100
Raw Context 2026.03		77	40	60	89	90	87	100	90	82	98	83.4	84.5	0	100
PII Redaction 2026.03		77	40	67	94	95	95	100	93	78	94	85.5	87.2	83.5	16.5
PlanTwin 2026.03		75	69	67	83	100	70	86	73	72	81	79.2	79.9	100	59
PlanTwin 2026.03		71	68	57	72	93	48	68	70	66	77	68.9	60.4	100	58.2
PII Redaction 2026.03		65	75	73	77	78	79	87	75	91	83	78.8	74.7	83.5	16.5
Local (Gemma-3-1B) 2026.03		55	86	50	54	60	57	55	55	55	44	56.7	55.9	100	0
Heuristic (Oracle) 2026.03		53	50	50	78	90	70	59	54	43	84	65.3	78.2	100	55.8
Local (Qwen 3.5-4B) 2026.03		27	40	50	46	58	50	52	70	54	30	48.4	65.6	100	0
Local (SmolLM2-360M)‡ 2026.03		10	50	50	12	38	50	17	33	17	17	28.3	56.7	100	0
Local (Qwen 3.5-0.8B)‡ 2026.03		10	50	50	12	38	50	17	33	17	17	28.3	56.7	100	0
Local (Qwen 3.5-2B) 2026.03		10	50	50	12	38	50	17	33	17	17	28.3	56.7	100	0
Local (Qwen 3.5-9B) 2026.03		10	50	86	12	64	93	73	50	45	60	53.1	64.2	100	0