Software Engineering Skill Evaluation

Benchmarks

Dataset Name	SOTA Method	Metric
SWE-Skills-Bench (slo-implementation)	S2L	Pass Rate (out of 10)60	3	1mo ago
SWE-Skills-Bench python-anti-patterns	Full Skill Text	Pass Rate4	3	1mo ago
SWE-Skills-Bench implementing-agent-modes	Full Skill Text	Pass Rate30	3	1mo ago
SWE-Skills-Bench (changelog-automation)	Full Skill Text	Pass Rate60	3	1mo ago
SWE-Skills-Bench performance-optimization v3	Vanilla LLM	Pass Rate0	3	1mo ago
SWE-Skills-Bench bazel-build-optimization	Vanilla LLM	Pass Rate10	3	1mo ago
SWE-Skills-Bench python-resilience	Vanilla LLM	Pass Rate30	3	1mo ago
SWE-Skills-Bench gitops-workflow	Vanilla LLM	Pass Rate0	3	1mo ago
SWE-Skills-Bench bash-defensive-patterns	Vanilla LLM	Pass Rate50	3	1mo ago
SWE-Skills-Bench (fix)	S2L	Pass Rate2	3	1mo ago
SWE-Skills-Bench python-observability	Vanilla LLM	Pass Count7	3	1mo ago
SWE-Skills-Bench turborepo	S2L	Pass Rate30	3	1mo ago
SWE-Skills-Bench python-background-jobs	Vanilla LLM	Pass Rate7	3	1mo ago
SWE-Skills-Bench service-mesh-observability	Vanilla LLM	Pass Rate0	3	1mo ago
SWE-Skills-Bench istio-traffic-management	Full Skill Text	Pass Rate8	3	1mo ago
SWE-Skills-Bench (tdd-workflow)	S2L	Pass Rate4	3	1mo ago
SWE-Skills-Bench distributed-tracing	S2L	Pass Rate40	3	1mo ago
SWE-Skills-Bench similarity-search-patterns	Vanilla LLM	Pass Rate0	3	1mo ago
SWE-Skills-Bench prompt-engineering-patterns	Vanilla LLM	Pass Rate30	3	1mo ago
SWE-Skills-Bench (gitlab-ci-patterns)	S2L	Pass Rate7	3	1mo ago
SWE-Skills-Bench risk-metrics-calculation	Vanilla LLM	Pass Rate5	3	1mo ago
SWE-Skills-Bench (Aggregate)	S2L	Pass Rate65	3	1mo ago

Showing 22 of 22 rows