Calibrating Claims From Coding-Agent Benchmarks: An Evidence Taxonomy
Coding-agent leaderboards increasingly invite broad claims about autonomous software engineering, yet benchmark evidence differs sharply in task unit, repository access, oracle design, interface permissions, curation, and domain coverage. This paper asks what kinds of claims are justified by current public benchmark evidence and where stronger operational claims require additional support. I performed a scoping taxonomy using AlexandrAI graph papers, primary benchmark papers, official SWE-bench documentation, and benchmark-risk studies screened on 2026-06-23. The synthesis yields a seven-level claim ladder from isolated function synthesis through validated issue repair to operational readiness, plus an evidence matrix showing that executable oracles and repository repair are now well represented while visual evidence, cross-language repair, contamination controls, and agent-boundary disclosure remain uneven. The result is not a new leaderboard metric; it is a calibration framework for keeping pass rates proportional to the evidence behind them. The central conclusion is that coding-agent evaluations should report the strongest claim their benchmark design supports, not the strongest claim a resolved-rate number can rhetorically suggest.
Introduction
Software-engineering agents are now evaluated with artifacts that range from isolated function synthesis tasks to issue-to-patch repair in full repositories. That breadth is productive, but it also creates a claim-calibration problem. A pass rate on HumanEval, RepoBench, SWE-bench Verified, or a current SWE-bench leaderboard entry does not mean the same thing because each benchmark supplies different evidence about context use, oracle validity, action interface, domain coverage, and human quality control [[cite:chen2021humaneval,liu2023repobench,jimenez2024swebench,openai2024verified,swebench2026site]].
Two AlexandrAI graph papers motivated this study. The first found that repository-scale coding-agent benchmarks cover real maintenance workflows unevenly: execution checks were common, while repository context, issue-to-patch repair, multimodal evidence, and human validation were much less uniformly represented [[cite:alexCoverage]]. The second proposed benchmark cards for coding agents, arguing that pass rates should be accompanied by structured disclosure of task scope, corpus provenance, oracle design, harness permissions, contamination controls, domain coverage, and cost [[cite:alexCards]].
This paper extends that graph context with a narrower question: what claims about autonomous coding agents are justified by current benchmark evidence, and where should benchmark reports stop short? The contribution is a taxonomy rather than an empirical benchmark. It maps benchmark families to seven claim levels and identifies confounds that block direct inference from a single resolved-rate number to broad operational ability.
Method
The study mode is a scoping taxonomy with conceptual synthesis. On 2026-06-23 UTC, I searched the AlexandrAI graph with six short keyword angles: agent systems, software engineering, benchmark evaluation, language model, coding agents, and SWE bench. I screened only abstracts first, fetched the two directly relevant English papers, read their front matter, methods, results, figures, and references, and retained both as graph context [[cite:alexCoverage,alexCards]].
The external sweep used primary sources and official pages. Included benchmark artifacts were HumanEval, MBPP, APPS, CodeXGLUE, MultiPL-E, DS-1000, RepoBench, SWE-bench, SWE-bench Verified, SWE-bench Multimodal, and the current SWE-bench official leaderboard family [[cite:chen2021humaneval,austin2021mbpp,hendrycks2021apps,lu2021codexglue,cassano2022multiple,lai2023ds1000,liu2023repobench,jimenez2024swebench,openai2024verified,yang2024multimodal,swebench2026site]]. Included agent-system and benchmark-risk sources were SWE-agent, Agentless, a code-benchmark prompt-quality study, and a contamination-control paper [[cite:yang2024sweagent,xia2024agentless,siddiq2024fault,jacovi2023contamination]].
I coded each source against eight evidence dimensions: executable oracle, repository context, issue-to-patch repair, language breadth, visual evidence, human quality control, explicit agent boundary, and contamination controls. The coding was conservative: a dimension was marked present only when it was central to the artifact or explicitly discussed by the source. Aggregated counts are therefore descriptive evidence maps, not statistical estimates of field prevalence.
support(c) = Σ d=1 8 I(evidence dimension d is required for claim level c)
Equation 1 defines the minimum support count used in the claim ladder. A benchmark may be excellent for a narrow claim while scoring low on broader dimensions. For example, HumanEval is strong evidence for bounded function synthesis but weak evidence for repository repair; SWE-bench is stronger evidence for issue repair but still limited by public data, Python skew, static test oracles, and agent-scaffold variation [[cite:chen2021humaneval,jimenez2024swebench,openai2024verified,swebench2026site]].
Results
The first result is a seven-level claim ladder. The ladder separates claims about short-program synthesis, benchmark-suite competence, repository context use, issue repair, validated issue repair, multimodal or multilingual repair, and operational readiness. Table 2 states the strongest proportional claim for each level and the evidence still needed before moving to the next level.
The second result is an evidence matrix across sources. Executable oracles are now common in code-generation and repair benchmarks, while repository context and issue-to-patch repair are concentrated in the SWE-bench family. The narrowest evidence dimensions are visual evidence, explicit contamination controls, and benchmark reports that clearly separate the base model from the agent scaffold [[cite:alexCoverage,alexCards,siddiq2024fault,jacovi2023contamination]].
The third result is that four confounds recur across the literature. First, oracle and solvability defects can make a benchmark underestimate or misclassify ability, which is why SWE-bench Verified filtered 500 instances for clearer problem descriptions, correct tests, and solvability [[cite:openai2024verified]]. Second, agent-interface design changes measured performance: SWE-agent improves results through a custom agent-computer interface, while Agentless shows that a simpler localization-repair-validation pipeline can be a strong baseline [[cite:yang2024sweagent,xia2024agentless]]. Third, static public benchmarks are exposed to prompt-quality and contamination issues [[cite:siddiq2024fault,jacovi2023contamination]]. Fourth, domain coverage remains partial: SWE-bench Multimodal and the current SWE-bench Multilingual view address visual and language gaps, but they also show that the original Python text-only repair setting was not representative of all maintenance work [[cite:yang2024multimodal,swebench2026site]].
Discussion
The synthesis supports a conservative rule for reading coding-agent leaderboards: the benchmark defines the claim. A static task with hidden tests can establish useful evidence about a system under a particular harness, but it cannot by itself establish that the system is ready for production maintenance, secure deployment, ambiguous product work, or sustained collaboration with developers. This is not a weakness of narrow benchmarks; it is a reason to state their scope precisely.
The claim ladder also reconciles apparently conflicting findings. SWE-agent and Agentless disagree about the amount of agentic complexity needed for strong SWE-bench performance, but together they show that benchmark scores are properties of composite systems rather than pure model properties [[cite:yang2024sweagent,xia2024agentless]]. OpenAI's SWE-bench Verified work and the official SWE-bench leaderboard similarly show that curation, scaffold choice, benchmark version, and cost views are part of the evidence, not presentation details [[cite:openai2024verified,swebench2026site]].
The implication for benchmark authors is practical. A report should name the claim level it supports, disclose the benchmark card fields proposed by prior graph work, and avoid phrasing that moves silently from a narrow score to broad operational readiness [[cite:alexCards]]. A report should also separate model-only comparisons from scaffold-level or full-system comparisons, because agent permissions, retries, retrieval, validation, and human review can dominate a score in ways that a base-model name alone does not reveal.
This paper has limitations. The corpus is a scoped review, not a complete systematic review of every code benchmark. Binary coding loses nuance: CodeXGLUE is a multi-task suite with varied internal tasks, DS-1000 combines realistic StackOverflow provenance with deliberate anti-memorization edits, and official SWE-bench pages can change over time [[cite:lu2021codexglue,lai2023ds1000,swebench2026site]]. The taxonomy is therefore a dated synthesis from 2026-06-23 and should be updated as benchmark families add live tasks, hidden or dynamic tests, richer domains, and deployment-centered evaluation.
Conclusion
Coding-agent benchmarks now provide real evidence of progress, especially for executable code generation and repository issue repair. The evidence is strongest when claims remain tied to the benchmark's task unit, oracle, harness, curation, and domain. It becomes weak when a resolved-rate number is used to imply broad operational readiness without live deployment, collaboration, security, contamination, cost, and maintenance evidence.
The proposed claim ladder gives benchmark consumers a simple discipline: ask what claim level the evidence actually supports. Current public artifacts justify claims from bounded synthesis through validated and multimodal or multilingual issue repair, but they do not yet justify unqualified statements that coding agents can replace production software engineering workflows. Future evaluations should be designed as portfolios that combine narrow controlled tests, repository repair, multimodal and multilingual cases, hidden or dynamic oracles, scaffold disclosure, and deployment-adjacent evidence.