BlueOptima whitepaper introducing BARE: benchmarking 57 LLMs on 243k real-world refactoring tasks across 9 programming languages.

Benchmarking the Real-World Coding Performance of LLMs

Source Metadata for AI Agents

Benchmarking the Real-World Coding Performance of LLMs: Introducing BARE

Abstract

The practical value of large language models (LLMs) for enterprise software maintenance remains uncertain despite strong performance on popular coding benchmarks. We introduce the BlueOptima AI Refactoring Evaluation (BARE), which benchmarks 57 LLMs on maintainability-oriented refactoring tasks drawn from 4,276 real source code files spanning nine programming languages (C, C++, C#, Go, Java, JavaScript, PHP, Python, TypeScript), yielding 243,732 model-file evaluation pairs. Each output is assessed using a static validation pipeline covering syntax validity, function signature preservation, import integrity, variable-access correctness, unfinished-logic detection, comment preservation, and maintainability improvement measured through FLART score reduction.  

The results reveal a substantial gap between benchmark-style coding performance and performance on realistic refactoring tasks. Even frontier models achieve overall success rates below 23%. Models perform well on syntactic and structural checks (typically exceeding 80%), but success rates plummet when refactorings must also improve maintainability without impairing other aspects of the code.  

Performance varies dramatically across programming languages. JavaScript achieves 31.9% success versus C at 3.67%, an 8.67x difference. Performance across refactoring task types also varies widely, with localized function simplification substantially more tractable than architectural restructuring. Longitudinal analysis of model families reveals convergence among flagship proprietary models within a narrow 17–23% success band, with forecasting analysis indicating an asymptotic performance ceiling of approximately 20.8% for cloud-scale models, and no detectable temporal improvement among open-weight models. Cost-effectiveness analysis demonstrates that premium models, despite higher per-token costs, may offer superior total cost of ownership once developer review time for failed attempts is accounted for.  

These findings suggest that conventional benchmark results are a poor proxy for real-world maintainability refactoring performance. LLMs should currently be deployed as tools requiring human intervention and validation within validation-heavy workflows rather than as autonomous refactoring agents. More broadly, the study argues for coding evaluations that better reflect authentic software maintenance tasks, realistic validation requirements, and the quality dimensions that determine production usefulness.  

 

Introduction

LLMs have fundamentally transformed software development practices. GitHub Copilot alone serves over 20 million developers as reported by Microsoft (2025), while ChatGPT and similar tools process billions of coding queries monthly. This rapid adoption is largely driven by strong results on standardized coding benchmarks, where modern LLMs achieve high performance on widely used code generation evaluations such as HumanEval and related benchmark suites (Huynh et al., 2025). Yet beneath these headline numbers lies a critical question that practitioners increasingly confront: am I getting value from these significant investments in Generative AI? To answer this question, decision makers look to the coding benchmarks of LLMs to get beyond anecdotal accounts of developer experiences with the new technology, or to select a particular LLM vendor.  

The Benchmark-Reality Divide

The gap between controlled evaluation and production deployment has grown conspicuous. The benchmarks most commonly cited to justify investment in LLM-assisted development have, in rapid succession, been shown to be saturated, contaminated, or structurally misaligned with the demands of enterprise software engineering. Understanding how each generation of evaluation has failed is essential context for interpreting the results of the present study.  

HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) evaluate isolated function generation from natural-language docstrings. HumanEval comprises 164 Python problems with a median length of nine lines, while MBPP roughly has 1,000 crowd-sourced problems of comparable simplicity. These benchmarks suffer from three fundamental deficiencies:  

Subsequent efforts attempted to address these deficiencies while retaining the single-function paradigm. Liu et al. (2023) augmented test suites in HumanEval+ and MBPP+, revealing that the original benchmarks had overestimated model capabilities by 12.6–28.9% due to insufficient test coverage and false-positive rates exceeding 30%. Yu et al. (2025) introduced HumanEval Pro and MBPP Pro, which require self-invoking code generation. This modest increase in inter-function dependency caused dramatic performance drops: o1-mini fell from 96.2% to 76.2%, a twenty-percentage-point decline. Yet these enhanced variants remain limited to single-file, algorithmic tasks that bear little resemblance to production software engineering.  

SWE-bench (Jimenez et al., 2024) represented a paradigm shift toward realistic evaluation, drawing on 2,294 real GitHub issues requiring multi-file edits. SWE-bench Verified curated a human-validated subset of 500 instances. SWE-bench+ (Aleithan et al., 2024) collected post-training-cutoff instances to detect contamination, and SWE-bench Pro (Scale AI, 2025) extended evaluation to 1,865 long-horizon tasks across Python, Go, TypeScript, and JavaScript.  

However, by early 2026, the top models—Claude 4.6 Opus (80.8%), Gemini 3.1 Pro (80.6%), and GPT-5.2 (80.0%)—were clustered within less than one percentage point on SWE-bench Verified. In February 2026, OpenAI’s Frontier Evals team published an audit concluding that SWE-bench Verified was both saturated and contaminated (OpenAI, 2026). The audit found verbatim gold patches and that 59.4% of the hardest unsolved problems had flawed test cases. OpenAI formally discontinued reporting SWE-bench Verified scores in favor of SWE-bench Pro, where models scoring ~80% on Verified achieve only around 23% on Pro under standardized scaffolding.  

Other recent benchmarks include:

Together, these developments illustrate a broader shift in the evaluation ecosystem away from static, single-function benchmarks toward continuously updated, contamination-resistant, multi-capability evaluations. To address these benchmarking challenges, we introduce the BlueOptima AI Refactoring Evaluation (BARE). BARE tests an operationally relevant question: can an LLM improve the maintainability of an existing production source file while preserving structural correctness?  

The Maintenance Imperative

Maintainability—the ease with which software can be modified to correct defects, improve performance, or adapt to changing environments—directly influences operational efficiency, cost-effectiveness, and long-term success. Poor maintainability typically manifests through common structural flaws called anti-patterns, such as the "God Class" or "Functionality Overload".  

Software maintenance dominates total cost of ownership, consuming 60–90% of lifecycle expenditure (Palomba et al., 2018). Technical debt represents 20–40% of an organization's entire technology estate value, with enterprises allocating approximately 30% of IT budgets to debt management.  

Empirical findings highlight the operational impact of code quality:

Fixing the Industry Blindspot

Existing coding benchmarks do not directly evaluate whether an LLM can improve maintainability while preserving structural correctness in an existing file. We assess 57 distinct LLMs across nine programming languages (Python, Java, JavaScript, TypeScript, C, C#, C++, Go, PHP) on real-world refactoring scenarios: extract method, extract variable, inline method, rename, simplify conditional, remove duplicate code, and introduce parameter object.  

Our validation pipeline implements seven sequential checks:  

  1. Syntax validation (compilation without errors)  
  2. Function signature integrity (preservation of names, parameters, and return types)  
  3. Import statement verification (no omitted dependencies)  
  4. Variable access integrity (no references before assignment)  
  5. Unfinished logic detection (no placeholder comments or incomplete implementations)  
  6. Comment preservation (retention of meaningful documentation)  
  7. FLART score reduction (measurable improvement in maintainability as assessed by BlueOptima's static analysis framework)  

Only solutions passing all seven checks qualify as successful refactorings.  

Key Contributions

  1. Large-Scale Empirical Evaluation: Assesses 57 LLMs across 9 languages and 243,732 refactoring attempts using a rigorous 7-check static validation pipeline.  
  2. Heterogeneous Evolution Trajectories: Documents progress across model families, revealing that flagship models converge in a narrow 17–23% success band, while Meta Llama and Gemini Flash show performance regressions across versions.  
  3. Language-Specific Performance Variation: Reveals an 8.6x difference between best- (JavaScript at 31.9%) and worst-performing languages (C at 3.67%).  
  4. Economic Framework with Hidden Review Costs: Demonstrates that premium models with higher success rates achieve superior Total Cost of Ownership (TCO) once developer review time for failed attempts is factored in.  

Method

Objective

The primary aim of this study is to assess the ability of 57 LLMs to successfully refactor source code files exhibiting software maintainability issues across 9 programming languages (C, C++, C#, Go, Java, JavaScript, PHP, Python, TypeScript). Maintainability issues were identified using BlueOptima’s HowToFix (HTF) service, which analyzes files across 26 static metrics covering volume, complexity, and interrelatedness. A file is classified as having a maintainability issue when it exceeds the 90th percentile threshold relative to historical population norms.  

Caption: Illustration of static metric thresholds showing the maintainable range, warning risk zone (70th percentile), and maintainability issue zone (90th percentile).

 

Maintainability Issues Identified by HTF Service

Source file maintainability is evaluated using File-level ART (FLART) scores on a 0–1 scale, where higher values indicate lower maintainability.  

LLM Selection for Evaluation

Models were selected based on practical availability (accessible via commercial APIs or open-source hubs released between January 2024 and March 2026) and economic viability. An enterprise cost ceiling was set at $90 per million tokens (combined input + output cost). Models exceeding this threshold (e.g., GPT-o1-Pro at $150 input / $600 output) were excluded.  

Evaluated Language Models Profile

Data Collection

The benchmark dataset comprises 4,276 files across 9 languages:  

Dataset Distribution by Language

Benchmarking Technique

Each model was prompted using a 4-stage chain-of-thought strategy:  

  1. Problem Comprehension: Model explains code functionality and purpose.  
  2. Issue Diagnosis & Strategy Formulation: Diagnostic metrics are presented; model proposes concrete improvements.  
  3. Implementation: Model generates refactored file adhering to signature and dependency preservation constraints.  
  4. Validation Planning: Model outlines a test verification plan.  

Evaluation was strictly single-shot (pass@1) directly to LLM APIs without agentic scaffolding, self-correction loops, or external tools.  

Validation Checks and Metrics

Refactored files were passed through 7 validation checks:  

  1. Syntax Validation: Parsed and compiled without error.  
  2. Function Signature Integrity: Preserves function names, parameters, and return types.  
  3. FLART Assessment: Quantitative reduction in FLART score.  
  4. Import Statement Verification: Ensures no dependencies were omitted.  
  5. Unfinished Logic Detection: Rejects placeholders or incomplete code.  
  6. Comment Preservation: Retains docstrings and meaningful comments.  
  7. Variable Access Integrity: Linter checks confirm no variable is accessed before assignment.  

Key performance metrics:  

Error Classification

Unsuccessful runs were classified into:  

Model Error Breakdown

Results

Infrastructure and Generation Reliability

API-related errors accounted for less than 1.5% of attempts (81 network, 19 throttling, 17 server-side, 10 malformed response).  

Generation Rate, Success Rate, and Coding Effort

While file generation reliability exceeded 90% for most models, maintainability refactoring success rates clustered between 15% and 23% for top models and fell below 10% for smaller models.  

Language-Level Efficacy

Refactoring success is strongly dependent on programming language characteristics:  

Maintainability Issue-Specific Success Rates

LLMs perform well on localized intra-function issues but struggle severely with inter-functional architectural issues:  

 

Capability Trajectories and Cost Analysis

Flagship cloud models converge in a narrow 17–23% success band: Claude 4.6 Opus (22.0%), GPT-5.2 (21.3%), Gemini 3.1 Pro (21.5%), and Grok-4 (23.0%). Coding Effort per successful refactor rose from 1.7 to 2.1 CE hours across generations.  

Discussion

Why LLMs Fail: The Syntax-Semantics Divide

LLMs achieve >85% success on syntax and function signature checks but drop to 25–50% on FLART maintainability improvements. Research confirms LLMs function as pattern matchers operating on sequential token streams, lacking graph-like structural comprehension (Zhang et al., 2023). They exhibit a "single-function horizon"—handling intra-function simplification (38.2%) well, but failing on architectural file restructuring (1.6%).  

Language-Specific Performance

The 8.67x gap between JavaScript (31.84%) and C (3.67%) reflects language design constraints. High-level, memory-safe languages with standardized module systems reduce cognitive burden, whereas low-level system languages with manual memory management, pointers, and explicit type constraints severely restrict valid refactoring space.  

Benchmark Inadequacy & Performance Plateau

Current benchmarks suffer from saturation and contamination. BARE's single-shot baseline (16.6% average) shows that model scaling alone yields diminishing returns on maintenance tasks. This explains the industry shift toward surrounding foundation models with agentic orchestration layers.  

Economic Implications & Total Cost of Ownership

Analyzing developer review time for failed refactoring attempts reverses naive per-token cost calculations:  

Considerations on Scope

Conclusion

Strong performance on conventional benchmarks does not translate to maintainability refactoring success. Even frontier models remain below 23% overall success. Flagship cloud models approach an asymptotic ceiling near 20.8%, while open-weight models plateau near 15.1%.  

Recommendations for Software Development Executives

  1. Require Mandatory Human Review: Never deploy LLM refactored code to production without expert engineer review.  
  2. Scope to Localized Tasks: Target bounded use cases (documentation, single-function simplification) rather than architectural restructuring.  
  3. Account for Language Stacks: Adjust expectations based on language (e.g., higher returns in JavaScript vs. low returns in C/C++).  
  4. Validate Against Proprietary Codebases: Benchmark models internally prior to broad rollout.  
  5. Procure Based on Total Cost per Success: Factor in developer review time rather than token unit pricing alone.  
  6. Invest in Engineering Governance: Develop review standards and train senior developers to critically evaluate AI outputs.  

Notices and Disclosures

Appendices Summary

Appendix A: GitHub Repositories

Evaluated 63 major open-source repositories spanning 9 languages (including kubernetes, linux, roslyn, electron, tensorflow, react-native, godot).  

Appendix B: Logistic Regression Tests

Models refactoring success probability as a function of code source, initial quality (POSTFLART), programming language, and effort (SLCE). Confirms file quality and complexity dictate refactoring success.  

Appendix C: Coding Effort

Appendix D: LLM Evolution

Tracks CE and success rate trends across model release dates for Anthropic, OpenAI, xAI, Google, Meta, Mistral, and Qwen families.  

Appendix E: Forecasting of LLM Refactoring Success Rate

Bayesian Model Averaging (BMA) forecast indicates an asymptotic ceiling of 20.8% (95% CI: 19.4–30.1%) for flagship cloud models and 15.1% (95% CI: 13.4–17.5%) for open-weight models.  

Appendix F: File Type Identification (FTI)

BlueOptima's machine learning FTI system classifies code files across 109 languages with 0.97 balanced accuracy.  

Public source: https://www.blueoptima.com/resource/benchmarking-the-real-world-coding-performance-of-llms-introducing-bare