How to Evaluate a New AI Coding Tool

When a new AI coding tool looks fast in a demo or ranks highly on a public leaderboard, it is tempting to adopt it immediately. The practical question is narrower: for which tasks in my repository does it succeed, how often, and how much review is required before the result is acceptable?

This note separates the model from the harness, fixes representative tasks, and runs them more than once. The measurements will vary by environment, so this is an evaluation procedure rather than a ranking of products.

Decompose the comparison

Observed result = model
                × reasoning setting
                × harness
                × prompt / project instructions
                × tools / permissions
                × environment

Separate a test that compares one model in two harnesses from a test that compares two models in the same harness. If both change at once, the result has no clear explanation.

Build representative tasks

Public benchmarks are useful for screening, but adoption decisions need tasks from the repository that matters. Prepare at least one task of each kind:

1. A small, well-defined bug fix
2. A feature spanning several files
3. A refactor that must preserve existing behavior
4. Investigation of a failing test
5. Turning an ambiguous request into a plan
6. A task with files or operations that must not be touched
7. Documentation or configuration work

Record the starting commit, prompt, allowed tools, timeout, expected checks, and forbidden changes for each task. Score required behavior and acceptable boundaries instead of requiring one exact patch.

Choose metrics before running the test

Metric	Observation
Functional success	Required tests and acceptance conditions
Regression	Existing tests and diff review
Change quality	Minimality, readability, design consistency
Autonomy	Number and quality of human interventions
Time	Wall-clock time and human review time
Cost	Input/output tokens, subscription allowance, tool charges
Safety	Forbidden operations, permission violations, secret handling
Reproducibility	Success rate and variance under the same conditions

One run cannot distinguish a reliable result from luck. Start with three to five runs when the cost allows, and keep the worst case and the amount of human repair, not only the average.

Fix the execution conditions

Log the following values:

• Product and version
• Model ID and snapshot
• Provider
• Reasoning effort
• System and project instructions
• Available tools and permissions
• Files included in context
• Maximum turns, timeout, and retries
• Token usage, price, and timestamp

The OpenAI model guidance also recommends comparing quality, final answers, tokens, latency, and cost on representative tasks. Using the same dimensions for other products makes the result easier to explain.

Read public benchmarks as conditional evidence

SWE-bench generates patches from real GitHub issues and evaluates them with tests. Terminal-Bench verifies end-to-end terminal tasks.

Both are useful, but a score is not a model-only measurement. It includes a harness, prompt, tools, budget, retries, and dataset version. When reading a leaderboard, check:

• Whether the dataset subset is the same
• Whether the result is verified or live
• Whether it is pass@1 or a multi-run result
• Whether reasoning and tool budgets match
• Whether a human edited the patch
• Whether logs and settings are available
• How training contamination was handled

A public score can narrow the candidates; it cannot guarantee success in your repository.

Introduce the tool in three stages

1. Screening

Check public documentation, license, supported operating systems, authentication, price, and update activity. Drop candidates that fail a mandatory requirement.

2. Controlled evaluation

Run the representative tasks from an isolated worktree and fixed commit. Do not allow external sending, deletion, publication, or billing actions. Record the result in the scorecard.

3. Pilot

Limit the tool to low-risk production work for two to four weeks. Measure waiting time, rate limits, configuration maintenance, and review fatigue that a controlled benchmark may not reveal.

Define adoption criteria in advance

For example, write criteria such as these before running the comparison:

• At least 80% success on mandatory tasks
• Zero regression-test failures
• At least 20% less human review time than the current tool
• Monthly cost within budget
• Zero forbidden operations
• Important results reproducible from logs

Adjust the numbers to the team and the task. Creating the criteria after seeing the results makes it easy to move the goalposts for a favorite tool.

Keep the conclusion revisable

Models, pricing, default prompts, permissions, and rate limits change. Do not turn an adoption result into a permanent ranking; rerun the same suite after a major version change or quarterly.

A good evaluation does not identify “the strongest tool.” It defines which tasks can be delegated under which conditions and cost. Keeping failures and human repairs in the scorecard gives the next demo something concrete to compare against.