A change to a web research agent can improve coverage while reducing accuracy, or produce better answers at an impractical cost. Before shipping, engineers need to know whether it improved the intended workload and whether it broke anything else.

At Nimble, we built a shared evaluation framework to answer both questions. Every change, from prompts to models to search providers, goes through the same process: verify answers against live evidence, score multiple dimensions of quality, and compare versions under controlled conditions.

Across roughly 440 runs in under three months, it has also changed how we find what to improve: instead of relying on isolated examples, we measure performance by research category, such as lead generation, e-commerce, and real estate, and focus effort where quality is degrading.

Those measurements are only as trustworthy as the judge behind them. In early calibration, its agreement with human reviewers (within ±1 point on a 1–10 scale) rose from 7% to 67%, and its average error fell from 4.5 points to 1.3.

This post explains how the judge works, how we keep it calibrated, and how we use it to monitor quality and test changes.

Why web research resists simple evaluation

Consider a request: “Find logistics companies headquartered in Germany that announced a Series B between January and June 2026. Include the funding date and a source for each.”

Evaluating the answer presents three distinct challenges.

We encountered the first problem directly. Our original LLM judge marked correct, recent news as hallucinated because it fell beyond the model’s training cutoff. The agent had found information the evaluator did not know.

Verified reference answers help, but building and maintaining exhaustive answers across research tasks is expensive. Even a carefully prepared company list may become outdated. We needed an evaluator that could investigate the answer’s claims while recognizing the limits of what it could establish.

What a trustworthy research evaluation must establish

Web research depends on changing evidence and often lacks a complete reference answer. A trustworthy evaluation must account for both when assessing answers and comparing agent versions.

• Verify claims against evidence: Check whether sources support the answer’s specific claims, including facts beyond the evaluator’s training knowledge.
• Respect the relevant date: Assess claims against what was true during the period the question concerns.
• Separate correctness from completeness: Verifying returned results does not establish what the agent missed. Assess coverage separately and acknowledge limits where no exhaustive reference set exists.
• Account for a changing web: Compare versions on the same questions, close in time, with consistent settings and a fixed judge to reduce the influence of changing evidence.

Quality gains must also be weighed against completion rate, latency, and cost.

Building a judge that meets those requirements

Our evaluator gathers evidence before assigning scores. It operates in two stages:

1. Gather evidence. Identify the question’s requirements and the answer’s claims, then search and fetch web pages to check them. Record unsupported claims and missing requirements.
2. Score the answer. A second model uses the collected evidence to evaluate six dimensions of quality.

The judge receives a larger time and compute budget because independently checking an answer can require substantial research of its own. The production agent must balance answer quality with customer-facing latency and cost. The judge has more room to investigate claims and gather evidence before deciding whether that answer deserves a high score.

This approach balances two requirements:

1. The need to reduce dependence on hand-maintained reference answers
2. The need to create a trusted ground truth answer to benchmark against

How we evaluate research quality

Each applicable dimension receives a score from 1 to 10, with higher scores indicating better performance.

We combine the judge’s individual scores into a single North Star score, ranging from 1 to 10. It is the equally weighted average of the applicable quality dimensions above (not all of these dimensions are applicable to every research).

North Star gives the team a common measure for comparing agent versions and tracking overall quality, while the six individual scores help engineers understand what drove a change.

A recurring loop for improving the judge

The evaluator also needs evaluation. Human reviewers label real answers across the same six dimensions, and disagreements between their labels and the judge’s scores drive candidate updates.

The calibration loop is:

The current and candidate judges are compared against the same held-out examples. Promotion requires statistical evidence and a meaningful improvement in agreement. Corrections from earlier rounds carry forward into subsequent candidates.

In the first calibration cycles, the judge’s agreement with human reviewers (within ±1 point on the 1–10 scale) rose from 7% to 67%, and its average error fell from 4.5 points to 1.3. The loop also rejected 3 of 11 candidate judges because they would have scored worse than the current version.

Each judge update has a version identifier. Comparisons of current and candidate agents use the same judge version. Weekly benchmark and production checks use the latest approved version, with updates marked on charts because a new judge can change scores even when the agent hasn’t changed.

Using evaluation to identify what needs improvement

Individual experiments answer whether a proposed change helped. Weekly monitoring identifies areas that deserve investigation, which runs two checks to track quality:

The benchmark also serves as a regression check during individual experiments. Those experiments primarily use questions selected for the workload the change is intended to improve.

For example, the October 19 report recorded a production North Star of 7.51, with recruiting at 6.77 and lead generation at 7.15. These category scores help identify priorities for further investigation, considered alongside usage and sample size.

The relationship between benchmark and production performance is also informative. If a category performs well on benchmark questions but poorly in customer traffic, we investigate whether the benchmark captures customers’ actual constraints and task complexity.

Fixed questions provide consistency, but the web still changes. Production samples also differ from week to week. Monitoring therefore helps direct investigation; it does not isolate the effect of an individual release.

Turning findings into tested improvements

Once an engineer proposes a change, the framework provides a common comparison process.

First, we define a dataset representing the intended workload and reserve test questions before development. Keeping those questions outside iteration helps distinguish general improvement from adaptation to familiar examples.

We then run the current and candidate agents on the same development questions, with matched settings and a pinned judge version. Running them close together reduces the opportunity for changes in the web to explain the result.

We bootstrap per-question score differences to estimate the average change and its 95% confidence interval. In the illustrative chart, dots show averages and bars show intervals; crossing zero means the result is inconclusive. We also check all six quality dimensions using Bonferroni-adjusted intervals.

The selected candidate is evaluated on the held-out questions and broader benchmark. Completion rates, missing evaluations, latency, and cost accompany the quality results. Engineers inspect the answers and judge explanations to check whether the observed differences correspond to the intended change.

Examples: How our eval system informed product releases

The model update illustrates why the target metric cannot determine the decision alone. Accuracy improved, but the latency measurement exposed a setting that would have made the agent substantially slower. Fixing that setting allowed the team to retain the quality improvement.

Lessons and next steps

Building this framework taught us a few things that apply to any team evaluating agents:

• Give the judge what the agent has. A judge needs the same live information as the agent, and more budget to check it.
• Decide on tradeoffs, not averages. Score multiple dimensions and weigh quality against completion rate, latency, and cost.
• Compare under matched conditions. Use the same questions, the same time window, and a pinned judge, and lock the test set before development starts.
• Test the judge like the agent. Calibrate against human labels, promote a new version only when it beats the current one on held-out examples, and mark every judge change.
• Watch real usage, not just the benchmark. Measuring by research category shows where quality is slipping, and gaps between benchmark and production show where the benchmark falls short.

Scores still need interpretation. Running comparisons close together limits web drift but doesn’t eliminate it, and human calibration improves the judge without making it infallible.

Next, we are exploring automated experimentation, following the pattern of Andrej Karpathy’s autoresearch: a coding agent proposes harness changes, such as prompts, tool descriptions, or search budgets, and tests them within a fixed budget before held-out evaluation and human review. We are also studying what a research agent with more time and compute does differently, and adapting those strategies to the production agent’s latency and cost limits.

To evaluate Nimble on your own workload, start with the questions your application actually needs answered and run them through the Web Search Agent documentation via AI plugins, MCP, or API.

Appendix: Method Details, Costs, and Limits

How the evaluations run

• Benchmark: 66 fixed questions, answered weekly by the agent from the main branch at low, medium, and high effort. High is the official trend series.
• Production sample: 100 real customer answers per week, sampled to reflect usage. A different sample each week.
• Judge: two stages, web evidence then scoring by a second model. One named default version; experiments pin their version.
• Experiments: paired per-question deltas, 95% bootstrap confidence interval on the mean, Bonferroni-adjusted intervals for the six dimensions, completion rate and missing evaluations reported with quality.
• Judge promotion: one-sided Wilcoxon signed-rank on per-item error deltas against human labels, bootstrap on agreement-rate and F1 deltas, α = 0.05 plus a minimum effect size, held-out labels only.
• Infrastructure: web-access failures tracked per question; degraded runs excluded from the official quality series.

Limitations

• Changing evidence: Running comparisons close together reduces web drift without eliminating it.
• Comparability over time: Different production samples, judge versions, and effort settings can move scores independently of agent improvements.
• Imperfect verification: Human calibration improves agreement with reviewers but cannot guarantee factual correctness or exhaustive coverage.
• Overfitting: Repeated experimentation can optimize for familiar questions or weaknesses in the judge. Held-out tests, component scores, broader benchmarks, and inspection of the evidence help constrain this risk.

Evaluation cost

First measurement, October 2026, LLM spend only (web calls not included):

• Benchmark run: about $59, or about $0.90 per question. The judge is about 80% of that; the web fact-check stage alone is about 72%.
• Production sample: about $70 per week (judge only).
• Judge calibration cycle (20 held-out pairs): about $32.
• All three: about $160 per week.