Do you trust the data your agents find on the web? Our findings from 100k queries
How often AI search APIs give agents on-topic results from sources a professional would not trust.


Do you trust the data your agents find on the web? Our findings from 100k queries
How often AI search APIs give agents on-topic results from sources a professional would not trust.


When people search online, we implicitly do quality control. We skip the listicle, notice the page belongs to a company selling what we're asking about, and scroll down to the regulator's page we trust. Agents don't. Whatever ranks first goes into the model's context, and their answer sounds just as confident whether it came from a food-safety regulator or a company selling the product.
Search quality is usually measured as relevance: did a page about the right thing come back? We wanted to know how often agents get a page about the right thing from a source with no authority on it, like a seller, an SEO page or a forum, and whether that depends on what they're asking about. So we took 100,000 real search queries written by agents in production and ran each through 12 search configurations from five providers. Then we classified all results by what the question was about and where the answer came from.
What counts as a bad source
We looked for results that come from a source a professional wouldn't accept for that kind of question. Results that are simply off topic, like a page about a different company with a similar name, were counted separately. Search engines are built to catch those and push them down in rankings. A few examples include:
- Sellers answering their own questions. A company that sells a product or service, answering whether it's safe, effective or required.
- Content built to rank. SEO and affiliate pages, "top 10" lists, auto-generated stock forecasts and rate aggregators.
- Content outside its owner's expertise. A payroll vendor stating a city's minimum wage, a travel blog listing a bank's fees.
- Social posts, forums and AI-generated pages used as the source for facts.
Whether a source is acceptable depends on the question. A blog is a fine source for "how do I set up a home office". It is a bad source for a central bank's interest rate decision. So we judged every result against the question it was answering, not against a list of good and bad websites.
How often it happens
As expected, most agent searches get information from trusted sources. However, about 1 in 8 results (12.2%) were on topic, but came from a source that shouldn't be trusted for that question.

The questions with the highest stakes get the worst sources. Finance, health and legal questions got low-trust sources, mostly SEO pages and company blogs, twice as often as all other topics. In more than half of finance and health searches, at least one of the top five results was a source a professional wouldn't use. In 30% of health searches and 22% of finance searches the first result was a low-trust source.
Switching providers doesn't fix this, every provider has the problem. All twelve configurations returned low-trust sources roughly twice as often for finance, health and legal questions as for other questions. Faster, cheaper tiers were worse. Even the best configuration put a low-trust source somewhere in the top five for a third of these searches.
What these results look like
The examples below are real searches from the sample, paraphrased, with company and site names removed.
Sellers answering safety questions
One of the most worrying patterns were health questions answered by the people selling the answer.
"Is an IV drip at home safe?" All twelve search configurations put a business that sells at-home IV drips in first place. Nine of the twelve returned no independent medical or nursing source anywhere in their top five. An agent answering this question would answer it entirely from the sellers.
"Is raw pet food a bacteria risk?" The FDA warns that raw pet food can carry Salmonella and Listeria, a risk to pets and to the people who handle it. Only one configuration put that guidance first. Six providers put a page calling the risk a myth in first place: a raw-food seller, a dog breeder and a pet hotel.

Copies ranking above the original
Sometimes the right source exists, and something else outranks it.
- A wire-service story about a commodity regulator's price decision. Seven of twelve configurations ranked an AI-generated rewrite of the story above the original article. Only one put the original first. An agent briefing a commodities desk would quote a rewrite whose figures no one has checked, and cite it as the wire.
- Verifying a manufacturing license on a state regulator's portal. Nine of twelve configurations put the same consultancy sales page first. Only two returned the regulator's own portal anywhere in their top five. An agent vetting a supplier would mark the license as checked without ever seeing the regulator's record.
- Clinical genetic testing providers in a major city. None of the twelve configurations returned an authoritative source in its top five. One put a gambling spam site first: a copy of a genetics lab's website, re-skinned to promote online slot games. An agent helping a patient find a test would have no lab it could safely recommend, and one would even send them to a gambling site.
Numbers that are early, old or answered from forums
In financial questions, the problem is often timing or trusted sources: the page is about the right thing but has the wrong numbers.
- Financial advice from forums. All twelve configurations answered questions on what to do with retirement funds returning 3-5 of the top 5 answers with content from forums, one even stating to put all your fund into one stock because it was the top voted answer.
- Quarterly earnings, the day they came out. Eight of twelve configurations put the wrong kind of source first, including earnings previews written before the results and a report from two quarters earlier.
- A national health service's pay scales for the new year. One configuration's first result was titled with the new year's pay scales but showed the previous year's figures. Two others put an Instagram reel first. An agent using these would give a nurse the wrong salary.
- "Recommend a retirement calculator." Two configurations put a gold-IRA sales site's "top 10 for 2024" list first, two years out of date and written to sell a specific investment product.
Why this happens
AI search APIs are built for agents, but they still rank mainly for relevance: does the page match the question? On money, health and law, many of the most relevant-looking pages are written to capture that traffic: affiliate content, SEO pages and sellers' blogs. These topics have the most commercial incentive to publish content that ranks, and relevance ranking can't tell those pages from a regulator's. In our data, wrong-source results were as common at position 1 as at position 5, at every provider we tested.

Agents remove the human filter. They take the top results as they come. And instructions like "only use reliable sources" don't reliably help: the agent can only choose among the results it was given, and it has no way to know the regulator's page was never in the list. In long agent runs, the instruction also fades. After dozens of searches it sits under thousands of tokens of results, or is summarized out of the context entirely, while the results keep coming in.
How we solve it
Nimble treats source quality as a filter, not a ranking signal. You define one source policy for each agent, or one for all agents, set in one place. Domain allowlists and blocklists, safe search and more. A finance agent can be limited to filings, regulators and approved publishers, while a marketing agent keeps the open web.
Your agents get answers from sources you'd trust yourself, not from whoever ranks first.
FAQ
Answers to frequently asked questions






.webp)
.webp)

.webp)
.avif)