The API is not the ChatGPT your clients use

The short answer: there are two ways to measure whether AI recommends your firm. The cheap way is to call the vendor API, the developer product, with web search switched on, and treat its answers as a stand-in. The honest way is to ask the consumer product itself, the ChatGPT window your client actually types into. They are different systems that happen to share a name, and a firm evaluating an AI visibility tool should know which one is being measured.

What "checking the API" means

OpenAI, Google, and others sell developer APIs: you pick a model, send a question, optionally let it search the web ("grounding"), and get an answer back. It is fast, cheap, and easy to run at scale, which is why measurement tools reach for it. The result looks like a ChatGPT answer. It is not one.

Five concrete differences

1. Different model, different instructions

The consumer product decides which model variant answers, with what settings, under what system instructions, and changes all three without notice. An API caller makes those choices themselves. Same question, two differently configured systems.

2. Different retrieval

When the real ChatGPT searches the web, it uses OpenAI's own search stack. API grounding uses whatever search tool the developer wired in. Which pages get read decides which firms get named, so a different retrieval path means a different recommendation pool.

3. The product layers only exist in the product

Consumer surfaces attach things the API never sees: local business cards, maps modules, review snippets, follow-up suggestions. For a local-intent question like finding a lawyer, those layers shape the answer materially.

4. Session context

The consumer product carries memory, chat history, and signals about the user; the API is stateless. We sample in fresh anonymous sessions precisely to hold this constant, but a tool measuring the API is not holding it constant, it is measuring a system where the variable does not exist.

5. Google AI Overviews has no API at all

Google offers no API that returns the AI Overview a searcher sees. The Gemini API's "grounding with Google Search" is a different feature of a different product, and the third-party tools that do return AI Overviews get them by scraping the real search page, which concedes the point: the consumer surface is the only source. Measuring AI Overviews, including whether one appears at all for a question (in our tracking it triggers only some of the time), means running the real search and looking. A vendor-API-only tool cannot measure the AI surface with the largest audience of the three.

Why tools use the API anyway

Cost and convenience. API calls are metered, predictable, and run in milliseconds; sampling real surfaces is slower, more expensive, and harder to engineer. For some questions the API is a fine proxy. But a proxy is exactly what it is, and nothing guarantees the two systems drift together as vendors ship product changes weekly.

How Ayo measures

We ask the real ChatGPT, the real Gemini, and the real Google, in fresh anonymous sessions, with the questions New Yorkers actually type, many samples per question per day. It is the slow way. It is also the only way to report what your prospective client actually sees, which is the thing a law firm is paying to know.

What to ask any vendor

• "Do you query the API or the consumer product?" If the answer is the API, every number is a proxy for the thing you care about.
• "Can you measure Google AI Overviews?" If yes, ask how, since no API returns it.
• "How many samples per question?" Answers vary between identical asks; a single check misleads.