Can AI assistants find your business?

Two different doors

People talk about "AI crawlers" as one thing. They are two, and confusing them is the most common expensive mistake here.

The left column feeds models in advance. Whether that ever causes you to be recommended is genuinely unclear, and you have very little control over it.

The right column is the one that matters commercially. Those agents fetch pages at the moment a question is asked, to answer it and cite sources. If they cannot read your site, you are not among the things the assistant can name — today, in that answer.

There is a third door, and it is older

Several assistants lean on a conventional search index underneath. ChatGPT and Microsoft Copilot both draw on Bing.

Which means being indexed by Bing matters far more than Bing's share of direct searches suggests. Most small businesses have never thought about Bing at all. There is a protocol called IndexNow that lets a site tell Bing about a new page immediately rather than waiting to be crawled — it takes an afternoon to set up once and then runs by itself. Google does not participate, which is exactly why people skip it and exactly why it is worth doing.

Should you let them in?

For a local business, we think yes, and it is not a close call.

Blocking makes sense if your content is the product — a newspaper, a course, a paid archive. A hardware store, a tint shop or a builder is in the opposite position: you are trying to be recommended, and you cannot be recommended by something that is not permitted to read you.

It costs you nothing. Your opening hours are not intellectual property.

What about llms.txt?

An emerging convention: a plain Markdown file at your site root summarising the business and linking to the pages that matter, so an assistant can get an accurate picture in one fetch instead of piecing it together.

Being straight about this: it is not an official standard, and no major AI company has publicly committed to reading it. Anyone telling you it is a ranking factor is guessing. Treat it as cheap insurance — an hour of work, cannot hurt, and it doubles as the clearest one-page description of the business you will ever write. Ours is here, if you want to see the shape of one.

The thing that is not speculative is structured data: the machine-readable description of what you are, where, and what you charge. That has been read by search engines for years and is increasingly what feeds richer answers.

The question nobody can answer

Here is the part that surprised us most.

Ask almost any business, or any agency, whether AI assistants are actually reading their site. They cannot tell you. Not because they have not looked — because their analytics structurally cannot see it.

Nearly all website analytics is a JavaScript snippet that runs in the visitor's browser. AI crawlers generally do not execute JavaScript. They request the page, take the HTML and leave. The snippet never runs, so the tool records nothing and reports a confident zero.

Answering it requires logging the request itself — before the page is rendered, not after. On a modern static site that means logging at the edge, because the page is served from a CDN and never touches a server you control.

We built that for this site so we could answer the question for ourselves before offering it to anyone else. The most useful part of the report turned out to be the inverse: not which assistants came, but which ones we invited and never saw. A list of arrivals cannot show an absence, and the absence is usually the finding.

How to check your own site in five minutes

1. Open yoursite.com/robots.txt. If it 404s, nothing is blocked — fine, if accidental. If it lists GPTBot, ClaudeBot or PerplexityBot under Disallow, you are shut to them. Some themes and hosts add these by default without telling you.
2. Check for a blanket block. User-agent: * followed by Disallow: / blocks everything, search engines included. It happens more often than you would think, usually left over from a staging site.
3. Look at yoursite.com/llms.txt. Almost certainly a 404. Not urgent, but cheap.
4. Ask an assistant about your business by name and see what it says. Wrong hours and a wrong phone number are common, and they come from somewhere — usually an old directory listing that is easier to fix than to argue with.
5. Ask your web person whether crawler hits are logged. The answer is usually no. That is the gap.

The honest summary

Nobody credible can tell you how much business AI assistants will send you, and anyone quoting a number is inventing it. What can be said plainly:

• The doors are either open or shut, and shut is usually accidental.
• Retrieval agents matter more than training crawlers, and they are different user-agents.
• Bing sits underneath more of this than its search share suggests.
• Structured data is proven; llms.txt is cheap and unproven.
• If you are not logging crawler hits, you are guessing — and so is anyone charging you for this.

None of it is expensive. Most of it is a one-off afternoon. The reason to do it now is that it is much easier than trying to work out, two years from now, why you were never mentioned.