Bot access is not enough: why an open site remains uncited

Why is my site accessible to AI bots but never cited?

A pricing page can be open to AI bots and remain absent from answers. Permission to read it guarantees neither its usefulness for the question asked nor its selection as a source. For answer engine optimization, the question remains: what does this page contribute to the answer being sought?

Aggarwal et al. (2024) formalized content optimization for generative search engines in a study presented at KDD 2024 and proposed methods whose effect on visibility they measured.

What proves that a bot actually accessed your page?

Permission granted by your robots.txt file does not prove a visit. The evidence comes from a request attributable to the bot in the server logs. To establish what the server served it, you also need to examine the recorded response. RFC 9309, published by Koster et al. (2022), standardizes the Robots Exclusion Protocol and defines, in particular, how to interpret the User-agent, Allow and Disallow directives: these rules help explain the instructions addressed to bots, but their presence in a file does not document any actual visit to a specific address. Server logs record incoming requests. An entry matching the page in question, with its date and the available information about the requester, provides a record to examine. This record tells you more than the configuration alone.

The scope of this record remains limited. An incoming request may result in an error or a redirect. The name declared by the requester is not enough on its own to authenticate the bot. RFC 9110, published by Fielding et al. (2022), standardizes HTTP semantics, including status codes and redirects; this information helps characterize the server’s response without establishing that the content was selected. Google also publishes a list of its search engine crawlers, with the names they use to identify themselves during visits. The record ends with this exchange.

Ask the person managing your hosting for an extract from the logs. Specify the exact page address and the period under review. Keep the information needed to verify the bot’s identity, along with the status code returned. A screenshot of the robots.txt file alone does not meet this request. Ask them to clarify whether the server served the requested page or redirected the bot to another address. If there was a redirect, also look for a request to the destination. Without this check, do not conclude that the bot accessed the final content. If no matching entry appears, record “visit not established during the period under review”. Check that the logs cover this period before stating that the bot never visited.

Does your page answer your customer’s question?

A page details your support offering. It precisely describes how the service works and the skills involved. But it does not explain what the customer needs to prepare before the first appointment. The question “What documents do I need to provide to get started?” remains unanswered. The description of the offering leaves information missing here. To check your own page, write down a specific question received from a customer. Copy the passage meant to answer it alongside it. Keep only the published words. Do not add the explanations you usually give verbally. They may fill a gap for you that remains for someone discovering your offering. Then reread this extract on its own. Does the customer have the information needed to act? In this example, the passage should specify the documents required. Or explicitly state that no documents are needed beforehand, if that reflects your practice. A phrase such as “support tailored to your needs” leaves the question open. The reader still needs to contact you to get the answer.

What can you verify with the published sample diagnostic?

The published sample diagnostic lets you verify what the observations presented establish as of their date, within the scope examined. It does not predict the citations your company could obtain, even when a case seems similar to yours. Similarity is not enough. To read this document as evidence, connect each conclusion to the passage that supports it. An observation about the site studied retains that specific scope, even if you recognize a difficulty your company faces and want to use it to inform a decision.

HENUSSE reports its results as dated N/M fractions, rather than a score out of 100. This presentation requires reading the number of cases counted in relation to the full set observed. You therefore need to check what the report actually counts. Before judging a fraction as good or bad, look for the definitions of its numerator and denominator. The letters alone do not tell you precisely what the calculation counts or leaves out. Never cite a fraction without its denominator. If you use an AI citation measurement from the sample in a management briefing, keep its date and the definition of the observed set alongside the result. The person reading your briefing must be able to understand the scope of the figure without assigning it a general significance that the published case does not demonstrate.

Then examine the link between the finding and the proposed conclusion. When a diagnostic recommends a change, what does the document show? Only the reason for proposing it? Or also an observation after its implementation? These elements support different conclusions about the case studied. To judge what the sample allows you to decide, retain only conclusions for which you can find support in the document. Leave open the questions it does not settle.

Who confirms your claims outside your site?

An article published elsewhere that repeats your sales presentation does not independently confirm your claims. To call it external corroboration, you need to identify what its author personally verified and which specific claim that verification relates to. The origin of the text matters. If a publication repeats a result announced in your press release without explaining how it checked it, the reader finds your statement on another site, with a different layout. Your company remains the sole source of the information. Similarly, a positive testimonial about the quality of your interactions leaves a claim about the results of your service unaddressed. A confirmation applies to the exact point it supports.

Before keeping a mention as public corroboration, place your claim beside the external passage meant to confirm it. Look for what the author relies on. An account describing a personal experience can support the facts of that experience. Its scope stops at what its author actually observed. If the passage refers only to your own statements, classify the mention as a repetition. Exclude it from independent confirmations, even if the title presents you favorably and you would like to display the site’s name among your references. If a source provides limited confirmation, narrow the associated claim. Keep, for example, a testimonial about support to discuss that support. Any promise of results that this testimonial does not document remains without external confirmation.

How can you tell whether your AI citation measurement allows a comparison?

Do the questions differ from the previous measurement? Your AI citation measurement does not allow a comparison with the questions held constant. Keep their exact wording. Record the engines used, along with the conditions and date of each test. The scope matters as much as the result. To track your answer engine optimization, set these rules before the first measurement. Then keep a record of every deviation from the protocol.

1. Keep the questions fixed. Keep a reference list. Repeat each question word for word in the next measurement. A rewording counts as a new question in your tracking, even if its intent seems identical to you. Keep additions in a separate series. You can then compare results for the questions common to the two dates.
2. Describe the test conditions. Note the engine and mode used. In particular, indicate whether the test uses Web search. Specify the language and indicate whether the conversation starts fresh or already contains exchanges. If the interface displays a model version, record it. Information that is unavailable remains marked as such: do not reconstruct it from memory.
3. Archive each answer with its date. Keep the full text and associated source links, including when your site is absent. Plan the same number of runs per question for each measurement. Keep all results obtained. Keeping only the answer that cites you would make the assessment depend on your selection.
4. Define what you count. Decide in advance whether your metric counts a link to your site or a simple mention of your name. Count these two elements separately if you want to track both. Present each total with the number of answers examined. Then flag missing questions and interrupted tests.

If the protocol changes, limit the comparison to the part that remained the same or establish a new starting point. Even with a consistent protocol, an observed increase is not enough to attribute citations to a specific change to your site. The record establishes a difference. Its cause remains to be verified.

What should you check now to understand why your site remains uncited despite allowing bots?

Even if your site allows bots, first check that no access issue remains to be corrected. This check comes before work on answer pages and their external corroboration. Then examine gaps in the accessible content to choose a change whose effect you can verify. After this change, repeat the AI citation measurement with the same questions. Keep the protocol. Do not automatically attribute every observed change to your intervention.