How Llms.txt And Robots.txt Affect AI Crawlers

The problem is not that the tools are dishonest. It is that the vendor controls both the number and the prompt set that produces it, so the score can improve without anything happening to your business, and a client has no way to audit the difference.

That matters most for the facts that establish identity, because those are the facts that let scattered mentions of you resolve into one record. It matters far less for content, where the model is going to read the prose anyway and is reasonably good at it.

Get the Basics Right Before Anything Clever Once access is confirmed, check that content actually exists for a crawler to read. Load your important pages with JavaScript disabled. If your specifications, pricing, service areas or contact details vanish, they are effectively absent from this channel regardless of how permissive your robots file is.

This matters because the prompt set is built from it, and a prompt set written from segment language measures your positioning rather than your market. If your brief says mid market operations leaders, the prompts will use that phrase and no buyer ever will.

What llms.txt Proposes It is a proposed convention: a file at your root offering a curated, plain text guide to your site for language model consumers, pointing at the documents you consider authoritative.

It is also worth asking for the report a day before the meeting rather than seeing it in the room. A document presented live is experienced as a narrative and approved on the strength of the delivery. The same document read beforehand is experienced as evidence, and the questions that occur to you reading it alone are usually the ones worth asking.

Connect It to Something in the Business Referral traffic from assistant domains should be segmented in analytics and tracked, with the understanding that it undercounts. Some assistants strip referrer data and some visits arrive looking direct.

State who signs off, how fast, and what is off limits. An agency that knows the constraints will build a plan that fits them. One that finds out gradually will spend the retainer producing work that never ships. llm seo

What Honest Reporting Contains The prompt set, versioned and unchanged since last month. The raw answers, kept in full rather than summarised. Which competitors were named. Which sources were cited. What work was done. What moved, and the specific claim about which work caused it.

Keep the raw text of every answer, not just a tally. Six months in, the archive is the most useful thing you own, because it lets you see exactly when a competitor entered the shortlist, which source appeared alongside them, and whether your own description shifted from something a marketer wrote to something a customer would recognise. A score with no working behind it cannot tell you any of that. llm seo

Assertions with nothing behind them are weaker than silence, because they introduce a detail that fails verification. The pattern that works is reciprocal: your site names the profile, the profile links to your site, and some independent source associates the two without either of you being involved.

Put someone's name against this. Crawler rules sit between marketing, development and whoever administers the content delivery network, which in most organisations means nobody checks them. The failures documented here are not difficult to find, they are simply nobody's job, and a quarterly review taking half an hour prevents the most complete form of invisibility available.

This section sounds procedural and it is the foundation of everything after it. A prompt set quietly edited between runs makes every trend line in the document meaningless, and it is the easiest way to manufacture improvement without doing anything.

This is the least interesting subject in the discipline and the one that most often explains a total absence from generated answers. A brand can do everything else correctly and remain invisible because a line in a text file, or a setting nobody remembers enabling, is turning the relevant crawlers away.

Keep the brief to something you would be willing to send to three suppliers unchanged. The temptation is to tailor each one, which feels attentive and makes the resulting proposals impossible to compare. Identical briefs produce differences that reflect the agencies rather than the instructions, which is the entire point of asking more than one.

Name the Buyer, Not the Segment Marketing documents describe segments. Briefs need people. Who specifically buys from you, what situation are they in when they start looking, and what have they already tried before they arrive.

One test separates a report written to inform from one written to reassure. Read it and try to write down a question it does not answer. In a good report you will find several, because it contains enough specifics to make new questions obvious. In a padded one you will struggle, not because everything is covered but because there is nothing specific enough to interrogate.