AI crawlers, robots.txt and llms.txt: what practices should set up

ChatGPT, Perplexity, Gemini and Claude answer patients' questions with content from the web. To do this, the providers send out programs that read websites: AI crawlers. Which of them may read your practice website is set out in a small text file, the robots.txt. If it contains the wrong rules, your practice won't appear as a source in AI answers, no matter how good your content is.

In this article we show which crawlers exist, which ones a practice should allow, what the entries look like and what the new llms.txt file is all about.

What are AI crawlers?

AI crawlers

Programs that providers such as OpenAI, Anthropic, Perplexity or Google use to read websites. Some collect material for training their models, others fetch content for answers in AI search.

For practices, this distinction is the crux: search crawlers decide whether your website can be cited as a source in an AI answer. Training crawlers collect texts that future models learn from. Blocking the first costs you visibility; blocking the second is a matter of your own stance.

Important for Google: the AI Overviews in Google Search are generated from the normal Google index. Google states explicitly that Google-Extended has no effect on inclusion in Google Search (Google Search Central). If you allow Googlebot, you can appear in AI Overviews.

What the robots.txt looks like

The robots.txt sits in the root directory of your website. Each block names a crawler and says what it may do. For a practice that wants to appear in AI answers, a few lines are enough:

Can I prohibit training?

Yes. Under § 44b of the German Copyright Act (UrhG), freely accessible content may be used for text and data mining unless the rights holder has reserved this in machine-readable form (§ 44b UrhG). An entry in the robots.txt for training crawlers is the usual way of declaring this reservation. Whether it is legally sufficient in every case has not yet been finally settled. Many website operators therefore also declare it in their legal notice (Impressum).

What is llms.txt?

llms.txt

A text file in the root directory of a website that shows language models, in short form, what the website offers and which pages are important.

llms.txt is a proposal from 2024 (llmstxt.org), not a binding standard. Whether the major AI providers actually use it has not yet been proven. We still maintain one on med-direction.de, with our services and the terms from our glossary, and treat it as a bonus. For a practice, the effort is small and the benefit uncertain. robots.txt and readable pages come first.

Readable without JavaScript

Not all AI crawlers execute JavaScript. If a website only assembles its content in the browser, such a crawler sees an almost empty page. What is safe is what the server delivers as finished HTML. How a practice website should be built technically is described in the article Websites for doctors.

Five-minute self-check

• Search crawlers allowed? Otherwise you cannot be cited as a source.
• No block at your hosting provider or CDN? Otherwise your robots.txt has no effect.
• Content readable without JavaScript? Otherwise the crawler sees an empty page.

Not every practice has the time or patience to click through hosting menus and CDN settings. You don't have to: we check your robots.txt, any blocks and the readability of your pages with professional tools, and set everything up to match your stance.

Conclusion and outlook

Whether a practice can appear as a source in AI answers is decided first in an inconspicuous place: the question of who may read the website. For most practices, three steps are enough:

• Allow search crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot.
• Check for blocks at your hosting provider or CDN: otherwise even the best robots.txt has no effect.
• Deliver content as finished HTML: so that crawlers without JavaScript can read everything too.

Whether you allow training crawlers is a question of your stance, not your visibility. llms.txt is a bonus, not a must. After that, content, reviews and consistent practice details decide whether your practice is actually named. If you want it done faster, we take care of it as part of our SEO and AI visibility service, from the technical setup to checking whether your practice ultimately shows up in the answers.

The outlook: the list of crawlers is getting longer, not shorter. Alongside search and training crawlers, AI agents are arriving that compare practices on behalf of patients and will soon request appointments too. If you clearly define today who may read your website, you are prepared. So check your robots.txt every six months, because providers occasionally rename their crawlers or introduce new ones. Whether llms.txt becomes a standard remains open. We keep an eye on it for our clients.