Not every AI bot crawling your website is doing the same job. Some of them are building search answers that mention your practice by name. Others are vacuuming up your content to train a language model and will never send a single patient your way.
If you don't know the difference, you might block the wrong ones -- and disappear from the AI search tools your patients are already using.
Two categories, two completely different goals
AI bots split into two camps: search crawlers and training crawlers. The names sound similar. What they do with your content couldn't be more different.
Search crawlers read your site so they can cite it in real-time answers. When a patient asks Perplexity "NaProTechnology doctor near me" or uses ChatGPT's search feature to find a FertilityCare practitioner, these bots are the reason your site can appear in the response. The major ones are OAI-SearchBot (OpenAI's search crawler), PerplexityBot, and ChatGPT-User. They retrieve your pages, pull relevant facts, and link back to you as a source.
Training crawlers scrape your content to improve AI models. GPTBot (also OpenAI, but separate from their search bot), CCBot (Common Crawl's open dataset builder), Google-Extended, and ClaudeBot all fall into this category. They're collecting text to make language models smarter. They don't cite you. They don't link to you. They don't send traffic.
Here's how stark the difference is: Cloudflare's 2025 analysis found that Anthropic's training crawler had a ratio of 38,000 crawls for every single referral visit back to publishers. Google Search, by comparison, runs at about 14 crawls per referral. That's a 2,700x difference in how much value flows back to your site.
Why this matters for your practice website
The practical question is simple: which bots should your site welcome, and which should it turn away?
If you block PerplexityBot because you're worried about "AI scraping," you've just pulled yourself out of one of the fastest-growing search tools in healthcare. If you block OAI-SearchBot, patients using ChatGPT search won't see your practice in their results. That's not a theoretical problem. Cloudflare reports that AI crawler traffic surged dramatically in 2025, with GPTBot's share jumping from 5% to 30% of all AI crawling between May 2024 and May 2025.
On the other hand, there's no patient-facing benefit to letting GPTBot, CCBot, or Google-Extended scrape your clinical content for model training. A page you wrote about endometriosis treatment using restorative reproductive medicine gets absorbed into a training dataset, and you get nothing back. Not a citation, not a link, not a patient.
The robots.txt solution
Your robots.txt file is where you draw the line. It's a simple text file at the root of your website that tells bots what they can and can't access. Here's what a sensible setup looks like for a medical practice:
Allow these (search crawlers that cite you):
PerplexityBot, OAI-SearchBot, ChatGPT-User -- these power the AI search tools patients are using right now.
Block these (training crawlers that don't):
GPTBot, CCBot, Google-Extended, ClaudeBot, Bytespider -- these scrape for model training with no meaningful referral traffic in return.
One important caveat: OpenAI's documentation notes that OAI-SearchBot and GPTBot share crawl data when both are allowed. If you allow OAI-SearchBot but block GPTBot, your content stays available for search results without contributing to model training. That's the right configuration for most practices.
One more thing worth knowing
Vercel analyzed over 1.3 billion AI crawler requests and found that none of the major AI crawlers render JavaScript. If your site is built on a JavaScript-heavy platform where content loads dynamically, AI search bots can't read it. They see a blank page. This is separate from the robots.txt issue, but it compounds the problem -- even if you allow the right crawlers, they still need to be able to read what's on the page.
For RRM and NaPro practice sites, this is usually less of an issue since most run on WordPress or static site builders that serve HTML directly. But it's worth confirming with whoever built your site.
The bottom line
You don't have to choose between protecting your content and being visible to patients. The two types of AI bots have different user-agent strings, which means your robots.txt can treat them differently. Block the training bots. Welcome the search bots. It takes about five minutes to set up, and it's one of the highest-leverage changes a practice can make for AI search visibility right now.
The patients searching for NaProTechnology, restorative reproductive medicine, and fertility awareness-based methods on AI platforms are already out there. The only question is whether your site is part of the answer they get back.
Frequently asked questions
Does blocking GPTBot affect my Google search rankings?
No. GPTBot is OpenAI's training crawler and has nothing to do with Google Search. Blocking it won't affect your position in Google results. Google uses Googlebot for search rankings and Google-Extended for AI training. These are separate systems.
Should I block all AI crawlers in my robots.txt?
No. Blocking all AI crawlers removes your site from AI-powered search tools like Perplexity and ChatGPT search, where patients are increasingly looking for practitioners. Block the training crawlers (GPTBot, CCBot, Google-Extended) and allow the search crawlers (PerplexityBot, OAI-SearchBot, ChatGPT-User).
Can AI crawlers read JavaScript-rendered content on my website?
Most cannot. Vercel's analysis of over 1.3 billion AI crawler requests found that none of the major AI crawlers render JavaScript. If your site relies on client-side rendering, AI search systems may not be able to read your content at all.
How do I know which AI bots are crawling my medical practice website?
Check your server logs or Cloudflare dashboard for user-agent strings like GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and CCBot. Cloudflare's bot analytics can show you exactly which crawlers are hitting your site and how often.