Shield your website from unwanted AI training
Scan your site for AI crawler exposure, robots.txt rules, legal notices, hosting setup, and practical protection steps — while preserving normal search visibility.
Free scan
What the free scan checks
A superficial but real check of your site's public signals. No login, no payment, no access to private data.
Hosting provider
Detects hosting platform and CDN signals such as Wix, Squarespace, Vercel, Cloudflare, and more.
robots.txt analysis
Looks for a robots.txt file at your site root and inspects its directives.
AI crawler permissions
Checks for rules covering GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot, and others.
Search crawler permissions
Confirms that classic search crawlers like Googlebot and Bingbot are still allowed.
Impressum & legal
Detects the presence of impressum, legal notice, terms, and privacy pages.
AI training language
Scans visible text for AI / training / TDM / IP-rights language (EN + DE).
CDN / provider hints
Reads response headers and asset URLs to infer the platform powering your site.
Ownership hints
Looks for public metadata and ownership signals visible on the site.
CrawlFence Protection Report
What you get in the full protection report
Provider-specific instructions, copy-paste snippets, and a plain-language creator/IP-rights checklist designed for non-technical site owners.
- ✓Exact provider-specific protection steps
- ✓Suggested robots.txt block tailored to your site
- ✓Suggested legal wording prompts (not legal advice)
- ✓CMS / hoster instructions (Wix, WordPress, Webflow, Cloudflare, …)
- ✓SEO- and GEO-preserving setup
- ✓Plain-language risk explanation
- ✓Creator and IP-rights protection checklist
Preview
CrawlFence Protection Report
Full instructions are unlocked after checkout. Nothing is hidden in the page — paid content is generated only after payment.
Built for creators
CrawlFence is designed for people who publish original work online and want practical protection without becoming web infrastructure experts. It helps artists, writers, photographers, and designers put visible technical and legal-structural boundaries around their work.
Protect without disappearing from search
CrawlFence helps you reduce unwanted AI training exposure while keeping normal search engines and discoverability where you want them. You decide which AI search and discovery crawlers may still see your content.
AI training crawlers like GPTBot, CCBot, ClaudeBot
Classic SEO crawlers like Googlebot and Bingbot
AI answer engine crawlers per case (GEO visibility)
Frequently asked questions
Can CrawlFence stop all AI scraping?+
No. robots.txt and crawler instructions are signals. Compliant crawlers may respect them; non-compliant crawlers may ignore them. CrawlFence helps you put clear technical and legal-structural barriers in place, but cannot guarantee that every crawler will comply.
Will this hurt my Google SEO?+
No, when configured correctly. CrawlFence focuses on restricting AI training crawlers (like GPTBot, Google-Extended, CCBot) while keeping classic search crawlers (Googlebot, Bingbot) fully allowed.
What is robots.txt?+
robots.txt is a small text file at the root of your website that tells well-behaved crawlers which areas they may or may not visit. It is a request, not enforcement.
What is GPTBot?+
GPTBot is OpenAI's web crawler used to collect training data. You can ask it not to crawl your site by adding a rule for GPTBot in robots.txt.
What is Google-Extended?+
Google-Extended is a separate user-agent token Google uses to control whether your content is used for Gemini and Vertex AI training, independent of Google Search.
What does "AI training blocked" mean?+
It means your robots.txt asks AI training crawlers not to use your content for model training. Compliant bots will follow this signal; non-compliant bots may not.
Does this protect my copyright or IP rights?+
CrawlFence helps make your intent visible through technical signals and clear legal-structural language on your site. It does not replace a lawyer or grant legal enforcement.
Do I need a lawyer?+
For binding legal wording, yes. CrawlFence gives you a practical checklist and draft suggestions, but a lawyer should review final terms.
Can I protect images?+
You can ask compliant AI crawlers to skip your images and request that providers not use them for training. Determined non-compliant actors may still attempt extraction.
Why are the full instructions paid?+
The free scan is a real, useful check. The full CrawlFence Protection Report is provider-specific, hand-tailored guidance that takes real work to maintain.
CrawlFence provides technical and legal-structural information, not legal advice. robots.txt and crawler instructions are signals, not guaranteed enforcement.