Yes — if you haven’t touched your robots.txt since you launched your site, there’s a real chance you’re blocking GPTBot, ClaudeBot, or PerplexityBot right now. These are the AI crawlers that feed ChatGPT Search, Claude, and Perplexity the content they cite. Blocking them doesn’t protect you. It just makes you invisible. Here’s the checklist to find out, and to fix it.
Quotable definition: A robots.txt file is a plain-text instruction set that tells web crawlers which pages they may and may not access. AI answer engines — including ChatGPT Search, Claude, and Perplexity — use dedicated crawlers (GPTBot, ClaudeBot, PerplexityBot) that read this file before indexing your content. If your robots.txt blocks these crawlers, your pages cannot be cited by AI systems, regardless of how well-written or authoritative they are.
Why This Is Suddenly a Real Problem for SG Sites
Most Singapore business websites were built when the only crawlers that mattered were Googlebot and Bingbot. Developers routinely added broad Disallow rules — often blocking entire site sections, staging paths, or using a blanket User-agent: * rule that catches everything, including crawlers that didn’t exist yet.
The AI crawler wave arrived after most of those robots.txt files were written. GPTBot launched in August 2023. ClaudeBot followed. PerplexityBot, OAI-SearchBot — the list kept growing. None of these were in anyone’s default template.
The result: many SG sites are handing potential AI citations to competitors who simply left their robots.txt alone. It’s not a strategy. It’s a default setting that no one reviewed.
The Four AI Crawlers You Actually Need to Know
There are dozens of bots out there, but for most Singapore SMEs, four matter for AI citation probability right now.
| Crawler | Powers | User-agent string | Respects robots.txt? |
|---|---|---|---|
| GPTBot | ChatGPT (training + search) | GPTBot |
Yes |
| OAI-SearchBot | ChatGPT Search (live retrieval) | OAI-SearchBot |
Yes |
| ClaudeBot | Anthropic / Claude | ClaudeBot |
Yes |
| PerplexityBot | Perplexity AI | PerplexityBot |
Yes |
All four crawlers read raw HTML. They do not execute JavaScript. If your content lives inside a React or Vue app and the server returns an empty shell until JS hydrates it, these bots see nothing — even if your robots.txt is perfectly configured. That’s a separate fix, covered below.
One Critical Dependency Most Guides Skip
ChatGPT Search is built on Bing’s index. That’s not a guess — OpenAI confirmed the arrangement. Which means if Bingbot can’t crawl you, OAI-SearchBot’s real-time retrieval layer has nothing to pull. Being indexed in Bing is a hard prerequisite for ChatGPT Search citations, not a nice-to-have.
Check your Bing Webmaster Tools now. [VERIFY: current percentage of SG sites that have verified in Bing Webmaster Tools vs Google Search Console] Many Singapore site owners have never set it up — because historically, Bing traffic was modest and easy to deprioritise. In 2026, that’s a more expensive oversight than it used to be.
Fixing Bing indexation doesn’t require a separate content strategy. Submit your sitemap at bing.com/webmasters, verify ownership, and confirm Bingbot isn’t blocked in your robots.txt. That’s the minimum.
The AI-Crawler Checklist: 8 Checks, 30 Minutes
Work through these in order. Each check takes under five minutes if you have site access.
- Retrieve your live robots.txt. Go to
yourdomain.com.sg/robots.txtin a browser. If you get a 404 or blank page, you have no file — which means all crawlers are allowed by default. Not necessarily a problem, but confirm it’s intentional. - Check for a blanket wildcard block. Look for
User-agent: *followed byDisallow: /. This blocks every crawler including all four AI bots. It’s the nuclear option. Remove it or scope it narrowly. - Search for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot by name. If any appear under a
Disallow: /rule, they’re blocked. Add an explicitAllow: /rule under each, or remove the disallow entirely. - Check your CMS plugin settings. Yoast SEO, Rank Math, and similar plugins can generate or override robots.txt entries. If you’ve ticked “Discourage search engines” in WordPress Settings → Reading at any point and forgot to untick it, that flag likely generated a blocking rule. Check it.
- Test with Google’s robots.txt Tester (Search Console → Legacy Tools). Enter each AI crawler’s user-agent string and your key page URLs. It’ll tell you pass or block. It tests Google’s parser, which is close enough to how most bots interpret the file.
- Check for JavaScript rendering dependency. Use
curl -A "GPTBot" https://yourdomain.com.sg/your-key-pagein a terminal, or use a fetch-as-bot tool. If the HTML returned is thin — no body content, just script tags — AI crawlers can’t read your page regardless of robots.txt. - Verify Bing indexation. Search
site:yourdomain.com.sgon Bing. If fewer pages appear than on Google, or none, set up Bing Webmaster Tools and submit your sitemap. - Check your sitemap is referenced. Your robots.txt should include a line:
Sitemap: https://yourdomain.com.sg/sitemap.xml. AI crawlers use this to discover content efficiently. If it’s missing, add it — it’s one line.
The llms.txt File: Useful or Just Admin?
You may have seen recommendations to add an llms.txt file — a plain-text document that summarises what your site does, intended to help LLMs understand your brand quickly. It’s a reasonable idea in principle.
Here’s the honest picture: Ahrefs found that 97% of domains with a valid llms.txt received zero requests for the file from any crawler. The spec is genuinely new, adoption among AI systems is patchy, and the absence of llms.txt will not hurt you today. The presence of it won’t move the needle in any measurable way yet.
Kaizenaire’s view: spend your 30 minutes on the eight-point checklist above before touching llms.txt. Fix the basics. The experimental layer can wait.
What the Robots.txt Fix Actually Gets You
Opening the door to AI crawlers improves your probability of citation. It doesn’t guarantee it. There’s a meaningful difference, and any agency that presents it otherwise is overselling.
Citation depends on crawl access, yes — but also on whether your content actually answers questions better than competitors, whether you’re treated as an authoritative entity by the AI system, and whether your pages load cleanly without JavaScript walls. Robots.txt is the bouncer at the door. Getting past the bouncer is necessary. It’s not sufficient.
Think of it this way: if your robots.txt is blocking GPTBot, fixing it is table stakes — the floor, not the ceiling. The ceiling requires proper AEO and GEO work on top of clean technical foundations.
Who Should NOT Spend Time on This Right Now
If your business operates entirely on referrals, is pre-revenue, or has fewer than five pages of real content, AI crawler access is not your constraint. You don’t have enough indexable substance for citation even if every bot can reach you freely.
Fix robots.txt when you have content worth citing. If you’re still building the content layer, that’s the prior problem. [VERIFY: average number of indexed pages on SG SME websites below 50 employees — useful benchmark if available]
This also isn’t urgent if you’re in a category where AI answer engines don’t yet drive discovery — some B2B niches with highly specialised, relationship-driven buying still see negligible AI-referral traffic. Check your own analytics for referrals from ChatGPT, Perplexity, or Claude before investing time here.
The Correct robots.txt Snippet for AI Crawlers
If your current file has no AI-specific rules and you want to be explicit, here’s a clean addition. Place it at the bottom of your existing robots.txt — don’t overwrite existing rules:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://yourdomain.com.sg/sitemap.xmlIf you want to allow AI crawlers but block specific sections (staging, client portals, internal tools), use Disallow: /staging/ under each agent rather than a blanket block. Surgical beats nuclear every time — though “surgical robots.txt” sounds like a procedure you’d rather not explain to your accounts team.
Frequently Asked Questions
Does blocking GPTBot affect my Google rankings?
No. GPTBot is OpenAI’s crawler — it has no relationship with Google’s ranking algorithm. Blocking it won’t hurt your Google position. It simply prevents your content from being fed into ChatGPT’s training data and, separately, reduces the chance of ChatGPT Search citing you in real-time results. These are independent systems.
My developer says robots.txt is already “open.” Is that enough?
It’s necessary but not sufficient. Open robots.txt means crawlers can attempt access — but if your pages are JavaScript-rendered and return an empty HTML shell to bots, AI crawlers still see nothing. Confirm your key pages return full content in the raw HTML response, without JavaScript execution.
Should I block AI crawlers to protect my content?
That’s a legitimate choice, particularly for publishers worried about training-data use. The trade-off is clear: blocking GPTBot and ClaudeBot protects your content from being used in model training but removes any chance of AI citation. For most Singapore SMEs trying to attract new enquiries, blocking AI crawlers is the wrong default in 2026.
Does this work for Perplexity and Claude as well, or just ChatGPT?
The same logic applies to all four major AI crawlers. Each has its own user-agent string, each respects robots.txt, and each requires unblocked access plus readable HTML to cite your content. Fixing one without the others gives you partial coverage — worth doing, but the full checklist covers all four.
How long does it take AI crawlers to re-crawl after I fix robots.txt?
There’s no published crawl-schedule commitment from OpenAI, Anthropic, or Perplexity. [VERIFY: any published recrawl frequency from GPTBot or ClaudeBot]. In practice, important pages on active domains tend to get recrawled within days to a few weeks. Submitting your sitemap in Bing Webmaster Tools may accelerate OAI-SearchBot’s access, given ChatGPT Search’s Bing dependency.
Is there a free way to check if AI crawlers have actually visited my site?
Yes — check your server access logs for user-agent strings containing “GPTBot,” “ClaudeBot,” “PerplexityBot,” or “OAI-SearchBot.” Most hosting control panels (cPanel, Kinsta, WP Engine) let you download raw access logs. Alternatively, a crawl-monitoring plugin or Cloudflare’s bot analytics can surface this data without log-parsing.
If I fix all this, will I definitely appear in ChatGPT answers?
No. Clean technical access improves your probability of being crawled and considered for citation. Whether you’re actually cited depends on your content quality, your topical authority relative to competitors, how well your pages answer specific questions, and how the AI model weights sources. Technical access is the entry requirement — it’s not the selection criterion.
If you’re not sure where your site stands right now, the fastest starting point is a structured audit. Kaizenaire’s free AI-Visibility Check looks at robots.txt configuration, Bing indexation, JavaScript rendering, and content-level citation signals — and tells you specifically what’s blocking you, not just whether something is. No commitment required.