For most Singapore SME owners, blocking AI crawlers is a mistake. Unless you’re a publisher whose entire revenue model depends on people clicking through to read your content, refusing AI access means you simply won’t appear in ChatGPT, Perplexity, Google’s AI Overviews, or any of the answer engines your next customer is already using. The decision deserves more than a one-line robots.txt edit.
Quotable definition: AI crawlers are automated bots — operated by companies such as OpenAI (GPTBot), Google (Google-Extended), Anthropic (ClaudeBot), and Perplexity (PerplexityBot) — that read your web pages to train large language models or populate real-time AI answer panels. Blocking them prevents those systems from learning about your business, reducing the probability that an AI cites you when a prospective customer asks a relevant question.
Why this question is appearing now
AI Overviews appear on roughly 48% of Google queries as of mid-2026. That’s not a niche feature — it’s the default experience for nearly half the searches your customers run. Alongside that, zero-click searches hit approximately 68% of all Google searches in 2026 (SparkToro). Put those two numbers together and you get a picture where the answer engine increasingly is the destination. Your website doesn’t get the visit; the AI panel absorbs the answer.
In that environment, the question isn’t just “should I block crawlers?” It’s “do I want to exist in AI-generated answers at all?” Those are the same question.
What blocking actually does — mechanically
When you add a crawler to your robots.txt disallow list, you’re issuing a polite instruction. Most reputable AI crawlers — GPTBot, Google-Extended, ClaudeBot — respect it. Some do not. The ones that don’t are typically the ones you can’t control anyway, so the block primarily filters the legitimate players: the very systems whose answer panels you’d want to appear in.
Blocking GPTBot, for instance, tells OpenAI’s training and retrieval systems to skip your site. ChatGPT’s browsing and citation features may then default to a competitor who didn’t block. You haven’t protected your content so much as redirected traffic to someone else’s content. This is the mechanism that most “block everything” advice skips entirely.
The comparison: block vs. allow, by business type
| Business type | Block AI crawlers? | Reasoning |
|---|---|---|
| SG SME selling services (legal, accounting, F&B, education, trade) | No — allow | Discovery is the goal. AI citation builds brand presence at zero marginal cost per impression. |
| SG e-commerce store (product pages, prices, specs) | Mostly allow | Product data in AI answers can drive purchase intent. Block only if you have dynamic pricing you don’t want scraped. |
| Publisher / media site (paywalled or ad-funded content) | Consider selective blocking | If click-through is your monetisation unit, AI consumption without referral traffic is a real cost. Evaluate per crawler. |
| B2B SaaS / professional services with gated IP | Block gated pages only | Block your proprietary tools, client dashboards, unpublished research. Allow your public marketing and content pages. |
| Anyone with sensitive customer data or legal constraints | Block relevant sections | PDPA obligations apply. Don’t expose personal data pages to any crawler — AI or otherwise. |
The citation signal most businesses are ignoring
Here’s the part that surprises most people. Brand web mentions correlate approximately 0.66 with AI citation frequency, versus roughly 0.22 for traditional backlinks (Ahrefs). That’s a significant gap. The implication: AI systems weight brand recognition — how often your brand name appears across the web in credible contexts — far more heavily than the link graph that classic SEO optimised for.
Blocking crawlers doesn’t just affect whether GPT reads your site once. It affects whether your brand accumulates the kind of web presence that makes AI systems treat you as a credible, citable entity over time. It’s a compounding decision, not a one-off toggle.
This is also why AEO and GEO services focus heavily on structured content and brand mention density, not just technical robots.txt hygiene.
The honest case for blocking — it exists
Kaizenaire’s view is that selective blocking is legitimate and sometimes correct. If you run a specialist research firm and your competitive advantage is a proprietary methodology you’ve spent three years developing, training a commercial LLM on it for free isn’t a gift you’re obliged to give. Block the relevant pages. Put a summary on a public landing page instead — you get the citation signal without giving away the detailed IP.
Similarly, if you’re a local media business with 40 staff whose revenue depends on page views and display advertising, an AI that summarises your articles without sending you traffic is eating your lunch without paying for it. The argument for selective blocking there is economically coherent. The answer isn’t “block everything” — it’s “block the paywall, publish a structured summary, and extract what citation value you can from the public portion.”
Robots.txt: the specific crawlers and what they do
Not all AI bots are equal. Here’s what the main ones actually do when they visit your site:
- GPTBot (OpenAI) — used for both training data and ChatGPT’s real-time browsing. Blocking it affects both. It respects robots.txt with an accuracy rate that OpenAI describes as near-complete for compliant crawls — though this is self-reported. [VERIFY: independent audit of GPTBot robots.txt compliance]
- Google-Extended — separate from Googlebot. Controls whether your content feeds Google’s AI training (Bard/Gemini). Blocking it does not affect your standard Google Search rankings — that’s Googlebot, a different agent entirely.
- ClaudeBot (Anthropic) — used for Claude model training. Respects robots.txt. Blocking it affects Claude’s knowledge of your brand, not your Google presence.
- PerplexityBot — powers Perplexity’s real-time citations. If you block it, you won’t appear in Perplexity answers. Given Perplexity’s growth trajectory in research-heavy B2B use, this matters more than most SG SMEs currently realise.
There is a moderately absurd specificity to this: you can now write a robots.txt that permits exactly 2.3 crawlers while excluding the other 1.7, measured in fractions only if you’re doing something genuinely unusual with conditional logic and path-based rules. Most businesses don’t need that level of precision. A blanket “allow all reputable AI crawlers, block sensitive paths” policy handles 95% of cases cleanly.
The inconvenient truth
AI citation currently drives a very small share of direct referral clicks. If your goal is traffic to your website this quarter, optimising for AI crawler access is not your most urgent lever — your Google rankings, paid search, and conversion rate still matter more in the short term. What AI citation builds is brand presence in zero-click answers: someone asks ChatGPT “which accounting firm in Singapore handles XBRL filing?” and your name appears, without a click ever happening. That’s real value — but it’s brand value, not session volume. If you’re measuring success purely in GA4 sessions, you’ll undervalue this channel and potentially make the wrong blocking decision for the wrong reason.
What to do in the next 30 minutes
- Open your robots.txt (yourdomain.com/robots.txt) and check whether GPTBot, Google-Extended, ClaudeBot, or PerplexityBot are currently blocked. Many sites have blanket “Disallow: /” rules added by developers who were thinking about scraping, not AI citation.
- Identify sensitive paths — client portals, gated tools, any page containing personal data under PDPA scope. Add specific disallow rules for those paths only.
- Allow public marketing and content pages explicitly. A simple “User-agent: GPTBot / Allow: /” for your public pages is sufficient.
- Check your brand mention footprint. Search your brand name in ChatGPT and Perplexity right now. If you don’t appear when you ask a question your business should own, the issue may not be the robots.txt at all — it may be that you lack the structured, citable content to appear in the first place.
- Review quarterly. The AI crawler landscape is evolving fast enough that a policy set today needs a check-in every 90 days.
Frequently Asked Questions
Real questions we hear from Singapore business owners on this topic.
Will blocking AI crawlers protect my content from being copied?
Partially. Reputable crawlers like GPTBot and ClaudeBot respect robots.txt. But bad actors — scrapers, spam bots, less scrupulous data aggregators — generally don’t. Blocking via robots.txt gives you control over the legitimate AI systems, which are also the ones that would have cited you favourably. It doesn’t protect you from the ones you actually distrust.
Does blocking Google-Extended affect my Google Search ranking?
No. Google-Extended is a separate crawler from Googlebot. Blocking it affects whether your content trains Google’s AI products (Gemini, AI Overviews’ underlying models). Your standard organic search ranking is governed by Googlebot, which isn’t affected by a Google-Extended disallow rule. These are genuinely separate systems.
I run an F&B business in Singapore. Should I care about AI crawlers?
Yes, more than you might think. When someone asks ChatGPT or Perplexity “best laksa near Tanjong Pagar” or “halal catering in Tampines,” those answers pull from structured web content. If your site is blocked or lacks the right structured information, you won’t appear — even if you’re the obvious answer. Allowing crawlers and adding structured content (schema markup, clear location data, menu details) improves your probability of citation.
How do I know if an AI crawler is already visiting my site?
Check your server logs or analytics. Look for user-agent strings containing “GPTBot,” “ClaudeBot,” “PerplexityBot,” or “Google-Extended.” Most managed hosting dashboards (SiteGround, Cloudflare, Kinsta) show bot traffic at the request level. If you’re on a basic shared host, your server log file (access.log) will contain the raw entries — ask your developer to pull a week’s worth.
Can I block AI training crawlers but still appear in AI search results?
Sometimes. GPTBot serves both training and real-time browsing, so blocking it affects both. Google-Extended primarily affects training rather than retrieval, so blocking it may reduce your influence on future Gemini outputs without immediately removing you from existing AI Overviews. The clean answer: it depends on the crawler. There’s no universal switch that separates “train on my content” from “cite my content” across all systems.
Is there a right robots.txt template for Singapore SMEs?
A reasonable starting position: allow all named AI crawlers at the root level, then add specific disallow rules for any path containing customer data, gated content, or proprietary tools. Don’t copy a generic “block all bots” template from a 2019 SEO guide — that era’s threat model was scraping and duplicate content penalties, not AI citation. The context has changed materially.
What if I’m not sure whether my content is appearing in AI answers?
Test it manually: open ChatGPT, Perplexity, and Google with AI Overviews active, and ask the questions your customers actually ask about your category. If your brand doesn’t appear in any of them, robots.txt may be one factor — but structured content, brand mention density, and content authority are typically bigger gaps. A proper AI-Visibility Check maps all of those factors at once, which is more useful than fixing robots.txt in isolation.
If you want to know exactly where you stand — which AI systems are citing you, which are ignoring you, and what’s causing the gap — run the free AI-Visibility Check. It takes about two minutes to submit, and we’ll return a structured read of your current AI footprint with specific, actionable gaps identified. No pitch call required to get the results.