Are Kenyan Websites Blocking ChatGPT? We Checked 401 Top Sites

By Manasseh Adina, KaziBora Digital · Data collected 12 November 2026 · Sources checked 13 November 2026
Clients keep asking us one question: “Can ChatGPT see my website?” So we checked the robots.txt files of the most-visited Kenyan websites to see which AI crawlers they let in.
The short answer is that very few Kenyan sites block AI search. But one in eight blocks AI training bots, and most of them probably didn’t write that rule themselves.
- 12.2% of 401 top .ke homepages block at least one AI crawler.
- Almost all of those block training bots (GPTBot, ClaudeBot, Google-Extended). Only 2% block any bot that AI search tools use to find and cite pages.
- 35 of the 49 blocking sites (71%) do it through Cloudflare’s managed robots.txt, a one-click setting.
- Sites behind Cloudflare block AI training bots at 29.9%. Everyone else: 3.4%.
- 8 sites block AI bots while also publishing an llms.txt file written to help AI tools. Those two choices contradict each other.
First, there are two kinds of AI bot
Training bots learn from your pages. Search bots send you customers. They are not the same.
Most robots.txt advice treats “AI bots” as one thing. They are not. Each AI company runs at least two crawlers, and they do different jobs.
- Training bots collect pages that may be used to train future AI models. Blocking them says “don’t learn from my content”.
- Search and answer bots fetch pages so an AI tool can show, quote and link to them when someone asks a question. Blocking them can keep you out of those answers.
| Bot | Company | Job | If you block it |
|---|---|---|---|
GPTBot |
OpenAI | Training | Your pages shouldn’t be used to train OpenAI models. |
OAI-SearchBot |
OpenAI | Search | OpenAI says opted-out sites won’t be shown in ChatGPT search answers. |
ChatGPT-User |
OpenAI | Fetches a page when a user asks | OpenAI says robots.txt rules may not apply, because a person triggered the visit. |
ClaudeBot |
Anthropic | Training | Your pages shouldn’t be used to train Claude. |
Claude-SearchBot |
Anthropic | Search | Anthropic says it may reduce your visibility in Claude’s search results. |
Google-Extended |
Gemini training and grounding | Google says it does not affect inclusion in Google Search and is not a ranking signal. | |
PerplexityBot |
Perplexity | Search | Perplexity recommends allowing it if you want to appear in its results. |
CCBot, Bytespider, Applebot-Extended |
Common Crawl, ByteDance, Apple | Training datasets | Your pages shouldn’t go into those datasets. |
So “blocking AI” can mean two very different decisions. One is about whether AI companies may learn from your content. The other is about whether AI tools may send you customers.
What we found on Kenya’s most-visited websites
12.2% block AI training bots, but only 2% block the bots that cite you in answers.
We took every .ke domain in the Tranco list of the world’s top 1 million websites (515 domains) and read each site’s robots.txt on 12 November 2026. 401 homepages could be analysed. The data comes from our Kenya SEO Benchmark 2026, which also checks titles, headings, sitemaps and more.
| Bot | Type | Sites blocking it | Share |
|---|---|---|---|
| CCBot | Training | 46 | 11.5% |
| ClaudeBot | Training | 45 | 11.2% |
| GPTBot | Training | 44 | 11.0% |
| Bytespider | Training | 44 | 11.0% |
| Google-Extended | Training | 43 | 10.7% |
| Applebot-Extended | Training | 41 | 10.2% |
| ChatGPT-User | Answer | 8 | 2.0% |
| PerplexityBot | Search | 4 | 1.0% |
| OAI-SearchBot | Search | 2 | 0.5% |
| Claude-SearchBot | Search | 2 | 0.5% |
The pattern is clear. The six training bots are blocked by almost the same group of about 45 sites. The search and answer bots are blocked by 8 sites in total (2.0%). For 41 of the 49 blocking sites, the block covers training only.
Who blocks? Businesses far more than public institutions:
| Group | Sites analysed | Block an AI bot |
|---|---|---|
| Businesses (.co.ke) | 223 | 16.6% |
| Government (.go.ke) | 46 | 4.3% |
| Universities and colleges (.ac.ke) | 86 | 3.5% |
Cloudflare is making the choice for many sites
71% of the blocks come from one Cloudflare setting that someone switched on.
When we opened the robots.txt files of the 49 blocking sites, 35 of them (71%) carried Cloudflare’s managed block. That is a robots.txt that Cloudflare writes and updates for you once the setting is switched on.
Cloudflare’s documentation lists the bots its managed file disallows: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Every other bot gets a “Content-Signal” line saying search yes, AI training no. That matches our data almost exactly: training bots blocked, search bots left alone.
The effect shows up when you split the sites by who serves them:
| Served through | Sites | Block AI training bots |
|---|---|---|
| Cloudflare | 134 | 29.9% |
| Anything else | 267 | 3.4% |
Cloudflare’s docs say the managed robots.txt is off by default, so someone turned it on for each of these sites. That might have been the owner, the developer or an agency. Either way, we suspect many Kenyan business owners don’t know their site has opted out of AI training. If you made that choice on purpose, fine. Our point is that it should be a decision, not an accident.
The contradiction: blocking AI while inviting it
8 sites block AI bots while inviting them in with an llms.txt file.
56 of the 401 sites (14.0%) publish an llms.txt file. That’s a short guide to the site written for AI tools, often switched on in Yoast or Rank Math. 8 of those 56 also block AI training bots in robots.txt.
Both files are allowed, and the two signals cover slightly different things. But it usually means two different people, or two different plugins, made the two decisions without talking to each other. That’s worth a five-minute review.
Another blind spot: pages AI bots can’t read
Nearly one in five homepages looks almost empty to AI crawlers that skip JavaScript.
robots.txt is only half the story. 18.2% of the homepages we checked sent fewer than 100 words of text in their HTML. The rest of the page is drawn by JavaScript after it loads. A December 2024 study by Vercel and MERJ found that the major AI crawlers, including OpenAI’s and Anthropic’s, do not run JavaScript. To those bots, such a page looks nearly empty, even with every bot allowed.
We also left out 69 sites that wouldn’t show us their homepage at all. 27 refused our request (22 of them through Cloudflare) and 42 showed a “One moment, please…” bot-check page. We can’t tell from outside whether those walls also stop AI search crawlers. If your site uses a bot-protection setting, check its logs for blocked AI bots.
How to check your own site in two minutes
Four steps, two minutes, nothing to install.
- Open
yourdomain.co.ke/robots.txtin your browser. - Look for
User-agent:lines naming the bots in the table above, followed byDisallow: /. - If you see
Content-Signalor a block marked as managed by Cloudflare, the rule comes from Cloudflare. In the Cloudflare dashboard, open your domain, go to Security → Settings and look for the AI bot and robots.txt options. (Cloudflare renames menus often. If you can’t find it, search the dashboard for “robots.txt”.) - If you use Yoast or Rank Math, check whether they also publish an
llms.txt, and decide whether that matches your robots.txt.
Three robots.txt setups, and who each one suits
Open, search-only or closed: pick the setup that fits your business.
1. Open to all AI (what we do). Simply don’t name any AI bots. Your normal User-agent: * rules apply to them. This suits most Kenyan service businesses that want to be recommended and are not selling content. We switched kaziboradigital.com to this in October 2026.
2. Found in AI search, not used for training. This is the setup Cloudflare’s managed file roughly gives you. Written out by hand, it looks like this:
# AI search and answer bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# AI training bots: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
Disallow: /
This suits publishers, course creators and anyone whose content is the product.
3. Block all AI. This adds the search bots to the blocked list. It’s a legitimate choice, but know the price. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers. Very few businesses that want customers should choose this.
Methodology and Sources
Methodology
Sample: every .ke domain in the Tranco top-1M list (515 domains), part of our Kenya SEO Benchmark scan. 401 homepages could be analysed. Excluded: 27 that refused our request, 42 that showed a bot-protection page, 7 server placeholders and 38 unreachable or erroring sites.
What we checked: each site’s /robots.txt on 12 November 2026. A bot counts as blocked when its group (or a group naming it) disallows / without allowing /. If a site has no rule naming the bot, its User-agent: * rule applies, so the 2 sites that block every crawler are counted too. We marked a block as Cloudflare-managed when the file carried Cloudflare’s Content-Signal or managed-block markers (re-checked the same day). “Served through Cloudflare” means Cloudflare response headers. llms.txt counts only files that returned real text, not an HTML error page.
Limits: one snapshot of each site’s main domain. robots.txt shows what a site asks for, not what crawlers actually do. We publish percentages, not a list of sites.
Corrections: if you run a .ke site and think we read your robots.txt wrong, email [email protected]. We’ll recheck it and correct this page.
Sources
- OpenAI, Overview of OpenAI crawlers (checked 13 Nov 2026)
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (checked 13 Nov 2026)
- Google, Google’s common crawlers: Google-Extended (checked 13 Nov 2026)
- Perplexity, Perplexity crawlers (checked 13 Nov 2026)
- Cloudflare, Managed robots.txt (checked 13 Nov 2026)
- Vercel and MERJ, The rise of the AI crawler (17 Dec 2024)
- Tranco, top-1M list PY2NJ
Recent Posts
- Every Screenshot on Our New Page Came Out as a Thin Slice November 19, 2026
- A Sponsorship Deck Looked Great. We Found Four Gaps Before Our Client Paid November 17, 2026
- Black Friday 2026 in Kenya: A Playbook for Small Businesses November 16, 2026
- Are Kenyan Websites Blocking ChatGPT? We Checked 401 Top Sites November 13, 2026
- We checked our own portfolio. 7 of 16 client sites were offline. November 12, 2026
Categories
- Business Guides (9)
- Client Stories (4)
- SEO Kenya (3)
- Uncategorized (2)
