{"id":5071,"date":"2026-09-10T00:31:47","date_gmt":"2026-09-10T03:31:47","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/09\/10\/your-robots-txt-might-not-say-what-you-think-it-says\/"},"modified":"2026-09-10T00:31:47","modified_gmt":"2026-09-10T03:31:47","slug":"your-robots-txt-might-not-say-what-you-think-it-says","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/09\/10\/your-robots-txt-might-not-say-what-you-think-it-says\/","title":{"rendered":"Your robots.txt might not say what you think it says"},"content":{"rendered":"
\n
<\/header>\n
\n

I ship a bunch of small free tools, and last week I wanted a quick way to double-check what robots.txt<\/code> a site was actually<\/em> serving to crawlers \u2014 not what’s sitting in the repo, what’s actually on the wire.<\/p>\n

Turns out those aren’t always the same thing. Cloudflare has a feature called AI Crawl Control that can rewrite robots.txt<\/code> at the edge \u2014 inject its own rules for AI crawlers \u2014 without touching your origin file at all. If you’re only ever checking the file in your repo (or your CMS, or wherever it’s generated), you’d never see that. The version Googlebot, GPTBot, or your own CI sees can be different from the one you think you’re serving.<\/p>\n

Every robots.txt<\/code> tool I could find is a generator<\/em> \u2014 it helps you build a static file. None of them check what’s actually live. So I built a tiny one that does: <\/p>\n

\n
npx github:janibert1\/robots-check yourdomain.com <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Zero dependencies, just fetches the live file and tells you what’s there, including whether anything looks CDN-injected. There’s also a --ci<\/code> flag for build pipelines (exits 1 if the whole site is accidentally disallowed) and a stricter --fail-on-cloudflare-managed<\/code> flag if you want CI to fail on any Cloudflare-injected rewrite specifically, plus a matching GitHub Action (janibert1\/robots-check@v1<\/code>) if you’d rather wire it into a workflow than run it by hand.<\/p>\n
Repo (MIT): https:\/\/github.com\/janibert1\/robots-check<\/a><\/p>\n
If you run a site behind Cloudflare, it’s worth a 10-second check \u2014 worst case it confirms nothing weird is happening.<\/p>\n<\/p><\/div>\n<\/article>\n
  <\/div>\n
 <\/main> <\/div>\n<\/div>\n<\/div>\n<\/div>\n
Fuente: Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
I ship a bunch of small free tools, and last week I wanted a quick way to double-check what robots.txt a site was actually serving to crawlers \u2014 not what’s sitting in the repo, what’s actually on the wire. Turns out those aren’t always the same thing. Cloudflare has a feature called AI Crawl Control […]<\/p>\n","protected":false},"author":1,"featured_media":5070,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"webixso_pending_account_ids":""},"categories":[41],"tags":[],"class_list":["post-5071","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/5071","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=5071"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/5071\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/5070"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=5071"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=5071"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=5071"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}