大网数据广告活动
网站运营与故障问答

How can I use robots.txt to disallow subdomain only?

· 52CE 我爱测

My code base is shared between several environments (live, staging, dev) & sub-domains ( staging.example , dev.example , etc.) and only two should be allowed to be crawled (ie. www.example and example ). Normally I'd modify /robo…

问题原文

原帖链接 · alexus · CC BY-SA 3.0

My code base is shared between several environments (live, staging, dev) & sub-domains (staging.example, dev.example, etc.) and only two should be allowed to be crawled (ie. www.example and example). Normally I'd modify /robots.txt and add Disallow: /, but due to shared code base I cannot modify /robots.txt without affecting all (sub)domains.

Any ideas how to go about it?

已采纳解答

原帖链接 · MrWhite · CC BY-SA 3.0

You can serve a different robots.txt file based on the subdomain through which the site has been accessed. One way of doing this on Apache is by internally rewriting the URL using mod_rewrite in .htaccess. Something like:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^(www\.)?example\.com$ [NC]
RewriteRule ^robots\.txt$ robots-disallow.txt [L]

The above states that for all requests to robots.txt where the host is anything other than www.example.com or example.com, then internally rewrite the request to robots-disallow.txt. And robots-disallow.txt will then contain the Disallow: / directive.

If you have other directives in your .htaccess file then this directive will need to be nearer the top, before any routing directives.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-08-18)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。