大网数据广告活动
网站运营与故障问答

Can robots.txt redirect to a different domain?

· 52CE 我爱测

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer. example: https://www.mydomain.example/robots.txt redirects to https://dif…

问题原文

原帖链接 · Deepthi R · CC BY-SA 4.0

I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer.

example: https://www.mydomain.example/robots.txt redirects to https://differentdomain.example/example/robots.txt

社区高分解答

原帖链接 · Maximillian Laumeister · CC BY-SA 4.0

Google allows the robots.txt 301 redirection you're talking about:

Google follows at least five redirect hops as defined by RFC 1945 for HTTP/1.0 and then stops and treats it as a 404.

https://developers.google.com/search/reference/robots_txt

I couldn't find any information about Bing's crawler.

It's my suspicion that smaller, poorly-written, non-search-engine bots might not be smart enough to follow the redirect, and might take the 301 status code as a blank check to hit your website.

If this is about keeping your robots.txt automatically up to date with the robots.txt on the other domain, I'd consider a reverse-proxy setup if possible.

On the other hand if this is about lessening the strain on your server by offloading your robots.txt to a CDN, well, your server is still going to have to serve all of those 301 status codes, which probably won't be much of a lighter load than just serving up the robots.txt file in the first place. And if serving your robots.txt file even tens of thousands of times per day is putting any real strain on your server, that's a surefire sign that there's an issue with your server.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-03-11)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。