问题原文
原帖链接 · Deepthi R · CC BY-SA 4.0
I would like to know if I can put a 301 redirect on robots.txt file and have it hosted in a different location. I'd like to have it on a cloud front layer.
example: https://www.mydomain.example/robots.txt redirects to https://differentdomain.example/example/robots.txt
社区高分解答
原帖链接 · Maximillian Laumeister · CC BY-SA 4.0
Google allows the robots.txt 301 redirection you're talking about:
Google follows at least five redirect hops as defined by RFC 1945 for HTTP/1.0 and then stops and treats it as a 404.
I couldn't find any information about Bing's crawler.
It's my suspicion that smaller, poorly-written, non-search-engine bots might not be smart enough to follow the redirect, and might take the 301 status code as a blank check to hit your website.
If this is about keeping your robots.txt automatically up to date with the robots.txt on the other domain, I'd consider a reverse-proxy setup if possible.
On the other hand if this is about lessening the strain on your server by offloading your robots.txt to a CDN, well, your server is still going to have to serve all of those 301 status codes, which probably won't be much of a lighter load than just serving up the robots.txt file in the first place. And if serving your robots.txt file even tens of thousands of times per day is putting any real strain on your server, that's a surefire sign that there's an issue with your server.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
