问题原文
原帖链接 · clickbait · CC BY-SA 4.0
User-agent: *
Disallow: /robots.txt
What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself.
It is actually very common for robots.txt to disallow itself. Many websites disallow everything:
User-Agent: *
Disallow: /
That drective to disallow everything would include robots.txt. I myself have some websites like this. Despite disallowing everything including robots.txt, search engine bots refresh the robots.txt file periodically.
Google's John Mueller recently confirmed that Googlebot still crawls a disallowed robots.txt: Disallowing Robots.txt In Robots.txt Doesn't Impact How Google Processes It. So even if you specifically called out Disallow: /robots.txt, Google (and I suspect other search engines) wouldn't change their behavior.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
