大网数据广告活动
网站运营与故障问答

What if robots.txt disallows itself?

· 52CE 我爱测

User-agent: * Disallow: /robots.txt What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?

问题原文

原帖链接 · clickbait · CC BY-SA 4.0

User-agent: *
Disallow: /robots.txt

What happens if you do this? Will search engines crawl robots.txt once and then never crawl it again?

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Robots.txt directives don't apply to robots.txt itself. Crawlers may fetch robots.txt even if it disallows itself.

It is actually very common for robots.txt to disallow itself. Many websites disallow everything:

User-Agent: *
Disallow: /

That drective to disallow everything would include robots.txt. I myself have some websites like this. Despite disallowing everything including robots.txt, search engine bots refresh the robots.txt file periodically.

Google's John Mueller recently confirmed that Googlebot still crawls a disallowed robots.txt: Disallowing Robots.txt In Robots.txt Doesn't Impact How Google Processes It. So even if you specifically called out Disallow: /robots.txt, Google (and I suspect other search engines) wouldn't change their behavior.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-08-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。