问题原文
原帖链接 · arthur · CC BY-SA 3.0
I have a robots.txt file:
User-agent:*
Disallow:/path/page
Disallow:/path/
Allow:/
The disallowed path is still getting crawled.
I have searched this problem and what they said, order of precedence in google does not matter. So technically, the disallowed should work but now I'm wondering if it is because Allow:/ is overriding it?
社区高分解答
原帖链接 · Stephen Ostermiller · CC BY-SA 3.0
Putting your page through Google's robots.txt tester reveals two problems:
- You have no
User-agentline so your rules don't apply to any crawler: - Once you put in the
User-agentline, theAllowline overrides the disallow:
The correct robots.txt file would be:
User-agent: *
Disallow: /path/page
Disallow: /path/
DO NOT use Allow:. Allowing crawling is the default. You only need to include the items you don't want crawled.
Include the User-Agent: line to specify that it applies to all crawlers. Otherwise, it will apply to none.
I don't think that having a space after the colon actually matters, but all the examples I see have it.
I should also add that robots.txt is only for crawlers that choose to obey it. Search engine spiders like Googlebot should obey robots.txt, however not all other spiders will.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
