问题原文
原帖链接 · jII · CC BY-SA 4.0
It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.:
Using both
X-Robots-Tagandmeta robotson a URL is redundant because they are equivalent, and using bothrobots.txtand either of the others for a URL can cause issues becauserobots.txtblocks crawling, and crawling is required for a bot to even see either of the other ones since they are document-level directives.
However, I cannot see what the "issue" is. Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if foo.html is followed from another site. However, if we add a meta robots in foo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be in robots.txt.
Therefore, while it might be redundant to use both methods, I am not sure why doing so can cause foo.html to be indexed.
已采纳解答
原帖链接 · Maximillian Laumeister · CC BY-SA 4.0
The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it!
Suppose I have a URL
/foo.htmlfor which crawling is blocked usingrobots.txt. Then the bot will not crawl the page, but it might still be indexed iffoo.htmlis followed from another site.
Correct.
However, if we add a
meta robotsinfoo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be inrobots.txt.
There is a subtle misunderstanding here - the bot never actually reaches the page. What happens is the following:
The bot sees the link to
foo.htmlon an external site.The bot prepares to crawl
foo.html, but first retrieves therobots.txtfile from the domain on whichfoo.htmlis hosted, to make sure it is allowed to crawlfoo.html.The bot notices that crawling
foo.htmlis prohibited, so it does not requestfoo.htmlat all. It is unable to see any of the contents offoo.html, including therobots noindexmeta tag.Unable to see whether the page has a
noindexmeta tag or not, the crawler makes its own decision for whether to index the page. It may decide to index the page, which it's allowed to do because it hasn't seen thenoindexdirective telling it not to index.You end up with the situation where both
robots.txtandnoindexwere used, but the page still gets indexed.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
