大网数据广告活动
网站运营与故障问答

Why does it cause issues to use robots.txt and a meta robots tag at the same time?

· 52CE 我爱测

It seems common wisdom that using X-Robots-Tag / meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.: Using both X-Robots-Tag and meta robots on a URL is redundant because they are equivalent, a…

问题原文

原帖链接 · jII · CC BY-SA 4.0

It seems common wisdom that using X-Robots-Tag/meta robots with robots.txt to block a URL from being indexed "can cause issues", e.g.:

Using both X-Robots-Tag and meta robots on a URL is redundant because they are equivalent, and using both robots.txt and either of the others for a URL can cause issues because robots.txt blocks crawling, and crawling is required for a bot to even see either of the other ones since they are document-level directives.

Source: https://webmasters.stackexchange.com/a/130710

However, I cannot see what the "issue" is. Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if foo.html is followed from another site. However, if we add a meta robots in foo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be in robots.txt.

Therefore, while it might be redundant to use both methods, I am not sure why doing so can cause foo.html to be indexed.

已采纳解答

原帖链接 · Maximillian Laumeister · CC BY-SA 4.0

The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it!

Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if foo.html is followed from another site.

Correct.

However, if we add a meta robots in foo.html, indexing will be blocked when the bot reaches the page from another site, independently of what happens to be in robots.txt.

There is a subtle misunderstanding here - the bot never actually reaches the page. What happens is the following:

  1. The bot sees the link to foo.html on an external site.

  2. The bot prepares to crawl foo.html, but first retrieves the robots.txt file from the domain on which foo.html is hosted, to make sure it is allowed to crawl foo.html.

  3. The bot notices that crawling foo.html is prohibited, so it does not request foo.html at all. It is unable to see any of the contents of foo.html, including the robots noindex meta tag.

  4. Unable to see whether the page has a noindex meta tag or not, the crawler makes its own decision for whether to index the page. It may decide to index the page, which it's allowed to do because it hasn't seen the noindex directive telling it not to index.

  5. You end up with the situation where both robots.txt and noindex were used, but the page still gets indexed.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2021-10-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。