大网数据广告活动
网站运营与故障问答

How does "Noindex:" in robots.txt work?

· 52CE 我爱测

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt . Disallow: /page-one.html Noindex: /page-two.html Seems li…

问题原文

原帖链接 · Stephen Ostermiller · CC BY-SA 3.0

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.

Disallow: /page-one.html
Noindex: /page-two.html

Seems like it would prevent search engines from crawling page one, and prevent them from indexing page two.

Is this robots.txt directive supported by Google and other search engines? Does it work? Is it documented?

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Google used to unofficially support a Noindex directive in robots.txt, however in 2019, they announced that the directive will no longer work.

Here is what Google's John Mueller says about Noindex: in robots.txt:

We used to support the no-index directive in robots.txt as an experimental feature. But it's something that I wouldn't rely on. And I don't think other search engines are using that at all.

Before Google announced the feature was discontinued, deepcrawl.com did some testing of the feature and discovered that:

  • Before 2019, it still worked with Google
  • It prevented URLs from appearing in the search index
  • URLs that have been noindexed in robots.txt were marked as such in Google Search Console

Given that Google discontinued the feature, it shouldn't be used anymore.

Instead, use robots meta tags that are well supported and documented to prevent indexing:

<meta name="robots" content="noindex" />

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2015-09-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0 / CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。