大网数据广告活动
网站运营与故障问答

Link rel="nofollow" vs robots.txt disallow for "/search" URLs?

· 52CE 我爱测

There's a /search path on my web app that I don't want robots to crawl or index. What would be the best approach for this scenario? 1: Block them from accessing by adding a Disallow for /search on robots.txt 2: Add a rel="noindex…

问题原文

原帖链接 · cbdeveloper · CC BY-SA 4.0

There's a /search path on my web app that I don't want robots to crawl or index.

What would be the best approach for this scenario?

  • 1: Block them from accessing by adding a Disallow for /search on robots.txt
  • 2: Add a rel="noindex" to the links that lead to the /search URLs.

What is the best practice in this case?

已采纳解答

原帖链接 · MrWhite · CC BY-SA 4.0

Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt.

However, that doesn't necessarily prevent the /search pages from appearing as link only results in the SERPS (if other pages link to them) - but this is unlikely. And they are very unlikely to rank higher than other search results anyway.

2: Add a rel="noindex" to the links that lead to the /search route.

noindex is not a recognised rel value for Google, maybe you mean rel="nofollow"?

You can set rel="nofollow" on the link to the search pages. However, this won't necessarily prevent the target pages from appearing in the SERPs (if there are other links to them) or even from being crawled, although it should. (This value appears to have been demoted to an advisory value by Google).

To prevent a page from being indexed then you need to include a <meta name="robots" content="noindex"> tag on those pages. Or serve with an X-Robots-Tag HTTP response header. But in this case, you can't block crawling in robots.txt, otherwise the bot won't see the meta tag (or response header).

Depending on your use case, it might be beneficial to allow Google to crawl your search pages (in order to find internal pages), but not to index them. However, you don't necessarily want to waste Google's crawl budget on your search pages either, so it can depend on the nature of your site.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-11-17)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。