问题原文
原帖链接 · cbdeveloper · CC BY-SA 4.0
There's a /search path on my web app that I don't want robots to crawl or index.
What would be the best approach for this scenario?
- 1: Block them from accessing by adding a
Disallowfor/searchon robots.txt - 2: Add a
rel="noindex"to the links that lead to the/searchURLs.
What is the best practice in this case?
已采纳解答
原帖链接 · MrWhite · CC BY-SA 4.0
Since you don't want bots to crawl these /search pages then you have no option other than to block them in robots.txt.
However, that doesn't necessarily prevent the /search pages from appearing as link only results in the SERPS (if other pages link to them) - but this is unlikely. And they are very unlikely to rank higher than other search results anyway.
2: Add a
rel="noindex"to the links that lead to the/searchroute.
noindex is not a recognised rel value for Google, maybe you mean rel="nofollow"?
You can set rel="nofollow" on the link to the search pages. However, this won't necessarily prevent the target pages from appearing in the SERPs (if there are other links to them) or even from being crawled, although it should. (This value appears to have been demoted to an advisory value by Google).
To prevent a page from being indexed then you need to include a <meta name="robots" content="noindex"> tag on those pages. Or serve with an X-Robots-Tag HTTP response header. But in this case, you can't block crawling in robots.txt, otherwise the bot won't see the meta tag (or response header).
Depending on your use case, it might be beneficial to allow Google to crawl your search pages (in order to find internal pages), but not to index them. However, you don't necessarily want to waste Google's crawl budget on your search pages either, so it can depend on the nature of your site.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
