大网数据广告活动
网站运营与故障问答

Do search engines still crawl a noindex page

· 52CE 我爱测

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex' and I was wondering if these…

问题原文

原帖链接 · GrapeSoda · CC BY-SA 3.0

Do search engines crawl a page that has a 'noindex' meta attribute?

The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex' and I was wondering if these pages would still be detected as duplicates?

Do I have to add a 'nofollow' attribute to the link whilst we make these pages unique?

已采纳解答

原帖链接 · MrWhite · CC BY-SA 3.0

As Goyllo has already stated, search engine bots will crawl pages that have a noindex meta tag. If you think about it, they need to crawl the page in order to see the noindex meta tag in the first place. (You could use an X-Robots-Tag HTTP response header instead and, in theory, a bot would only need to do a HEAD request in order to see the noindex attribute - but that's not how Google rolls.)

If a page is noindex, it can still be follow (which it would be by default, unless you explicitly state nofollow as well), so the page would obviously need to be crawled in order to discover any links to follow.

Do I have to add a 'nofollow' attribute to the link whilst we make these pages unique?

That simply discounts that particular link from the ranking algorithm. So, that particular link will not be used as a ranking factor for the target URL. I assume it's highly likely that there are other inbound links to that page as well?

...pages have stated 'noindex' and I was wondering if these pages would still be detected as duplicates?

Duplicate of what? A page can only be considered a duplicate (in the eyes of the search engine index) if it is indexed. If it's not indexed then it can't be a duplicate.

The duplicate content "problem" is if you have two (or more) duplicate pages that have been crawled and indexed then the search engine must decide which page to return in the SERPs. Unless you resolve this duplicate content yourself (redirect, canonical tag or simply making the content unique) then it's out of your control - the search engine makes the decision for you. You are also potentially diluting your search ranking as users discover different pages and link back to one or the other.

To prevent a page from being crawled (ie. not even requested) then you can include an entry in your robots.txt file. However, this will mean the search engines will be unable to see your noindex meta tag. Whilst this should prevent the page appearing in normal search results, it doesn't necessarily prevent the page from appearing as a link-only result in the SERPs (ie. "indexed") if it is linked to. However, it still can't be considered "duplicate" because it's content won't have been read and indexed.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-11-03)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。