大网数据广告活动
网站运营与故障问答

Why is preventing Googlebot from crawling some pages on my site good for SEO?

· 52CE 我爱测

Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site. https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://developers.google.com/search/docs…

问题原文

原帖链接 · ekusiadadus · CC BY-SA 4.0

Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site.

https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://developers.google.com/search/docs/advanced/crawling/block-indexing

But I could not figure it out. Why is it effective to prevent Google Crawling Bot from crawling some useless pages to enhance SEO?

社区高分解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several types of pages that Google doesn't want to index:

  • URLs without content
    • Error pages
    • Redirects
    • Blank pages
  • Thin content
    • Stub pages
    • Tag or category pages that contain very few links
    • User profile pages with little info filled out by the user
  • Poor quality content
    • Auto generated text
    • Machine translated text
    • Bad spelling
    • Bad grammar
    • Broken links
    • Spam
  • Duplicate content
    • Internal duplication, URLs that duplicate another page on your site
    • Scraped content
    • Syndicated content
  • Infinite (or very large) URL spaces. Such as pages for:
    • Site search
    • Each number
    • Every person on earth
    • Every phone number of IP address
    • Large combinations (like how to get from here to there for every combination of locations).
  • Temporary content
    • Pages that will go away shortly
    • Pages only useful when they are first viewed
  • Private content
    • Pages requiring users to log in
    • Admin pages
    • Pages with sensitive data

For each of those, you actually have four options:

  • Let Googlebot crawl and index as it sees fit
  • Prevent Googlebot from crawling it using robots.txt
  • Prevent Googlebot from indexing it using a noindex meta tag or header
  • Tell Google about an alternative using redirects or canonical meta tags

Google will never index error pages or redirects so it is fine to let Googlebot crawl them decide not to index them.

For duplicate content it is usually best to redirect or tell Googlebot about the duplication with canonical tags.

I recommend preventing Googlebot from crawling very large (or infinite) URL spaces.

For other types of content you have to strike a balance between allowing lots of crawling, vs preventing indexing. Googlebot may occasionally index pages that it can't crawl, but to see the noindex Googlebot has to be able to crawl the URL.

If your site has only rich, original, informative URLs, then you can let Googlebot crawl the whole thing and index all of it.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2021-09-06)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。