问题原文
原帖链接 · ekusiadadus · CC BY-SA 4.0
Google says that it is effective to prevent Google Crawling Bot from crawling some useless pages in my site.
https://developers.google.com/search/docs/advanced/guidelines/how-search-works https://developers.google.com/search/docs/advanced/crawling/block-indexing
But I could not figure it out. Why is it effective to prevent Google Crawling Bot from crawling some useless pages to enhance SEO?
社区高分解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
Some pages can hurt the SEO of your entire site if you let Google crawl and index them. From my answer to How should I choose which URLs I want indexed to include in my sitemap? there are several types of pages that Google doesn't want to index:
- URLs without content
- Error pages
- Redirects
- Blank pages
- Thin content
- Stub pages
- Tag or category pages that contain very few links
- User profile pages with little info filled out by the user
- Poor quality content
- Auto generated text
- Machine translated text
- Bad spelling
- Bad grammar
- Broken links
- Spam
- Duplicate content
- Internal duplication, URLs that duplicate another page on your site
- Scraped content
- Syndicated content
- Infinite (or very large) URL spaces. Such as pages for:
- Site search
- Each number
- Every person on earth
- Every phone number of IP address
- Large combinations (like how to get from here to there for every combination of locations).
- Temporary content
- Pages that will go away shortly
- Pages only useful when they are first viewed
- Private content
- Pages requiring users to log in
- Admin pages
- Pages with sensitive data
For each of those, you actually have four options:
- Let Googlebot crawl and index as it sees fit
- Prevent Googlebot from crawling it using
robots.txt - Prevent Googlebot from indexing it using a
noindexmeta tag or header - Tell Google about an alternative using redirects or
canonicalmeta tags
Google will never index error pages or redirects so it is fine to let Googlebot crawl them decide not to index them.
For duplicate content it is usually best to redirect or tell Googlebot about the duplication with canonical tags.
I recommend preventing Googlebot from crawling very large (or infinite) URL spaces.
For other types of content you have to strike a balance between allowing lots of crawling, vs preventing indexing. Googlebot may occasionally index pages that it can't crawl, but to see the noindex Googlebot has to be able to crawl the URL.
If your site has only rich, original, informative URLs, then you can let Googlebot crawl the whole thing and index all of it.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
