问题原文
原帖链接 · Craig · CC BY-SA 3.0
A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed.
When a search engine returns for future crawls of a website, does it systematically go through all of the links in the Sitemap, regardless of whether there have been any changes since the last crawl? Conversely, does the relevant Bot visit the Sitemap, identifies the date under the 'Last Modified' and then only crawls the URL, if the date is after the last time the bot visited/Crawled the URL? The latter, making more sense to me.
I am curious about this as when I submitted the Sitemap, I noticed that in the first few weeks, there was no consistencies in each crawl. Some days, over 1,000 pages would be Crawled while other days, only 10-20 pages would be crawled.
The 'Crawl Rate' has now steadied out now but there are some pages still not having been indexed, despite having been created before other pages which are getting picked up quickly by Search Bots.
I have checked the .htaccess file as well as ensuring that the pages have not been noindexed.
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 3.0
Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do.
Search bots almost never crawl the entire site in one sitting. Rather they re-crawl pages each on their own schedule. They schedule page re-crawls based on how popular a page is and how often the search bot observes the page changing.
Google has said that they pretty much ignore the lastmod and changefreq fields in sitemaps because so many webmasters don't keep them up to date reliably. In fact, Google doesn't use your sitemap for very much at all. The main benefit of having a sitemap is seeing extra stats in Google Search Console. Sitemaps don't typically get Googlebot to crawl pages it wouldn't otherwise crawl and they don't help with rankings. See The Sitemap Paradox.
Most sites end up with some pages that don't get indexed. It isn't usually because Google hasn't gotten around to crawling. Rather, Google just chooses not to index some pages. See Why aren't search engines indexing my content? Pages that Google chooses not to index are typically one of:
- Duplicate
- Not much content
- Poor quality
- Not targeting topics for which users search
- Low reputation (PageRank)
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
