大网数据广告活动
网站运营与故障问答

How often do search bots typically visit a website?

· 52CE 我爱测

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed. When a search engine returns for future crawls of a website, does i…

问题原文

原帖链接 · Craig · CC BY-SA 3.0

A few weeks ago, I completed a website and submitted its Sitemap to Google. To date, I have had no real problems, with most of the URLs now successfully indexed.

When a search engine returns for future crawls of a website, does it systematically go through all of the links in the Sitemap, regardless of whether there have been any changes since the last crawl? Conversely, does the relevant Bot visit the Sitemap, identifies the date under the 'Last Modified' and then only crawls the URL, if the date is after the last time the bot visited/Crawled the URL? The latter, making more sense to me.

I am curious about this as when I submitted the Sitemap, I noticed that in the first few weeks, there was no consistencies in each crawl. Some days, over 1,000 pages would be Crawled while other days, only 10-20 pages would be crawled.

The 'Crawl Rate' has now steadied out now but there are some pages still not having been indexed, despite having been created before other pages which are getting picked up quickly by Search Bots.

I have checked the .htaccess file as well as ensuring that the pages have not been noindexed.

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 3.0

Search bots typically never stop visiting a website. Googlebot will typically download some pages every day. It may even download nearly as many pages per day as real users do.

Search bots almost never crawl the entire site in one sitting. Rather they re-crawl pages each on their own schedule. They schedule page re-crawls based on how popular a page is and how often the search bot observes the page changing.

Google has said that they pretty much ignore the lastmod and changefreq fields in sitemaps because so many webmasters don't keep them up to date reliably. In fact, Google doesn't use your sitemap for very much at all. The main benefit of having a sitemap is seeing extra stats in Google Search Console. Sitemaps don't typically get Googlebot to crawl pages it wouldn't otherwise crawl and they don't help with rankings. See The Sitemap Paradox.

Most sites end up with some pages that don't get indexed. It isn't usually because Google hasn't gotten around to crawling. Rather, Google just chooses not to index some pages. See Why aren't search engines indexing my content? Pages that Google chooses not to index are typically one of:

  • Duplicate
  • Not much content
  • Poor quality
  • Not targeting topics for which users search
  • Low reputation (PageRank)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-03-16)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。