大网数据广告活动
网站运营与故障问答

How to limit Google crawling certain URL patterns?

· 52CE 我爱测

recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These attribute filters are implemented as…

问题原文

原帖链接 · Frodik · CC BY-SA 4.0

recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These attribute filters are implemented as URL parts added to product category URL like so:

Simple product category URL: www.domain.com/product-category

Product category URL with one attribute filter: www.domain.com/product-category/filter1

Product category URL with two attribute filters: www.domain.com/product-category/filter1/filter2

And so on.....

So having around 30 attribute filters there can be A LOT of their combinations and therefore exponentialy increasing number of all available URLs to Google, for example: www.domain.com/product-category/filter2/filter4 or www.domain.com/product-category/filter5/filter9/filter12 and so on ....

Google's traffic hit us very hard, because right now is continuous traffic rate around 30Mbits only to Google alone, indexing right now around 100 million pages from our domain (before this filter feature we had around 5000 indexed pages).

What I need to do is to find a way how to tell Google not to crawl all the 100 million pages over and over again, but only, let's say, once a week or so.... Because right now, Google is crawling all the pages over and over again.

I have tried to set the HTTP headers cache-control: public, max-age:604800 and expires: header one week in future. But I have no idea if Google will use them at all. I don't want to block Google from entirely indexing these pages with attribute filter in URL, but I just want Google not to crawl them too often.

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

It is not a good idea to let Googlebot crawl URLs that can be constructed with combinations of parameters. When you allow your categories to be filtered, my advice is to only allow Googlebot to crawl pages with a single filter applied, even if you allow users to view multiple filters at the same time.

To implement this, you need to change your URLs:

  • /product-category/filter1 is fine
  • /product-category/filter1/filter2 needs to have a non-search-engine friendly URL like /view-filters?category=product-category&filters=filter1/filter2

to change your URLs cleanly:

  • Change all the links to filter pages to use the new URL structure
  • Redirect your current URLs with multiple filters applied to the new URLs
  • Add Disallow: /view-filters to robots.txt

The reason to change your URLs is because robots.txt rules are "starts with" rules. The best way to prevent bots from crawling some URLs but not others is to give the uncrawlable URLs a common prefix.

Unlike most bots, Googlebot allows wildcard rules. If you don't want to change your URLs, an option that could work for Googlebot would be to disallow any page with three slashes: Disallow: /*/*/*. That would disallow crawling of /product-category/filter1/filter2 but not /product-category/filter1. Of course it would also disallow crawling of any deep directory structure on your site that isn't related to product category filters as well.

There is no way to tell Googlebot how often to come back and crawl a URL. Your cache control headers are not effective for bots. Bots don't use them. Rather Googlebot determines how often it should crawl a URL based on how many other pages link to it. A popular page like the CNN home page will get crawled every few minutes. A page that is 6 hops away from your home page and only linked once may only get crawled every few months. The only way to control Googlebot's crawl frequency is to very carefully control the Pagerank passed to each URL, which isn't very feasible.

Your only real way of controlling Googlbot is the binary option given to you by robots.txt: "please crawl this", or "never crawl this".

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-04-02)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。