问题原文
原帖链接 · Frodik · CC BY-SA 4.0
recently we have updated our e-commerce website, where each product category has product filter. There is around 30 attributes which can be used for filtering products shown in category. These attribute filters are implemented as URL parts added to product category URL like so:
Simple product category URL: www.domain.com/product-category
Product category URL with one attribute filter: www.domain.com/product-category/filter1
Product category URL with two attribute filters: www.domain.com/product-category/filter1/filter2
And so on.....
So having around 30 attribute filters there can be A LOT of their combinations and therefore exponentialy increasing number of all available URLs to Google, for example: www.domain.com/product-category/filter2/filter4 or www.domain.com/product-category/filter5/filter9/filter12 and so on ....
Google's traffic hit us very hard, because right now is continuous traffic rate around 30Mbits only to Google alone, indexing right now around 100 million pages from our domain (before this filter feature we had around 5000 indexed pages).
What I need to do is to find a way how to tell Google not to crawl all the 100 million pages over and over again, but only, let's say, once a week or so.... Because right now, Google is crawling all the pages over and over again.
I have tried to set the HTTP headers cache-control: public, max-age:604800 and expires: header one week in future. But I have no idea if Google will use them at all. I don't want to block Google from entirely indexing these pages with attribute filter in URL, but I just want Google not to crawl them too often.
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
It is not a good idea to let Googlebot crawl URLs that can be constructed with combinations of parameters. When you allow your categories to be filtered, my advice is to only allow Googlebot to crawl pages with a single filter applied, even if you allow users to view multiple filters at the same time.
To implement this, you need to change your URLs:
/product-category/filter1is fine/product-category/filter1/filter2needs to have a non-search-engine friendly URL like/view-filters?category=product-category&filters=filter1/filter2
to change your URLs cleanly:
- Change all the links to filter pages to use the new URL structure
- Redirect your current URLs with multiple filters applied to the new URLs
- Add
Disallow: /view-filtersto robots.txt
The reason to change your URLs is because robots.txt rules are "starts with" rules. The best way to prevent bots from crawling some URLs but not others is to give the uncrawlable URLs a common prefix.
Unlike most bots, Googlebot allows wildcard rules. If you don't want to change your URLs, an option that could work for Googlebot would be to disallow any page with three slashes: Disallow: /*/*/*. That would disallow crawling of /product-category/filter1/filter2 but not /product-category/filter1. Of course it would also disallow crawling of any deep directory structure on your site that isn't related to product category filters as well.
There is no way to tell Googlebot how often to come back and crawl a URL. Your cache control headers are not effective for bots. Bots don't use them. Rather Googlebot determines how often it should crawl a URL based on how many other pages link to it. A popular page like the CNN home page will get crawled every few minutes. A page that is 6 hops away from your home page and only linked once may only get crawled every few months. The only way to control Googlebot's crawl frequency is to very carefully control the Pagerank passed to each URL, which isn't very feasible.
Your only real way of controlling Googlbot is the binary option given to you by robots.txt: "please crawl this", or "never crawl this".
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
