问题原文
原帖链接 · czmudzin · CC BY-SA 3.0
I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).
An example URL is:
catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200
Can I update my robots.txt to say:
Disallow: /;jsessionid=*
Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.
已采纳解答
原帖链接 · Evgeniy · CC BY-SA 3.0
you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.
set the robots rule like:
disallow: /*jsessionid*
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
