大网数据广告活动
网站运营与故障问答

Using robots.txt to block sessionID URLs

· 52CE 我爱测

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment). An example URL is: catalo…

问题原文

原帖链接 · czmudzin · CC BY-SA 3.0

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work environment).

An example URL is:

catalog/product-category/product/;jsessionid=Mf87s+Xw2P8ByQYz2CyQjEJh.prod-14?f=p%3A100-200

Can I update my robots.txt to say:

Disallow: /;jsessionid=*

Does anyone see an issue with doing that? I can also canonicalize the pages, but I feel that using robots is a better solution so Googlebot won't have to waste any resources crawling the URLs in the first place.

已采纳解答

原帖链接 · Evgeniy · CC BY-SA 3.0

you should exclude such urls from crawling, because you spend your whole crawling budget for those useless urls. Read this topic at Google Product Forum.

set the robots rule like:

disallow: /*jsessionid*

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2015-10-27)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。