大网数据广告活动
网站运营与故障问答

Disallow URLs with an ID from being crawled

· 52CE 我爱测

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example.com/node/sport/category/ How can…

问题原文

原帖链接 · Diego Wais · CC BY-SA 4.0

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern:

www.example.com/node/sport/category/id

but not those without the ID:

www.example.com/node/sport/category/

How can I approach this in a robots.txt?

UPDATE:
the ID are some numbers like /12343/ or /12345/ etc

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

You don't have a lot of great options. robots.txt rules are usually prefix matching.

One option would be to change all of your URLs that end in an id so that they have a common prefix: /node/sport/category/ would get crawled but /private/node/sport/category/id would not because you would add Disallow: /private/ to robots.txt.

The major search engine bots (Google and Bing) do support pattern matching in robots.txt files. Unfortunately, they don't support regular expression matching, so you are limited much less powerful globs. Even the globs won't be supported by most other crawlers, so this will only work for major search engine bots, not for all crawlers. I think the best you can do is disallow URLs ending in digits with 10 different rules:

Disallow: *0$
Disallow: *1$
Disallow: *2$
Disallow: *3$
Disallow: *4$
Disallow: *5$
Disallow: *6$
Disallow: *7$
Disallow: *8$
Disallow: *9$

Another option is to allow those URLs to be crawled, but prevent them from being indexed. Instead of changing your robots.txt file, you could add a meta tag to those documents so that search engines won't index them after they get crawled:

<meta name="robots" content="noindex">

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2019-05-26)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。