问题原文
原帖链接 · Diego Wais · CC BY-SA 4.0
I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern:
www.example.com/node/sport/category/id
but not those without the ID:
www.example.com/node/sport/category/
How can I approach this in a robots.txt?
UPDATE:
the ID are some numbers like /12343/ or /12345/ etc
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
You don't have a lot of great options. robots.txt rules are usually prefix matching.
One option would be to change all of your URLs that end in an id so that they have a common prefix: /node/sport/category/ would get crawled but /private/node/sport/category/id would not because you would add Disallow: /private/ to robots.txt.
The major search engine bots (Google and Bing) do support pattern matching in robots.txt files. Unfortunately, they don't support regular expression matching, so you are limited much less powerful globs. Even the globs won't be supported by most other crawlers, so this will only work for major search engine bots, not for all crawlers. I think the best you can do is disallow URLs ending in digits with 10 different rules:
Disallow: *0$
Disallow: *1$
Disallow: *2$
Disallow: *3$
Disallow: *4$
Disallow: *5$
Disallow: *6$
Disallow: *7$
Disallow: *8$
Disallow: *9$
Another option is to allow those URLs to be crawled, but prevent them from being indexed. Instead of changing your robots.txt file, you could add a meta tag to those documents so that search engines won't index them after they get crawled:
<meta name="robots" content="noindex">
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
