大网数据广告活动
网站运营与故障问答

Are 'robots.txt' rules "starts with" or "contains" rules (clarification on confusing Google documentation)

· 52CE 我爱测

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says the /fish rule matches any path that starts with /fish (...) it doesn't match the path /desert/fish …

问题原文

原帖链接 · Déjà vu · CC BY-SA 4.0

Searching for robots.txt pattern matching rules (inc. the non official Google rules), I found this Google dev page where it says

the /fish rule matches any path that starts with /fish (...) it doesn't match the path /desert/fish

but they also say

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/

That means that as long as a path is defined within the rule (like /something/), a match occurs wherever that pattern is found within the URL path. But if the rule doesn't define a path, having a leading / but no trailing /, the path starts from root?

(regexes would be something like

  • /fish ⇨ ^/fish.*
  • /fish/ ⇨ ^.*/fish/.*

the .* and the second ^ are not necessary, it's just to emphasize the regex patterns)

That sounds quite counter-intuitive to me (I thought that could be a mistake in the Google doc). Can someone confirm this understanding of the syntax?

已采纳解答

原帖链接 · MrWhite · CC BY-SA 4.0

the /fish/ rule matches anything in the /fish/ folder (...) it matches the path /animals/fish/

This does look like a mistake in the Google docs IMO (as you suggest) for several reasons:

  • That example does not work in Google's own robots.txt Tester:

  • That would seem to be the only example (AFAICS) that does not follow the expected prefix-matching nature of robots.txt pattern matching. Patterns always match from the document root (from the left). So this is indeed counter-intuitive.

  • If you needed to match /fish/ anywhere in the URL-path then you can use an explicit wildcard, eg. /*/fish/ (or even */fish/). So having /fish/ also match anywhere is not required and again counter-intuitive.

  • If the pattern /fish/ matches anywhere then you are going to have to always have rules that negate matching /fish/ in other parts of the URL. This would create more work and is counter-intuitive. If for instance you had Disallow: /fish/ and I link to your homepage with /?/fish/ then I've effectively created a nofollow link that can be shared. (This should not be possible, at least not so easily/accidentally.)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2022-02-25)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。