问题原文
原帖链接 · Yen Deng · CC BY-SA 3.0
Can spiders crawl the sites if there are no asterisks used on the robots.txt?
For example:
User-agent: *
Disallow: .*/ping/.*
then
User-agent: *
Disallow: /ping/
Please can someone confirm this, many thanks
已采纳解答
原帖链接 · Goyllo · CC BY-SA 3.0
As @w3dk said you can't use dot asterisk (.*) in robots.txt with directory, it will probably gives you an error or you will not going to accomplish that you want.
.dot prefix used to block specific kind of file, like this.
User-agent: *
Disallow: *.pdf
This will block all pdf files to crawl in your website. No matter it is in root level or in sub folder.
Your second robots.txt clarification.
User-agent: *
Disallow: /ping/
Here all polite bots including Googlebot, MSNBot, Yandexbot allowed to crawl your all directory except /ping/ directory, but your /ping/ directory should be placed in root level, otherwise it won't work.
First example - www.example.com/ping/
Second example (This won't work) - www.example.com/sub/ping/
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
