问题原文
原帖链接 · Sathiya Kumar V M · CC BY-SA 3.0
In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section.
User-agent: Mediapartners-Google
Disallow: /search
Allow: /
I know that Disallow will prevent Googlebot from indexing a webpage, but I don't understand the usage of Disallow: /search.
What is the exact meaning of Disallow: /search?
已采纳解答
原帖链接 · unor · CC BY-SA 3.0
In the Disallow field you specify the beginning of URL paths of URLs that should be blocked.
So if you have Disallow: /, it blocks everything, as every URL path starts with /.
If you have Disallow: /a, it blocks all URLs whose paths begin with /a. That could be /a.html, /a/b/c/hello, or /about.
In the same sense, if you have Disallow: /search, it blocks all URLs whose paths begin with the string /search. So it would block the following URLs, for example (if the robots.txt is in http://example.com/):
http://example.com/searchhttp://example.com/search.htmlhttp://example.com/searchenginehttp://example.com/search/http://example.com/search/index.html
While the following URLs would still be allowed:
http://example.com/foo/searchhttp://example.com/sea
Note that robots.txt doesn’t know/bother if the string matches a directory, a file or nothing at all. It only looks at the characters in the URL.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
