问题原文
原帖链接 · ashutosh · CC BY-SA 3.0
I have written the following code inside my robots.txt file:
User-Agent: Googlebot
Disallow:
User-agent: Mediapartners-Google
Disallow:
Sitemap: http://example.com/sitemap.xml
Is my robots.txt is correct? I only want two user agent to follow my site (i.e., Googlebot & Mediapartners-Google).
已采纳解答
原帖链接 · dan · CC BY-SA 3.0
Almost, see the bottom of the section named, "Blocking user-agents" in the following: Google Webmaster Tools: Block or remove pages using a robots.txt file
According that, you should have:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Mediapartners-Google
Allow: /
sitemap: http://example.com/sitemap.xml
See the bottom of the page in the above link for how to test your robots.txt file.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
