问题原文
原帖链接 · JohnJ · CC BY-SA 3.0
I have a small website, for which the current robots.txt looks like so:
User-agent: *
Disallow:
Sitemap: https://www.myawesomesite.com/sitemap.xml
Now, I want to add another rule like so (with the aim to disallow SomeStupidBot for crawling my website):
User-agent: SomeStupidBot
Disallow: /
Sitemap: https://www.myawesomesite.com/sitemap.xml
I am unsure how I combine the above two rules into one robots.txt.
已采纳解答
原帖链接 · MrWhite · CC BY-SA 3.0
To combine them you simply append the second rule block (group) to the file, with a line gap, like so:
User-agent: *
Disallow:
User-agent: SomeStupidBot
Disallow: /
Sitemap: https://www.myawesomesite.com/sitemap.xml
The order does not matter (the most specific - based on length - wins). Only one block is processed by the bot.
Note that the Sitemap directive is not part of the group. It is an independent directive that often appears at the end of the file (although the location does not really matter).
As @MEGA has already stated, the first group is superfluous in this instance. However, it is up to you whether you keep it (there will usually be something you don't want other bots to crawl).
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
