问题原文
原帖链接 · the · CC BY-SA 3.0
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
Today, according to archive.org I have to add the following in my robots.txt to allow their bots:
User-agent: ia_archiver
Disallow:
But, I already had done what they indicated a couple of years ago, at least, I added the following:
User-agent: archive.org_bot
Disallow:
Then there's another source claiming that you have to add the two above Disallows, plus another one:
User-agent: ia_archiver-web.archive.org
Disallow:
Note that you need to put Disallow: / if you don't want the bot to archive your site.
Has there been a change with the IA bot? If so, when?
What is the recommended way? Should I just allow all three for now and hope that IA will not change their bot name again in the future?
已采纳解答
原帖链接 · unor · CC BY-SA 3.0
Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014).
Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver.
So this record should allow their bot to crawl your entire site:
User-agent: ia_archiver
Disallow:
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
