大网数据广告活动
网站运营与故障问答

How to properly (dis)allow the archive.org bot? Did things change, if so when?

· 52CE 我爱测

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / Today, according to archive.org I h…

问题原文

原帖链接 · the · CC BY-SA 3.0

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:

User-agent: *
Disallow: /

Today, according to archive.org I have to add the following in my robots.txt to allow their bots:

User-agent: ia_archiver
Disallow:

But, I already had done what they indicated a couple of years ago, at least, I added the following:

User-agent: archive.org_bot
Disallow:

Then there's another source claiming that you have to add the two above Disallows, plus another one:

User-agent: ia_archiver-web.archive.org 
Disallow:

Note that you need to put Disallow: / if you don't want the bot to archive your site.

Has there been a change with the IA bot? If so, when?

What is the recommended way? Should I just allow all three for now and hope that IA will not change their bot name again in the future?

已采纳解答

原帖链接 · unor · CC BY-SA 3.0

Update: As @KevinFegan notes in the comments, their documentation changed. The below part describes how the Internet Archive handled it in the past (at least in 2014).


Their FAQ How can I have my site's pages excluded from the Wayback Machine? refers to Removing Documents From the Wayback Machine, which documents that their bot is called ia_archiver.

So this record should allow their bot to crawl your entire site:

User-agent: ia_archiver
Disallow:

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2014-10-25)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。