大网数据广告活动
网站运营与故障问答

What is a minimum valid robots.txt file?

· 52CE 我爱测

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a simple robots.txt file that will…

问题原文

原帖链接 · bessarabov · CC BY-SA 3.0

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a simple robots.txt file that will prevent the 404 errors from appearing in my log file.

What is a minimum valid robots.txt file that will allow everything on the site to be crawled?

已采纳解答

原帖链接 · dan · CC BY-SA 3.0

As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add:

User-agent: *
Disallow:

If you want robots to crawl everything. If not, then see the above link for more examples.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2014-01-05)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。