问题原文
原帖链接 · bessarabov · CC BY-SA 3.0
I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a simple robots.txt file that will prevent the 404 errors from appearing in my log file.
What is a minimum valid robots.txt file that will allow everything on the site to be crawled?
已采纳解答
原帖链接 · dan · CC BY-SA 3.0
As indicated here, create a text file named robots.txt in the top-level directory of your web server. You can leave it empty, or add:
User-agent: *
Disallow:
If you want robots to crawl everything. If not, then see the above link for more examples.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
