大网数据广告活动
Web 标准与前端开发

robots.txt configuration

· 52CE 我爱测

`robots.txt` is a text file that tells robots (such as search engine indexers) how to behave, by instructing them not to crawl certain paths on the website. It is placed within the root directory of a website.

robots.txt is a text file that tells robots (such as search engine indexers) how to behave, by instructing them not to crawl certain paths on the website. It is placed within the root directory of a website.

Problem

Many factors can increase the load on your website; this includes web crawlers. In addition, if allowed to crawl the entirety of a site, web crawlers can cause pollution of search results with resources that don't benefit from being searchable.

Solution

Use robots.txt to reduce website load and stop unsuitable content appearing in search results. Using this file is optional and should only be used for these purposes. It should not be used as a way to prevent the disclosure of private information or to hide portions of a website.

While using this file can prevent pages from appearing in search engine results, it does not secure websites against attackers. On the contrary, it can unintentionally help them: robots.txt is publicly accessible, and by adding your sensitive page paths to it, you are showing their locations to potential attackers.

Also be aware that some robots, such as malware robots and email address harvesters, will ignore your robots.txt file.

Examples

Stop all search engines from crawling a site:

User-agent: *
Disallow: /

Hide certain directories (this is not recommended):

User-agent: *
Disallow: /secret/admin-interface

See also

来源:MDN Web Docs / Mozilla 贡献者 · 作者与贡献记录 · CC BY-SA 2.5。离线整理了版式、链接和媒体显示。

资料来源与适用范围

资料来源
MDN Web Docs / Mozilla 贡献者
适用范围
请按文中版本与环境说明核对
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 2.5

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。