大网数据广告活动
网站运营与故障问答

How do I "noindex" a text (.txt) file?

· 52CE 我爱测

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as part of the application functio…

问题原文

原帖链接 · Bangkokian · CC BY-SA 4.0

I have a directory full of .txt files that are used as part of a larger javascript application. The text files are not inteded to be directly accessed, and are used as includes on separate pages as part of the application functionality).

So.. I added the directory to my robots.txt file because I don't want the files to show up independently in search results.

However, robots.txt simply makes the .txt files inaccessible to robots. It doesn't actually make the .txt files un-indexable.

What I really want to do is "noindex" the .txt files. But .txt files aren't like .html files.. a noindex meta tag will be visible in the text.

So how does one "noindex" a file that's raw text without corrupting the text with a tag?

Is there some other way to noindex a text file, externally? Or noindex the overall directory?

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

The solution is the same as for X-Robots noindex a folder of PDFs and Prevent XML sitemaps from showing up in Google search results. You use the X-Robots-Tag HTTP header rather than a meta tag. The HTTP header served with the txt file should look like:

X-Robots-Tag: noindex

After implementing the HTTP header, remove the disallow from your robots.txt. In order for crawlers to be able to see and honor the header, you need to make the txt files crawlable.


Under Apache web servers (most shared hosting), put the following code in the .htaccess file in the directory containing the .txt files:

<FilesMatch ".txt$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

If you are using nginx, your configuration would need to contain (source):

location ~* ^/txtfiledir/.*\.txt$ {
  add_header  X-Robots-Tag "noindex";
}

If you are using IIS as your web server, the configuration would need to contain (source):

<system.webServer>
  <rewrite>
    <outboundRules>
      <rule name="Set custom HTTP response header">
        <match serverVariable="RESPONSE_X_Robots_Tag" pattern=".*" />
        <conditions>
          <add input="{REQUEST_URI}" pattern="^/txtfiledir/.*\.txt$" />
        </conditions>
        <action type="Rewrite" value="noindex"/>
      </rule>
    </outboundRules>
  </rewrite>
</system.webServer>

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-10-26)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。