问题原文
原帖链接 · Mustafa · CC BY-SA 4.0
Google Search is indexing my robots.txt file. I know this because when I search site:example.com on Google, my robots.txt shows in the list of results. I don't want my robots.txt to show in Google.
The content of my robots.txt is
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
已采纳解答
原帖链接 · Maximillian Laumeister · CC BY-SA 4.0
Why is Google indexing my
robots.txtfile?
It is normal behavior for Google to index your robots.txt file. It's a URL on your website just like any other.
I don't want my robots.txt to show in Google.
Your robots.txt file already does not show on Google, in the general sense that Google Search users could stumble upon it without specifically looking for it. Google is smart enough to know that it's generally irrelevant to searchers. Nobody will see it in search results unless they are trying to comb through your site's index using the site: operator.
To answer the question as written: To prevent your robots.txt file from being indexed, you can configure your web server to serve it up with a X-Robots-Tag: noindex header. However there is no practical benefit to doing so, and depending on your server software it may involve some extra setup/configuration.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
