大网数据广告活动
网站运营与故障问答

Prevent search bots from indexing server (sub)domain name

· 52CE 我爱测

A web application I wrote is hosted on an in-house server with the name myserver , which is under my university's domain ( department.uni.edu ), resulting in the server's address being myserver.department.uni.edu . When I Google …

问题原文

原帖链接 · Chris Cirefice · CC BY-SA 3.0

A web application I wrote is hosted on an in-house server with the name myserver, which is under my university's domain (department.uni.edu), resulting in the server's address being myserver.department.uni.edu. When I Google myserver, the first result is that exact server hosting the web application.

I have a robots.txt file for the application (root directory) with the following contents:

User-agent: *
Disallow: /

It's the actual server domain name that was indexed, and not anything in the web application.

I know that I can remove search results with Google Webmaster Tools, but how do I prevent Google from indexing a server's domain name (or address)? I believe the server is running Nginx on Ubuntu 14.10 (I am not the person in charge of the server, just coding the web application).

The desire here is to prevent the server from being indexed by web searching tools such as Google, Bing, Yahoo, etc. - basically block all known search engine crawlers.

Perhaps a solution is to block all crawlers to the subdomain's root (mysever.department.university.edu) using an Nginx rewrite rule such as:

map $http_user_agent $limit_bots {
     default 0;
     ~*(google|bing|yandex|msnbot) 1;
     ~*(AltaVista|Googlebot|Slurp|BlackWidow|Bot|ChinaClaw|Custo|DISCo|Download|Demon|eCatch|EirGrabber|EmailSiphon|EmailWolf|SuperHTTP|Surfbot|WebWhacker) 1;
     ~*(Express|WebPictures|ExtractorPro|EyeNetIE|FlashGet|GetRight|GetWeb!|Go!Zilla|Go-Ahead-Got-It|GrabNet|Grafula|HMView|Go!Zilla|Go-Ahead-Got-It) 1;
     ~*(rafula|HMView|HTTrack|Stripper|Sucker|Indy|InterGET|Ninja|JetCar|Spider|larbin|LeechFTP|Downloader|tool|Navroad|NearSite|NetAnts|tAkeOut|WWWOFFLE) 1;
     ~*(GrabNet|NetSpider|Vampire|NetZIP|Octopus|Offline|PageGrabber|Foto|pavuk|pcBrowser|RealDownload|ReGet|SiteSnagger|SmartDownload|SuperBot|WebSpider) 1;
     ~*(Teleport|VoidEYE|Collector|WebAuto|WebCopier|WebFetch|WebGo|WebLeacher|WebReaper|WebSauger|eXtractor|Quester|WebStripper|WebZIP|Wget|Widow|Zeus) 1;
     ~*(Twengabot|htmlparser|libwww|Python|perl|urllib|scan|Curl|email|PycURL|Pyth|PyQ|WebCollector|WebCopy|webcraw) 1;
 }

location / {
    if ($limit_bots = 1) {
        return 403;
    }
}

(borrowed from GD Hussle)

but, would this be sufficient or would something more sophisticated be necessary?

已采纳解答

原帖链接 · unor · CC BY-SA 3.0

With robots.txt you can control crawling, not indexing. If a search engine is not allowed to crawl a document on your host, it might still index its URL, e.g. if it found the link on an external site.

You can control indexing with the meta-robots element or the X-Robots-Tag HTTP header (see examples).

You have to decide if you want to allow search engines to crawl but not to index, or to index but not to crawl. Because if you disallow crawling in robots.txt, search engines won’t be able to reach your documents, so they’ll never learn that you don’t want these to get indexed.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-01-25)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。