大网数据广告活动
网站运营与故障问答

How does robots.txt work with sites in subfolders?

· 52CE 我爱测

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2.com is one of the parked sites, b…

问题原文

原帖链接 · Coomie · CC BY-SA 3.0

I have a single web host with a number of other parked domains/sites in sub-directories, like this:

example.com is the primary site and root directory of the web hosting.

example.com/www.example2.com is one of the parked sites, but it is just a subfolder of the primary site.

Both www.example2.com and www.example.com/www.example2.com are accessible as the same content, but I want to block access the later while allowing access to the former.

Will a robots.txt file in the primary site disallowing www.* allow www.example2.com to be crawled?

已采纳解答

原帖链接 · Hello Fishy · CC BY-SA 3.0

I guess what you are looking for is a robots.txt entry like this:

User-agent: *
Disallow: /www.example2.com

Let's suggestion you have more than 100 'parked' exampleNR.com URLs, but don't want to write a line for every single one of them...use this:

User-agent: *
Disallow: /www.example

The problem is, it actually is not officially supported, but many robots like Googlebot are able to understand those easy wildcards. RegEx are definitely not supported. for additonal information

UPDATE

Deleted the trailing asterisk since robots.txt uses simple prefix matching anyway. Thanks for your attention, w3dk

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2017-03-23)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。