大网数据广告活动
网站运营与故障问答

How to configure robots.txt to disallow any URL/file/directory with a certain word in it?

· 52CE 我爱测

I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt , but I have a more detailed need than that. Google search console keeps trying to discover/crawl certain URLs on my site that it sho…

问题原文

原帖链接 · ravl13 · CC BY-SA 4.0

I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.

Google search console keeps trying to discover/crawl certain URLs on my site that it shouldn't be doing. It's not a security hazard, but essentially what it's crawling are completely redundant URLs that don't exist in actuality. I have an idea of why it's happening, but I don't want to try and restructure my pages to accommodate Google in this regard.

These unnecessary page crawls always contain a certain phrase in the URL. Suppose this phrase is "unnecessaryPage", as in info-unnecessaryPage.htm is a URL that Google attempts to crawl, and I don't want it to.

How can I create a "disallow" line to do this? For example, if I wanted to disallow an entire directory, I'd put this:

Disallow: /DirectoryA

But I want to disallow everything that includes the phrase "unnecessaryPage" anywhere in the filename, directory, or URL. How would I make that Disallow line?

已采纳解答

原帖链接 · Henry Visotski · CC BY-SA 4.0

The directive would be something like this:

user-agent: *
disallow: /*unnecessaryPage

Thus, anything that comes before the phrase "unnecessaryPage" will be included via wildcard. No trailing wildcard: it is optional because it is ignored. Further examples can be found in Google's documentation on the matter.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2022-09-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。