问题原文
原帖链接 · ravl13 · CC BY-SA 4.0
I've seen basic articles online about allowing/disallowing certain subdirectories in robots.txt, but I have a more detailed need than that.
Google search console keeps trying to discover/crawl certain URLs on my site that it shouldn't be doing. It's not a security hazard, but essentially what it's crawling are completely redundant URLs that don't exist in actuality. I have an idea of why it's happening, but I don't want to try and restructure my pages to accommodate Google in this regard.
These unnecessary page crawls always contain a certain phrase in the URL. Suppose this phrase is "unnecessaryPage", as in info-unnecessaryPage.htm is a URL that Google attempts to crawl, and I don't want it to.
How can I create a "disallow" line to do this? For example, if I wanted to disallow an entire directory, I'd put this:
Disallow: /DirectoryA
But I want to disallow everything that includes the phrase "unnecessaryPage" anywhere in the filename, directory, or URL. How would I make that Disallow line?
已采纳解答
原帖链接 · Henry Visotski · CC BY-SA 4.0
The directive would be something like this:
user-agent: *
disallow: /*unnecessaryPage
Thus, anything that comes before the phrase "unnecessaryPage" will be included via wildcard. No trailing wildcard: it is optional because it is ignored. Further examples can be found in Google's documentation on the matter.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
