问题原文
原帖链接 · Quillion · CC BY-SA 4.0
So I am trying to learn SEO and I am honestly confused and have following 8 questions.
Do I tell a bot not to visit a certain link through
X-Robots-Tagor throughrobot meta tagorrobots.txt?Is it ok to include all 3 (robots.txt, robot meta tag, and X-Robots-Tag header) or I should always only provide 1?
Do I get penalized if I show same info in
X-Robots-Tagand inrobotsmeta tag androbots.txt?Let's say for
/test1myrobots.txtsaysDisallowbut myrobots meta tagsaysindex, followand myX-Robots-Tagsaysindex, nofollow, noarchive. Do I get penalized if those values are different?Let's say for
/test1myrobots.txtsaysDisallowbut myrobotsmeta tag saysindex, followand myX-Robots-Tagsaysindex,nofollow,noarchive. Which rule will be followed by the bot? What is the importance here?Let's say my
robots.txthas a rule sayingDisallow: /andAllow: /link_one/link_twoand myX-Robots-Tagandrobot meta tagfor every link except/link_one/link_twosaysnofollow,noindex,noarchive. From what I understand bot will never get to/link_one/link_twosince I prevented it from crawling at root level. Now if I provide asitemap.xmlin therobots.txtthat has/link_one/link_twothere, will it actually end up being crawled?Will bot crawl into the directory provided by
sitemap.(xml/txt)even though it is not accessible through home page or any pages following the home page?And overall I would appreciate some clarification on what is the difference between
robots.txt,X-Robots-Tagandrobot meta tagandsitemap.(xml/txt). To me they seem like they do the exact same thing.
I already saw that there are some questions that answer a small subset of what I asked. But I want the whole big explanation.
已采纳解答
原帖链接 · Maximillian Laumeister · CC BY-SA 4.0
While X-Robots-Tag and meta robots are equivalent, robots.txt is different. The former is about indexing, while the latter is about crawling/visiting.
Tell bots not to visit a URL by using
robots.txt.Use only one of the three for each URL. Using both
X-Robots-Tagandmeta robotson a URL is redundant because they are equivalent, and using bothrobots.txtand either of the others for a URL can cause issues becauserobots.txtblocks crawling, and crawling is required for a bot to even see either of the other ones since they are document-level directives.You don't get penalized, but it doesn't do anything more than just having a page in
robots.txt. As I said in 2,robots.txtwill block a page from crawling, but a bot would need to crawl the page to see either of the other 2, so it can't see them.robots.txtprevents the bot from crawling and finding the other two.robots.txtprevents the bot from crawling and finding the other two.At least for Google, "the most specific rule based on the length of the [path] entry trumps the less specific (shorter) rule." This means that your
robots.txtfile will allow crawling of/link_one/link_two, even though you disallow/.See The Sitemap Paradox. Short answer is that if your site can't properly be crawled without the sitemap, it may run into SEO issues anyways. In other words, that could cause a problem.
X-Robots-Tagandmeta robotsare exactly equivalent, just one is a way to do it at the HTTP level, and the other is a way to do the same thing at the HTML level. They prevent indexing, they don't prevent crawling. Use them for pages you don't want in search results. In contrast,robots.txtprevents crawling, but not indexing. Userobots.txtfor pages you don't want bots to waste time crawling, but that wouldn't be catastrophic if they showed up in search results (Google is known to index URLs without crawling them if they are considered important enough). If you use bothrobots.txtand one of the others,robots.txtwill prevent the bot from visiting the page and even seeing the others, rendering them useless.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
