问题原文
原帖链接 · Carlos Torrecillas · CC BY-SA 4.0
I have a site which offers two languages, English and Spanish. When the user navigates to the home page, let's say www.example.com the page redirects you to either /es if your browser language is Spanish or English otherwise.
At the moment the robots.txt I have is:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap_index.xml
because I'm defining all hreflang alternate URLs in the sitemap_languages.xml and all URLs are listed also in the sitemap.xml. My question is more towards the configuration of the robots.txt because I'm not sure if I should be allowing any user agent to crawl the / page. As that page always redirects to the home of either /en or /es I believe that should be disallowed.
Should I then do:
User-agent: *
Disallow: /
Allow: /es
Allow: /en
Sitemap: https://www.example.com/sitemap_index.xml
I'm not sure if that could cause a crawl issue or whether there is another way to achieve the same result.
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
The best way to handle this situation is to allow crawling of the root page, but to show a page with links to all languages when the Accept-Language header is not present or does not match one of your supported languages. Web crawlers typically crawl without an Accept-Language header and they will almost always get this page.
That will allow search engines to pass link juice to your site appropriately. If you disallow crawling on your root page, you are throwing away the link juice from all links to the root. Typically sites get more links to the root than any where else.
Disallowing crawling won't prevent the root URL from getting indexed. Google will include disallowed pages in the index if they have links to them. When that happens Google will show unfriendly messages such as "A snippet cannot be shown because the page was blocked by robots.txt." It is far better to let the search engines crawl the page and see what is actually there.
The page can be quite simple:
<title>example.com</title>
<a href=/es>Español</a> | <a href=/en>English</a>
Most users won't see it, they will be automatically redirected away from it if they land on that URL. Search engine crawlers will get the links they need to crawl both languages and assign link juice to them.
In your sitemap the root page should be listed as x-default:
<url>
<loc>http://www.example.com/</loc>
<xhtml:link rel="alternate" hreflang="x-default" href="http://www.example.com/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://www.example.com/en"/>
<xhtml:link rel="alternate" hreflang="es" href="http://www.example.com/es"/>
</url>
<url>
<loc>http://www.example.com/en</loc>
<xhtml:link rel="alternate" hreflang="x-default" href="http://www.example.com/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://www.example.com/en"/>
<xhtml:link rel="alternate" hreflang="es" href="http://www.example.com/es"/>
</url>
<url>
<loc>http://www.example.com/es</loc>
<xhtml:link rel="alternate" hreflang="x-default" href="http://www.example.com/"/>
<xhtml:link rel="alternate" hreflang="en" href="http://www.example.com/en"/>
<xhtml:link rel="alternate" hreflang="es" href="http://www.example.com/es"/>
</url>
It is good that you allow people to view any language based on the URL regardless of their browser setting. That is especially important to web crawlers. If you were to redirect away from URLs based on Accept-Language it would make your site uncrawable by search engine spiders.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
