问题原文
原帖链接 · pixelbash · CC BY-SA 4.0
my client has to inform it's customers about some new regulations that the Googlebot should NOT crawl. It is not possible to place this information on a separate page and disallow Google to crawl it. So the idea is to place a button/link on the page, that will AJAX-load the corresponding information only when the user clicks it. My assumption is, that Google is unable to click the link and crawl that specific AJAX content.
Am I right? And if yes, is there an official documentation that proofs my point on this?
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
Googlebot does not interact with the page like a user. It does not click on anything. It does not scroll. If content is loaded into the page when users click, Google is not going to index it as part of that page.
However, Googlebot still may find AJAX content to which users have to click. That is because Googlebot scans the page source, the rendered document object model, and loaded JavaScript files for things that look like links. Googlebot is likely to discover the URL for the AJAX that way and then Googlebot may crawl it.
The only reliable way to keep Googlebot from crawling something is to use robots.txt. You could put the AJAX URL itself into robots.txt. Googlebot would still be allowed to crawl the page, but would not be allowed to load the AJAX. So you could use Disallow: /regulations.json in robots.txt but still have Googlebot crawl and index /mypage.html which calls regulations.json via AJAX.
You can prevent Googlebot from seeing portions of the page like this. It doesn't matter if the content is loaded via AJAX on click, or on document load. If Googlebot isn't allowed to crawl the AJAX URL, Google won't see the content. See Preventing robots from crawling specific part of a page for full details.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
