问题原文
原帖链接 · Jean-Baptiste Rudant · CC BY-SA 4.0
Might crawlers visit a page if there is no link referencing this page anywhere but a URL to this page is generated client side with JavaScript ?
Context:
Let's say I have a SPA with Server Side Rendering. Some pages show a list of items and offer a filtering facility. When the user selects some options or fills in some input field to filter the list, I'd like to embed this information in the URL (eg. /items?sort=price&order=desc&q=something) via the history API (client side routing). Behind the scene, an API call is made to get the results.
Since I do SSR, the server will also be able to understand these URLs and render these pages (hence the user can bookmark the page or share it). But nowhere in the HTML pages these URLs will appear, there are only generated client side in response to user events.
In this context, I think crawlers won't know these pages exist, and so, they should have no impact on SEO. Even if crawlers are now able to run JavaScript, they don't use it to simulate user events.
Am I wrong ?
(I guess if someone shares publicly that kind of URL, it could suffice to make this page crawled ? In any case, what I'm worried about is the cost on the crawling budget if all these pages are visited, but I'm ok with a few pages being crawled, they could be marked as "noindex" for instance).
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
You don't need to worry about crawl budget. Your site should have plenty of crawl budget even if Googlebot does start crawling those pages. Pages that have just one way to get to them are going to be very low priority. Google would still crawl the main pages of your site well and far more often.
Googlebot will crawl pages that you don't link to in some cases. see How can I figure out how a search engine is finding hidden pages?
- Googlebot can process JavaScript and it has heuristics to pick things that look like URLs out of JavaScript code. If you have JS code like
var url = "/items?sort=price&order=desc&q=something"that would trigger Google to crawl that page. However if the URL is built from smaller pieces, Googlebot shouldn't try to piece together the URL. - Googlebot executes JavaScript that happens on page load. If that JavaScript creates HTML
<a hrefs in the DOM, Googlebot will crawl those URLs. It is worth noting that Googlebot doesn't simulate user actions such as scrolling or clicking when it executes JavaScript. Googlebot only executes code that runs when the page loads and then analyzes the output of that. - Googlbot will try to fill in forms by picking and choosing combinations of the various inputs. Google announced that here. Google says that they limit this type of crawling to "a small amount" per site.
- If your pages have external links, the pages you link to will get notified of your URL in the HTTP Referrer header. Many sites publish inbound links with features like WordPress trackbacks or analytics data that isn't password protected. Googlebot often finds out about pages through their external links because of this.
- Users may create links or bookmarks to pages that Google finds (as you already noted.)
Google could penalize your site if you allow site search to be crawled. Your q=something parameter is worrying. Google doesn't want to index anything on your site that could look like search results because it is bad user experience for a user to click from Google search results onto other search results. See https://www.mattcutts.com/blog/search-results-in-search-results/
Having lots of crawlable pages based on permutations of parameters can lead to lots of duplicate content and low quality pages. It is usually best to ensure that crawlers see content at only one preferred URL. See What is duplicate content and how can I avoid being penalized for it on my site?
I would recommend disallowing these URLs in robots.txt. Something like this would work. It would allow crawling of /items but disallow crawling of the same URL with parameters.
User-agent: *
Disallow: /items?
If you want to disallow Googlebot from URLs with just a specific parameter (like q=) you could use the following. It useds wildcard rules in Disallow: directives which Googlebot understands, but most robots do not.
User-agent: Googlebot
Disallow: /items?*&q=
Disallow: /items?q=
When pages are blocked by robots.txt, Googlebot won't crawl them, but Google still may show them in the search results based on external links. See Why do Google search results include pages disallowed in robots.txt?
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
