大网数据广告活动
网站运营与故障问答

How does Google discover and index URLs that aren't linked or in the sitemap

· 52CE 我爱测

I can see multiple URLs of my website are crawled by google. I see it using site: in Google search. I was wondering what are all possible places from where Google picks these URLs? I checked many of my crawled URLs are not in the…

问题原文

原帖链接 · maverick · CC BY-SA 3.0

I can see multiple URLs of my website are crawled by google. I see it using site: in Google search.

I was wondering what are all possible places from where Google picks these URLs? I checked many of my crawled URLs are not in the sitemap, and we haven't put link of these URLs on any other page too. How would Google discover such content?

Is there anyway I can check all my Google indexed URLs and get information regarding how Google discovered those pages?

已采纳解答

原帖链接 · Henry Visotski · CC BY-SA 3.0

There are many places Google can go to index your site pages. Your sitemap, and what's on your live site, are only a small part of it. Your XML sitemap is merely a signal to Google, Bing, and other search engines to index your most important pages and to take note of new content (if you're using a CMS and a plugin that automatically updates the sitemap.)

When Google gets into your site, it follows all kinds of links, not just page-level links. It can index files, taxonomies, multiple versions of pages... In a CMS like Drupal, where everything is a node, it can even index portions of pages.

This is why it's important that you know your CMS and how it works on the backend. You have to use a combination of noindex meta, canonicalization, redirects, robots.txt, and Search Console / Bing Webmaster to control what's being crawled/indexed and what isn't.

Using Search Console to look at inbound links, Moz's Open Site Explorer to analyze the linkscape of any individual page, and a tool like Screaming Frog SEO Spider (the first one is free, the second and third are freemium) will allow you to analyze both internal and external links. Between all of these, you should be able to diagnose the source.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2017-11-18)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。