大网数据广告活动
网站运营与故障问答

Do search engines index folders and subfolders regardless of pages?

· 52CE 我爱测

I know search engines first index the main page, then follow by its links and do the same with the pages by the links as the main page before. Ok, but what about folders and subfolders? Do search engines index them separately eve…

问题原文

原帖链接 · stckvrw · CC BY-SA 4.0

I know search engines first index the main page, then follow by its links and do the same with the pages by the links as the main page before. Ok, but what about folders and subfolders? Do search engines index them separately even if a folder is not mentioned in the source code of a page? For example I have the folder /some_folder and it's nowhere to be mentioned in the source code of pages, will it be indexed too?

If yes and I use static html pages and I have a page with URL like category/page (i.e. the page.html file in the /category folder), does it mean search engine will index this folder with html files and there will be two copies of the /category/page: one from indexing of the page after following by the link /category/page and one from direct indexing the /category folder with the same page as html file?

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Search engines only index URLs that have unique content on them. If your folder URL doesn't return any useful content, search engines won't index it. That will be the case if the folder URL:

  • Returns a blank page
  • Has a error code such as 403 Forbidden or 404 Not Found
  • Redirects to some other URL

Search engines do try to discover content by crawling folders. If you use the URL /category/page on your site, search engines may very well try to crawl /category/ to see if they can find content there.

If you are powering your website with static files, directory URLs may have content that is automatically generated by your web server. Most web servers are configured to serve a directory listing. That is often enough "content" for search engines to index. Further more, search engines will follow the links in that directory listing and find all the other files you have in the directory.

If you don't want your directory listing indexed you can add an index.html to that folder to replace the directory listing with content of your own choosing. Alternately you can use Options -Indexes on Apache to turn off the generation of directory indexes and instead serve an error page.

Search engines never index the same URL multiple times, even if they find multiple ways to get to the same URL. If you link to /category/ and search engines also try find it because they expect it from /category/page they will still crawl and index that URL only once.

For directories there are some issues with URL canonicalization. All three of these URLs may return the same content:

  • /category
  • /category/
  • /category/index.html

Web servers usually redirect to add the slash to directory URLs. Even if you power the directory content with an index.html file you shouldn't use index.html in your links. Just link to the directory URL ending in the trailing slash.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2019-05-13)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。