大网数据广告活动
网站运营与故障问答

Accidentally submitted broken URLs in a sitemap. How can I stop Google crawling them?

· 52CE 我爱测

We recently added a sitemap to our site but didn't exclude unpublished items (which, unless you're logged in as an admin, will 404). Google started trying to crawl them and as a result, we saw a huge increase in the number of 404…

问题原文

原帖链接 · Tim · CC BY-SA 3.0

We recently added a sitemap to our site but didn't exclude unpublished items (which, unless you're logged in as an admin, will 404). Google started trying to crawl them and as a result, we saw a huge increase in the number of 404s in the Google Webmaster tools.

We've since corrected the sitemap and now those items are not included, however the graph still indicates there are a high number of 404s. We can mark these URLs as "fixed" in the webmaster tools, but they aren't fixed - they're still 404ing, they just shouldn't be crawled and indexed.

Is it possible to tell Google not to crawl these? Will I need to add these 40,000 links to our robots file? And will these 404s be causing an issue for SEO?

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Googlebot generally never stops crawling pages once it starts. It is possible that Googlebot will return to check on those URLs every once in a while indefinitely. For reference here is an article about Google's big memory for 404 pages.

You say that you already removed the URLs from the sitemap. That is good. If you hadn't already done so, that would be the first step.

You don't want to add 40,000 individual URLs to your robots.txt file. That could produce a file that is too big. The max robots.txt size is 500KB for Google. Other crawlers may not process even that much.

Having "404 Not Found" error pages on your site is not a problem. Google's John Mueller says:

404 errors on invalid URLs do not harm your site’s indexing or ranking in any way. It doesn’t matter if there are 100 or 10 million, they won’t harm your site’s ranking. http://googlewebmastercentral.blogspot.ch/2011/05/do-404s-hurt-my-site.html

Having that many 404 errors will make that report in Google Search Console much less usable. One way to remove them from it would be to return a more appropriate status. I would suggest: "401 Unauthorized". That indicates that there is content there, but that the user would be required to login to see it. If the user is logged in, but not an admin, then a "403 Forbidden" status would be appropriate.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-02-26)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0 / CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。