大网数据广告活动
网站运营与故障问答

How to tell bots to forget a website and reindex it from scratch

· 52CE 我爱测

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch? Context: After replacing a website done in CMS-x with a …

问题原文

原帖链接 · Pit · CC BY-SA 4.0

I was wondering if there was anything I could add to robots.txt or sitemap to tell bots to completely forget everything they know about a website and index it from scratch?

Context: After replacing a website done in CMS-x with a new one done in CMS-y, 99% of pages/links/resources will be gone or moved to different locations, and even though there are proper 404/410 redirects in place, it still would be better if any bots indexing the website would not try to access old stuff.

Basically this: How to tell google a blog article has been updated? but site wide

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience.

Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, you need to implement the page level redirects

If there is no equivalent for some particular page you can let it 404 and Google will remove it from the index. Using "410 Gone" instead gets Google to drop the URLs from the index as soon as they are crawled without the 24 hour grace period that Google uses for "404 Not Found."

There is no directive to tell bots to forget about an old site either in search console or robots.txt.

What if you don't redirect?

It may be too much work to redirect, or your new CMS may not make the redirect implementation easy.

If you choose not to implement the redirects it will be something like starting over. Google will see that your old URLs return 404 status and it will remove them from the search index.

Your new URLs will eventually get indexed, but it may take a while. Changing all your URLs without redirects is a big sign that your site isn't stable and can't be trusted. All your rankings will be lost and your site will start over.

Googlebot will continue to crawl the old URLs for years. For it, hope springs eternal that you may someday put those pages back up.

If you do redirect, all your inbound links, users' bookmarks, and most of your current rankings will be preserved.

Why?

So why don't search engines have a "reset" button? Because there are almost always better options. In your case it is much better to redirect.

In the case when a site is penalized, Google doesn't provide a reset button because that might remove all penalties.

How?

So how do you implement the redirects? You need a list of your old URLs. You may have a sitemap from your old site that you can start with. You can also get the list from your server logs, Google Analytics, or even from Google Search console.

If you planned ahead, your URLs in your new CMS will be similar and you can implement a rewrite rule to handle them. If there is a pattern between the old and new URLs, it can be a one liner in a .htaccess file to issue the redirects for the entire site.

If you have to manually find the new URLs and map thousands of them one by one, you could look into RewriteMap functionality.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2019-01-22)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。