大网数据广告活动
网站运营与故障问答

Remove large number of useless page from Google index using robots.txt

· 52CE 我爱测

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error. I'm trying to clean this in order to be sure that only the most high quality content is on Google index. So I have adde…

问题原文

原帖链接 · Jonathan Meyer · CC BY-SA 3.0

A large number of auto-generated pages from the search of my site has been indexed by Google due to an error.

I'm trying to clean this in order to be sure that only the most high quality content is on Google index.

So I have added a line to disallow this directory in robots.txt so googlebot doesn't waste time crawling:

Disallow: /search/

After some months it worked as expected and most pages are indexed on Google only as URLs with this message:

A description for this result is not available because of this site's robots.txt

Is this is enough for preventing future penalties like Panda, or since they are still on the index (even as only URLs due to the disallow) Google will still have the pages in their "cache" from the last crawl and it may cause problems?

已采纳解答

原帖链接 · Mike -- No longer here · CC BY-SA 3.0

What I would suggest is to create a brand new set of URLs for search result pages and then for anyone requesting the old URLs, produce an error page with an HTTP 410 status code. Also, make your search pages only accessible via the POST request method.

Google won't crawl pages that are requested via POST if it is requested as a result from filling out a proper form. An example is logging in to a specific section of a website.

For example:

If your current search result URLs are in the form of:

http://example.com/results.php?query=abc

then that needs to return a 410 status code with an error indicating the page is no longer available.

What you need to do is create a proper search form. In HTML, this will work:

<form action="searchfor.php" method="post">
Query: <input type="text" name="query">
<input type="submit" value="search">
</form>

When the user clicks search, the page requested will be searchfor.php and the data posted will be query=whatever (replace whatever with the text user entered) and the value can be extracted in the server script. In all search result pages, the URL in the address bar will always remain the same but the page will be different based on the query.

For best results, make sure the form action value points to a different script name. The script however must exist on the server. You can insert the full URL to the script if that makes things easier for you.

Just make sure there is no other way for people to access the search results pages and then google will not try to access them. If you must allow users to access them another way via a hyperlink, you should include a rel="nofollow" inside the anchor tag and in the particular result page, make it non-indexable. See other answers in this thread for instructions on how to make the page non-indexable.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2016-06-28)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。