问题原文
原帖链接 · Jonathan Meyer · CC BY-SA 3.0
A large number of auto-generated pages from the search of my site has been indexed by Google due to an error.
I'm trying to clean this in order to be sure that only the most high quality content is on Google index.
So I have added a line to disallow this directory in robots.txt so googlebot doesn't waste time crawling:
Disallow: /search/
After some months it worked as expected and most pages are indexed on Google only as URLs with this message:
A description for this result is not available because of this site's robots.txt
Is this is enough for preventing future penalties like Panda, or since they are still on the index (even as only URLs due to the disallow) Google will still have the pages in their "cache" from the last crawl and it may cause problems?
已采纳解答
原帖链接 · Mike -- No longer here · CC BY-SA 3.0
What I would suggest is to create a brand new set of URLs for search result pages and then for anyone requesting the old URLs, produce an error page with an HTTP 410 status code. Also, make your search pages only accessible via the POST request method.
Google won't crawl pages that are requested via POST if it is requested as a result from filling out a proper form. An example is logging in to a specific section of a website.
For example:
If your current search result URLs are in the form of:
http://example.com/results.php?query=abc
then that needs to return a 410 status code with an error indicating the page is no longer available.
What you need to do is create a proper search form. In HTML, this will work:
<form action="searchfor.php" method="post">
Query: <input type="text" name="query">
<input type="submit" value="search">
</form>
When the user clicks search, the page requested will be searchfor.php and the data posted will be query=whatever (replace whatever with the text user entered) and the value can be extracted in the server script. In all search result pages, the URL in the address bar will always remain the same but the page will be different based on the query.
For best results, make sure the form action value points to a different script name. The script however must exist on the server. You can insert the full URL to the script if that makes things easier for you.
Just make sure there is no other way for people to access the search results pages and then google will not try to access them. If you must allow users to access them another way via a hyperlink, you should include a rel="nofollow" inside the anchor tag and in the particular result page, make it non-indexable. See other answers in this thread for instructions on how to make the page non-indexable.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
