问题原文
原帖链接 · abhinsit · CC BY-SA 3.0
We have added following meta tag on some of our pages:
<meta content="noindex" name="robots"/>
Does google crawl the content and links on this page and just don't show it on the google search results.
If yes then how can i stop bot not to crawl the content and the links on this page ?
已采纳解答
原帖链接 · Raul Reyes · CC BY-SA 3.0
By default Google or any other modern search engine will crawl, index and follow the links in a page when the above meta tag is absent.
In your example Google will find the noindex instruction and this is what is expected to happen:
- Google bot will crawl the page (if robots txt allows it)
- Your page will not get indexed and will be removed from SERP, eventually.
- The search engine will follow all the links in the page.
- linked pages will get indexed.
Add nofollow
Add this instruction to the meta tag, this will tell Google not to follow the links. But, if these pages are linked from other pages without this instruction the pages will be indexed.
Do not allow Google to crawl the page
Make sure to Disallow o block access via robots txt. However, do this after Google has crawled the page and is aware of the noindex, nofollow instruccions.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
