问题原文
原帖链接 · Mivaweb · CC BY-SA 3.0
We have a website where we have a round 30 pages that are only visible for authorized users. in Google WMT we see errors appearing about this that Google bot can't read this.
Do we have to add them into our robots file to not let them be indexed or can we leave it this way?
已采纳解答
原帖链接 · closetnoc · CC BY-SA 3.0
You can do either. There is no harm in 401 error notices. They are just notices for your benefit.
I would still advise, if possible, adding them to your robots.txt file so that Google does not look for them. I assume these can easily be identified by the URL.
Others allow Google to index these pages to encourage new users, however, from a user experience (UX) point of view, I am not convinced this is a good thing.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
