问题原文
原帖链接 · Muhammad Ahmad · CC BY-SA 4.0
I was going through the google support website and there they said:
"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your robots.txt file:
User-agent: Mediapartners-Google
Disallow: /
This change will allow our crawler to index the content of your site and provide you with Google ads."
Can somebody explain to me why I need to remove the above two lines from my blogger website? I saw a guy on YouTube that was adding these lines for SEO purpose.
已采纳解答
原帖链接 · Henry Visotski · CC BY-SA 4.0
Whether you should leave the directive or remove it depends on if you run Google's AdSense ads.
The directive above states that for the Mediapartners-Google user agent (used for the AdSense program), everything on your site (from the root onward) should be disallowed. Removing the two lines would allow this user agent to crawl your website, or at least allow as much as your directive for all bots, commonly denoted by a wildcard character (*), otherwise permits.
Google AdSense is Google's ad network for web properties. You can run ads on your website, or a select set of your webpages, when you are approved for the AdSense program and place the necessary code on your website.
In order to run appropriate ads on your website, Google needs to crawl your content and determine its context. This is where their ads crawler comes into play. Google has many crawlers, and this is just one of them.
So if you run Google AdSense ads on your site, get rid of the two lines. If you do not, then you don't need AdSense to crawl your site, and you can leave them in. AdSense has a lightweight crawler that won't overwhelm your website either way, so if you allow it to crawl your site, you probably won't notice much of a difference, as long as you monitor its activity on your site once in a while.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
