大网数据广告活动
网站运营与故障问答

What does User-agent: Mediapartners-Google mean?

· 52CE 我爱测

I was going through the google support website and there they said: "To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your robots.txt file: User-agent: Mediapar…

问题原文

原帖链接 · Muhammad Ahmad · CC BY-SA 4.0

I was going through the google support website and there they said:

"To update your robots.txt file to grant our crawler access to your pages, remove the following two lines of text from your robots.txt file:

User-agent: Mediapartners-Google
Disallow: /

This change will allow our crawler to index the content of your site and provide you with Google ads."

Can somebody explain to me why I need to remove the above two lines from my blogger website? I saw a guy on YouTube that was adding these lines for SEO purpose.

已采纳解答

原帖链接 · Henry Visotski · CC BY-SA 4.0

Whether you should leave the directive or remove it depends on if you run Google's AdSense ads.

The directive above states that for the Mediapartners-Google user agent (used for the AdSense program), everything on your site (from the root onward) should be disallowed. Removing the two lines would allow this user agent to crawl your website, or at least allow as much as your directive for all bots, commonly denoted by a wildcard character (*), otherwise permits.

Google AdSense is Google's ad network for web properties. You can run ads on your website, or a select set of your webpages, when you are approved for the AdSense program and place the necessary code on your website.

In order to run appropriate ads on your website, Google needs to crawl your content and determine its context. This is where their ads crawler comes into play. Google has many crawlers, and this is just one of them.

So if you run Google AdSense ads on your site, get rid of the two lines. If you do not, then you don't need AdSense to crawl your site, and you can leave them in. AdSense has a lightweight crawler that won't overwhelm your website either way, so if you allow it to crawl your site, you probably won't notice much of a difference, as long as you monitor its activity on your site once in a while.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2023-01-14)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。