大网数据广告活动
网站运营与故障问答

Does Googlebot execute Google Tag Manager?

· 52CE 我爱测

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google Tag Manager that detects the…

问题原文

原帖链接 · Ralph · CC BY-SA 3.0

I wanted to understand how Googlebot (and other crawlers) crawls my site. Specifically whether it passes a document.referrer and if it maintains localStorage keys, so I implemented a script via Google Tag Manager that detects these crawlers and logs data to Logstash.

This is the condition I'm using to detect crawler user agents (returns true for crawlers):

function() {
if(navigator.userAgent.indexOf('robot de Google') < 0 &&
   navigator.userAgent.indexOf('Googlebot') < 0 &&
   navigator.userAgent.indexOf('bingbot') < 0 &&
   navigator.userAgent.indexOf('msnbot') < 0 &&
   navigator.userAgent.indexOf('BingPreview') < 0 &&
   navigator.userAgent.indexOf('Yahoo! Slurp') < 0) {
      return false;
   } else {
      return true;
   }
}

And this is the tag that sends sends a request to Logstash via an image pixel on the GTM Pageview event:

<script type="text/javascript">
(function (d) {

var pagePath = encodeURIComponent(document.location.pathname);
var pageReferrer = encodeURIComponent(document.referrer) || "null";
var userAgent = encodeURIComponent(navigator.userAgent);

var viewCount = Number(localStorage.getItem("preview_view_count")) + 1 || 1;
localStorage.setItem("preview_view_count", viewCount);

var js;
js = d.createElement('img');
js.style = 'display:none;';
js.alt = 'tracking img';
js.src = 'http://MY_LOGSTASH_ENDPOINT_DOMAIN/pixel.gif?EVENT=LogCrawl&USER_AGENT=' + userAgent + '&PAGE_PATH=' + pagePath + '&PAGE_REFERRER=' + pageReferrer + '&VIEW_COUNT=' + viewCount;
d.body.appendChild(js);
})(window.document);
</script>

Now when I look at Logstash, I only see 40 hits in the last 4 days from Googlebot, but Search Console reports ~50,000 pages crawled per day.

Has anyone tried to log Googlebot with GTM before? I'm trying to figure out if there is something wrong with my script, or if Googlebot just doesn't execute Javascript most of the time.

Any ideas are highly appreciated. Thanks.

已采纳解答

原帖链接 · schwarzbrot · CC BY-SA 3.0

The Search Console reports the overall number of files (!) that are downloaded (source). This includes JS / CSS / image files etc. and leads to quite huge numbers, depending on how much caching is involved.

So you might get more accurate numbers for crawled HTML pages by analyzing your server logs (if you do that: don't forget to verify hits from Googlebot, there's a lot of pretenders out there).

Regarding your tag: localStorage should be supported by Googlebot, but is unlikely to work across multiple page impressions. So if you took those 40 hits straight from VIEW_COUNT, you might instead want to check back to your log and add those hits up by yourself.

(very interesting!)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-02-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。