大网数据广告活动
网站运营与故障问答

What headers used in request by google bot?

· 52CE 我爱测

What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache? The related question: Do browsers send different HTTP hea…

问题原文

原帖链接 · LeonidMew · CC BY-SA 4.0

What headers used in request by google bot? Must be user agent, but what else? I'm interested in cache control headers, does bot try to get fresh page, but not from cache?

The related question: Do browsers send different HTTP headers on page reload so that server side caches can be flushed?

I'm using server cache, what have option to refresh on Ctrl+F5, but don't want bot to refresh page. Code what detect Ctrl+F5 below:

function cacheHitDisplay($params)
  if ($_SERVER['HTTP_CACHE_CONTROL'] == 'no-cache') {
    // process/refresh page. ctrl-f5 pressed
    return 0;
  }
  getFromCache($params);
  exit 0
}
cacheHitDisplay(); // will show cached page and exit script
...here normal page processing, saving page to cache at end...

已采纳解答

原帖链接 · MrWhite · CC BY-SA 4.0

if ($_SERVER['HTTP_CACHE_CONTROL'] == 'no-cache') {

Also bear in mind that the Cache-Control HTTP request header (as with most request headers) are entirely optional, so may not be present as part of the request at all. (This line would then generate an E_NOTICE under these conditions.)

In fact, I don't believe Googlebot sends the Cache-Control header when performing it's regular website crawl (as opposed to Fetch and Render in GSC). This is from my own logs, and from those reported online. So, if you "don't want bot to refresh page" then you would seem to be correct to specifically check for "no-cache" in order to refresh your server-side cache.

The HTTP Request headers that I typically see from Googlebot are really quite minimal:

Host: www.example.com
Connection: keep-alive
Accept: text/plain,text/html,*/*
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate,br

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2019-11-29)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。