大网数据广告活动
服务器运维问答

MD5 $request_body

· 52CE 我爱测

I'm using nginx (and OpenResty) and trying to hash the $request_body for caching purposes (with srcache ). When trying to just echo the $request_body , it works flawlessly: # actually echoes the request body location /works { # f…

问题原文

原帖链接 · matan129 · CC BY-SA 3.0

I'm using nginx (and OpenResty) and trying to hash the $request_body for caching purposes (with srcache).

When trying to just echo the $request_body, it works flawlessly:

# actually echoes the request body
location /works {
    # force loading the body
    echo_read_request_body;

    echo $request_body;
}

But when trying to calculate the md5, I get the MD5 for an empty string, although I ensure the $request_body is loaded via echo_read_request_body;.

# echoes d41d8cd98f00b204e9800998ecf8427e, which is the MD5 for empty input.
location /doesnt-work-md5 {
    # force loading the body
    echo_read_request_body;

    set_md5 $digest $request_body;
    echo $digest;
}

The weirdest phenomenon is that it doesn't even work when just trying to copy the variable:

# echoes nothing - empty data
location /doesnt-work {
    # force loading the body
    echo_read_request_body;

    set $temp $request_body;
    echo $temp;
}

By the way, these latest snippets don't work when using $echo_request_body instead of $request_body.

Thanks in advance!

已采纳解答

原帖链接 · matan129 · CC BY-SA 3.0

With the help of the developers of OpenResty, I realized that the set_XYZ (i.e. set, set_md5) are evaluated during nginx's rewrite phase, while the $request_body/$echo_request_body are available only through the content phase.

So, when evaluating set_md5 $digest $request_body;, $request_body variable is empty, which explains the constant MD5 result.

Ultimately I implemented the actual cache key generation in my own API application (see example below), and accessed it with access_by_lua block.

The block runs in the access phase, before srcache_fetch and srcache_store are evaluated (they evaluate in post-access and output-filter, respectively).

Implementing it in my own API allowed greater control over the cache key generation logic, which would be hard to do with nginx alone (since I didn't want to become a full-pledged lua programmer).

For instance, I wanted to be able to deterministically cache POST requests with Json bodies. Json serialization is not deterministic since keys can be in any order. In my API, I sort the keys so the generated cache key is constant for the same data.

Also, it simplified handling of the $request_body, since the lua-issued subrequst just forwards it to the API, regardless of phase or disk buffering status.

The final config looks like

location /api {
    # proxy_pass ...
    # Force normal responses (no deflate, etc.) See https://github.com/openresty/srcache-nginx-module#srcache_ignore_content_encoding
    proxy_set_header  Accept-Encoding  "";

    set $cache_key "";
    access_by_lua_block {
        local res = ngx.location.capture('/generate-key' .. ngx.var.request_uri, {
                method=ngx.HTTP_POST,
                -- forwards the entire request body,
                -- regardless of disk buffering!
                always_forward_body=true,
                args=ngx.var.args
            })

        if res then
            ngx.var.cache_key = res.body
        end
    }

    # ... srcache options ...
}

location /generate-key {
    # proxy_pass ...
}

Example key generation API is as follows:

import flask
import json
import hashlib
import urllib


app = flask.Flask(__name__)

@app.route('/generate', defaults={'path': ''}, methods=['POST'])
@app.route('/generate/<path:path>', methods=['POST'])
def generate_cache_key(path):
    return urllib.quote('{}_{}_{}'.format(path,
        digest(stable_body()),
        digest(stable_json_dumps(flask.request.args))))


def stable_body():
    if flask.request.json:
        return stable_json_dumps(flask.request.json)

    return flask.request.data


def stable_json_dumps(data):
    return json.dumps(data, sort_keys=True)


def digest(data):
    return hashlib.md5(data).hexdigest()


if __name__ == '__main__':
    app.run()

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2018-01-24)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。