大网数据广告活动
服务器运维问答

Grok Issue with Multiple IP's in NginX Logstash

· 52CE 我爱测

i've got an issue with logging from my webservers, which has an elb and then a varnish layer in front of nginx layer. varnish is setup properly for X-Forwarded-For and logs come through normally with the correct 'client.ip' being…

问题原文

原帖链接 · geniestacks · CC BY-SA 3.0

i've got an issue with logging from my webservers, which has an elb and then a varnish layer in front of nginx layer.

varnish is setup properly for X-Forwarded-For and logs come through normally with the correct 'client.ip' being logged.

however, nginx logs are coming through with a whole list of IP's in the request. the default grok behaviour seems to set the client IP to the last in the list ie. the elb and varnish servers, which messes up my client.ip field for nginx logs. the correct client IP should be the first (or at least first few) in the list.

heres an example:

172.31.7.219 - - [28/Sep/2015:12:39:56 +1000] "GET /api/filter/14928/content?api_key=apikey&site=website HTTP/1.1" 403 101 "-" "-" "my.website.com" "1.144.97.102, 1.144.97.102, 1.144.97.102, 127.0.0.1, 172.31.26.59"

problem is i haven't been able to tweak the grok to handle such a result, the heroku grok debugger doesn't seem to work for this query and my grok -- but they are working in logstash ie. not tagging grok failure.

i've attempted to debug the specific parts but i haven't found a way to do what i need with IP/IPORHOST where there is a comma separated list of IP addresses. i need to be able to specify which IP it should use. ie. the first in the list should be the client.ip not the last.

my nginx grok is:

NGINXACCESS %{IP:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})(?:;|) %{QS:agent}

any ideas on grok to cover that log?

已采纳解答

原帖链接 · GregL · CC BY-SA 3.0

Not sure if you're still having this issue, but if so, here's what will work for you.

Given this log format:

log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$host" "$http_x_forwarded_for"';

the grok pattern you've specified doesn't take into account the addition of the "$host" "$http_x_forwarded_for" portion.

Not sure why your grok isn't failing, but it should.

In any event, this pattern will work with the log format above:

%{IP:clientip} %{NOTSPACE:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})(?:;|) %{QS:agent} "%{NOTSPACE:host}" "(?<x_forwarded_for>%{IP:xff_clientip}, .*)"

And results in the following fields

httpversion      1.1
request          /api/filter/14928/content?api_key=apikey&site=website
timestamp        28/Sep/2015:12:39:56·+1000
auth             -
host             my.website.com
agent            "-"
x_forwarded_for    1.144.97.102,·1.144.97.102,·1.144.97.102,·127.0.0.1,·172.31.26.59
clientip         172.31.7.219
bytes            101
response         403
xff_clientip     1.144.97.102
ident            -
port    
verb             GET
referrer    

Note that you've got a couple of new fields than you would have had before.

The first ("x_forward_for" => 1.144.97.102, 1.144.97.102, 1.144.97.102, 127.0.0.1, 172.31.26.59) is the contents of the last set of quotes, or $http_x_forwarded_for from the log format.
The second ("xff_clientip" => 1.144.97.102) is just the first IP in that list, which should translate to the actual source IP of the request.

If it were me, I'd also run the x_forwarded_for field through a mutate filter to break it into an array:

mutate {
  split  => { "x_forwarded_for" => ", " }
}

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2015-09-28)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。