大网数据广告活动
服务器运维问答

How to block bots with nginx

· 52CE 我爱测

I have a Rails app running on nginx on the aws elastic beanstalk platform. I've checked the nginx/access.log and found a number of suspicious GET requests like so: SUSPICIOUS_URL == '#m. #face #book #.com' 172.31.17.148 - - [24/J…

问题原文

原帖链接 · Ryan.lay · CC BY-SA 3.0

I have a Rails app running on nginx on the aws elastic beanstalk platform. I've checked the nginx/access.log and found a number of suspicious GET requests like so:

SUSPICIOUS_URL == '#m. #face #book #.com'

172.31.17.148 - - [24/Jul/2016:04:02:37 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:37 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:37 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:37 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:38 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:38 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http://SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

172.31.17.148 - - [24/Jul/2016:04:02:38 +0000] "GET /bisnis/read/2439204/beredar-di-medsos-bi-bantah-keluarkan-uang-pecahan-rp-200-ribu HTTP/1.1" 301 5 
"http:/SUSPICIOUS_URL/" "Mozilla/5.0 (Linux; Android 5.1.1; SM-E700H Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/51.0.2
704.81 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/86.0.0.19.69;]" "175.141.68.34"

I'm fairly certain that these are bots, how do I block these suspicious requests with nginx?


Edit:

Apparently this is a practice called referral spamming. Read more here. https://www.wikiwand.com/en/Referer_spam

已采纳解答

原帖链接 · chrwhm · CC BY-SA 3.0

Looks like a Samsung Galaxy using outdated dns-entry for m.facebook.com ;-)

Well, if you take a look at your access.log you will find a lot of requests like /pma, /webadmin, /admin/config.php, /w00tw00t, /vtigercrm and so on. It seems that there are a few bots checking for old / vulnerable versions of well-known opensource software like phpmyadmin, wordpress or something else.

Since we didn't like that very much, we tried to keep them out. First time, we only added a few locations to our nginx-config, sending an error 404 on requests which matched an entry from a manually-filled list. That was nice to confuse them but didn't really decrease the amount of "bad requests". Later on, we moved the list to "fail2ban". Whenever a GET-request appeared on access.log with one of our predefined URIs, the client IP was blocked for 48 hours by iptables. since they couldn't reach our server anymore, that helped very much. You only need to install fail2ban, add a filter (on debian at /etc/fil2an/filter.d/xxx.conf) with a simple list of regex values and add the filter to jail.conf.

For example, /etc/fail2ban/filter.d/nginx-bots.conf:

[INCLUDES]

[Definition]
failregex =     ^<HOST> - .* "GET /?/phpmyadmin/
                ^<HOST> - .* "GET /?/phpMyAdmin/
                ^<HOST> - .* "GET /?/pma/
                ^<HOST> - .* "GET /?/PMA/
                ^<HOST> - .* "GET /?/sqlmanager/
                ^<HOST> - .* "GET /?/phpmyadmin-?2/
                ^<HOST> - .* "GET /?/phpMyAdmin-?2/
                ^<HOST> - .* "GET /?/mysqlmanager/
                ^<HOST> - .* "GET /?/webadmin/
                ^<HOST> - .* "GET /?/php-my-admin/

                ^<HOST> - .* "GET /?/myadmin
                ^<HOST> - .* "GET /?/MyAdmin
                ^<HOST> - .* "GET /?/cgi-bin/
                ^<HOST> - .* "POST /?/%%
                ^<HOST> - .* "GET /admin/config.php
                ^<HOST> - .* "GET /idssvc/iesvc
                ^<HOST> - .* "GET /vtigercrm/
                ^<HOST> - .* "GET /wstats/wstats
                ^<HOST> - .* "GET /CluJaNul/
                ^<HOST> - .* "GET /bynazi/
                ^<HOST> - .* "GET /invoker/
                ^<HOST> - .* "GET \\x
                ^<HOST> - .* "GET /jmx-console/
                ^<HOST> - .* "GET /zecmd/
                ^<HOST> - .* "GET /?/manager/html/upload

                ^<HOST> - .* "GET /w00tw00t
                ^<HOST> - .* "GET /muieblackcat

ignoreregex =

Added on /etc/fail2ban/jail.conf:

[nginx-bots] 
enable   = true
port     = http,https
filter   = nginx-bots
logpath  = /var/log/nginx/access.log*
maxretry = 1
bantime  = 172800
findtime = 86400

You also could think about another way to filter out those bots. If you add the status-code to nginx access log format you are able to set up the fail2ban/filter.conf to block users with more than 10 404-errors per minute - or somthing like that.

But you have to keep in mind, that it might block "good" users, if your fail2ban-configuration isn't good enough.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2016-07-24)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。