大网数据广告活动
网站运营与故障问答

Will forbidding "bad bot" user agents via .htaccess add a layer of protection against email address scraping beyond the existing captcha?

· 52CE 我爱测

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot protection ( article ). I do …

问题原文

原帖链接 · Peter K. · CC BY-SA 4.0

I was reviewing the security/protection measures for the websites I manage (one is for a small consulting firm, the other for a not-for-profit art association), and I came across the topic of bad-bot protection (article). I do not have the resources to spend much time or money for a self-built solution (e.g. custom made honeypot), so I discovered the Apache Ultimate Bad Bot Blocker. So my questions is whether it is actually useful/helpful and not too performance impacting (adding 8000 lines in htaccess or virtual host file and 5000 lines in robot.txt)?

My skepticism both comes from the fact that this is obviously public information, so any "bad bot" writer would of course not use any of the User Agents or Referrers present in that list ?? I am not claiming that the information is wrong, but I assume this is collected ex-post, i.e. through honeypots or other means, but any "bad bot" would just use these parameters for a few days, and then move on with other parameters, which initially are not part of that list?

This recent (2018) article also explains why trying to prevent bad bots is a waste of time and resources.

Any real-world advice on what to do?

已采纳解答

原帖链接 · Don King · CC BY-SA 4.0

To answer your question, no, I don't think adding that many lines in htaccess etc. is a good solution. As you suspected you only block already detected bots.

Its more efficient to use a list like spamcop https://www.spamcop.net/fom-serve/cache/291.html and Apache Spamassasin https://spamassassin.apache.org/ directly on mailserver level to protect you from incoming spam.

In the end the brain is your best protection, in combination with the tools available. Unfortunately you need to spend resources to stay safe long-term.

1. Limit the amount of emails displayed on site(s).

If possible change to a submit form where visitor enters his or hers email and you contact them rather then showing an email in clear-text. If not possible, then make sure to have only 1 email displayed on the website, and then monitor that inbox like a real webmaster should. Anytime you receive spam or infected emails make sure to add the specific unique thing in that email to a spam filter so that any future email with same characteristics gets completely ignored in the future. This takes some practice but works great. You will be amazed how predictable most spam is. In time you get less and less "bad" emails. Be careful not to block legitimate emails creating too sensible filters though.

2. Educate the email users.

You can never stop "bad bots" completely (they are not bad at all, they are great, but the people that use them are bad) and you can spend endless resources trying to do it. It is on the other hand easier to educate the users of the emails connected to your servers. Make sure they understand to never open an attachment or click a file or even download one unless they have previously confirmed that they know the sender and are expecting such file, and even then be suspicious. If an email is unexpected, don't open it, but if possible forward it to you as webmaster etc. Educating people about how to use email safely is easier then stopping hackers from sending infected emails.

3. Don't worry, but be serious about security.

Don't go around paranoid worrying about bad bots and such. It's nothing to worry about unless you have something to steal. If you do have something valuable, then learn how to protect it. Make backups. Store backups on external servers in other data centers. Be serious about using latest version of scripts and keep an eye on security patches etc etc. Stay up to date!

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2018-11-05)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。