大网数据广告活动
网站运营与故障问答

How I can ensure that a link sent via email is opened only via user clicks from a mail client and not by bots?

· 52CE 我爱测

In my web application I generate links in the following format: https://example.com/^token^ Each link according to my specs is sent via email and should not allow robots to scrape it. In order to avoid visits via robot I placed t…

问题原文

原帖链接 · Dimitrios Desyllas · CC BY-SA 4.0

In my web application I generate links in the following format:

https://example.com/^token^

Each link according to my specs is sent via email and should not allow robots to scrape it. In order to avoid visits via robot I placed the following robots.txt:

User-agent: *
Disallow: /

And on page's <header></header> tag I placed:

    <meta name="robots" content="noindex">
    <meta name="robots" content="nofollow">

The question is how I can ensure that a link is opened only when user has clicked upon it and not a random bot/spider scrapes it? Does the length and the randomness of the ^token^ in url factor into bot visit prevention?

In my application the ^token^ is a cryptographically-random 5 byte value that once generates is hex-econded. So in my case, if the token length takes a significant role on non-scrappines of this link, which is the recommended length of the random token?

My application is in Laravel and I use nginx and php-fpm combo in order to serve the content to the browser.

已采纳解答

原帖链接 · Stephen Ostermiller · CC BY-SA 4.0

Incorrectly used meta tags

Instead of using two meta tags, you should put both values into a single tag. With two tags, some search engines may choose to obey only one of the two.

<meta name="robots" content="noindex,nofollow">

Robots.txt and robots meta tags are mutually exclusive

Disallowing pages in robots.txt prevents robots from downloading the page and seeing the meta tags. There are approximately zero bots that wouldn't obey robots.txt but would obey meta tags. You should choose one or the other, but not both.

Robots.txt won't prevent search engines from indexing URLs

If Google finds enough links to a URL, it may include that URL in its search index, even if that URL is disallowed in robots.txt. If your fear is that some of these URLs will get indexed in search engines, you should allow crawling in robots.txt but disallow indexing via the meta tag.

Meta tags won't prevent bots from hitting the URLs

If your fear is that bots will mess up your stats or cause other undesired effects when bots hit them, then you should use robots.txt. Search engines might still index a URL occasionally, but most bots will obey robots.txt and not even request the URLs

No way to prevent indexing and bot hits with robots.txt and meta tags

If you want to prevent indexing of the URLs and bot hits to the URLs, you are out of luck. There is no way to use robots.txt and meta tags to do both at the same time.

Are your tokens long enough?

A five byte token gives you 256^5 or 1.1E12 (1.1 trillion) possible URLs. If you send out a million email messages, that leaves a 1 in 1 million chance of getting an in-use token for each guess. If you send out a billion email messages, the odds of getting a in-use token rise to 1 in 1 thousand. You'd certainly want to increase the length of your token after sending out 1 million emails.

You could also get a lot more security without increasing the length of your URLs. Five bytes hex encoded is 10 characters. It would be smarter to use 10 characters that are randomly chosen from:

  • 10 digits
  • 26 lowercase letters
  • 26 uppercase letters
  • 4 URL safe symbols (-._~)

If you did that you could increase the possible number of tokens to 66^10 or 1.5E18 (1.5 quintillion). That would give you enough token space no matter how many emails you sent out.

Other ways to increase security

You could also employ any of the following tactics to further ensure that bots don't get access to this content:

  • Use server side configuration that gives an error code if a suspected robot hits these URLs. You could detect robots based on:
    • User agent
    • What other things that IP address is requesting
    • Can it pass captcha?
  • Require that users log in to see this content when clicking from an email.
  • Expire old tokens: require that clicks happen in the hours or days after email is sent out.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-12-08)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。