大网数据广告活动
服务器运维问答

NGINX - Send request to backend if the request is coming from a crawler/bot

· 52CE 我爱测

I have a Single Page Application without server-side rendering. So whenever someone shares a URL from the site on Facebook or Twitter it does not show the dynamic title of the page. I would like to send crawler/bot requests to a …

问题原文

原帖链接 · IHateToxicPlatforms · CC BY-SA 4.0

I have a Single Page Application without server-side rendering. So whenever someone shares a URL from the site on Facebook or Twitter it does not show the dynamic title of the page.

I would like to send crawler/bot requests to a special backend that serves a page with OpenGraph meta tags.

Front-end is hosted on NGINX and I have this configuration so far:

server {
    listen       80;
    listen  [::]:80;

    location / {
    
        if ($http_user_agent ~* "linkedinbot|googlebot|yahoo|bingbot|baiduspider|yandex|yeti|yodaobot|gigabot|ia_archiver|facebookexternalhit|twitterbot|developers\.google\.com") {
            return 307 https://renderer_service_domain$request_uri;
    }

    root   /usr/share/nginx/html;
    index  index.html;
}

This works, but Angular's URL has the # before the route url, I need to remove it before sending the request to the special backend.

So, how can I remove the hash from the $request_uri ?

I tried couple rewrite examples, but it sends the request to the same domain, I need to send the request to another server/domain name.

已采纳解答

原帖链接 · Tero Kilkanen · CC BY-SA 4.0

The hash # and contents after it are not sent to the HTTP server, so they are not a part of $request_uri variable.

With your example configuration, what is the exact request your backend receives?

I don't have experience with single page apps and Google, but I don't think it is a good approach to send different content to crawler bots than to end users.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2021-09-11)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。