大网数据广告活动
网站运营与故障问答

How to avoid robots from indexing pages of my app through alternate URLs?

· 52CE 我爱测

I have this web app that is accessible through those different URLs. URL_1: www.mycustomdomain.com // THIS IS MY CUSTOM DOMAIN (PUBLIC TO THE USERS) URL_2: www.my-project.firebaseapp.com // THIS IS A DEFAULT FIREBASE URL URL_3: m…

问题原文

原帖链接 · cbdeveloper · CC BY-SA 4.0

I have this web app that is accessible through those different URLs.

URL_1: www.mycustomdomain.com            // THIS IS MY CUSTOM DOMAIN (PUBLIC TO THE USERS)
URL_2: www.my-project.firebaseapp.com    // THIS IS A DEFAULT FIREBASE URL
URL_3: my-project.web.app                // THIS IS A DEFAULT FIREBASE URL

Obviously I want Google to index my pages using my custom domain, and not the Firebase default URLs.

I don't publicize the default URLs anywhere, but what if Google discovers them at some point? What is the proper way of letting Google know that those pages/URLs should NOT be indexed?

OPTION #1

Should I always add a <link rel="canonical" href={"https://www.mycustomdomain.com/some-page"}/> for each page?

This way, if Google happens to crawl the default URLs, it should see that canonical pointing to the custom domain, right? Is this the correct approach?

OPTION #2

Since I'm dynamically generating the robots.txt, I could detect that the request is coming from URL_2 or 3 and simply block the entire website from being crawled, by adding Disallow: /.

How would you solve this?

已采纳解答

原帖链接 · Itai · CC BY-SA 4.0

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you suggest.

The other way is to simply issue 301 Permanent Redirect so that all traffic uses the official URLs. This will also allow you to enforce the use of HTTPS as it also happens that Googlebot indexes both the HTTP and HTTPS version when they exist and there is no redirect between them.

Given that you control the site, you are even free to not issue the redirect for internal traffic. This is common if the internal server does not have an SSL certificate. You can redirect when the IP address is internal to your network, for example.

Robots is generally not a good option because bots can choose to ignore it. Googlebot will respect it but you may have to cover for other bots.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2020-12-30)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。