大网数据广告活动
网站运营与故障问答

How to prevent discovery of URLs by search engines?

· 52CE 我爱测

I crawled my Wordpress site using the Screaming Frog program and noticed a few issues which might have SEO implications and was hoping someone could help identify the problem (if there is one). N.B : I have obscured the domain na…

问题原文

原帖链接 · jrcollins · CC BY-SA 4.0

I crawled my Wordpress site using the Screaming Frog program and noticed a few issues which might have SEO implications and was hoping someone could help identify the problem (if there is one).

N.B: I have obscured the domain name in the image as I do not want it made public.

The canonical URL for my domain is prefixed by https://www but as you can see from the tree graph image below, there are a number of URLs accessible under the http protocol and the non-www version of the domain. Those URLs are non-indexable but I'm curious as to why they're accessible at all. I was thinking that shouldn't happen if the proper redirects were in place.

In the Wordpress admin settings I have entered the correct version of my domain for the Wordpress and site addresses (i.e., https://www)

It's possible my .htaccess file might require editing so I have pasted it below:

# BEGIN LSCACHE
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
<IfModule LiteSpeed>
RewriteEngine on
CacheLookup on
RewriteRule .* - [E=Cache-Control:no-autoflush]
RewriteRule \.object-cache\.ini - [F,L]

### marker CACHE RESOURCE start ###
RewriteRule wp-content/.*/[^/]*(responsive|css|js|dynamic|loader|fonts)\.php - [E=cache-control:max-age=3600]
### marker CACHE RESOURCE end ###

### marker FAVICON start ###
RewriteRule favicon\.ico$ - [E=cache-control:max-age=86400]
### marker FAVICON end ###

### marker DROPQS start ###
CacheKeyModify -qs:fbclid
CacheKeyModify -qs:gclid
CacheKeyModify -qs:utm*
CacheKeyModify -qs:_ga
### marker DROPQS end ###

</IfModule>
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
# END LSCACHE
# BEGIN NON_LSCACHE
## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
### marker BROWSER CACHE start ###
<IfModule mod_expires.c>
ExpiresActive on
ExpiresByType application/pdf A31557600
ExpiresByType image/x-icon A31557600
ExpiresByType image/vnd.microsoft.icon A31557600
ExpiresByType image/svg+xml A31557600

ExpiresByType image/jpg A31557600
ExpiresByType image/jpeg A31557600
ExpiresByType image/png A31557600
ExpiresByType image/gif A31557600
ExpiresByType image/webp A31557600

ExpiresByType video/ogg A31557600
ExpiresByType audio/ogg A31557600
ExpiresByType video/mp4 A31557600
ExpiresByType video/webm A31557600

ExpiresByType text/css A31557600
ExpiresByType text/javascript A31557600
ExpiresByType application/javascript A31557600
ExpiresByType application/x-javascript A31557600

ExpiresByType application/x-font-ttf A31557600
ExpiresByType application/x-font-woff A31557600
ExpiresByType application/font-woff A31557600
ExpiresByType application/font-woff2 A31557600
ExpiresByType application/vnd.ms-fontobject A31557600
ExpiresByType font/ttf A31557600
ExpiresByType font/otf A31557600
ExpiresByType font/woff A31557600
ExpiresByType font/woff2 A31557600

</IfModule>
### marker BROWSER CACHE end ###

## LITESPEED WP CACHE PLUGIN - Do not edit the contents of this block! ##
# END NON_LSCACHE
#This Apache config file was created by Duplicator Installer on 2021-02-17 10:08:29.
#The original can be found in archived file with the name .htaccess__[HASH]

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

已采纳解答

原帖链接 · Itai · CC BY-SA 4.0

The best way to not make the site accessible is not to serve it. You can block bots by denying access via the user-agent string but that can be spoofed by malicious crawlers.

The example provided makes not attempts at redirecting traffic through the desired URL which lets users and bots crawl the site via 4 versions (https://www.example.com, http://www.example.com, https://example.com and http://example.com). By adding a few lines to your .htaccess file, you can permanently redirect traffic to the three undesired protocol and host combinations which means you will no longer be serving pages with URLs you don't want. There are different ways to do this but here is one:

RewriteEngine on

RewriteCond %{HTTPS} off
RewriteRule ^/?(.*)$ https://www.example.com/$1 [R=301,L]

RewriteCond %{HTTPS} on
RewriteCond %{HTTP_HOST} !^www.example.com$
RewriteRule ^/?(.*)$ https://www.example.com/$1 [R=301,L]

Without the above or something equivalent, all 4 versions are accessible and even without links to the URLs you don't want, I found that they eventually get accessed occasionally. It's not clear why but I've seen it many times in logs.

Since this is a new site, you may even not redirect at all and return 404 not found since there is no incoming links to preserve but it's usually good to redirect in case users type the site from memory and don't enter it the way you expect.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2021-03-05)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。