大网数据广告活动
网站运营与故障问答

Should I block the '_next' directory in a site using Next.js with robots.txt?

· 52CE 我爱测

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx where xxx can be different JSON…

问题原文

原帖链接 · checklist · CC BY-SA 4.0

We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx where xxx can be different JSON files or images.

My question is whether I should block the /_next directory so that I don't run into duplicate content issues or stale content (as these JSON files change for each deployment).

社区高分解答

原帖链接 · Brendan · CC BY-SA 4.0

I wrote an entire blog post on the topic:

Robots.txt - Block _Next Folder in Next.js Application

I tried to block the entire /_next/ directory in my robots.txt file to test out the idea.

I then ran into issues with the CSS not being able to be rendered by Google bot.

So I did the next best thing and disallowed files that were only used for User Experience (Metadata files):

User-agent: *

# Next.JS Crawl Budget Performance Updates
# Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
# The asterisks allows any file name
# The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html) 
Disallow: /*.json$
Disallow: /*_buildManifest.js$
Disallow: /*_middlewareManifest.js$
Disallow: /*_ssgManifest.js$
Disallow: /*.js$

We now save some additional Google Crawl Budget for the files we actually want Google to crawl :)

PS - If you are looking for a great explanation on blocking specific files in your robots.txt file, use this reference:

https://martijnoud.com/robots-txt/#Disallow_a_certain_file

EDIT: 3/13/22 - I decided to block all JS files, I originally was only blocking buildManifest, middlewareManifest, and ssgManifest.. But Google just really likes to check to make sure expired JS is still a 404. So just blocking all JS for now!

EDIT: 8/10/2022 - This is still working like a charm after 5 months of implementing this change. No crawl/indexing issues from Google and my crawl budget is being used much more efficiently.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Webmasters Stack Exchange
适用范围
以原帖系统、软件版本与发布日期为准(2021-11-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。