问题原文
原帖链接 · checklist · CC BY-SA 4.0
We are using Next.js for our website. A major part of its advantage is that it loads all linked pages on the fly as the user is viewing the page. It does so by making different calls to: /_next/xxx where xxx can be different JSON files or images.
My question is whether I should block the /_next directory so that I don't run into duplicate content issues or stale content (as these JSON files change for each deployment).
社区高分解答
原帖链接 · Brendan · CC BY-SA 4.0
I wrote an entire blog post on the topic:
Robots.txt - Block _Next Folder in Next.js Application
I tried to block the entire /_next/ directory in my robots.txt file to test out the idea.
I then ran into issues with the CSS not being able to be rendered by Google bot.
So I did the next best thing and disallowed files that were only used for User Experience (Metadata files):
User-agent: *
# Next.JS Crawl Budget Performance Updates
# Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
# The asterisks allows any file name
# The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Disallow: /*.json$
Disallow: /*_buildManifest.js$
Disallow: /*_middlewareManifest.js$
Disallow: /*_ssgManifest.js$
Disallow: /*.js$
We now save some additional Google Crawl Budget for the files we actually want Google to crawl :)
PS - If you are looking for a great explanation on blocking specific files in your robots.txt file, use this reference:
https://martijnoud.com/robots-txt/#Disallow_a_certain_file
EDIT: 3/13/22 - I decided to block all JS files, I originally was only blocking buildManifest, middlewareManifest, and ssgManifest.. But Google just really likes to check to make sure expired JS is still a 404. So just blocking all JS for now!
EDIT: 8/10/2022 - This is still working like a charm after 5 months of implementing this change. No crawl/indexing issues from Google and my crawl budget is being used much more efficiently.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
