问题原文
原帖链接 · Behseini · CC BY-SA 4.0
I have created robots.txt in my root directory and when I navigate to my website address like https://example.ca/robots.txt I am able to see the rules
User-agent:*
Disallow: /captcha/
Disallow: /mailing/
Disallow: /assets/
Allow: /
Sitemap: https://domain.ca/sitemap.xml
Sitemap: http://domain.ca/sitemap.xml
But when I want to test the file in Google robots.txt Tester
I am getting this error
I am however able to test the file at bottom of the page or see the live robots.txt
but still getting 404 error when entering to the page!
已采纳解答
原帖链接 · Quitxotic · CC BY-SA 4.0
Can take up to 24 hours, the robot tester seems to a bit flakey after just putting your file up. Good thing at least is that you're not getting a BLOCKED response from the tester.
https://developers.google.com/search/docs/advanced/robots/submit-updated-robots-txt
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
