问题原文
原帖链接 · Quillion · CC BY-SA 4.0
I have a project and I would like to disallow everything starting with root.
From what I understand I think I can do so by doing this
Disallow: /
Disallow: /*
However I would like to allow 4 subdirectories and everything under those subdirectories.
This is how I think it should be done
Allow: /directory_one/
Allow: /directory_one/*
Allow: /directory_two/
Allow: /directory_two/*
Allow: /directory_six/
Allow: /directory_six/*
Allow: /about/
Allow: /about/*
So how would I go about disallowing everything starting from root but allowing only those 4 directories and everything under them?
Also if I want to allow specific directory and everything under it, do I have to declare it twice?
Will webcrawler be able to navigate to those subdirectories if root is disallowed?
已采纳解答
原帖链接 · Stephen Ostermiller · CC BY-SA 4.0
I would not recommend trying to set up your site to disallow everything except certain directories.
- The home page of your site will be blocked from crawling. Because most sites get many links to their home page, your site will be throwing away SEO value from a lot of incoming links. You will need to have direct external links to your deep crawlable pages to make your SEO work.
- Futhermore, the home page is usually where bots find navigation to the other pages on your site. Without allowing crawling of the home page, bots may not be able to crawl all the pages of your site. You'd need to ensure that there are links between your deep crawlable pages.
- Even though your home page wouldn't be crawled, it would likely still be indexed. When somebody searches for your brand name, the Google search results would likely have link to your home page with the unfriendly message "A description for this result is not available because of the site's robots.txt -- learn more"
- Most crawlers don't know how to deal with
Allowdirectives. Most crawlers will be disallowed from crawling your entire site. Only a few crawlers will know they are allowed to crawl those subdirectories. Luckily the major search engine bots processAllowdirectives.
I would recommend moving all disallowed content into its own subdirectory and disallowing it. For example, put all your blocked content into /private/ and use the following in robots.txt:
User-Agent: *
Disallow: /private
If you are comfortable blocking most bots other than search engine crawlers and with having worse SEO because your home page is not crawled, your idea of using Allow for specific directories could work. There is no need to use the wildcard * at the end of any directive. All robots.txt rules without a wildcard are "starts with" rules. There is an implied * at the end of each and every one. All you would need would be:
User-Agent: *
Disallow: /
Allow: /directory_one
Allow: /directory_two
Allow: /directory_six
Allow: /about
The order of the rules shouldn't matter. When multiple rules match, the longest rule should apply. So:
/(the home page) matchesDisallow: /and crawling is not allowed/foomatchesDisallow: /and crawling is not allowed/about/bobmatches bothDisallow: /andAllow: /about. The longer rule will apply and crawling would be allowed.
You can test this with Google's robots.txt testing tool that is part of search console.
摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。
