Grav/robots.txt

User-agent: *
Disallow: /.github/
Disallow: /.phan/
Disallow: /assets/
Disallow: /backup/
Disallow: /bin/
Disallow: /cache/
Disallow: /logs/
Disallow: /system/
Disallow: /tests/
Disallow: /tmp/
Disallow: /user/
Disallow: /vendor/
Disallow: /webserver-configs/
Allow: /user/pages/
Allow: /user/themes/
Allow: /user/images/
Allow: /
Allow: *.css$
Allow: *.js$
Allow: /system/*.js$
Added a default all-access robots.txt 2014-12-22 10:51:10 -07:00			`User-agent: *`
Update robots.txt reflecting actual folders (#3625) * Update robots.txt "grav" folder no longer exists * Add folders now existing 2023-03-06 18:21:03 +01:00			`Disallow: /.github/`
			`Disallow: /.phan/`
			`Disallow: /assets/`
Prevent crawling of unnecessary directories Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render. 2015-11-20 19:33:24 -07:00			`Disallow: /backup/`
			`Disallow: /bin/`
			`Disallow: /cache/`
			`Disallow: /logs/`
			`Disallow: /system/`
Update robots.txt reflecting actual folders (#3625) * Update robots.txt "grav" folder no longer exists * Add folders now existing 2023-03-06 18:21:03 +01:00			`Disallow: /tests/`
			`Disallow: /tmp/`
Prevent crawling of unnecessary directories Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render. 2015-11-20 19:33:24 -07:00			`Disallow: /user/`
Update robots.txt reflecting actual folders (#3625) * Update robots.txt "grav" folder no longer exists * Add folders now existing 2023-03-06 18:21:03 +01:00			`Disallow: /vendor/`
			`Disallow: /webserver-configs/`
Prevent crawling of unnecessary directories Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render. 2015-11-20 19:33:24 -07:00			`Allow: /user/pages/`
			`Allow: /user/themes/`
Update robots.txt (#2043) When using "Fetch and Render as Google" in Google Search console it will report "partial" rendering due to the blocked images in /user/images directory which is blocked because of the Disallow /user/ rule. Proposing this small change as it improves google rendering of the page. 2018-05-30 19:47:21 +02:00			`Allow: /user/images/`
Update robots.txt (#2632) I have found that Bing/Yahoo/DuckDuckGo, Yandex and Google report crawl errors when using the default robots.txt. Specifically their bots will not crawl the the path '/' or any sub-paths. I agree that the current robots.txt should work and properly implements the specification. However it still does not work. In my experience explicitly permitting the path '/' by adding the directive Allow: / resolves the issue. More details can be found in a blog post about the issue here: https://www.dfoley.ie/blog/starting-with-the-indieweb 2019-08-19 03:22:33 +10:00			`Allow: /`
Allow all css and js to be seached via robots.txt by default 2020-12-10 10:41:42 -07:00			`Allow: *.css$`
updated robots.txt 2021-01-04 13:41:32 -07:00			`Allow: *.js$`
Update robots.txt reflecting actual folders (#3625) * Update robots.txt "grav" folder no longer exists * Add folders now existing 2023-03-06 18:21:03 +01:00			`Allow: /system/*.js$`