Grav/robots.txt

User-agent: *
Disallow: /backup/
Disallow: /bin/
Disallow: /cache/
Disallow: /grav/
Disallow: /logs/
Disallow: /system/
Disallow: /vendor/
Disallow: /user/
Allow: /user/pages/
Allow: /user/themes/
Added a default all-access robots.txt 2014-12-22 10:51:10 -07:00			`User-agent: *`
Prevent crawling of unnecessary directories Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render. 2015-11-20 19:33:24 -07:00			`Disallow: /backup/`
			`Disallow: /bin/`
			`Disallow: /cache/`
			`Disallow: /grav/`
			`Disallow: /logs/`
			`Disallow: /system/`
			`Disallow: /vendor/`
			`Disallow: /user/`
			`Allow: /user/pages/`
			`Allow: /user/themes/`