Prevent crawling of unnecessary directories

Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render.

robots.txt

@@ -1,2 +1,11 @@
 User-agent: *
-Disallow:
+Disallow: /backup/
+Disallow: /bin/
+Disallow: /cache/
+Disallow: /grav/
+Disallow: /logs/
+Disallow: /system/
+Disallow: /vendor/
+Disallow: /user/
+Allow: /user/pages/
+Allow: /user/themes/