Logo
Explore Help
Register Sign In
Nemcio/Grav
1
0
Fork 0
You've already forked Grav
mirror of https://github.com/getgrav/grav.git synced 2025-10-26 07:56:07 +01:00
Code Issues Packages Projects Releases Activity
Files
20b9ca56fa80fc0000801ea6a0534a154b8e0b46
Grav/robots.txt

13 lines
218 B
Plaintext
Raw Normal View History

Added a default all-access robots.txt
2014-12-22 10:51:10 -07:00
User-agent: *
Prevent crawling of unnecessary directories Prevent search engines from crawling and indexing unnecessary files and directories. The "/user/plugins/" directory may need to be added to the Allow list if plugins use frontend accessible assets. This is tested at a basic level using Google Fetch and Render.
2015-11-20 19:33:24 -07:00
Disallow: /backup/
Disallow: /bin/
Disallow: /cache/
Disallow: /grav/
Disallow: /logs/
Disallow: /system/
Disallow: /vendor/
Disallow: /user/
Allow: /user/pages/
Allow: /user/themes/
Update robots.txt (#2043) When using "Fetch and Render as Google" in Google Search console it will report "partial" rendering due to the blocked images in /user/images directory which is blocked because of the Disallow /user/ rule. Proposing this small change as it improves google rendering of the page.
2018-05-30 19:47:21 +02:00
Allow: /user/images/
Reference in New Issue Copy Permalink
Powered by Gitea Version: 1.24.7 Page: 56ms Template: 1ms
English
Bahasa Indonesia Deutsch English Español Français Gaeilge Italiano Latviešu Magyar nyelv Nederlands Polski Português de Portugal Português do Brasil Suomi Svenska Türkçe Čeština Ελληνικά Български Русский Українська فارسی മലയാളം 日本語 简体中文 繁體中文(台灣) 繁體中文(香港) 한국어
Licenses API