Merge branch 'block_ai_scrappers' of https://github.com/ubyssey/ubyss…

…ey.ca into develop
ubyssey · May 22, 2024 · 6104d34 · 6104d34
2 parents d207e8a + 2f661c8
commit 6104d34
Showing 1 changed file with 8 additions and 3 deletions.
diff --git a/robots.txt b/robots.txt
@@ -1,3 +1,8 @@
-User-agent: *
-Disallow: /api/
-Disallow: /admin/
+User-agent: Amazonbot
+User-agent: anthropic-ai
+User-agent: AwarioSmartBot
+User-agent: ChatGPT-User
+User-agent: ClaudeBot
+User-agent: Meltwater
+User-agent: Bytespider
+Disallow: /