Merge pull request #60 from NeuromatchAcademy/robots.txt-update

Update robots.txt
NeuromatchAcademy · Aug 25, 2024 · d416ae6 · d416ae6
2 parents da5a3cd + 1f11423
commit d416ae6
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/public/robots.txt b/public/robots.txt
@@ -140,6 +140,12 @@ Disallow: /
 User-agent: Timpibot
 Disallow: /
 
+# AI Data Scraper
+# https://darkvisitors.com/agents/webzio-extended
+
+User-agent: Webzio-Extended
+Disallow: /
+
 # AI Search Crawler
 # https://darkvisitors.com/agents/youbot