danvics/debrid-services-comparison
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(robots.txt): remove disallow rules for cgi-bin, admin, and JSON files

Browse source
This commit is contained in:
fynks 2025-10-27 10:37:44 +05:00
parent 4eef89c2b0
commit 7e2566b493
1 changed files with 3 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
dist/robots.txt vendored
View file

@ -1,9 +1,6 @@
# robots.txt for debridcompare.xyz
User-agent: *
Allow: /
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /*.json$
# Crawl-delay for respectful crawling
Crawl-delay: 1
@ -14,11 +11,11 @@ Sitemap: https://debridcompare.xyz/sitemap.xml
# Specific user agents
User-agent: Googlebot
Allow: /
Crawl-delay: 0
Crawl-delay: 1
User-agent: Bingbot
Allow: /
Crawl-delay: 0
Crawl-delay: 1
User-agent: Slurp
Allow: /
@ -28,4 +25,4 @@ Allow: /
User-agent: Baiduspider
Allow: /
Crawl-delay: 2
Crawl-delay: 1