danvics/debrid-services-comparison
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(robots.txt): remove disallow rules for cgi-bin, admin, and JSON files

Browse source
This commit is contained in:
fynks 2025-10-27 10:37:44 +05:00
parent 4eef89c2b0
commit 7e2566b493
1 changed files with 3 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
dist/robots.txt vendored
View file

@ -1,9 +1,6 @@
# robots.txt for debridcompare.xyz # robots.txt for debridcompare.xyz
User-agent: * User-agent: *
Allow: / Allow: /
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /*.json$
# Crawl-delay for respectful crawling # Crawl-delay for respectful crawling
Crawl-delay: 1 Crawl-delay: 1
@ -14,11 +11,11 @@ Sitemap: https://debridcompare.xyz/sitemap.xml
# Specific user agents # Specific user agents
User-agent: Googlebot User-agent: Googlebot
Allow: / Allow: /
Crawl-delay: 0 Crawl-delay: 1
User-agent: Bingbot User-agent: Bingbot
Allow: / Allow: /
Crawl-delay: 0 Crawl-delay: 1
User-agent: Slurp User-agent: Slurp
Allow: / Allow: /
@ -28,4 +25,4 @@ Allow: /
User-agent: Baiduspider User-agent: Baiduspider
Allow: / Allow: /
Crawl-delay: 2 Crawl-delay: 1