AI Permissions File for Boostlingo.com
AI and LLM Crawler Policies
Allow AI/LLM providers to crawl publicly available content
User-Agent: GPTBot
Allow: /
User-Agent: Googlebot
Allow: /
User-Agent: ChatGPT-User
Allow: /
User-Agent: ClaudeBot
Allow: /
User-Agent: Gemini
Allow: /
User-Agent: anthropic-ai
Allow: /
User-Agent: PerplexityBot
Allow: /
Disallow crawling of private or sensitive pages
Disallow: /login
Disallow: /account
Disallow: /wp-admin/
Disallow: /privacy-policy
Disallow: /terms-of-service
Request for AI training usage
Training-Data-Use: opt-in
Comment: Boostlingo consents to the use of publicly available pages for AI model training, excluding user-generated or customer-specific content.
Rate Limiting
Crawl-Delay: 10