# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /scripts/ Disallow: /toolbox/ Disallow: /admin/ User-agent: BLEXBot Disallow:/ User-agent: dotbot Disallow:/ User-agent: AhrefsBot Disallow:/ User-agent: YandexBot Disallow:/ User-agent: Yandex Disallow:/ User-agent: Goodzer Disallow:/ User-agent: Baiduspider Disallow:/ User-agent: MJ12bot Disallow:/ User-agent: SemrushBot Disallow:/ User-agent: SemrushBot-SA Disallow:/ User-agent: ZoominfoBot Disallow:/