# Robots.txt for The Appliance Directory User-agent: * Allow: / # Block admin areas Disallow: /admin/ # Block parameterized AI search results (duplicate / infinite variations) # Disallow: /search?* Sitemap: https://www.theappliancedirectory.com/sitemap.xml # Block aggressive scrapers to reduce server load User-agent: SemrushBot Disallow: / User-agent: DotBot Disallow: / User-agent: MJ12bot Disallow: /