Home » Free Tools » Robots.txt Generator & Validator Tool

Robots.txt Generator & Validator Tool

by Akshay Goswami
Free Tool · WritoryBuzz

Build a valid robots.txt file visually, or paste an existing one to validate rules and test if a specific URL is blocked.

Site Settings

Blocking these keeps your content out of future model training. It does not affect whether AI chatbots can cite you today.
Blocking these removes your content from AI-generated answers right now. Most sites should leave these unchecked.

Rules

Your robots.txt

Upload to your site root as robots.txt 

        

      

    


    
    

      

        
Paste Your robots.txt

        

          

            
            
Paste your robots.txt content above, then test a URL below.

          

          

            

              
              
            

            

              
              
            

          

          
          


          

          
Syntax Check

          
          

        

      

    


    
    

      Technical SEO
      
What Is a Robots.txt File?

      
A robots.txt file is a plain-text file placed at the root of your domain that tells web crawlers which pages they can access and which to skip. It follows the Robots Exclusion Protocol and is read by Google, Bing, and most major AI crawlers before they begin fetching your site.

      
A correctly configured robots.txt file prevents crawl budget waste on low-value pages such as admin directories, internal search results, and duplicate pagination, while making sure your important content is crawled efficiently. In 2026 it also plays a second role: controlling which AI systems can access your content for training and for live citation.


      
AI Crawlers Are Not All the Same

      
Most major AI providers run two separate bots: a training crawler that feeds future model updates, and a search or retrieval crawler used at query time to answer a live question. Blocking the training bot does not stop the search bot, and blocking the search bot removes you from that engine's cited answers immediately.

      

        

          
User-agentOwnerPurposeBlocking it means

          
            
GPTBotOpenAITrainingOpt out of training only

            
OAI-SearchBotOpenAIChatGPT SearchLose ChatGPT citations

            
ChatGPT-UserOpenAIUser-triggered browsingLose ChatGPT citations

            
ClaudeBot / anthropic-aiAnthropicTrainingOpt out of training only

            
Claude-Web / Claude-SearchBotAnthropicClaude citationsLose Claude citations

            
PerplexityBotPerplexitySearch indexLose Perplexity citations

            
Google-ExtendedGoogleGemini training onlyNo effect on Search ranking

            
CCBotCommon CrawlDataset used by many modelsOpt out of training only

          
        

      

      
A common and reasonable posture in 2026 is to block the training-only bots (GPTBot, anthropic-ai, Google-Extended, CCBot) while leaving the search and citation bots (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot) allowed, so you stop feeding future models without losing today's AI search visibility.


      
A caveat on Perplexity: Cloudflare reported in August 2025 that Perplexity was running undeclared crawlers that rotate user agents, IPs, and ASNs to bypass robots.txt disallow rules. robots.txt is a voluntary convention, not enforcement, so if a crawler chooses not to honour it, blocking it there will not stop that specific bot. Server-level or firewall blocking is the only reliable control against crawlers that do not comply.


      
What to Block and What to Allow

      
Block admin directories such as /wp-admin/, internal search result pages, login and registration paths, staging or test directories, and duplicate parameter-based URLs. Never block your XML sitemap, CSS files, or JavaScript files that affect rendering, since Google needs them to render your pages correctly for indexing.

      
Always add your sitemap URL at the bottom of your robots.txt using the Sitemap: directive. This helps every crawler, including AI bots, find your sitemap immediately without a manual submission in Search Console.


      
The most damaging robots.txt mistake: a single line Disallow: / under User-agent: * removes your entire site from every search index and every AI citation source at once. Always validate your robots.txt with the tester above after any site change, and check Google Search Console for crawl errors afterward.


      
Common Robots.txt Mistakes

      
Blocking CSS and JavaScript. Google needs these files to render your pages correctly. Blocking them degrades Core Web Vitals assessment and can make pages appear broken in Search Console's rendering tool.

      
Missing the Sitemap directive. Without Sitemap: https://yourdomain.com/sitemap.xml in your robots.txt, crawlers must discover your sitemap through a manual submission or by following links.

      
Misspelled or outdated user-agent names. User-agent names in robots.txt are case-sensitive. Always use the exact, current string from the crawler's official documentation rather than a name copied from an old blog post, since these tokens do change over time.


      

      
Frequently Asked Questions

      

        

          
What is a robots.txt file?+

          
A robots.txt file is a plain-text file at your domain root that tells web crawlers which pages and directories they can access. It follows the Robots Exclusion Protocol and is read by Google, Bing, and AI crawlers before indexing or citation begins. A correctly configured file prevents crawl budget waste on low-value pages while ensuring your important content is crawled efficiently.

        

        

          
Does blocking GPTBot stop ChatGPT from citing my site?+

          
No. GPTBot is OpenAI's training crawler, so blocking it only opts you out of future model training. ChatGPT's live citations come from a separate crawler, OAI-SearchBot, along with ChatGPT-User for user-triggered browsing. To stop appearing in ChatGPT's cited answers you would need to block those crawlers specifically, which most sites should not do if they want AI search visibility.

        

        

          
Will robots.txt reliably block Perplexity?+

          
Not entirely. Cloudflare reported in August 2025 that Perplexity was operating undeclared crawlers that rotate user agents, IPs, and ASNs specifically to get around robots.txt disallow rules. Declaring PerplexityBot as disallowed still blocks the crawler that identifies itself honestly, but it is not a guaranteed defence against every Perplexity-operated bot. Server-level or firewall rules are the more reliable control if full exclusion matters to you.

        

        

          
What should I block in robots.txt?+

          
Block admin directories such as /wp-admin/, internal search result pages, login and registration paths, staging or test directories, and duplicate parameter-based URLs. Never block your XML sitemap, CSS files, or JavaScript files that affect rendering, since blocking critical rendering resources is one of the most damaging and common robots.txt mistakes.

        

        

          
How do I add robots.txt to WordPress?+

          
WordPress generates a virtual robots.txt file automatically. Edit it through an SEO plugin such as Yoast SEO or Rank Math in their settings panels. Alternatively, upload a physical robots.txt file to your public_html directory via FTP, which overrides the virtual version. Verify the file is accessible at https://yourdomain.com/robots.txt after uploading, and whitelist the filename in any security plugin that might block .txt files at the root.

        

        

          
Will a misconfigured robots.txt hurt my SEO?+

          
Yes, seriously. Blocking Googlebot from key pages removes them from search results entirely. Blocking CSS or JavaScript files prevents proper rendering, degrading Core Web Vitals scores. A common mistake during site migrations is accidentally leaving Disallow: / under User-agent: * in place after launch. Always validate your robots.txt using this free tool before uploading, and check Google Search Console for crawl errors afterward.

        

        

          
Where does the sitemap directive go in robots.txt?+

          
Add your sitemap URL anywhere in your robots.txt file using: Sitemap: https://yourdomain.com/sitemap.xml. This helps all crawlers find your sitemap immediately without manual submission. You can list multiple sitemaps if you maintain separate XML files for posts, pages, or images. The Sitemap directive does not interfere with Allow or Disallow rules and is supported by Google, Bing, and most major crawlers.

        

      

    


    
    

      More Tools
      
Complete Your Crawler Control Setup

      

        
          📜AI Searchllms.txt Generator
          
Guide AI crawlers to your best content. Pair with robots.txt for complete crawler control.

          Open Tool →
        
        
          🗺SEOXML Sitemap Generator
          
Declare your sitemap inside robots.txt for instant discovery by all crawlers.

          Open Tool →
        
        
          🧩Technical SEOSchema Markup Generator
          
Add structured data so search engines and AI systems understand your pages.

          Open Tool →
        
        
          🌐SEOHreflang Generator
          
Build multilingual SEO tags for international sites alongside your crawler rules.

          Open Tool →
        
        
          🎯AI SearchAEO Readiness Checker
          
Score your content against 15 AI citation signals to see if it is likely to be cited.

          Open Tool →
        
        
          🔍SEOGoogle SERP Simulator
          
Preview your title and meta description in Google results before you publish.

          Open Tool →