Guides 9 min read

How to Create Robots.txt for WordPress (Beginner Guide)

Every WordPress site gets visited by dozens of automated crawlers every day — search engines like Google and Bing, SEO tools like Ahrefs and Semrush, and a new generation of AI bots such as OpenAI’s GPTBot, Anthropic’s ClaudeBot, and PerplexityBot. The single file that tells all of them what they may and may not fetch is robots.txt. In this guide you will learn how to create robots.txt for WordPress the right way: what the file does, which rules matter in 2026, and how to generate a valid file in under two minutes.

What Is a Robots.txt File?

Robots.txt is a plain text file that lives in the root directory of your website. Before almost any well-behaved crawler requests a single page, it first requests https://yoursite.com/robots.txt and reads the instructions inside. Those instructions follow a standard called the Robots Exclusion Protocol, which is understood by Google, Bing, DuckDuckGo, Yandex, and the major AI vendors.

A minimal WordPress robots.txt looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Translated into plain English: “For every user-agent (*), do not crawl anything under /wp-admin/ — except /wp-admin/admin-ajax.php, which many WordPress themes and plugins need to function.”

Each block of rules starts with a User-agent: line followed by one or more Disallow: or Allow: lines. You can have as many blocks as you like, and a file can also declare your Sitemap: location so crawlers can discover your content faster. Comments start with the # character and are ignored by crawlers.

Why Robots.txt Matters for WordPress Sites

It protects sensitive areas. Your WordPress dashboard, login page, and internal search results rarely belong in a search index. A few well-chosen Disallow rules keep crawlers out of them.

It improves crawl efficiency. Google allocates each site a finite crawl budget. If a bot wastes that budget crawling faceted search results, tag archives, or reply-to-comment links, your important pages get discovered and refreshed more slowly.

It prevents duplicate content. WordPress can generate many URLs for the same content — for example, /?s=keyword search pages or URLs with ?replytocom parameters. Blocking these patterns in robots.txt keeps Google from crawling those duplicate URLs (and if you ever need them out of the index entirely, a canonical tag or noindex directive is the right tool).

It decides how AI models treat your content. This is the newest and often most strategic reason. OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, and Apple all operate crawlers that either train models on your pages or cite them inside AI answers. Whether you want that visibility (Generative Engine Optimization, or GEO) or want to opt out, robots.txt is where the choice is recorded: if you never write rules for a specific bot, crawlers like GPTBot simply follow your general User-agent: * rules — only an explicit, bot-specific declaration lets you control an individual crawler differently.

One honest caveat: robots.txt is a courtesy protocol, not a security mechanism. Legitimate companies honor it; malicious scrapers ignore it completely. Never use robots.txt as your only line of defense for genuinely private data — use authentication for that.

How to Create Robots.txt for WordPress, Step by Step

Step 1: Check whether you already have a robots.txt file

Open your browser and visit https://yoursite.com/robots.txt. One of three things will happen. If you see rules, a physical robots.txt file already exists in your site root. If you see a simple three-line file that blocks /wp-admin/ while allowing /wp-admin/admin-ajax.php, WordPress is generating a virtual robots.txt on the fly — this happens automatically whenever pretty permalinks are enabled and no physical file exists. If you get a 404, there is no file yet and you should create one.

Step 2: Decide your policy for search engines and AI crawlers

Before writing rules, decide what you actually want. There are three common strategies. A WordPress standard policy keeps the site open to all search engines while blocking the dashboard, login page, and search results. A GEO / AI-friendly policy does the same but explicitly allows AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended, signalling that AI engines may train on and cite your content — attractive for publishers who want traffic from ChatGPT, Claude, Perplexity, and AI Overviews. A strict protection policy allows only the major search engine bots and blocks AI, training, and SEO crawlers outright.

Step 3: Generate the rules with a free Robots.txt Generator

The fastest way to produce a correct file is to use the free Robots.txt Generator. It ships with all three presets described above — WordPress Standard, GEO / AI-Friendly, and Strict Protection — so the hard thinking is already done for you. Select a preset, enter your sitemap URL (for most WordPress sites running an SEO plugin it is https://yoursite.com/sitemap_index.xml), and the preview updates live. You can copy the result or download it as robots.txt directly.

Step 4: Add custom rules for specific bots

Preset not enough? The generator lets you add custom user-agent groups, each with its own multi-line Disallow and Allow lists — for example, a group that blocks AhrefsBot and SemrushBot to save server resources, or one that allows Googlebot but nothing else. It also includes a reference of common crawlers with a one-line description of what each one does, plus one-click Allow and Block buttons. If you prefer to write rules by hand, remember the essentials: rules are case-sensitive, a bare Disallow: with no value means “allow everything”, and Disallow: / means “block the entire site” — a rule you should only ever publish on purpose.

Step 5: Upload robots.txt to your site root

Robots.txt must live in the document root of your domain and must be named exactly robots.txt in lowercase. For a standard WordPress install that is the same folder that contains wp-config.php and the wp-content directory. You can upload the downloaded file with your hosting provider’s file manager, over SFTP, or with WP-CLI: wp eval 'file_put_contents(ABSPATH."robots.txt", file_get_contents("robots.txt"));'. Some SEO plugins such as Yoast SEO or Rank Math can also host a virtual robots.txt for you in their settings — useful if you manage everything inside WordPress.

Step 6: Validate the file and monitor it

After uploading, fetch https://yoursite.com/robots.txt in a private browser window to confirm the new version is served. Then check the robots.txt report in Google Search Console (in the Settings section), which shows when Google last fetched your file and its status. Note that the old robots.txt Tester — where you typed in a URL, picked a user-agent, and tested it — was retired in September 2023, so to test your rules, use a third-party validator or compare your file against the vetted presets in our Robots.txt Generator as a reference template. If a critical page suddenly disappears from search, a typo in robots.txt is one of the first things to rule out. Revalidate after every deploy, since a stray Disallow: / can quietly de-index an entire site.

Robots.txt Best Practices for WordPress

A small, correct robots.txt beats a large, clever one. These rules of thumb keep you out of trouble:

  • Never block CSS or JavaScript files. Google renders pages with a full browser; if it cannot load your theme’s assets, rankings can drop even though the pages are technically “allowed”.
  • Keep the file small. Most sites need fewer than 20 lines. Every extra rule is another chance for a mistake.
  • Always declare your sitemap. A Sitemap: line helps new crawlers discover posts and pages faster, especially on fresh sites with few backlinks.
  • Decide your AI policy explicitly. Silent defaults are being interpreted unpredictably across vendors. Whether you allow or block GPTBot, ClaudeBot, and friends, write it down — the Robots.txt Generator presets make this a one-click decision.
  • Do not rely on robots.txt for privacy. Blocking a URL in robots.txt leaves the URL itself crawlable by rogue bots and may even let it appear in search results without a description. Private content needs a login, not a Disallow rule.
  • Review quarterly. New bots launch constantly and your site structure evolves. Re-check the file every few months, and after any migration or redesign.

Frequently Asked Questions

Where does the robots.txt file go in WordPress?

The file belongs in the document root of your domain — the same directory that holds wp-config.php, usually named public_html or www on shared hosts. It must be reachable at exactly https://yoursite.com/robots.txt. If it is nested in a subfolder or named with uppercase letters, crawlers will ignore it.

Does WordPress create a robots.txt file automatically?

Yes. When pretty permalinks are enabled and no physical robots.txt exists, WordPress generates a virtual one on the fly that blocks /wp-admin/ but allows /wp-admin/admin-ajax.php. It is intentionally minimal. To add sitemap declarations, AI-crawler rules, or custom Disallow paths, create a real file (or use an SEO plugin) and upload it to your site root.

How do I block AI crawlers like GPTBot in robots.txt?

Add a dedicated group for each bot you want to stop, with a full-site disallow, and place these groups before your default User-agent: * block. For example: User-agent: GPTBot followed by Disallow: / opts you out of OpenAI’s training data. Repeat for ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, and meta-externalagent — or use the Strict Protection preset in the free Robots.txt Generator, which emits all of these blocks for you.

Will robots.txt remove a page from Google search results?

No. Blocking a URL with robots.txt prevents crawling, not indexing. Google can still list the URL if other sites link to it — it will simply show without a snippet or description. To remove a page from search entirely, use a noindex meta tag or an X-Robots-Tag HTTP header instead, and let crawlers reach the page so they can see the directive.

Can a bad robots.txt file hurt my WordPress SEO?

Absolutely. A single Disallow: / under User-agent: * blocks the entire site from every compliant crawler, and blocking /wp-content/ or /wp-includes/ can break Google’s rendering of your pages. Because robots.txt changes take effect immediately and are cached by search engines, always test the file — with Search Console’s robots.txt report or by generating it from vetted presets — before you upload it.

Start in Two Minutes

You now know what robots.txt does, why it matters for both classic SEO and AI visibility, and how to create, upload, and validate one for WordPress. Head over to the Robots.txt Generator, pick the preset that matches your strategy, add your sitemap, and download a production-ready file — your future crawler traffic (and your AI-era visibility) will thank you.