Your robots.txt file controls which parts of your WordPress site search engine crawlers can access. Used correctly, it helps you manage crawl budget and keep low-value admin or technical URLs out of the crawling process.
In this guide, you’ll learn what robots.txt does, what to block and allow on a WordPress site, see safe robots.txt examples, learn how to add the file with Yoast SEO, and avoid mistakes that can accidentally block important pages from being crawled.
📋 Table of Contents
What Is Robots.txt?
A robots.txt file is a text file that tells search engine crawlers which URLs they can access on your website. It is placed at the root of your domain and must be named exactly robots.txt.
For RoyalBackup, the robots.txt file would be located at:
https://royalbackup.com/robots.txt
The file uses simple rules. User-agent identifies the crawler, while Allow and Disallow define which URL paths may or may not be crawled.
User-agent: *Disallow: /wp-admin/In this example, the asterisk means the rule applies to all crawlers, and the rule asks them not to crawl URLs beginning with /wp-admin/.
Robots.txt vs Noindex
A common SEO mistake is using robots.txt to remove a page from Google. Robots.txt controls crawling, while noindex is used to ask search engines not to index a page. Google specifically states that robots.txt is not a mechanism for keeping a page out of Google.
| Feature | Robots.txt | Noindex |
|---|---|---|
| Main purpose | Controls crawling | Requests removal from the index |
| Where it is used | robots.txt file | Page-level meta robots tag or X-Robots-Tag |
| Can Google crawl the page? | No, if disallowed | Yes, unless also blocked by robots.txt |
| Can the URL still appear in Google? | Yes, if externally linked | Usually no, after Google processes it |
| Best use | Admin paths, technical URLs, crawl control | Thank-you pages, internal search results, private pages |
⚠️ Important
Do not use robots.txt to hide private content. Use password protection, user permissions, or proper authentication instead.
Where Is Robots.txt Located?
The robots.txt file must be in the root directory of your domain. You can usually view it by adding /robots.txt to your domain.
Correct location:
https://royalbackup.com/robots.txt
Incorrect location:
https://royalbackup.com/wp-content/robots.txt
What to Block in WordPress Robots.txt
For most WordPress sites, block only technical or low-value URLs that do not need to be crawled. Do not block pages, posts, categories, images, CSS, JavaScript, or your XML sitemap.
1. WordPress admin login area
You can block the WordPress admin area, but keep admin-ajax.php accessible because some themes and plugins may rely on it.
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.php2. WordPress core technical folders
Some WordPress installations include technical folders that do not need to be crawled. Only block these if they exist on your site and do not contain assets needed to render pages.
Disallow: /wp-includes/Disallow: /cgi-bin/Use caution: blocking CSS or JavaScript files can prevent Google from rendering pages correctly.
3. Internal WordPress search results
WordPress search URLs can create many low-value crawlable pages. You can block the search parameter path:
Disallow: /?s=For stronger control, also use a noindex rule on internal search results where appropriate.
4. Private or staging environments
For a staging site, you may temporarily block all crawling:
User-agent: *Disallow: /Never leave this rule on a live production website. It can prevent Google from crawling your entire site.
What to Allow in WordPress Robots.txt
Allow crawlers to access the content and resources they need to understand and render your pages. This includes posts, pages, categories, images, CSS, JavaScript, and your XML sitemap.
1. Allow your main content
Do not block your blog posts, pages, categories, or important landing pages. If you want to explicitly allow the whole site, use an empty Disallow rule:
User-agent: *Disallow:Google explains that an empty Disallow rule allows crawling of the entire site.
2. Allow WordPress uploads
Do not block your uploads folder if it contains images, PDFs, or other media used in your content.
Allow: /wp-content/uploads/3. Allow CSS and JavaScript
Google may need CSS and JavaScript files to render pages correctly. Avoid broad rules that block /wp-content/ or /wp-includes/ if they contain required assets.
Allow: /*.css$Allow: /*.js$4. Add your XML sitemap
Add your XML sitemap to robots.txt using its full URL. Google says the sitemap directive must use a fully qualified URL.
Sitemap: https://royalbackup.com/sitemap_index.xmlIf you use Yoast SEO, your sitemap is commonly located at /sitemap_index.xml.
WordPress Robots.txt Example
Here is a safe starting robots.txt example for a typical WordPress website. Review it before using it, because your hosting setup, plugins, and site structure may require changes.
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpDisallow: /?s=Disallow: /search/Sitemap: https://royalbackup.com/sitemap_index.xml⚠️ Before using this example
Only add Disallow: /search/ if your WordPress search results actually use that URL structure. Do not copy rules blindly; test your site after saving the file.
How to Add Robots.txt With Yoast
Yoast SEO lets you create or edit robots.txt directly from WordPress. Yoast’s File Editor can edit both robots.txt and, if present, .htaccess.
✅ Steps:
- Go to Yoast SEO → Tools → File Editor.
- Click Create robots.txt file if one does not already exist.
- Paste your robots.txt rules into the editor.
- Click Save changes to robots.txt.
- Visit
https://royalbackup.com/robots.txtto confirm the file is live. - Test important URLs in Google Search Console.
⚠️ Backup first
Before editing robots.txt, create a recent WordPress backup. A single incorrect rule can block important pages or resources from being crawled.
How to Test Robots.txt
1. Check the live file
Open https://royalbackup.com/robots.txt in your browser. Confirm the rules and sitemap URL appear exactly as intended.
2. Use Google Search Console
In Google Search Console, use the URL Inspection tool to check whether Google can crawl important URLs. Also review the Pages report for indexing issues.
3. Test after every change
After changing robots.txt, test your homepage, key blog posts, product pages, images, and XML sitemap. This helps you catch accidental blocks quickly.
Common Robots.txt Mistakes
1. Blocking your entire website
The rule Disallow: / blocks crawling of the whole site. Never use it on a live website unless the site is private or staging.
2. Blocking CSS and JavaScript
Blocking required CSS or JavaScript can prevent Google from rendering your pages properly. Avoid broad blocks on /wp-content/ or /wp-includes/.
3. Using robots.txt to remove a page from Google
Robots.txt blocks crawling, not indexing. To request that a page not be indexed, use a noindex directive or password-protect the page.
4. Blocking your XML sitemap
Do not block your sitemap. Add it using a full URL, such as Sitemap: https://royalbackup.com/sitemap_index.xml.
5. Copying rules from another website
Another site’s robots.txt may block paths that are important on your WordPress installation. Always adapt rules to your own site structure.
Robots.txt for WordPress FAQ
Frequently Asked Questions
Q: Does every WordPress site need a robots.txt file?
A: No. A WordPress site can work without one, but a correctly configured robots.txt file gives you more control over crawling.
Q: Should I block /wp-admin/ in robots.txt?
A: You can block /wp-admin/, but allow /wp-admin/admin-ajax.php because some WordPress features may depend on it.
Q: Does robots.txt improve SEO rankings?
A: Robots.txt does not directly improve rankings. It helps control crawling and can prevent crawlers from wasting time on low-value URLs.
Q: Can robots.txt remove a page from Google?
A: No. Robots.txt controls crawling, not indexing. Use a noindex directive or password protection if you want a page excluded from Google’s index.
Q: Where should I add my XML sitemap?
A: Add it at the end of robots.txt using a full URL, for example: Sitemap: https://royalbackup.com/sitemap_index.xml.
Robots.txt Optimization Checklist
- Place robots.txt at the root of your domain
- Block only technical or low-value URLs
- Allow important posts, pages, categories, images, CSS, and JavaScript
- Allow
/wp-admin/admin-ajax.phpif you block/wp-admin/ - Add your XML sitemap using a full URL
- Do not use robots.txt to remove pages from Google
- Never leave
Disallow: /on a live website - Back up your site before editing robots.txt
- Test important URLs after every change
A safe robots.txt file blocks only what does not need to be crawled and allows Google to access the content and resources it needs. Use the examples above as a starting point, test your setup, and keep your XML sitemap accessible.
🚀 Related WordPress SEO Resources
Enjoy this post?
