The robots.txt file tells search engine crawlers which parts of your website they should not fetch. It is a small text file with a large effect: one wrong line can hide an entire site from search. This tutorial explains the syntax, gives you a safe starting file and shows how to test it.
What robots.txt does and does not do
- It controls crawling. It asks crawlers not to request certain addresses.
- It does not control indexing. A blocked page can still appear in results, without a description, if other sites link to it. To keep a page out of search results, use a noindex tag and leave the page crawlable so the tag can be read.
- It is not security. The file is public and compliance is voluntary. Never use it to hide private content. Protect that with a password.
Where the file lives
The file must be named robots.txt and sit at the root of the host: https://www.example.com/robots.txt. Each subdomain needs its own. A file in a subfolder is ignored.
WordPress generates a virtual robots.txt when no physical file exists. SEO plugins usually provide an editor for it, which is the easiest way to make changes.
The syntax
A file consists of groups. Each group starts with a User-agent line naming the crawler it applies to, followed by rules.
- User-agent: which crawler. An asterisk means all crawlers.
- Disallow: a path that should not be crawled.
- Allow: an exception inside a disallowed path.
- Sitemap: the full address of your sitemap.
Paths are case-sensitive and matched from the start of the address. An asterisk matches any sequence of characters, and a dollar sign marks the end of an address.
A safe starting file
For most small websites, this is enough:
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap_index.xml
An empty Disallow means nothing is blocked. The WordPress default is similar and also sensible:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xml
The one line to fear
User-agent: *
Disallow: /
This blocks the entire site. It is appropriate on a staging copy and disastrous on a live site. It is the first thing to check when a launched site does not appear in search.
Step 1: Look at what you have
Open yourdomain/robots.txt in a browser and read it. Note any Disallow lines and ask what each one is for. If nobody knows, it probably should not be there.
Step 2: Decide what, if anything, to block
Most small sites need to block very little. Reasonable candidates:
- Internal search result pages
- Cart, checkout and account pages in a shop
- Endless filter or sorting combinations that create near-duplicate addresses
Do not block:
- CSS, JavaScript or image folders. Search engines need them to render your pages.
- Pages you want removed from results. Blocking stops the noindex tag from being seen.
- Your whole uploads or theme directory.
Step 3: Edit the file
- Use your SEO plugin’s robots.txt editor, or create a plain text file named robots.txt and upload it to the site root.
- Add your rules and the Sitemap line.
- Save as plain text. Do not use a word processor.
Step 4: Test
- Reload yourdomain/robots.txt and confirm the new content is shown.
- In Google Search Console, open Settings and find the robots.txt report. It shows the version Google fetched and any errors.
- Use the URL inspection tool on an important page and confirm that crawling is allowed.
- Inspect a page you meant to block and confirm it is reported as blocked.
Example: a small online shop
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /?s=
Sitemap: https://www.example.com/sitemap_index.xml
Common mistakes
- Leaving Disallow: / in place after moving from staging to live.
- Blocking a page and expecting it to disappear from results.
- Blocking scripts and styles, so pages cannot be rendered.
- Using the wrong letter case in a path.
- Placing the file in a subfolder.
- Believing the file keeps anything private.
Next step
With crawling under control, make sure your pages are listed for discovery: How to Create and Submit an XML Sitemap. For a full review, use the Technical SEO Checklist for a New Website.
