How to Create a Robots.txt File Without Blocking Important Pages

Blue plastic robot toy

The robots.txt file tells search engine crawlers which parts of your website they should not fetch. It is a small text file with a large effect: one wrong line can hide an entire site from search. This tutorial explains the syntax, gives you a safe starting file and shows how to test it.

What robots.txt does and does not do

  • It controls crawling. It asks crawlers not to request certain addresses.
  • It does not control indexing. A blocked page can still appear in results, without a description, if other sites link to it. To keep a page out of search results, use a noindex tag and leave the page crawlable so the tag can be read.
  • It is not security. The file is public and compliance is voluntary. Never use it to hide private content. Protect that with a password.

Where the file lives

The file must be named robots.txt and sit at the root of the host: https://www.example.com/robots.txt. Each subdomain needs its own. A file in a subfolder is ignored.

WordPress generates a virtual robots.txt when no physical file exists. SEO plugins usually provide an editor for it, which is the easiest way to make changes.

The syntax

A file consists of groups. Each group starts with a User-agent line naming the crawler it applies to, followed by rules.

  • User-agent: which crawler. An asterisk means all crawlers.
  • Disallow: a path that should not be crawled.
  • Allow: an exception inside a disallowed path.
  • Sitemap: the full address of your sitemap.

Paths are case-sensitive and matched from the start of the address. An asterisk matches any sequence of characters, and a dollar sign marks the end of an address.

A safe starting file

For most small websites, this is enough:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap_index.xml

An empty Disallow means nothing is blocked. The WordPress default is similar and also sensible:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

The one line to fear

User-agent: *
Disallow: /

This blocks the entire site. It is appropriate on a staging copy and disastrous on a live site. It is the first thing to check when a launched site does not appear in search.

Step 1: Look at what you have

Open yourdomain/robots.txt in a browser and read it. Note any Disallow lines and ask what each one is for. If nobody knows, it probably should not be there.

Step 2: Decide what, if anything, to block

Most small sites need to block very little. Reasonable candidates:

  • Internal search result pages
  • Cart, checkout and account pages in a shop
  • Endless filter or sorting combinations that create near-duplicate addresses

Do not block:

  • CSS, JavaScript or image folders. Search engines need them to render your pages.
  • Pages you want removed from results. Blocking stops the noindex tag from being seen.
  • Your whole uploads or theme directory.

Step 3: Edit the file

  1. Use your SEO plugin’s robots.txt editor, or create a plain text file named robots.txt and upload it to the site root.
  2. Add your rules and the Sitemap line.
  3. Save as plain text. Do not use a word processor.

Step 4: Test

  1. Reload yourdomain/robots.txt and confirm the new content is shown.
  2. In Google Search Console, open Settings and find the robots.txt report. It shows the version Google fetched and any errors.
  3. Use the URL inspection tool on an important page and confirm that crawling is allowed.
  4. Inspect a page you meant to block and confirm it is reported as blocked.

Example: a small online shop

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /?s=

Sitemap: https://www.example.com/sitemap_index.xml

Common mistakes

  • Leaving Disallow: / in place after moving from staging to live.
  • Blocking a page and expecting it to disappear from results.
  • Blocking scripts and styles, so pages cannot be rendered.
  • Using the wrong letter case in a path.
  • Placing the file in a subfolder.
  • Believing the file keeps anything private.

Next step

With crawling under control, make sure your pages are listed for discovery: How to Create and Submit an XML Sitemap. For a full review, use the Technical SEO Checklist for a New Website.