Technical SEO Basics: Crawling, Indexing and Ranking Explained

Spider web in close up photography

Before a page can appear in search results, a search engine has to find it, read it and decide to keep it. Technical SEO is the work of making those steps possible and easy. This guide explains the three stages, crawling, indexing and ranking, and what a site owner can check at each one.

The three stages at a glance

  1. Crawling. A program called a crawler follows links and fetches pages.
  2. Indexing. The search engine analyses each fetched page and decides whether to store it in its index.
  3. Ranking. When someone searches, the engine selects pages from the index and orders them.

A page that is not crawled cannot be indexed. A page that is not indexed cannot rank. Problems are easier to solve when you know which stage is failing.

Crawling: can search engines reach your pages?

Crawlers discover pages mainly by following links, from other websites and from your own pages, and by reading sitemaps.

What helps crawling:

  • Internal links. Every important page should be reachable through links from other pages. A page nothing links to is called an orphan and is easily missed. See How to Audit Internal Links on a Small Website.
  • An XML sitemap. A file listing the pages you want found. Submit it in Search Console.
  • A reliable server. Pages that load slowly or return errors are crawled less.

What blocks crawling:

  • robots.txt. A file at the root of your site that tells crawlers which paths not to fetch. A single wrong line can block the entire site. Note that robots.txt controls crawling, not indexing. A blocked page can still appear in results without a description if other sites link to it.
  • Login walls and coming-soon modes. Crawlers cannot see what requires a password.
  • Broken links and long redirect chains. They waste the crawler’s time.

Very large sites need to think about how much a search engine is willing to crawl. For a small business site with a few hundred pages, this is rarely a concern.

Indexing: will the page be stored?

After fetching a page, the search engine processes its content, including what JavaScript adds, and decides whether to index it.

Signals you control:

  • The noindex tag. A robots meta tag with noindex tells search engines not to index a page. It is useful for thank-you pages and internal search results, and harmful when left on by accident after launch.
  • Canonical tags. When the same or very similar content is available at several addresses, a canonical tag names the preferred one. This consolidates signals instead of splitting them.
  • Status codes. A working page returns 200. A permanent move uses a 301 redirect. A missing page returns 404. A page that looks like an error but returns 200 confuses search engines.
  • Content quality. Search engines may choose not to index pages that are empty, duplicated or add little. Indexing is not guaranteed for every page you publish.

How to check: in Google Search Console, the Page indexing report lists indexed pages and the reason for each page that is not. The URL inspection tool shows the status of a single address. Our tutorial How to Set Up Google Search Console for a New Website covers both.

Ranking: where the page appears

Ranking is decided per search. Engines use many signals, and they do not publish a complete list or their weights. The areas that consistently matter are:

  • Relevance. Does the page answer what the searcher wants? Search Intent Explained shows how to match content to a query.
  • Quality and trust. Accurate, original, useful content from an identifiable source.
  • Links. Links from other reputable sites act as recommendations.
  • Page experience. Mobile-friendly layout, secure HTTPS delivery, reasonable loading speed and no intrusive pop-ups.
  • Context. Location, language and device of the searcher.

Technical SEO does not make weak content rank. It removes the obstacles that stop good content from being found.

The technical foundations

HTTPS

Serve every page securely and redirect http:// to https://. See HTTPS and SSL Certificates Explained.

Mobile-friendly pages

Google primarily uses the mobile version of a page for indexing and ranking. Content that is missing or hidden on mobile may not count.

Speed and Core Web Vitals

Core Web Vitals measure loading, responsiveness and visual stability as experienced by real visitors. Compressing images, limiting heavy scripts and using caching improve them.

Clean URLs and one address per page

Use short, readable addresses. Make sure www and non-www, and versions with and without a trailing slash, resolve to a single address.

Structured data

Structured data is a standard vocabulary added to a page’s code that describes what the page is about, such as a product, an article or a business. It can make a result eligible for enhanced display. It does not guarantee it.

Titles and descriptions

Each page needs a unique title. A good meta description does not affect ranking directly, but it influences whether people click. See How to Write a Meta Description That Matches the Page.

A basic technical check

  1. Open yourdomain/robots.txt and confirm it does not disallow the whole site.
  2. Open your sitemap and confirm it lists your real pages.
  3. View the source of a key page and search for noindex.
  4. Test http, https, www and non-www. All should end at one address.
  5. Load the site on a phone and try the menu and a form.
  6. Check the Page indexing report for unexpected exclusions.
  7. Click through the main pages looking for broken links.

Common problems on new sites

  • The option to discourage search engines is still switched on after launch.
  • A staging or test copy of the site is open to search engines and competes with the real one.
  • The same content is reachable at several addresses with no canonical tag.
  • Old addresses were changed without redirects.
  • Large uncompressed images slow every page.

Next step

Work through SEO for a New Website: A Practical Beginner’s Checklist, then plan which page should answer which search with Keyword Mapping.