Robots.txt: Technical SEO Guide to Crawl Control 

Have you ever searched for something on Google, found a website, and then wondered why some pages appear in search while other areas of the same website don’t? Behind the scenes, websites can give search engine crawlers instructions about which areas they should crawl. One of the main ways they do this is through robots.txt.

If you’re learning Technical SEO, robots.txt is something you need to understand properly, not just memorize. During an audit, you should be able to look at its rules, understand what they allow or block, identify any SEO risk, and recommend the right action.

What is Robots.txt?

Robots.txt is a simple text file that gives instructions to search engine crawlers about which parts of a website they can or cannot crawl. It is usually placed at the root of a website, for example:

A basic robots.txt file might look like this:

Here, User-agent: * applies the rule broadly to crawlers, while Disallow: /admin/ tells them not to crawl URLs under the /admin/ path. In simple terms, robots.txt is mainly used for crawl control, helping search engines understand which areas of a website they should or should not access.

What is the Purpose of Robots.txt?

The main purpose of robots.txt is to control crawling. It is not primarily an indexing control tool. This difference is very important in Technical SEO.

Think about the two separately:

Robots.txt

Crawling → Can the crawler access this URL?

Noindex

Indexing → Should this page be included in the search index?

So:

robots.txt ≠ noindex

A blocked URL is not automatically removed from Google’s index. This is one of the most common mistakes beginners make.

Where does Robots.txt live?

Robots.txt should normally be placed at the root of the host.

Correct:

Not:

There is another important point here: robots.txt works at the host level.

For example:

example.com/robots.txt

and

shop.example.com/robots.txt

belong to different hosts, so they can have different robots.txt files.

SEO takeaway

When auditing robots.txt, always check the correct hostname.

A rule on the main domain does not automatically mean the same rule applies to a subdomain.

Basic Robots.txt structure

A basic robots.txt file can contain several directives.

You do not need to memorize complicated syntax on Day 3. First, understand what each directive is trying to do.

User-agent

User-agent identifies the crawler that a group of rules applies to.

For example:

The * means the rules apply broadly to crawlers.

You can also create rules for a specific crawler:

Here, the rule is specifically aimed at Googlebot.

Interview-level explanation

User-agent defines which crawler a particular robots.txt rule group applies to.

That is the simple definition you should remember.

Disallow

Disallow tells a crawler not to crawl a particular path or URL pattern.

Example:

This can block crawling of URLs such as:

example.com/admin/

example.com/admin/users

example.com/admin/settings

The important part is understanding what the path represents.

If /admin/ contains non-SEO pages, blocking it may make sense.

If /products/ contains all your product pages, blocking it could create a serious SEO problem.

Disallow does NOT mean “remove from Google”

This is one of the most important things to understand about robots.txt.

Suppose your robots.txt contains:

A beginner might think:

Google will not index anything under /products/.

That’s not necessarily true.

Blocking crawling does not guarantee that a URL will never appear in Google’s index.

Google may discover a blocked URL through other sources, such as links pointing to it. If Google cannot crawl the URL, it may not be able to see the page content, but it can still know that the URL exists.

That is why robots.txt should not be treated as an indexing-removal tool.

Robots.txt vs Noindex

This is one of the most important Technical SEO concepts.

With robots.txt:

The instruction is essentially:

Do not crawl this URL or path.

With a noindex directive:

The instruction is:

Do not include this page in the search index.

Here is where things get important.

Imagine a page contains a noindex tag, but robots.txt blocks Google from crawling it:

robots.txt

      ↓

Blocks crawl

      ↓

Google cannot crawl the page

      ↓

Google may not see the noindex directive

So if your actual goal is index control, blocking the page in robots.txt can be the wrong approach.

Professional rule

Use robots.txt mainly for crawl control.

Use noindex when the goal is index control, while making sure search engines can actually crawl the page to see the directive.

To Read In Deep-Dive– Google Crawls Your Page Despite Your Noindex Tag: Why It Happens and How to Actually Stop It

Allow

Allow can be used to permit crawling of a path that would otherwise be blocked by a broader rule.

For example:

Conceptually, this means:

/products/

    ↓

Blocked

/products/featured/

    ↓

Allowed

This becomes useful when you want a broad rule but need to make an exception.

For complicated robots.txt patterns, don’t guess. Test the rules and confirm how the target crawler interprets them.

Wildcards

robots.txt can use pattern matching to target groups of URLs.

One common character is:

*

It generally works as a wildcard.

For example:

Disallow: /*?sort=

This can be used to target URLs containing a pattern such as:

/products/?sort=price

/products/?sort=rating

This can be useful on websites that generate large numbers of URL variations through parameters.

The important thing for SEO is not becoming a programming expert.

You need to understand which URLs the pattern is targeting and whether blocking them is actually safe.

$ — end-of-URL matching

The $ symbol can be used to indicate that a pattern should match the end of the URL.

For example:

Disallow: /*.pdf$

This is intended to match URLs that end in .pdf.

So this would match:

example.com/file.pdf

But:

example.com/file.pdf?download=1

does not end exactly with .pdf.

That’s where $ becomes important.

SEO takeaway

You don’t need to memorize every pattern immediately.

What matters is being able to look at a rule and understand what type of URLs it is trying to block.

Sitemap directive

Robots.txt can also tell crawlers where the XML sitemap is located.

For example:

This gives crawlers the sitemap location.

But remember, the sitemap and robots.txt have different jobs.

A sitemap says:

These URLs are important and available for discovery.

robots.txt says:

Do or do not crawl these paths.

A sitemap does not override a robots.txt block.

For example, if robots.txt says:

Disallow: /products/

and the sitemap contains:

Adding the URL to the sitemap does not force Google to crawl it.

Common Robots.txt mistake #1 — Blocking Important Pages

Imagine an e-commerce website has:

But every product page lives under /products/.

You have potentially blocked Google from crawling the entire product section.

That’s a major SEO issue.

During an audit, don’t just check whether the robots.txt file looks technically valid.

Ask:

Which important SEO URLs are being blocked?

Check pages such as:

• Product pages
• Category pages
• Blog pages
• Important landing pages
• Other pages that generate organic traffic

A technically valid robots.txt file can still be bad for SEO.

Common mistake #2 — Blocking CSS or JavaScript unnecessarily

You may come across old robots.txt files containing rules such as:

Disallow: /css/

Disallow: /js/

This can cause problems if those resources are needed for Google to properly render and understand important pages.

Remember the basic process:

Crawl → Render → Understand → Index

If important resources are blocked, search engines may have a harder time seeing the page as users see it.

That does not mean every CSS or JavaScript file must be crawlable in every situation.

The better question is:

Does Google need this resource to properly render or understand important pages?

Don’t block resources just because they look unnecessary from a quick technical check.

Common mistake #3 — Thinking robots.txt protects sensitive information

Robots.txt is not a security tool.

Suppose you have:

This does not make /secret/ private.

People can still try to visit:

example.com/secret/

They can also look at:

example.com/robots.txt

So never use robots.txt to protect passwords, confidential files, private information, or sensitive systems.

Security should be handled through proper access controls and authentication.

Common mistake #4 — Blocking URLs you actually want indexed

Imagine an SEO team says:

Our blog pages aren’t getting indexed.

You check robots.txt and find:

That’s immediately worth investigating. The problem is not necessarily that the robots.txt file is broken. The problem is that the rule may be blocking a section of the website that SEO depends on.

This is why robots.txt should always be reviewed against the website’s actual SEO URL inventory.

Common mistake #5 — Accidentally blocking the entire site

One of the most dangerous rules is:

This effectively blocks crawling across the site.

Think about what that includes:

A production website accidentally deploying this rule can create a serious SEO problem.

This is one of the first things worth checking when a website suddenly experiences major crawling or indexing problems.

Robots.txt and Crawl Budget

This is where robots.txt connects with larger Technical SEO concepts.

Imagine a large e-commerce website with millions of URLs.

Many of those URLs may be created by filters, sorting options, sessions, or other parameters:

/product/shoes?color=red

/product/shoes?color=blue

/product/shoes?sort=price

/product/shoes?sort=rating

Not every URL variation needs to be crawled.

If search engine crawlers spend a large amount of their resources crawling low-value URL variations, crawl efficiency can become an issue, especially on very large websites.

Robots.txt can sometimes be part of the solution. But don’t jump straight to blocking everything.

First understand:

What URLs are being crawled?

Why are they being crawled?

Are they useful for SEO?

Could blocking them create another problem?

That is the difference between simply editing robots.txt and making an informed Technical SEO recommendation.

A realistic example

Imagine an e-commerce website has this robots.txt:

At first glance, several of these rules make sense.

Pages such as:

/admin/

/cart/

/checkout/

/account/

usually aren’t intended to be organic landing pages.

But /search/ needs more investigation.

Suppose the website creates URLs like:

/search?q=shoes

/search?q=laptops

/search?q=red+shoes

Thousands of these URLs could potentially be created.

Blocking them may be part of a deliberate crawl-control strategy.

But a proper SEO audit would still ask how these URLs are being discovered, whether any are indexed, and whether internal links are creating unnecessary crawl paths.

That’s the real skill: understanding the SEO reason behind the rule instead of simply calling the rule good or bad.

Robots.txt Audit Process

When auditing robots.txt, follow a simple process.

Open the file

Go to:

First, see what is actually there.

Identify the user-agents

Check whether the file has general rules or specific rules for crawlers such as Googlebot.

Review Disallow rules

Look at every blocked directory and URL pattern.

Ask yourself:

What exactly is being blocked?

Compare the rules with important SEO URLs

Check important areas such as:

• Homepage
• Categories
• Products
• Blog
• Landing pages
• Important resources

Look for accidental blocking

Pay special attention to:

Disallow: /

and rules blocking major sections of the website.

Look for crawl-waste patterns

On large websites, investigate patterns involving URLs such as:

?sort=

?filter=

?session=

?tracking=

Don’t automatically block them. First understand how the website generates and uses them.

Check the sitemap

Make sure the sitemap location is correctly declared when appropriate.

Check the SEO intent

This is the most important part. Don’t write an audit recommendation like:

Robots.txt contains a Disallow rule.

That’s technically true but not very useful.

A better recommendation would be:

The /products/ directory is currently blocked for Googlebot. Because product pages are important organic landing pages, Google may be unable to crawl and properly process these URLs. Review and remove the block if these pages are intended to be crawled and indexed, then validate the affected URLs.

That’s the kind of recommendation that shows actual SEO understanding.

The Most Important Distinctions for Today

Keep these concepts clear in your head.

These concepts work together, but they are not interchangeable.

A common Technical SEO mistake is using one tool to solve a problem that belongs to another.

For example, using robots.txt when the actual goal is to remove a page from Google’s index.

We’ll get deeper into canonical tags and redirects later.

What Happens When You Have Rules for All Crawlers and a Specific Crawler?

This is where robots.txt can get a little confusing.

First, remember one important principle:

By default, URLs are allowed to be crawled unless a robots.txt rule tells a crawler not to crawl them.

Now imagine you have this robots.txt:

At first, you might think:

Googlebot is specifically allowed to crawl /products/, so does that mean the * rule no longer applies to Googlebot?

Not exactly.

The * group is a general rule, while the Googlebot group is a specific rule for Googlebot. When a crawler has its own matching User-agent group, that specific group is used rather than combining it with the wildcard group.

So for this example:

This applies to crawlers that don’t have a more specific group.

Then:

is specifically for Googlebot.

The important point is that you should not read these groups as being combined together for Googlebot.

A clearer example

Suppose your robots.txt is:

The intention might be:

Other crawlers

    ↓

/admin/ → blocked

Other URLs → allowed

Googlebot

    ↓

/products/ → allowed

But if your actual intention is:

Block /admin/ for everyone, including Googlebot, while allowing Googlebot to crawl /products/, you should write the Googlebot rules explicitly:

Now the instruction is much clearer:

Googlebot

├── /admin/     → Blocked

└── /products/  → Allowed

The simple rule to remember

Think of User-agent groups like this:

User-agent: * = general instructions

User-agent: Googlebot = specific instructions for Googlebot

And remember:

Specific crawler rules don’t automatically inherit the rules from the * group.

So when you create a specific Googlebot group, make sure it contains every restriction or permission you actually want Googlebot to follow.

This is why complicated robots.txt files should be tested carefully rather than interpreted by simply reading each line independently.

Interview Question You Should be Able to Answer

If an interviewer asks:

What’s the difference between robots.txt and noindex?

A weak answer would be:

Both tell Google not to index pages.

That’s incorrect.

A good answer is:

Robots.txt controls whether crawlers can crawl a URL, while noindex is an indexing directive that tells search engines not to include a page in their index.

A stronger professional answer would be:

robots.txt is mainly used for crawl control, while noindex is used for indexing control. If a URL is blocked in robots.txt, Google may not be able to crawl the page and therefore may not see a noindex directive on it. So robots.txt should not be treated as a replacement for noindex when the goal is index control.

That’s the level of understanding you should aim for.

Your Technical SEO mental model

Keep this basic flow in your head:

Once you understand that difference, robots.txt becomes much easier to audit.

The goal is not to memorize dozens of rules. The goal is to look at a robots.txt file and understand what it blocks, why it blocks it, and whether that decision is actually good for SEO.

Available next action: Create a downloadable DOCX file here in this chat containing the editable prose above.

Quick Doubts About robots.txt

Can I open a URL that is blocked in robots.txt?
Yes. robots.txt controls crawler access, not whether a person can directly open the URL in a browser.

What if the sitemap URL itself is blocked?
Avoid it. If /sitemap.xml is disallowed, crawlers may not be able to fetch the sitemap properly.

What if a URL is in the sitemap but blocked by robots.txt?
The sitemap can help Google discover the URL, but it does not override the Disallow rule. Google may know the URL exists but cannot crawl it.

Does User-agent: Googlebot block other crawlers?
No. It targets Googlebot specifically. Other crawlers are not affected by that particular rule.

Does User-agent: * block everything?
No. * only means the rules apply to all crawlers. A restriction happens only when you add something like Disallow.

Easy rule to remember:
User-agent = Who?
Disallow = What should they not crawl?