Key Takeaways
- Robots.txt is a small text file that tells crawlers where they can go and where they shouldn’t bother.
- A single stray slash can hide your whole site from Google, and many owners only notice weeks later.
- It controls crawling and not indexing, so a page you truly want hidden needs a noindex tag instead.
- Put the file in your root folder, link your sitemap inside it, and test before anything goes live.
Picture a stranger walking into your shop who first stops to read a small sign at the door. That sign is robots.txt for your website, and every search engine bot reads it before anything else. Here’s how it works, how to build one, and what to do when something quietly breaks on you.
What is Robots.txt?
Robots.txt is a plain text file that lives on your server and hands crawlers a few basic rules. Search engines grab it first, long before they bother looking at even a single page of yours. Google and Bing respect it, though shady bots often don’t, so never rely on it to hide private files.
What Robots.txt Looks Like on a Live Website?
It looks harder than it is, since you only name a bot and then list what it can’t touch. Every rule gets its own line, and a blank line splits one bot group from the next one.
Sitemap: https://www.example.com/sitemap.xml
User-agent: *
Disallow: /admin/
Allow: /admin/help.html
Here, every bot is told to skip the admin folder, except for one public help page inside it.
Robots.txt user-agents Explained for Beginners
A user-agent is just the name a crawler gives your server when it knocks on the door. Use the asterisk when you want one set of rules to cover every single bot at once.
- Googlebot for Google web search
- Googlebot-Image for Google Images
- Bingbot for Microsoft Bing
- DuckDuckBot for DuckDuckGo
- Baiduspider for Baidu
Copy each name exactly as published, since a bot obeys only the group that fits it best.
Robots.txt Directives and What Each One Does?
Directives are the actual instructions inside each group, and Google only understands a few of them these days. This table shows what each one does and whether Google still pays any attention to it in 2026.
| Directive | What It Does | Google Support |
| Disallow | Blocks a path from being crawled | Yes |
| Allow | Permits a page inside a blocked folder | Yes |
| Sitemap | Points bots to your XML sitemap | Yes |
| Crawl-delay | Slows down the crawl speed | No, but Bing supports it |
| Noindex | Tries to hide a page from results | No |
Noindex and nofollow lines stopped working for Google back in 2019, so use meta robots tags for those jobs.
Why Does Robots.txt SEO Matter for Crawl Budget?
Google won’t crawl your site forever, so every visit wasted on junk pages is one your good pages lose. Smart robots.txt SEO steers bots away from filters, internal search results, and duplicate pages nobody needs to see. Site structure matters here too, and solid topic clusters point crawlers to your best pages much faster.
How to Find Robots.txt on a Website?
Finding it takes about ten seconds, since every robots.txt sits right at the root of its domain. Open a browser, type your domain, and tack /robots.txt onto the end, like example.com/robots.txt for a quick check. Seeing a blank page or a 404 usually means there’s no file, so crawlers assume they can go anywhere. Subdomains get their own file too, so blog.example.com needs a completely separate robots.txt from your main site.
How to Create a Robots.txt File?
You don’t need fancy software, because a basic editor like Notepad or TextEdit does the job nicely. There are only three steps, and most people finish all of them in well under ten minutes. Take them one at a time, and you’ll end up with a clean file that crawlers can read.
1. Write Your Rules
Open a blank document and add a user-agent line, then the rules you want that bot to follow. Put each directive on its own line, because bots can misread rules crammed together on one line. Start small and add blocks only when you need them, since short files are far easier to audit.
2. Save and Upload the File
Save it as robots.txt in plain text with lowercase letters, and use UTF-8 so nothing odd creeps in. Upload it to your root folder so it loads at yourdomain.com/robots.txt instead of hiding in a subfolder. Many hosting panels and WordPress SEO plugins let you edit the file directly, which saves a lot of hassle.
3. Test Before You Publish
Open the robots.txt report in Google Search Console and check that Google can fetch and read your file. Then drop a few key URLs into the URL Inspection tool to make sure none of them are blocked. Fix any warning straight away, because even one syntax slip can make bots skip rules you care about.
How to Add a Sitemap to Robots.txt?
A sitemap line helps search engines find your pages faster, especially on new sites or ones with weak internal links. Write the word “sitemap,” then the full absolute URL, and drop it at the top or bottom of the file.
Sitemap: https://www.example.com/sitemap.xml
You can list several sitemaps if you have them, and the line applies to every bot, so don’t repeat it.
Robots.txt File Examples for SEO
These patterns cover the situations most site owners run into, so grab whichever one fits your own setup. Swap in your real folder names first, because the paths below are only placeholders and won’t match your site. Then run each one through Search Console before it goes live, since every website has its own quirks.
Allow Every Crawler
User-agent: *
Disallow:
An empty disallow line means every bot can visit every page, which suits most small blogs just fine. Plenty of owners begin here and only add blocks once crawl waste starts showing up in their reports. You can still keep a sitemap line above this group, so all your new pages get discovered quickly.
Block Internal Search and Parameters
User-agent: *
Disallow: /search/
Disallow: /*?sort=
Internal search pages and sort parameters spawn endless duplicate URLs, so blocking them protects your crawl budget. The asterisk inside a path matches any characters, which keeps pattern rules short and easy to maintain. Afterward, check Search Console to confirm that no valuable landing page got caught by these patterns by mistake.
Robots.txt Best Practices to Follow
Most robots.txt disasters come from tiny slips, and the habits below help you avoid the usual traps. Run through them before launch and again after every redesign, so small problems never reach live pages. Each one takes only minutes to apply, yet it can protect months of hard-won ranking progress.
Use Specific Paths
Add a trailing slash to folder rules, because Disallow: /de blocks every URL starting with those two letters. A tighter rule like Disallow: /de/ blocks only that folder and spares unrelated pages such as /delivery-information. Double-check every broad rule, since short paths often match far more URLs than you’d ever expect.
Never Block CSS and JavaScript
Google needs your CSS and JavaScript files to render pages, so blocking them can distort how it sees your layout. Keep those files open, and recheck them whenever a developer changes your theme, plugins, or build setup. The URL Inspection tool shows a rendered view of any page, and that quickly reveals blocked resources.
Add Comments and Keep One Group Per Bot
Start a line with a hash symbol to leave notes for teammates, since bots skip every commented line. Declare each user-agent only once, so your rules stay tidy and nobody has to guess which group wins. A quick note on why each block exists will save hours when someone new inherits your file.
How to Use robots.txt for SEO?
Start by listing the low-value areas, such as cart pages, login screens, staging folders, and filtered category URLs. Block those paths, keep your money pages wide open, and revisit the file whenever your site structure changes. Just remember that blocking a page doesn’t remove it from search, so use a noindex tag for that.
How to Fix robots.txt for Errors?
Most problems show up in Search Console reports, so check them monthly and match each message to its fix. Good SEO analytics habits speed this up, because crawl trends tend to reveal blocked pages pretty quickly.
| Error Message | What It Means | How to Fix It |
| Submitted URL blocked by robots.txt | A sitemap page is blocked | Remove the matching Disallow rule |
| Blocked by robots.txt | A blocked page is not indexed | Unblock it if the page should rank |
| Indexed, though blocked by robots.txt | Links got the page indexed anyway | Unblock it and add a noindex tag |
Remove the crawl block before you add noindex, because Google can’t read a tag on a page it can’t open.
Is a Robots.txt File for SEO Really Necessary?
Tiny sites can get by without one, but there’s rarely a good reason to skip it altogether. A robots.txt file for SEO gives you control over crawl waste, private folders, and server load as you grow. Even a three-line file with a sitemap reference beats leaving crawlers to wander around with no guidance. Businesses can use a technical SEO audit to identify the issues in the robots.txt file that can hamper performance and organic visibility.
Conclusion
Robots.txt is a tiny file with a surprisingly big say in how search engines explore your website. Keep the rules simple, test every change, and pair the file with noindex tags and a clean sitemap. Give it a quick review every quarter, and your best pages will stay open to the bots that matter.
Frequently Asked Questions
What does robots.txt do for a website?
It gives search engine crawlers simple rules about which folders and pages they may visit on your site. Used well, it saves crawl budget, protects low-value areas, and points bots clearly toward your sitemap.
Does robots.txt stop a page from appearing in Google?
No, it only blocks crawling, so a blocked page can still appear when other sites link to it. Use a noindex tag instead, and keep the page crawlable so Google can actually see that tag.
Where should I place my robots.txt file?
Place it in the root directory of each host, for example yourdomain.com/robots.txt, and nowhere else on your server. Subdomains need their own separate file, and crawlers will simply ignore any copy stored inside a subfolder.
How often should I check my robots.txt file?
Look at it any time something big changes on your site, like a migration, a redesign, or a new plugin. It’s also worth a quick glance at your Search Console reports once a month. A tiny edit can quietly block important pages without anyone noticing, and by the time your rankings drop, the damage is already done.
Are robots.txt rules case-sensitive?
They are, at least when it comes to URL paths. If you write Disallow: /Blog/, it won’t stop crawlers from getting to /blog/, because those count as two different paths. The easiest fix is to copy the path straight from your live URLs every time you add or change a rule, so the capitalization always matches.