There is a text file of a few lines, called robots.txt and placed at the root of your website, that every search engine (Google, Bing, Yandex and the others) reads first, before crawling a single page.
This file is the robots.txt file, and its importance for SEO is often underestimated. Badly written, it can cause errors with dramatic consequences: inaccessible files, incorrect indexing, strategic pages vanishing from search results.
In computing terms, a file called robots.txt works as a system of directives addressed to crawlers: a list of instructions these agents must read and respect under the Robots Exclusion Protocol. And a single badly written line is enough to accidentally block the indexing of your entire site.
Without you knowing. For weeks. With an SEO impact that can last for months.
That file is robots.txt. Yet in many web agency audits, the robots.txt file is one of the most neglected parts of technical SEO. Too often left with its default configuration since the site was built, edited in a hurry during a migration, or simply ignored for lack of documentation or knowledge of what it does, robots.txt files deserve particular attention.
Used well, robots.txt guides user agents towards the resources that matter for your rankings. Badly configured, it can make files inaccessible, cause incorrect indexing and weaken site protection. Note that creating and writing a good robots.txt file (adding comments, handling Allow/Disallow directives, giving the address of the XML sitemap) requires no special licence, no special permission and no complex system command.
It is part of your SEO strategy and is accessible to any webmaster who takes the time to understand how it works. Whether you manage sites in France or internationally, whether your main menu is in English or in French, whether your sources are internal or come from an intranet, this guide will help you write, test and add directives to your file, and make sure it works as intended at all times.
In this complete guide we explain everything: the definition and role of the robots.txt file, its structure and syntax, instructions for creating your file, Allow and Disallow directives, ways to test your file, common mistakes to avoid and best practices to optimise your site’s SEO through good management of bot access.
Whether you are a webmaster, a developer, an SEO consultant or simply the owner of a website, you will come away with the knowledge you need to master this essential technical SEO tool.
Robots.txt: definition and role
What is the robots.txt file?
The robots.txt file, also called the “robot exclusion file” or “exclusion protocol”, is a text file used by webmasters to tell search engine crawlers which pages or resources of their website they are allowed or not allowed to crawl and index.
It is a file named robots.txt, placed at the root of a website (yourdomain.com/robots.txt), which crawlers systematically check first before exploring the rest of the site.
The definition of robots.txt in SEO vocabulary can be summed up like this: it is a text file containing instructions for bots on how to crawl your site. These instructions say which folders, directories, pages or file types may be crawled and which should be ignored.
The robots.txt file is based on the Robots Exclusion Protocol (REP), an internet standard set in 1994 and regularly updated since.
It is important to understand from the start what robots.txt is, and what it is not. Robots.txt is a crawl management tool: it tells crawlers what to crawl or not to crawl. It is not a security tool: the information in a folder or page blocked by robots.txt remains accessible to users who know its direct URL.
To protect sensitive data, other security and privacy mechanisms are needed.
What is the robots.txt file for?
The main role of the robots.txt file is to guide crawlers (Googlebot for Google, Bingbot for Bing, YandexBot for Yandex and the many other crawlers on the internet) as they explore your website. Without this file, bots would crawl your whole site indiscriminately, including pages and resources that are of no interest for organic search.
The robots.txt file is mainly used for three things. First, to prevent crawling of pages or resources that shouldn’t be indexed: admin pages, login pages, technical folders, configuration files, internal data. Second, to preserve crawl budget by stopping Googlebot from wasting its crawl quota on pages with no SEO value, which lets you concentrate crawling on the important pages of your site.
Finally, to tell crawlers where your XML sitemap is, which makes it easier to discover and index all your web pages.
Robots.txt vs the noindex tag: two complementary approaches
A common confusion in technical SEO is mixing up the robots.txt file and the noindex meta tag. These two tools serve different purposes and work in a complementary way.
The robots.txt file controls crawling: it tells bots which pages they may or may not visit. A page blocked in robots.txt can still be indexed by Google if it receives inbound links from other pages, because Google can know it exists without having visited it directly.
The noindex meta tag (placed in a page’s HTML code: <meta name="robots" content="noindex">) controls indexing: it tells Google not to include the page in its search results, even if the bot has visited it. It is the safest method to permanently exclude a page from Google’s index.
Combining the two is sometimes done in specific cases, but beware: a page blocked in robots.txt cannot be read by Googlebot, which means the noindex tag it contains can’t be read either. To exclude a page from indexing, prefer the noindex tag over a robots.txt rule.
The impact of robots.txt on crawl budget
Crawl budget is the number of pages Googlebot is willing to crawl on your site over a given period. For small sites, it is usually not a problem. For large ecommerce sites with thousands of pages, sites with lots of URL parameters or sites with dynamically generated content, managing crawl budget becomes a strategic SEO issue.
A well-configured robots.txt file lets you optimise the use of crawl budget by blocking crawling of pages with no SEO value: internal search result pages, pages with URL parameters, login pages, technical files. By steering Googlebot towards the important pages, you make sure your strategic content is crawled and indexed regularly, which has a direct impact on your site’s visibility in Google search results.
Structure and syntax of the robots.txt file
The text format of the robots.txt file
The robots.txt file is a simple, unformatted text file encoded in UTF-8. It contains no HTML code, no tags, no scripts, only plain text organised according to precise rules. Its structure is based on blocks called “records”, each made up of one or more directives that apply to a specific user agent.
Each record starts with a User-agent directive that says which bot the following instructions are addressed to, followed by one or more Disallow (forbid access) or Allow (permit access) directives. Lines starting with the # symbol are comments: they are ignored by bots but let you annotate the file to make it easier to maintain.
The main directives
The User-agent directive says which bot the instructions are for. The * (asterisk) means all bots. Googlebot means specifically Google’s bot. Bingbot is Bing’s. YandexBot is Yandex’s. You can create specific records for each agent or one general record that applies to all of them.
The Disallow directive tells the bot not to crawl the specified path. A “Disallow: /admin/” line blocks access to the /admin/ folder and all its contents. A “Disallow: /” line blocks access to the entire site; it is the most dangerous directive and the cause of the most frequent SEO disasters.
The Allow directive, less well known, explicitly permits access to a specific resource, even if it sits inside a folder blocked by a Disallow directive. It is useful in specific cases such as allowing access to a CSS file in a folder that is otherwise blocked.
The Sitemap directive gives the URL of the site’s XML sitemap file, a good practice that makes it easier for bots to discover and index all your pages.
The Crawl-delay directive lets you set a delay (in seconds) between each bot request, useful for sites with performance problems caused by intensive crawling. Beware: Google does not officially honour this directive, but other engines such as Yandex do take it into account.
Examples of robots.txt files
Here are a few concrete examples of robots.txt files to illustrate the different possible configurations.
Example 1 — A basic robots.txt file that allows everything:User-agent: *Allow: /Sitemap: https://www.yoursite.com/sitemap.xml
Example 2 — Blocking access to the admin folder and login pages:User-agent: *Disallow: /admin/Disallow: /wp-admin/Disallow: /login/Allow: /wp-admin/admin-ajax.phpSitemap: https://www.yoursite.com/sitemap.xml
Example 3 — Blocking a specific bot (for example a scraping bot):User-agent: BadBotDisallow: /
User-agent: *Allow: /Sitemap: https://www.yoursite.com/sitemap.xml
Example 4 — Blocking pages with URL parameters on an ecommerce site:User-agent: *Disallow: /?Allow: /Sitemap: https://www.yoursite.com/sitemap.xml
These robots.txt examples illustrate the most common configurations. In practice, every site has specific needs and the robots.txt file must be customised accordingly.
How to create a robots.txt file
Where to put the robots.txt file
The robots.txt file must be placed at the root of the website, that is, at the very top level of your domain, reachable at the URL yourdomain.com/robots.txt. It cannot be placed in a subfolder or a subdomain, or bots won’t find it.
On most web servers, the site root corresponds to the public_html folder (or www, or htdocs depending on your hosting setup). This root folder is where you upload your robots.txt file via FTP or through the file manager of your hosting provider.
To check that your file is correctly placed and accessible, simply type yourdomain.com/robots.txt into your browser. If the file’s contents appear, all is well. If you get a 404 error, the file is in the wrong place or doesn’t exist yet.
Creating the file with a text editor
The simplest way to create a robots.txt file is to use a text editor: Notepad on Windows, TextEdit on Mac (in plain text mode), or any other simple text editor. Open a new file, type your directives following the correct syntax, and save the file as “robots.txt” (lowercase, with no spaces or special characters in the name).
Be careful not to use a word processor such as Microsoft Word, which can add invisible formatting characters that may corrupt the file. A robots.txt file must be in pure text format, with no formatting and no special characters, encoded in UTF-8.
Robots.txt generators
For those who prefer to avoid writing it by hand, free robots.txt generators are available online. These tools let you create a robots.txt file by selecting the options you want through a graphical interface, without writing the directives manually.
These generators are useful for beginners or standard setups, but they are no substitute for understanding the directives in specific cases. For sites with complex needs (ecommerce sites with many URL parameters, multilingual sites, sites with several subdomains), a thoughtful manual configuration is still necessary.
Creating your robots.txt on WordPress with Yoast SEO
For WordPress sites, the Yoast SEO plugin offers a dedicated interface to create and edit the robots.txt file straight from the WordPress dashboard, without having to access the server via FTP. Go to Yoast SEO > Tools > File editor to view and edit the contents of your robots.txt file.
Other WordPress SEO plugins such as Rank Math or All in One SEO offer similar features. These tools make managing the robots.txt file easier for non-technical users, but always check that the configuration generated by the plugin really matches your specific needs.
Accessing and editing the file via FTP
To access the robots.txt file directly on your server, use an FTP client such as FileZilla. Connect to your server with your FTP credentials (provided by your host), go to your site’s root folder (usually public_html), and you will find the robots.txt file.
Download it, edit it with your text editor, then upload it to the server again, overwriting the previous version.
This method is the most direct and reliable way to edit the code of your robots.txt file, and particularly useful when your CMS dashboard is inaccessible or when you need to make urgent changes.
Allow and Disallow directives: mastering access
What to block with Disallow
The Disallow directive is the most used in a robots.txt file. It lets you forbid access to specific folders, directories or pages. Here are the resources it is generally recommended to block, to optimise crawl budget and protect the sensitive areas of your site.
Admin folders (/admin/, /wp-admin/ except admin-ajax.php, /dashboard/) should be blocked so bots don’t crawl your site’s management interfaces. These pages have no SEO value and needlessly waste crawl budget.
Login and registration pages (/login/, /register/, /account/) aren’t meant to appear in search results. Blocking them lets you concentrate crawling on your content pages.
On ecommerce sites, pages generated by filters and sorting (/search?, /?sort=, /?filter=*) create hundreds or thousands of duplicate URLs that dilute crawl budget. Blocking them with Disallow rules on URL parameters is a common good practice.
Technical folders (/tmp/, /cache/, /logs/) contain no content meant for users and should be blocked.
What to allow with Allow
The Allow directive is used less often but can be necessary in specific cases. It is particularly useful for allowing access to a precise resource in a folder that is otherwise blocked.
The classic example is WordPress: the /wp-admin/ folder is usually blocked, but the /wp-admin/admin-ajax.php file must be accessible to bots for some site features to work properly. The right configuration is therefore:
Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.php
CSS and JavaScript files are another case where Allow is useful: if you block a folder that contains resources needed to render your pages, you risk stopping Google from properly understanding the design and layout of your site, which can harm its assessment of the quality of the user experience.
Resources you should never block
Some resources must never be blocked in the robots.txt file, or you will seriously harm your SEO. The main content pages of your site (home page, service pages, blog posts, product pages) must be accessible to bots.
The CSS and JavaScript files essential to rendering your pages must be crawlable. Images that matter for image search must not be blocked. And above all, your XML sitemap must never be blocked.
Best practices for ecommerce sites
Ecommerce sites have specific needs when it comes to configuring the robots.txt file. Handling URL parameters is particularly important: filters, sorting, internal search result pages and session parameters generate a large number of unique URLs that can quickly saturate crawl budget.
The good practice is to block URLs with parameters through targeted Disallow rules, while keeping access to the canonical URLs of product and category pages. Particular attention should be paid to the Crawl-delay directive on high-traffic sites, so that intensive crawling by Googlebot doesn’t slow server performance.
Testing and validating your robots.txt file
The Google Search Console testing tool
Google Search Console offers a built-in robots.txt testing tool that lets you check whether a specific URL is blocked or allowed by your current robots.txt file. To reach it, go to Google Search Console, in the “Legacy tools and reports” section or directly in the URL inspection tool.
This tool lets you test your robots.txt file in real conditions: enter the URL you want to check, select the user agent (Googlebot, Googlebot-Image, etc.) and see whether the URL is accessible or blocked under the rules of your file. It is the reference tool for validating your configuration before putting it live.
Testing your file in real conditions
Beyond Google Search Console, several methods let you test your robots.txt file. The first is simply to visit yourdomain.com/robots.txt in your browser to check that the file is accessible and that its contents are correct.
Free online tools such as Google’s robots.txt tester or third-party tools let you simulate a bot’s crawl and check which URLs are accessible or blocked under your file’s rules. These tests are particularly useful after every change to the file, to make sure the new rules work as intended without creating unwanted side effects.
Monitoring the impact on indexing after a change
After editing your robots.txt file, closely monitor how your site’s indexing evolves in Google Search Console. The “Coverage” section tells you how many pages are indexed, which pages are excluded and any errors detected.
If you see a sudden drop in the number of indexed pages after a robots.txt change, immediately check that you haven’t accidentally blocked important pages.
The delay between changing the file and seeing the impact on indexing can range from a few hours to several days, depending on how often your site is crawled. Always test your changes in a development environment before deploying them to production, and keep a version history of your robots.txt file so you can roll back to a previous version if something goes wrong.
Robots.txt mistakes that sink your SEO
Accidentally blocking the whole site
This is the most serious mistake and, unfortunately, not that rare. A single line in the robots.txt file is enough to block Googlebot’s access to your entire site:
User-agent: *Disallow: /
This configuration, which looks like “forbid everything”, can end up in your robots.txt file after a badly handled migration, a copy-paste error, or an automatic setting from a badly configured plugin. The result: Googlebot can no longer crawl your site, pages gradually disappear from Google’s index, and your organic traffic collapses.
It is a catastrophic change whose effects can take weeks to be felt, and further weeks to fix.
The wrong directive that blocks key pages
An overly broad Disallow directive can block important pages without you realising. For example, a “Disallow: /fr/” rule meant to block a specific subfolder can in fact block every URL starting with /fr/, including service pages or blog posts that make up most of your organic traffic.
Precision when writing Disallow rules is essential. Always check the impact of a new rule with Google Search Console’s testing tool before deploying it.
Syntax errors in the robots.txt file
A syntax error in the robots.txt file can make the whole file unreadable for bots, or create unexpected behaviour. Common errors include: spaces at the start of a line (robots.txt is sensitive to spaces), special characters in blocked URLs, a missing User-agent directive before the Disallow/Allow directives, or instructions on the same line instead of separate lines.
Always test your robots.txt file with Google Search Console’s validation tool before deploying it, to catch any syntax errors.
Confusing robots.txt with security
A frequent design mistake is using the robots.txt file to “protect” sensitive data, thinking that blocking a folder in robots.txt makes it inaccessible. This is a dangerous misconception: robots.txt is an indication that well-behaved bots respect, but it doesn’t prevent a user or a malicious bot from accessing the URL directly.
To protect sensitive data (configuration pages, critical PHP files, intranet data), use authentication mechanisms, server access restrictions (.htaccess) or appropriate privacy settings, not the robots.txt file.
Web Opteam: your SEO agency for technical audits
Auditing the robots.txt file in our SEO approach
At Web Opteam, a Paris-based digital agency working remotely with clients worldwide, auditing the robots.txt file is an integral part of every technical SEO audit. Our approach is to analyse the file’s current configuration, and identify incorrect directives, unwanted blocks and opportunities to optimise crawl budget.
A badly configured robots.txt file can undo months of SEO work, and it is often one of the first causes of indexing problems we detect in our audits.
Based on this audit, we provide concrete recommendations and an optimised robots.txt file for your site, with detailed explanations of each directive so that you understand and control the configuration of your own site.
Technical support and training
Web Opteam offers complete technical support on technical SEO (robots.txt, XML sitemap, canonicalisation, structured data, Core Web Vitals) for businesses that want to optimise their organic search in depth.
Our technical SEO training helps your teams understand and master these essential aspects of SEO.
For an audit of your robots.txt file or complete technical SEO support, contact our team directly. We are here to make sure your site is perfectly accessible to Google’s bots, and that nothing needlessly blocks your visibility in search results.
Robots.txt: a simple file, considerable stakes
The robots.txt file is a text file of a few lines that can have a considerable impact, positive or negative, on your site’s organic search performance. Well configured, it optimises crawl budget, guides Googlebot to your important pages and protects the parts of your site that shouldn’t be crawled.
Badly configured, it can block the indexing of key pages, or even of your entire site.
The golden rules to remember: put your file at the root of your site, check its syntax with Google Search Console’s testing tool, test every change before deploying it, and never confuse robots.txt with security. Review your robots.txt file regularly during your SEO audits; it is a file that evolves with your site and must be kept up to date.
And if you have any doubt about your file’s configuration or want a complete technical SEO audit of your site, the Web Opteam team is here to help. Get in touch: we have the tools, the expertise and the experience to make your robots.txt an ally of your SEO rather than an obstacle.
