How to Optimise Your Robots.txt for SEO in WordPress
In technical SEO, learning to optimise a WordPress robots.txt file is key in stopping search engines from ‘crawling’ pages you want to keep from public view. Whether that’s a placeholder page, your plug-ins or your admin, these files get the job done.
You might be tempted to block WordPress tags, archive pages or categories in a bid to improve your crawl rate. Our experts warn against this, as any promise of faster indexing and higher rankings is a myth – and against Google’s Search Essentials.
Instead, this guide will help you understand the robots.txt format as well as creating a file for your own website, helping you keep those key pages under wraps while still ranking for your chosen keywords.
So, what is a Robots.txt file?
Essentially, a robots.txt is a type of text file created by a website owner that tells search engine crawlers how to do their job.
Typically stored in a text file that website owners can create to tell search engine bots how to crawl and index pages on their sites.
You’ll usually find it in the root directory – or the main folder – of your website, with the basic format going as follows:
- User-agent: [user-agent name, which tells the robots.txt file to apply the next rule to all crawlers]
- Disallow: [URL string not to be crawled]
- User-agent: [user-agent name]
- Allow: [This is an optional URL string to be crawled – even if disallowed by a previous rule]
- Sitemap: [URL of your XML Sitemap which tells crawlers where it is located, helping them navigate your website]
Why should your WordPress site have a Robots.txt file?
Without a robots.txt file, search engines will crawl and index your website – but you’ll have no say in which pages they choose to do so.
In terms of control, you’ll want a bigger say in what Googlebot (Google’s dedicated crawler) deems important on your website. As crawlers have a set ‘quota’ for each website, once they use up their ‘budget’ before a session is up, they will come back another time and pick up where they left off.
The problem? This can majorly slow down your indexing rate.
Luckily, WordPress robots.txt files can help disallow search bots from crawling useless pages on your website, saving crawl quota and getting those ‘good’ pages indexed sooner rather than later.
You can also use a robots.txt file as a preventative measure, stopping search engine crawlers from indexing a post, page or entire sections of your website.
While it’s not the slickest way of hiding content from your user’s eyes – as many people can stumble across ‘backend’ pages through natural site exploration – it will help you prevent unwanted content from cropping up in search results.
What Does an Ideal Robots.txt File Look Like?
To clean up your website, all it takes is a fairly basic robots.txt file. This can vary depending on your site’s individual needs, but we’re usually looking for the following layout:
- User-agent: *
- Disallow:
- Sitemap: http://www.example.com/post-sitemap.xml
- Sitemap: http://www.example.com/page-sitemap.xml
This robots.txt file gives crawlers permission to index the entire website and links to the XML sitemaps.
For WordPress sites, where you’ll want to keep folders such as the WordPress admin area, readme file, and cloaked affiliate links private, the following rules might help:
- User-Agent: *
- Allow: /wp-content/uploads/
- Disallow: /wp-admin/
- Disallow: /readme.html
- Disallow: /refer/
- Sitemap: http://www.example.com/post-sitemap.xml
- Sitemap: http://www.example.com/page-sitemap.xml
By adding sitemaps in this format to your robots.txt file, it makes it clear for crawlers to find all the pages on your site – so more can be indexed in a shorter time span.
To create or edit selected robots.txt, you can always turn to plugins like Yoast SEO or All in One SEO to tackle the file. Or, if enabled, you can always use the built-in WordPress file editor to alter your permissions.
Final thoughts
When it comes to guiding search engines, their crawlers and even users through your WordPress website, creating and optimising a robots.txt file is a straightforward and effective way of signposting what should be seen – and hiding what doesn’t. The result? Boosting your user’s experience, as well as improving your SEO efforts and hopefully, rankings and overall visibility across the SERPs.
However, if you need a little extra help – or would like us to talk about all things SEO from a technical standpoint – our experts can help.
