How To Add Custom Robots.txt File In Blogger

Robots.txt is a text file which contains few lines of simple code. It is saved on the website or blog’s server which instruct the web crawlers to how to index and crawl your blog in the search results.

That means you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog labels page or any other pages that are not as important to get indexed.

In one of my previous posts I have discussed about Custom Robots Header Tags for blogger. If you have read that post then I hope you guys are aware with its importance in search rankings.

Each blog hosted on blogger have its default robots.txt file which looks like this:

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: http://example.blogspot.com/feeds/posts/default?orderby=UPDATED

Let’s first study the code after that we will learn how to add custom robots.txt file in blogspot blogs.

Robots.txt Explanation:

1. User-agent: Mediapartners-Google is for Google AdSense robots which help them to serve better ads on your blog. Either you are using Google AdSense on your blog or not simply leave it as it is.

2. User-agent: * is for all robots marked with asterisk (*). In default settings our blog’s labels links are restricted to indexed by search crawlers that means the web crawlers will not index our labels page links because of below code.

3. Disallow: /search means the links having keyword search just after the domain name will be ignored. See below example which is a link of label page named SEO.

And if we remove Disallow: /search from the above code then crawlers will access our entire blog to index and crawl all of its content and web pages.

Disallow Particular Post: Now suppose if we want to exclude a particular post from indexing then we can add below lines in the code.

Disallow: /y/m/post-url.html

Here y and m refers to the publishing year and month of the post respectively. For example if we have published a post in year 2014 in month of March then we have to use below format.

Disallow: /2014/03/post-url.html

To make this task easy, you can simply copy the post URL and remove the blog name from the beginning.

Disallow Particular Page: If we need to disallow a particular page then we can use the same method as above. Simply copy the page URL and remove blog address from it which will something look like this:

Disallow: /p/page-url.html

4. Allow: / refers to the Homepage that means web crawlers can crawl and index our blog’s homepage.

5. Sitemap: http://example.blogspot.com/feeds/posts/default?orderby= UPDATED refers to the sitemap of our blog. Means whenever the web crawlers scan our robots.txt file they will find a path to our sitemap where all the links of our published posts present.

But this sitemap will only tell the web crawlers about the recent 25 posts. If you want to increase the number of link in your sitemap then replace default sitemap with below one.

It will work for first 500 recent posts :

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

If you have more than 500 published posts in your blog then you can use two sitemaps like below :

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=500&max-results=1000

Adding Custom Robots.Txt to Blogger:

• Navigate to Settings >> Search Preferences >> Crawlers and indexing >> Custom robots.txt >> Edit >> Yes
• Now paste this robots.txt file code in the box.

User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /
Sitemap: http://example.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500

• Click on Save Changes button.

You are done!

This was the tutorial on how to add custom robots.txt file in blogger. If you have any doubt or query then feel free to ask me.