One important factor on how to earn money from your blog is doing SEO or search engine optimization. SEO has two separate but related methods: on-site and off-site optimization. I will talk now one essential part of off-site SEO, the robots.txt installation and how it will improve your blog’s ranking in search engine results.
Robots.txt is a way to instruct search engine robots like Googlebot on which pages or posts of your blog it should crawl and index. It is saying to these robots, you should not go to this page or you should crawl this post only. As a default, it is programmed to allow crawling and indexing all pages, posts and contents of your site by all search engine robots. You can see the default robots.txt as below:
User-agent: *
Disallow:
The above text means all search engine robots will be allowed to crawl your sites anything in it without restriction. However, you could remove a page or pages from indexing or crawling by changing the text as the following:
User-agent: *
Disallow: /page/*
How to Install Robots.txt on your Website or other Blogs?
If you want to know your blog’s robots.txt file, you can type this in your Internet browser address: http://yourblogsite.com/robots.txt and it will show your current setting. In addition, if you like to change it, you must generate or create first or you can copy my own robots.txt if you want. Here it is,
Sitemap: http://yourblog.com/sitemap.xml
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /trackback/
Disallow: /*/trackback/
Disallow: /cgi-bin/
Disallow: /weblog/*
Disallow: /email/
Allow: /wp-content/uploads/
User-agent: Googlebot
Disallow: /*.pdf$
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.php*
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
Disallow: /*?*
Disallow: /wp-*
Disallow: */trackback/
Disallow: /cgi-bin/
Disallow: /weblog/*
Disallow: /*&lang=en
Disallow: /email/
Disallow: /&rurl*
Allow: /wp-content/uploads/
User-agent: Googlebot-Image
Allow: /*
User-agent: Mediapartners-Google*
Disallow:
In the sitemap section, you must enter your blog’s sitemap and then just copy the rest. You can make your robots.txt in Notepad and then save it as “robots.” After saving it, log in at your blog’s cPanel to install it. You can do this by going to the File Manager after your successful log in. When you click the File Manager, select the Home Directory button and then choose the public_html folder where you should install the robots.txt. Select the upload button to install it in the public_html directory. After that, you must wait for several minutes to see the new one.
Is Robots.txt really Important for SEO?
In my experience I would say it is important because you could instruct which content or pages search engine should focus or crawl in your blog. In this way, your content will be seen clearly by bots than the other contents of your blog. You could block images, uploaded contents, archives or comment pages if you like so your post will stand out for search engines.
{ 0 comments… add one now }