Generate , verify and validate google robots txt with site for webmaster tool

robots.txt file can make huge impact traffic and search engine ranking of your website or blog, it is actually a coding that allows search engine bots, spider or robots(i.e google bot) to access your web pages also it can be used to restrict your pages from search engine bots for privacy.

Example of robots txt is something like this

User-agent:  *

# disallow all files in these directories

Disallow: /cgi-bin/

Disallow: /z/j/

Disallow: /z/c/

Disallow: /stats/

Disallow: /dh_

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /contact/

Disallow: /tag/

Disallow: /wp-content/b

Disallow: /wp-content/p

Disallow: /wp-content/themes/askapache/4

Meta robots.txt file is actually top link of your website directory, i.e www.yoursite.com/robots.txt

But when you will follow this link you will not see any coding it will instead redirect to your main page, these robots.txt file allows google bot and other search engine bots to read content and index them in there search engines, that is why they are very important, google webmaster tools also allow you to create, manage and validate your robots txt.

If you are using wordpress, then there is a plugin all in one seo which will automatically generate robots txt.

Using google webmaster tool to generate and manage robots txt

First you have to sing in for google account, if you don’t have one then create account, after that click settings and then select google account settings.

On google account settings page select google webmaster tools, this tool is best robots txt generator, robots txt seo, robots txt checker and robots txt validator.

You will see a page like this, then click on add site and enter URL of the website you wish to generate and manage robots txt.

Then you have to validate your website by three or four methods, but I prefer using tracking code(html) code to verify your account it is easy method.

Add the meta code just after head tag as shown in example, then click verify.

After verifying your site select it from main page of webmaster tools, select drop menu site configuration and then select crawler access, here you can create, manage and test your robots txt file, on clicking on generate robots txt you can grant access or restrict any of your page.

I recommend you using attracta.com it will automatically detect seo problems and will tell you what to do and how to fix.

Wishing you success, have a happy life.