I’ve just come across a great little tool to help you build a robots.txt file for your website.
What is a robots.txt file? It’s basically a little text file you upload to your server which tells search engines which pages or directories of your site you want them to spider (or not spider). In other words, it tells search engines which pages to look at (please) and which pages to please ignore and not list in the Search Engine Results Pages.
This can be more important than one might think. You have files in your server that are not real webpages and you don’t want to distract search engines by allowing them to inspect and crawl over these boring unimportant (for SEO) files. Examples of this are image files in your “images” folder, or WordPress plugins and themes, etc.
You might also have a file which you don’t want indexed for other reasons. Such as the download page for ab E-Book you are selling. You would want people to only find that page after they have paid. If it shows up in the SERPs, they can just find it by accident and get it for free (gasp)!
So here is the little tool you can use. If you have more questions – well, Google it. But also leave me a comment so I can add some more information later when I have some more time!
Tags:
is this really helful? i find robots.txt is pretty easy to make manually
die guys last blog post..Exchange
Yes it is – especially for people who aren’t familiar with how to make robots.txt files and/or who don’t always remember how to write them and what to write.
But what is follow or nonfollow tag in the robots.txt file ?