Robots.txt Generator

I’ve just come across a great little tool to help you build a robots.txt file for your website.

Robots.txt Generator

What is a robots.txt file? It’s basically a little text file you upload to your server which tells search engines which pages or directories of your site you want them to spider (or not spider). In other words, it tells search engines which pages to look at (please) and which pages to please ignore and not list in the Search Engine Results Pages.

This can be more important than one might think. You have files in your server that are not real webpages and you don’t want to distract search engines by allowing them to inspect and crawl over these boring unimportant (for SEO) files. Examples of this are image files in your “images” folder, or WordPress plugins and themes, etc.

You might also have a file which you don’t want indexed for other reasons. Such as the download page for ab E-Book you are selling. You would want people to only find that page after they have paid. If it shows up in the SERPs, they can just find it by accident and get it for free (gasp)!

So here is the little tool you can use.  If you have more questions – well, Google it.  But also leave me a comment so I can add some more information later when I have some more time!

Robots.txt Generator


Robots.txt, SEO, Search Engine Optimization

, , , , , , , , , , ,

3 Responses to “Robots.txt Generator”

  1. die guy
    August 19, 2008 at 1:13 pm #

    is this really helful? i find robots.txt is pretty easy to make manually

    die guys last blog post..Exchange

  2. Anna
    August 19, 2008 at 2:40 pm #

    Yes it is – especially for people who aren’t familiar with how to make robots.txt files and/or who don’t always remember how to write them and what to write.

  3. invisible
    September 20, 2010 at 10:36 pm #

    But what is follow or nonfollow tag in the robots.txt file ?

Leave a Reply

Leave your opinion here. Please be nice. Your Email address will be kept private.