All Search Engines Love Spiders: How Meta Commands Can Help You Love Them Too
http://www.site-reference.com/articles/ … m-Too.html

I disagree with most of what is said.

The article is about meta no/follow; meta no/index and meta no/archive tags that appear on the page, in the head section

A robots.txt file is the proper place to put these commands

Firstly because it is, and secondly because of the paradox of putting a no-index tag on a page that can only be read once indexed.

In short, meta tags about indexing and archiving on a page hold less weight than if they were in a robots.txt file

If you don't have a robots.txt file then look at your web logs and see how many 404 errors you have with requests for a robots.txt file. Every search engine looks for one before it crawls your site; so give them one!!!! It's easier to update in just one place than in all your pages