CATEGORIES Auto & Trucks Business & Finance Computers & Internet Education Family Food & Drink Gadgets and Gizmos Health Hobbies Home Improvement Humor Kids & Teens Legal Marketing Men Music and Movies Online Business Parenting Pets and Animals Politics and Government Recreation & Sports Relationships Religion and Faith Self Improvement Site Promotion Travel & Leisure Web Development Women Writing Keyword Search Article Title Author's Name SITE MENU Search Submit An Article Submit Articles in Bulk Submit Press Release Syndicate Articles Distribute Articles Privacy Policy Terms of Use Link To Us Contact Us Site Map Home

The Proper Way To Use The robots.txt File Update  by: Jimmy Whisenhunt In my last article about the robots.txt file I had spelled it wrong. It should have been robots.txt instead of robot.txt. The article should read like this: When optimizing your web site most webmasters don’t consider using the robots.txt file.This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder. Here is a list of variables that you can include in a robot.txt file and there meaning: 1) User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example. 2) Disallow: In the field you specify the files and folders not to include in the crawl. 3) The # is to represent comments Here are some examples of a robots.txt file User-agent: * Disallow: The above would let all spiders index all content. Here another example User-agent: * Disallow: /cgi-bin/ The above would block all spiders from indexing the cgi-bin directory. User-agent: googlebot Disallow: User-agent: * Disallow: /admin.php Disallow: /cgi-bin/ Disallow: /admin/ Disallow: /stats/ In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php. About The Author Jimmy Whisenhunt is the webmaster at VIP Enterprises http://www.vipenterprises.org. vipenter@vipenterprises.org This article was posted on February 16, 2005 Read More Articles from the "Web Design and Development" Category: Choosing the Right Web Designer by Andy Quick What is Windows Hosting? by Mer Amer Create a Google Sitemap by Robin Eldred What is Managed Hosting? by Mer Amer 8 Ways to Build a Really Bad Web Site for Search Engines by Gareth Davies Denver Internet Marketing by Sytsma Morris-Reeves Optimizing for Visitors not Search Engines by Michael Medeiros Keep Your Navigation Simple! by Jamie Kiley Improve Your Google AdSense Ad Relevancy with Section Targeting by Jos Jongejan Google Adsense - Responsible For The Decline Of Web Publishing Standards? by Stuart Drew << Back to "Web Design And Development" Index

Disclaimer: The information presented and opinions expressed herein are those of the authors
and do not necessarily represent the views of ArticleCity.com and/or its partners.

	Web www.articlecity.com