Sep 022011
 

Robot Text

First let me explain what a robot text is, then ask the question “is it necessary”? Search engines use ‘spiders’ or ‘bots’, short for robot. Whatever you call them they go to sites and bring back criteria to the search engine (google, yahoo, bing ect) to index and rate the pages on your site. Whats the criteria? I wrote a little about that in the ‘Meta Tags’ post. The criteria changes from search engine to search engine and from year to year. Mine is much longer but looks a little like this: User-agent: Alexibot Disallow: / User-agent: Aqua_Products Disallow: / User-agent: asterias Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: * Disallow: /cgi-bin/

Robot Text

It’s just a text file placed at the root level and says what search engines are allowed to surf your site and which are not. You can also restrict the ‘spider’ access to key parts of your site if needed. I mean after all, what the ‘spider’ sees the search engine sees and what the search engine sees is made public. Mr.Wu at Google said in a post that a robot text was not needed for a site. I’ll need to read the full article but he addressed someone having issues with her robot text file. He told her to just not put it in place. Yep, you can live without a robot text file. Sites will work just fine. Should you live without it? I say no. You want to block bad robots, aid the good ones to index your site and one day I’m pretty sure a search engine like google.com, yahoo.com, or bing.com will use a robot text or even the date of the robot text in it’s ranking system. It’s just a theory but what if I’m right? Besides it only take a few moments to do. I say do it. Enjoy the song: Girl_Use_a_Robot_Text_File