Robots.txt System Set for Major Revamp

[RJD]

The new proposal, to be unveiled Thursday by a consortium of publishers at the global headquarters of The Associated Press, seeks to have those extra commands -- and more -- apply across the board. Sites, for instance, could try to limit how long search engines may retain copies in their indexes, or tell the crawler not to follow any of the links that appear within a Web page.

The current system doesn't give sites "enough flexibility to express our terms and conditions on access and use of content," said Angela Mills Wade, executive director of the European Publishers Council, one of the groups behind the proposal. "That is not surprising. It was invented in the 1990s and things move on."

Full story at http://www.newsfactor.com/story.xhtml?story_id=13100C1F8V0J&nl=2

 

[Rabbit]

I like the idea of having more control over the robots. I am currently working on my website and just have test pages but noticed in my server logs the frequency at which these pesky robots visit... not to mention the hacks in China, Korea, Taiwan, Russia. For some of the "brick & morter shops" I suppose the robot crawlers would give them better page ranking and relevency during a search.

My website is being created for mostly my own client base and not necessarily for public exposure... so... limiting the depth of the crawlers would be a good thing for me. Some of my friends were pushing me to create all manner of things for the robots which I don't like at all. I wouldn't want people to do a search for Rabbit Floral and find "diapers for rabbits... peter rabbit floral aprons... flower garden pests - rabbits..." LOL

Well... back to dynamic content creation... it's been quite a challenge as I am not a web master or SQL database person. I'm just a Rabbit... "garden pest"... LOL