How do I use a Robots file or do you have an example of one?

[dealsforums]

Hi Every One,

Can any one please suggest me, How do I use a Robots file ?

if anyone have example please post a small one here..

Thanks in advance for reply

[whitelee5]

Inktomi is no longer. In December, 2002, Yahoo! announced that it would acquire Inktomi. Shortly thereafter, Inktomi was shut down entirely.

[whitelee5]

The robots.txt is a TEXT file (not HTML!) which has a section for each robot to be controlled. Each section has a user-agent line which names the robot to be controlled and has a list of "disallows" and "allows". Each disallow will prevent any address that starts with the disallowed string from being accessed. Similarly, each allow will permit any address that starts with the allowed string from being accessed. The (dis)allows are scanned in order, with the last match encountered determining whether an address is allowed to be used or not. If there are no matches at all then the address will be used.

[dustin]

I don't have much information.

[Portable]

Thanks, I enjoyed reading your post.

[bugzy]

There are generators that will automatically do that for you. Just Google it and you'll fins one.