Need help!

Post by **Duvel78** » 25 Oct 2004 01:09 pm

I need some help for the robots.txt file, to avoid a duplicate content problem with my url rewriting... But I'm lost

I've found that code for example in a tutorial:

Disallow: /your-forum-folder/sutra*.html$ 
Disallow: /your-forum-folder/ptopic*.html$ 
Disallow: /your-forum-folder/ntopic*.html$ 
Disallow: /your-forum-folder/ftopic*asc*.html$

But this is not allowed, a "*" after a "disallow" instruction...

Post by **5lab** » 25 Oct 2004 01:24 pm

why not? surely robots.txt just tells bots what to do?

Post by **Duvel78** » 25 Oct 2004 01:28 pm

Yep it tells the bots what to do and not to do

But in the code, a "*" can't after a "disallow"

Post by **5lab** » 25 Oct 2004 01:47 pm

different bots use slightly different rules. how do you know you cant use a * after a dissalow?

Post by **Duvel78** » 25 Oct 2004 02:13 pm

It's a general rule ( http://www.robotstxt.org )

Note also that regular expression are not supported in either the User-agent or Disallow lines. The '*' in the User-agent field is a special value meaning "any robot". Specifically, you cannot have lines like "Disallow: /tmp/*" or "Disallow: *.gif".

whole article: http://www.robotstxt.org/wc/exclusion-admin.html

Post by **5lab** » 25 Oct 2004 02:18 pm

ah gotcha. is there another character, something like ? you can use?

Post by **foggyjames** » 31 Oct 2004 06:41 pm

Presumably the * is a 'wildcard'...which doesn't work with this board....in which case the 'wildcard' for this board must be something else....
....he says from a complete position of ignorance

cheers

James

Post by **5lab** » 31 Oct 2004 06:46 pm

you kinda had teh right idea, but not entirely right. never mind eh