<<<
Chronological Index
>>> <<<
Thread Index
>>>
Re: [ga] DNSO.org web site and robots.txt
Saturday, May 18, 2002, 1:29:37 PM, George Kirikos wrote:
> Hello,
> I was curious as to why the DNSO website has a very restrictive
> robots.txt file:
> http://www.dnso.org/robots.txt
> With the top 3 lines of:
> User-agent: *
> Disallow: /constituency/work/
> Disallow: /clubpublic/
> it means that search engines like Google can't index the website
> (including mailing lists), and since the website itself lacks search
> capabilities, it's hard to find material (e.g. old posts) quickly.
> PROPOSAL: That the DNSO webmaster delete the current robot.txt file
> entirely.
> In this day and age with cheap bandwidth, I'm sure allowing the website
> to be indexed by Google and others wouldn't be costly, and has so many
> benefits.
I agree. I've just changed the configuration of the dnspolicy search
indexer to ignore the robots.txt file on the dnso.org website.
--
Best regards,
William X Walsh <william@wxsoft.info>
--
Save Internet Radio!
CARP will kill Webcasting!
http://www.saveinternetradio.org/
--
This message was passed to you via the ga-full@dnso.org list.
Send mail to majordomo@dnso.org to unsubscribe
("unsubscribe ga-full" in the body of the message).
Archives at http://www.dnso.org/archives.html
<<<
Chronological Index
>>> <<<
Thread Index
>>>
|