Log In | Register
Skip to main content
Topic: Search engines blocked (Read 794 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.
Search engines blocked
Greetings,

A bit of a question for the head nerd(s) around here, being one myself...but I'm curious why the forums are blocking search engines from indexing content? The robots.txt excludes Google and all others from crawling the site, forcing searches to be done here.

Certainly the prerogative of those running the site, and the native search capabilities are better than many places I've been to. Still, I've yet to find any forum software search function that works as well as Google - especially when doing advanced searches.

Otherwise, thanks much to those behind the move from the Yahoo forums. We're ex-Lazy Daze owners, had to sell our 24TK a while back due to other obligations. We're ready to dive back in, and are happy to see this new home.

Bill
2011 RB
Re: Search engines blocked
Reply #1 –
To eliminate many problems, LDOF is a private, members-only site.
This prevent the robots from searching the site... until they figure out how to join.
Other forums whose postings show up on Google searches are usually from forums open to viewing by non members .

FYI, I often search the old Yahoo LD site, finding it's search engine to work better.
This is one of the reasons why the old forum is still in existence and has not been totally shut down.

Larry

Larry
2003 23.5' Front Lounge, since new.  Previously 1983 22' Front Lounge.
Tow vehicles  2020 Jeep Wrangler Rubicon, 2001 Jeep Cherokee
Photo Collection: Lazy Daze
Re: Search engines blocked
Reply #2 –
Quote
finding it's search engine to work better
Example please.    To be fair, you should select to only search in the Yahoo Archive when looking for just that.

One thing to keep in mind is when using search, use the full function http://www.lazydazeowners.com/index.php?action=search and not just the quick search at the top right, as it has many more options (by member, by board, by date, subject only, etc)
Re: Search engines blocked
Reply #3 –
Quote from: Steve C  - 
One thing to keep in mind is when using search, use the full function http://www.lazydazeowners.com/index.php?action=search and not just the quick search at the top right, as it has many more options (by member, by board, by date, subject only, etc)

Steve

I shouldn't have said it that Yahoo search works better. I should have said that there have been times when I have been unable to find an old posting, using the LDOF search engine's 'advanced' search, where going next to Yahoo search brought it up. I have no idea why.

I'm reasonably good at searching and this is just another tool in the box, that has worked for me.
Naturally, this only works on pre-LDOF postings.

Sorry for any misinformation.

Larry

Larry
2003 23.5' Front Lounge, since new.  Previously 1983 22' Front Lounge.
Tow vehicles  2020 Jeep Wrangler Rubicon, 2001 Jeep Cherokee
Photo Collection: Lazy Daze
Re: Search engines blocked
Reply #4 –
Speaking of members-only, I tried to look at someone's tool list PDF from the Yahoo site.  I get as far as "Oops.  You need to be a member to perform this action." from Yahoo.  Do I need to sign up for that, too, even though it's theoretically replicated here?
Same deal when trying to access from The Companion's link, too.
It's Larry's list from here:  Tool kit ideas for fulltimers

Chip
2000 Front Lounge
Re: Search engines blocked
Reply #5 –
Larry,

Thanks for the details.  I thought it may have been a "relevance" issue that I could look at over time.  There are a number of ways the relevance and weighting of a search term are handheld, its both location (LCS) in the document and density of usage (BM25) in the document that drive a results overall weight. 

Since I was looking, I did enable a basic morphology stemming preprocessor that may help a bit, it allows for some reduction of search words to a valid natural language root word. For example, "running" may be reduced to "run" during the search.

Anyway ranking /relevance is often in the eye of the searcher anyway.

Quote from: Chip Chester  - 
Speaking of members-only, I tried to look at someone's tool list PDF from the Yahoo site.  I get as far as "Oops.  You need to be a member to perform this action." from Yahoo.  Do I need to sign up for that, too, even though it's theoretically replicated here?
Same deal when trying to access from The Companion's link, too.
It's Larry's list from here:  Tool kit ideas for fulltimers

Chip

I think that means you need a yahoo account and have to have "joined" that group.

 
Re: Search engines blocked
Reply #6 –
Quote from: Steve C  - 
Larry,

Thanks for the details.  I thought it may have been a "relevance" issue that I could look at over time.  There are a number of ways the relevance and weighting of a search term are handheld, its both location (LCS) in the document and density of usage (BM25) in the document that drive a results overall weight. 

Since I was looking, I did enable a basic morphology stemming preprocessor that may help a bit, it allows for some reduction of search words to a valid natural language root word. For example, "running" may be reduced to "run" during the search.

Anyway ranking /relevance is often in the eye of the searcher anyway.
I think that means you need a yahoo account and have to have "joined" that group.
"Since I was looking, I did enable a basic morphology stemming preprocessor that may help a bit, it allows for some reduction of search words to a valid natural language root word."

Every time I read tech talk like this, I am so grateful we have Steve Cook ministering to this forum!  ;D

Chris
Formerly: 2002 30' IB