Re: searchengine in php

On Sat, 28 Oct 2000 13:32:34 -0700, Joe Stump (joe@earth.care2.com) wrote: >I don't have code laying around that I'd be willing to share with >everyone, but >I will explain a simple concept that most people would enjoy hearing >about. > >Making a simple search engine with relevancy is actually easier than >it sounds. >First let me describe some terms: > >1.) "Stop Words" - words that the search engine will ignore (ie >"the") >2.) "Relevancy" - an equation used to determine how "relevant" a >document is > to a given search. >3.) "Index" - a table that usually holds wordID, documentID, and a >relvency code > relating that word and that document. > >An industry standard for simple search engines is a "btree index" or >"reverse >index". What you do is take your text fiels that you want to index >(ie docTitle >and docBody) and run them through and indexing script - mine usually >take a >document, create a list of words in that document, strip out all >punctuation, >ignore stop words, count up the number of instances of each word in >that >document, and then insert the result in my index. Before this you >will want to >have created a "dictionary" of words from your documents - meaning a >table with >word, wordID and consisting of all the unique words in your >documents. > >After you have created your index you can then search for "dog" and >all you have >to do to get all of those documents with dog in them is look up the >wordID for >"dog" and do a join in your index table with that wordID and the >documents that >have that word in it - you can order by "hits" or "instances" for >simple >relevancy also. > >I have done this on DMOZ (www.dmoz.org) with PHP/MySQL and I know it >scales >beyond 1.7 Million documents - so it works well. or you could just drop all your data into a MySQL table with a FULLTEXT index on the keyword field and all this gets taken care of for you. this is definitely an easier way to go. - Mark