Re: Search engine development

Hiya Front-end isn't your big problem, more the indexer (if it indexes the right way, your frontend won't be that much work (ecept layout)) First you have to decide what you want in your database.. possible options (my things): - all pages striped of html java etc. - all words with counters to pages (one big index) - the pages you have/will index (manual, spider, submit) - stats for words - timestamps and checksum to reindex good - etc. things to think good of: - database layout - keys - speed things i have... php4 with mysql apache linux wrote my own http class to retrieve the pages of the servers (php doesn;t support max page size timeouts and headers) wrote my own database class (not nessacery but i allready had it :) i have made some functions to detect if a page is a error page (do NOT relay on the 404 page, because if for example apache has a custom 404 page it will give you status 200 @#$*@) i even made it detect custom/dynamic error pages (kinda like this: 404 page <some_page> doesn;t exsist) i show the searcher a part of the page to show the context of the word they where looking for speed, i curently track 220 servers completely (there submited and check by human to ensure the comes no garbage in the db and to make it index all in on week) my indexer indexes 2 pages/sec (with all bells surrounding it) feel free to take a look @ http://www.zoekzorg.nl Any more questions feel free to ask P.S. i did 6 months to make it (rebuild it 3 times for speed, changed layout more then i liked (@)*($#@ boss ;)) so i hope you got a long breath I don't know perl, but i could recommend php, we have a small server (p450-256mb) and it keeps all fresh in a week Greetz, Wico At 06:19 13-7-00 -0700, Thomas Edison Jr. wrote:

Wico, What help do you want? What are you going to build an indexer or the front end or both? Well we're designing both, front end & an indexer. First of all we want to know where to begin? What kind of stuff do we actually require to make a search engine? Let me give u a fair idea of what i have in mind. A database with few coloumns like keywords, site description etc. Then the search field picking up stuff from the database as per the query entered by the user. Something like yahoo - which i believe is totally made in perl. Now hwo to go about this really. Thanx a lot for this. Regards, T. Edison jr. ===== Rahul S. Johari (Director) ****************************************** Abraxas Technologies Inc. Homepage : http://www.abraxastech.com Email : abraxastech@yahoo.com Tel : 91-4546512/4522124 ******************************************* __________________________________________________ Do You Yahoo!? Get Yahoo! Mail ­ Free email you can access from anywhere! http://mail.yahoo.com/ -- PHP General Mailing List (http://www.php.net/) To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net For additional commands, e-mail: php-general-help@lists.php.net To contact the list administrators, e-mail: php-list-admin@lists.php.net