New Search Engine Development (Was: Re: [PHP] [PHP-GENERAL] Search engine development)

My company, Jatosoft LLC, is working on a new kind of search engine designed for small to medium sized sites. You simply provide the engine with a list of URLs, and it indexes them. No robots, spiders, etc. are necessary. Also, it sends page requests via HTTP ( just like a web browser ) so if there is dynamic content on your pages, it will get indexed properly. Granted, listing every URL to be searched can get to be a pain on larger sites, but it is great for small to medium-sized sites. For example, a small business selling PCs could allow their site, as well as the site of their supplier(s), to be searched from the same location. Also, the webmaster doesn't have to go around adding robots.txt files to every directory - if you do not want a page indexed for searching, simply omit it from the URL list. Currently, we are working on adding relevance ranking to this search engine. We are also looking into indexing selective portions of a page's HTML - all the <img> tags, for example. This would allow searching for a particular image on a site by looking for its filename. We are using PHP4 and MySQL to store the indexes. A benefit of this is that advanced SQL syntax can be used for searching; for example, the soundex() function in MySQL will find words that sound alike ( John and Jon ). We would like to make this search engine available for public consumption when it nears completion. We plan to provide hosting on our servers as well as allowing others to download it and install it on their servers. If you'd like more information, please email me at ben@jatosoft.com. You may also visit http://search.jatosoft.com/ ( although this page hasn't been updated in a while, it will be soon ). If any searching gurus have suggestions about relevance algorithms, I'd love to hear them :). It would be cool if we could build in several different relevance schemes, and allow the webmaster ( or even the user ) to choose which one they prefer. Ben Gollmer Jatosoft, LLC http://www.jatosoft.com/