Re: Search engine

It's not that complicated, after crawling your website at a specific time you can check for content created in your database after that time. Then you can search specific fields from that content for words and index them. If you need to find <a href=""> tags you'ld best use regular expressions to search through the text and then follow those links. You can keep an index of links crawled and date they were crawled to ensure you don't crawl pages that haven't changed. I wrote a crawling engine for bittorrent site so if you need some sample code let me know. Ron Piggott (PHP) wrote:

What I haven't figured out is how to crawl my site. If I knew how to crawl my site I could write a basic search engine and develop it over time. This is the crucial issue I need help with. I have a really basic search engine I wrote last year. I told my web site what pages the user could search ... as in I keyed in the URL's I am giving my site a face lift and making some serious changes to it ... I have my database out of date ... I would like to crawl my site once a day to update what changes I made during the past 24 hours and delete HTML files which no longer exist. If someone knew how to retrieve various <a href=""> then I could start with something and build on it over time ... Ron