note 78693 rejected from function.file-get-contents by james

Note Submitter: Harvie ---- Hi everybody! I am trying to write WebCrawler/Spider (as school project, and -of course- I am trying to be more wealthy than google ;o) So... I have big/small problem: I am using file_get_contents() (i've tryed fopen() too...). Crawler works 100% great, but sometimes it freezing. I have tryed to trace what function freezes, and i found it, it's file_get_contents()... So, i googled and found default_socket_timeout setting, i set it to 1, but sometimes its freezes and never get up again. I've done this example, so you can see, that it freezes after few iterations. I have supplyed URL, that causes freeze of my crawler (im not sure why...): #!/usr/bin/php <?php ini_set('default_socket_timeout',1); set_time_limit(0); //$url='http://ad.doubleclick.net/click'; $url='http://w.moreover.com/'; while(1) { @file_get_contents($url, false, null, 0, 10000); echo "#"; } ?> Of course, if somebody want be better than google, he have to have very good crawler. So I need very solid code, that can run and crawl a days without crash (like this one). Yeah, it's true, that this worked 1 or 2 hours before it crashes, or i stoped it, but the file_get_contents() doesn't work like i need. If you are interested to crawling, you can write me to YouSeekMe: 283//782//978 ;D And there are few statistics from my last session: +5431 URLs; 19292 Downloaded; UpTime: 21.5 mins; Buffered: 30 URLs; History: 1000 URLs; Speed: 4.22 URLs/s, 14.98 Downloads/s THX4AnyHelp ;o)