Emergency! Performance downloading big files
| From: | Brian Dunning | Date: | Tue, 01 Dec 2009 22:48:45 +0000 |
| Subject: | Emergency! Performance downloading big files | ||
| Groups: | php.general | ||
| Request: | Send a blank email to php-general+get-300187@lists.php.net to get a copy of this message | ||
This is a holiday-crunch emergency.
I'm dealing with a client from whom we need to download many large PDF docs 24x7, several
thousand per hour, all between a few hundred K and about 50 MB. Their security process requires the
files to be downloaded via https using a big long URL with lots of credential parameters.
Here's how I'm doing it. This is on Windows, a quad Xeon with 16GB RAM:
$ctx = stream_context_create(array('http' => array('timeout' => 1200)));
$contents = file_get_contents($full_url, 0, $ctx);
$fp = fopen('D:\\DocShare\\'.$filename, "w");
$bytes_written = fwrite($fp, $contents);
fclose($fp);
It's WAY TOO SLOW. I can paste the URL into a browser and download even the largest files quite
quickly, but the PHP method bottlenecks and cannot keep up.
Is there a SUBSTANTIALLY faster way to download and save these files? Keep in mind the client's
requirements cannot be changed. Thanks for any suggestions.