Re: How to jump to line number in large file

Robert Cummings wrote:

On Sat, 2008-04-05 at 19:09 +0100, Steve McGill wrote:

"Richard Heyes" <richardh@phpguru.org> wrote in message news:47F75D2A.3020101@phpguru.org...

Thanks for the heads up on fgetc() incrementing by one. I hadn't actually tested that code yet, I was using the original fseek($handle,$pos). strpos would be ideal but it needs to work on a string and not a file - I don't want to load a 100Mb file into memory if I don't have to. Perhaps I should test how quick the fgets() and ftell() method is because at least it loads in one line at a time. Does anybody know any other ways to go about the problem?

Haven't read the rest of the thread, and so going by the subject alone, fgets() finishes when it encounters a newline, so you can use this wondrous fact to seek to a specific line: <?php

    $fp  = fopen('filename', 'r');
    $num = 18; // Desired line number

    for ($i=0; $i<$num; $i++)
        $line = fgets($fp);

    echo $line;

?> It works because fgets() stops when it encounters a newline (\n). So it's just a case of counting the calls to fgets().

fgets() would work but as I'm constantly jumping around a 500,000 line file I thought it was better to maintain a cache of line number positions. As a final update to anybody following: - Taking away the unnecessary fseek() made the script execute in 63 seconds - Using a buffer system, (reading in 1Mb of the text file at a time and then looping through the string in memory) made the script execute in 36 seconds. Huge improvement, but... - Porting the code to C++, doing a shell_exec and reading the results back in to PHP, took less than 2 seconds. As fgetc() etc are all effectively C wrappers I was quite surprised at the speed increase....

It really depends on how you write your code... I ran the following script on a 150 meg text log file containing 1905883 lines in 4 seconds (note that it performs caching). Here's the script: <?php $path = $argv[1]; if( ($fPtr = fopen( $path, 'r' )) === false ) {

    echo "Couldn't open for reading: $path\n";
    exit();

} $line = 1; $lines[$line] = 0; while( fgets( $fPtr ) !== false ) {

    $lines[++$line] = ftell( $fPtr );

}

couldn't you get away from incrementing a counter variable by simply starting the array at index #1 ?? $lines[1] = 0; while( fgets( $fPtr ) !== false ) {

    $lines[] = ftell( $fPtr );

} Wouldn't this make it faster?

fclose( $fPtr ); ?> Here's the run times on several iterations (Athlon 2400+):

real    0m4.065s
user    0m3.488s
sys     0m0.464s

real    0m4.005s
user    0m3.464s
sys     0m0.436s

real    0m5.816s
user    0m3.336s
sys     0m0.536s

real    0m3.994s
user    0m3.384s
sys     0m0.504s

real    0m4.069s
user    0m3.512s
sys     0m0.444s

real    0m4.009s
user    0m3.344s
sys     0m0.552s

Cheers, Rob.