Bug #13598 Updated: Arrays taking up to much memory

ID: 13598 Updated by: jeroen Reported By: renze@datalink.nl Old Status: Open Status: Bogus Bug Type: Arrays related Operating System: Debian 2.2.17 PHP Version: 4.0CVS-2001-10-08 New Comment: 9040*84=768400. So you've got 759360 bytes of tabs, and about the same amount of data. So each field in your file is probably about 1 byte large. PHP doesn't support 2D arrays, as you can read in the manual, rather, it is possible to have an array as a value of another array, thus faking multidimensional arrays. So, when you want a 2D array, you'll need 9040 array's, each containing 85 1-byte values. In any case, you need at least 768400 PHP variables for the strings themselves. Dividing 22M by 768400, it turns out that about 28 bytes is used for each variable, and that's not extremely much (12 bytes for the variable itself, 8 bytes for the reference from the array, and 2 bytes for the data alone already makes 22 bytes... PHP is a language meant to support a lot of different uses, which has it's drawback on memory. Usually this isn't noticable, but when you try to read 3/4 of a million single bytes in memory... I don't know what the layout of your file is, and what you want to do with it, but I suggest to use a smarter approach to save the data in memory. Ask php-general for hints. Previous Comments: ------------------------------------------------------------------------ [2001-10-08 10:15:01] renze@datalink.nl I'm trying to process a tab separated file. This file is about 1.5M large, contains 9040 lines that each contain 85 "columns". Very happy with the fgetcsv() function I started working on this. Then, when I finished, I didn't get my wanted requested output, but only errors in my logfile. "memory exhausted". So I started trying all kinds of stuff. The piece of script I use for the 'processing' is like: if (!($fd = fopen ($file, "r"))) { // Some error and exit } $data = array(); $total = array(); while ($data = fgetcsv ($fd, 580, $delimiter)) { array_push ($total, $data); } if (!fclose ($fd)) { // Some error and exit } Now I've configured Apache to log the peak memory usage of the script and guess what?! 28M!!! It's take up 28M to read a file of hardly 1.5M! Really weird. So I tried something else. $contents = file ($file); No problem! Memory usage: 2M. So, I thought, maybe there's some bug in fgetcsv(). Let's try it myself: $contents = file($file); $total = array(); $record = array(); foreach ($contents as $line) { $record = split ("\t", $line); array_push ($total, $record); } Memory usage: 28M. So that's not the solution. So... it might just be that array_push doesn't work correct. Next try: if (!($fd = fopen ($file, "r"))) { // Some error and exit } $data = array(); $total = array(); $counter = 0; while ($data = fgetcsv ($fd, 580, $delimiter)) { $total[$counter++] = $data; } if (!fclose ($fd)) { // Some error and exit } Memory usage: 24.7M So that's no solution neither. Then I've tried the following change to the while-loop. while ($total[] = fgetcsv ($fd, 580, $delimiter)) { } No luck! Same memory usage. Okay... well... that one was predictable. So... Then I came up with the following very nasty solution: if (!($fd = fopen ($file, "r"))) { // Some error and exit } $data = array(); $counter = 0; while ($data = fgetcsv ($fd, 580, "r")) { $var_name = "myVar_$counter"; $$var_name = $data; $counter++; } if (!close ($fd)) { // Some error and exit } Nasty ain't it?! Didn't work either. Memory usage: 24.8M. So quickly threw that one out again :) I've tried several combinations of the above constructions. None worked correctly. Memory usage way to high! Some other experiment: if (!($fd = fopen ($file, "r"))) { // Some error and exit } $data = array(); $total = array(); while ($data = fgetcsv ($fd, 580, $delimiter)) { array_push ($total, implode ("\t", $data)); } if (!fclose ($fd)) { // Some error and exit } Yep. That worked. Memory usage: 2M. But, well... it's the same result as just: $total = file ($file). That's not the way! So... well... I've tried about everything, but everything that produces the correct result (a 2D array with all the 'records' in it) also produces a memory usage that way to high. Anyone knows where this bug comes from? *R&zE: Btw... PHP version 4.0.8-dev: --prefix=/usr/local/php --with-config-file-path=/usr/local/php/etc --with-exec-dir=/usr/local/php/safe --with-apxs=/usr/local/Apache/bin/apxs --without-mysql --with-solid=/home/solid --with-pgsql=/usr/local/pgsql --with-pdflib=/usr/local --with-db3 --enable-ftp --with-mm --with-zlib --with-bz2 --with-openssl --with-gd --enable-gd-native-ttf --with-jpeg-dir --with-png-dir=/usr --with-zlib-dir=/usr --with-xpm-dir=/usr/X11R6 --with-ttf --with-t1lib --with-pcre-regex --enable-sysvsem --enable-sysvshm --enable-memory-limit --enable-inline-optimization --enable-versioning Apache version 1.3.14 ------------------------------------------------------------------------ Edit this bug report at http://bugs.php.net/?id=13598&edit=1