Re: Re: refernces, arrays, and why does it take up so much memory?

On 9/3/2013 1:09 AM, Daevid Vincent wrote:

-----Original Message----- From: Jim Giner [mailto:jim.giner@albanyhandball.com] Sent: Monday, September 02, 2013 8:14 PM To: php-general@lists.php.net Subject: [PHP] Re: refernces, arrays, and why does it take up so much memory? On 9/2/2013 9:30 PM, Daevid Vincent wrote:

I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want

to

do

is make the array key the "id" of the result set row. This is the basic gist of it:

         private function _normalize_result_set()
         {
                foreach($this->tmp_results as $k => $v)
                {
                       $id = $v['id'];
                       $new_tmp_results[$id] =& $v; //2013-08-29 [dv]

using

a

reference here cuts the memory usage in half!

                       unset($this->tmp_results[$k]);

                       /*
                       if ($i++ % 1000 == 0)
                       {
                             gc_enable(); // Enable Garbage Collector
                             var_dump(gc_enabled()); // true
                             var_dump(gc_collect_cycles()); // # of

elements

cleaned up

                             gc_disable(); // Disable Garbage Collector
                       }
                       */
                }
                $this->tmp_results = $new_tmp_results;
                //var_dump($this->tmp_results); exit;
                unset($new_tmp_results);
         }

Without using the =& reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552)

    6904 =>
      array (size=4)
        'id' => string '6904' (length=4)
        'studio_id' => string '5' (length=1)
        'genres' => string '34|' (length=3)
    6905 =>
      array (size=4)
        'id' => string '6905' (length=4)
        'studio_id' => string '5' (length=1)
        'genres' => string '6|37|' (length=5)

However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd

EXPECT

it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552)

    6904 => &
      array (size=4)
        'id' => string '86260' (length=5)
        'studio_id' => string '210' (length=3)
        'genres' => string '8|9|10|29|58|' (length=13)
    6905 => &
      array (size=4)
        'id' => string '86260' (length=5)
        'studio_id' => string '210' (length=3)
        'genres' => string '8|9|10|29|58|' (length=13)

Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this->tmp_results as $k => $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All

I

can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results

array

in

a tricky way to avoid a second array. The concept being I'd add 1

million

to the ['id'] (which we want as the new array key), then unset the

existing

sequential key, then when all done, loop through and shift all the keys

by

1

million thereby they'd be the right index ID. So add one and unset one immediately after. Clever right? 'cept it too made no difference on

memory.

Same thing is happening as above where the gc() isn't running or

something

is holding all that memory until the end. *sigh* Then I tried a different way using array_combine() and noticed something very disturbing. http://www.php.net/manual/en/function.array-combine.php

         private function _normalize_result_set()
         {
                if (!$this->tmp_results || count($this->tmp_results) < 1)

return;

                $D_start_mem_usage = memory_get_usage();
                foreach($this->tmp_results as $k => $v)
                {
                       $id = $v['id'];
                       $tmp_keys[] = $id;

                       if ($v['genres'])
                       {
                              $g = explode('|', $v['genres']);
                             $this->tmp_results[$k]['g'] = $g; //this

causes a

massive spike in memory usage

                       }
                }
                //var_dump($tmp_keys, $this->tmp_results); exit;
                echo "\nMEMORY USED BEFORE array_combine:

".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK: (".number_format(memory_get_peak_usage(true)).")<br>\n";

                $this->tmp_results = array_combine($tmp_keys,

$this->tmp_results);

                echo "\nMEMORY USED FOR array_combine:

".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK: (".number_format(memory_get_peak_usage(true)).")<br>\n";

                var_dump($tmp_keys, $this->tmp_results); exit;
         }

Just the simple act of adding that 'g' variable element to the array

causes

a massive change in memory usage. WHAT THE F!? MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264) MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264) And taking out the $this->tmp_results[$k]['g'] = $g; Results in MEMORY USED BEFORE array_combine: 8,050,456 PEAK: (78,118,912) MEMORY USED FOR array_combine: 8,050,376 PEAK: (86,507,520) Just as a wild guess, I also added 'g' to my SQL so that PHP would

already

have a placeholder variable there in tmp_results, but that made no difference. And still used up nearly double the memory as above. SELECT DISTINCT id, sag.genres, 'g' FROM.

Are you sure that the data is what you expect? I've never used an object to hold the results of a query, but I'm picturing that your foreach may not be working at all unless an object as a result is totally different than the type of query results I always process. You're taking each object of the query results and assigning the key and value to vars. But aren't query results 'keyless'? So what are you actually assigning to $k and $v? If I'm wrong and using an object is quite different, forget I said anything. :)

Positive the data is right. It all works. It's just the memory usage is the issue. I'm not assigning an object. I'm assigning an array to a property (array) IN an object ($this). The query itself is using mysql_fetch_array() and MYSQL_ASSOC http://php.net/manual/en/function.mysql-fetch-array.php Results are certainly NOT keyless as you wouldn't be able to use them otherwise. They are a sequential multi-dimensional array starting at [0] ... [x] with column names as the sub-array hash keys and column values as the corresponding hash value. Hard to know that from your code sample.

Please ignore my my post. :) Although - why not just use the query results instead of first putting them into an array only to put some of them (?) into a second array?