Bug #78808 [Com]: [LMDB] MDB_MAP_FULL: Environment mapsize limit reached

Edit report at https://bugs.php.net/bug.php?id=78808&edit=1 ID: 78808 Comment by: progr-d at yandex dot ru Reported by: progr-d at yandex dot ru Summary: [LMDB] MDB_MAP_FULL: Environment mapsize limit reached Status: Open Type: Bug Package: DBM/DBA related Operating System: macOS 10.13.6 PHP Version: 7.3.11 Block user comment: N Private report: N New Comment: I can send a patch that will add mdb_env_set_mapsize() at the specified location, but with a fixed size (e.g. 1GB). But as far as I understand it is much more correct to give users the opportunity to change this behavior themselves (through the fourth argument). And for this, you already need to write tests (in which I am not strong for PHP tests). For macOS homebrew users I made hotfix (hacker-way style), which increases the size of the database in the library itself (by the way, there is an error - it really is 1MB in size, and the documentation says 10MB): https://gist.github.com/saippuakauppias/95b6f7b139b3d117b7dac4aeead9f5b4 Previous Comments: ------------------------------------------------------------------------ [2019-11-13 13:34:50] cmb@php.net > contains the fourth argument, but nowhere is there a description > of how to use it Arguments following the $handler are converted to string and are passed to the individual DBA drivers, and are driver dependent. This is indeed not yet documented; let alone which drivers support which parameters. > It turns out that this needs to be done here: Yes. info->argc and info->argv provide the driver specific arguments which have been passed to dba_open(). ------------------------------------------------------------------------ [2019-11-13 11:46:03] progr-d at yandex dot ru Yes, I found a description about this: > The size should be a multiple of the OS page size. The default is 10485760 bytes. The size of > the memory map is also the maximum size of the database. The value should be chosen as large as > possible, to accommodate future growth of the database. This function should be called after > mdb_env_create() and before mdb_env_open(). It may be called at later times if no transactions are > active in this process. Note that the library does not check for this condition, the caller must > ensure it explicitly. (from http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5 ) It turns out that this needs to be done here: https://github.com/php/php-src/blob/PHP-7.2.0/ext/dba/dba_lmdb.c#L61 --------------- Strange thing: 10485760 bytes is 10MB, but my Test script create file only with 1MB size (you can check it). --------------- > there likely should be some dba_open() argument dba_open ( string $path , string $mode [, string $handler [, mixed $... ]] ) contains the fourth argument, but nowhere is there a description of how to use it: https://www.php.net/manual/en/function.dba-open.php ------------------------------------------------------------------------ [2019-11-13 11:30:26] cmb@php.net From the docs[1]: | Default size of memory map. This is certainly too small for any | actual applications. Apps should always set the size explicitly | using mdb_env_set_mapsize(). So yes, there likely should be some dba_open() argument to allow for a custom map size. [1] <http://www.lmdb.tech/doc/group__internal.html#gac08cad5b096925642ca359a6d6f0562a> ------------------------------------------------------------------------ [2019-11-13 10:21:18] progr-d at yandex dot ru Description: ------------ Presently php not provide any way to change mapsize for lmdb databases and they limited by size. How can I fix this? Now, probably, I can only recompile the lmdb library... PS: build php in macos with lmdb support: https://gist.github.com/saippuakauppias/fac2a33dfe420adb4a6cf19c257eef46 Test script: --------------- <?php $lmdb_h = dba_open('test.db', 'c', 'lmdb'); for ($i = 0; $i < 50000; $i++) { dba_insert('key' . $i, 'value '. $i, $lmdb_h); } dba_close($lmdb_h); Actual result: -------------- ➜ php prefill_lmdb.php PHP Warning: dba_insert(key18505,value 18505): MDB_MAP_FULL: Environment mapsize limit reached in /Users/usr/prefill_lmdb.php on line 6 Warning: dba_insert(key18505,value 18505): MDB_MAP_FULL: Environment mapsize limit reached in /Users/usr/prefill_lmdb.php on line 6 [1] 42370 segmentation fault php prefill_lmdb.php ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=78808&edit=1