Mnogo indexing test results, status report

Hello! I ran a full indexing with Mnogo last night. The test environment is a normal webserver, hu2.php.net ======================================================= cpuinfo: Intel(R) Pentium(R) 4 CPU 1.50GHz meminfo: MemTotal: 512960 kB SwapTotal: 779112 kB db backend: mysql-max-3.23.52 db type: crc-multi character encoding: utf-8 disallowed urls: - robots.txt contents - */print/* - */printwn/* - */manual/howto/* - */cal.php* - */search.php* - */from/a/mirror/* - */manual/add-note.php* - */source.php* - *.pdb *.ico *.chm *.js *.rss And the results are: ==================== total indexing time: 49 332 seconds = 13.703 hours urls in database: 72 478 total mysql db size: 1.1G (htdig was 1.2G) Used mysql tables: ================== table | rows | size =================================== ndict2 | 2 867 599 | 93.1 MB ndict3 | 5 089 785 | 165.5 MB ndict4 | 5 210 276 | 169.3 MB ndict5 | 3 818 095 | 124.0 MB ndict6 | 4 277 167 | 139.0 MB ndict7 | 4 132 472 | 134.4 MB ndict8 | 2 272 980 | 73.9 MB ndict9 | 1 628 203 | 52.8 MB ndict10 | 896 741 | 29.1 MB ndict11 | 721 978 | 23.5 MB ndict12 | 293 370 | 9.5 MB ndict16 | 494 637 | 16.1 MB ndict32 | 204 808 | 6.7 MB url | 72 478 | 76.3 MB That sizes can be lowered with good stopwords files. For example there are no hungarian stopwords file. Stopwords are words that indexer is not storing. For example in english: "the, a, and, but". But i think that words have to be removed from english stopwords list: "while, even, try, for, each". So, we have to distribute special stopwords too, not just stopwords.conf. Now i'm playing with php-mnogosearch on my mirror. I'm preparing to reorganize mnogosearch's results with PHP to show only the primary language version of a page function.mysql_connect.php for example. And there list all possible language variant, linking to manual/lang/samedocument.php. -- ============================================================= Heilig (Cece) Szabolcs - cece@php.net - http://www.phphost.hu =============================================================