RE: Mnogo indexing test results, status report

Do you have search results tests? Is mnogosearch really faster than htdig? Just for information purposes. Personally I don't believe that the mirror admins (like myself) are going to object about the size of the database. Since it is 100MB less maybe there is no point of wasting time to create stop words. If the search time is good then we should probably just go for it. Maybe we should just take Shimi's guide and your tests and merge them to create a manual on how to set it up. It will take some time to finalize the guide since people have to test it on different platforms and for example I want to test it with two different mnogo dbs on the same box. Spyros -----Original Message----- From: Heilig (Cece) Szabolcs [mailto:cece@phphost.hu] Sent: Sunday, June 01, 2003 1:56 PM To: php-mirrors@lists.php.net Cc: shimi@shimi.net; rasmus@lerdorf.com; goba@php.net; jacques@php.net Subject: Mnogo indexing test results, status report Hello! I ran a full indexing with Mnogo last night. The test environment is a normal webserver, hu2.php.net ======================================================= cpuinfo: Intel(R) Pentium(R) 4 CPU 1.50GHz meminfo: MemTotal: 512960 kB SwapTotal: 779112 kB db backend: mysql-max-3.23.52 db type: crc-multi character encoding: utf-8 disallowed urls: - robots.txt contents - */print/* - */printwn/* - */manual/howto/* - */cal.php* - */search.php* - */from/a/mirror/* - */manual/add-note.php* - */source.php* - *.pdb *.ico *.chm *.js *.rss And the results are: ==================== total indexing time: 49 332 seconds = 13.703 hours urls in database: 72 478 total mysql db size: 1.1G (htdig was 1.2G) Used mysql tables: ================== table | rows | size =================================== ndict2 | 2 867 599 | 93.1 MB ndict3 | 5 089 785 | 165.5 MB ndict4 | 5 210 276 | 169.3 MB ndict5 | 3 818 095 | 124.0 MB ndict6 | 4 277 167 | 139.0 MB ndict7 | 4 132 472 | 134.4 MB ndict8 | 2 272 980 | 73.9 MB ndict9 | 1 628 203 | 52.8 MB ndict10 | 896 741 | 29.1 MB ndict11 | 721 978 | 23.5 MB ndict12 | 293 370 | 9.5 MB ndict16 | 494 637 | 16.1 MB ndict32 | 204 808 | 6.7 MB url | 72 478 | 76.3 MB That sizes can be lowered with good stopwords files. For example there are no hungarian stopwords file. Stopwords are words that indexer is not storing. For example in english: "the, a, and, but". But i think that words have to be removed from english stopwords list: "while, even, try, for, each". So, we have to distribute special stopwords too, not just stopwords.conf. Now i'm playing with php-mnogosearch on my mirror. I'm preparing to reorganize mnogosearch's results with PHP to show only the primary language version of a page function.mysql_connect.php for example. And there list all possible language variant, linking to manual/lang/samedocument.php. -- ============================================================= Heilig (Cece) Szabolcs - cece@php.net - http://www.phphost.hu =============================================================