Categorizing search results

Hi I've looked into improving the search to also provide categorization and descriptions. There is a minor issue with the Docbook sources, not all <refentry>s are actually methods/functions, and there is minor lack of meta info, but thats something we can "easily fix". The index is generated by PhD after rendering the docs (takes less then a second so thats fine). It dumps 2files; the entire index and a description file. Rather then forcing all mirrors to update their configs, we output these files from /search-index.php which compresses the files (if the browser supports it), and will (in the future) add proper cache headers. To test the descriptions, I added a sleep(5) to it.. just for proofing that the search will work fine without having retrieved the descriptions, and once downloaded, they will popup on your next keystroke. The patch only iterates through the variables, classes, exceptions, and function/methods, but the index contains a lot of more stuff. Attached is the patch for phpweb & PhD, and as a bonus I've attached the generated index files in case you don't have PhD installed :P The index files should be located under /manual/en/ I killed the actual order-by-score, and also lowered the minScore value (didn't work so great for long function/variable names where you typed only few characters..). That should probably be updated if we decide to commit this. After applying the patch, and throwing the index files into correct dir, try searching for 'arg', 'server', or prettymuch whatever :] What do you guys think, is it worth perusing? Its not searching in the descriptions.. Unsure if its worth it.. Maybe provide an option in my.php search in: [x] function/method names [x] classes [ ] foo [ ] bar [ ] exceptions And chunk the index files into index.functions.json, index.classnames.json, .., and only push the data the user wants to search in? The entire thing is only 270kb, compressed, with descriptions.. so chunking the json file itself may not be worth it, but providing the searching options could be nice though? -Hannes

Index: styles/structure.css =================================================================== --- styles/structure.css (revision 306812) +++ styles/structure.css (working copy) @@ -178,7 +178,7 @@ * Auto-complete */ .ui-autocomplete { - width: 187px; + max-width: 350px; max-height: 250px; overflow-y: auto; overflow-x: hidden; @@ -207,12 +207,21 @@ */ line-height: 16px; overflow: hidden; + border-bottom: 1px solid #c3add9 ; } .ui-autocomplete a { display: block; padding: 2px 5px; } +.ui-autocomplete-category { + font-weight: bold; + padding: .2em .4em; + margin: .8em 0 .2em; + line-height: 1.5; + background-color: #987db3; + color: #fff; +} aside.tips { Index: search-index.php =================================================================== --- search-index.php (revision 0) +++ search-index.php (revision 0) @@ -0,0 +1,23 @@ +<?php +require dirname(__FILE__) . "/include/languages.inc"; + +$langcode = language_convert(isset($_GET["lang"]) ? $_GET["lang"] : "en"); + +if (isset($_GET["description"])) { + $filename = "search-description"; + $varname = "descriptionIndex"; + sleep(5); +} +else { + $filename = "search-index"; + $varname = "searchIndex"; +} + +header("Content-Type: application/javascript"); +if (strpos($_SERVER["ACCEPT_ENCODING"], "gzip") !== false) { + header("Content-Encoding: gzip"); + ob_start("ob_gzhandler"); +} +echo "$varname = "; +readfile(dirname(__FILE__) . "/manual/$langcode/$filename.json"); + Index: js/common.js =================================================================== --- js/common.js (revision 306812) +++ js/common.js (working copy) @@ -44,35 +44,71 @@ }); // load the search index and enable auto-complete. - jQuery.getScript("/js/search-index-" + getLanguage() + ".js", function(){ - $('#headsearch-keywords').autocomplete({ + jQuery.getScript("/search-index.php?lang=" + getLanguage(), function(){ + var haveDesc = 0; + jQuery.getScript("/search-index.php?lang=" + getLanguage() + "&description=1", function(){ + haveDesc = 1; + }); + $.widget("custom.catcomplete", $.ui.autocomplete, { + _renderMenu: function(ul, items) { + var self = this, currentCategory = ""; + $.each(items, function(index, item) { + if (item.category != currentCategory) { + ul.append("<li class='ui-autocomplete-category'>" + item.category + "</li>"); + currentCategory = item.category; + } + self._renderItem(ul, item); + + }); + }, + _renderItem: function(ul, item) { + var n = $("<li></li>").data("item.autocomplete", item); + if (item.desc) { + n.append("<a>" + item.label + "<br>" + item.desc + "</a>"); + } + else { + n.append("<a>" + item.label + "</a>"); + } + + return n.appendTo(ul); + } + }); + $('#headsearch-keywords').catcomplete({ delay: 50, - minScore: 75, + minScore: 50, maxResults: 50, source: function(request, response){ - // sort the search index by similarity to the user's query var term = request.term; + var minScore = this.options.minScore; + + // Score an possible match var score = function(item){ - var match = item.name.search(new RegExp(term, "i")); + var match = item.search(new RegExp(term, "i")); if (match < 0) return 0; - return 100 - (match * 2) - (item.name.length - term.length); + return 100 - (match * 2) - (item.length - term.length); }; - searchIndex.sort(function(a, b){ - return score(b) - score(a); + + var results = []; + // Loop through our indexes and score every match + var indexes = { + 'phpdoc:varentry': 'Variables', + 'phpdoc:classref': 'Classes', + 'phpdoc:exceptionref': 'Exceptions', + 'refentry':'Functions' + }; + $.each(indexes, function(key, category) { + $.each(searchIndex[key], function(idx, item) { + if (item && score(item[0]) > minScore) { + results.push({ + label: item[0], + value: item[1], + desc: haveDesc ? descriptionIndex[item[1]] : '', + category: category + }); + } + }); }); - - // display the best matches - var results = []; - for (var i = 0; i < this.options.maxResults; i++){ - var item = searchIndex[i]; - if (item && score(item) > this.options.minScore){ - results.push({ - label: item.name, - value: item.page - }); - } - } - response(results); + response(results.slice(0, this.options.maxResults)); }, focus: function(event, ui) { $('#headsearch-keywords').val(ui.item.label); Attachment: [application/x-gzip] index.tar.gz
Index: phpdotnet/phd/Package/PHP/Web.php =================================================================== --- phpdotnet/phd/Package/PHP/Web.php (revision 306187) +++ phpdotnet/phd/Package/PHP/Web.php (working copy) @@ -56,6 +56,10 @@ public function update($event, $val = null) { switch($event) { + case Render::FINALIZE: + $this->writeJsonIndex(); + break; + case Render::CHUNK: $this->flags = $val; break; @@ -190,6 +194,19 @@ return "<?php manual_footer(); ?>"; } + protected function writeJsonIndex() { + v("Writing search indexes..", VERBOSE_FORMAT_RENDERING); + $ids = array(); + $desc = array(); + foreach($this->indexes as $id => $index) { + $ids[$index["element"]][] = array($index["sdesc"], $index["filename"]); + $desc[$id] = $index["ldesc"]; + } + file_put_contents($this->getOutputDir() . "search-index.json", json_encode($ids)); + file_put_contents($this->getOutputDir() . "search-description.json", json_encode($desc)); + v("Index written", VERBOSE_FORMAT_RENDERING); + } + } /*