Re: what would a c extension buy me

joseph wrote:

hi, my site is www.myowndictionary.com i use javascript to create definitions for words (from open source dictionaries) as pop-ups and hook word-lists for vocabulary study with that. i can now parse the html from rss feeds and match only text of interest. i just wrote my own multibyte html parser by combining various other code i already had on hand for the parsing of multibyte text anyhow. but, that's not the slowest part. that takes maybe 3 seconds to get the page, 3 seconds to parse it, but it sometimes takes 5 minutes for the next step on huge rss files. (I know the time because they are cached) that last step... The slow part that's left is the dictionary searching itself. It's the parsing of complicated php code -- 550 lines of complex data structure loops, function calls, looping through arrays to make sql queeries -- lots of sql queeries.... the seconds ~ 5 minute it takes it all takes so much time.

use the 'EXPLAIN' syntax of MySQL to figure out if there are any bottlenecks related to crap or 'missing' indexes with regard to the sql queries you mention above. - chances are adding a couple of well placed indexes will probably shave massive ammounts from the process time.

i want to ask for opinion about: 1) rewriting the php as C extension. Would it slow my down to the below 15 second range?

How Long is a chinaman. How Long is a chinaman? yes the C extension would be much faster (assuming you wrote it correctly - but it's much harder to write and maintain)

2) does mysql have server-side functions yet?

there are stored procedures in mysql5.

3) if i moved the php code off mysql to postgresql and wrote server-side functions (which i've been trained to do), how much of a time improvement would we be talking about?

how on gods' (or whoever runs the place these days) earth can we tell whether an unwritten stored procedure in postgres would be faster or slower than an unknown (to us) piece of php code? that said I doubt postgres will beat mysql in terms of pure speed assuming you db schema is sound (indexes are good, etc).

please send me your input! i am at a growing stage and need direction here. because any of those steps will require considerable time.

it's monday morning here too ;-) interesting language tool your building, you might consider repackaging it in a modular form so that other people could implement in their own site. (you do call it 'open source') rgds, Jochem

thank you. joseph.