Re: How would you do this ?

Jad madi wrote:

I'm building an RSS aggregator so I'm trying to find out the best way to parse users account feeds equally so Lets say we have 20.000 user with average of 10 feeds in account so we have about 200.000 feed How would you schedule the parsing process to keep all accounts always updated without killing the server? NOTE: that some of the 200.000 feeds might be shared between more than one user

this is just an idle thought, but could you base it on prime numbers? for instance, let's say you have 6 feeds which in the last week have generated (a=10, b=8, c=5, d=7, e=13, f=2) articles. order them by article amount descending in last week: e, a, b, d, c, f assign primes to them: e=1, a=2, b=3, d=5, c=7, f=11 let $i=1 every 6 hours, check feeds who's primes are divisible by $i, then add 1 to $i at the end of the week, recalculate the primes. in this example, the feed checks would be:

monday    (1,2,3,4)     = (e),     (e,a),   (e,b),   (e,a)
tuesday   (5,6,7,8)     = (e,d),   (e,a,b), (e,c),   (e,a)

wednesday (9,10,11,12) = (e,b), (e,a,d), (e,f), (e,a,b)

thursday  (13,14,15,16) = (e),     (e,a,c), (e,b,d), (e,a)
friday    (17,18,19,20) = (e),     (e,a,b), (e),     (e,a,d)
saturday  (21,22,23,24) = (e,b,c), (e,a,f), (e),     (e,a,b)
sunday    (25,26,27,28) = (e,d),   (e,a),   (e,a,b), (e,a,c)

in this example, at most three out of six are being checked for at a time, but everything gets at least a chance. e had 13 articles, is checked 28 times a had 10 articles, is checked 14 times b had 8 articles, is checked 9 times d had 7 articles, is checked 5 times c had 5 articles, is checked 4 times f had 2 articles, is checked 2 times Kae