Re: Any clever ideas on how to find some random records?

Brian Dunning wrote:

I have a MySQL database with about a million records. I'd like to use the SQL command "order by RAND()" but my ISP won't let me: whenever the server gets spidered, Google overloads their MySQL server because of all the overhead of that command. I can't just cloak the spiders because I need them to find the random pages. So...what I've been doing for small sets of records is to use PHP to generate a bunch of random record ID's, then I construct a long SQL statement to find all the matching records. This works, but if I want to generate a big index page to list a hundred or a thousand records, it could get pretty clunky. Anyone have any better suggestions? :)

Not sure that this would work (since I've never done it :) but perhaps you could create a static page specifically for the web spiders. The basic plan is this: - Create a web page with the record set (as you're already doing it) - Save this web page to your server's cache (either roll your own or you can use Cache_lite or whatever) - Use an 8 day expiration on this file - Then once a week you update this cache file from a cron job - Now the key part: you alter your robots.txt file so that the spiders will not go to the dynamically created page, but rather they follow to the cached / static page. Oh and it this actually works for you... it would be nice to get some feedback.