Re: Script to Fetch Elephpants

On Sun, Sep 18, 2011 at 07:17, Stewart Lord <stewey@ambitious.ca> wrote: > > On 2011-09-14, at 11:57 PM, Hannes Magnusson wrote: > >> Is url_sq guaranteed never to contain urlencoded paths, or >> non-windows-friendly-filenames? >> I doubt we have any Windows mirrors though.. but who knows. >> >> I notice you never cleanup the fetched images, so we will quickly have >> bucketload of images.. maybe thats the idea - then we can shuffle them >> on the mirrors? >> >> Its also missing sanitychecks around the file_get_contents() for the >> json data, and ensuring $decoded actually contains anything before >> overwriting photos.json. >> Same when fetching the file, it needs to verify it actually fetched something. >> ..And maybe check if the file already exists before overwriting it? >> >> -Hannes > > > Hi Hannes, > > Thanks again for the code review. I've updated the script to incorporate your feedback: > > http://pastebin.com/BBu39Rv4 > > The url_sq basename seems to only ever contain alpha-numerics, '_' and '.', > but I've added a filter to make sure. I added a bit of logic at the end of the script to remove > any stale images (as long as we set the limit reasonably high we should have enough to shuffle > through). Also added sanity checks. > Looks good. Is the result set random, or the "latest images"? Just wondering how often new pics would be downloaded when we run this every hour. -Hannes