Re: Large(ish) scale pdf file cacheing
| From: | George Wilson | Date: | Wed, 05 Mar 2014 00:49:16 +0000 |
| Subject: | Re: Large(ish) scale pdf file cacheing | ||
| References: | 1 2 3 4 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-323029@lists.php.net to get a copy of this message | ||
On Tue, Mar 4, 2014 at 1:56 PM, Bastien Koert <phpster@gmail.com> wrote:
> Ok, so what I proposed should work fairly well. To keep spiders off, add a
> robots.txt file to the webserver to block them, but hopefully this process
> is hidden behind a password or session to prevent unnecessary generation.
> This note from google might help ..
> https://support.google.com/webmasters/answer/156449?hl=en
>
I had suggested using a robots.txt. I am not certain why but the project
manager wants the robots to have the ability to crawl the pdf links. Thank
you for the reference, I will check it out!