Re: Loading large volumes of data

On Thu, 7 Oct 2004 09:05:55 +0200, Evan Morris <evan@exclusivebooks.co.za> wrote: > I have a text file containing strings. The text file is pretty massive, > about 895 MB. I need to load the words in the text file into the database in > such a way that there is a single occurrence of each word in the table. I'm glad you found that the indexes worked for you. I was curious if your file already had one word per line? If so, you could do a cat | sort | uniq to pre-process the file and rid yourself of any duplicates to eliminate the need for the check altogether. I think that getting the data into a ready-to-load format before involving the database would be quicker overall. Not to disparage PHP, but I would use Perl for such an endeavor. PHP seems to have a lot of memory overhead associated with text parsing.