slightly OT, need help with wget, was RE:php web spider
| From: | Marc Swanson | Date: | Mon, 31 Jul 2000 23:24:59 +0000 |
| Subject: | slightly OT, need help with wget, was RE:php web spider | ||
| Groups: | php.general | ||
| Request: | Send a blank email to php-general+get-9307@lists.php.net to get a copy of this message | ||
A while ago I posted a question regarding web spiders written in PHP. I
then decided on using wget to do the page retrieval to save some time. It
has worked great except for one annoying case that I can't seem to fix.
The problem comes when I try to pull pages that contain a tilde '~'
charachter in the URL, a directory will be created on the local filesystem
with a %7E instead of a ~. EG if I'm trying to download
http://www.foo.com/~someguy, my directory tree will look
like
/some/local/dir/www.foo.com/%7Esomeguy. I've poured over the docs for wget
(http://www.gnu.org/manual/wget/ ) with no aparent command line option to
fix this. It is rather annoying since the links on the pages that are
downloaded are set to point to /some/local/dir/www.foo.com/~someguy but that
dir does not exist! I tried writing a hack to examine the file system after
the wget call but when I try to do a readdir on a directory containing a %7E
instead of a ~, readdir shows it as a ~?????? What gives?
Any other suggestions?
Thanks (and sorry for the off topic WOB)
-Marc-
#/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/#
#Marc Swanson | \|||/ #
#MSwanson Consulting | /o o\ #
#Phone: (603)868-1721 |-ooo----O----ooo-#
#Fax: (603)868-1730 | Solutions in: #
#marcswanson@mediaone.net | 'PHP 'Perl #
# | 'SQL 'C++ #
# | 'HTML 'Sh/Csh#
# http://www.mswanson.com | 'Javascript #
#/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/#