My-Proxy Forum

Proxy Help => Proxy Questions => Topic started by: perlgoodies on March 23, 2006, 08:18:45 PM

Title: Some technical problems when creating a proxy web crawler
Post by: perlgoodies on March 23, 2006, 08:18:45 PM
Hey guys, I recently created a proxy web crawler for commercial release (see

Code: [Select]
www.spyderscripts.com
and on the upper right side of the screen, click the proxy crawler).

Anyway, I am a programmer but when it comes to technical things, I just don't get it.  I am totally new to proxies myself.

I setup a bot that scrapes a number of proxy sites and it collects all the checked/verified proxies.  But when I run them through my script, many of them error out for one reason or another (only a few of the errors are actually timeouts).

Is that typical?  Do proxies go bad quickly?

I'm using the proxies to load a web site in (to load as a unique visitors), are all proxies able to do that?  Does the port number at the end make any difference?  Like can only some ports connect to web sites or something?

Any help would be much appreciated.
Title: Hey guys
Post by: Terry on March 23, 2006, 09:09:52 PM
1) Why proxies always die quickly?
http://www.my-proxy.com/blog/working-proxy-live-shortly/

2) Yes, the port number at the end makes difference.
Title: Hey guys
Post by: perlgoodies on March 23, 2006, 10:01:45 PM
Can you tell me which ports ARE allowed to surf web sites?  Or if it's easier, which ones to filter out that won't work?

I'd appreciate it.
Title: Hey guys
Post by: Terry on March 24, 2006, 04:53:28 AM
Sorry, every proxy has different setting, it all depends on the proxy administrator.
The usual ports of a proxy: 8080, 80, 3128, 8000, 3382, 8888...

http://www.my-proxy.com/blog/proxy-port-proxy-type/
Title: Re: Hey guys
Post by: aabee on June 25, 2007, 12:32:36 PM
thanks.