Author Topic: Some technical problems when creating a proxy web crawler  (Read 3809 times)

perlgoodies

  • Newbie
  • *
  • Posts: 3
    • View Profile
Some technical problems when creating a proxy web crawler
« on: March 23, 2006, 08:18:45 PM »
Hey guys, I recently created a proxy web crawler for commercial release (see

Code: [Select]
www.spyderscripts.com
and on the upper right side of the screen, click the proxy crawler).

Anyway, I am a programmer but when it comes to technical things, I just don't get it.  I am totally new to proxies myself.

I setup a bot that scrapes a number of proxy sites and it collects all the checked/verified proxies.  But when I run them through my script, many of them error out for one reason or another (only a few of the errors are actually timeouts).

Is that typical?  Do proxies go bad quickly?

I'm using the proxies to load a web site in (to load as a unique visitors), are all proxies able to do that?  Does the port number at the end make any difference?  Like can only some ports connect to web sites or something?

Any help would be much appreciated.

Terry

  • Administrator
  • *****
  • Posts: 1669
    • View Profile
Hey guys
« Reply #1 on: March 23, 2006, 09:09:52 PM »
1) Why proxies always die quickly?
http://www.my-proxy.com/blog/working-proxy-live-shortly/

2) Yes, the port number at the end makes difference.
Elite Proxy Switcher - Easily testing and using proxies.
Socks Proxy Checker - To be your chief socks proxy checker.
Buy Proxy - Buy checked proxies from My-Proxy easily and quickly.

perlgoodies

  • Newbie
  • *
  • Posts: 3
    • View Profile
Hey guys
« Reply #2 on: March 23, 2006, 10:01:45 PM »
Can you tell me which ports ARE allowed to surf web sites?  Or if it's easier, which ones to filter out that won't work?

I'd appreciate it.

Terry

  • Administrator
  • *****
  • Posts: 1669
    • View Profile
Hey guys
« Reply #3 on: March 24, 2006, 04:53:28 AM »
Sorry, every proxy has different setting, it all depends on the proxy administrator.
The usual ports of a proxy: 8080, 80, 3128, 8000, 3382, 8888...

http://www.my-proxy.com/blog/proxy-port-proxy-type/
Elite Proxy Switcher - Easily testing and using proxies.
Socks Proxy Checker - To be your chief socks proxy checker.
Buy Proxy - Buy checked proxies from My-Proxy easily and quickly.

aabee

  • Post Rank: 2
  • **
  • Posts: 29
    • View Profile
Re: Hey guys
« Reply #4 on: June 25, 2007, 12:32:36 PM »
thanks.