Golang实现的IP代理池

Golang实现的IP代理池

采集免费的代理资源为爬虫提供有效的IP代理

Travis Status for henson/proxypool Go Report Card

Stargazers over time

Stargazers over time

版本更新

  • 2019年12月18日 v2.4 感谢 @sndnvaps
    • 添加代理网站 ip3306, plp-ssl 两个
    • 更新 数据库结构,添加创建时间和更新时间
    • 更新 ip.go当中的Update(), x.Id()方法将会被x.ID()代替,所以直接更换为x.ID()
    • 更新 在插入数据时先检查库中是否存在,如果存在就更新,不存在则插入到数据库
    • 更新 https proxy api的相关参数
  • 2019年3月28日 v2.3 感谢 @sndnvaps
    • 修复数据库中不存在https代理源的时候查询出错的问题。解决问题#31
  • 2019年2月2日 v2.2 感谢 @sndnvaps
    • 添加MAC Darwin平台的支持
    • 添加完全的 sqlite3 支持
    • 添加新代理平台【feiyi,89ip】
    • fix一个bug【当数据库中无数据时,不能写入数据】
  • 2018年8月17日 v2.1 感谢 @harrybi
    • 对代理可用性的验证,增加speed字段,验证代理的速度(毫秒)
    • 调用API获取代理IP时自动过滤掉慢的代理(>=1s)
  • 2018年7月17日 v2.0 感谢 @sndnvaps
    • 使用 xorm 来处理数据库,支持 mysql、mssql、postgres 和 sqlite3
    • 更新相应爬虫程序
    • 加入日志
  • 2017年3月30日 v1.0
    • 采用 mongo 作为数据持久化
    • 结构简洁,适合二次开发

1、代理池设计

  代理池由四部分组成:

  • Getter:

  代理获取接口,目前有9个免费代理源,每调用一次就会抓取这些网站最新的100个代理放入Channel,可自行添加额外的代理获取接口

  • Channel:

  临时存放采集来的代理,通过访问稳定的网站去验证代理的有效性,有效则存入数据库;

  • Schedule:

  用定时的计划任务去检测数据库中代理IP的可用性,删除不可用的代理。同时也会主动通过Getter去获取最新代理;

  • Api:

  代理池的访问接口,提供get接口输出JSON,方便爬虫直接使用。

2、代码实现

  • Api:

  api接口相关代码,提供get接口,输出JSON;

  • Getter:

  代理获取接口,目前抓取这九个网站的免费代理,当然也支持自己扩展代理接口;

  1. 快代理
  2. 代理66
  3. IP181
  4. 有代理
  5. 西刺代理
  6. guobanjia
  7. 讯代理
  8. 无忧代理
  9. Proxylist+
  • Pkg:

  存放一些公共的模块、方法或函数;

  • 其他:

  配置文件:conf/app.ini,数据库、日志配置和代理获取接口配置;

; App name
APP_NAME = ProxyPool

[server]
HTTP_ADDR = 0.0.0.0
HTTP_PORT = 3000
;Session expires time
SESSION_EXPIRES =

[database]
; Either "mysql", "postgres" or "sqlite3", you can connect to TiDB with MySQL protocol
DB_TYPE = postgres
HOST = 127.0.0.1:5432
NAME = ProxyPool
USER = postgres
PASSWD =
; For "postgres" only, either "disable", "require" or "verify-full"
SSL_MODE = disable
; For "sqlite3" and "tidb", use absolute path when you start as service
PATH = data/ProxyPool.db

[log]
; Can be "console" and "file", default is "console"
; ; ; Use comma to separate multiple modes, e.g. "console, file"
MODE       = file
; Buffer length of channel, keep it as it is if you don't know what it is.
BUFFER_LEN = 100
; Either "Trace", "Info", "Warn", "Error", "Fatal", default is "Trace"
LEVEL      = Info
; Root path of log files, align will fill it automatically.
ROOT_PATH  =  

; For "console" mode only
[log.console]
; leave empty to inherit
LEVEL = Trace

; For "file" mode only
[log.file]
; leave empty to inherit
LEVEL          = Info
; This enables automated log rotate (switch of following options)
LOG_ROTATE     = true
; Segment log daily
DAILY_ROTATE   = true
; Max size shift of single file, default is 28 means 1 << 28, 256MB
MAX_SIZE_SHIFT = 28
; Max line number of single file
MAX_LINES      = 1000000
; Expired days of log file (delete after max days)
MAX_DAYS       = 7

[log.xorm]
; Enable file rotation
ROTATE = true
; Rotate every day
ROTATE_DAILY = true
; Rotate once file size excesses x MB
MAX_SIZE = 100
; Maximum days to keep logger files
MAX_DAYS = 3

[security]
INSTALL_LOCK = false

3、安装及使用

因为有些代理网站使用了加密页面、混淆代码等反爬技术,要正确采集到代理数据得用到 PhantomJS ,必须提前先装好。

另外,本项目用到的依赖库有:

clog "unknwon.dev/clog/v2"
github.com/go-ini/ini
github.com/go-xorm/xorm
github.com/go-xorm/core
github.com/go-sql-driver/mysql
github.com/lib/pq
github.com/Aiicy/htmlquery
github.com/PuerkitoBio/goquery
github.com/parnurzeal/gorequest
github.com/nladuo/go-phantomjs-fetcher

下载本项目:

go get -u github.com/henson/proxypool

然后配置好相应的app.ini并启动:

go build
./ProxyPool

随机输出可用的代理:

GET http://localhost:8080/v2/ip

HTTP

随机输出HTTPS代理:

GET http://localhost:8080/v2/https

HTTPS

4、添加自定义代理采集接口

其实很简单,只需要在getter包下新增一个采集函数(如例子的Data5u()),甚至可以不需要新建一个go文件(新建文件是为了方便归档采集函数,如例子5u.go)。

// 5u.go
// Data5u get ip from data5u.com
func Data5u() (result []*models.IP) {
    //处理逻辑
    ...
    log.Println("Data5u done.")
    return
}

然后在main.go的run函数中添加、删除或注释掉该采集函数的调用即可。

func run(ipChan chan<- *models.IP) {
    var wg sync.WaitGroup
    funs := []func() []*models.IP{
        getter.Data5u,
        getter.IP66,
        getter.KDL,
        getter.GBJ,
        getter.Xici,
        getter.XDL,
        //getter.IP181,
        //getter.YDL,
        getter.PLP,
    }
    ...
}

5、异常恢复

之前,偶尔会有朋友跟我反映程序无法编译,经过检查发现都是代理网站发生了变化(或修改了页面或关闭了网站),以致于采集程序原先设计的爬虫不能正常工作而导致了错误的发生。为此,我修改了代码,加入了容错机制,即便爬虫出错了也不会影响到主体程序的运行。出错的采集进程会被主线程忽略,其它正常的采集进程仍将继续工作。

6、诚挚的感谢

  • 首先感谢您的使用,如果觉得程序还不错也能帮助您解决实际问题,不妨添个赞以鼓励本人继续努力,谢谢!
  • 如果您对程序有任何建议和意见,也欢迎提交issue。
  • 当然,如果您愿意贡献代码和我一起改进本程序,那再好不过了。
Owner
Henson Lu
Make Something Amazing
Henson Lu
Comments
  • run error

    run error

    phantomjs 装了,环境看似没问题,就是起不起来,求教 -。-

    ➜  ProxyPool git:(master) ✗ ./ProxyPool
    2017/10/23 12:55:07 Starting server :8080
    2017/10/23 12:55:07 Chan: 0, IP: 35
    2017/10/23 12:55:07 Before check, DB has: 35 records.
    2017/10/23 12:55:07 IP66 done.
    2017/10/23 12:55:07 Data5u done.
    2017/10/23 12:55:07 IP181 done.
    2017/10/23 12:55:08 PLP done.
    panic: Post http://localhost:2015: dial tcp 127.0.0.1:2015: getsockopt: connection refused
    
    goroutine 86 [running]:
    panic(0x319340, 0xc4201afd40)
      /usr/local/go/src/runtime/panic.go:500 +0x1a1
    github.com/nladuo/go-phantomjs-fetcher.(*Fetcher).GetWithOption(0xc4200dc270, 0x37bed8, 0x23, 0x38393c, 0x6c, 0x371cac, 0xc, 0xc42057c880, 0x6f15f, 0xc420036e30, ...)
      /Users/xxx/Dev/go/src/github.com/nladuo/go-phantomjs-fetcher/fetcher.go:134 +0x436
    github.com/nladuo/go-phantomjs-fetcher.(*Fetcher).GetWithJS(0xc4200dc270, 0x37bed8, 0x23, 0x38393c, 0x6c, 0x371cac, 0xc, 0x0, 0x0, 0x0)
      /Users/xxx/Dev/go/src/github.com/nladuo/go-phantomjs-fetcher/fetcher.go:97 +0x80
    github.com/henson/ProxyPool/getter.XDL(0x0, 0x0, 0x0)
      /Users/xxx/Dev/go/src/github.com/henson/ProxyPool/getter/xdl.go:25 +0x1dc
    main.run.func1(0xc42006e480, 0xc420158070, 0x3a5c50)
      /Users/xxx/code/test/ProxyPool/main.go:66 +0x2b
    created by main.run
      /Users/xxx/code/test/ProxyPool/main.go:71 +0x14c
    ➜  ProxyPool git:(master) ✗ Error: Could not create web server listening on port 2016
    Error: Could not create web server listening on port 2017
    
    
    
    
  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    启动报错 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xc8 pc=0x7fdf60]

    goroutine 19 [running]: github.com/go-xorm/xorm.(*Session).Init(0xc0000ed080) /home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/session.go:94 +0x200 github.com/go-xorm/xorm.(*Engine).NewSession(...) /home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/engine.go:317 github.com/go-xorm/xorm.(*Engine).Where(0x0, 0xa025a0, 0xb8acd0, 0xc0001425d0, 0x1, 0x1, 0x0) /home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/engine.go:594 +0x57 github.com/henson/proxypool/pkg/models.countIps(0x0) /home/chen/work/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:39 +0x97 github.com/henson/proxypool/pkg/models.CountIPs(...) /home/chen/work/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:45 github.com/henson/proxypool/pkg/storage.CheckProxyDB() /home/chen/work/gopath/src/github.com/henson/proxypool/pkg/storage/filter.go:60 +0x35 main.main.func2() /home/chen/work/gopath/src/github.com/henson/proxypool/main.go:31 +0x20 created by main.main /home/chen/work/gopath/src/github.com/henson/proxypool/main.go:30 +0x8a

  • 已经正常启动,获取https代理时报错

    已经正常启动,获取https代理时报错

    2019/03/20 17:25:33 http: panic serving [::1]:14121: runtime error: invalid memory address or nil pointer dereference goroutine 1023 [running]: net/http.(*conn).serve.func1(0xc000056b40) C:/Go/src/net/http/server.go:1769 +0x140 panic(0xb127a0, 0xa68160) C:/Go/src/runtime/panic.go:522 +0x1c3 github.com/henson/proxypool/pkg/storage.ProxyFind(0xb9f032, 0x5, 0xba5f21) D:/goworker/src/github.com/henson/proxypool/pkg/storage/filter.go:102 +0x5a github.com/henson/proxypool/api.FindHandler(0xc7a400, 0xc0001e0540, 0xc0000e2800) D:/goworker/src/github.com/henson/proxypool/api/api.go:41 +0x145 net/http.HandlerFunc.ServeHTTP(0xbd41a0, 0xc7a400, 0xc0001e0540, 0xc0000e2800) C:/Go/src/net/http/server.go:1995 +0x4b net/http.(*ServeMux).ServeHTTP(0xc0001b0100, 0xc7a400, 0xc0001e0540, 0xc0000e2800) C:/Go/src/net/http/server.go:2375 +0x1dd net/http.serverHandler.ServeHTTP(0xc000286000, 0xc7a400, 0xc0001e0540, 0xc0000e2800) C:/Go/src/net/http/server.go:2774 +0xaf net/http.(*conn).serve(0xc000056b40, 0xc7b8c0, 0xc0001b0fc0) C:/Go/src/net/http/server.go:1878 +0x858 created by net/http.(*Server).Serve C:/Go/src/net/http/server.go:2884 +0x2fb

  • go get 时候报错 undefined: clog.CONSOLE

    go get 时候报错 undefined: clog.CONSOLE

    PS D:\WorkSpace\GO\proxy> go get -u github.com/henson/proxypool github.com/henson/proxypool/pkg/setting

    D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:61:11: undefined: clog.CONSOLE D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:61:43: cannot use clog.ConsoleConfig literal (type clog.ConsoleConfig) as type clog.Initer in argument to clog.New D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:65:14: cannot use 2 (type int) as type string in argument to clog.Fatal D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:89:14: cannot use 2 (type int) as type string in argument to clog.Fatal D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:98:14: cannot use 2 (type int) as type string in argument to clog.Fatal D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:105:14: cannot use 2 (type int) as type string in argument to clog.Fatal D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:137:27: undefined: clog.LEVEL D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:148:15: cannot use 2 (type int) as type string in argument to clog.Fatal D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:162:10: undefined: clog.MODE D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:163:8: undefined: clog.CONSOLE D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:163:8: too many errors

  • Zombie Process(phantomjs) 过多

    Zombie Process(phantomjs) 过多

    运行了大约 24 小时左右,查看系统有 297 个 Zombie Process

    ➜  ~ phantomjs --version
    2.1.1
    ➜  ~ ps -ef | grep defunct
    root       502 20310  0 13:15 pts/7    00:00:00 [phantomjs] <defunct>
    root       503 20310  0 13:15 pts/7    00:00:00 [phantomjs] <defunct>
    root       504 20310  0 13:15 pts/7    00:00:00 [phantomjs] <defunct>
    root       876 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root       877 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root       878 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root       940 20310  0 05:45 pts/7    00:00:00 [phantomjs] <defunct>
    root       941 20310  0 05:45 pts/7    00:00:00 [phantomjs] <defunct>
    root       942 20310  0 05:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      1323 20310  0 13:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      1324 20310  0 13:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      1325 20310  0 13:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      1608 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      1609 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      1610 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      1691 20310  0 05:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      1692 20310  0 05:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      1693 20310  0 05:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      2363 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      2364 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      2365 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      2429 20310  0 06:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      2430 20310  0 06:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      2431 20310  0 06:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      3082 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3083 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3084 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3147 20310  0 06:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      3148 20310  0 06:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      3149 20310  0 06:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      3430 20310  0 13:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      3431 20310  0 13:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      3432 20310  0 13:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      3796 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3797 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3798 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      3964 20310  0 06:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      3965 20310  0 06:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      3966 20310  0 06:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      4154 20310  0 14:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      4155 20310  0 14:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      4156 20310  0 14:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      4513 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      4514 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      4515 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      4681 20310  0 06:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      4682 20310  0 06:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      4683 20310  0 06:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      5227 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      5228 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      5229 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      5401 20310  0 06:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      5402 20310  0 06:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      5403 20310  0 06:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      5947 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      5948 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      5949 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      6119 20310  0 06:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      6120 20310  0 06:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      6121 20310  0 06:55 pts/7    00:00:00 [phantomjs] <defunct>
    root      6198 20310  0 14:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      6199 20310  0 14:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      6200 20310  0 14:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      6661 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      6662 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      6663 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      6839 20310  0 07:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      6840 20310  0 07:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      6841 20310  0 07:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      6921 20310  0 14:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      6922 20310  0 14:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      6923 20310  0 14:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      7379 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      7380 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      7381 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      7558 20310  0 07:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      7559 20310  0 07:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      7560 20310  0 07:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      8101 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      8102 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      8103 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root      8282 20310  0 07:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      8283 20310  0 07:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      8284 20310  0 07:25 pts/7    00:00:00 [phantomjs] <defunct>
    root      8827 20310  0 00:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      8828 20310  0 00:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      8829 20310  0 00:05 pts/7    00:00:00 [phantomjs] <defunct>
    root      8964 20310  0 15:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      8965 20310  0 15:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      8966 20310  0 15:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      9000 20310  0 07:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      9001 20310  0 07:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      9002 20310  0 07:35 pts/7    00:00:00 [phantomjs] <defunct>
    root      9551 20310  0 00:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      9552 20310  0 00:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      9553 20310  0 00:15 pts/7    00:00:00 [phantomjs] <defunct>
    root      9730 20310  0 07:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      9731 20310  0 07:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      9732 20310  0 07:45 pts/7    00:00:00 [phantomjs] <defunct>
    root      9810  9531  0 15:26 pts/5    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn defunct
    root     10272 20310  0 00:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     10273 20310  0 00:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     10274 20310  0 00:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     10986 20310  0 00:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     10987 20310  0 00:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     10988 20310  0 00:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     11108 20310  0 08:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     11109 20310  0 08:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     11110 20310  0 08:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     11704 20310  0 00:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     11705 20310  0 00:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     11706 20310  0 00:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     11825 20310  0 08:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     11826 20310  0 08:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     11827 20310  0 08:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     12423 20310  0 00:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     12424 20310  0 00:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     12425 20310  0 00:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     12553 20310  0 08:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     12554 20310  0 08:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     12555 20310  0 08:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     13146 20310  0 01:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     13147 20310  0 01:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     13148 20310  0 01:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     13280 20310  0 08:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     13281 20310  0 08:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     13282 20310  0 08:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     13874 20310  0 01:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     13875 20310  0 01:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     13876 20310  0 01:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     14000 20310  0 08:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     14001 20310  0 08:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     14002 20310  0 08:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     14601 20310  0 01:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     14602 20310  0 01:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     14603 20310  0 01:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     15315 20310  0 01:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     15316 20310  0 01:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     15317 20310  0 01:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     15379 20310  0 09:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     15380 20310  0 09:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     15381 20310  0 09:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     16033 20310  0 01:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     16034 20310  0 01:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     16035 20310  0 01:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     16098 20310  0 09:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     16099 20310  0 09:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     16100 20310  0 09:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     16747 20310  0 01:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     16748 20310  0 01:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     16749 20310  0 01:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     17466 20310  0 02:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     17467 20310  0 02:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     17468 20310  0 02:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     17477 20310  0 09:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     17478 20310  0 09:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     17479 20310  0 09:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     18182 20310  0 02:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     18183 20310  0 02:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     18184 20310  0 02:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     18197 20310  0 09:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     18198 20310  0 09:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     18199 20310  0 09:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     18902 20310  0 02:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     18903 20310  0 02:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     18904 20310  0 02:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     18915 20310  0 09:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     18916 20310  0 09:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     18917 20310  0 09:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     19617 20310  0 02:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     19618 20310  0 02:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     19619 20310  0 02:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     20291 20310  0 10:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     20292 20310  0 10:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     20293 20310  0 10:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     20317 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     20318 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     20319 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     20356 20310  0 02:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     20357 20310  0 02:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     20358 20310  0 02:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     21024 20310  0 10:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     21025 20310  0 10:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     21026 20310  0 10:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     21048 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21049 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21050 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21093 20310  0 02:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     21094 20310  0 02:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     21095 20310  0 02:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     21787 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21788 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21789 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     21813 20310  0 03:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     21814 20310  0 03:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     21815 20310  0 03:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     22410 20310  0 10:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     22411 20310  0 10:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     22412 20310  0 10:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     22514 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     22515 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     22516 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     22539 20310  0 03:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     22540 20310  0 03:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     22541 20310  0 03:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     23135 20310  0 10:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     23136 20310  0 10:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     23137 20310  0 10:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     23233 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23234 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23235 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23267 20310  0 03:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     23268 20310  0 03:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     23269 20310  0 03:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     23969 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23970 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23971 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     23996 20310  0 03:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     23997 20310  0 03:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     23998 20310  0 03:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     24531 20310  0 11:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     24532 20310  0 11:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     24533 20310  0 11:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     24685 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     24686 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     24687 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     24718 20310  0 03:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     24719 20310  0 03:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     24720 20310  0 03:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     25261 20310  0 11:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     25262 20310  0 11:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     25263 20310  0 11:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     25407 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     25408 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     25409 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     25449 20310  0 03:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     25450 20310  0 03:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     25451 20310  0 03:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     26123 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26124 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26125 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26174 20310  0 04:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     26175 20310  0 04:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     26176 20310  0 04:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     26643 20310  0 11:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     26644 20310  0 11:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     26645 20310  0 11:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     26843 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26844 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26845 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     26895 20310  0 04:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     26896 20310  0 04:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     26897 20310  0 04:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     27364 20310  0 11:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     27365 20310  0 11:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     27366 20310  0 11:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     27559 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     27560 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     27561 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     27620 20310  0 04:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     27621 20310  0 04:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     27622 20310  0 04:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     28276 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     28277 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     28278 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     28337 20310  0 04:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     28338 20310  0 04:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     28339 20310  0 04:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     28748 20310  0 12:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     28749 20310  0 12:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     28750 20310  0 12:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     29003 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29004 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29005 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29057 20310  0 04:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     29058 20310  0 04:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     29059 20310  0 04:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     29474 20310  0 12:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     29475 20310  0 12:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     29476 20310  0 12:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     29731 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29732 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29733 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     29788 20310  0 04:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     29789 20310  0 04:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     29790 20310  0 04:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     30457 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     30458 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     30459 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     30514 20310  0 05:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     30515 20310  0 05:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     30516 20310  0 05:05 pts/7    00:00:00 [phantomjs] <defunct>
    root     30863 20310  0 12:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     30864 20310  0 12:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     30865 20310  0 12:45 pts/7    00:00:00 [phantomjs] <defunct>
    root     31180 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31181 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31182 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31231 20310  0 05:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     31232 20310  0 05:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     31233 20310  0 05:15 pts/7    00:00:00 [phantomjs] <defunct>
    root     31584 20310  0 12:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     31585 20310  0 12:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     31586 20310  0 12:55 pts/7    00:00:00 [phantomjs] <defunct>
    root     31894 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31895 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31896 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     31954 20310  0 05:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     31955 20310  0 05:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     31956 20310  0 05:25 pts/7    00:00:00 [phantomjs] <defunct>
    root     32616 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     32617 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     32618 20310  0 Jul09 pts/7    00:00:00 [phantomjs] <defunct>
    root     32674 20310  0 05:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     32675 20310  0 05:35 pts/7    00:00:00 [phantomjs] <defunct>
    root     32676 20310  0 05:35 pts/7    00:00:00 [phantomjs] <defunct>
    
  • 跑了2个小时自己挂了

    跑了2个小时自己挂了

    2017/05/10 10:57:55 All getters finished. 2017/05/10 10:59:28 Chan: 144, IP: 275 2017/05/10 11:01:28 Chan: 0, IP: 277 panic: runtime error: index out of range

    goroutine 8554 [running]: panic(0x6daf40, 0xc042006060) C:/Go/src/runtime/panic.go:500 +0x1af github.com/henson/ProxyPool/getter.IP66(0x0, 0x0, 0x0) E:/golang/src/github.com/henson/ProxyPool/getter/66ip.go:20 +0x759 main.run.func1(0xc042172060, 0xc04222a4b0, 0x768160) E:/golang/src/github.com/henson/ProxyPool/main.go:66 +0x32 created by main.run E:/golang/src/github.com/henson/ProxyPool/main.go:71 +0x16b exit status 2

  • 这里有个拼写错误

    这里有个拼写错误

    https://github.com/henson/proxypool/blob/bd58d75b5642dd2ae3f7a1e2756d73b66ae0cfd1/pkg/rpc/grpc_server.go#L5

    应该是: github.com/henson/proxyppol---->github.com/henson/proxypool

  • 麻烦看一下,这报错了,ubuntu编译后,第一次运行

    麻烦看一下,这报错了,ubuntu编译后,第一次运行

    root@cloud:~/gopath/src/github.com/henson/proxypool# ./proxypool 2019/04/27 23:34:38 [TRACE] Log path: 2019/04/27 23:34:38 [TRACE] Log Mode: File (Info) 2019/04/27 23:34:38 [ INFO] ProxyPool panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0xc8 pc=0x81a450]

    goroutine 1 [running]: github.com/go-xorm/xorm.(*Session).Init(0xc0000fab00) /root/gopath/src/github.com/go-xorm/xorm/session.go:94 +0x200 github.com/go-xorm/xorm.(*Engine).NewSession(...) /root/gopath/src/github.com/go-xorm/xorm/engine.go:315 github.com/go-xorm/xorm.(*Engine).Where(0x0, 0x9fc1a0, 0xb84060, 0xc0001527f0, 0x1, 0x1, 0xc0001fff30) /root/gopath/src/github.com/go-xorm/xorm/engine.go:592 +0x57 github.com/henson/proxypool/pkg/models.countIps(0x8) /root/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:39 +0x97 github.com/henson/proxypool/pkg/models.CountIPs(...) /root/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:45 main.main() /root/gopath/src/github.com/henson/proxypool/main.go:45 +0xde

  • 运行出现错误

    运行出现错误

    你好,运行出现如下报错

    C:\Users\AppData\Local\Programs\Python\Python36\python.exe F:/code/ProxyPool/run.py
    Traceback (most recent call last):
      File "F:/code/ProxyPool/run.py", line 1, in <module>
        from proxypool.api import app
      File "F:\code\ProxyPool\proxypool\api.py", line 1, in <module>
        from flask import Flask,g
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\__init__.py", line 17, in <module>
        from werkzeug.exceptions import abort
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\__init__.py", line 151, in <module>
        __import__('werkzeug.exceptions')
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\exceptions.py", line 71, in <module>
        from werkzeug.wrappers import Response
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\wrappers.py", line 37, in <module>
        from werkzeug.formparser import FormDataParser, default_stream_factory
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\formparser.py", line 28, in <module>
        from werkzeug.wsgi import make_line_iter, \
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\wsgi.py", line 13, in <module>
        import httplib
      File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\httplib.py", line 345
        print "reply:", repr(line)
                     ^
    SyntaxError: invalid syntax
    
    Process finished with exit code 1
    
  • memory error on windows

    memory error on windows

    panic: runtime error: invalid memory address or nil pointer dereference [signal 0xc0000005 code=0x0 addr=0x0 pc=0x6815f9]

    goroutine 21 [running]: gopkg.in/mgo%2ev2.(*Session).Clone(0x0, 0xc042123f78) C:/Users/xxx/go/src/gopkg.in/mgo.v2/session.go:1603 +0x29 github.com/henson/ProxyPool/storage.(*Storage).GetDBSession(0xc042123f38, 0xc042123fac) C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/storage.go:30 +0x36 github.com/henson/ProxyPool/storage.(*Storage).Count(0xc042123f38, 0x0) C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/storage.go:58 +0x53 github.com/henson/ProxyPool/storage.CheckProxyDB() C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/filter.go:36 +0xd3 main.main.func2() E:/git/ProxyPool/main.go:27 +0x27 created by main.main E:/git/ProxyPool/main.go:26 +0x110

  • bug ?

    bug ?

    2022/11/26 18:20:29 [TRACE] Log Mode: Console (Trace)
    2022/11/26 18:20:29 [ INFO] ProxyPool 
    2022/11/26 18:20:29 [TRACE] Log path: 
    2022/11/26 18:20:29 [ INFO] SQLite3 Supported
    2022/11/26 18:20:29 Starting server 0.0.0.0:3000
    2022/11/26 18:20:29 Chan: 0, IP: 0
    2022/11/26 18:20:29 [ INFO] Before check, DB has: 0 records.
    2022/11/26 18:20:29 [ INFO] 89IP] start test
    2022/11/26 18:20:29 [ INFO] [IP3306]] start Get IpProxy
    2022/11/26 18:20:29 [ INFO] After check, DB has: 0 records.
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x88bc3c]
    
    goroutine 83 [running]:
    github.com/Aiicy/htmlquery.(*NodeNavigator).NodeType(0x30?)
            /home/xxx/go/pkg/mod/github.com/!aiicy/[email protected]/query.go:169 +0x1c
    github.com/antchfx/xpath.axisPredicate.func1({0xb002c0, 0xc0002e2150})
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/build.go:45 +0x47
    github.com/antchfx/xpath.(*descendantQuery).Select.func1()
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/query.go:251 +0x62
    github.com/antchfx/xpath.(*descendantQuery).Select(0xc0001beb70, {0xaf9d20, 0xc00012e9c0})
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/query.go:283 +0x170
    github.com/antchfx/xpath.(*filterQuery).Select(0xc0001bebd0, {0xaf9d20, 0xc00012e9c0})
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/query.go:569 +0x50
    github.com/antchfx/xpath.(*descendantQuery).Select(0xc0001bec00, {0xaf9d20, 0xc00012e9c0})
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/query.go:240 +0x5f
    github.com/antchfx/xpath.(*descendantQuery).Select(0xc0001bec30, {0xaf9d20, 0xc00012e9c0})
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/query.go:240 +0x5f
    github.com/antchfx/xpath.(*NodeIterator).MoveNext(0xc00012e9c0)
            /home/xxx/go/pkg/mod/github.com/antchfx/[email protected]/xpath.go:85 +0x38
    github.com/Aiicy/htmlquery.Find(0x0, {0xa41217?, 0x0?})
            /home/xxx/go/pkg/mod/github.com/!aiicy/[email protected]/query.go:33 +0x175
    github.com/henson/proxypool/getter.FQDL()
            /home/xxx/proxypool/getter/fanqiedl.go:14 +0x48
    main.run.func1(0x0?)
            /home/xxx/proxypool/main.go:76 +0x32
    created by main.run
            /home/xxx/proxypool/main.go:75 +0x94
    
    

    conf/app.ini

    ; App name
    APP_NAME = ProxyPool
    
    [server]
    HTTP_ADDR       = 0.0.0.0
    HTTP_PORT       = 3000
    ; Session expires time
    SESSION_EXPIRES = 168h0m0s
    
    [database]
    ; Either "mysql", "postgres" or "sqlite3", you can connect to TiDB with MySQL protocol
    DB_TYPE  = sqlite3
    HOST     = 192.168.13.105:3306
    NAME     = ProxyPool
    USER     = root
    PASSWD   = 5XSwBxGx
    ; For "postgres" only, either "disable", "require" or "verify-full"
    SSL_MODE = disable
    ; For "sqlite3" and "tidb", use absolute path when you start as service
    PATH     = data/ProxyPool.db
    
    [log]
    ; Can be "console" and "file", default is "console"
    ; ; ; Use comma to separate multiple modes, e.g. "console, file"
    MODE       = console
    ; Buffer length of channel, keep it as it is if you don't know what it is.
    BUFFER_LEN = 100
    ; Either "Trace", "Info", "Warn", "Error", "Fatal", default is "Trace"
    LEVEL      = Info
    ; Root path of log files, align will fill it automatically.
    ROOT_PATH  = 
    
    ; For "console" mode only
    [log.console]
    ; leave empty to inherit
    LEVEL = Trace
    
    ; For "file" mode only
    [log.file]
    ; leave empty to inherit
    LEVEL          = Info
    ; This enables automated log rotate (switch of following options)
    LOG_ROTATE     = true
    ; Segment log daily
    DAILY_ROTATE   = true
    ; Max size shift of single file, default is 28 means 1 << 28, 256MB
    MAX_SIZE_SHIFT = 28
    ; Max line number of single file
    MAX_LINES      = 1000000
    ; Expired days of log file (delete after max days)
    MAX_DAYS       = 7
    
    [log.xorm]
    ; Enable file rotation
    ROTATE       = true
    ; Rotate every day
    ROTATE_DAILY = true
    ; Rotate once file size excesses x MB
    MAX_SIZE     = 100
    ; Maximum days to keep logger files
    MAX_DAYS     = 3
    
    [security]
    INSTALL_LOCK = true
    
  • 一点建议

    一点建议

    proxypool的运行位置,一定要在censor内网才是可靠的结果,但又会导致subscribe catch失败,这是一个相互制约的问题。 为此建议如下: 1、抓取独立设计,并且支持临时的proxy方式,独立于proxy check,支持触发式或者定期抓取。 2、proxy check方式采用灵活策略可定制,而不是固定的crontab方式,只能定期。比如:当连续多少个上一次有效的地址失败(random检测),然后启动全check。

    这两个功能完成,才会有一定的生命力。目前的功能,只能说不好用。

  • 数据库没有数据,访问端口也不出来数据

    数据库没有数据,访问端口也不出来数据

    控制台显示在3000端口运行

    cjp@MacBook-Pro proxypool % ./proxypool
    2021/02/05 00:01:17 [TRACE] Log Mode: Console (Trace)
    2021/02/05 00:01:17 [ INFO] ProxyPool 
    2021/02/05 00:01:17 [TRACE] Log path: 
    2021/02/05 00:01:17 Starting server 127.0.0.1:3000
    2021/02/05 00:01:17 Chan: 0, IP: 0
    2021/02/05 00:01:17 [ INFO] 89IP] start test
    2021/02/05 00:01:17 [ INFO] [IP3306]
    

    但是实际上却没有数据 image

    数据库也没有数据

  • 程序能运行,但数据库一直没数据

    程序能运行,但数据库一直没数据

    程序运行半天,数据库 mysql 一直是空的,成功的创建了 ip

    直接访问 localhost:8080/v2/ip 这个页面无法访问

    This page isn’t working
    localhost didn’t send any data.
    ERR_EMPTY_RESPONSE
    

    运行环境: macos 11.2 数据库: mysql docker启动 go version go1.15.7 darwin/amd64

  • http://ip138.com

    http://ip138.com

    [ WARN] [CheckIP] testIP = http://45.76.236.83:3128, pollURL = http://ip138.com: Error = [Get "http://ip138.com": proxyconnect tcp: dial tcp 45.76.236.83:3128: connect: connection refused]

Related tags