Dragonfly is an intelligent P2P based image and file distribution system.

Last update: Jan 9, 2023

Comments: 16

Dragonfly

Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order to get stable binaries.

Introduction

Dragonfly is an open source intelligent P2P based image and file distribution system. Its goal is to tackle all distribution problems in cloud native scenarios. Currently Dragonfly focuses on being:

Simple: well-defined user-facing API (HTTP), non-invasive to all container engines;
Efficient: CDN support, P2P based file distribution to save enterprise bandwidth;
Intelligent: host level speed limit, intelligent flow control due to host detection;
Secure: block transmission encryption, HTTPS connection support.

Dragonfly is now hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project. Originally it was born to solve all kinds of distribution at very large scales, such as application distribution, cache distribution, log distribution, image distribution, and so on.

Dragonfly has finished refactoring in Golang. Now versions > 0.4.0 are totally in Golang, while those < 0.4.0 are in Java. We encourage adopters to try Golang version first, since Java versions will be out of support in the next few releases.

Features

In details, Dragonfly has the following features:

P2P based file distribution: Using P2P technology for file transmission, which can make full use of the bandwidth resources of each peer to improve download efficiency, saves a lot of cross-IDC bandwidth, especially costly cross-board bandwidth
Non-invasive support for all kinds of container technologies: Dragonfly can seamlessly support various containers for distributing images.
Host level speed limit: Many downloading tools(wget/curl) only have rate limit for the current download task, but dragonfly also provides rate limit for the entire host.
Passive CDN: The CDN mechanism can avoid repetitive remote downloads.
Strong consistency: Dragonfly can guarantee that all downloaded files must be consistent even if users do not provide any check code(MD5).
Disk protection and high efficient IO: Precheck Disk space, delay synchronization, write file-block in the best order, split net-read / disk-write, and so on.
High performance: Cluster Manager is completely closed-loop, which means, it does not rely on any DB or distributed cache, processing requests with extremely high performance.
Exception auto isolation: Dragonfly will automatically isolate exception nodes(peer or Cluster Manager) to improve download stability.
No pressure on file source: Generally, as long as a few Cluster Managers download file from the source.
Support standard http header: Support http header, Submit authentication information through http header.
Effective concurrency control of Registry Auth: Reduce the pressure of the Registry Auth Service.
Simple and easy to use: Very few configurations are needed.

Comparison

For Dragonfly, no matter how many clients start the file downloading, the average downloading time is almost stable without increasement (12s in experiment, which means it only takes 12s in total for all client to finish downloading file/image).

And for wget, the downloading time keeps increasing when you have more clients. As the number of wget clients reaches 1200 (in following experiment), the file source will crash, then it can not serve any client.

The following table shows the testing environment and the graph shows the comparison result.

Test Environment	Statistics
Dragonfly server	2 * (24core 64GB 2000Mb/s)
File Source server	2 * (24core 64GB 2000Mb/s)
Client	4core 8GB 200Mb/s
Target file size	200MB

Roadmap

For more details about roadmap, please refer to file ROADMAP.md.

Community

You are encouraged to communicate most things via GitHub issues or pull requests.

Other active channels:

Twitter: @dragonfly_oss
Dingtalk Group(钉钉群)

Contributing

You are warmly welcomed to hack on Dragonfly. We have prepared a detailed guide CONTRIBUTING.md.

License

Dragonfly is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Owner

dragonflyoss

https://github.com/dragonflyoss/Dragonfly https://d7y.io

Comments

Add Continuous Fuzzing via Fuzzit

This PR adds continuous fuzzing via Fuzzit.

Description is at https://github.com/dragonflyoss/Dragonfly/issues/846

fixes https://github.com/dragonflyoss/Dragonfly/issues/846

cc @lowzj

Dragonfly does not work

Ⅰ. Issue Description

Ⅱ. Describe what happened

Follow Dragonfly Quick Start to install Dragonfly, but it does not work.

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Env:

SuperNode: 192.168.21.103
Clients: 192.168.21.102, 192.168.21.101

Installation:

SuperNode

$ docker run --name dragonfly-supernode --restart=always -d -p 8001:8001 -p 8002:8002 registry.cn-hangzhou.aliyuncs.com/dragonflyoss/supernode:0.3.0

Clients

$ cat <<EOD >/etc/dragonfly.conf
[node]
address=192.168.21.103
EOD
$ docker run --name dragonfly-dfclient --restart=always -d -p 65001:65001 -v /root/.small-dragonfly:/root/.small-dragonfly -v /etc/dragonfly.conf:/etc/dragonfly.conf dragonflyoss/dfclient:v0.3.0 --registry=https://registry.xx.com
# Add "registry-mirrors": ["http://127.0.0.1:65001"] in /etc/docker/daemon.json
$ systemctl restart docker

Test:

I can pull images on client node, but there is on datas and logs from Dragonfly.

$ du -sh /root/.small-dragonfly/dfdaemon/data/
0	/root/.small-dragonfly/dfdaemon/data/

$ cat /root/.small-dragonfly/logs/dfdaemon.log
2019-03-15 12:31:22.357 INFO sign:1 : init...
2019-03-15 12:31:22.361 INFO sign:1 : rotate log routine start...
2019-03-15 12:31:22.396 INFO sign:1 : dfget version:
2019-03-15 12:31:22.396 ERRO sign:1 : init properties failed:open /etc/dragonfly/dfdaemon.yml: no such file or directory
2019-03-15 12:31:22.397 INFO sign:1 : init properties:{"Registries":[{"Schema":"https","Host":"cargo.caicloudprivatetest.com","Certs":null,"Regx":"(^localhost$)|(^127.0.0.1$)|(^127.0.0.1$)"}]}
2019-03-15 12:31:22.397 INFO sign:1 : init finish
2019-03-15 12:31:22.397 INFO sign:1 : start dfdaemon param: &{DfPath:/dfclient/dfget DFRepo:/root/.small-dragonfly/dfdaemon/data/ RateLimit:20M CallSystem:com_ops_dragonfly URLFilter:Signature&Expires&OSSAccessKeyId Notbs:true MaxProcs:4 Version:false Verbose:false HostIP:127.0.0.1 Port:65001 Registry:https://cargo.caicloudprivatetest.com DownRule: CertFile: KeyFile: TrustHosts:[] ConfigPath:/etc/dragonfly/dfdaemon.yml}
2019-03-15 12:31:22.397 INFO sign:1 : launch dfdaemon http server on 127.0.0.1:65001
2019-03-15 12:33:22.398 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:33:22.399 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file
2019-03-15 12:35:22.400 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:35:22.401 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file
2019-03-15 12:37:22.403 INFO sign:1 : scan repo and clean expired files
2019-03-15 12:37:22.404 INFO sign:1 : ignore /root/.small-dragonfly/dfdaemon/data/: not a regular file

$ docker logs -f --tail 50 dragonfly-dfclient
# there is no log from dfclient

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

dragonfly version: v0.3.0
OS (e.g. from /etc/os-release): Linux
Kernel (e.g. uname -a): Linux 3.10.0-693.el7.x86_64
Install tools:
Others:

Fix an issue where HTTPS is not supported

Ⅰ. Describe what this PR did

Fix bug

Ⅱ. Does this pull request fix one issue?

Issue: #1231

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Need integration testing

Ⅳ. Describe how to verify it

Don't config docker mirrors，pull the images from a non-local address（eg：docker pull 192.168.1.100:8443/alpine:3.9）

Ⅴ. Special notes for reviews
When dfget specifies -- node node, dfget still synchronizes old files after the supernode file is updated

Question

问题描述：想测试一下，用dfget 同步过程中，supernode（也是file server）中的文件修改后，是否下次马上同步新文件，发现用dfget 同步的依旧是修改前的老文件，使用命令： dfget --url --output /data/qiyu/indexing.data --node ** -s 50M --totallimit 50M -b dfget --version ：0.2.0 start download by dragonfly ====================start==================== progress[3092019/3092019 in 0.000s] =====================end===================== download SUCCESS(0) cost(0.191s) length:3092019 reason:0 fail to download indexing.data at Thu Jun 27 15:30:02 CST 2019 remote file length 3092024 local file length 3092019

[Question] dfget fails to pull images

Question

After installed all component of dragonfly and finished all configuration according to this document, I failed to pull images via dragonfly with great possibility.

when failed, check the dfclient.log:

[2018-11-30 16:14:54,289] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:['127.0.0.1']
[2018-11-30 16:14:54,771] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:2,cdn status:RUNNING,cdn sucCount:2', u'code': 602} and sleep 1.142 ...
[2018-11-30 16:14:54,916] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.890 ...
[2018-11-30 16:14:55,921] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.001
[2018-11-30 16:14:55,957] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.064 ...
[2018-11-30 16:14:56,813] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.013 ...
[2018-11-30 16:14:57,026] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.577 ...
[2018-11-30 16:14:57,832] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.743 ...
[2018-11-30 16:14:58,610] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:3,cdn status:RUNNING,cdn sucCount:3', u'code': 602} and sleep 1.613 ...
[2018-11-30 16:14:59,295] ERROR sign:8797-1543565691.582 lineno:81 : register to node:127.0.0.1 error
Traceback (most recent call last):
  File "/usr/local/bin/component/httputil.py", line 67, in register
    schema, node), data=params, timeout=(2.0, 5.0))
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/bin/vendor/requests-2.18.4-py2.7.egg/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8002): Read timed out. (read timeout=5.0)
[2018-11-30 16:14:59,300] INFO sign:8797-1543565691.582 lineno:64 : do register to 127.0.0.1,remainder:[]
[2018-11-30 16:14:59,582] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.895 ...
[2018-11-30 16:15:00,084] ERROR sign:8797-1543565691.582 lineno:136 : init fail but try back down
Traceback (most recent call last):
  File "/usr/local/bin/dfget", line 128, in <module>
    register_result = env.init()
  File "/usr/local/bin/env.py", line 148, in init
    paramparser.cmdparam.identifier)
  File "/usr/local/bin/env.py", line 84, in register
    nodes, paramparser.cmdparam.url, task_url, port, http_path, md5, identifier)
  File "/usr/local/bin/component/httputil.py", line 86, in register
    raise Exception("register result:%s" % result)
Exception: register result:{u'code': 607}
[2018-11-30 16:15:00,085] INFO sign:8797-1543565691.582 lineno:599 : download fail and not back source
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:94 : |UNKNOWN|http://hub.c.163.com/v2/library/php/blobs/sha256:2871138c7173567814c6bda10b07b3d772bfafb53ffefe66a4e44e530f4127bc|-1|0|UNKNOWN|com_ops_dragonfly|8.505|
[2018-11-30 16:15:00,087] INFO sign:8797-1543565691.582 lineno:111 : download FAIL cost:8.505s length:-1 reason:1001
[2018-11-30 16:15:00,234] INFO sign:8626-1543565671.974 lineno:60 : local http result:20971520 for path:/rate/ and cost:0.002
[2018-11-30 16:15:00,271] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 0.956 ...
[2018-11-30 16:15:01,240] INFO sign:8626-1543565671.974 lineno:110 : pull piece task result:{u'msg': u'client sucCount:4,cdn status:RUNNING,cdn sucCount:4', u'code': 602} and sleep 1.511 ...
[2018-11-30 16:15:01,484] INFO sign:8625-1543565671.979 lineno:110 : pull piece task result:{u'msg': u'client sucCount:0,cdn status:RUNNING,cdn sucCount:0', u'code': 602} and sleep 1.554 ...

dfget 本地下载报错

RT，我在本地启动两个dfget进程，同时下载一个在线文件，报了如下错误，请问是什么原因呢

本地通过docker，启了supernode进程

docker run -d -p 8001:8001 -p 8002:8002 f28c73445883

我的dragonfly.conf配置如下 [node] address=127.0.0.1,127.0.0.1

dfclient 报错日志：

[2017-12-15 18:56:47,844] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,848] WARNING sign:70462-1513335389.992 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,851] INFO sign:70462-1513335389.992 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 1.432 ... [2017-12-15 18:56:47,938] ERROR sign:70469-1513335392.125 lineno:332 : piece range:0-5242879 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:436a148497d25f524d222019b174f54f,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,943] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,944] ERROR sign:70469-1513335392.125 lineno:332 : piece range:5242880-10485759 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:d99f987ccca3083eaac61338d800da60,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,947] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,958] WARNING sign:70469-1513335392.125 lineno:167 : has not available pieceTask,maybe resource lack [2017-12-15 18:56:47,965] ERROR sign:70469-1513335392.125 lineno:332 : piece range:10485760-15728639 error,realMd5:70461da8b94c6ca5d2fda3260c5a8c3b,expectedMd5:7d05eb939cd37dc6f612d09a9b6319cc,dstIp:127.0.0.1,total:162 [2017-12-15 18:56:47,970] INFO sign:70469-1513335392.125 lineno:110 : pull piece task result:{u'msg': u'piece resource lack', u'code': 602} and sleep 0.665 ...

求大神帮忙看下是什么原因，谢谢了^_^
docker proxy error

when docker pull, there are some errors as follows: root@iZ2ze7tbgo9lk7jubiy6tcZ:~# df-daemon -registry https://docker.acmcoder.com -port 18001

launch df-daemon on port:18001 2017/12/12 18:21:28 http: proxy error: context canceled 2017/12/12 18:21:32 http: proxy error: context canceled 2017/12/12 18:21:34 http: proxy error: context canceled 2017/12/12 18:21:35 http: proxy error: context canceled 2017/12/12 18:21:36 http: proxy error: context canceled 2017/12/12 18:21:36 http: proxy error: dial tcp: i/o timeout 2017/12/12 18:21:37 http: proxy error: context canceled 2017/12/12 18:21:38 http: proxy error: context canceled Killed root@iZ2ze7tbgo9lk7jubiy6tcZ:~# curl https://docker.acmcoder.com
Harbor
Loading...
root@iZ2ze7tbgo9lk7jubiy6tcZ:~#
Using GOPROXY environment variables if set instead of default value

Ⅰ. Describe what this PR did

Using GOPROXY environment variables if set instead of default value

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews
feature: support non-ipv4 format node address for Dragonfly dfclient

Signed-off-by: Starnop [email protected]

Ⅰ. Describe what this PR did

Make it work normally when specifies non-ipv4 format node address for dfclient.

Ⅱ. Does this pull request fix one issue?

#679

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

None.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

BTW. update the shell scripts.
feature: update API definition to be more accurate

Signed-off-by: Allen Sun [email protected]

Ⅰ. Describe what this PR did

update API deinition to be more accurate

Ⅱ. Does this pull request fix one issue?

none

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

just definition of API, no need to test.

Ⅳ. Describe how to verify it

none

Ⅴ. Special notes for reviews

none
fix valid ip not support ipv6

Ⅰ. Describe what this PR did

fix valid ip not support ipv6

Ⅱ. Does this pull request fix one issue?

fixes #334

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

unit test

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews
ipv6 support
Ⅰ. Issue Description

The server bind ip '0.0.0.0' only useing for ipv4. ipv6 should be support。

Ⅱ. Describe what happened

can not support ipv6 listen & advertiseIP not support ipv6

Ⅲ. Describe what you expected to happen

export --bind-ip flags or change default bind ip ‘0.0.0.0’ to ‘::’ advertise-ip support ipv6

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

dragonfly version:

OS (e.g. from /etc/os-release):

Kernel (e.g. uname -a):

Install tools:

Others:
chore(deps): bump github.com/valyala/fasthttp from 1.3.0 to 1.34.0
Bumps github.com/valyala/fasthttp from 1.3.0 to 1.34.0.

Release notes

Sourced from github.com/valyala/fasthttp's releases.

v1.34.0

59f94a3 Update github.com/klauspost/compress (#1237) (Mikhail Faraponov)

62c15a5 Don't reset RequestCtx.s (#1234) (Erik Dubbelboer)

7670c6e Fix windows tests (#1235) (Erik Dubbelboer)

f54ffa1 feature: Keep the memory usage of the service at a stable level (#1216) (Rennbon)

15262ec Warn about unsafe ServeFile usage (#1228) (Erik Dubbelboer)

1116d03 Fix panic while reading invalid trailers (Erik Dubbelboer)

856ca8e Update dependencies (#1230) (Mikhail Faraponov)

6b5bc7b Add windows support to normalizePath (Erik Dubbelboer)

f0b0cfe Don't log ErrBadTrailer by default (Erik Dubbelboer)

6937fee fix: (useless check), skip Response body if http method HEAD (#1224) (Pavel Burak)

b85d2a2 Fix http proxy behavior (#1221) (Aoang)

ad8a07a RequestHeader support set no default ContentType (#1218) (Jack.Ju)

c94581c support configure HostClient (#1214) (lin longhjui)

632e222 Client examples (#1208) (Sergey Ponomarev)

6a3cc23 uri_test.go use example.com for clearness (#1212) (Sergey Ponomarev)

9d665e0 Update dependencies (#1204) (Mikhail Faraponov)

8d7953e Fix scheme check for not yet parsed requests (#1203) (ArminBTVS)

v1.33.0

61aa8b1 remove redundant code (#1202) (tyltr)

4369776 fix(hijack): reuse RequestCtx (#1201) (Sergio VS)

2aca3e8 fix(hijack): reset userValues after hijack handler execution (#1199) (Sergio VS)

9123060 Updated dependencies (#1194) (Mikhail Faraponov)

v1.32.0

7eeb00e Make tests less flaky (#1189) (Erik Dubbelboer)

d19b872 Update tcpdialer.go (#1188) (Mikhail Faraponov)

c727b99 Release UseHostHeader in ReleaseRequest() (#1185) (Tolyar)

6c0518b Fix UseHostHeader for DoTimeout + tests (#1184) (Tolyar)

6b55811 Add MaxIdleWorkerDuration to Server. (#1183) (Kilos Liu)

4517204 Allow to set Host header for Client (#1169) (Tolyar)

258a4c1 fix: reset response after reset user values on keep-alive connections (#1176) (Sergio VS)

e9db537 Use %w to wrap errors (#1175) (Erik Dubbelboer)

7db0597 Fix bad request trailer panic (Erik Dubbelboer)

4aadf9a Fix parseTrailer panic (Erik Dubbelboer)

da7ff7a Add trailer support (#1165) (ichx)

017f0aa fix: reset request after reset user values on keep-alive connections (#1162) (Sergio VS)

3b117f8 feat: close idle connections when server shutdown (#1155) (ichx)

a94a2c3 Remove redundant code (#1154) (ichx)

f7c354c Fix race condition in Client.mCleaner (Erik Dubbelboer)

c078a9d Add string and bytes buffer convert trick in README (#1151) (ichx)

3ff6aaa uri: isHttps() and isHttp() (#1150) (Sergey Ponomarev)

8febad0 http.go: Request.SetURI() (Fix #1141) (#1148) (Sergey Ponomarev)

2ca01c7 fix: Status Line parsing and writing (#1135) (Shivansh Vij)

931d0a4 Fix lint (Erik Dubbelboer)

d613502 use sync.map is better (#1145) (halst)

c15e642 Don't run all race tests on windows (#1143) (Erik Dubbelboer)

6006c87 chore (#1137) (tyltr)

6d4db9b Fix race condition in getTCPAddrs (Erik Dubbelboer)

... (truncated)

Commits

59f94a3 Update github.com/klauspost/compress (#1237)

62c15a5 Don't reset RequestCtx.s (#1234)

7670c6e Fix windows tests (#1235)

f54ffa1 feature: Keep the memory usage of the service at a stable level (#1216)

15262ec Warn about unsafe ServeFile usage (#1228)

1116d03 Fix panic while reading invalid trailers

856ca8e Update dependencies (#1230)

6b5bc7b Add windows support to normalizePath

f0b0cfe Don't log ErrBadTrailer by default

6937fee fix: (useless check), skip Response body if http method HEAD (#1224)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.
knative pull image by Dragonfly is error

Question

my deploy knative service yaml :

apiVersion: serving.knative.dev/v1 kind: Service metadata: name: nginx namespace: cvd spec: template: metadata: name: nginx-v3 annotations: autoscaling.knative.dev/minScale: "1" autoscaling.knative.dev/maxScale: "11" autoscaling.knative.dev/target: "122" spec: containers: - env: value: "2" image: tools/nginx:1.9.1 imagePullPolicy: IfNotPresent name: user-container

/etc/docker/daemon.json:

{ "registry-mirrors": [ "http://127.0.0.1:65001" ], "max-concurrent-downloads": 10, "log-driver": "json-file", "log-level": "warn", "log-opts": { "max-size": "10m", "max-file": "3" }, "data-root":"/apps/docker" }

error: Unable to fetch image "tools/nginx:1.9.1": failed to resolve image to digest: Get "https://index.docker.io/v2/": dial tcp: lookup index.docker.io on 10.96.0.10:53: no such host

docker info :

Client: Debug Mode: false

Server: Containers: 50 Running: 22 Paused: 0 Stopped: 28 Images: 24 Server Version: 19.03.0 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-1160.el7.x86_64 Operating System: BigCloud Enterprise Linux 7.6 (Core) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 62.42GiB Name: 10.253.173.21 ID: NZR5:RHYS:MZDG:X3SU:JDEF:ZCG7:QPPV:VYVK:PHL2:SA3D:JKKY:44NH Docker Root Dir: /apps/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: docker.dep.devops.cmit.cloud:20100 127.0.0.0/8 Registry Mirrors: http://127.0.0.1:65001/ Live Restore Enabled: false Product License: Community Engine

create pod by Dragonfly is success but knative is erro

dfget变僵尸进程

Ⅰ. dfget变僵尸进程

[root@700 ~]# ps aux  | grep dfget
root      5289  0.0  0.0  12132  1156 pts/1    S+   10:07   0:00 grep --color=auto dfget
root     29631  0.0  0.0      0     0 ?        Z    Dec09   0:01 [dfget] <defunct>
[root@700 ~]#

Ⅱ. 错误日志

dfclient.log

2021-12-09 22:17:45.987 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:46.541 ERRO sign:158-1639058589.746 : failed to report piece{taskid:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8,range:352321536-358612991},err: dialing to the given TCP address timed out
2021-12-09 22:17:46.541 WARN sign:158-1639058589.746 : failed to report piece to supernode with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) for (1) times and will retry after sleep 0.359s
2021-12-09 22:17:48.388 WARN sign:158-1639058589.746 : success to report piece with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) after retrying (2) times
2021-12-09 22:17:48.988 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : register fail:{"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : failed to download piece: {"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.989 ERRO sign:158-1639058589.746 : failed to download by dragonfly: failed to download with p2p pattern, reason: 3, and start try to download from source
2021-12-09 22:17:48.989 INFO sign:158-1639058589.746 : start download b4dffbe3-7409-450f-84e1-1f1f7dfbacec from the source station
2021-12-10 00:10:22.990 INFO sign:158-1639058589.746 : download FAIL cost:7633.244s length:442008134 reason:3 error:failed to download file from source: download timeout(6754.000s)
2021-12-09 22:17:46.541 ERRO sign:158-1639058589.746 : failed to report piece{taskid:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8,range:352321536-358612991},err: dialing to the given TCP address timed out
2021-12-09 22:17:46.541 WARN sign:158-1639058589.746 : failed to report piece to supernode with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) for (1) times and will retry after sleep 0.359s
2021-12-09 22:17:48.388 WARN sign:158-1639058589.746 : success to report piece with request(&{TaskID:5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 Cid:dockerip-158-1639058589.746 DstCid:cdnnode:dockerip~5e7d99190ab19eeb996bd25ecd612fa7fbedf045b32b398d754d94ed4bff9fd8 PieceRange:352321536-358612991}) after retrying (2) times
2021-12-09 22:17:48.988 INFO sign:158-1639058589.746 : do register to x.x.x.x:8002, res:null error:dialing to the given TCP address timed out
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : register fail:{"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.988 ERRO sign:158-1639058589.746 : failed to download piece: {"Code":-100,"Msg":"dialing to the given TCP address timed out"}
2021-12-09 22:17:48.989 ERRO sign:158-1639058589.746 : failed to download by dragonfly: failed to download with p2p pattern, reason: 3, and start try to download from source
2021-12-09 22:17:48.989 INFO sign:158-1639058589.746 : start download b4dffbe3-7409-450f-84e1-1f1f7dfbacec from the source station
2021-12-10 00:10:22.990 INFO sign:158-1639058589.746 : download FAIL cost:7633.244s length:442008134 reason:3 error:failed to download file from source: download timeout(6754.000s)

dfserver.log

2021-12-09 22:03:09.456 INFO sign:63-1639058328.796 : update total limit to 209715200
2021-12-09 22:03:09.526 ERRO sign:63-1639058328.796 : failed to open file:4416a668-508a-45c4-b1f3-7480a641197d-112-1639058534.194, failed to get taskPath: 4416a668-508a-45c4-b1f3-7480a641197d-112-1639058534.194
2021-12-09 22:03:09.899 INFO sign:63-1639058328.796 : update total limit to 209715200
2021-12-09 22:03:10.384 ERRO sign:63-1639058328.796 : failed to open file:b3fbc506-56b8-4b30-bff9-d01b37e22408-85-1639058322.810, failed to get taskPath: b3fbc506-56b8-4b30-bff9-d01b37e22408-85-1639058322.810
2021-12-09 22:03:10.384 ERRO sign:63-1639058328.796 : failed to open file:8e772c19-3fac-4910-b98d-3b628986ca39-86-1639058333.073, failed to get taskPath: 8e772c19-3fac-4910-b98d-3b628986ca39-86-1639058333.073
2021-12-09 22:03:10.385 ERRO sign:63-1639058328.796 : failed to open file:060b5cc5-1a6d-4425-a83f-d957b4368b1c-88-1639058325.280, failed to get taskPath: 060b5cc5-1a6d-4425-a83f-d957b4368b1c-88-1639058325.280
2021-12-09 22:03:10.385 ERRO sign:63-1639058328.796 : failed to open file:50623dd5-9c9e-4422-8e22-3cba78c5ed52-86-1639058371.813, failed to get taskPath: 50623dd5-9c9e-4422-8e22-3cba78c5ed52-86-1639058371.813
2021-12-09 22:03:10.567 ERRO sign:63-1639058328.796 : failed to open file:396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241, failed to get taskPath: 396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241
2021-12-09 22:03:10.729 ERRO sign:63-1639058328.796 : failed to open file:396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241, failed to get taskPath: 396baa34-edc9-4ae5-9629-1f1049a6c036-87-1639058323.241
2021-12-09 22:03:10.885 ERRO sign:63-1639058328.796 : failed to open file:1f005af6-5e1d-4e72-8f79-98c1db778d1e-88-1639058333.660, failed to get taskPath: 1f005af6-5e1d-4e72-8f79-98c1db778d1e-88-1639058333.660
2021-12-09 22:03:11.045 ERRO sign:63-1639058328.796 : failed to open file:fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307, failed to get taskPath: fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307
2021-12-09 22:03:11.203 ERRO sign:63-1639058328.796 : failed to open file:fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307, failed to get taskPath: fc13b11b-7655-4fa6-9793-4e0967ed8749-87-1639058327.307
2021-12-09 22:06:23.581 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/1864bb3f-b17d-470a-bfc7-e61b2446967b-26-1639058328.552.service
2021-12-09 22:06:23.727 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/a0006b05-0c17-47ec-9e63-ee8689f12c55-146-1639058589.423.service
2021-12-09 22:20:25.299 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/cae62168-b4ca-4c50-932d-77e8a73421eb-134-1639058385.453
2021-12-09 22:20:25.303 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/cae62168-b4ca-4c50-932d-77e8a73421eb-134-1639058385.453.service
2021-12-09 22:20:55.832 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/b4dffbe3-7409-450f-84e1-1f1f7dfbacec-158-1639058589.746
2021-12-09 22:20:55.832 INFO sign:63-1639058328.796 : server gc, delete file:/root/.small-dragonfly/data/b4dffbe3-7409-450f-84e1-1f1f7dfbacec-158-1639058589.746.service
2021-12-09 22:22:00.464 INFO sign:63-1639058328.796 : no more task, peer server will stop...
2021-12-09 22:22:00.466 INFO sign:63-1639058328.796 : peer server is shutdown.

Ⅲ. Environment:

dragonfly version:

dragonflyoss/dfclient:latest
dragonflyoss/supernode:latest

OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Kernel (e.g. uname -a):

Linux 700.link.china 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Wed Feb 5 02:00:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Install tools:
Others:

supernode 3台
下载镜像1.4G
dfclient 50台

Fix goroutine leak bug in DoDownloadTimeout

Ⅰ. Describe what this PR did

This PR fix a goroutine leak bug in DoDownloadTimeout

Ⅱ. Does this pull request fix one issue?

NONE

Ⅲ. Why don't you add test cases (unit test/integration test)?

This fix is already covered by TestDoDownloadTimeout

Ⅳ. Describe how to verify it

In the original code, DoDownloadTimeout has a potential goroutine leak in rare goroutine intervening. It can be verified by using https://github.com/uber-go/goleak if we instrument the entry of func Test(t *testing.T) in dfget/core/downloader/downloader_test.go and then run TestDoDownloadTimeout enough times:

go test github.com/dragonflyoss/Dragonfly/dfget/core/downloader -gocheck.f=TestDoDownloadTimeout

This command results in :

time="2021-11-01T13:37:27+08:00" level=warning msg="invalid download timeout(0.000s), use default:(300.000s)"
leaks.go:78: found unexpected goroutines:
    [Goroutine 8 in state chan send, with github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout.func1 on top of the stack:
    goroutine 8 [chan send]:
    github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout.func1()
            /home/yuanting/work/dev/goprojects/dragonfly/dfget/core/downloader/downloader.go:56 +0x12f
    created by github.com/dragonflyoss/Dragonfly/dfget/core/downloader.DoDownloadTimeout
            /home/yuanting/work/dev/goprojects/dragonfly/dfget/core/downloader/downloader.go:55 +0x205
        ]

Dragonfly is an intelligent P2P based image and file distribution system.

Dragonfly

Contents

Introduction

Features

Comparison

Roadmap

Community

Contributing

License

Owner

dragonflyoss

Comments

Add Continuous Fuzzing via Fuzzit

Dragonfly does not work

Ⅰ. Issue Description

Ⅱ. Describe what happened

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

Fix an issue where HTTPS is not supported

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

When dfget specifies -- node node, dfget still synchronizes old files after the supernode file is updated

Question

[Question] dfget fails to pull images

Question

dfget 本地下载报错

docker proxy error

Using GOPROXY environment variables if set instead of default value

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

feature: support non-ipv4 format node address for Dragonfly dfclient

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

feature: update API definition to be more accurate

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

fix valid ip not support ipv6

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)? (你真的觉得不需要加测试吗？)

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

ipv6 support

Ⅰ. Issue Description

Ⅱ. Describe what happened

Ⅲ. Describe what you expected to happen

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

chore(deps): bump github.com/valyala/fasthttp from 1.3.0 to 1.34.0

v1.34.0

v1.33.0

v1.32.0

knative pull image by Dragonfly is error

Question

dfget变僵尸进程

Ⅰ. dfget变僵尸进程

Ⅱ. 错误日志

Ⅲ. Environment:

Fix goroutine leak bug in DoDownloadTimeout

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews