Without the bootstrapDns parameter in the config DoH queries take a very long time (>2000ms).

And additionally I saw a lot of i/o timouts when making a lot of requests to blocky without a bootstrap DNS.

Configuration:

upstream:
  default:
    - https://dns.quad9.net/dns-query

queryLog:
  type: console

#bootstrapDns: tcp+udp:1.1.1.1
port: 53

As soon as the bootstrap DNS is enabled the the durations of the queries are at a normal level again and the i/o timeouts are gone.

• release workflow doesn't run on forks anymore
• stale workflow doesn't run on forks anymore
• added .git folder to .dockerignore and substituted it with build-args(VERSION & BUILD_TIME)
• removed grype in favor of trivy(grype seams to have problems with scratch images)
• development-docker workflow can be enabled on forks by setting a secret
• ~~automatically initialize buildcache tag on forks~~
• docker layer caching optimizations
• crosscompile arm binaries with go + zig
• test run docker images
• added Architecture to version command
• ci-build as matrix

Hello, I have a problem that has been bothering me for a long time, I hope you can help me

This is an issue I left with the original author, but the author has not replied to me at present, I am currently using a macos device

This simple dns I wrote can use dig but not nslookup, including curl and so on can not be resolved

This is a simple dns server, dig can be used, but nslookup can't be used, curl can't be used either https://gist.github.com/walm/0d67b4fb2d5daf3edd4fad3e13b162cb

https://github.com/miekg/dns/issues/1396

As title, I've noticed that Blocky seems to keep query upstream domain via bootstrap DNS even after bootstrapped, and it's frequently, not sure if it's expected? Thinking about maybe once the query to encrypted DNS upstream is established, should use the encrypted DNS to look up the upstream's IP addresses?

Here's the partial of the timestamp records that I observed the query to of upstream encrypted DNS's domain name. I use only single one upstream in this case, and its TTL of the upstream domain is 300 secs, it looks like the query is much more frequent than the TTL.

15:46:31
15:46:33
15:46:34
15:46:34
15:46:35
15:46:36
15:46:36
15:46:37
15:46:38
15:47:36
15:48:27
15:48:42
15:48:45
15:48:47
15:49:37
15:49:54
15:50:39
15:50:49
15:50:51
15:50:52
15:50:54
15:51:37
15:51:38
15:52:09
15:52:18
15:52:19
15:52:20
15:52:21
15:52:22
15:52:23
15:52:24
15:52:25
15:53:07
15:53:18
15:53:42
15:53:43
15:53:44
15:53:45
15:53:58
15:54:00
15:54:02
15:54:02
15:54:08
15:54:25
15:54:29
15:54:50
15:55:36
15:55:52
15:55:55
15:56:22
15:57:23
15:57:36
15:57:40
15:57:40
15:57:41

I have caching/prefetching set up pretty generously, however it seems that some queries are slow to look up when it should already be cached.

logLevel: warn

upstream:
  default:
    - https://8.8.8.8/dns-query
    - https://8.8.4.4/dns-query

caching:
  minTime: 5m
  maxTime: 30m
  prefetching: true
  prefetchExpires: 24h
  prefetchThreshold: 2

If blocky is deployed on multiple instances for concurrency and/or failsafe the cache most likely will differ. This will cause spikes in response time during instance switches.

I'd like to propose an external second level cache for blocky. Redis would be logical solution as it's already been used in similar scenarios(unbound cache db).

If activated this feature would include:

• populate blocky cache from redis during startup
• query redis after cache miss
• update redis entry on cache insertion/update

Feature request:

Resolve bootstrap DNS queries which aren't FQDN against 127.0.0.11 instead of bootstrapDNS configuration(enable via configuration and disabled as default).

Background:

In docker stacks service names are resolved through 127.0.0.11. Currently it has to be the bootstrap DNS configuration if a service is used as an upstream resolver. With this workaround the list download DNS resolution is done through the host DNS configuration. This is a problem if blocky is set as DNS resolver for the host. The proposed feature would eliminate this problem as service names aren't typically FQDN.

Possible FQDN check function: link

Using blocky in the latest version in docker...

the website ravelry.com does not work. The host style-cdn.ravelrycache.com resolves as CNAME to g2s2z8r3.stackpathcdn.com but that does not work with blocky. The domain is not blocked (blacklisted), there is simply no entry that this host was queried.

With pihole, there is no problem.....

Error received:

This app can't run on your PC To find a version for your PC, check with the software publisher.

Latest 0.8 release AMD64 Windows build.

This is a cpu graph from observium.

The host is a vm (i run arch btw..) 4 cpus, 4G, xanmod lts 5.15.40 vmxnet3, etc..

about 4k (+/- 10%) chromebooks, via wifi with this vm as their only recursor..

(blocky is sitting in front of dnscrypt-proxy)

---

port: 10.20.8.29:53
httpPort: 4000
bootstrapDns: tcp+udp:10.20.8.29:40

upstream:
  default:
    - tcp+udp:10.20.8.29:40

upstreamTimeout: 5s

filtering:
  queryTypes:
    - AAAA
    - HTTPS
    - ANY

caching:
  minTime: 2400s
  maxTime: 9600s
  maxItemsCount: 32768
  prefetching: true

logLevel: warn
# optional: Log level (one from debug, info, warn, error). Default: info
logTimestamp: false

Yesterday was the first day of only blocky as the recursor..

Previously it was coredns..

Will see how it looks today..

also.. can't seem to get the sigusr1 things to dump with the logLevel at warn..

 sv status blocky
run: blocky: (pid 163871) 832s; run: log: (pid 31930) 65530s

kill -s SIGUSR1 163871


tail -F /var/log/blocky/current
ERROR server: error on processing request:resolution was not successful, used resolvers: 'upstream 'tcp+udp:10.20.8.29:40'' and 'upstream 'tcp+udp:10.20.8.29:40'' errors: [can't resolve request via upstream server 10.20.8.29:40: dns: overflowing header size can't resolve request via upstream server 10.20.8.29:40: dns: overflowing header size]
ERROR server: error on processing request:resolution was not successful, used resolvers: 'upstream 'tcp+udp:10.20.8.29:40'' and 'upstream 'tcp+udp:10.20.8.29:40'' errors: [can't resolve request via upstream server 10.20.8.29:40: dns: overflowing header size can't resolve request via upstream server 10.20.8.29:40: dns: overflowing header size]
ERROR can't write message: write tcp 10.20.8.29:53->10.120.46.224:37166: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34184: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34244: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34284: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34208: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34378: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34498: write: broken pipe
ERROR can't write message: write tcp 10.20.8.29:53->10.120.63.169:34324: write: broken pipe

set the logLevel to info..

 INFO server: current configuration:
 INFO server: -> resolver: 'FilteringResolver'
 INFO server:      filtering query Types: 'AAAA, ANY, HTTPS'
 INFO server: -> resolver: 'ClientNamesResolver'
 INFO server:      deactivated, use only IP address
 INFO server: -> resolver: 'QueryLoggingResolver'
 INFO server:      type: "console"
 INFO server:      target: ""
 INFO server:      logRetentionDays: 0
 INFO server: -> resolver: 'MetricsResolver'
 INFO server:      metrics:
 INFO server:        Enable = false
 INFO server:        Path   = /metrics
 INFO server: -> resolver: 'CustomDNSResolver'
 INFO server:      deactivated
 INFO server: -> resolver: 'HostsFileResolver'
 INFO server:      deactivated
 INFO server: -> resolver: 'BlockingResolver'
 INFO server:      deactivated
 INFO server: -> resolver: 'CachingResolver'
 INFO server:      minCacheTimeInSec = 2400
 INFO server:      maxCacheTimeSec = 9600
 INFO server:      cacheTimeNegative = 30 minutes
 INFO server:      prefetching = true
 INFO server:      prefetchExpires = 2 hours
 INFO server:      prefetchThreshold = 5
 INFO server:      cache items count = 624
 INFO server: -> resolver: 'ConditionalUpstreamResolver'
 INFO server:      deactivated
 INFO server: -> resolver: 'ParallelBestResolver'
 INFO server:      upstream resolvers:
 INFO server:      - default
 INFO server:        - upstream 'tcp+udp:10.20.8.29:40'
 INFO server: - DNS listening on addrs/ports: [10.20.8.29:53]
 INFO server: - TLS listening on addrs/ports: []
 INFO server: - HTTP listening on addrs/ports: [4000]
 INFO server: - HTTPS listening on addrs/ports: []
 INFO server: runtime information:
 INFO server: MEM Alloc =                 1 MB
 INFO server: MEM HeapAlloc =             1 MB
 INFO server: MEM Sys =                  15 MB
 INFO server: MEM NumGC =                95
 INFO server: RUN NumCPU =                4
 INFO server: RUN NumGoroutine =         17

Open to suggestions..

Currently, bootstrap DNS can be plain DNS only (UDP/TCP via 53). It should be possible to use DoT/DoH as bootstrap DNS. If host system has no valid DNS configuration, bootstrap should be defined as IP address only.

Current situation:

• Implementation of REST endpoints (e.g.https://github.com/0xERR0R/blocky/blob/development/api/api_endpoints.go#L65) is a standard http implementation with some "annotations" as comment
• with swaggo (see Makefile) we generate "swagger.json". This is manual step and must be done on each REST endpoint change
• with "bootprint.js" we generate a static HTML page with Swagger description
• with "html-inline" we generate a single HTML with embedded swagger JS files (docs/swagger.html)
• "docs/swagger.html" is committed and versioned with GIT
• Blocky's main HTML page references swagger.html on Github with "htmlpreview.github.io" as proxy

This approach works but it has too much moving parts and also some drawbacks:

• it needs a bunch of external tools
• manual build steps required
• no technical API definition file provided
• requires external services and internet to show the API definition
• swagger JSON definition is not "the state of the art", it was "replaced" with OpenAPI

It would be better to use the API-first approach

• Definition of OpenAPI.yaml
• automatic code generation with OpenAPI generator
• Embed OpenAPI.yaml in binary and documentation

The link to the API in the HTTP server root (Swagger Rest API Documentation (Online @GitHub)) is pointing to https://htmlpreview.github.io/?https://github.com/0xERR0R/blocky/blob/0.20/docs/swagger.html which returns a 404.

Blocky is my first attempt at grafana and prometheus

for starters this was great: https://grafana.com/grafana/dashboards/13768-blocky/

this was not great: https://github.com/0xERR0R/blocky-grafana-prometheus-example

As I've never had a working grafana/prometheus I'm not sure what is a problem and what might be 'self inflicted'

(running void linux (glibc not musl) fwiw)

 tail -F /var/log/socklog/everything/current
2022-12-12T18:52:08.87858 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=0 orgId=0 uname= t=2022-12-12T13:52:08.87-0500 lvl=info msg="Request Completed" method=GET path=/d/JvOqE4gRk/blocky status=302 remote_addr=10.20.250.18 time_ms=0 duration=242.444µs size=29 referer= traceID=00000000000000000000000000000000
2022-12-12T18:52:18.66284 daemon.info: Dec 12 13:20:33 grafana: logger=http.server t=2022-12-12T13:52:18.66-0500 lvl=info msg="Successful Login" User=admin@localhost
2022-12-12T18:52:19.32945 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T13:52:19.32-0500 lvl=info msg="Request Completed" method=GET path=/api/live/ws status=0 remote_addr=10.20.250.18 time_ms=2 duration=2.864183ms size=0 referer= traceID=00000000000000000000000000000000
2022-12-12T18:53:01.16633 daemon.info: Dec 12 13:20:33 grafana: logger=tsdb.prometheus t=2022-12-12T13:53:01.16-0500 lvl=eror msg="Range query failed" query="sum(increase(blocky_prefetch_hit_count[300s])) / (sum(increase(blocky_cache_hit_count[300s])))" err="Post \"http://localhost:9090/api/v1/query_range\": context canceled"
2022-12-12T18:53:01.16670 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T13:53:01.16-0500 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.20.250.18 time_ms=4 duration=4.369778ms size=98 referer="http://10.20.0.15:3000/d/JvOqE4gRk/blocky?orgId=1&from=now-5m&to=now&refresh=10s" traceID=00000000000000000000000000000000
2022-12-12T18:53:01.17926 daemon.info: Dec 12 13:20:33 grafana: logger=tsdb.prometheus t=2022-12-12T13:53:01.17-0500 lvl=eror msg="Range query failed" query="sum(blocky_prefetch_domain_name_cache_count)/ sum(up{job=\"blocky\"})" err="Post \"http://localhost:9090/api/v1/query_range\": context canceled"
2022-12-12T18:53:01.17937 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T13:53:01.17-0500 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.20.250.18 time_ms=1 duration=1.254691ms size=98 referer="http://10.20.0.15:3000/d/JvOqE4gRk/blocky?orgId=1&from=now-5m&to=now&refresh=10s" traceID=00000000000000000000000000000000
2022-12-12T18:54:24.32408 daemon.info: Dec 12 13:20:33 grafana: logger=tsdb.prometheus t=2022-12-12T13:54:24.32-0500 lvl=eror msg="Range query failed" query="sum by (client) (rate(blocky_query_total[5m])) * 60" err="Post \"http://localhost:9090/api/v1/query_range\": context canceled"
2022-12-12T18:54:24.32420 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T13:54:24.32-0500 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.20.250.18 time_ms=242 duration=242.716032ms size=98 referer="http://10.20.0.15:3000/d/JvOqE4gRk/blocky?orgId=1&from=now-5m&to=now&refresh=10s" traceID=00000000000000000000000000000000
2022-12-12T18:54:24.41804 daemon.info: Dec 12 13:37:49 prometheus: ts=2022-12-12T18:54:24.417Z caller=api.go:1540 level=error component=web msg="error writing response" bytesWritten=0 err="write tcp 127.0.0.1:9090->127.0.0.1:23280: write: broken pipe"
2022-12-12T20:13:55.65922 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T15:13:55.65-0500 lvl=info msg="Request Completed" method=GET path=/api/live/ws status=0 remote_addr=10.20.250.18 time_ms=4 duration=4.738284ms size=0 referer= traceID=00000000000000000000000000000000

250.18 is where I am.. viewing the instance at 0.15. Grafana, Prometheus, and Blocky are all on 0.15

I cannot print out as a pdf the grafana dashboard..

Is there a problem here? or did I do something wrong?

this looks and seems accurate

this also seems accurate and correct

This is a three hour view, aside from the html; the Cache Hit/Miss ratio is that 97.4% hit or miss?

and this happens a lot..

logs:

2022-12-12T20:13:55.65922 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T15:13:55.65-0500 lvl=info msg="Request Completed" method=GET path=/api/live/ws status=0 remote_addr=10.20.250.18 time_ms=4 duration=4.738284ms size=0 referer= traceID=00000000000000000000000000000000
2022-12-12T20:18:26.87177 daemon.info: Dec 12 13:20:33 grafana: logger=tsdb.prometheus t=2022-12-12T15:18:26.86-0500 lvl=eror msg="Range query failed" query="sum by (client) (rate(blocky_query_total[5m])) * 60" err="Post \"http://localhost:9090/api/v1/query_range\": context canceled"
2022-12-12T20:18:26.87634 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T15:18:26.87-0500 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.20.250.18 time_ms=84 duration=84.325457ms size=98 referer="http://10.20.0.15:3000/d/JvOqE4gRk/blocky?orgId=1&from=now-5m&to=now&refresh=10s" traceID=00000000000000000000000000000000
2022-12-12T20:21:36.12626 daemon.info: Dec 12 13:20:33 grafana: logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-12-12T15:21:36.12-0500 lvl=info msg="Request Completed" method=GET path=/api/live/ws status=0 remote_addr=10.20.250.18 time_ms=1 duration=1.460355ms size=0 referer= traceID=00000000000000000000000000000000

curl -v --stderr - http://127.0.0.1:4000/metrics | grep -c client
3933

in case this is important

xbps-query -l | grep 'grafana\|prometheus'
ii grafana-8.5.3_1                     Open platform for beautiful analytics and monitoring
ii prometheus-2.38.0_1                 Monitoring system and time series database

machine is a vm, two cores, 2G ram, xfs, 6.0.9 kernel.

As I said this is the closest I've been to having a working grafana instance, so thank you for that.

preface.. I am not a git aficionado.. I only pretend to use git..

 docker exec blocky /app/blocky version
[2022-12-05 19:11:43]  WARN 'logTimestamp' is deprecated. Please use 'log.timestamp' instead.
[2022-12-05 19:11:43]  WARN 'port' is deprecated. Please use 'ports.dns' instead.
blocky
Version: v0.18-447-gd501d86
Build time: 20221205-015954
Architecture: amd64

The commit tag in the Version is correct.. but the 0.18..

I had this happen on a few machines as well.. I needed to rm the clone dir and start over..

then I got a build version without the 0.18.. release says 0.20

right side is git clone --depth 1

left is is git clone

git tag shows up to v0.18 from git clone

git tag shows nothing in --depth 1

Not sure if this is helpful or noise..

sorry if it is noise..

If more than one upstream is configured, they are raced via a ParallelBestResolver.

Other improvements:

• feat(bootstrap): support IP only encrypted DNS Also make tcp+udp upstreams use any IPs provided.
• feat: always prefetch upstream IPs to avoid stalling user queries Otherwise, a request to blocky could end up waiting for 2 DNS requests:
  1. lookup the DNS server IP
  2. forward the user request to the server looked-up in 1
• feat: stack log prefixes to differentiate between log emitters The goal is to be able to tell apart logs from difference sources, such as bootstrap.parallel_best_resolver and parallel_best_resolver.

Also includes #781 so the lint passes here.

Fixes #760.