package main

import (
	"log"
	"net/http"

	"github.com/gocolly/colly/v2"
)

func main() {
	// we have to disable Accept-Encoding: gzip,
	// because the remote server might send compressed
	// 404 page, avoiding the bug
	tr := &http.Transport{
		DisableCompression: true,
	}

	c := colly.NewCollector()
	c.WithTransport(tr)

	c.OnError(func(resp *colly.Response, err error) {
		log.Printf("err=%v", err)
	})

	c.Visit("https://example.org/sitemap.xml.gz")
}

Results in:

2023/01/05 22:30:27 err=gzip: invalid header

I'm using Request Context to store information about the parsed body on various c.OnHTML callbacks..

So what happens is, if I use the e.Request.Visit() for following up on hrefs, then the request context is also being passed. I wanted to avoid this. So instead of using e.Request.Visit() I used c.Visit() directly. This made sure that I got new context for each request.

However, I would like to use the MaxDepth option as well. But that only works if I use the e.Request.Visit().

It would work for me to use the e.Request.Visit() but give new context for each request. This is currently not possible. Is that correct?

If yes, this feature request would be great to have as a configuration option - to determine if the request context has to be passed along or not..

For now, I have manually made the change for local purposes..

index 6beef834..524bb77c 100644
--- a/vendor/github.com/gocolly/colly/v2/request.go
+++ b/vendor/github.com/gocolly/colly/v2/request.go
@@ -117,7 +117,7 @@ func (r *Request) AbsoluteURL(u string) string {
 // request and preserves the Context of the previous request.
 // Visit also calls the previously provided callbacks
 func (r *Request) Visit(URL string) error {
-	return r.collector.scrape(r.AbsoluteURL(URL), "GET", r.Depth+1, nil, r.Ctx, nil, true)
+	return r.collector.scrape(r.AbsoluteURL(URL), "GET", r.Depth+1, nil, nil, nil, true)
 }
 
 // HasVisited checks if the provided URL has been visited```

I'm running a test project on localhost:8000 and when I access it over https, it fails (which is expected)

Get "https://localhost:8000/": tls: first record does not look like a TLS handshake

The above is correctly caught in OnError. However, when I set ignoreRobots to false, then it tries to fetch the robots.txt and the below failure

Get "https://localhost:8000/robots.txt": tls: first record does not look like a TLS handshake

Is not propogated to OnError - as it is really not originating from the request that I had started, but colly tries to first fetch the robots which fails.. Could this also be propogated either to OnError or can be caught with a known Error Code from Colly such as

ErrRobotsTxtBlocked = errors.New("URL blocked by robots.txt")
ErrRobotsTxtFetchFailed = errors.New("Unable to fetch robots.txt") // New Error Code

https://github.com/gocolly/colly/blob/947eeead97b39d46ce2c89b06164c78b39d25759/queue/queue.go#L113

stuck on q.wake <- struct{}{}

because q.wake already not used by queue.Run().