simple_server.zip Reading the response of a GET request using .String() arbitrarily cuts off the full response when the response is quite large (+5MB). The issue appears to occur when reading the internal buffer from the response object; the full response string is not returned.

Using http.Get() works fine:

Attached is a standalone project that reproduces the issue. The attached project has a simple server that returns a hard coded json string:

src/simple_server

There are also two clients:

src/grequests_client (reproduces the issue) src/http_client (works fine)

The error occurs in grequests_client when trying to decode the json response: 'use of closed network connection'. But inspecting the json response using resp.String() or resp.Byte() shows that only part of the json response is returned.

POST json body now use json.Encoder#Encode()

This will result a trailing newline. It's mostly for json stream. Not useful in HTTP and will cause some problem.

Should use json.Marshal() instead.

resp, err := grequests.Post(url, options)
if err != nil || resp.StatusCode != 200 {
    ...
    return
}

above is my code, and after about 2 weeks, I could get an error as socket: too many open files. I don't know whether it due to the response unclosed. resp only used to get StatusCode, no resp.JSON or any other usages.
Should i must close the connection? like defer resp.Close(). Or what should I do to avoid this socket error?

session := grequests.NewSession(nil)        // session.HTTPClient.Transport initialized
session.RequestOptions.Proxies = proxie  // here will not rebuild HTTPClient.Transport, so proxy does not work

Look like problem in doFollowingRedirects()

...
        if redirect != 0 {
            nreq := new(Request)
            nreq.Method = ireq.Method
            if ireq.Method == "POST" || ireq.Method == "PUT" {
                nreq.Method = "GET"
            }
            nreq.Header = make(Header)
            nreq.URL, err = base.Parse(urlStr)
            if err != nil {
                break
            }
            if len(via) > 0 {
                // Add the Referer header.
                lastReq := via[len(via)-1]
                if ref := refererForURL(lastReq.URL, nreq.URL); ref != "" {
                    nreq.Header.Set("Referer", ref)
                }
...

There is only Referer header is set.

Is there a reason why this library doesn't provide the "go way" of error handling ?

resp := grequests.Get("http://httpbin.org/xml", nil)

if resp.Error != nil {
    log.Fatalln("Unable to make request", resp.Error)
}

VS

resp, err := grequests.Get("http://httpbin.org/xml", nil)

if err != nil {
    log.Fatalln("Unable to make request", err)
}

        proxyURL, err := url.Parse("http://127.0.0.1:8080") // Proxy URL
	if err != nil {
		log.Panicln(err)
	}

	resp, err := grequests.Get("http://www.google.com",
		&grequests.RequestOptions{Proxies: map[string]*url.URL{proxyURL.Scheme: proxyURL}})

	if err != nil {
		log.Println(err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp)

The code above will work as expected.

But, if you replace http://www.google.com to https://www.google.com, it will not work, it access google directly without the proxy you set!

As much as I like Grequests and I truly want more people to find about it, forcing default User-Agent is not the way.

https://github.com/levigross/grequests/blob/fd07961896a93d37e169271bb23ee423c42033d3/request.go#L558

Many platform will change behavior based on the User-Agent. Some of their WAF (web application firewall) will try to single out requests originating from the same IP, User-Agent, etc. There are situation where an empty User-Agent needs to be sent, literally User-Agent: and Grequests is preventing that from happening.

So I would suggest two solutions.

1. Create custom struct that can be set to empty, e.g.

// NullString represents a string that may be null.
type NullString struct {
	String string
	Valid  bool // Valid is true if String is not NULL
}

and if Valid is true then use the String provided in the struct.

2. Remove the check.

Requests allows you to create requests using the request function, instead of having to call the wrapper for the http method. I am trying to achieve the same functionality in grequests, but doRegularRequest is private.

working well as follow:

var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)

session := grequests.NewSession(&grequests.RequestOptions{
	Proxies:map[string]*url.URL{prox.Scheme:prox},
})
rsp,err := session.Get(ourl, nil)

fmt.Println(err, rsp.String())

not working as follow:

var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)

session := grequests.NewSession(nil)
rsp,err := session.Get(ourl, &grequests.RequestOptions{
	Proxies:map[string]*url.URL{prox.Scheme:prox},
})

fmt.Println(err, rsp.String())

Since we can configure the timeouts, I think we don't need the http.DefaultClient anymore. I'd like to know what you think about that @levigross.

BTW, now all tests are passing :)

Hey there!

I belong to an open source security research community, and a member (@ranjit-git) has found an issue, but doesn’t know the best way to disclose it.

If not a hassle, might you kindly add a SECURITY.md file with an email, or another contact method? GitHub recommends this best practice to ensure security issues are responsibly disclosed, and it would serve as a simple instruction for security researchers in the future.

Thank you for your consideration, and I look forward to hearing from you!

(cc @huntr-helper)

Hi, So basically if Proxies are added at the time of creating Session it works but on sending request Like Get or Post it doesn't work

Using Proxy while creating Request in Session

proxyURL, _ := url.Parse("http://127.0.0.1:8080")
session := grequests.NewSession(nil)

res , _ := session.Get("https://api.ipify.org?format=json",&grequests.RequestOptions{
        Proxies: map[string]*url.URL{
		"http": proxyURL,
		"https": proxyURL,
	},
})
fmt.Println(res.String())

This above request gives my own IP

Using Proxy while Creating Session

proxyURL, _ := url.Parse("http://127.0.0.1:8080")
session := grequests.NewSession(&grequests.RequestOptions{
	Proxies: map[string]*url.URL{
		"http": proxyURL,
		"https": proxyURL,
	},
})
res , _ := session.Get("https://api.ipify.org?format=json",nil)
fmt.Println(res.String())

This Above Request works

grequests is a great http client library. I like to use it to replace golang default http client. But in some case, I want to build Response from http.Response. I could not to find any way to do this. So, it is any way to export buildResponse function or create new function call BuildResponse?

Currently grequests always sets a filename when creating a multipart post request https://github.com/levigross/grequests/blob/master/request.go#L312 . This leads to inconsistent behaviors with python's requests.

More specifically, lots of web servers (e.g., go's net/http, python's flask) rely on the existence of 'filename' when parsing a multipart form. If 'filename' exists, the field will be parsed as a file (and put in Request.FormFile, for example); otherwise it will be parsed as a normal value (and put in Request.FormValue, for example).

As grequests always sets a filename, all fields will be parsed as files, while python's requests allows providing non-file fields using syntax like requests.post(xxx, files={'key': (None, json.dumps("{}"), "application/json")}).