1. Current workflow
   • Request to url + /favicon.ico
   • Doesn't check if the faviconResp.data is html or it's image
2. Implemented workflow
   • Request to url + /favicon.ico if failed, will scrape root path for link element with rel (icon or shortcut icon or mask-icon or apple-touch-icon)
   • Supports checking if the response content is actually an image

Do let me know if I need to make any changes :)

To replicate the issue of missing valid host with CIDR input

with prips:-

> prips 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | grep 1.1.1.1:80

http://1.1.1.1:80 [301] [186] [301 Moved Permanently]

with httpx internal CIDR handler: -

> echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | grep 1.1.1.1:80

To replicate the duplication issue with CIDR input

> echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | sort | grep http://1.1.1.24:80
http://1.1.1.24:80 [403] [16] []

echo 1.1.1.0/24 | httpx -title -content-length -status-code -ports 80,443 -silent | sort | grep http://1.1.1.24:80

http://1.1.1.24:80 [403] [16] []
http://1.1.1.24:80 [403] [16] []
http://1.1.1.24:80 [403] [16] []

I have opened ports 80 and 443 on 192.168.8.1

echo 192.168.8.1 | ./httpx  -title -content-length -status-code -ports 80,443

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   / 
 / / / / /_/ /_/ /_/ /   |  
/_/ /_/\__/\__/ .___/_/|_|  
             /_/              v1           

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://192.168.8.1:443 [307] [13] []
http://192.168.8.1:80 [307] [13] []

But when I run CIDR scan, the httpx does not return any open port on 192.168.8.1

echo 192.168.8.0/24 | ./httpx  -title -content-length -status-code -ports 80,443

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   / 
 / / / / /_/ /_/ /_/ /   |  
/_/ /_/\__/\__/ .___/_/|_|  
             /_/              v1           

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions
[WRN] Developers assume no liability and are not responsible for any misuse or damage.

Added flag to support probe all ips #245.

Example output: For default output, I just implicitly enable IP address output:

$ echo hackerone.com | ./httpx -sc -scan-all-ips
https://hackerone.com [302] [104.16.99.52]
https://hackerone.com [302] [104.16.100.52]

JSON (as described in the original issue):

{
  "timestamp": "2021-10-23T19:10:52.740432132+03:00",
  "scheme": "https",
  "port": "443",
  "path": "/",
  "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
  "header-sha256": "883abfaf1ec8e7c9270389ceb45e0c4b8a16264c933174b93b14766b6f66c85c",
  "a": [
    "104.16.100.52",
    "104.16.99.52"
  ],
  "url": "https://hackerone.com:443",
  "input": "hackerone.com",
  "location": "https://www.hackerone.com/",
  "webserver": "cloudflare",
  "content-type": "text/html",
  "method": "GET",
  "host": "104.16.100.52",
  "content-length": 92,
  "status-code": 302,
  "csp": {
    "domains": [
      "a5s.hackerone-ext-content.com",
      "b5s.hackerone-ext-content.com",
      "errors.hackerone.net",
      "profile-photos.hackerone-user-content.com",
      "www.youtube-nocookie.com",
      "www.google-analytics.com",
      "cover-photos.hackerone-user-content.com",
      "hackathon-photos.hackerone-user-content.com",
      "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
      "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d"
    ]
  },
  "response-time": "963.997528ms",
  "failed": false
}
{
  "timestamp": "2021-10-23T19:10:52.774901606+03:00",
  "scheme": "https",
  "port": "443",
  "path": "/",
  "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
  "header-sha256": "e129249144cc96d0bcf0273eb03e42f002169966d76179ffbd5d359264e5ea35",
  "a": [
    "104.16.100.52",
    "104.16.99.52"
  ],
  "url": "https://hackerone.com:443",
  "input": "hackerone.com",
  "location": "https://www.hackerone.com/",
  "webserver": "cloudflare",
  "content-type": "text/html",
  "method": "GET",
  "host": "104.16.99.52",
  "content-length": 92,
  "status-code": 302,
  "csp": {
    "domains": [
      "profile-photos.hackerone-user-content.com",
      "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d",
      "www.youtube-nocookie.com",
      "b5s.hackerone-ext-content.com",
      "www.google-analytics.com",
      "errors.hackerone.net",
      "cover-photos.hackerone-user-content.com",
      "hackathon-photos.hackerone-user-content.com",
      "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
      "a5s.hackerone-ext-content.com"
    ]
  },
  "response-time": "1.005377776s",
  "failed": false
}

httpx version:

Tested version: v1.2.3, v1.2.0

Current Behavior:

Some urls return 404 http response for valid 200 response urls. Was able to reproduce this on twitter site.

Expected Behavior:

It should return 200 response.

Steps To Reproduce:

echo "https://twitter.com/twitter" | httpx -fr -sc 

Response:
https://twitter.com/twitter [302,404]  [https://mobile.twitter.com/twitter]

Same command with redirected url return same response

echo https://mobile.twitter.com/twitter | httpx -fr -sc

https://mobile.twitter.com/twitter [404]

--------

echo "https://twitter.com/twitter" | httpx -fr -sc -x HEAD 

https://twitter.com/twitter [302,404] [HEAD] [https://mobile.twitter.com/twitter]

--------

echo https://mobile.twitter.com/twitter | httpx -fr -sc -http2

https://mobile.twitter.com/twitter [404] [http2]

Curl returns 200 status code for this url

curl -ik "https://mobile.twitter.com/twitter" -X HEAD

HTTP/2 200

Anything else:

The results are as expected

echo "www.baidu.com"|httpx   -title -content-length -status-code #auto use https
https://www.baidu.com [302] [161] [302 Found]

echo "http://www.baidu.com:80"|httpx   -title -content-length -status-code #use  http
http://www.baidu.com:80 [200] [300813] [百度一下，你就知道]

Results did not meet expectations

echo "http://www.baidu.com"|httpx   -title -content-length -status-code #auto jump https
https://www.baidu.com [302] [161] [302 Found]

Modified program

echo "http://www.baidu.com"|httpx   -title -content-length -status-code #use http
http://www.baidu.com:80 [200] [300813] [百度一下，你就知道]

In pr #108 seems that cname probe has been added, but when I try it locally it seems that not work,

Use the test case in the original pr:

https://www.hackerone.com
https://api.hackerone.com
https://mta-sts.hackerone.com
https://mta-sts.managed.hackerone.com
https://support.hackerone.com
https://mta-sts.forwarding.hackerone.com
https://docs.hackerone.com
https://resources.hackerone.com

./httpx -l domains.txt -cname

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.0.5

		projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
https://mta-sts.forwarding.hackerone.com
http://mta-sts.managed.hackerone.com
http://mta-sts.hackerone.com
http://api.hackerone.com
http://docs.hackerone.com
http://www.hackerone.com
http://resources.hackerone.com
http://support.hackerone.com

When I dig into source code I find that in internal/runner/runner.go:

dnsData, err := hp.Dialer.GetDNSData(domain)

and in fast dialer GetDNSData function:

		data, err = d.dnsclient.Resolve(hostname)
		if err != nil && d.options.EnableFallback {
			data, err = d.dnsclient.ResolveWithSyscall(hostname)
		}
		if err != nil {
			return nil, err
		}
		if data == nil {
			return nil, errors.New("could not resolve host")
		}
		b, _ := data.Marshal()
		err = d.hm.Set(hostname, b)
		if err != nil {
			return nil, err
		}
		return data, nil

It call retryabledns's Resolve function which only query for the A record:

func (c *Client) Resolve(host string) (*DNSData, error) {
	return c.Query(host, dns.TypeA)
}

So I think it is the root cause? I think the fastdialer need to add an function for A|CNAME type record.

while beautifying the output of json in any online website i encounter an error saying

Error: Parse error on line 22: ...": "500.794337ms"} { "ips": ["13.227.1 ----------------------^ Expecting 'EOF', '}', ',', ']', got '{'

If a server is listening on both 80 and 443 ports and the response is identical, there should be an option to avoid saving http://target.com in output file. httprobe from tomnomnom has the cool feature. You may check it out for better understanding of what I am saying.

Description

Hello Team

Thank you very much for making this tool for us it's been very helpful for us as I was poking around some features and flag I noticed that there is an issue with the mc flag it actually renders an error when you try to match a 200 or 302 status code

Test

Let's say I want to check the js file that att.com has in the archive I came with this oneliner echo "att.com" | ~/go/bin/gau | grep '\.js$' | httpx -mc 200

Expected behavior

Checking results that has 200 status code and display

Issue

What I tried

I tried to copy paste the same command you are using in the example section  httpx -status-code -mc 200,302

Ended up with the same message

Config

Ubuntu 18.04 Golang1.14 GO111MODULE=on httpx version 0.0.7

it's possible to let users specify TLS SNI like this one https://github.com/ffuf/ffuf/issues/440 and make HTTPS have the priority.

Add a new -sni flag that will be used when establishing TLS connections:

echo hackerone.com | httpx -sni www.hackerone.com

httpx version:[INF] Current Version: v1.1.4

Current Behavior:httpx -l target.txt -path "///////../../../../../../etc/passwd"

Expected Behavior:

Steps To Reproduce:

Run 'httpx -l target.txt -path "///////../../../../../../etc/passwd"' See error

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x9002d3]

goroutine 14 [running]:
github.com/projectdiscovery/httpx/runner.(*Runner).analyze(_, _, {_, _}, {_, _}, {_, _}, {0xc000eea780, 0x1b}, ...)
        /home/runner/work/httpx/httpx/runner/runner.go:874 +0xe93
github.com/projectdiscovery/httpx/runner.(*Runner).process.func1({0xc000eea780, 0x1b}, {0x9fe1ef, 0x3}, {0xa042ab, 0xa})
        /home/runner/work/httpx/httpx/runner/runner.go:651 +0x125
created by github.com/projectdiscovery/httpx/runner.(*Runner).process
        /home/runner/work/httpx/httpx/runner/runner.go:649 +0x965

Anything else:

Run: httpx -l in.txt -path dir.txt -sc -fc 200,201,300,301,401,500,501,502,503 -o out.txt

The output includes other status codes not listed in the -fc parameter (e.g. 400).

httpx version:

1.2.5

Current Behavior:

httpx 1.2.5 ignores -proxy when -unsafe is used.

Expected Behavior:

httpx should respect -proxy regardless of -unsafe usage.

Steps To Reproduce:

• setup a proxy to listen on 127.0.0.1:3128
• httpx -title -sc -td -unsafe -proxy http://127.0.0.1:3128 -u http://example.com

while scanning wide scope there are a lot of urls have the same base code so they have the same http response so it's time to let httpx exclude these urls based on comparing every http response with others

https://github.com/projectdiscovery/nuclei/issues/2284

echo https://redirect-localhost.free.beeceptor.com| httpx -sni redirect-localhost.free.beeceptor.com -fr

Please describe your feature request:

echo hackerone.com | httpx -user-agents mobile,headless,desktop-new-versions

echo hackerone.com | httpx -user-agents desktop-old-versions

Describe the use case of this feature:

Use a random user agent, but in the chosen category. This feature should be added in nuclei too

Hello, v1.2.1

echo "http://testphp.vulnweb.com/redir.php" | httpx -status-code -content-length -ip -cdn -title -tech-detect -random-agent

nothing

curl http://testphp.vulnweb.com/redir.php -v

> GET /redir.php HTTP/1.1
> Host: testphp.vulnweb.com
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved Temporarily
< Server: nginx/1.19.0
< Date: Sat, 30 Apr 2022 04:51:09 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-Powered-By: PHP/5.6.40-38+ubuntu20.04.1+deb.sury.org+1
< Location:
<
* Connection #0 to host testphp.vulnweb.com left intact