A passive reconnaissance tool for known URLs discovery

Last update: Dec 20, 2022

Comments: 5

sigurlfind3r

A passive reconnaissance tool for known URLs discovery - it gathers a list of URLs passively using various online sources.

Resource

Features
Installation
Post Installation
Usage
- Examples
Contribution

Features

Collect known URLs:
- Fetches from AlienVault's OTX, Common Crawl, URLScan, Github and the Wayback Machine.
- Fetches disallowed paths from robots.txt found on your target domain and snapshotted by the Wayback Machine.
Reduce noise:
- Regex filter URLs.
- Removes duplicate pages in the sense of URL patterns that are probably repetitive and points to the same web template.
Output to stdout for piping or save to file.

Installation

From Binary

You can download the pre-built binary for your platform from this repository's releases page, extract, then move it to your $PATHand you're ready to go.

From Source

sigurlfind3r requires go1.14+ to install successfully. Run the following command to get the repo

GO111MODULE=on go get -u -v github.com/signedsecurity/sigurlfind3r/cmd/sigurlfind3r

From Github

git clone https://github.com/signedsecurity/sigurlfind3r.git && \
cd sigurlfind3r/cmd/sigurlfind3r/ && \
go build; mv sigurlfind3r /usr/local/bin/ && \
sigurlfind3r -h

Post Installation

sigurlfind3r will work after installation. However, to configure sigurlfind3r to work with certain services - currently github - you will need to have setup API keys. The API keys are stored in the $HOME/.config/sigurlfind3r/conf.yaml file - created upon first run - and uses the YAML format. Multiple API keys can be specified for each of these services.

Example:

version: 1.5.0
sources:
    - commoncrawl
    - github
    - otx
    - urlscan
    - wayback
    - waybackrobots
keys:
    github:
        - d23a554bbc1aabb208c9acfbd2dd41ce7fc9db39
        - asdsd54bbc1aabb208c9acfbd2dd41ce7fc9db39

Usage

DiSCLAIMER: fetching urls from github is a bit slow.

sigurlfind3r -h

This will display help for the tool.

     _                  _  __ _           _ _____
 ___(_) __ _ _   _ _ __| |/ _(_)_ __   __| |___ / _ __
/ __| |/ _` | | | | '__| | |_| | '_ \ / _` | |_ \| '__|
\__ \ | (_| | |_| | |  | |  _| | | | | (_| |___) | |
|___/_|\__, |\__,_|_|  |_|_| |_|_| |_|\__,_|____/|_| 1.5.0
       |___/

USAGE:
  sigurlfind3r [OPTIONS]

OPTIONS:
   -d, --domain            domain to fetch urls for
  -eS, --exclude-sources   comma(,) separated list of sources to exclude
   -f, --filter            URL filtering regex
  -iS, --include-subs      include subdomains' urls
  -lS, --list-sources      list all the available sources
  -nC, --no-color          no color mode
   -s  --silent            silent mode: output urls only
  -uS, --use-sources       comma(,) separated list of sources to use
   -o, --output            output file

Examples

Basic

sigurlfind3r -d tesla.com

Regex filter URLs

sigurlfind3r -d tesla.com -f ".(jpg|jpeg|gif|png|ico|css|eot|tif|tiff|ttf|woff|woff2)"

Include Subdomains' URLs

sigurlfind3r -d tesla.com -iS

Contribution

Issues and Pull Requests are welcome!

Owner

Signed Security

https://github.com/signedsecurity/sigurlfind3r https://github.com/signedsecurity/sigurlfind3r

Comments

Output save in File

Hello , Really great tool but I noticed that there is no option for saving output in a file that causes a lot of problem for me. It would be really really best if you kindly add this feature to this tool for saving output in a file.
[BUG]

Describe the bug panic: runtime error: index out of range [0] with length 0

Steps To Reproduce ./sigurlfind3r -d example.com -iS Output [ INF ] fetching urls for

panic: runtime error: index out of range [0] with length 0

goroutine 18 [running]: github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run.func1() /Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:82 +0x7bc created by github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl.(*Source).Run /Users/emilaltynbaev/Documents/bugbounty/tools/sigurlfind3r/pkg/sigurlfind3r/scraping/sources/commoncrawl/commoncrawl.go:63 +0xf2

Additional context MacOS 12.1
feat: Extract URLs from wayback robots.txt files

This feature adds a new source, waybackrobots. It extracts endpoints marked as disallowed from robots.txt found on your target domain and snapshotted by the Wayback Machine.
Question
I see this alot but figured i would ask you, why would you use import "github.com/signedsecurity/sigurlfind3r/pkg/sigurlfind3r/session"

when you can just

import crawler_utils "path_to_File_name_of_package"
[BUG]

Description Hi, thanks for the amazing tool. When I playing around using 'commoncrawl' as the source I found it sometimes can print out urls but sometimes can't. So I modified the code to print out the errors alongside then I found commoncrawl returned tons of 503 and timeout

Steps To Reproduce ./sigurlfind3r -d tesla.com --include-subs -uS commoncrawl

Screenshots

Additional context I tested over my home network as well as a VPS located on Los Angeles. I changed a bit of the code to avoid parallel processing and increased the timeout to 60s from 10s. Then it worked as expected. I haven't found any official documented rate limit of commoncrawl though, but reliability is the top concern in my opinion, especially in terms of automation. Just for your reference and thanks again for your work :)

A tool to find redirection chains in multiple URLs

UnChain A tool to find redirection chains in multiple URLs Introduction UnChain automates process of finding and following `30X` redirects by extracti

Dec 12, 2022

A tool to filter URLs by parameter count or size

GoFilter A tool to filter URLs by parameter count or size. This tool requires unique sorted URL list. For example: cat hosts.txt | sort -u > sorted &&

Dec 18, 2022

Otx - otx tool can scrap to find sensitive information and vulnerable endpoint urls.

otx Description This tool is base on AlienVault Open Threat Exchange (OTX)? and this tool can help you to extract all the urls endpoints which can be

Sep 24, 2022

efaceconv - Code generation tool for high performance conversion from interface{} to immutable type without allocations.

efaceconv High performance conversion from interface{} to immutable types without additional allocations This is tool for go generate and common lib (

May 14, 2022

GoWrap is a command line tool for generating decorators for Go interfaces

GoWrap GoWrap is a command line tool that generates decorators for Go interface types using simple templates. With GoWrap you can easily add metrics,

Dec 30, 2022

Perforator is a tool for recording performance metrics over subregions of a program using the Linux "perf" interface.

Perforator Perforator is a tool for recording performance metrics over subregions of a program (e.g., functions) using the Linux "perf" interface.

Dec 15, 2022

a tool for creating exploited media files for discord

Discord-Exploits A program for creating exploited media files for discord written in Go. Usage discord-exploits is a command line utility, meaning you

Dec 29, 2021

A full-featured license tool to check and fix license headers and resolve dependencies' licenses.

SkyWalking Eyes A full-featured license tool to check and fix license headers and resolve dependencies' licenses. Usage You can use License-Eye in Git

Dec 26, 2022

A tool and library for using structural regular expressions.

Structural Regular Expressions sregx is a package and tool for using structural regular expressions as described by Rob Pike (link).

Dec 7, 2022

TUI grep tool respect for IntelliJ

ilse TUI grep tool respect for IntelliJ Requirements ripgrep for fast grep bat for beautiful preview Features support HeadMatch(FirstMatch), WordMatch

Sep 27, 2022

A tool to check problems about meta files of Unity

A tool to check problems about meta files of Unity on Git repositories, and also the tool can do limited autofix for meta files of auto-generated files.

Dec 22, 2022

⚖️ A tool for transpiling C to Go.

A tool for converting C to Go. The goals of this project are: To create a generic tool that can convert C to Go. To be cross platform (linux and mac)

Dec 29, 2022

Go tool to modify struct field tags

Go tool to modify/update field tags in structs. gomodifytags makes it easy to update, add or delete the tags in a struct field. You can easily add new tags, update existing tags (such as appending a new key, i.e: db, xml, etc..) or remove existing tags

Jan 1, 2023

🤖🤝A tool to test and analyze storage and retrieval deal capability on the Filecoin network.

Dealbot A tool to test and analyze storage and retrieval deal capability on the Filecoin network. Getting Started Clone the repo and build: git clone

Sep 10, 2022

A simple tool to auto remove unused imports in rust

rust_strip A simple tool to remove unused imports in RUST. Call Cargo build/test Process the warning logs of unused imports Then replace the involved

Oct 15, 2022

A simple business indicator tool that uses a sliding window to detect whether the indicator exceeds the threshold

melon A simple business indicator tool that uses a sliding window to detect whether the indicator exceeds the threshold Usage //create the metric //th

Jul 11, 2021

A Golang tool to whitelist ASN's based on organization name

A Golang tool to whitelist ASN's based on organization name. This works by providing a list of ASN org names. This tool uses goPacket to monitor incoming traffic, capturing the IP's and checking the IP to see if it is a part of a whitelisted ASN. If it is not, it blocks that connection and future connections using iptables.

Jul 23, 2022

Chaos Engineering tool for introducing failure into syscalls

Syscall monkey Chaos Engineering tool for tampering with syscalls.

Jun 11, 2022

dontgo403 is a tool to bypass 40X errors.

dontgo403 is a tool to bypass 40X errors. Installation git clone https://github.com/devploit/dontgo403; cd dontgo403; go get; go build Options ./dontg

Dec 28, 2022