Crawls web pages and prints any link it can find.

Build Go Report Card Maintainability Test Coverage License

crawley

Crawls web pages and prints any link it can find.

Scan depth (by default - 0) can be configured.

features

  • fast SAX-parser (powered by golang.org/x/net/html)
  • small (<1000 SLOC), idiomatic, 100% test covered codebase
  • grabs most of useful resources links (pics, videos, audios, etc...)
  • found links are streamed to stdout and guranteed to be unique

usage

crawley [flags] url

possible flags:

-delay duration
    per-request delay
-depth int
    scan depth
-skip-ssl
    skip ssl verification
-user-agent string
    user-agent string
-version
    show version
-workers int
    number of workers
Owner
Alexei Shevchenko
developer
Alexei Shevchenko
Comments
  • Bump goreleaser/goreleaser-action from 2 to 3

    Bump goreleaser/goreleaser-action from 2 to 3

    Bumps goreleaser/goreleaser-action from 2 to 3.

    Release notes

    Sourced from goreleaser/goreleaser-action's releases.

    v3.0.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v2.9.1...v3.0.0

    v2.9.1

    What's Changed

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v2...v2.9.1

    v2.9.0

    What's Changed

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v2.8.1...v2.9.0

    v2.8.1

    What's Changed

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v2.8.0...v2.8.1

    v2.8.0

    What's Changed

    ... (truncated)

    Commits
    • 68acf3b chore(deps): bump @​actions/tool-cache from 1.7.2 to 2.0.1 (#355)
    • 46da113 chore: node 16 as default runtime (#343)
    • 223909a chore: update
    • c56d8df Revert "chore(deps): bump @​actions/core from 1.6.0 to 1.8.2 (#354)"
    • d1c2f83 chore(deps): bump @​actions/core from 1.6.0 to 1.8.2 (#354)
    • 5c65fd8 chore(deps): bump @​actions/http-client from 1.0.11 to 2.0.1 (#353)
    • 46cd12b chore(deps): bump yargs from 17.4.1 to 17.5.1 (#352)
    • 822d1bf chore(deps): bump docker/bake-action from 1 to 2 (#346)
    • c25888f chore: update dev dependencies and workflow (#342)
    • ec57748 chore(deps): bump yargs from 17.4.0 to 17.4.1 (#339)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump github/codeql-action from 1 to 2

    Bump github/codeql-action from 1 to 2

    Bumps github/codeql-action from 1 to 2.

    Changelog

    Sourced from github/codeql-action's changelog.

    2.1.8 - 08 Apr 2022

    • Update default CodeQL bundle version to 2.8.5. #1014
    • Fix error where the init action would fail due to a GitHub API request that was taking too long to complete #1025

    2.1.7 - 05 Apr 2022

    • A bug where additional queries specified in the workflow file would sometimes not be respected has been fixed. #1018

    2.1.6 - 30 Mar 2022

    • [v2+ only] The CodeQL Action now runs on Node.js v16. #1000
    • Update default CodeQL bundle version to 2.8.4. #990
    • Fix a bug where an invalid commit_oid was being sent to code scanning when a custom checkout path was being used. #956
    Commits
    • 2c03704 Allow the version of the ML-powered pack to depend on the CLI version
    • dd6b592 Simplify ML-powered query status report definition
    • a90d8bf Merge pull request #1011 from github/henrymercer/ml-powered-queries-pr-check
    • dc0338e Use latest major version of actions/upload-artifact
    • 57096fe Add a PR check to validate that ML-powered queries are run correctly
    • b0ddf36 Merge pull request #1012 from github/henrymercer/update-actions-major-versions
    • 1ea2f2d Merge branch 'main' into henrymercer/update-actions-major-versions
    • 9dcc141 Merge pull request #1010 from github/henrymercer/stop-running-ml-powered-quer...
    • ea751a9 Update other Actions from v2 to v3
    • a2949f4 Update actions/checkout from v2 to v3
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump actions/setup-go from 2 to 3

    Bump actions/setup-go from 2 to 3

    Bumps actions/setup-go from 2 to 3.

    Release notes

    Sourced from actions/setup-go's releases.

    v3.0.0

    What's Changed

    Breaking Changes

    With the update to Node 16, all scripts will now be run with Node 16 rather than Node 12.

    This new major release removes the stable input, so there is no need to specify additional input to use pre-release versions. This release also corrects the pre-release versions syntax to satisfy the SemVer notation (1.18.0-beta1 -> 1.18.0-beta.1, 1.18.0-rc1 -> 1.18.0-rc.1).

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v3
        with:
          go-version: '1.18.0-rc.1' 
      - run: go version
    

    Add check-latest input

    In scope of this release we add the check-latest input. If check-latest is set to true, the action first checks if the cached version is the latest one. If the locally cached version is not the most up-to-date, a Go version will then be downloaded from go-versions repository. By default check-latest is set to false. Example of usage:

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
        with:
          go-version: '1.16'
          check-latest: true
      - run: go version
    

    Moreover, we updated @actions/core from 1.2.6 to 1.6.0

    v2.1.5

    In scope of this release we updated matchers.json to improve the problem matcher pattern. For more information please refer to this pull request

    v2.1.4

    What's Changed

    New Contributors

    Full Changelog: https://github.com/actions/setup-go/compare/v2.1.3...v2.1.4

    v2.1.3

    • Updated communication with runner to use environment files rather then workflow commands

    v2.1.2

    This release includes vendored licenses for this action's npm dependencies.

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump golangci/golangci-lint-action from 2 to 3

    Bump golangci/golangci-lint-action from 2 to 3

    Bumps golangci/golangci-lint-action from 2 to 3.

    Release notes

    Sourced from golangci/golangci-lint-action's releases.

    v3.0.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/golangci/golangci-lint-action/compare/v2...v3.0.0

    Bump version v2.5.2

    Bug fixes

    • 5c56cd6 Extract and don't mangle User Args. (#200)

    Dependencies

    • e3c53fe bump @​typescript-eslint/eslint-plugin (#194)
    • 3b9f80e bump @​typescript-eslint/parser from 4.18.0 to 4.19.0 (#195)
    • 9845713 bump @​types/node from 14.14.35 to 14.14.37 (#197)
    • e789ee1 bump eslint from 7.22.0 to 7.23.0 (#196)
    • f2e9a96 bump @​typescript-eslint/eslint-plugin (#188)
    • 818081a bump @​types/node from 14.14.34 to 14.14.35 (#189)
    • 6671836 bump @​typescript-eslint/parser from 4.17.0 to 4.18.0 (#190)
    • 526907e bump @​typescript-eslint/parser from 4.16.1 to 4.17.0 (#185)
    • 6b6ba16 bump @​typescript-eslint/eslint-plugin (#186)
    • 9cab4ef bump eslint from 7.21.0 to 7.22.0 (#187)
    • 0c76572 bump @​types/node from 14.14.32 to 14.14.34 (#184)
    • 0dfde21 bump @​typescript-eslint/parser from 4.15.2 to 4.16.1 (#182)
    • 9dcf389 bump typescript from 4.2.2 to 4.2.3 (#181)
    • 34d3904 bump @​types/node from 14.14.31 to 14.14.32 (#180)
    • e30b22f bump @​typescript-eslint/eslint-plugin (#179)
    • 8f30d25 bump eslint from 7.20.0 to 7.21.0 (#177)
    • 0b64a40 bump @​typescript-eslint/parser from 4.15.1 to 4.15.2 (#176)
    • 973b3a3 bump eslint-config-prettier from 8.0.0 to 8.1.0 (#178)
    • 6ea3de1 bump @​typescript-eslint/eslint-plugin (#175)
    • 6eec6af bump typescript from 4.1.5 to 4.2.2 (#174)

    v2.5.1

    Bug fixes:

    • d9f0e73 Check that go.mod exists in reading the version (#173)

    v2.5.0

    New Features:

    • 51485a4 Try to get version from go.mod file (#118)

    ... (truncated)

    Commits
    • b517f99 fix version in package-lock.json (#407)
    • 9636c5b Update version to 3.1.0 in package.json (#406)
    • 03e4bef ci(dep): Add step to commit changes if PR has dependencies label (#108)
    • cdfc708 Allow to disable caching completely (#351)
    • 7d5614c build(deps-dev): bump eslint from 8.9.0 to 8.10.0 (#405)
    • c675eb7 Update all direct dependencies (#404)
    • 423fbaf Remove Setup-Go (#403)
    • bcfc6f9 build(deps-dev): bump eslint-plugin-import from 2.25.3 to 2.25.4 (#402)
    • d34ac2a build(deps): bump setup-go from v2.1.4 to v2.2.0 (#401)
    • e4b538e build(deps-dev): bump @​types/node from 16.11.10 to 17.0.19 (#400)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump actions/checkout from 2 to 3

    Bump actions/checkout from 2 to 3

    Bumps actions/checkout from 2 to 3.

    Release notes

    Sourced from actions/checkout's releases.

    v3.0.0

    • Update default runtime to node16

    v2.4.0

    • Convert SSH URLs like org-<ORG_ID>@github.com: to https://github.com/ - pr

    v2.3.5

    Update dependencies

    v2.3.4

    v2.3.3

    v2.3.2

    Add Third Party License Information to Dist Files

    v2.3.1

    Fix default branch resolution for .wiki and when using SSH

    v2.3.0

    Fallback to the default branch

    v2.2.0

    Fetch all history for all tags and branches when fetch-depth=0

    v2.1.1

    Changes to support GHES (here and here)

    v2.1.0

    Changelog

    Sourced from actions/checkout's changelog.

    Changelog

    v2.3.1

    v2.3.0

    v2.2.0

    v2.1.1

    • Changes to support GHES (here and here)

    v2.1.0

    v2.0.0

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump golangci/golangci-lint-action from 2 to 3.1.0

    Bump golangci/golangci-lint-action from 2 to 3.1.0

    Bumps golangci/golangci-lint-action from 2 to 3.1.0.

    Release notes

    Sourced from golangci/golangci-lint-action's releases.

    v3.1.0

    What's Changed

    New features

    CI

    Dependabot

    Misc

    New Contributors

    Full Changelog: https://github.com/golangci/golangci-lint-action/compare/v3...v3.1.0

    v3.0.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/golangci/golangci-lint-action/compare/v2...v3.0.0

    Bump version v2.5.2

    Bug fixes

    • 5c56cd6 Extract and don't mangle User Args. (#200)

    Dependencies

    • e3c53fe bump @​typescript-eslint/eslint-plugin (#194)
    • 3b9f80e bump @​typescript-eslint/parser from 4.18.0 to 4.19.0 (#195)
    • 9845713 bump @​types/node from 14.14.35 to 14.14.37 (#197)
    • e789ee1 bump eslint from 7.22.0 to 7.23.0 (#196)
    • f2e9a96 bump @​typescript-eslint/eslint-plugin (#188)
    • 818081a bump @​types/node from 14.14.34 to 14.14.35 (#189)
    • 6671836 bump @​typescript-eslint/parser from 4.17.0 to 4.18.0 (#190)

    ... (truncated)

    Commits
    • b517f99 fix version in package-lock.json (#407)
    • 9636c5b Update version to 3.1.0 in package.json (#406)
    • 03e4bef ci(dep): Add step to commit changes if PR has dependencies label (#108)
    • cdfc708 Allow to disable caching completely (#351)
    • 7d5614c build(deps-dev): bump eslint from 8.9.0 to 8.10.0 (#405)
    • c675eb7 Update all direct dependencies (#404)
    • 423fbaf Remove Setup-Go (#403)
    • bcfc6f9 build(deps-dev): bump eslint-plugin-import from 2.25.3 to 2.25.4 (#402)
    • d34ac2a build(deps): bump setup-go from v2.1.4 to v2.2.0 (#401)
    • e4b538e build(deps-dev): bump @​types/node from 16.11.10 to 17.0.19 (#400)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump goreleaser/goreleaser-action from 3 to 4

    Bump goreleaser/goreleaser-action from 3 to 4

    Bumps goreleaser/goreleaser-action from 3 to 4.

    Release notes

    Sourced from goreleaser/goreleaser-action's releases.

    v4.0.0

    What's Changed

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v3...v4.0.0

    v3.2.0

    What's Changed

    • chore: remove workaround for setOutput by @​crazy-max (#374)
    • chore(deps): bump @​actions/core from 1.9.1 to 1.10.0 (#372)
    • chore(deps): bump yargs from 17.5.1 to 17.6.0 (#373)

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v3.1.0...v3.2.0

    v3.1.0

    What's Changed

    • fix: dist resolution from config file by @​crazy-max (#369)
    • ci: fix workflow by @​crazy-max (#357)
    • docs: bump actions to latest major by @​crazy-max (#356)
    • chore(deps): bump crazy-max/ghaction-import-gpg from 4 to 5 (#360)
    • chore(deps): bump ghaction-import-gpg to v5 (#359)
    • chore(deps): bump @​actions/core from 1.6.0 to 1.8.2 (#358)
    • chore(deps): bump @​actions/core from 1.8.2 to 1.9.1 (#367)

    Full Changelog: https://github.com/goreleaser/goreleaser-action/compare/v3.0.0...v3.1.0

    Commits
    • 8f67e59 chore: regenerate
    • 78df308 chore(deps): bump minimatch from 3.0.4 to 3.1.2 (#383)
    • 66134d9 Merge remote-tracking branch 'origin/master' into flarco/master
    • 3c08cfd chore(deps): bump yargs from 17.6.0 to 17.6.2
    • 5dc579b docs: add example when using workdir along with upload-artifact (#366)
    • 3b7d1ba feat!: remove auto-snapshot on dirty tag (#382)
    • 23e0ed5 fix: do not override GORELEASER_CURRENT_TAG (#370)
    • 1315dab update build
    • b60ea88 improve install
    • 4d25ab4 Update goreleaser.ts
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump paambaati/codeclimate-action from 3.1.0 to 3.2.0

    Bump paambaati/codeclimate-action from 3.1.0 to 3.2.0

    Bumps paambaati/codeclimate-action from 3.1.0 to 3.2.0.

    Release notes

    Sourced from paambaati/codeclimate-action's releases.

    v3.2.0

    3.2.0 (2022-10-21)

    Bug Fixes

    • ci: rebuild and republish to use new branch and workflows (0c99fb3)
    • ci: use the un-broken latest version of the workflow (0fe0643)

    Features

    v3.1.1

    3.1.1 (2022-10-20)

    Bug Fixes

    • ci: rebuild and republish to use new branch and workflows (0c99fb3)
    Changelog

    Sourced from paambaati/codeclimate-action's changelog.

    3.2.0 (2022-10-21)

    Bug Fixes

    • ci: rebuild and republish to use new branch and workflows (0c99fb3)
    • ci: use the un-broken latest version of the workflow (0fe0643)

    Features

    3.1.1 (2022-10-20)

    Bug Fixes

    • ci: rebuild and republish to use new branch and workflows (0c99fb3)
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump paambaati/codeclimate-action from 3.0.0 to 3.1.0

    Bump paambaati/codeclimate-action from 3.0.0 to 3.1.0

    Bumps paambaati/codeclimate-action from 3.0.0 to 3.1.0.

    Release notes

    Sourced from paambaati/codeclimate-action's releases.

    v3.1.0

    [3.1.0] - 2022-10-14

    Changed

    Changelog

    Sourced from paambaati/codeclimate-action's changelog.

    [3.1.0] - 2022-10-14

    Changed

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files

Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values

Dec 29, 2022
WebWalker - Fast Script To Walk Web for find urls...

WebWalker send http request to url to get all urls in url and send http request to urls and again .... WebWalker can find 10,000 urls in 10 seconds.

Nov 28, 2021
Fast website link checker in Go
Fast website link checker in Go

Muffet Muffet is a website link checker which scrapes and inspects all pages in a website recursively. Features Massive speed Colored outputs Differen

Dec 26, 2022
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号:337344607 Features Concurrent

Jan 6, 2023
🔎 scan the internet to find "private" proxies.

?? scan the internet to find "private" proxies. ?? HTTP/SOCKS4/SOCKS5 Proxies. ?? Installation: - sudo apt-get install git zmap golang

Oct 31, 2022
Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL
Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL

Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL

Nov 9, 2022
Go-site-crawler - a simple application written in go that can fetch contentfrom a url endpoint

Go Site Crawler Go Site Crawler is a simple application written in go that can f

Feb 5, 2022
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Crawlab 中文 | English Installation | Run | Screenshot | Architecture | Integration | Compare | Community & Sponsorship | CHANGELOG | Disclaimer Golang-

Jan 7, 2023
Apollo 💎 A Unix-style personal search engine and web crawler for your digital footprint.
Apollo 💎 A Unix-style personal search engine and web crawler for your digital footprint.

Apollo ?? A Unix-style personal search engine and web crawler for your digital footprint Demo apollodemo.mp4 Contents Background Thesis Design Archite

Dec 27, 2022
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

Dec 14, 2022
Youtube tutorial about web scraping using golang and Gocolly

This is an example project I wrote for a youtube tutorial about webscraping using golang and gocolly It extracts data from a tracking differences webs

Mar 26, 2022
Fast golang web crawler for gathering URLs and JavaSript file locations.

Fast golang web crawler for gathering URLs and JavaSript file locations. This is basically a simple implementation of the awesome Gocolly library.

Sep 24, 2022
ant (alpha) is a web crawler for Go.

The package includes functions that can scan data from the page into your structs or slice of structs, this allows you to reduce the noise and complexity in your source-code.

Dec 30, 2022
Declarative web scraping
Declarative web scraping

Ferret Try it! Docs CLI Test runner Web worker What is it? ferret is a web scraping system. It aims to simplify data extraction from the web for UI te

Jan 4, 2023
Web Scraper in Go, similar to BeautifulSoup

soup Web Scraper in Go, similar to BeautifulSoup soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSou

Jan 9, 2023
Gospider - Fast web spider written in Go
Gospider - Fast web spider written in Go

GoSpider GoSpider - Fast web spider written in Go Painless integrate Gospider into your recon workflow? Enjoying this tool? Support it's development a

Dec 31, 2022
Fast, highly configurable, cloud native dark web crawler.

Bathyscaphe dark web crawler Bathyscaphe is a Go written, fast, highly configurable, cloud-native dark web crawler. How to start the crawler To start

Nov 22, 2022
Just a web crawler
Just a web crawler

gh-dependents gh command extension to see dependents of your repository. See The GitHub Blog: GitHub CLI 2.0 includes extensions! Install gh extension

Sep 27, 2022
Golang based web site opengraph data scraper with caching
Golang based web site opengraph data scraper with caching

Snapper A Web microservice for capturing a website's OpenGraph data built in Golang Building Snapper building the binary git clone https://github.com/

Oct 5, 2022