Scraper to download school attendance data from the DfE's statistics website

dfe-attendance

latest version maintainability code coverage

  • πŸ’ͺ Simple to use. Scrape attendance data with a single command!
  • πŸ‡ Super fast. Automatically scrapes, downloads ZIP archives, and extracts CSV files in seconds!
  • πŸŽ‰ Support for Windows, MacOS (Intel & Apple Silicon), Linux, and FreeBSD!
  • 🌐 Web-based API. We also offer a web-based API (can be used as a source in Power BI!) at https://dfe-attendance.vercel.app!

πŸš€ Quick Start

Install

Head over to the Releases page and download the latest version for your target OS.

Alternatively, you can install the dfe-attendance binary using go install:

go install github.com/lukecarr/dfe-attendance@latest

Scrape Daily Attendance (Example Usage)

Now you can invoke the dfe-attendance binary and start scraping data from the DfE's statistics website:

/path/to/your/dfe-attendance daily --output /path/to/output/daily.csv

This will scrape daily attendance data for schools and save the data to /path/to/output/daily.csv.

βš– License

dfe-attendance is licensed under the Apache-2.0 License.

Attribution

dfe-attendance's banner background is sourced from SVGBackgrounds.com.

Owner
Luke Carr
Digital Learning Engineer @ Ted Wragg Trust
Luke Carr
Comments
  • Bump minimist from 1.2.5 to 1.2.6 in /web

    Bump minimist from 1.2.5 to 1.2.6 in /web

    Bumps minimist from 1.2.5 to 1.2.6.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump nanoid from 3.1.30 to 3.3.2 in /web

    Bump nanoid from 3.1.30 to 3.3.2 in /web

    Bumps nanoid from 3.1.30 to 3.3.2.

    Changelog

    Sourced from nanoid's changelog.

    3.3.2

    • Fixed enhanced-resolve support.

    3.3.1

    • Reduced package size.

    3.3

    • Added size argument to function from customAlphabet (by Stefan Sundin).

    3.2

    • Added --size and --alphabet arguments to binary (by Vitaly Baev).

    3.1.32

    • Reduced async exports size (by Artyom Arutyunyan).
    • Moved from Jest to uvu (by Vitaly Baev).

    3.1.31

    • Fixed collision vulnerability on object in size (by Artyom Arutyunyan).
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump next from 12.0.9 to 12.1.0 in /web

    Bump next from 12.0.9 to 12.1.0 in /web

    Bumps next from 12.0.9 to 12.1.0.

    Release notes

    Sourced from next's releases.

    v12.1.0

    Core Changes

    • Relay Support in Rust Compiler: #33702
    • fix eslint link-passhref rule: #33857
    • update webpack: #33831
    • Flush buffered vitals metrics on page mount: #33867
    • fix problem with HMR when middleware and page reference the same node_module: #33873
    • Refactor page component getter in web server: #33759
    • update NextResponse default redirect status to 307 to match docs: #33505
    • Bug fix: dynamic page should not be interpreted as predefined page: #33808
    • Group streaming experimental apis: #33878
    • Encapsulate routing and initial hydration: #33875
    • Optimize offline condition judgment: #33238
    • Ensure external beforeFiles rewrites are handled with next/link: #33888
    • Fix parsing params for i18n optional route in minimal mode: #33896
    • Ensure browserslist extends works properly: #33890
    • Fix image cache race condition: #33883
    • Add support for Relay projects without artifactDirectory: #33918
    • fix: handle jsxspreadattribute in inline-script-id eslint rule: #32421
    • feat(next-swc): Update swc: #33724
    • Update to latest version of amphtml-validator: #33967
    • Warn in dev mode when script tags are added with next/head: #33968
    • Ensure optional chaining in swc matches babel: #33995
    • Use react-dom/server.browser in Node.js: #33950
    • Ensure external middleware rewrite is handled correctly: #33962
    • Update Terser to v5.10.0, fix minification issues: #33045
    • Warn in dev mode when stylesheets are added using next/head: #34004
    • Use ReadableStream in RenderResult: #34005
    • Fix suffix ordering while streaming: #34011
    • Don't use yarn if a package-lock.json file is found: #31926
    • Do not warn when application/ld+json scripts are used with next/head: #34021
    • Babel & next-swc: Fix exporting page config with AsExpression: #32702
    • Detect per page runtime config for functions manifest: #33945
    • Add JSDoc to config options: #32915
    • Update font-stylesheet-gathering-plugin.ts: #30709
    • Add decoratorMetadata flag if enabled by tsconfig: #32914
    • fix: data url handling in css-loader: #34034
    • Place 'charset' element at the top of : #28119
    • Fix detection of anchor click events inside svg: #23272
    • Allow passing nothing as custom jest config: #32328
    • Fixes #31240: Adding a recursive addPackagePath function in webpack-config: #31264
    • Require component rendered as child of Link to pass event to onClick handler: #27723
    • Allow scroll prevention on hash change: #31921
    • Add support for async fn / promise in next.config.js/.mjs: #33662
    • Fix lazyRoot functionality for next/image: #33933
    • Change SWC minify from beta to release candidate: #34056
    • Make Router state immutable: #33925
    • Stop exposing internal render and renderError methods from next/client: #34069
    • Add api-utils helper for testing: #34078

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump follow-redirects from 1.14.6 to 1.14.8 in /web

    Bump follow-redirects from 1.14.6 to 1.14.8 in /web

    Bumps follow-redirects from 1.14.6 to 1.14.8.

    Commits
    • 3d81dc3 Release version 1.14.8 of the npm package.
    • 62e546a Drop confidential headers across schemes.
    • 2ede36d Release version 1.14.7 of the npm package.
    • 8b347cb Drop Cookie header across domains.
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump nanoid from 3.1.30 to 3.2.0 in /web

    Bump nanoid from 3.1.30 to 3.2.0 in /web

    Bumps nanoid from 3.1.30 to 3.2.0.

    Changelog

    Sourced from nanoid's changelog.

    Change Log

    This project adheres to Semantic Versioning.

    3.2

    • Added --size and --alphabet arguments to binary (by Vitaly Baev).

    3.1.32

    • Reduced async exports size (by Artyom Arutyunyan).
    • Moved from Jest to uvu (by Vitaly Baev).

    3.1.31

    • Fixed collision vulnerability on object in size (by Artyom Arutyunyan).
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump follow-redirects from 1.14.6 to 1.14.7 in /web

    Bump follow-redirects from 1.14.6 to 1.14.7 in /web

    Bumps follow-redirects from 1.14.6 to 1.14.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump next from 12.0.7 to 12.0.9 in /web

    Bump next from 12.0.7 to 12.0.9 in /web

    Bumps next from 12.0.7 to 12.0.9.

    Release notes

    Sourced from next's releases.

    v12.0.9

    This upgrade is completely backward-compatible and recommended for all users on versions below 12.0.9

    Vulnerable code could allow a bad actor to trigger a denial of service attack via the /${locale}/_next/ route for anyone running a Next.js app at version >= 12.0.0, and using built-in i18n routing functionality.

    How to Upgrade

    • We have released patch versions for both the stable and canary channels of Next.js.
    • To upgrade run npm install next@latest --save

    Impact

    • Affected:Β All of the following must be true to be affected by this CVE
      • Next.js versions between v12.0.0 and v12.0.9
      • Using next start or a custom server
      • Using the built-in i18n support
    • Not affected:
      • Deployments on Vercel (vercel.com) are not affected along with similar environments where invalid requests are filtered before reaching Next.js.

    We recommend everyone to upgrade regardless of whether you can reproduce the issue or not.

    How to Assess Impact

    If your server has seen requests to any route under the prefix /${locale}/_next/ that have triggered a heap overflow error, this was caused by the patched issue.

    What is Being Done

    As Next.js has grown in popularity and usage by enterprises, it has received the attention of security researchers and auditors. We are thankful to our users for their investigation and responsible disclosure of the original bug.

    We've landed a patch that ensures this is handled properly so the requested route no longer crashes and triggers a heap overflow.

    Regression tests for this attack were added to the i18n integration test suite

    • A public CVE was released.
    • We encourage responsible disclosure of future reports. Please email us at [email protected]. We are actively monitoring this mailbox.

    Core Changes

    • middlewares: limit process.env to inferred usage: #33186
    • update webpack: #33207
    • Abstract out native filesystem usage from the base server: #33226
    • use text data url instead of base64 for shorter encoding: #33218
    • chore(deps): upgrade postcss: #33142
    • Fix global process testing for the process polyfill: #33220
    • Update swc: #33201
    • improve full refresh overlay: #33301
    • Custom app for server components: #33149
    • Update yarn PnP tests and disable swc file reading for PnP: #33236
    • Base Http for BaseServer: #32999
    • Update swc: #33342

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Configure Renovate

    Configure Renovate

    Mend Renovate

    Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

    🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.


    Detected Package Files

    • .github/workflows/release.yml (github-actions)
    • go.mod (gomod)
    • web/.node-version (nodenv)
    • web/package.json (npm)

    Configuration

    πŸ”‘ Renovate has detected a custom config for this PR. Feel free to ask for help if you have any doubts and would like it reviewed.

    Important: Now that this branch is edited, Renovate can't rebase it from the base branch any more. If you make changes to the base branch that could impact this onboarding PR, please merge them manually.

    What to Expect

    With your current configuration, Renovate will create 20 Pull Requests:

    Update module github.com/pterm/pterm to v0.12.49
    • Schedule: ["at any time"]
    • Branch name: renovate/github.com-pterm-pterm-0.x
    • Merge into: main
    • Upgrade github.com/pterm/pterm to v0.12.49
    Update Node.js to v16.18.1
    • Schedule: ["at any time"]
    • Branch name: renovate/node-16.x
    • Merge into: main
    • Upgrade node to 16.18.1
    Update dependency @​types/adm-zip to ^0.5.0
    • Schedule: ["at any time"]
    • Branch name: renovate/adm-zip-0.x
    • Merge into: main
    • Upgrade @types/adm-zip to ^0.5.0
    Update dependency axios to ^0.27.0
    • Schedule: ["at any time"]
    • Branch name: renovate/axios-0.x
    • Merge into: main
    • Upgrade axios to ^0.27.0
    Update dependency eslint to v8.28.0
    • Schedule: ["at any time"]
    • Branch name: renovate/eslint-8.x
    • Merge into: main
    • Upgrade eslint to 8.28.0
    Update dependency next-themes to ^0.2.0
    • Schedule: ["at any time"]
    • Branch name: renovate/next-themes-0.x
    • Merge into: main
    • Upgrade next-themes to ^0.2.0
    Update dependency use-immer to ^0.8.0
    • Schedule: ["at any time"]
    • Branch name: renovate/use-immer-0.x
    • Merge into: main
    • Upgrade use-immer to ^0.8.0
    Update module github.com/spf13/cobra to v1.6.1
    • Schedule: ["at any time"]
    • Branch name: renovate/github.com-spf13-cobra-1.x
    • Merge into: main
    • Upgrade github.com/spf13/cobra to v1.6.1
    Update module go to 1.19
    • Schedule: ["at any time"]
    • Branch name: renovate/go-1.x
    • Merge into: main
    • Upgrade go to 1.19
    Update nextjs monorepo to v12.3.3
    • Schedule: ["at any time"]
    • Branch name: renovate/nextjs-monorepo
    • Merge into: main
    • Upgrade eslint-config-next to 12.3.3
    • Upgrade next to 12.3.3
    Update pnpm to v6.35.1
    • Schedule: ["at any time"]
    • Branch name: renovate/pnpm-6.x
    • Merge into: main
    • Upgrade pnpm to 6.35.1
    Update Node.js to v18
    • Schedule: ["at any time"]
    • Branch name: renovate/node-18.x
    • Merge into: main
    • Upgrade node to 18.12.1
    Update actions/checkout action to v3
    • Schedule: ["at any time"]
    • Branch name: renovate/actions-checkout-3.x
    • Merge into: main
    • Upgrade actions/checkout to v3
    Update actions/setup-go action to v3
    • Schedule: ["at any time"]
    • Branch name: renovate/actions-setup-go-3.x
    • Merge into: main
    • Upgrade actions/setup-go to v3
    Update dependency @​types/react to v18
    • Schedule: ["at any time"]
    • Branch name: renovate/react-18.x
    • Merge into: main
    • Upgrade @types/react to ^18.0.0
    Update dependency axios to v1
    • Schedule: ["at any time"]
    • Branch name: renovate/axios-1.x
    • Merge into: main
    • Upgrade axios to ^1.0.0
    Update goreleaser/goreleaser-action action to v3
    • Schedule: ["at any time"]
    • Branch name: renovate/goreleaser-goreleaser-action-3.x
    • Merge into: main
    • Upgrade goreleaser/goreleaser-action to v3
    Update nextjs monorepo to v13 (major)
    • Schedule: ["at any time"]
    • Branch name: renovate/major-nextjs-monorepo
    • Merge into: main
    • Upgrade eslint-config-next to 13.0.4
    • Upgrade next to 13.0.4
    Update pnpm to v7
    • Schedule: ["at any time"]
    • Branch name: renovate/pnpm-7.x
    • Merge into: main
    • Upgrade pnpm to 7.16.1
    Update react monorepo to v18 (major)
    • Schedule: ["at any time"]
    • Branch name: renovate/major-react-monorepo
    • Merge into: main
    • Upgrade react to 18.2.0
    • Upgrade react-dom to 18.2.0

    🚸 Branch creation will be limited to maximum 2 per hour, so it doesn't swamp any CI resources or spam the project. See docs for prhourlylimit for details.


    ❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section. If you need any further assistance then you can also request help here.


    This PR has been generated by Mend Renovate. View repository job log here.

Ratemyprof scraper - Ratemyprof scraper with golang

ratemyprof scraper visit https://ratemyprof-api.vercel.app/api/getProf to try ou

Jan 18, 2022
Golang based web site opengraph data scraper with caching
Golang based web site opengraph data scraper with caching

Snapper A Web microservice for capturing a website's OpenGraph data built in Golang Building Snapper building the binary git clone https://github.com/

Oct 5, 2022
A simple scraper to export data from buildkite to honeycomb using opentelemetry SDK
A simple scraper to export data from buildkite to honeycomb using opentelemetry SDK

A quick scraper program that let you export builds on BuildKite as OpenTelemetry data and then send them to honeycomb.io for slice-n-dice high cardinality analysis.

Jul 7, 2022
Elegant Scraper and Crawler Framework for Golang

Colly Lightning Fast and Elegant Scraping Framework for Gophers Colly provides a clean interface to write any kind of crawler/scraper/spider. With Col

Jan 9, 2023
Web Scraper in Go, similar to BeautifulSoup

soup Web Scraper in Go, similar to BeautifulSoup soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSou

Jan 9, 2023
A crawler/scraper based on golang + colly, configurable via JSON

A crawler/scraper based on golang + colly, configurable via JSON

Aug 21, 2022
A crawler/scraper based on golang + colly, configurable via JSON

Super-Simple Scraper This a very thin layer on top of Colly which allows configuration from a JSON file. The output is JSONL which is ready to be impo

Aug 21, 2022
Simple price scraper with HTTP server/exporter for use with Prometheus

priceserver v0.3 Simple price scraper with HTTP server/exporter for use with Prometheus Currently working with Bitrue.com exchange but easily adaptabl

Nov 16, 2021
A cli scraper of gocomics.com made in go

goComic goComic is a cli tool written in go that scrapes your favorite childhood favorite comic from gocomics.com. It will give you a single days comi

Dec 24, 2021
Best Room Price Scraper from Booking.com

Best Room Price Scraper from Booking.com This repo is a tutorial of Large Scale

Nov 11, 2022
Warhammer40K faction scraper written in Golang, powered by colly.

Wascra Description Wascra is a tool written in Golang, which lets you extract all relevant Datasheet info from a Warhammer40K (9th edition) faction fr

Feb 8, 2022
Fast website link checker in Go
Fast website link checker in Go

Muffet Muffet is a website link checker which scrapes and inspects all pages in a website recursively. Features Massive speed Colored outputs Differen

Dec 26, 2022
Download Vimeo videos and retrieve metadata in Go.

vimego Download Vimeo videos and retrieve metadata. Largely based on yashrathi's vimeo_downloader. Installing go get github.com/raitonoberu/vimego Ple

Dec 30, 2022
πŸ¦™ acaoοΌˆι˜Ώθ‰οΌ‰, the tool man for data scraping of https://asoul.video/.

?? acao acaoοΌˆι˜Ώθ‰οΌ‰, the tool man for data scraping of https://asoul.video/. Deploy to Aliyun serverless function with Raika update_member Update A-SOUL

Jul 25, 2022
Go-Yahoo-Finance-Daily-Actives - Scrape for the daily actives on yh Finance and save the data to a CSV, and optionally send it to yourself as an email
Go-Yahoo-Finance-Daily-Actives - Scrape for the daily actives on yh Finance and save the data to a CSV, and optionally send it to yourself as an email

Go-Yahoo-Finance-Daily-Actives - Scrape for the daily actives on yh Finance and save the data to a CSV, and optionally send it to yourself as an email

Dec 13, 2022
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.
DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

DataHen Till is a standalone tool that instantly makes your existing web scraper scalable, maintainable, and more unblockable, with minimal code changes on your scraper.

Dec 14, 2022
Ratemyprof scraper - Ratemyprof scraper with golang

ratemyprof scraper visit https://ratemyprof-api.vercel.app/api/getProf to try ou

Jan 18, 2022
Download an entire website with Go

Go Download Web A simple command-line application to download an entire online website, including CSS, JSS, and other assets. Coded with Go. Project s

Nov 23, 2022
An API Client package for Studyplus for School SYNC API

Studyplus for School SYNC API Client This project is currently alpha, possibility having breaking changes. studyplus_for_school_sync_go is a API clien

Aug 2, 2021
Code examples for Algorithm Analysis and design (CS311) [School project]

Introduction Algorithm Analysis and design 2021/2022 Code examples implemeneted using golang Why Golang? Low Level programming language Awesome garbag

Dec 5, 2021