Skip to content
View dngr2's full-sized avatar

Block or report dngr2

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
dngr2/README.md

Python automation, scraping, and things that keep running

I build tools for the boring failure modes β€” the ones where nothing crashes, nothing alerts, and the data has been quietly wrong for three weeks.

A scraper that returns 0 results because it parsed a block page as data. A bot that died on one API timeout at 3am. A monitor that alerts on the timestamp instead of the price. Every repo below exists because of one of those.

Everything here runs. Most have a one-command demo that needs no key, no signup, and no network.


Config-driven scraping for sites that don't want to be scraped. Adding a site is a YAML file, not code. The core idea: a block page returns 200 OK, so every response is classified before extraction β€” WAF, captcha, rate-limit, empty shell β€” and a layout change fails loudly instead of silently writing blanks. Falls back to search-engine discovery when a site's own search is closed off.

python -m stealth_scrape --site olx --query "rtx 3090" --out results.csv

Real extraction output from live product pages, plus a write-up of four layout traps that produce a scraper which looks correct on your test page and returns nulls across a catalogue β€” including the price living in a visually-hidden node that Selenium's .text returns empty for.

Real-time dashboard: FastAPI, WebSocket streaming, live candlestick charts, dark/light themes. Instrument switching without reconnecting. Tabular figures so columns don't jitter twice a second β€” the detail you only notice after sitting in front of one for an hour.

pip install -r requirements.txt && uvicorn app.main:app

A bot built around what happens when the API doesn't answer. Transient and permanent failures are separate types β€” a 502 retries, a 401 never does. Backoff uses full jitter so retries don't land as one synchronised stampede. Exhausting the retry budget logs and continues; only a permanent error exits.

python -m bot.main --demo    # no token β€” watch it recover from a flaky API

πŸ‘ pagewatch

Alerts you won't mute. Every page differs on every fetch β€” timestamps, view counters, session tokens β€” so it watches a CSS-targeted value, normalises known noise, and never mistakes a WAF challenge for a change.

python -m pagewatch --demo   # baseline β†’ noise (silent) β†’ real change β†’ blocked

Python Β· Selenium Β· FastAPI Β· Linux Β· PostgreSQL Β· Docker

Available for scraping, automation, dashboards and monitoring work. Send me a URL and I'll tell you whether it's extractable before you commit to anything.

Popular repositories Loading

  1. stealth-scrape stealth-scrape Public

    Config-driven Selenium scraping that survives bot protection β€” a block page returns 200 OK, so every response is classified before extraction. New sites are a YAML file, not code.

    Python

  2. amazon-product-extraction amazon-product-extraction Public

    Sample output from live Amazon product pages β€” details, specs, pricing and reviews as structured JSON, plus the page-layout traps that silently break most scrapers.

  3. fastapi-trading-dashboard fastapi-trading-dashboard Public

    Real-time dashboard in FastAPI β€” WebSocket streaming, live candlestick charts, dark and light themes. What you move to when a Jupyter/Voila front-end runs out of road.

    Python

  4. resilient-telegram-bot resilient-telegram-bot Public

    A Telegram bot built around what happens when the API doesn't answer β€” retry with jittered backoff, structured logging, systemd deployment. Runs with --demo, no token needed.

    Python

  5. pagewatch pagewatch Public

    Watch a web page, get alerted only when the thing you care about actually changes β€” CSS-targeted extraction, noise filtering, and block-page detection so a WAF challenge is never mistaken for a cha…

    Python

  6. dngr2 dngr2 Public