scrape

A layered media extraction and downloading tool for difficult video pages.

Paste a URL, choose a format, and let scrape handle the rest.

scrape started as a personal project after getting tired of downloaders that were either painfully slow, unreliable, or simply unable to download certain websites. Instead of constantly looking for another downloader that might work, I decided to build one myself.

The idea is simple: give it a URL, try the easiest and most reliable method first, and keep going when that method fails.

The project is still evolving, but the goal is to turn it into a dependable tool that I can use myself and eventually share with anyone who wants to run, modify, or build on it.

How it works

scrape uses a layered approach rather than relying on a single extraction method.

URL
↓
Platform detection
↓
Direct HTTP
↓
Browser / Cloudflare
↓
HTML + iframe extraction
↓
Network interception
↓
yt-dlp fallback
↓
ffmpeg
↓
Download

YouTube and X are routed directly through yt-dlp. Other sites are given progressively heavier extraction methods, with a real Chrome session available for pages that require JavaScript or Cloudflare handling.

The goal is not to force every website through the same method. It's to find a method that works.

What it currently handles

The extraction layer currently looks for:

mp4, m3u8, mpd, webm, ts, and mp3 URLs in static HTML
iframe player chains, including recursive re-scanning
base64 encoded media URLs inside player pages
data-id player parameters using vid=, s=, and i= keys
token-bound CDN URLs, which are routed to browser interception
media URLs exposed through browser network requests, matched by URL pattern or, for tokenized/extensionless CDN URLs, by the response's actual Content-Type
HLS streams with separate audio and video tracks
portrait/vertical video (reels, shorts, and similar formats), with quality selection mapped to the correct dimension instead of assuming landscape
yt-dlp as a fallback for its large extractor library
DNS failures with preflight checks and fallback handling
browser TLS impersonation through yt-dlp where required

Known platforms with dedicated handling include:

YouTube
Twitter / X
Vimeo
Dailymotion
Reddit
TikTok
Twitch
Spotify (audio, via spotdl)

For sites supported by yt-dlp, scrape can fall back to its extractors through the ytdlp_probe and ytdlp_download layers. That includes sites like Instagram, which don't get a dedicated shortcut but still work through this path.

Install

git clone https://github.com/Kulungey/Scrape.git
cd Scrape
pip install -r requirements.txt
pip install -e .

Docker / Cloudflare solver

scrape can use an external browser solver for Cloudflare-protected pages. The solver runs separately from the Python application, so Docker is not required for normal downloads.

On Windows, install Docker Desktop first.

Official Docker Desktop installation instructions:

https://docs.docker.com/desktop/setup/install/windows-install/

After Docker Desktop is installed and running, start the Byparr solver:

docker run -d --name byparr -p 8191:8191 --restart unless-stopped ghcr.io/thephaseless/byparr:latest

The solver will then be available locally at:

http://localhost:8191

Verify that the container is running:

docker ps

If the container is running, scrape can use the local solver service for its Cloudflare fallback.

To stop it:

docker stop byparr

To start it again later:

docker start byparr

To remove the container completely:

docker rm -f byparr

Docker is not required for YouTube, X, Spotify, normal direct extraction, or the regular browser/yt-dlp fallback chain. It is an additional dependency for the external Cloudflare-solving path.

Cloudflare support is best-effort. A solver being available does not guarantee that every Cloudflare challenge will be bypassed, and websites can change their protection at any time.

Requires Python 3.10+, ffmpeg, and Chrome for browser based extraction.

Usage

Run without arguments for the interactive experience:

scrape

Or give it a URL directly:

scrape https://example.com/video

You can also use the original Python entry point:

python scraper.py https://example.com/video

Choose a format:

scrape https://example.com/video -f mp4
scrape https://example.com/video -f mp3

Choose an output directory:

scrape https://example.com/video --output downloads

For troubleshooting:

scrape https://example.com/video --debug

Run scrape --help for all available options.

Features

Layered media extraction
YouTube, X, Vimeo, Dailymotion, Reddit, TikTok, Twitch, and Spotify support
Direct HTTP extraction
HTML and iframe extraction
Recursive iframe player scanning
Base64 encoded URL detection
Player parameter extraction
HLS and DASH manifest detection
Separate audio and video handling
Portrait/vertical video quality selection (reels, shorts, and similar)
Cloudflare handling through a real Chrome session
Browser network interception, with an idle cutoff so it stops early once the page goes quiet instead of always waiting out the full timeout
Content-Type based media matching for tokenized CDN URLs that don't carry a file extension
yt-dlp fallback
yt-dlp TLS impersonation
DNS preflight and fallback handling
ffmpeg based conversion
MP4, MP3, and other output formats
Interactive terminal interface
Command line interface for scripting
Debug logging
Tokenized CDN URLs redacted from normal logs
Pluggable extraction chain
Automated tests for core logic

Known gaps

scrape now covers the main extraction paths and the currently tested platforms, but there are still several engineering gaps.

HLS behind a login wall does not currently pass cookies into the extractor layer
DASH manifests requiring authentication headers have the same limitation
Some sites expose the media URL only after an actual play button is clicked
The main flow does not currently simulate play clicks before waiting for network interception
Interrupted large file downloads resume from zero rather than continuing from the partial file
Non-English bot walls are not currently detected by the browser block check
Token-bound CDN detection currently covers specific token formats rather than every possible signed CDN URL
The original format flag still needs proper handling for formats that do not map directly to an ffmpeg container
MPD / DASH manifests can be detected and classified but are not yet handled as cleanly as HLS
Authentication and cookie handling still need to be expanded
Live HLS has not yet been fully validated
Master playlist handling needs more real-world testing

These are known engineering gaps, not promises of permanent site support.

Project layout

scrape/
├── main.py
├── cli.py
├── config.py
├── media.py
├── patterns.py
├── extractors.py
├── browser.py
├── ytdlp.py
├── downloader.py
├── pipeline.py
├── ui.py
└── logging_setup.py

The project is deliberately split into separate layers so the extraction logic, browser handling, downloading, command line interface, and user interface can evolve independently.

Finished 8-29-2026

The core scraper architecture, extraction pipeline, platform compatibility layer, browser fallback system, yt-dlp fallback, DNS handling, TLS impersonation, and initial automated test coverage are now in place.

Updated 8-31-2026

A round of fixes aimed at extraction speed and portrait video support:

Removed a leftover debug HTML dump that ran on every single search
The browser layer now checks the page's static HTML for the media URL before waiting on network capture, so searches where the answer is already on the page finish much faster
The network listener now cuts off early once traffic goes quiet instead of always waiting the full timeout
Added mp3 to the patterns the extractor and browser layer look for
Added a Content-Type based fallback for CDN URLs that don't have a file extension in the URL itself
Fixed quality selection for portrait/vertical video (reels, shorts, and similar). Quality tiers were only ever checked against height, which works for landscape video but not portrait, where height is the long edge. This was causing "requested format not available" errors on sites like Instagram. Quality selection now falls back to matching on width when height doesn't turn anything up, so the same 1080p/720p/480p/360p tiers apply correctly regardless of orientation

The project has also been manually tested against the main supported platforms:

YouTube
YouTube Shorts
YouTube playlists
X / Twitter
Reddit
Vimeo
Dailymotion
TikTok
Twitch
Instagram Reels (via yt-dlp fallback, portrait quality selection confirmed working)

Roadmap

This is an ambitious one man project, so development is intentionally gradual. The priority is to make each layer reliable before piling more features on top of it.

Foundation

Platform Compatibility

Extraction & Reliability

Open Source Polish

Add continuous integration
Improve documentation
Add contribution guidelines
Improve examples
Establish a release workflow
Add automated compatibility testing where practical

Standalone Builds

Build a Windows .exe
Test packaged builds on clean systems
Make runtime dependencies clear
Automate release builds

GUI

The GUI is intended to be a frontend for the existing backend rather than a separate downloader. The same extraction and download pipeline should power both interfaces.

Batch & Quality

Further Ideas

Features will be added when they solve an actual problem rather than just to make the project bigger.

Philosophy

scrape is built around a fairly simple idea:

If one method can't get the media, try another.

A website might expose a direct video URL. Another might hide it behind an iframe. Another might only reveal it after JavaScript runs. Another might require a browser session entirely.

Instead of treating those cases as completely different applications, scrape tries to give them a common pipeline.

I built this because I wanted a downloader I could actually rely on, without paying for something locked down or fighting with ad-riddled sites that barely worked half the time. It's free, it's open, and it's meant to be accessible to anyone who wants to use it, fork it, or build something different on top of it.

If it becomes useful to other people too, even better.

Real World Testing

Manual testing against real pages. Results can change as websites update their players or protection.

[x] passed · [!] tested but currently fails · [ ] not tested

Platforms

Media Sources

Browser & Fallbacks

Cloudflare protected page*-> requires docker quite a hassle but have implemented feature to open and close after each sessions so just install docker thats all
Network interception
Tokenized CDN URL detection
Direct HTTP → browser fallback
Browser → extractor fallback
Extractor → yt-dlp fallback
DNS preflight / fallback
yt-dlp TLS impersonation
Non-English bot wall detection

Downloads

These are manual compatibility tests, not guarantees of permanent site support.

What's Next

The core extraction system is now in place and the last round of fixes cleaned up speed and portrait video handling. The next stage is about turning scrape from a capable developer tool into something that can be distributed and used comfortably by other people.

The immediate priorities are:

Standalone Windows .exe
GUI frontend
Batch downloading
Format and quality probing
Better download reliability
DASH and authentication improvements
Automated releases and CI

The GUI will use the existing backend rather than replacing it. The goal is to keep the extraction pipeline independent from whatever interface sits on top of it.

For now, let me cook. One problem at a time, one improvement at a time.

Contributing

The project is currently developed as a personal project, but contributions, ideas, bug reports, and improvements are welcome.

If you find a website that scrape cannot handle, an issue describing what happened and how the page behaves is especially useful.

License

Released under the MIT License. You are free to use, modify, and distribute this project for your own purposes.

I made this to be accessible, not to gatekeep it. It's free to fork, free to build on, and free to use however you want. Any legal responsibility for what you do with it is yours, not mine.

Use it responsibly. The software is provided as is, and I am not responsible for any damage, data loss, legal issues, or other consequences resulting from its use or modification.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
scrape		scrape
tests		tests
README.md		README.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
scraper.py		scraper.py

Navigation Menu

Repository files navigation

scrape

How it works

What it currently handles

Install

Usage

Features

Known gaps

Project layout

Roadmap

Foundation

Platform Compatibility

Extraction & Reliability

Open Source Polish

Standalone Builds

GUI

Batch & Quality

Further Ideas

Philosophy

Real World Testing

Platforms

Media Sources

Browser & Fallbacks

Downloads

What's Next

Contributing

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

scrape

How it works

What it currently handles

Install

Usage

Features

Known gaps

Project layout

Roadmap

Foundation

Platform Compatibility

Extraction & Reliability

Open Source Polish

Standalone Builds

GUI

Batch & Quality

Further Ideas

Philosophy

Real World Testing

Platforms

Media Sources

Browser & Fallbacks

Downloads

What's Next

Contributing

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages