Data Collection Engineer — I turn the messy web into clean, structured data, at scale.
- Web scraping — robust, distributed crawlers built on Scrapy, Requests, httpx
- Browser automation — Playwright, Selenium, Puppeteer for the hard-to-reach pages
- Reverse engineering — JS & app reverse, captcha solving, anti-bot countermeasures
- Data pipelines — ETL from raw HTML to queryable datasets with Pandas & Spark
- Distributed systems — Kafka, Redis, and message queues keeping crawlers in sync
Languages Python
Scraping Scrapy · Requests · httpx · aiohttp
Automation Playwright · Selenium · Puppeteer
Distributed Kafka · Redis · RabbitMQ · Scrapy-Redis
Processing Pandas · Spark · Airflow
Storage MySQL · MongoDB · Elasticsearch · ClickHouse
Open-source work in progress — crawler frameworks, anti-bot toolkits, and pipeline templates coming soon.
- Blog: minner.fun
- Telegram: @MinnerMin
- Email: minner.fun@gmail.com
Data is out there — I just fetch it, politely and at scale.