14 days. 14 scraping projects. Real code, real mistakes, real fixes.
Each day covers a different tool or technique — from bare HTTP requests to full browser automation. Every project includes working Python code + a Streamlit UI + a written breakdown of what broke and why.
| Day | Project | Stack | Status |
|---|---|---|---|
| 01 | LinkedIn Job Scraper | httpx + BeautifulSoup4 + Streamlit | ✅ Done |
| 02 | Book Price Tracker | Scrapy + Streamlit | ✅ Done |
| 03 | Trustpilot Review Scraper | curl_cffi + BeautifulSoup4 + Streamlit | ✅ Done |
| 🎁 | Scrapy + curl_cffi Combined | Scrapy + scrapy-impersonate + curl_cffi + Streamlit | ✅ Done |
| 04 | AI-powered crawler | crawl4ai | 🔜 Coming |
| 05 | Agent-based scraping | browser-use | 🔜 Coming |
| 06 | Auto-detect scraper | autoscraper | 🔜 Coming |
| 07 | Managed scraping API | Firecrawl | 🔜 Coming |
| 08 | Dynamic pages | Playwright | 🔜 Coming |
| 09 | Cloudflare bypass (cookies + request mirroring) | CloudflareBypassForScraping | 🔜 Coming |
| 10 | Large-scale crawling | crawlee-python | 🔜 Coming |
| 11 | Markdown from any URL | markitdown | 🔜 Coming |
| 12 | News article extraction | newspaper4k | 🔜 Coming |
| 13 | CAPTCHA solving | 2captcha | 🔜 Coming |
| 14 | Rotating proxies | proxy rotator | 🔜 Coming |
| 15 | Lightweight headless browser for AI agents | Obscura (Rust) | 🔜 Coming |
python-scraping-series/
├── day01_linkedin_jobs/
│ ├── scraper.py ← async scraper (httpx + BeautifulSoup4)
│ ├── app.py ← Streamlit UI
│ ├── locations.py ← country/city/geoId data + reliability ratings
│ ├── find_geoid.py ← CLI tool to look up verified LinkedIn geoIds
│ ├── test_subdomain.py ← experiment: do country subdomains help?
│ └── requirements.txt
├── day02_scrapy/
│ ├── booktracker/ ← Scrapy project
│ ├── app.py ← Streamlit UI
│ └── requirements.txt
├── day03_curl_cffi/
│ ├── scraper.py ← curl_cffi session + __NEXT_DATA__ parser
│ ├── app.py ← Streamlit UI
│ └── requirements.txt
└── ...