diff --git a/README.md b/README.md index 3137f9b..094d406 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,22 @@ python main.py --dashboard # Opens at http://localhost:8501 ``` +### ๐Ÿ“‹ Requirements + +- **Python 3.8+** +- **ffmpeg** (optional, for video with audio) + +```bash +# Windows (via chocolatey) +choco install ffmpeg + +# macOS +brew install ffmpeg + +# Ubuntu/Debian +sudo apt install ffmpeg +``` + --- ## ๐Ÿ“– All Commands diff --git a/docs/BLOG.md b/docs/BLOG.md new file mode 100644 index 0000000..2f659b6 --- /dev/null +++ b/docs/BLOG.md @@ -0,0 +1,536 @@ +# Building the Ultimate Reddit Scraper: A Full-Featured, API-Free Data Collection Suite + +![Reddit Scraper](https://img.shields.io/badge/Reddit-Scraper-FF4500?style=for-the-badge&logo=reddit&logoColor=white) +![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&logo=python&logoColor=white) +![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?style=for-the-badge&logo=docker&logoColor=white) + +**December 2024** | By Sanjeev Kumar + +--- + +## TL;DR + +I built a **complete Reddit scraper suite** that requires **zero API keys**. It comes with a beautiful Streamlit dashboard, REST API for integration with tools like Grafana and Metabase, plugin system for post-processing, scheduled scraping, notifications, and much more. Best of allโ€”it's completely open source. + +๐Ÿ”— **GitHub**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper) + +--- + +## The Problem + +If you've ever tried to scrape Reddit data for analysis, research, or just personal projects, you know the pain: + +1. **Reddit's API is heavily rate-limited** (especially after the 2023 API changes) +2. **API keys require approval** and are increasingly restricted +3. **Existing scrapers are often single-purpose** - scrape posts OR comments, not both +4. **No easy way to visualize or analyze the data** after scraping +5. **Running scrapes manually is tedious** - you want automation + +I decided to solve all of these problems at once. + +--- + +## The Solution: Universal Reddit Scraper Suite + +After weeks of development, I created a full-featured scraper that: + +| Feature | What It Does | +|---------|--------------| +| ๐Ÿ“Š **Full Scraping** | Posts, comments, images, videos, galleriesโ€”everything | +| ๐Ÿšซ **No API Keys** | Uses Reddit's public JSON endpoints and mirrors | +| ๐Ÿ“ˆ **Web Dashboard** | Beautiful 7-tab Streamlit UI for analysis | +| ๐Ÿš€ **REST API** | Connect Metabase, Grafana, DuckDB, and more | +| ๐Ÿ”Œ **Plugin System** | Extensible post-processing (sentiment analysis, deduplication, keywords) | +| ๐Ÿ“… **Scheduled Scraping** | Cron-style automation | +| ๐Ÿ“ง **Notifications** | Discord & Telegram alerts when scrapes complete | +| ๐Ÿณ **Docker Ready** | One command to deploy anywhere | + +--- + +## Architecture Deep Dive + +### How It Works Without API Keys + +The secret sauce is in the approach. Instead of using Reddit's official (and restricted) API, I leverage: + +1. **Reddit's public JSON endpoints**: Every Reddit page has a `.json` suffix that returns structured data +2. **Multiple mirror fallbacks**: When one source is rate-limited, the scraper automatically rotates through alternatives like Redlib instances +3. **Smart rate limiting**: Built-in delays and cool-down periods to stay under the radar + +```python +MIRRORS = [ + "https://old.reddit.com", + "https://redlib.catsarch.com", + "https://redlib.vsls.cz", + "https://r.nf", + "https://libreddit.northboot.xyz", + "https://redlib.tux.pizza" +] +``` + +When one source fails, it automatically tries the next. No manual intervention needed. + +### The Core Scraping Engine + +The scraper operates in three modes: + +**1. Full Mode** - The complete package +```bash +python main.py python --mode full --limit 100 +``` +This scrapes posts, downloads all media (images, videos, galleries), and fetches comments with their full thread hierarchy. + +**2. History Mode** - Fast metadata-only +```bash +python main.py python --mode history --limit 500 +``` +Perfect for quickly building a dataset of post metadata without the overhead of media downloads. + +**3. Monitor Mode** - Live watching +```bash +python main.py python --mode monitor +``` +Continuously checks for new posts every 5 minutes. Ideal for tracking breaking news or trending discussions. + +--- + +## The Dashboard Experience + +One of the standout features is the **7-tab Streamlit dashboard** that makes data exploration a joy: + +### ๐Ÿ“Š Overview Tab +At a glance, see: +- Total posts and comments +- Cumulative score across all posts +- Media post breakdown +- Posts-over-time chart +- Top 10 posts by score + +### ๐Ÿ“ˆ Analytics Tab +This is where it gets interesting: +- **Sentiment Analysis**: Run VADER-based sentiment scoring on your entire dataset +- **Keyword Cloud**: See the most frequently used terms +- **Best Posting Times**: Data-driven insights on when posts get the most engagement + +### ๐Ÿ” Search Tab +Full-text search across all scraped data with filters for: +- Minimum score +- Post type (text, image, video, gallery, link) +- Author +- Custom sorting + +### ๐Ÿ’ฌ Comments Analysis +- View top-scoring comments +- See who the most active commenters are +- Track comment patterns over time + +### โš™๏ธ Scraper Controls +Start new scrapes right from the dashboard! Configure: +- Target subreddit/user +- Post limits +- Mode (full/history) +- Media and comment toggles + +### ๐Ÿ“‹ Job History +Full observability into every scrape job: +- Status tracking (running, completed, failed) +- Duration metrics +- Post/comment/media counts +- Error logging + +### ๐Ÿ”Œ Integrations +Pre-configured instructions for connecting: +- Metabase +- Grafana +- DreamFactory +- DuckDB + +--- + +## The Plugin Architecture + +I designed a plugin system to allow extensible post-processing. The architecture is simple but powerful: + +```python +class Plugin: + """Base class for all plugins.""" + name = "base" + description = "Base plugin" + enabled = True + + def process_posts(self, posts): + return posts + + def process_comments(self, comments): + return comments +``` + +### Built-in Plugins + +**1. Sentiment Tagger** +Analyzes the emotional tone of every post and comment using VADER sentiment analysis: + +```python +class SentimentTagger(Plugin): + name = "sentiment_tagger" + description = "Adds sentiment scores and labels to posts" + + def process_posts(self, posts): + for post in posts: + text = f"{post.get('title', '')} {post.get('selftext', '')}" + score, label = analyze_sentiment(text) + post['sentiment_score'] = score + post['sentiment_label'] = label + return posts +``` + +**2. Deduplicator** +Removes duplicate posts that may appear across multiple scraping sessions. + +**3. Keyword Extractor** +Pulls out the most significant terms from your scraped content for trend analysis. + +### Creating Your Own Plugin + +Drop a new Python file in the `plugins/` directory: + +```python +from plugins import Plugin + +class MyCustomPlugin(Plugin): + name = "my_plugin" + description = "Does something cool" + enabled = True + + def process_posts(self, posts): + # Your logic here + return posts +``` + +Enable plugins during scraping: +```bash +python main.py python --mode full --plugins +``` + +--- + +## REST API for External Integrations + +The REST API opens up the scraper to a whole ecosystem of tools: + +```bash +python main.py --api +# API at http://localhost:8000 +# Docs at http://localhost:8000/docs +``` + +### Key Endpoints + +| Endpoint | Description | +|----------|-------------| +| `GET /posts` | List posts with filters (subreddit, limit, offset) | +| `GET /comments` | List comments | +| `GET /subreddits` | All scraped subreddits | +| `GET /jobs` | Job history | +| `GET /query?sql=...` | Raw SQL queries for power users | +| `GET /grafana/query` | Grafana-compatible time-series data | + +### Real-World Integration: Grafana Dashboard + +1. Install the "JSON API" or "Infinity" plugin in Grafana +2. Add datasource pointing to `http://localhost:8000` +3. Use the `/grafana/query` endpoint for time-series panels + +```sql +SELECT date(created_utc) as time, COUNT(*) as posts +FROM posts GROUP BY date(created_utc) +``` + +Now you have a real-time dashboard tracking Reddit activity! + +--- + +## Scheduled Scraping & Notifications + +### Automation Made Easy + +Set up recurring scrapes with cron-style scheduling: + +```bash +# Scrape every 60 minutes +python main.py --schedule delhi --every 60 + +# With custom options +python main.py --schedule delhi --every 30 --mode full --limit 50 +``` + +### Get Notified + +Configure Discord or Telegram alerts when scrapes complete: + +```bash +# Environment variables +export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." +export TELEGRAM_BOT_TOKEN="123456:ABC..." +export TELEGRAM_CHAT_ID="987654321" +``` + +Now you get notified with scrape summaries directly in your preferred platform. + +--- + +## Dry Run Mode: Test Before You Commit + +One of my favorite features is **dry run mode**. It simulates the entire scrape without saving any data: + +```bash +python main.py python --mode full --limit 50 --dry-run +``` + +Output: +``` +๐Ÿงช DRY RUN MODE - No data will be saved +๐Ÿงช DRY RUN COMPLETE! + ๐Ÿ“Š Would scrape: 100 posts + ๐Ÿ’ฌ Would scrape: 245 comments +``` + +Perfect for: +- Testing your scrape configuration +- Estimating data volume before committing +- Debugging without cluttering your dataset + +--- + +## Docker Deployment + +### Quick Start + +```bash +# Build +docker build -t reddit-scraper . + +# Run a scrape +docker run -v ./data:/app/data reddit-scraper python --limit 100 + +# Run with plugins +docker run -v ./data:/app/data reddit-scraper python --plugins +``` + +### Full Stack with Docker Compose + +```bash +docker-compose up -d +``` + +This spins up: +- Dashboard at `http://localhost:8501` +- REST API at `http://localhost:8000` + +### Deploy to Any VPS + +```bash +ssh user@your-server-ip +git clone https://github.com/ksanjeev284/reddit-universal-scraper.git +cd reddit-universal-scraper +docker-compose up -d +``` + +Open the firewall: +```bash +sudo ufw allow 8000 +sudo ufw allow 8501 +``` + +You now have a production-ready Reddit scraping platform! + +--- + +## Data Export Options + +### CSV (Default) +All scraped data is saved as CSV files: +- `data/r_/posts.csv` +- `data/r_/comments.csv` + +### Parquet (Analytics-Optimized) +Export to columnar format for analytics tools: + +```bash +python main.py --export-parquet python +``` + +Query directly with DuckDB: +```python +import duckdb +duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df() +``` + +### Database Maintenance + +```bash +# Backup +python main.py --backup + +# Optimize/vacuum +python main.py --vacuum + +# View job history +python main.py --job-history +``` + +--- + +## Data Schema + +### Posts Table + +| Column | Description | +|--------|-------------| +| `id` | Reddit post ID | +| `title` | Post title | +| `author` | Username | +| `score` | Net upvotes | +| `num_comments` | Comment count | +| `post_type` | text/image/video/gallery/link | +| `selftext` | Post body (for text posts) | +| `created_utc` | Timestamp | +| `permalink` | Reddit URL | +| `is_nsfw` | NSFW flag | +| `flair` | Post flair | +| `sentiment_score` | -1.0 to 1.0 (with plugins) | + +### Comments Table + +| Column | Description | +|--------|-------------| +| `comment_id` | Comment ID | +| `post_permalink` | Parent post URL | +| `author` | Username | +| `body` | Comment text | +| `score` | Upvotes | +| `depth` | Nesting level | +| `is_submitter` | Whether author is OP | + +--- + +## Use Cases + +### 1. Academic Research +- Analyze subreddit community dynamics +- Track sentiment over time during events +- Study user engagement patterns + +### 2. Market Research +- Monitor brand mentions +- Track product feedback +- Identify emerging trends + +### 3. Content Creation +- Find popular topics in your niche +- Analyze what makes posts go viral +- Discover optimal posting times + +### 4. Data Journalism +- Archive discussions around breaking news +- Analyze public sentiment during events +- Track narrative evolution + +### 5. Personal Projects +- Build a dataset for ML training +- Create Reddit-based recommendation systems +- Archive communities you care about + +--- + +## Performance Considerations + +### Respect Reddit's Servers +The scraper includes built-in delays: +- **3 second cooldown** between API requests +- **30 second wait** if all mirrors fail +- **Automatic mirror rotation** to distribute load + +### Optimize Your Scrapes +- Use `--mode history` for faster metadata-only scrapes +- Use `--no-media` if you don't need images/videos +- Use `--no-comments` for post-only data + +### Handle Large Datasets +- Parquet export for analytics queries +- SQLite database for structured storage +- Automatic deduplication to avoid bloat + +--- + +## What's Next? Roadmap + +I'm actively developing new features: + +- [ ] **Async scraping** for even faster data collection +- [ ] **Multi-subreddit monitoring** in a single command +- [ ] **Email notifications** in addition to Discord/Telegram +- [ ] **Cloud deployment templates** (AWS, GCP, Azure) +- [ ] **Web-based scraper configuration** (no CLI needed) + +--- + +## Getting Started + +### Prerequisites +- Python 3.10+ +- pip + +### Installation + +```bash +# Clone the repo +git clone https://github.com/ksanjeev284/reddit-universal-scraper.git +cd reddit-universal-scraper + +# Install dependencies +pip install -r requirements.txt + +# Your first scrape +python main.py python --mode full --limit 50 + +# Launch the dashboard +python main.py --dashboard +``` + +That's it! You're now scraping Reddit like a pro. + +--- + +## Contributing + +This is an open-source project and contributions are welcome! Whether it's: +- Bug fixes +- New plugins +- Documentation improvements +- Feature suggestions + +Open an issue or submit a PR on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper). + +--- + +## Conclusion + +The Universal Reddit Scraper Suite represents months of work solving a problem that many data enthusiasts face. By combining a robust scraping engine with analytics capabilities, a beautiful dashboard, and extensive integration optionsโ€”all without requiring API keysโ€”I hope this tool empowers you to unlock insights from Reddit's vast treasure trove of community discussions. + +**Happy scraping!** ๐Ÿค– + +--- + +*If you found this useful, consider giving the project a โญ on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper)!* + +--- + +## Connect + +- **GitHub**: [@ksanjeev284](https://github.com/ksanjeev284) +- **Project**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper) + +--- + +*Tags: Reddit, Web Scraping, Python, Data Analysis, Streamlit, REST API, Docker, Open Source* diff --git a/main.py b/main.py index a2eb991..3d43ee0 100644 --- a/main.py +++ b/main.py @@ -12,6 +12,8 @@ import argparse import random import sys import json +import subprocess +import tempfile from urllib.parse import urlparse from pathlib import Path @@ -166,6 +168,97 @@ def download_media(url, save_path, media_type="image"): pass return False +def download_reddit_video_with_audio(video_url, save_path): + """ + Downloads Reddit video with audio by fetching both streams and merging. + Reddit stores video and audio separately - this combines them. + """ + try: + if os.path.exists(save_path): + return True + + # Try to find the audio URL by replacing video quality with audio + # Reddit videos have audio at URLs like .../DASH_audio.mp4 or .../DASH_AUDIO_128.mp4 + base_url = video_url.rsplit('/', 1)[0] + + # Common audio URL patterns + audio_urls = [ + f"{base_url}/DASH_audio.mp4", + f"{base_url}/DASH_AUDIO_128.mp4", + f"{base_url}/DASH_AUDIO_64.mp4", + f"{base_url}/audio.mp4", + f"{base_url}/audio" + ] + + # Download video to temp file first + with tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False) as video_temp: + video_temp_path = video_temp.name + response = SESSION.get(video_url, timeout=60, stream=True) + if response.status_code != 200: + return False + for chunk in response.iter_content(chunk_size=8192): + video_temp.write(chunk) + + # Try to download audio + audio_temp_path = None + for audio_url in audio_urls: + try: + response = SESSION.get(audio_url, timeout=30, stream=True) + if response.status_code == 200: + with tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False) as audio_temp: + audio_temp_path = audio_temp.name + for chunk in response.iter_content(chunk_size=8192): + audio_temp.write(chunk) + break + except: + continue + + if audio_temp_path: + # Merge video and audio using ffmpeg + try: + cmd = [ + 'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error', + '-i', video_temp_path, + '-i', audio_temp_path, + '-c:v', 'copy', '-c:a', 'aac', + '-shortest', save_path + ] + result = subprocess.run(cmd, capture_output=True, timeout=120) + + if result.returncode == 0: + # Cleanup temp files + os.unlink(video_temp_path) + os.unlink(audio_temp_path) + return True + else: + # ffmpeg failed, fall back to video only + print(f" โš ๏ธ ffmpeg merge failed, saving video without audio") + os.rename(video_temp_path, save_path) + os.unlink(audio_temp_path) + return True + except FileNotFoundError: + # ffmpeg not installed, save video only + print(f" โš ๏ธ ffmpeg not found, saving video without audio") + os.rename(video_temp_path, save_path) + if audio_temp_path: + os.unlink(audio_temp_path) + return True + except Exception as e: + # Other error, save video only + os.rename(video_temp_path, save_path) + if audio_temp_path and os.path.exists(audio_temp_path): + os.unlink(audio_temp_path) + return True + else: + # No audio found, just use video + os.rename(video_temp_path, save_path) + return True + + except Exception as e: + # Cleanup any temp files on error + pass + return False + def download_post_media(post_data, dirs, post_id): """Downloads all media from a post.""" media = get_media_urls(post_data) @@ -187,7 +280,11 @@ def download_post_media(post_data, dirs, post_id): if 'youtube' not in vid_url: ext = '.mp4' save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}") - if download_media(vid_url, save_path, "video"): + # Use enhanced download for Reddit videos (includes audio) + if 'v.redd.it' in vid_url or 'reddit.com' in vid_url: + if download_reddit_video_with_audio(vid_url, save_path): + downloaded["videos"] += 1 + elif download_media(vid_url, save_path, "video"): downloaded["videos"] += 1 return downloaded @@ -354,7 +451,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, else: path = f"/r/{target}/new.json" - target_url = f"{base_url}{path}?limit=100&raw_json=1" + # Use proper batch size - min of remaining posts needed or 100 (Reddit's max per request) + batch_size = min(100, limit - total_posts) + target_url = f"{base_url}{path}?limit={batch_size}&raw_json=1" if after: target_url += f"&after={after}" diff --git a/scraper/async_scraper.py b/scraper/async_scraper.py index be146f7..c848419 100644 --- a/scraper/async_scraper.py +++ b/scraper/async_scraper.py @@ -15,6 +15,8 @@ import sys sys.path.insert(0, str(Path(__file__).parent.parent)) from config import USER_AGENT, MIRRORS, ASYNC_MAX_CONCURRENT, ASYNC_BATCH_SIZE +import subprocess +import tempfile # Semaphore to limit concurrent requests semaphore = None @@ -33,14 +35,14 @@ async def fetch_json(session, url, retries=3): await asyncio.sleep(2) return None -async def fetch_posts_page(session, base_url, target, after=None, is_user=False): +async def fetch_posts_page(session, base_url, target, after=None, is_user=False, batch_size=100): """Fetch a single page of posts.""" if is_user: path = f"/user/{target}/submitted.json" else: path = f"/r/{target}/new.json" - url = f"{base_url}{path}?limit=100&raw_json=1" + url = f"{base_url}{path}?limit={batch_size}&raw_json=1" if after: url += f"&after={after}" @@ -65,6 +67,107 @@ async def download_media_async(session, url, save_path): pass return False +async def download_reddit_video_with_audio_async(session, video_url, save_path): + """ + Downloads Reddit video with audio asynchronously. + Reddit stores video and audio separately - this combines them using ffmpeg. + """ + global semaphore + + if os.path.exists(save_path): + return True + + async with semaphore: + try: + # Find audio URL by replacing video quality with audio + base_url = video_url.rsplit('/', 1)[0] + audio_urls = [ + f"{base_url}/DASH_audio.mp4", + f"{base_url}/DASH_AUDIO_128.mp4", + f"{base_url}/DASH_AUDIO_64.mp4", + f"{base_url}/audio.mp4", + f"{base_url}/audio" + ] + + # Download video to temp file + video_temp = tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False) + video_temp_path = video_temp.name + video_temp.close() + + try: + async with session.get(video_url, timeout=aiohttp.ClientTimeout(total=60)) as response: + if response.status != 200: + return False + async with aiofiles.open(video_temp_path, 'wb') as f: + async for chunk in response.content.iter_chunked(8192): + await f.write(chunk) + except: + if os.path.exists(video_temp_path): + os.unlink(video_temp_path) + return False + + # Try to download audio + audio_temp_path = None + for audio_url in audio_urls: + try: + async with session.get(audio_url, timeout=aiohttp.ClientTimeout(total=30)) as response: + if response.status == 200: + audio_temp = tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False) + audio_temp_path = audio_temp.name + audio_temp.close() + async with aiofiles.open(audio_temp_path, 'wb') as f: + async for chunk in response.content.iter_chunked(8192): + await f.write(chunk) + break + except: + continue + + if audio_temp_path: + # Merge video and audio using ffmpeg + try: + cmd = [ + 'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error', + '-i', video_temp_path, + '-i', audio_temp_path, + '-c:v', 'copy', '-c:a', 'aac', + '-shortest', save_path + ] + proc = await asyncio.create_subprocess_exec( + *cmd, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE + ) + await asyncio.wait_for(proc.wait(), timeout=120) + + if proc.returncode == 0: + os.unlink(video_temp_path) + os.unlink(audio_temp_path) + return True + else: + # ffmpeg failed, use video only + os.rename(video_temp_path, save_path) + os.unlink(audio_temp_path) + return True + except FileNotFoundError: + # ffmpeg not installed + os.rename(video_temp_path, save_path) + if audio_temp_path and os.path.exists(audio_temp_path): + os.unlink(audio_temp_path) + return True + except Exception: + os.rename(video_temp_path, save_path) + if audio_temp_path and os.path.exists(audio_temp_path): + os.unlink(audio_temp_path) + return True + else: + # No audio found, just use video + os.rename(video_temp_path, save_path) + return True + + except Exception: + pass + return False + async def fetch_comments_async(session, permalink): """Fetch comments asynchronously.""" global semaphore @@ -242,7 +345,9 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc data = None for mirror in mirrors: - data = await fetch_posts_page(session, mirror, target, after, is_user) + # Use proper batch size + batch_size = min(100, limit - total_fetched) + data = await fetch_posts_page(session, mirror, target, after, is_user, batch_size) if data: print(f"โœ… Fetched from {mirror}") break @@ -288,7 +393,11 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc for i, vid_url in enumerate(media['videos'][:2]): if 'youtube' not in vid_url: save_path = f"{videos_dir}/{post['id']}_{i}.mp4" - media_tasks.append(download_media_async(session, vid_url, save_path)) + # Use enhanced download for Reddit videos (includes audio) + if 'v.redd.it' in vid_url or 'reddit.com' in vid_url: + media_tasks.append(download_reddit_video_with_audio_async(session, vid_url, save_path)) + else: + media_tasks.append(download_media_async(session, vid_url, save_path)) # Queue comment fetching if scrape_comments and post['num_comments'] > 0: