Merge pull request #4 from ksanjeev284/feature/observability-enhancements
fix: --limit flag now works correctly + video download with audio
This commit is contained in:
Sanjeev Kumar
GitHub
commit
89991bcc4a
4 changed files with 766 additions and 6 deletions
16
README.md
16
README.md
|
|
@ -38,6 +38,22 @@ python main.py --dashboard
|
||||||
# Opens at http://localhost:8501
|
# Opens at http://localhost:8501
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### 📋 Requirements
|
||||||
|
|
||||||
|
- **Python 3.8+**
|
||||||
|
- **ffmpeg** (optional, for video with audio)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Windows (via chocolatey)
|
||||||
|
choco install ffmpeg
|
||||||
|
|
||||||
|
# macOS
|
||||||
|
brew install ffmpeg
|
||||||
|
|
||||||
|
# Ubuntu/Debian
|
||||||
|
sudo apt install ffmpeg
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📖 All Commands
|
## 📖 All Commands
|
||||||
|
|
|
||||||
536
docs/BLOG.md
Normal file
536
docs/BLOG.md
Normal file
|
|
@ -0,0 +1,536 @@
|
||||||
|
# Building the Ultimate Reddit Scraper: A Full-Featured, API-Free Data Collection Suite
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**December 2024** | By Sanjeev Kumar
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
I built a **complete Reddit scraper suite** that requires **zero API keys**. It comes with a beautiful Streamlit dashboard, REST API for integration with tools like Grafana and Metabase, plugin system for post-processing, scheduled scraping, notifications, and much more. Best of all—it's completely open source.
|
||||||
|
|
||||||
|
🔗 **GitHub**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Problem
|
||||||
|
|
||||||
|
If you've ever tried to scrape Reddit data for analysis, research, or just personal projects, you know the pain:
|
||||||
|
|
||||||
|
1. **Reddit's API is heavily rate-limited** (especially after the 2023 API changes)
|
||||||
|
2. **API keys require approval** and are increasingly restricted
|
||||||
|
3. **Existing scrapers are often single-purpose** - scrape posts OR comments, not both
|
||||||
|
4. **No easy way to visualize or analyze the data** after scraping
|
||||||
|
5. **Running scrapes manually is tedious** - you want automation
|
||||||
|
|
||||||
|
I decided to solve all of these problems at once.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Solution: Universal Reddit Scraper Suite
|
||||||
|
|
||||||
|
After weeks of development, I created a full-featured scraper that:
|
||||||
|
|
||||||
|
| Feature | What It Does |
|
||||||
|
|---------|--------------|
|
||||||
|
| 📊 **Full Scraping** | Posts, comments, images, videos, galleries—everything |
|
||||||
|
| 🚫 **No API Keys** | Uses Reddit's public JSON endpoints and mirrors |
|
||||||
|
| 📈 **Web Dashboard** | Beautiful 7-tab Streamlit UI for analysis |
|
||||||
|
| 🚀 **REST API** | Connect Metabase, Grafana, DuckDB, and more |
|
||||||
|
| 🔌 **Plugin System** | Extensible post-processing (sentiment analysis, deduplication, keywords) |
|
||||||
|
| 📅 **Scheduled Scraping** | Cron-style automation |
|
||||||
|
| 📧 **Notifications** | Discord & Telegram alerts when scrapes complete |
|
||||||
|
| 🐳 **Docker Ready** | One command to deploy anywhere |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Deep Dive
|
||||||
|
|
||||||
|
### How It Works Without API Keys
|
||||||
|
|
||||||
|
The secret sauce is in the approach. Instead of using Reddit's official (and restricted) API, I leverage:
|
||||||
|
|
||||||
|
1. **Reddit's public JSON endpoints**: Every Reddit page has a `.json` suffix that returns structured data
|
||||||
|
2. **Multiple mirror fallbacks**: When one source is rate-limited, the scraper automatically rotates through alternatives like Redlib instances
|
||||||
|
3. **Smart rate limiting**: Built-in delays and cool-down periods to stay under the radar
|
||||||
|
|
||||||
|
```python
|
||||||
|
MIRRORS = [
|
||||||
|
"https://old.reddit.com",
|
||||||
|
"https://redlib.catsarch.com",
|
||||||
|
"https://redlib.vsls.cz",
|
||||||
|
"https://r.nf",
|
||||||
|
"https://libreddit.northboot.xyz",
|
||||||
|
"https://redlib.tux.pizza"
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
When one source fails, it automatically tries the next. No manual intervention needed.
|
||||||
|
|
||||||
|
### The Core Scraping Engine
|
||||||
|
|
||||||
|
The scraper operates in three modes:
|
||||||
|
|
||||||
|
**1. Full Mode** - The complete package
|
||||||
|
```bash
|
||||||
|
python main.py python --mode full --limit 100
|
||||||
|
```
|
||||||
|
This scrapes posts, downloads all media (images, videos, galleries), and fetches comments with their full thread hierarchy.
|
||||||
|
|
||||||
|
**2. History Mode** - Fast metadata-only
|
||||||
|
```bash
|
||||||
|
python main.py python --mode history --limit 500
|
||||||
|
```
|
||||||
|
Perfect for quickly building a dataset of post metadata without the overhead of media downloads.
|
||||||
|
|
||||||
|
**3. Monitor Mode** - Live watching
|
||||||
|
```bash
|
||||||
|
python main.py python --mode monitor
|
||||||
|
```
|
||||||
|
Continuously checks for new posts every 5 minutes. Ideal for tracking breaking news or trending discussions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Dashboard Experience
|
||||||
|
|
||||||
|
One of the standout features is the **7-tab Streamlit dashboard** that makes data exploration a joy:
|
||||||
|
|
||||||
|
### 📊 Overview Tab
|
||||||
|
At a glance, see:
|
||||||
|
- Total posts and comments
|
||||||
|
- Cumulative score across all posts
|
||||||
|
- Media post breakdown
|
||||||
|
- Posts-over-time chart
|
||||||
|
- Top 10 posts by score
|
||||||
|
|
||||||
|
### 📈 Analytics Tab
|
||||||
|
This is where it gets interesting:
|
||||||
|
- **Sentiment Analysis**: Run VADER-based sentiment scoring on your entire dataset
|
||||||
|
- **Keyword Cloud**: See the most frequently used terms
|
||||||
|
- **Best Posting Times**: Data-driven insights on when posts get the most engagement
|
||||||
|
|
||||||
|
### 🔍 Search Tab
|
||||||
|
Full-text search across all scraped data with filters for:
|
||||||
|
- Minimum score
|
||||||
|
- Post type (text, image, video, gallery, link)
|
||||||
|
- Author
|
||||||
|
- Custom sorting
|
||||||
|
|
||||||
|
### 💬 Comments Analysis
|
||||||
|
- View top-scoring comments
|
||||||
|
- See who the most active commenters are
|
||||||
|
- Track comment patterns over time
|
||||||
|
|
||||||
|
### ⚙️ Scraper Controls
|
||||||
|
Start new scrapes right from the dashboard! Configure:
|
||||||
|
- Target subreddit/user
|
||||||
|
- Post limits
|
||||||
|
- Mode (full/history)
|
||||||
|
- Media and comment toggles
|
||||||
|
|
||||||
|
### 📋 Job History
|
||||||
|
Full observability into every scrape job:
|
||||||
|
- Status tracking (running, completed, failed)
|
||||||
|
- Duration metrics
|
||||||
|
- Post/comment/media counts
|
||||||
|
- Error logging
|
||||||
|
|
||||||
|
### 🔌 Integrations
|
||||||
|
Pre-configured instructions for connecting:
|
||||||
|
- Metabase
|
||||||
|
- Grafana
|
||||||
|
- DreamFactory
|
||||||
|
- DuckDB
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Plugin Architecture
|
||||||
|
|
||||||
|
I designed a plugin system to allow extensible post-processing. The architecture is simple but powerful:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class Plugin:
|
||||||
|
"""Base class for all plugins."""
|
||||||
|
name = "base"
|
||||||
|
description = "Base plugin"
|
||||||
|
enabled = True
|
||||||
|
|
||||||
|
def process_posts(self, posts):
|
||||||
|
return posts
|
||||||
|
|
||||||
|
def process_comments(self, comments):
|
||||||
|
return comments
|
||||||
|
```
|
||||||
|
|
||||||
|
### Built-in Plugins
|
||||||
|
|
||||||
|
**1. Sentiment Tagger**
|
||||||
|
Analyzes the emotional tone of every post and comment using VADER sentiment analysis:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class SentimentTagger(Plugin):
|
||||||
|
name = "sentiment_tagger"
|
||||||
|
description = "Adds sentiment scores and labels to posts"
|
||||||
|
|
||||||
|
def process_posts(self, posts):
|
||||||
|
for post in posts:
|
||||||
|
text = f"{post.get('title', '')} {post.get('selftext', '')}"
|
||||||
|
score, label = analyze_sentiment(text)
|
||||||
|
post['sentiment_score'] = score
|
||||||
|
post['sentiment_label'] = label
|
||||||
|
return posts
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Deduplicator**
|
||||||
|
Removes duplicate posts that may appear across multiple scraping sessions.
|
||||||
|
|
||||||
|
**3. Keyword Extractor**
|
||||||
|
Pulls out the most significant terms from your scraped content for trend analysis.
|
||||||
|
|
||||||
|
### Creating Your Own Plugin
|
||||||
|
|
||||||
|
Drop a new Python file in the `plugins/` directory:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from plugins import Plugin
|
||||||
|
|
||||||
|
class MyCustomPlugin(Plugin):
|
||||||
|
name = "my_plugin"
|
||||||
|
description = "Does something cool"
|
||||||
|
enabled = True
|
||||||
|
|
||||||
|
def process_posts(self, posts):
|
||||||
|
# Your logic here
|
||||||
|
return posts
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable plugins during scraping:
|
||||||
|
```bash
|
||||||
|
python main.py python --mode full --plugins
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## REST API for External Integrations
|
||||||
|
|
||||||
|
The REST API opens up the scraper to a whole ecosystem of tools:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python main.py --api
|
||||||
|
# API at http://localhost:8000
|
||||||
|
# Docs at http://localhost:8000/docs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Endpoints
|
||||||
|
|
||||||
|
| Endpoint | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| `GET /posts` | List posts with filters (subreddit, limit, offset) |
|
||||||
|
| `GET /comments` | List comments |
|
||||||
|
| `GET /subreddits` | All scraped subreddits |
|
||||||
|
| `GET /jobs` | Job history |
|
||||||
|
| `GET /query?sql=...` | Raw SQL queries for power users |
|
||||||
|
| `GET /grafana/query` | Grafana-compatible time-series data |
|
||||||
|
|
||||||
|
### Real-World Integration: Grafana Dashboard
|
||||||
|
|
||||||
|
1. Install the "JSON API" or "Infinity" plugin in Grafana
|
||||||
|
2. Add datasource pointing to `http://localhost:8000`
|
||||||
|
3. Use the `/grafana/query` endpoint for time-series panels
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT date(created_utc) as time, COUNT(*) as posts
|
||||||
|
FROM posts GROUP BY date(created_utc)
|
||||||
|
```
|
||||||
|
|
||||||
|
Now you have a real-time dashboard tracking Reddit activity!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Scheduled Scraping & Notifications
|
||||||
|
|
||||||
|
### Automation Made Easy
|
||||||
|
|
||||||
|
Set up recurring scrapes with cron-style scheduling:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Scrape every 60 minutes
|
||||||
|
python main.py --schedule delhi --every 60
|
||||||
|
|
||||||
|
# With custom options
|
||||||
|
python main.py --schedule delhi --every 30 --mode full --limit 50
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get Notified
|
||||||
|
|
||||||
|
Configure Discord or Telegram alerts when scrapes complete:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Environment variables
|
||||||
|
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
|
||||||
|
export TELEGRAM_BOT_TOKEN="123456:ABC..."
|
||||||
|
export TELEGRAM_CHAT_ID="987654321"
|
||||||
|
```
|
||||||
|
|
||||||
|
Now you get notified with scrape summaries directly in your preferred platform.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dry Run Mode: Test Before You Commit
|
||||||
|
|
||||||
|
One of my favorite features is **dry run mode**. It simulates the entire scrape without saving any data:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python main.py python --mode full --limit 50 --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Output:
|
||||||
|
```
|
||||||
|
🧪 DRY RUN MODE - No data will be saved
|
||||||
|
🧪 DRY RUN COMPLETE!
|
||||||
|
📊 Would scrape: 100 posts
|
||||||
|
💬 Would scrape: 245 comments
|
||||||
|
```
|
||||||
|
|
||||||
|
Perfect for:
|
||||||
|
- Testing your scrape configuration
|
||||||
|
- Estimating data volume before committing
|
||||||
|
- Debugging without cluttering your dataset
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Docker Deployment
|
||||||
|
|
||||||
|
### Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build
|
||||||
|
docker build -t reddit-scraper .
|
||||||
|
|
||||||
|
# Run a scrape
|
||||||
|
docker run -v ./data:/app/data reddit-scraper python --limit 100
|
||||||
|
|
||||||
|
# Run with plugins
|
||||||
|
docker run -v ./data:/app/data reddit-scraper python --plugins
|
||||||
|
```
|
||||||
|
|
||||||
|
### Full Stack with Docker Compose
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
This spins up:
|
||||||
|
- Dashboard at `http://localhost:8501`
|
||||||
|
- REST API at `http://localhost:8000`
|
||||||
|
|
||||||
|
### Deploy to Any VPS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh user@your-server-ip
|
||||||
|
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
|
||||||
|
cd reddit-universal-scraper
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
Open the firewall:
|
||||||
|
```bash
|
||||||
|
sudo ufw allow 8000
|
||||||
|
sudo ufw allow 8501
|
||||||
|
```
|
||||||
|
|
||||||
|
You now have a production-ready Reddit scraping platform!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Export Options
|
||||||
|
|
||||||
|
### CSV (Default)
|
||||||
|
All scraped data is saved as CSV files:
|
||||||
|
- `data/r_<subreddit>/posts.csv`
|
||||||
|
- `data/r_<subreddit>/comments.csv`
|
||||||
|
|
||||||
|
### Parquet (Analytics-Optimized)
|
||||||
|
Export to columnar format for analytics tools:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python main.py --export-parquet python
|
||||||
|
```
|
||||||
|
|
||||||
|
Query directly with DuckDB:
|
||||||
|
```python
|
||||||
|
import duckdb
|
||||||
|
duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Database Maintenance
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup
|
||||||
|
python main.py --backup
|
||||||
|
|
||||||
|
# Optimize/vacuum
|
||||||
|
python main.py --vacuum
|
||||||
|
|
||||||
|
# View job history
|
||||||
|
python main.py --job-history
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Schema
|
||||||
|
|
||||||
|
### Posts Table
|
||||||
|
|
||||||
|
| Column | Description |
|
||||||
|
|--------|-------------|
|
||||||
|
| `id` | Reddit post ID |
|
||||||
|
| `title` | Post title |
|
||||||
|
| `author` | Username |
|
||||||
|
| `score` | Net upvotes |
|
||||||
|
| `num_comments` | Comment count |
|
||||||
|
| `post_type` | text/image/video/gallery/link |
|
||||||
|
| `selftext` | Post body (for text posts) |
|
||||||
|
| `created_utc` | Timestamp |
|
||||||
|
| `permalink` | Reddit URL |
|
||||||
|
| `is_nsfw` | NSFW flag |
|
||||||
|
| `flair` | Post flair |
|
||||||
|
| `sentiment_score` | -1.0 to 1.0 (with plugins) |
|
||||||
|
|
||||||
|
### Comments Table
|
||||||
|
|
||||||
|
| Column | Description |
|
||||||
|
|--------|-------------|
|
||||||
|
| `comment_id` | Comment ID |
|
||||||
|
| `post_permalink` | Parent post URL |
|
||||||
|
| `author` | Username |
|
||||||
|
| `body` | Comment text |
|
||||||
|
| `score` | Upvotes |
|
||||||
|
| `depth` | Nesting level |
|
||||||
|
| `is_submitter` | Whether author is OP |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
### 1. Academic Research
|
||||||
|
- Analyze subreddit community dynamics
|
||||||
|
- Track sentiment over time during events
|
||||||
|
- Study user engagement patterns
|
||||||
|
|
||||||
|
### 2. Market Research
|
||||||
|
- Monitor brand mentions
|
||||||
|
- Track product feedback
|
||||||
|
- Identify emerging trends
|
||||||
|
|
||||||
|
### 3. Content Creation
|
||||||
|
- Find popular topics in your niche
|
||||||
|
- Analyze what makes posts go viral
|
||||||
|
- Discover optimal posting times
|
||||||
|
|
||||||
|
### 4. Data Journalism
|
||||||
|
- Archive discussions around breaking news
|
||||||
|
- Analyze public sentiment during events
|
||||||
|
- Track narrative evolution
|
||||||
|
|
||||||
|
### 5. Personal Projects
|
||||||
|
- Build a dataset for ML training
|
||||||
|
- Create Reddit-based recommendation systems
|
||||||
|
- Archive communities you care about
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Respect Reddit's Servers
|
||||||
|
The scraper includes built-in delays:
|
||||||
|
- **3 second cooldown** between API requests
|
||||||
|
- **30 second wait** if all mirrors fail
|
||||||
|
- **Automatic mirror rotation** to distribute load
|
||||||
|
|
||||||
|
### Optimize Your Scrapes
|
||||||
|
- Use `--mode history` for faster metadata-only scrapes
|
||||||
|
- Use `--no-media` if you don't need images/videos
|
||||||
|
- Use `--no-comments` for post-only data
|
||||||
|
|
||||||
|
### Handle Large Datasets
|
||||||
|
- Parquet export for analytics queries
|
||||||
|
- SQLite database for structured storage
|
||||||
|
- Automatic deduplication to avoid bloat
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What's Next? Roadmap
|
||||||
|
|
||||||
|
I'm actively developing new features:
|
||||||
|
|
||||||
|
- [ ] **Async scraping** for even faster data collection
|
||||||
|
- [ ] **Multi-subreddit monitoring** in a single command
|
||||||
|
- [ ] **Email notifications** in addition to Discord/Telegram
|
||||||
|
- [ ] **Cloud deployment templates** (AWS, GCP, Azure)
|
||||||
|
- [ ] **Web-based scraper configuration** (no CLI needed)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- Python 3.10+
|
||||||
|
- pip
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Clone the repo
|
||||||
|
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
|
||||||
|
cd reddit-universal-scraper
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
# Your first scrape
|
||||||
|
python main.py python --mode full --limit 50
|
||||||
|
|
||||||
|
# Launch the dashboard
|
||||||
|
python main.py --dashboard
|
||||||
|
```
|
||||||
|
|
||||||
|
That's it! You're now scraping Reddit like a pro.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
This is an open-source project and contributions are welcome! Whether it's:
|
||||||
|
- Bug fixes
|
||||||
|
- New plugins
|
||||||
|
- Documentation improvements
|
||||||
|
- Feature suggestions
|
||||||
|
|
||||||
|
Open an issue or submit a PR on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
The Universal Reddit Scraper Suite represents months of work solving a problem that many data enthusiasts face. By combining a robust scraping engine with analytics capabilities, a beautiful dashboard, and extensive integration options—all without requiring API keys—I hope this tool empowers you to unlock insights from Reddit's vast treasure trove of community discussions.
|
||||||
|
|
||||||
|
**Happy scraping!** 🤖
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*If you found this useful, consider giving the project a ⭐ on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper)!*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Connect
|
||||||
|
|
||||||
|
- **GitHub**: [@ksanjeev284](https://github.com/ksanjeev284)
|
||||||
|
- **Project**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Tags: Reddit, Web Scraping, Python, Data Analysis, Streamlit, REST API, Docker, Open Source*
|
||||||
103
main.py
103
main.py
|
|
@ -12,6 +12,8 @@ import argparse
|
||||||
import random
|
import random
|
||||||
import sys
|
import sys
|
||||||
import json
|
import json
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
@ -166,6 +168,97 @@ def download_media(url, save_path, media_type="image"):
|
||||||
pass
|
pass
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def download_reddit_video_with_audio(video_url, save_path):
|
||||||
|
"""
|
||||||
|
Downloads Reddit video with audio by fetching both streams and merging.
|
||||||
|
Reddit stores video and audio separately - this combines them.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if os.path.exists(save_path):
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Try to find the audio URL by replacing video quality with audio
|
||||||
|
# Reddit videos have audio at URLs like .../DASH_audio.mp4 or .../DASH_AUDIO_128.mp4
|
||||||
|
base_url = video_url.rsplit('/', 1)[0]
|
||||||
|
|
||||||
|
# Common audio URL patterns
|
||||||
|
audio_urls = [
|
||||||
|
f"{base_url}/DASH_audio.mp4",
|
||||||
|
f"{base_url}/DASH_AUDIO_128.mp4",
|
||||||
|
f"{base_url}/DASH_AUDIO_64.mp4",
|
||||||
|
f"{base_url}/audio.mp4",
|
||||||
|
f"{base_url}/audio"
|
||||||
|
]
|
||||||
|
|
||||||
|
# Download video to temp file first
|
||||||
|
with tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False) as video_temp:
|
||||||
|
video_temp_path = video_temp.name
|
||||||
|
response = SESSION.get(video_url, timeout=60, stream=True)
|
||||||
|
if response.status_code != 200:
|
||||||
|
return False
|
||||||
|
for chunk in response.iter_content(chunk_size=8192):
|
||||||
|
video_temp.write(chunk)
|
||||||
|
|
||||||
|
# Try to download audio
|
||||||
|
audio_temp_path = None
|
||||||
|
for audio_url in audio_urls:
|
||||||
|
try:
|
||||||
|
response = SESSION.get(audio_url, timeout=30, stream=True)
|
||||||
|
if response.status_code == 200:
|
||||||
|
with tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False) as audio_temp:
|
||||||
|
audio_temp_path = audio_temp.name
|
||||||
|
for chunk in response.iter_content(chunk_size=8192):
|
||||||
|
audio_temp.write(chunk)
|
||||||
|
break
|
||||||
|
except:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if audio_temp_path:
|
||||||
|
# Merge video and audio using ffmpeg
|
||||||
|
try:
|
||||||
|
cmd = [
|
||||||
|
'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error',
|
||||||
|
'-i', video_temp_path,
|
||||||
|
'-i', audio_temp_path,
|
||||||
|
'-c:v', 'copy', '-c:a', 'aac',
|
||||||
|
'-shortest', save_path
|
||||||
|
]
|
||||||
|
result = subprocess.run(cmd, capture_output=True, timeout=120)
|
||||||
|
|
||||||
|
if result.returncode == 0:
|
||||||
|
# Cleanup temp files
|
||||||
|
os.unlink(video_temp_path)
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
# ffmpeg failed, fall back to video only
|
||||||
|
print(f" ⚠️ ffmpeg merge failed, saving video without audio")
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
except FileNotFoundError:
|
||||||
|
# ffmpeg not installed, save video only
|
||||||
|
print(f" ⚠️ ffmpeg not found, saving video without audio")
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
if audio_temp_path:
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
# Other error, save video only
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
# No audio found, just use video
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
# Cleanup any temp files on error
|
||||||
|
pass
|
||||||
|
return False
|
||||||
|
|
||||||
def download_post_media(post_data, dirs, post_id):
|
def download_post_media(post_data, dirs, post_id):
|
||||||
"""Downloads all media from a post."""
|
"""Downloads all media from a post."""
|
||||||
media = get_media_urls(post_data)
|
media = get_media_urls(post_data)
|
||||||
|
|
@ -187,7 +280,11 @@ def download_post_media(post_data, dirs, post_id):
|
||||||
if 'youtube' not in vid_url:
|
if 'youtube' not in vid_url:
|
||||||
ext = '.mp4'
|
ext = '.mp4'
|
||||||
save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}")
|
save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}")
|
||||||
if download_media(vid_url, save_path, "video"):
|
# Use enhanced download for Reddit videos (includes audio)
|
||||||
|
if 'v.redd.it' in vid_url or 'reddit.com' in vid_url:
|
||||||
|
if download_reddit_video_with_audio(vid_url, save_path):
|
||||||
|
downloaded["videos"] += 1
|
||||||
|
elif download_media(vid_url, save_path, "video"):
|
||||||
downloaded["videos"] += 1
|
downloaded["videos"] += 1
|
||||||
|
|
||||||
return downloaded
|
return downloaded
|
||||||
|
|
@ -354,7 +451,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True,
|
||||||
else:
|
else:
|
||||||
path = f"/r/{target}/new.json"
|
path = f"/r/{target}/new.json"
|
||||||
|
|
||||||
target_url = f"{base_url}{path}?limit=100&raw_json=1"
|
# Use proper batch size - min of remaining posts needed or 100 (Reddit's max per request)
|
||||||
|
batch_size = min(100, limit - total_posts)
|
||||||
|
target_url = f"{base_url}{path}?limit={batch_size}&raw_json=1"
|
||||||
if after:
|
if after:
|
||||||
target_url += f"&after={after}"
|
target_url += f"&after={after}"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,6 +15,8 @@ import sys
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||||
from config import USER_AGENT, MIRRORS, ASYNC_MAX_CONCURRENT, ASYNC_BATCH_SIZE
|
from config import USER_AGENT, MIRRORS, ASYNC_MAX_CONCURRENT, ASYNC_BATCH_SIZE
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
|
||||||
# Semaphore to limit concurrent requests
|
# Semaphore to limit concurrent requests
|
||||||
semaphore = None
|
semaphore = None
|
||||||
|
|
@ -33,14 +35,14 @@ async def fetch_json(session, url, retries=3):
|
||||||
await asyncio.sleep(2)
|
await asyncio.sleep(2)
|
||||||
return None
|
return None
|
||||||
|
|
||||||
async def fetch_posts_page(session, base_url, target, after=None, is_user=False):
|
async def fetch_posts_page(session, base_url, target, after=None, is_user=False, batch_size=100):
|
||||||
"""Fetch a single page of posts."""
|
"""Fetch a single page of posts."""
|
||||||
if is_user:
|
if is_user:
|
||||||
path = f"/user/{target}/submitted.json"
|
path = f"/user/{target}/submitted.json"
|
||||||
else:
|
else:
|
||||||
path = f"/r/{target}/new.json"
|
path = f"/r/{target}/new.json"
|
||||||
|
|
||||||
url = f"{base_url}{path}?limit=100&raw_json=1"
|
url = f"{base_url}{path}?limit={batch_size}&raw_json=1"
|
||||||
if after:
|
if after:
|
||||||
url += f"&after={after}"
|
url += f"&after={after}"
|
||||||
|
|
||||||
|
|
@ -65,6 +67,107 @@ async def download_media_async(session, url, save_path):
|
||||||
pass
|
pass
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
async def download_reddit_video_with_audio_async(session, video_url, save_path):
|
||||||
|
"""
|
||||||
|
Downloads Reddit video with audio asynchronously.
|
||||||
|
Reddit stores video and audio separately - this combines them using ffmpeg.
|
||||||
|
"""
|
||||||
|
global semaphore
|
||||||
|
|
||||||
|
if os.path.exists(save_path):
|
||||||
|
return True
|
||||||
|
|
||||||
|
async with semaphore:
|
||||||
|
try:
|
||||||
|
# Find audio URL by replacing video quality with audio
|
||||||
|
base_url = video_url.rsplit('/', 1)[0]
|
||||||
|
audio_urls = [
|
||||||
|
f"{base_url}/DASH_audio.mp4",
|
||||||
|
f"{base_url}/DASH_AUDIO_128.mp4",
|
||||||
|
f"{base_url}/DASH_AUDIO_64.mp4",
|
||||||
|
f"{base_url}/audio.mp4",
|
||||||
|
f"{base_url}/audio"
|
||||||
|
]
|
||||||
|
|
||||||
|
# Download video to temp file
|
||||||
|
video_temp = tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False)
|
||||||
|
video_temp_path = video_temp.name
|
||||||
|
video_temp.close()
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with session.get(video_url, timeout=aiohttp.ClientTimeout(total=60)) as response:
|
||||||
|
if response.status != 200:
|
||||||
|
return False
|
||||||
|
async with aiofiles.open(video_temp_path, 'wb') as f:
|
||||||
|
async for chunk in response.content.iter_chunked(8192):
|
||||||
|
await f.write(chunk)
|
||||||
|
except:
|
||||||
|
if os.path.exists(video_temp_path):
|
||||||
|
os.unlink(video_temp_path)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Try to download audio
|
||||||
|
audio_temp_path = None
|
||||||
|
for audio_url in audio_urls:
|
||||||
|
try:
|
||||||
|
async with session.get(audio_url, timeout=aiohttp.ClientTimeout(total=30)) as response:
|
||||||
|
if response.status == 200:
|
||||||
|
audio_temp = tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False)
|
||||||
|
audio_temp_path = audio_temp.name
|
||||||
|
audio_temp.close()
|
||||||
|
async with aiofiles.open(audio_temp_path, 'wb') as f:
|
||||||
|
async for chunk in response.content.iter_chunked(8192):
|
||||||
|
await f.write(chunk)
|
||||||
|
break
|
||||||
|
except:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if audio_temp_path:
|
||||||
|
# Merge video and audio using ffmpeg
|
||||||
|
try:
|
||||||
|
cmd = [
|
||||||
|
'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error',
|
||||||
|
'-i', video_temp_path,
|
||||||
|
'-i', audio_temp_path,
|
||||||
|
'-c:v', 'copy', '-c:a', 'aac',
|
||||||
|
'-shortest', save_path
|
||||||
|
]
|
||||||
|
proc = await asyncio.create_subprocess_exec(
|
||||||
|
*cmd,
|
||||||
|
stdout=asyncio.subprocess.PIPE,
|
||||||
|
stderr=asyncio.subprocess.PIPE
|
||||||
|
)
|
||||||
|
await asyncio.wait_for(proc.wait(), timeout=120)
|
||||||
|
|
||||||
|
if proc.returncode == 0:
|
||||||
|
os.unlink(video_temp_path)
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
# ffmpeg failed, use video only
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
except FileNotFoundError:
|
||||||
|
# ffmpeg not installed
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||||
|
os.unlink(audio_temp_path)
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
# No audio found, just use video
|
||||||
|
os.rename(video_temp_path, save_path)
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return False
|
||||||
|
|
||||||
async def fetch_comments_async(session, permalink):
|
async def fetch_comments_async(session, permalink):
|
||||||
"""Fetch comments asynchronously."""
|
"""Fetch comments asynchronously."""
|
||||||
global semaphore
|
global semaphore
|
||||||
|
|
@ -242,7 +345,9 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc
|
||||||
|
|
||||||
data = None
|
data = None
|
||||||
for mirror in mirrors:
|
for mirror in mirrors:
|
||||||
data = await fetch_posts_page(session, mirror, target, after, is_user)
|
# Use proper batch size
|
||||||
|
batch_size = min(100, limit - total_fetched)
|
||||||
|
data = await fetch_posts_page(session, mirror, target, after, is_user, batch_size)
|
||||||
if data:
|
if data:
|
||||||
print(f"✅ Fetched from {mirror}")
|
print(f"✅ Fetched from {mirror}")
|
||||||
break
|
break
|
||||||
|
|
@ -288,7 +393,11 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc
|
||||||
for i, vid_url in enumerate(media['videos'][:2]):
|
for i, vid_url in enumerate(media['videos'][:2]):
|
||||||
if 'youtube' not in vid_url:
|
if 'youtube' not in vid_url:
|
||||||
save_path = f"{videos_dir}/{post['id']}_{i}.mp4"
|
save_path = f"{videos_dir}/{post['id']}_{i}.mp4"
|
||||||
media_tasks.append(download_media_async(session, vid_url, save_path))
|
# Use enhanced download for Reddit videos (includes audio)
|
||||||
|
if 'v.redd.it' in vid_url or 'reddit.com' in vid_url:
|
||||||
|
media_tasks.append(download_reddit_video_with_audio_async(session, vid_url, save_path))
|
||||||
|
else:
|
||||||
|
media_tasks.append(download_media_async(session, vid_url, save_path))
|
||||||
|
|
||||||
# Queue comment fetching
|
# Queue comment fetching
|
||||||
if scrape_comments and post['num_comments'] > 0:
|
if scrape_comments and post['num_comments'] > 0:
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue