fix: --limit flag now works correctly + video download with audio
- Fixed --limit flag being ignored (was always requesting 100 posts) - Added ffmpeg-based audio merging for Reddit videos - Updated README with ffmpeg requirement - Applied fixes to both main.py and async_scraper.py
This commit is contained in:
Sanjeev Kumar
parent
bf84b7ca64
commit
076404906d
4 changed files with 766 additions and 6 deletions
16
README.md
16
README.md
|
|
@ -36,6 +36,22 @@ python main.py --dashboard
|
|||
# Opens at http://localhost:8501
|
||||
```
|
||||
|
||||
### 📋 Requirements
|
||||
|
||||
- **Python 3.8+**
|
||||
- **ffmpeg** (optional, for video with audio)
|
||||
|
||||
```bash
|
||||
# Windows (via chocolatey)
|
||||
choco install ffmpeg
|
||||
|
||||
# macOS
|
||||
brew install ffmpeg
|
||||
|
||||
# Ubuntu/Debian
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📖 All Commands
|
||||
|
|
|
|||
536
docs/BLOG.md
Normal file
536
docs/BLOG.md
Normal file
|
|
@ -0,0 +1,536 @@
|
|||
# Building the Ultimate Reddit Scraper: A Full-Featured, API-Free Data Collection Suite
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**December 2024** | By Sanjeev Kumar
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
I built a **complete Reddit scraper suite** that requires **zero API keys**. It comes with a beautiful Streamlit dashboard, REST API for integration with tools like Grafana and Metabase, plugin system for post-processing, scheduled scraping, notifications, and much more. Best of all—it's completely open source.
|
||||
|
||||
🔗 **GitHub**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper)
|
||||
|
||||
---
|
||||
|
||||
## The Problem
|
||||
|
||||
If you've ever tried to scrape Reddit data for analysis, research, or just personal projects, you know the pain:
|
||||
|
||||
1. **Reddit's API is heavily rate-limited** (especially after the 2023 API changes)
|
||||
2. **API keys require approval** and are increasingly restricted
|
||||
3. **Existing scrapers are often single-purpose** - scrape posts OR comments, not both
|
||||
4. **No easy way to visualize or analyze the data** after scraping
|
||||
5. **Running scrapes manually is tedious** - you want automation
|
||||
|
||||
I decided to solve all of these problems at once.
|
||||
|
||||
---
|
||||
|
||||
## The Solution: Universal Reddit Scraper Suite
|
||||
|
||||
After weeks of development, I created a full-featured scraper that:
|
||||
|
||||
| Feature | What It Does |
|
||||
|---------|--------------|
|
||||
| 📊 **Full Scraping** | Posts, comments, images, videos, galleries—everything |
|
||||
| 🚫 **No API Keys** | Uses Reddit's public JSON endpoints and mirrors |
|
||||
| 📈 **Web Dashboard** | Beautiful 7-tab Streamlit UI for analysis |
|
||||
| 🚀 **REST API** | Connect Metabase, Grafana, DuckDB, and more |
|
||||
| 🔌 **Plugin System** | Extensible post-processing (sentiment analysis, deduplication, keywords) |
|
||||
| 📅 **Scheduled Scraping** | Cron-style automation |
|
||||
| 📧 **Notifications** | Discord & Telegram alerts when scrapes complete |
|
||||
| 🐳 **Docker Ready** | One command to deploy anywhere |
|
||||
|
||||
---
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
||||
### How It Works Without API Keys
|
||||
|
||||
The secret sauce is in the approach. Instead of using Reddit's official (and restricted) API, I leverage:
|
||||
|
||||
1. **Reddit's public JSON endpoints**: Every Reddit page has a `.json` suffix that returns structured data
|
||||
2. **Multiple mirror fallbacks**: When one source is rate-limited, the scraper automatically rotates through alternatives like Redlib instances
|
||||
3. **Smart rate limiting**: Built-in delays and cool-down periods to stay under the radar
|
||||
|
||||
```python
|
||||
MIRRORS = [
|
||||
"https://old.reddit.com",
|
||||
"https://redlib.catsarch.com",
|
||||
"https://redlib.vsls.cz",
|
||||
"https://r.nf",
|
||||
"https://libreddit.northboot.xyz",
|
||||
"https://redlib.tux.pizza"
|
||||
]
|
||||
```
|
||||
|
||||
When one source fails, it automatically tries the next. No manual intervention needed.
|
||||
|
||||
### The Core Scraping Engine
|
||||
|
||||
The scraper operates in three modes:
|
||||
|
||||
**1. Full Mode** - The complete package
|
||||
```bash
|
||||
python main.py python --mode full --limit 100
|
||||
```
|
||||
This scrapes posts, downloads all media (images, videos, galleries), and fetches comments with their full thread hierarchy.
|
||||
|
||||
**2. History Mode** - Fast metadata-only
|
||||
```bash
|
||||
python main.py python --mode history --limit 500
|
||||
```
|
||||
Perfect for quickly building a dataset of post metadata without the overhead of media downloads.
|
||||
|
||||
**3. Monitor Mode** - Live watching
|
||||
```bash
|
||||
python main.py python --mode monitor
|
||||
```
|
||||
Continuously checks for new posts every 5 minutes. Ideal for tracking breaking news or trending discussions.
|
||||
|
||||
---
|
||||
|
||||
## The Dashboard Experience
|
||||
|
||||
One of the standout features is the **7-tab Streamlit dashboard** that makes data exploration a joy:
|
||||
|
||||
### 📊 Overview Tab
|
||||
At a glance, see:
|
||||
- Total posts and comments
|
||||
- Cumulative score across all posts
|
||||
- Media post breakdown
|
||||
- Posts-over-time chart
|
||||
- Top 10 posts by score
|
||||
|
||||
### 📈 Analytics Tab
|
||||
This is where it gets interesting:
|
||||
- **Sentiment Analysis**: Run VADER-based sentiment scoring on your entire dataset
|
||||
- **Keyword Cloud**: See the most frequently used terms
|
||||
- **Best Posting Times**: Data-driven insights on when posts get the most engagement
|
||||
|
||||
### 🔍 Search Tab
|
||||
Full-text search across all scraped data with filters for:
|
||||
- Minimum score
|
||||
- Post type (text, image, video, gallery, link)
|
||||
- Author
|
||||
- Custom sorting
|
||||
|
||||
### 💬 Comments Analysis
|
||||
- View top-scoring comments
|
||||
- See who the most active commenters are
|
||||
- Track comment patterns over time
|
||||
|
||||
### ⚙️ Scraper Controls
|
||||
Start new scrapes right from the dashboard! Configure:
|
||||
- Target subreddit/user
|
||||
- Post limits
|
||||
- Mode (full/history)
|
||||
- Media and comment toggles
|
||||
|
||||
### 📋 Job History
|
||||
Full observability into every scrape job:
|
||||
- Status tracking (running, completed, failed)
|
||||
- Duration metrics
|
||||
- Post/comment/media counts
|
||||
- Error logging
|
||||
|
||||
### 🔌 Integrations
|
||||
Pre-configured instructions for connecting:
|
||||
- Metabase
|
||||
- Grafana
|
||||
- DreamFactory
|
||||
- DuckDB
|
||||
|
||||
---
|
||||
|
||||
## The Plugin Architecture
|
||||
|
||||
I designed a plugin system to allow extensible post-processing. The architecture is simple but powerful:
|
||||
|
||||
```python
|
||||
class Plugin:
|
||||
"""Base class for all plugins."""
|
||||
name = "base"
|
||||
description = "Base plugin"
|
||||
enabled = True
|
||||
|
||||
def process_posts(self, posts):
|
||||
return posts
|
||||
|
||||
def process_comments(self, comments):
|
||||
return comments
|
||||
```
|
||||
|
||||
### Built-in Plugins
|
||||
|
||||
**1. Sentiment Tagger**
|
||||
Analyzes the emotional tone of every post and comment using VADER sentiment analysis:
|
||||
|
||||
```python
|
||||
class SentimentTagger(Plugin):
|
||||
name = "sentiment_tagger"
|
||||
description = "Adds sentiment scores and labels to posts"
|
||||
|
||||
def process_posts(self, posts):
|
||||
for post in posts:
|
||||
text = f"{post.get('title', '')} {post.get('selftext', '')}"
|
||||
score, label = analyze_sentiment(text)
|
||||
post['sentiment_score'] = score
|
||||
post['sentiment_label'] = label
|
||||
return posts
|
||||
```
|
||||
|
||||
**2. Deduplicator**
|
||||
Removes duplicate posts that may appear across multiple scraping sessions.
|
||||
|
||||
**3. Keyword Extractor**
|
||||
Pulls out the most significant terms from your scraped content for trend analysis.
|
||||
|
||||
### Creating Your Own Plugin
|
||||
|
||||
Drop a new Python file in the `plugins/` directory:
|
||||
|
||||
```python
|
||||
from plugins import Plugin
|
||||
|
||||
class MyCustomPlugin(Plugin):
|
||||
name = "my_plugin"
|
||||
description = "Does something cool"
|
||||
enabled = True
|
||||
|
||||
def process_posts(self, posts):
|
||||
# Your logic here
|
||||
return posts
|
||||
```
|
||||
|
||||
Enable plugins during scraping:
|
||||
```bash
|
||||
python main.py python --mode full --plugins
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REST API for External Integrations
|
||||
|
||||
The REST API opens up the scraper to a whole ecosystem of tools:
|
||||
|
||||
```bash
|
||||
python main.py --api
|
||||
# API at http://localhost:8000
|
||||
# Docs at http://localhost:8000/docs
|
||||
```
|
||||
|
||||
### Key Endpoints
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /posts` | List posts with filters (subreddit, limit, offset) |
|
||||
| `GET /comments` | List comments |
|
||||
| `GET /subreddits` | All scraped subreddits |
|
||||
| `GET /jobs` | Job history |
|
||||
| `GET /query?sql=...` | Raw SQL queries for power users |
|
||||
| `GET /grafana/query` | Grafana-compatible time-series data |
|
||||
|
||||
### Real-World Integration: Grafana Dashboard
|
||||
|
||||
1. Install the "JSON API" or "Infinity" plugin in Grafana
|
||||
2. Add datasource pointing to `http://localhost:8000`
|
||||
3. Use the `/grafana/query` endpoint for time-series panels
|
||||
|
||||
```sql
|
||||
SELECT date(created_utc) as time, COUNT(*) as posts
|
||||
FROM posts GROUP BY date(created_utc)
|
||||
```
|
||||
|
||||
Now you have a real-time dashboard tracking Reddit activity!
|
||||
|
||||
---
|
||||
|
||||
## Scheduled Scraping & Notifications
|
||||
|
||||
### Automation Made Easy
|
||||
|
||||
Set up recurring scrapes with cron-style scheduling:
|
||||
|
||||
```bash
|
||||
# Scrape every 60 minutes
|
||||
python main.py --schedule delhi --every 60
|
||||
|
||||
# With custom options
|
||||
python main.py --schedule delhi --every 30 --mode full --limit 50
|
||||
```
|
||||
|
||||
### Get Notified
|
||||
|
||||
Configure Discord or Telegram alerts when scrapes complete:
|
||||
|
||||
```bash
|
||||
# Environment variables
|
||||
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
|
||||
export TELEGRAM_BOT_TOKEN="123456:ABC..."
|
||||
export TELEGRAM_CHAT_ID="987654321"
|
||||
```
|
||||
|
||||
Now you get notified with scrape summaries directly in your preferred platform.
|
||||
|
||||
---
|
||||
|
||||
## Dry Run Mode: Test Before You Commit
|
||||
|
||||
One of my favorite features is **dry run mode**. It simulates the entire scrape without saving any data:
|
||||
|
||||
```bash
|
||||
python main.py python --mode full --limit 50 --dry-run
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
🧪 DRY RUN MODE - No data will be saved
|
||||
🧪 DRY RUN COMPLETE!
|
||||
📊 Would scrape: 100 posts
|
||||
💬 Would scrape: 245 comments
|
||||
```
|
||||
|
||||
Perfect for:
|
||||
- Testing your scrape configuration
|
||||
- Estimating data volume before committing
|
||||
- Debugging without cluttering your dataset
|
||||
|
||||
---
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Build
|
||||
docker build -t reddit-scraper .
|
||||
|
||||
# Run a scrape
|
||||
docker run -v ./data:/app/data reddit-scraper python --limit 100
|
||||
|
||||
# Run with plugins
|
||||
docker run -v ./data:/app/data reddit-scraper python --plugins
|
||||
```
|
||||
|
||||
### Full Stack with Docker Compose
|
||||
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
This spins up:
|
||||
- Dashboard at `http://localhost:8501`
|
||||
- REST API at `http://localhost:8000`
|
||||
|
||||
### Deploy to Any VPS
|
||||
|
||||
```bash
|
||||
ssh user@your-server-ip
|
||||
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
|
||||
cd reddit-universal-scraper
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
Open the firewall:
|
||||
```bash
|
||||
sudo ufw allow 8000
|
||||
sudo ufw allow 8501
|
||||
```
|
||||
|
||||
You now have a production-ready Reddit scraping platform!
|
||||
|
||||
---
|
||||
|
||||
## Data Export Options
|
||||
|
||||
### CSV (Default)
|
||||
All scraped data is saved as CSV files:
|
||||
- `data/r_<subreddit>/posts.csv`
|
||||
- `data/r_<subreddit>/comments.csv`
|
||||
|
||||
### Parquet (Analytics-Optimized)
|
||||
Export to columnar format for analytics tools:
|
||||
|
||||
```bash
|
||||
python main.py --export-parquet python
|
||||
```
|
||||
|
||||
Query directly with DuckDB:
|
||||
```python
|
||||
import duckdb
|
||||
duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df()
|
||||
```
|
||||
|
||||
### Database Maintenance
|
||||
|
||||
```bash
|
||||
# Backup
|
||||
python main.py --backup
|
||||
|
||||
# Optimize/vacuum
|
||||
python main.py --vacuum
|
||||
|
||||
# View job history
|
||||
python main.py --job-history
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Schema
|
||||
|
||||
### Posts Table
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| `id` | Reddit post ID |
|
||||
| `title` | Post title |
|
||||
| `author` | Username |
|
||||
| `score` | Net upvotes |
|
||||
| `num_comments` | Comment count |
|
||||
| `post_type` | text/image/video/gallery/link |
|
||||
| `selftext` | Post body (for text posts) |
|
||||
| `created_utc` | Timestamp |
|
||||
| `permalink` | Reddit URL |
|
||||
| `is_nsfw` | NSFW flag |
|
||||
| `flair` | Post flair |
|
||||
| `sentiment_score` | -1.0 to 1.0 (with plugins) |
|
||||
|
||||
### Comments Table
|
||||
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| `comment_id` | Comment ID |
|
||||
| `post_permalink` | Parent post URL |
|
||||
| `author` | Username |
|
||||
| `body` | Comment text |
|
||||
| `score` | Upvotes |
|
||||
| `depth` | Nesting level |
|
||||
| `is_submitter` | Whether author is OP |
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 1. Academic Research
|
||||
- Analyze subreddit community dynamics
|
||||
- Track sentiment over time during events
|
||||
- Study user engagement patterns
|
||||
|
||||
### 2. Market Research
|
||||
- Monitor brand mentions
|
||||
- Track product feedback
|
||||
- Identify emerging trends
|
||||
|
||||
### 3. Content Creation
|
||||
- Find popular topics in your niche
|
||||
- Analyze what makes posts go viral
|
||||
- Discover optimal posting times
|
||||
|
||||
### 4. Data Journalism
|
||||
- Archive discussions around breaking news
|
||||
- Analyze public sentiment during events
|
||||
- Track narrative evolution
|
||||
|
||||
### 5. Personal Projects
|
||||
- Build a dataset for ML training
|
||||
- Create Reddit-based recommendation systems
|
||||
- Archive communities you care about
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Respect Reddit's Servers
|
||||
The scraper includes built-in delays:
|
||||
- **3 second cooldown** between API requests
|
||||
- **30 second wait** if all mirrors fail
|
||||
- **Automatic mirror rotation** to distribute load
|
||||
|
||||
### Optimize Your Scrapes
|
||||
- Use `--mode history` for faster metadata-only scrapes
|
||||
- Use `--no-media` if you don't need images/videos
|
||||
- Use `--no-comments` for post-only data
|
||||
|
||||
### Handle Large Datasets
|
||||
- Parquet export for analytics queries
|
||||
- SQLite database for structured storage
|
||||
- Automatic deduplication to avoid bloat
|
||||
|
||||
---
|
||||
|
||||
## What's Next? Roadmap
|
||||
|
||||
I'm actively developing new features:
|
||||
|
||||
- [ ] **Async scraping** for even faster data collection
|
||||
- [ ] **Multi-subreddit monitoring** in a single command
|
||||
- [ ] **Email notifications** in addition to Discord/Telegram
|
||||
- [ ] **Cloud deployment templates** (AWS, GCP, Azure)
|
||||
- [ ] **Web-based scraper configuration** (no CLI needed)
|
||||
|
||||
---
|
||||
|
||||
## Getting Started
|
||||
|
||||
### Prerequisites
|
||||
- Python 3.10+
|
||||
- pip
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
# Clone the repo
|
||||
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
|
||||
cd reddit-universal-scraper
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Your first scrape
|
||||
python main.py python --mode full --limit 50
|
||||
|
||||
# Launch the dashboard
|
||||
python main.py --dashboard
|
||||
```
|
||||
|
||||
That's it! You're now scraping Reddit like a pro.
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
This is an open-source project and contributions are welcome! Whether it's:
|
||||
- Bug fixes
|
||||
- New plugins
|
||||
- Documentation improvements
|
||||
- Feature suggestions
|
||||
|
||||
Open an issue or submit a PR on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper).
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
The Universal Reddit Scraper Suite represents months of work solving a problem that many data enthusiasts face. By combining a robust scraping engine with analytics capabilities, a beautiful dashboard, and extensive integration options—all without requiring API keys—I hope this tool empowers you to unlock insights from Reddit's vast treasure trove of community discussions.
|
||||
|
||||
**Happy scraping!** 🤖
|
||||
|
||||
---
|
||||
|
||||
*If you found this useful, consider giving the project a ⭐ on [GitHub](https://github.com/ksanjeev284/reddit-universal-scraper)!*
|
||||
|
||||
---
|
||||
|
||||
## Connect
|
||||
|
||||
- **GitHub**: [@ksanjeev284](https://github.com/ksanjeev284)
|
||||
- **Project**: [reddit-universal-scraper](https://github.com/ksanjeev284/reddit-universal-scraper)
|
||||
|
||||
---
|
||||
|
||||
*Tags: Reddit, Web Scraping, Python, Data Analysis, Streamlit, REST API, Docker, Open Source*
|
||||
103
main.py
103
main.py
|
|
@ -12,6 +12,8 @@ import argparse
|
|||
import random
|
||||
import sys
|
||||
import json
|
||||
import subprocess
|
||||
import tempfile
|
||||
from urllib.parse import urlparse
|
||||
from pathlib import Path
|
||||
|
||||
|
|
@ -166,6 +168,97 @@ def download_media(url, save_path, media_type="image"):
|
|||
pass
|
||||
return False
|
||||
|
||||
def download_reddit_video_with_audio(video_url, save_path):
|
||||
"""
|
||||
Downloads Reddit video with audio by fetching both streams and merging.
|
||||
Reddit stores video and audio separately - this combines them.
|
||||
"""
|
||||
try:
|
||||
if os.path.exists(save_path):
|
||||
return True
|
||||
|
||||
# Try to find the audio URL by replacing video quality with audio
|
||||
# Reddit videos have audio at URLs like .../DASH_audio.mp4 or .../DASH_AUDIO_128.mp4
|
||||
base_url = video_url.rsplit('/', 1)[0]
|
||||
|
||||
# Common audio URL patterns
|
||||
audio_urls = [
|
||||
f"{base_url}/DASH_audio.mp4",
|
||||
f"{base_url}/DASH_AUDIO_128.mp4",
|
||||
f"{base_url}/DASH_AUDIO_64.mp4",
|
||||
f"{base_url}/audio.mp4",
|
||||
f"{base_url}/audio"
|
||||
]
|
||||
|
||||
# Download video to temp file first
|
||||
with tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False) as video_temp:
|
||||
video_temp_path = video_temp.name
|
||||
response = SESSION.get(video_url, timeout=60, stream=True)
|
||||
if response.status_code != 200:
|
||||
return False
|
||||
for chunk in response.iter_content(chunk_size=8192):
|
||||
video_temp.write(chunk)
|
||||
|
||||
# Try to download audio
|
||||
audio_temp_path = None
|
||||
for audio_url in audio_urls:
|
||||
try:
|
||||
response = SESSION.get(audio_url, timeout=30, stream=True)
|
||||
if response.status_code == 200:
|
||||
with tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False) as audio_temp:
|
||||
audio_temp_path = audio_temp.name
|
||||
for chunk in response.iter_content(chunk_size=8192):
|
||||
audio_temp.write(chunk)
|
||||
break
|
||||
except:
|
||||
continue
|
||||
|
||||
if audio_temp_path:
|
||||
# Merge video and audio using ffmpeg
|
||||
try:
|
||||
cmd = [
|
||||
'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error',
|
||||
'-i', video_temp_path,
|
||||
'-i', audio_temp_path,
|
||||
'-c:v', 'copy', '-c:a', 'aac',
|
||||
'-shortest', save_path
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, timeout=120)
|
||||
|
||||
if result.returncode == 0:
|
||||
# Cleanup temp files
|
||||
os.unlink(video_temp_path)
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
else:
|
||||
# ffmpeg failed, fall back to video only
|
||||
print(f" ⚠️ ffmpeg merge failed, saving video without audio")
|
||||
os.rename(video_temp_path, save_path)
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
except FileNotFoundError:
|
||||
# ffmpeg not installed, save video only
|
||||
print(f" ⚠️ ffmpeg not found, saving video without audio")
|
||||
os.rename(video_temp_path, save_path)
|
||||
if audio_temp_path:
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
except Exception as e:
|
||||
# Other error, save video only
|
||||
os.rename(video_temp_path, save_path)
|
||||
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
else:
|
||||
# No audio found, just use video
|
||||
os.rename(video_temp_path, save_path)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
# Cleanup any temp files on error
|
||||
pass
|
||||
return False
|
||||
|
||||
def download_post_media(post_data, dirs, post_id):
|
||||
"""Downloads all media from a post."""
|
||||
media = get_media_urls(post_data)
|
||||
|
|
@ -187,7 +280,11 @@ def download_post_media(post_data, dirs, post_id):
|
|||
if 'youtube' not in vid_url:
|
||||
ext = '.mp4'
|
||||
save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}")
|
||||
if download_media(vid_url, save_path, "video"):
|
||||
# Use enhanced download for Reddit videos (includes audio)
|
||||
if 'v.redd.it' in vid_url or 'reddit.com' in vid_url:
|
||||
if download_reddit_video_with_audio(vid_url, save_path):
|
||||
downloaded["videos"] += 1
|
||||
elif download_media(vid_url, save_path, "video"):
|
||||
downloaded["videos"] += 1
|
||||
|
||||
return downloaded
|
||||
|
|
@ -354,7 +451,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True,
|
|||
else:
|
||||
path = f"/r/{target}/new.json"
|
||||
|
||||
target_url = f"{base_url}{path}?limit=100&raw_json=1"
|
||||
# Use proper batch size - min of remaining posts needed or 100 (Reddit's max per request)
|
||||
batch_size = min(100, limit - total_posts)
|
||||
target_url = f"{base_url}{path}?limit={batch_size}&raw_json=1"
|
||||
if after:
|
||||
target_url += f"&after={after}"
|
||||
|
||||
|
|
|
|||
|
|
@ -15,6 +15,8 @@ import sys
|
|||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from config import USER_AGENT, MIRRORS, ASYNC_MAX_CONCURRENT, ASYNC_BATCH_SIZE
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
# Semaphore to limit concurrent requests
|
||||
semaphore = None
|
||||
|
|
@ -33,14 +35,14 @@ async def fetch_json(session, url, retries=3):
|
|||
await asyncio.sleep(2)
|
||||
return None
|
||||
|
||||
async def fetch_posts_page(session, base_url, target, after=None, is_user=False):
|
||||
async def fetch_posts_page(session, base_url, target, after=None, is_user=False, batch_size=100):
|
||||
"""Fetch a single page of posts."""
|
||||
if is_user:
|
||||
path = f"/user/{target}/submitted.json"
|
||||
else:
|
||||
path = f"/r/{target}/new.json"
|
||||
|
||||
url = f"{base_url}{path}?limit=100&raw_json=1"
|
||||
url = f"{base_url}{path}?limit={batch_size}&raw_json=1"
|
||||
if after:
|
||||
url += f"&after={after}"
|
||||
|
||||
|
|
@ -65,6 +67,107 @@ async def download_media_async(session, url, save_path):
|
|||
pass
|
||||
return False
|
||||
|
||||
async def download_reddit_video_with_audio_async(session, video_url, save_path):
|
||||
"""
|
||||
Downloads Reddit video with audio asynchronously.
|
||||
Reddit stores video and audio separately - this combines them using ffmpeg.
|
||||
"""
|
||||
global semaphore
|
||||
|
||||
if os.path.exists(save_path):
|
||||
return True
|
||||
|
||||
async with semaphore:
|
||||
try:
|
||||
# Find audio URL by replacing video quality with audio
|
||||
base_url = video_url.rsplit('/', 1)[0]
|
||||
audio_urls = [
|
||||
f"{base_url}/DASH_audio.mp4",
|
||||
f"{base_url}/DASH_AUDIO_128.mp4",
|
||||
f"{base_url}/DASH_AUDIO_64.mp4",
|
||||
f"{base_url}/audio.mp4",
|
||||
f"{base_url}/audio"
|
||||
]
|
||||
|
||||
# Download video to temp file
|
||||
video_temp = tempfile.NamedTemporaryFile(suffix='_video.mp4', delete=False)
|
||||
video_temp_path = video_temp.name
|
||||
video_temp.close()
|
||||
|
||||
try:
|
||||
async with session.get(video_url, timeout=aiohttp.ClientTimeout(total=60)) as response:
|
||||
if response.status != 200:
|
||||
return False
|
||||
async with aiofiles.open(video_temp_path, 'wb') as f:
|
||||
async for chunk in response.content.iter_chunked(8192):
|
||||
await f.write(chunk)
|
||||
except:
|
||||
if os.path.exists(video_temp_path):
|
||||
os.unlink(video_temp_path)
|
||||
return False
|
||||
|
||||
# Try to download audio
|
||||
audio_temp_path = None
|
||||
for audio_url in audio_urls:
|
||||
try:
|
||||
async with session.get(audio_url, timeout=aiohttp.ClientTimeout(total=30)) as response:
|
||||
if response.status == 200:
|
||||
audio_temp = tempfile.NamedTemporaryFile(suffix='_audio.mp4', delete=False)
|
||||
audio_temp_path = audio_temp.name
|
||||
audio_temp.close()
|
||||
async with aiofiles.open(audio_temp_path, 'wb') as f:
|
||||
async for chunk in response.content.iter_chunked(8192):
|
||||
await f.write(chunk)
|
||||
break
|
||||
except:
|
||||
continue
|
||||
|
||||
if audio_temp_path:
|
||||
# Merge video and audio using ffmpeg
|
||||
try:
|
||||
cmd = [
|
||||
'ffmpeg', '-y', '-hide_banner', '-loglevel', 'error',
|
||||
'-i', video_temp_path,
|
||||
'-i', audio_temp_path,
|
||||
'-c:v', 'copy', '-c:a', 'aac',
|
||||
'-shortest', save_path
|
||||
]
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE
|
||||
)
|
||||
await asyncio.wait_for(proc.wait(), timeout=120)
|
||||
|
||||
if proc.returncode == 0:
|
||||
os.unlink(video_temp_path)
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
else:
|
||||
# ffmpeg failed, use video only
|
||||
os.rename(video_temp_path, save_path)
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
except FileNotFoundError:
|
||||
# ffmpeg not installed
|
||||
os.rename(video_temp_path, save_path)
|
||||
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
except Exception:
|
||||
os.rename(video_temp_path, save_path)
|
||||
if audio_temp_path and os.path.exists(audio_temp_path):
|
||||
os.unlink(audio_temp_path)
|
||||
return True
|
||||
else:
|
||||
# No audio found, just use video
|
||||
os.rename(video_temp_path, save_path)
|
||||
return True
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
async def fetch_comments_async(session, permalink):
|
||||
"""Fetch comments asynchronously."""
|
||||
global semaphore
|
||||
|
|
@ -242,7 +345,9 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc
|
|||
|
||||
data = None
|
||||
for mirror in mirrors:
|
||||
data = await fetch_posts_page(session, mirror, target, after, is_user)
|
||||
# Use proper batch size
|
||||
batch_size = min(100, limit - total_fetched)
|
||||
data = await fetch_posts_page(session, mirror, target, after, is_user, batch_size)
|
||||
if data:
|
||||
print(f"✅ Fetched from {mirror}")
|
||||
break
|
||||
|
|
@ -288,7 +393,11 @@ async def scrape_async(target, limit=100, is_user=False, download_media=True, sc
|
|||
for i, vid_url in enumerate(media['videos'][:2]):
|
||||
if 'youtube' not in vid_url:
|
||||
save_path = f"{videos_dir}/{post['id']}_{i}.mp4"
|
||||
media_tasks.append(download_media_async(session, vid_url, save_path))
|
||||
# Use enhanced download for Reddit videos (includes audio)
|
||||
if 'v.redd.it' in vid_url or 'reddit.com' in vid_url:
|
||||
media_tasks.append(download_reddit_video_with_audio_async(session, vid_url, save_path))
|
||||
else:
|
||||
media_tasks.append(download_media_async(session, vid_url, save_path))
|
||||
|
||||
# Queue comment fetching
|
||||
if scrape_comments and post['num_comments'] > 0:
|
||||
|
|
|
|||
Loading…
Reference in a new issue