141 lines
4.2 KiB
Markdown
141 lines
4.2 KiB
Markdown
# 🤖 Universal Reddit Scraper
|
|
|
|
[](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml)
|
|
|
|
A robust, full-featured Reddit scraper that downloads **posts, images, videos, galleries, and comments**. Designed to run on low-resource servers (like AWS Free Tier).
|
|
|
|
## 🐳 Quick Start (No Installation Needed!)
|
|
```bash
|
|
docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100
|
|
```
|
|
|
|
## ✨ Features
|
|
|
|
| Feature | Description |
|
|
|---------|-------------|
|
|
| 📊 **Full Metadata** | Title, author, score, upvotes, awards, flair, NSFW flags |
|
|
| 🖼️ **Image Download** | Automatically downloads all images from posts |
|
|
| 🎬 **Video Download** | Downloads Reddit-hosted videos |
|
|
| 🖼️ **Gallery Support** | Extracts and downloads all images from gallery posts |
|
|
| 💬 **Comment Scraping** | Recursively scrapes all comments with threading info |
|
|
| 🔄 **Dual Sources** | Uses old.reddit.com + Redlib mirrors for reliability |
|
|
| 📁 **Organized Output** | Clean folder structure per subreddit |
|
|
|
|
## 📁 Output Structure
|
|
|
|
```
|
|
data/
|
|
└── r_delhi/
|
|
├── posts.csv # All post metadata
|
|
├── comments.csv # All comments with threading
|
|
└── media/
|
|
├── images/ # Downloaded images & galleries
|
|
│ ├── abc123_0.jpg
|
|
│ ├── abc123_gallery_0.jpg
|
|
│ └── ...
|
|
└── videos/ # Downloaded videos
|
|
└── xyz789_0.mp4
|
|
```
|
|
|
|
## 🚀 Usage
|
|
|
|
### Full Scrape (Posts + Media + Comments)
|
|
```bash
|
|
# Scrape r/delhi with everything
|
|
python main.py delhi --mode full --limit 100
|
|
|
|
# Scrape a user's posts
|
|
python main.py spez --user --mode full --limit 50
|
|
```
|
|
|
|
### Posts Only (No Media Download)
|
|
```bash
|
|
python main.py python --mode full --no-media --limit 200
|
|
```
|
|
|
|
### Posts Only (No Comments)
|
|
```bash
|
|
python main.py india --mode full --no-comments --limit 100
|
|
```
|
|
|
|
### Live Monitor Mode
|
|
```bash
|
|
python main.py delhi --mode monitor
|
|
```
|
|
|
|
### Legacy History Mode (Posts Only, No Media)
|
|
```bash
|
|
python main.py delhi --mode history --limit 500
|
|
```
|
|
|
|
## 🐳 Docker Usage
|
|
|
|
```bash
|
|
# Build the image
|
|
docker build -t reddit-scraper .
|
|
|
|
# Full scrape with media
|
|
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100
|
|
|
|
# Scrape without media (faster)
|
|
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500
|
|
|
|
# Monitor mode (runs continuously)
|
|
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
|
|
```
|
|
|
|
## 📊 CSV Output Format
|
|
|
|
### posts.csv
|
|
| Column | Description |
|
|
|--------|-------------|
|
|
| id | Reddit post ID |
|
|
| title | Post title |
|
|
| author | Username |
|
|
| created_utc | Timestamp (ISO format) |
|
|
| permalink | Reddit URL path |
|
|
| url | External/media URL |
|
|
| score | Net upvotes |
|
|
| upvote_ratio | Percentage upvoted |
|
|
| num_comments | Comment count |
|
|
| selftext | Post body text |
|
|
| post_type | text/image/video/gallery/link |
|
|
| flair | Post flair text |
|
|
| has_media | Boolean |
|
|
| media_downloaded | Boolean |
|
|
|
|
### comments.csv
|
|
| Column | Description |
|
|
|--------|-------------|
|
|
| post_permalink | Parent post URL |
|
|
| comment_id | Reddit comment ID |
|
|
| parent_id | Parent comment/post ID |
|
|
| author | Username |
|
|
| body | Comment text |
|
|
| score | Net upvotes |
|
|
| created_utc | Timestamp |
|
|
| depth | Nesting level (0 = top-level) |
|
|
| is_submitter | Is the post author |
|
|
|
|
## ⚙️ Command Line Options
|
|
|
|
| Option | Description | Default |
|
|
|--------|-------------|---------|
|
|
| `target` | Subreddit or username | Required |
|
|
| `--mode` | `full`, `history`, or `monitor` | `full` |
|
|
| `--user` | Target is a user, not subreddit | `false` |
|
|
| `--limit` | Max posts to scrape | `100` |
|
|
| `--no-media` | Skip downloading images/videos | `false` |
|
|
| `--no-comments` | Skip scraping comments | `false` |
|
|
|
|
## 🛠️ Requirements
|
|
|
|
```bash
|
|
pip install pandas requests
|
|
```
|
|
|
|
## 📜 License
|
|
MIT License - Feel free to use, modify, and distribute.
|
|
|
|
## 🤝 Contributing
|
|
Pull requests are welcome! For major changes, please open an issue first.
|