🤖 Universal Reddit Scraper

A robust, full-featured Reddit scraper that downloads posts, images, videos, galleries, and comments. Designed to run on low-resource servers (like AWS Free Tier).

🐳 Quick Start (No Installation Needed!)

docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100

✨ Features

Feature	Description
📊 Full Metadata	Title, author, score, upvotes, awards, flair, NSFW flags
🖼️ Image Download	Automatically downloads all images from posts
🎬 Video Download	Downloads Reddit-hosted videos
🖼️ Gallery Support	Extracts and downloads all images from gallery posts
💬 Comment Scraping	Recursively scrapes all comments with threading info
🔄 Dual Sources	Uses old.reddit.com + Redlib mirrors for reliability
📁 Organized Output	Clean folder structure per subreddit

📁 Output Structure

data/
└── r_delhi/
    ├── posts.csv           # All post metadata
    ├── comments.csv        # All comments with threading
    └── media/
        ├── images/         # Downloaded images & galleries
        │   ├── abc123_0.jpg
        │   ├── abc123_gallery_0.jpg
        │   └── ...
        └── videos/         # Downloaded videos
            └── xyz789_0.mp4

🚀 Usage

Full Scrape (Posts + Media + Comments)

# Scrape r/delhi with everything
python main.py delhi --mode full --limit 100

# Scrape a user's posts
python main.py spez --user --mode full --limit 50

Posts Only (No Media Download)

python main.py python --mode full --no-media --limit 200

Posts Only (No Comments)

python main.py india --mode full --no-comments --limit 100

Live Monitor Mode

python main.py delhi --mode monitor

Legacy History Mode (Posts Only, No Media)

python main.py delhi --mode history --limit 500

🐳 Docker Usage

# Build the image
docker build -t reddit-scraper .

# Full scrape with media
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100

# Scrape without media (faster)
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500

# Monitor mode (runs continuously)
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor

📊 CSV Output Format

posts.csv

Column	Description
id	Reddit post ID
title	Post title
author	Username
created_utc	Timestamp (ISO format)
permalink	Reddit URL path
url	External/media URL
score	Net upvotes
upvote_ratio	Percentage upvoted
num_comments	Comment count
selftext	Post body text
post_type	text/image/video/gallery/link
flair	Post flair text
has_media	Boolean
media_downloaded	Boolean

comments.csv

Column	Description
post_permalink	Parent post URL
comment_id	Reddit comment ID
parent_id	Parent comment/post ID
author	Username
body	Comment text
score	Net upvotes
created_utc	Timestamp
depth	Nesting level (0 = top-level)
is_submitter	Is the post author

⚙️ Command Line Options

Option	Description	Default
`target`	Subreddit or username	Required
`--mode`	`full`, `history`, or `monitor`	`full`
`--user`	Target is a user, not subreddit	`false`
`--limit`	Max posts to scrape	`100`
`--no-media`	Skip downloading images/videos	`false`
`--no-comments`	Skip scraping comments	`false`

🛠️ Requirements

pip install pandas requests

📜 License

MIT License - Feel free to use, modify, and distribute.

🤝 Contributing

Pull requests are welcome! For major changes, please open an issue first.

4.2 KiB Raw Blame History