83 lines
2.8 KiB
Markdown
83 lines
2.8 KiB
Markdown
# 🤖 Universal Reddit Scraper
|
|
|
|
[](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml)
|
|
|
|
A robust, dual-mode Reddit scraper designed to run on low-resource servers (like AWS Free Tier).
|
|
|
|
## 🐳 Quick Start (No Installation Needed!)
|
|
```bash
|
|
# Pull the pre-built image and run
|
|
docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest CreditCardsIndia --mode monitor
|
|
```
|
|
|
|
## Features
|
|
- **Zero API Keys Needed:** Uses RSS feeds and Public Mirrors (Redlib) to bypass API limits.
|
|
- **Dual Modes:**
|
|
- `monitor`: Runs 24/7 to catch new posts via RSS.
|
|
- `history`: Digs into the past using mirror rotation to bypass IP blocks.
|
|
- **Universal:** Works on any Subreddit (`r/name`) or User (`u/name`).
|
|
- **Dockerized:** Ready to deploy in an isolated container.
|
|
|
|
## Usage
|
|
|
|
### 1. Run via Docker (Recommended)
|
|
```bash
|
|
# Build
|
|
docker build -t reddit-scraper .
|
|
|
|
# Monitor a Subreddit (e.g., r/CreditCardsIndia)
|
|
docker run -d -v $(pwd)/data:/app/data reddit-scraper python main.py CreditCardsIndia --mode monitor
|
|
|
|
# Monitor a User (e.g., u/spez)
|
|
docker run -d -v $(pwd)/data:/app/data reddit-scraper python main.py spez --user --mode monitor
|
|
|
|
# Scrape History (Last 1000 posts)
|
|
docker run --rm -v $(pwd)/data:/app/data reddit-scraper python main.py CreditCardsIndia --mode history --limit 1000
|
|
```
|
|
|
|
### 2. Run Locally (Without Docker)
|
|
```bash
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Monitor a Subreddit
|
|
python main.py python --mode monitor
|
|
|
|
# Monitor a User
|
|
python main.py spez --user --mode monitor
|
|
|
|
# Scrape History
|
|
python main.py python --mode history --limit 500
|
|
```
|
|
|
|
## Output
|
|
Data is saved to the `/data` folder in CSV format:
|
|
- `data/r_CreditCardsIndia.csv`
|
|
- `data/u_spez.csv`
|
|
|
|
## Command Line Options
|
|
| Option | Description | Default |
|
|
|--------|-------------|---------|
|
|
| `target` | Name of Subreddit or User to scrape | Required |
|
|
| `--mode` | `monitor` or `history` | `monitor` |
|
|
| `--user` | Flag if target is a User, not Subreddit | `false` |
|
|
| `--limit` | Max posts to scrape (History mode only) | `500` |
|
|
|
|
## How It Works
|
|
|
|
### Monitor Mode (RSS)
|
|
- Polls Reddit's public RSS feed every 5 minutes
|
|
- Catches new posts in real-time
|
|
- Uses official Reddit RSS endpoints (no rate limits)
|
|
|
|
### History Mode (Mirrors)
|
|
- Uses public Redlib mirrors to access historical data
|
|
- Rotates between multiple mirrors to avoid IP blocks
|
|
- Implements cooldown periods to respect rate limits
|
|
- Automatically resumes from where it left off
|
|
|
|
## License
|
|
MIT License - Feel free to use, modify, and distribute.
|
|
|
|
## Contributing
|
|
Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change.
|