diff --git a/README.md b/README.md index 3741895..395ebd1 100644 --- a/README.md +++ b/README.md @@ -2,82 +2,140 @@ [![Docker Build & Publish](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml) -A robust, dual-mode Reddit scraper designed to run on low-resource servers (like AWS Free Tier). +A robust, full-featured Reddit scraper that downloads **posts, images, videos, galleries, and comments**. Designed to run on low-resource servers (like AWS Free Tier). ## 🐳 Quick Start (No Installation Needed!) ```bash -# Pull the pre-built image and run -docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest CreditCardsIndia --mode monitor -``` +docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100 +``` -## Features -- **Zero API Keys Needed:** Uses RSS feeds and Public Mirrors (Redlib) to bypass API limits. -- **Dual Modes:** - - `monitor`: Runs 24/7 to catch new posts via RSS. - - `history`: Digs into the past using mirror rotation to bypass IP blocks. -- **Universal:** Works on any Subreddit (`r/name`) or User (`u/name`). -- **Dockerized:** Ready to deploy in an isolated container. +## ✨ Features -## Usage +| Feature | Description | +|---------|-------------| +| πŸ“Š **Full Metadata** | Title, author, score, upvotes, awards, flair, NSFW flags | +| πŸ–ΌοΈ **Image Download** | Automatically downloads all images from posts | +| 🎬 **Video Download** | Downloads Reddit-hosted videos | +| πŸ–ΌοΈ **Gallery Support** | Extracts and downloads all images from gallery posts | +| πŸ’¬ **Comment Scraping** | Recursively scrapes all comments with threading info | +| πŸ”„ **Dual Sources** | Uses old.reddit.com + Redlib mirrors for reliability | +| πŸ“ **Organized Output** | Clean folder structure per subreddit | -### 1. Run via Docker (Recommended) +## πŸ“ Output Structure + +``` +data/ +└── r_delhi/ + β”œβ”€β”€ posts.csv # All post metadata + β”œβ”€β”€ comments.csv # All comments with threading + └── media/ + β”œβ”€β”€ images/ # Downloaded images & galleries + β”‚ β”œβ”€β”€ abc123_0.jpg + β”‚ β”œβ”€β”€ abc123_gallery_0.jpg + β”‚ └── ... + └── videos/ # Downloaded videos + └── xyz789_0.mp4 +``` + +## πŸš€ Usage + +### Full Scrape (Posts + Media + Comments) ```bash -# Build +# Scrape r/delhi with everything +python main.py delhi --mode full --limit 100 + +# Scrape a user's posts +python main.py spez --user --mode full --limit 50 +``` + +### Posts Only (No Media Download) +```bash +python main.py python --mode full --no-media --limit 200 +``` + +### Posts Only (No Comments) +```bash +python main.py india --mode full --no-comments --limit 100 +``` + +### Live Monitor Mode +```bash +python main.py delhi --mode monitor +``` + +### Legacy History Mode (Posts Only, No Media) +```bash +python main.py delhi --mode history --limit 500 +``` + +## 🐳 Docker Usage + +```bash +# Build the image docker build -t reddit-scraper . -# Monitor a Subreddit (e.g., r/CreditCardsIndia) -docker run -d -v $(pwd)/data:/app/data reddit-scraper python main.py CreditCardsIndia --mode monitor +# Full scrape with media +docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100 -# Monitor a User (e.g., u/spez) -docker run -d -v $(pwd)/data:/app/data reddit-scraper python main.py spez --user --mode monitor +# Scrape without media (faster) +docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500 -# Scrape History (Last 1000 posts) -docker run --rm -v $(pwd)/data:/app/data reddit-scraper python main.py CreditCardsIndia --mode history --limit 1000 +# Monitor mode (runs continuously) +docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor ``` -### 2. Run Locally (Without Docker) -```bash -# Install dependencies -pip install -r requirements.txt +## πŸ“Š CSV Output Format -# Monitor a Subreddit -python main.py python --mode monitor +### posts.csv +| Column | Description | +|--------|-------------| +| id | Reddit post ID | +| title | Post title | +| author | Username | +| created_utc | Timestamp (ISO format) | +| permalink | Reddit URL path | +| url | External/media URL | +| score | Net upvotes | +| upvote_ratio | Percentage upvoted | +| num_comments | Comment count | +| selftext | Post body text | +| post_type | text/image/video/gallery/link | +| flair | Post flair text | +| has_media | Boolean | +| media_downloaded | Boolean | -# Monitor a User -python main.py spez --user --mode monitor +### comments.csv +| Column | Description | +|--------|-------------| +| post_permalink | Parent post URL | +| comment_id | Reddit comment ID | +| parent_id | Parent comment/post ID | +| author | Username | +| body | Comment text | +| score | Net upvotes | +| created_utc | Timestamp | +| depth | Nesting level (0 = top-level) | +| is_submitter | Is the post author | -# Scrape History -python main.py python --mode history --limit 500 -``` +## βš™οΈ Command Line Options -## Output -Data is saved to the `/data` folder in CSV format: -- `data/r_CreditCardsIndia.csv` -- `data/u_spez.csv` - -## Command Line Options | Option | Description | Default | |--------|-------------|---------| -| `target` | Name of Subreddit or User to scrape | Required | -| `--mode` | `monitor` or `history` | `monitor` | -| `--user` | Flag if target is a User, not Subreddit | `false` | -| `--limit` | Max posts to scrape (History mode only) | `500` | +| `target` | Subreddit or username | Required | +| `--mode` | `full`, `history`, or `monitor` | `full` | +| `--user` | Target is a user, not subreddit | `false` | +| `--limit` | Max posts to scrape | `100` | +| `--no-media` | Skip downloading images/videos | `false` | +| `--no-comments` | Skip scraping comments | `false` | -## How It Works +## πŸ› οΈ Requirements -### Monitor Mode (RSS) -- Polls Reddit's public RSS feed every 5 minutes -- Catches new posts in real-time -- Uses official Reddit RSS endpoints (no rate limits) +```bash +pip install pandas requests +``` -### History Mode (Mirrors) -- Uses public Redlib mirrors to access historical data -- Rotates between multiple mirrors to avoid IP blocks -- Implements cooldown periods to respect rate limits -- Automatically resumes from where it left off - -## License +## πŸ“œ License MIT License - Feel free to use, modify, and distribute. -## Contributing -Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change. +## 🀝 Contributing +Pull requests are welcome! For major changes, please open an issue first. diff --git a/main.py b/main.py index 77b0848..6f314fc 100644 --- a/main.py +++ b/main.py @@ -7,12 +7,15 @@ import xml.etree.ElementTree as ET import argparse import random import sys +import json +import re +from urllib.parse import urlparse +from concurrent.futures import ThreadPoolExecutor # --- CONFIGURATION --- USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -# Public Mirrors to bypass AWS/Data Center IP Blocks -# old.reddit.com works from residential IPs, mirrors for data centers +# Sources: old.reddit.com for residential IPs, mirrors for data centers MIRRORS = [ "https://old.reddit.com", "https://redlib.catsarch.com", @@ -23,68 +26,419 @@ MIRRORS = [ ] SEEN_URLS = set() +SESSION = requests.Session() +SESSION.headers.update({"User-Agent": USER_AGENT}) + +# --- DIRECTORY SETUP --- +def setup_directories(target, prefix): + """Creates organized folder structure for scraped data.""" + base_dir = f"data/{prefix}_{target}" + dirs = { + "base": base_dir, + "posts": f"{base_dir}/posts.csv", + "comments": f"{base_dir}/comments.csv", + "media": f"{base_dir}/media", + "images": f"{base_dir}/media/images", + "videos": f"{base_dir}/media/videos", + } + + for key in ["base", "media", "images", "videos"]: + if not os.path.exists(dirs[key]): + os.makedirs(dirs[key]) + + return dirs def get_file_path(target, type_prefix): - """Generates a dynamic filename (e.g., data/r_python.csv or data/u_elonmusk.csv)""" - # Ensure data directory exists + """Legacy function for backward compatibility.""" if not os.path.exists("data"): os.makedirs("data") - sanitized_target = target.replace("/", "_") return f"data/{type_prefix}_{sanitized_target}.csv" def load_history(filepath): """Loads existing CSV history to prevent duplicates.""" - SEEN_URLS.clear() # Reset for new target + SEEN_URLS.clear() if os.path.exists(filepath): try: df = pd.read_csv(filepath) - for url in df['URL']: + for url in df['permalink']: SEEN_URLS.add(str(url)) print(f"πŸ“š Loaded {len(SEEN_URLS)} existing items from {filepath}") except: pass -def save_to_csv(items, filepath): - """Appends new unique items to the specific CSV file.""" - if not items: +def save_posts_csv(posts, filepath): + """Saves posts to CSV with all metadata.""" + if not posts: return 0 - new_items = [i for i in items if i['URL'] not in SEEN_URLS] + new_posts = [p for p in posts if p['permalink'] not in SEEN_URLS] - if new_items: - df = pd.DataFrame(new_items) + if new_posts: + df = pd.DataFrame(new_posts) if os.path.exists(filepath): df.to_csv(filepath, mode='a', header=False, index=False) else: df.to_csv(filepath, index=False) - for i in new_items: - SEEN_URLS.add(i['URL']) - - print(f"βœ… Saved {len(new_items)} new items to {filepath}") - return len(new_items) + for p in new_posts: + SEEN_URLS.add(p['permalink']) + + print(f"βœ… Saved {len(new_posts)} new posts") + return len(new_posts) else: - print("πŸ’€ No new unique data found.") + print("πŸ’€ No new unique posts found.") return 0 -# --- MODE 1: LIVE MONITOR (RSS) --- +def save_comments_csv(comments, filepath): + """Saves comments to CSV.""" + if not comments: + return + + df = pd.DataFrame(comments) + if os.path.exists(filepath): + df.to_csv(filepath, mode='a', header=False, index=False) + else: + df.to_csv(filepath, index=False) + + print(f"πŸ’¬ Saved {len(comments)} comments") + +# --- MEDIA DOWNLOAD --- +def get_media_urls(post_data): + """Extracts all media URLs from a post.""" + media = {"images": [], "videos": [], "galleries": []} + + # Direct image link + url = post_data.get('url', '') + if any(ext in url.lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp']): + media["images"].append(url) + + # Reddit-hosted image + if 'i.redd.it' in url: + media["images"].append(url) + + # Reddit video + if post_data.get('is_video'): + reddit_video = post_data.get('media', {}) + if reddit_video and 'reddit_video' in reddit_video: + video_url = reddit_video['reddit_video'].get('fallback_url', '') + if video_url: + media["videos"].append(video_url.split('?')[0]) + + # Preview images + preview = post_data.get('preview', {}) + if preview and 'images' in preview: + for img in preview['images']: + source = img.get('source', {}) + if source.get('url'): + # Unescape HTML entities + clean_url = source['url'].replace('&', '&') + media["images"].append(clean_url) + + # Gallery posts + if post_data.get('is_gallery'): + gallery_data = post_data.get('gallery_data', {}) + media_metadata = post_data.get('media_metadata', {}) + + if gallery_data and media_metadata: + for item in gallery_data.get('items', []): + media_id = item.get('media_id') + if media_id and media_id in media_metadata: + meta = media_metadata[media_id] + if meta.get('s', {}).get('u'): + clean_url = meta['s']['u'].replace('&', '&') + media["galleries"].append(clean_url) + + # External video (YouTube, etc.) + if 'youtube.com' in url or 'youtu.be' in url: + media["videos"].append(url) + + return media + +def download_media(url, save_path, media_type="image"): + """Downloads a single media file.""" + try: + # Skip if already downloaded + if os.path.exists(save_path): + return True + + response = SESSION.get(url, timeout=30, stream=True) + if response.status_code == 200: + with open(save_path, 'wb') as f: + for chunk in response.iter_content(chunk_size=8192): + f.write(chunk) + return True + except Exception as e: + print(f"⚠️ Failed to download {media_type}: {e}") + return False + +def download_post_media(post_data, dirs, post_id): + """Downloads all media from a post.""" + media = get_media_urls(post_data) + downloaded = {"images": 0, "videos": 0} + + # Download images + for i, img_url in enumerate(media["images"][:5]): # Limit to 5 images per post + ext = os.path.splitext(urlparse(img_url).path)[1] or '.jpg' + save_path = os.path.join(dirs["images"], f"{post_id}_{i}{ext}") + if download_media(img_url, save_path, "image"): + downloaded["images"] += 1 + + # Download gallery images + for i, img_url in enumerate(media["galleries"][:10]): # Limit gallery to 10 + ext = '.jpg' + save_path = os.path.join(dirs["images"], f"{post_id}_gallery_{i}{ext}") + if download_media(img_url, save_path, "gallery"): + downloaded["images"] += 1 + + # Download videos + for i, vid_url in enumerate(media["videos"][:2]): # Limit to 2 videos + if 'youtube' not in vid_url: # Skip YouTube (can't direct download) + ext = '.mp4' + save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}") + if download_media(vid_url, save_path, "video"): + downloaded["videos"] += 1 + + return downloaded + +# --- COMMENT SCRAPING --- +def scrape_comments(permalink, max_depth=3): + """Scrapes comments from a post using Reddit JSON endpoint.""" + comments = [] + + try: + # Clean permalink and build URL + if not permalink.startswith('http'): + url = f"https://old.reddit.com{permalink}.json?limit=100" + else: + url = f"{permalink}.json?limit=100" + + response = SESSION.get(url, timeout=15) + if response.status_code != 200: + return comments + + data = response.json() + + # Comments are in the second element of the response + if len(data) > 1: + comment_data = data[1]['data']['children'] + comments = parse_comments(comment_data, permalink, depth=0, max_depth=max_depth) + + except Exception as e: + print(f"⚠️ Comment fetch error: {e}") + + return comments + +def parse_comments(comment_list, post_permalink, depth=0, max_depth=3): + """Recursively parses comments.""" + comments = [] + + if depth > max_depth: + return comments + + for item in comment_list: + if item['kind'] != 't1': # Skip non-comment items + continue + + c = item['data'] + + comment = { + "post_permalink": post_permalink, + "comment_id": c.get('id'), + "parent_id": c.get('parent_id'), + "author": c.get('author'), + "body": c.get('body', ''), + "score": c.get('score', 0), + "created_utc": datetime.datetime.fromtimestamp(c.get('created_utc', 0)).isoformat(), + "depth": depth, + "is_submitter": c.get('is_submitter', False), + } + comments.append(comment) + + # Parse replies recursively + replies = c.get('replies') + if replies and isinstance(replies, dict): + reply_children = replies.get('data', {}).get('children', []) + comments.extend(parse_comments(reply_children, post_permalink, depth + 1, max_depth)) + + return comments + +# --- ENHANCED POST EXTRACTION --- +def extract_post_data(post_json): + """Extracts comprehensive post data.""" + p = post_json + + # Determine post type + post_type = "text" + if p.get('is_video'): + post_type = "video" + elif p.get('is_gallery'): + post_type = "gallery" + elif any(ext in p.get('url', '').lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp']) or 'i.redd.it' in p.get('url', ''): + post_type = "image" + elif p.get('is_self'): + post_type = "text" + else: + post_type = "link" + + return { + # Basic Info + "id": p.get('id'), + "title": p.get('title'), + "author": p.get('author'), + "created_utc": datetime.datetime.fromtimestamp(p.get('created_utc', 0)).isoformat(), + "permalink": p.get('permalink'), + "url": p.get('url_overridden_by_dest', p.get('url')), + + # Engagement + "score": p.get('score', 0), + "upvote_ratio": p.get('upvote_ratio', 0), + "num_comments": p.get('num_comments', 0), + "num_crossposts": p.get('num_crossposts', 0), + + # Content + "selftext": p.get('selftext', ''), + "post_type": post_type, + "is_nsfw": p.get('over_18', False), + "is_spoiler": p.get('spoiler', False), + + # Flair & Awards + "flair": p.get('link_flair_text', ''), + "total_awards": p.get('total_awards_received', 0), + + # Media flags + "has_media": p.get('is_video', False) or p.get('is_gallery', False) or 'i.redd.it' in p.get('url', ''), + "media_downloaded": False, + + # Source tracking + "source": "History-Full" + } + +# --- MODE 2: FULL HISTORY SCRAPE --- +def run_full_history(target, limit, is_user=False, download_media_flag=True, scrape_comments_flag=True): + """Full scrape with images, videos, and comments.""" + prefix = "u" if is_user else "r" + print(f"πŸš€ Starting FULL HISTORY scrape for {prefix}/{target}") + print(f" πŸ“Š Target posts: {limit}") + print(f" πŸ–ΌοΈ Download media: {download_media_flag}") + print(f" πŸ’¬ Scrape comments: {scrape_comments_flag}") + print("-" * 50) + + dirs = setup_directories(target, prefix) + load_history(dirs["posts"]) + + after = None + total_posts = 0 + total_media = {"images": 0, "videos": 0} + total_comments = 0 + + while total_posts < limit: + random.shuffle(MIRRORS) + success = False + + for base_url in MIRRORS: + try: + if is_user: + path = f"/user/{target}/submitted.json" + else: + path = f"/r/{target}/new.json" + + target_url = f"{base_url}{path}?limit=100&raw_json=1" + if after: + target_url += f"&after={after}" + + print(f"\nπŸ“‘ Fetching from: {base_url}") + response = SESSION.get(target_url, timeout=15) + + if response.status_code == 200: + data = response.json() + posts = [] + all_comments = [] + + children = data['data']['children'] + print(f" Found {len(children)} posts in this batch") + + for child in children: + p = child['data'] + post = extract_post_data(p) + + # Skip if already seen + if post['permalink'] in SEEN_URLS: + continue + + # Download media + if download_media_flag: + downloaded = download_post_media(p, dirs, post['id']) + post['media_downloaded'] = downloaded['images'] > 0 or downloaded['videos'] > 0 + total_media['images'] += downloaded['images'] + total_media['videos'] += downloaded['videos'] + + posts.append(post) + + # Scrape comments + if scrape_comments_flag and post['num_comments'] > 0: + print(f" πŸ’¬ Fetching comments for: {post['title'][:40]}...") + comments = scrape_comments(post['permalink']) + all_comments.extend(comments) + total_comments += len(comments) + time.sleep(1) # Rate limiting for comment fetches + + # Save data + saved = save_posts_csv(posts, dirs["posts"]) + total_posts += saved + + if all_comments: + save_comments_csv(all_comments, dirs["comments"]) + + # Progress update + print(f"\nπŸ“Š Progress: {total_posts}/{limit} posts") + print(f" πŸ–ΌοΈ Images: {total_media['images']} | 🎬 Videos: {total_media['videos']}") + print(f" πŸ’¬ Comments: {total_comments}") + + after = data['data'].get('after') + if not after: + print("\n🏁 Reached end of available history.") + return + + success = True + break + + except Exception as e: + print(f" ⚠️ Error with {base_url}: {e}") + continue + + if not success: + print("\n❌ All sources failed. Waiting 30s...") + time.sleep(30) + else: + print(f"\n⏸️ Cooling down (3s)...") + time.sleep(3) + + print("\n" + "=" * 50) + print("βœ… SCRAPE COMPLETE!") + print(f" πŸ“ Data saved to: {dirs['base']}") + print(f" πŸ“Š Total posts: {total_posts}") + print(f" πŸ–ΌοΈ Total images: {total_media['images']}") + print(f" 🎬 Total videos: {total_media['videos']}") + print(f" πŸ’¬ Total comments: {total_comments}") + +# --- MODE 1: LIVE MONITOR (RSS) - Legacy --- def run_monitor(target, is_user=False): prefix = "u" if is_user else "r" - # Reddit RSS format: /r/sub/new.rss OR /user/name/submitted.rss if is_user: rss_url = f"https://www.reddit.com/user/{target}/submitted.rss?limit=100" else: rss_url = f"https://www.reddit.com/r/{target}/new.rss?limit=100" - print(f"[{datetime.datetime.now()}] πŸ“‘ Checking RSS Feed for {prefix}/{target}...") + print(f"[{datetime.datetime.now()}] πŸ“‘ Checking RSS for {prefix}/{target}...") try: - headers = {"User-Agent": USER_AGENT} - response = requests.get(rss_url, headers=headers, timeout=15) + response = SESSION.get(rss_url, timeout=15) if response.status_code != 200: - print(f"❌ Blocked/Error (Status {response.status_code})") + print(f"❌ RSS blocked (Status {response.status_code}), trying JSON...") + # Fallback to JSON + run_full_history(target, 25, is_user, download_media_flag=False, scrape_comments_flag=False) return root = ET.fromstring(response.content) @@ -93,98 +447,73 @@ def run_monitor(target, is_user=False): for entry in root.findall('atom:entry', namespace): posts.append({ - "Date": entry.find('atom:published', namespace).text, - "Title": entry.find('atom:title', namespace).text, - "URL": entry.find('atom:link', namespace).attrib['href'], - "Source": "Monitor-RSS" + "id": "", + "title": entry.find('atom:title', namespace).text, + "author": "", + "created_utc": entry.find('atom:published', namespace).text, + "permalink": entry.find('atom:link', namespace).attrib['href'], + "url": entry.find('atom:link', namespace).attrib['href'], + "score": 0, + "upvote_ratio": 0, + "num_comments": 0, + "num_crossposts": 0, + "selftext": "", + "post_type": "unknown", + "is_nsfw": False, + "is_spoiler": False, + "flair": "", + "total_awards": 0, + "has_media": False, + "media_downloaded": False, + "source": "Monitor-RSS" }) - - save_to_csv(posts, get_file_path(target, prefix)) + + dirs = setup_directories(target, prefix) + save_posts_csv(posts, dirs["posts"]) except Exception as e: print(f"❌ Monitor Error: {e}") -# --- MODE 2: HISTORY SCRAPE (Mirrors) --- -def run_history(target, limit, is_user=False): - prefix = "u" if is_user else "r" - print(f"πŸš€ Starting HISTORY scrape for {prefix}/{target} (Target: {limit})...") - - filepath = get_file_path(target, prefix) - load_history(filepath) - - after = None - count = 0 - - while count < limit: - random.shuffle(MIRRORS) - success = False - - for base_url in MIRRORS: - try: - # Mirror URL format differs slightly for users - if is_user: - path = f"/user/{target}/submitted.json" - else: - path = f"/r/{target}/new.json" - - target_url = f"{base_url}{path}?limit=100" - if after: - target_url += f"&after={after}" - - print(f"πŸ“‘ Requesting: {base_url} (Page Token: {after})") - response = requests.get(target_url, headers={"User-Agent": USER_AGENT}, timeout=10) - - if response.status_code == 200: - data = response.json() - posts = [] - - for child in data['data']['children']: - p = child['data'] - date_str = datetime.datetime.fromtimestamp(p['created_utc']).isoformat() - posts.append({ - "Date": date_str, - "Title": p['title'], - "URL": p.get('url_overridden_by_dest', p.get('url')), - "Source": "History-Mirror" - }) - - saved = save_to_csv(posts, filepath) - count += saved - - after = data['data'].get('after') - if not after: - print("🏁 Reached end of history.") - return - - success = True - break - except: - continue - - if not success: - print("❌ All mirrors failed. Waiting 30s...") - time.sleep(30) - else: - print(f"⏸️ Cooling down (5s)... Total: {count}") - time.sleep(5) - # --- CLI ARGS --- if __name__ == "__main__": - parser = argparse.ArgumentParser(description="Universal Reddit Scraper (Open Source)") - parser.add_argument("target", help="Name of Subreddit (e.g. 'python') or User (e.g. 'spez')") - parser.add_argument("--mode", choices=["monitor", "history"], default="monitor", help="Run mode") - parser.add_argument("--user", action="store_true", help="Flag if the target is a User, not a Subreddit") - parser.add_argument("--limit", type=int, default=500, help="Max posts to scrape (History mode only)") + parser = argparse.ArgumentParser( + description="πŸ€– Universal Reddit Scraper - Full Media & Comments Support", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + python main.py delhi --mode full --limit 100 + python main.py spez --user --mode full --limit 50 + python main.py python --mode full --no-media --limit 200 + python main.py india --mode monitor + """ + ) + parser.add_argument("target", help="Subreddit name (e.g. 'delhi') or Username (e.g. 'spez')") + parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full", + help="monitor=live RSS, history=posts only, full=posts+media+comments") + parser.add_argument("--user", action="store_true", help="Target is a User, not Subreddit") + parser.add_argument("--limit", type=int, default=100, help="Max posts to scrape") + parser.add_argument("--no-media", action="store_true", help="Skip downloading images/videos") + parser.add_argument("--no-comments", action="store_true", help="Skip scraping comments") args = parser.parse_args() - # Pre-load history for monitor mode + print("=" * 50) + print("πŸ€– UNIVERSAL REDDIT SCRAPER") + print("=" * 50) + if args.mode == "monitor": prefix = "u" if args.user else "r" - load_history(get_file_path(args.target, prefix)) - print(f"πŸ€– Monitoring {prefix}/{args.target} every 5 mins...") + dirs = setup_directories(args.target, prefix) + load_history(dirs["posts"]) + print(f"πŸ”„ Monitoring {prefix}/{args.target} every 5 mins...") while True: run_monitor(args.target, args.user) time.sleep(300) - else: - run_history(args.target, args.limit, args.user) + elif args.mode == "history": + # Legacy mode - posts only + run_full_history(args.target, args.limit, args.user, + download_media_flag=False, scrape_comments_flag=False) + else: # full mode + run_full_history(args.target, args.limit, args.user, + download_media_flag=not args.no_media, + scrape_comments_flag=not args.no_comments)