diff --git a/README.md b/README.md index 395ebd1..33fcea4 100644 --- a/README.md +++ b/README.md @@ -1,90 +1,148 @@ -# πŸ€– Universal Reddit Scraper +# πŸ€– Universal Reddit Scraper Suite [![Docker Build & Publish](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml) -A robust, full-featured Reddit scraper that downloads **posts, images, videos, galleries, and comments**. Designed to run on low-resource servers (like AWS Free Tier). - -## 🐳 Quick Start (No Installation Needed!) -```bash -docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100 -``` +A **full-featured** Reddit scraper suite with analytics dashboard, sentiment analysis, scheduled scraping, notifications, and more! ## ✨ Features | Feature | Description | |---------|-------------| -| πŸ“Š **Full Metadata** | Title, author, score, upvotes, awards, flair, NSFW flags | -| πŸ–ΌοΈ **Image Download** | Automatically downloads all images from posts | -| 🎬 **Video Download** | Downloads Reddit-hosted videos | -| πŸ–ΌοΈ **Gallery Support** | Extracts and downloads all images from gallery posts | -| πŸ’¬ **Comment Scraping** | Recursively scrapes all comments with threading info | -| πŸ”„ **Dual Sources** | Uses old.reddit.com + Redlib mirrors for reliability | -| πŸ“ **Organized Output** | Clean folder structure per subreddit | +| πŸ“Š **Full Scraping** | Posts, comments, images, videos, galleries | +| πŸ“ˆ **Analytics Dashboard** | Beautiful Streamlit web UI | +| πŸ˜€ **Sentiment Analysis** | Analyze post/comment sentiment | +| ☁️ **Keyword Extraction** | Generate word clouds | +| πŸ” **Search & Filter** | Query scraped data with filters | +| πŸ“… **Scheduled Scraping** | Cron-style job scheduling | +| πŸ“§ **Notifications** | Discord & Telegram alerts | +| πŸ—„οΈ **SQLite Database** | Structured data storage | +| πŸ“€ **Multiple Exports** | CSV, JSON, Excel | -## πŸ“ Output Structure +## πŸš€ Quick Start -``` -data/ -└── r_delhi/ - β”œβ”€β”€ posts.csv # All post metadata - β”œβ”€β”€ comments.csv # All comments with threading - └── media/ - β”œβ”€β”€ images/ # Downloaded images & galleries - β”‚ β”œβ”€β”€ abc123_0.jpg - β”‚ β”œβ”€β”€ abc123_gallery_0.jpg - β”‚ └── ... - └── videos/ # Downloaded videos - └── xyz789_0.mp4 -``` - -## πŸš€ Usage - -### Full Scrape (Posts + Media + Comments) ```bash -# Scrape r/delhi with everything +# Install dependencies +pip install -r requirements.txt + +# Scrape a subreddit (posts + media + comments) python main.py delhi --mode full --limit 100 +# Launch analytics dashboard +python main.py --dashboard +``` + +## πŸ“– Usage Guide + +### πŸ”„ Scraping Modes + +```bash +# Full scrape with everything +python main.py delhi --mode full --limit 100 + +# History only (no media/comments - faster) +python main.py delhi --mode history --limit 500 + +# Live monitor (checks every 5 min) +python main.py delhi --mode monitor + # Scrape a user's posts python main.py spez --user --mode full --limit 50 + +# Skip media or comments +python main.py delhi --mode full --no-media --limit 200 +python main.py delhi --mode full --no-comments --limit 200 ``` -### Posts Only (No Media Download) -```bash -python main.py python --mode full --no-media --limit 200 -``` - -### Posts Only (No Comments) -```bash -python main.py india --mode full --no-comments --limit 100 -``` - -### Live Monitor Mode -```bash -python main.py delhi --mode monitor -``` - -### Legacy History Mode (Posts Only, No Media) -```bash -python main.py delhi --mode history --limit 500 -``` - -## 🐳 Docker Usage +### πŸ“Š Analytics Dashboard ```bash -# Build the image -docker build -t reddit-scraper . +# Launch the web dashboard +python main.py --dashboard -# Full scrape with media -docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100 - -# Scrape without media (faster) -docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500 - -# Monitor mode (runs continuously) -docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor +# Opens at http://localhost:8501 ``` -## πŸ“Š CSV Output Format +**Dashboard Features:** +- πŸ“ˆ Post statistics & charts +- πŸ˜€ Sentiment analysis +- ☁️ Keyword extraction +- πŸ” Search & filter interface +- πŸ“€ Export data + +### πŸ” Search Data + +```bash +# Search all scraped data +python main.py --search "credit card" + +# Search with filters +python main.py --search "laptop" --min-score 100 +python main.py --search "advice" --author username +python main.py --search "help" --subreddit delhi +``` + +### πŸ˜€ Analytics + +```bash +# Run sentiment analysis +python main.py --analyze delhi --sentiment + +# Extract top keywords +python main.py --analyze delhi --keywords +``` + +### πŸ“… Scheduled Scraping + +```bash +# Scrape every 60 minutes +python main.py --schedule delhi --every 60 + +# Scrape with options +python main.py --schedule delhi --every 30 --mode full --limit 50 +``` + +### πŸ“§ Notifications (Discord/Telegram) + +**Discord:** +```bash +python main.py delhi --mode monitor --discord-webhook "YOUR_WEBHOOK_URL" +``` + +**Telegram:** +```bash +python main.py delhi --mode monitor \ + --telegram-token "YOUR_BOT_TOKEN" \ + --telegram-chat "YOUR_CHAT_ID" +``` + +## πŸ“ Project Structure + +``` +reddit-scraper/ +β”œβ”€β”€ main.py # Main CLI entry point +β”œβ”€β”€ config.py # Configuration settings +β”œβ”€β”€ analytics/ # Sentiment & keyword analysis +β”‚ └── sentiment.py +β”œβ”€β”€ alerts/ # Discord & Telegram notifications +β”‚ └── notifications.py +β”œβ”€β”€ dashboard/ # Streamlit web UI +β”‚ └── app.py +β”œβ”€β”€ export/ # Database & export functions +β”‚ └── database.py +β”œβ”€β”€ scheduler/ # Cron-style scheduling +β”‚ └── cron.py +β”œβ”€β”€ search/ # Search & filter engine +β”‚ └── query.py +└── data/ # Scraped data + └── r_subreddit/ + β”œβ”€β”€ posts.csv + β”œβ”€β”€ comments.csv + └── media/ + β”œβ”€β”€ images/ + └── videos/ +``` + +## πŸ“Š Data Output ### posts.csv | Column | Description | @@ -92,50 +150,51 @@ docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor | id | Reddit post ID | | title | Post title | | author | Username | -| created_utc | Timestamp (ISO format) | -| permalink | Reddit URL path | -| url | External/media URL | | score | Net upvotes | -| upvote_ratio | Percentage upvoted | | num_comments | Comment count | -| selftext | Post body text | -| post_type | text/image/video/gallery/link | -| flair | Post flair text | -| has_media | Boolean | -| media_downloaded | Boolean | +| post_type | text/image/video/gallery | +| selftext | Post body | +| flair | Post flair | +| is_nsfw | NSFW flag | +| created_utc | Timestamp | ### comments.csv | Column | Description | |--------|-------------| -| post_permalink | Parent post URL | -| comment_id | Reddit comment ID | -| parent_id | Parent comment/post ID | +| comment_id | Comment ID | +| post_permalink | Parent post | | author | Username | | body | Comment text | -| score | Net upvotes | -| created_utc | Timestamp | -| depth | Nesting level (0 = top-level) | -| is_submitter | Is the post author | +| score | Upvotes | +| depth | Nesting level | -## βš™οΈ Command Line Options - -| Option | Description | Default | -|--------|-------------|---------| -| `target` | Subreddit or username | Required | -| `--mode` | `full`, `history`, or `monitor` | `full` | -| `--user` | Target is a user, not subreddit | `false` | -| `--limit` | Max posts to scrape | `100` | -| `--no-media` | Skip downloading images/videos | `false` | -| `--no-comments` | Skip scraping comments | `false` | - -## πŸ› οΈ Requirements +## 🐳 Docker ```bash -pip install pandas requests +# Build +docker build -t reddit-scraper . + +# Full scrape +docker run -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100 + +# Monitor mode +docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor +``` + +## βš™οΈ Configuration + +Edit `config.py` or use environment variables: + +```bash +export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." +export TELEGRAM_BOT_TOKEN="123456:ABC..." +export TELEGRAM_CHAT_ID="987654321" ``` ## πŸ“œ License + MIT License - Feel free to use, modify, and distribute. ## 🀝 Contributing -Pull requests are welcome! For major changes, please open an issue first. + +Pull requests welcome! For major changes, please open an issue first. diff --git a/alerts/__init__.py b/alerts/__init__.py new file mode 100644 index 0000000..3e006e5 --- /dev/null +++ b/alerts/__init__.py @@ -0,0 +1,2 @@ +# Alerts module +from .notifications import * diff --git a/alerts/notifications.py b/alerts/notifications.py new file mode 100644 index 0000000..d92496c --- /dev/null +++ b/alerts/notifications.py @@ -0,0 +1,208 @@ +""" +Notification module - Discord & Telegram alerts +""" +import requests +import json +from datetime import datetime + +def send_discord_alert(webhook_url, title, message, posts=None, color=0x5865F2): + """ + Send alert to Discord via webhook. + + Args: + webhook_url: Discord webhook URL + title: Alert title + message: Alert message + posts: Optional list of posts to include + color: Embed color (default: Discord blue) + """ + if not webhook_url: + print("⚠️ Discord webhook URL not configured") + return False + + embeds = [{ + "title": f"πŸ€– {title}", + "description": message, + "color": color, + "timestamp": datetime.utcnow().isoformat(), + "footer": {"text": "Reddit Scraper Alert"} + }] + + # Add post previews + if posts: + fields = [] + for post in posts[:5]: # Max 5 posts + fields.append({ + "name": post.get('title', 'No Title')[:100], + "value": f"Score: {post.get('score', 0)} | Comments: {post.get('num_comments', 0)}\n[View Post](https://reddit.com{post.get('permalink', '')})", + "inline": False + }) + embeds[0]["fields"] = fields + + payload = {"embeds": embeds} + + try: + response = requests.post( + webhook_url, + json=payload, + headers={"Content-Type": "application/json"}, + timeout=10 + ) + if response.status_code == 204: + print("βœ… Discord alert sent!") + return True + else: + print(f"❌ Discord error: {response.status_code}") + return False + except Exception as e: + print(f"❌ Discord error: {e}") + return False + +def send_telegram_alert(bot_token, chat_id, title, message, posts=None): + """ + Send alert to Telegram via bot. + + Args: + bot_token: Telegram bot token + chat_id: Chat/Channel ID to send to + title: Alert title + message: Alert message + posts: Optional list of posts to include + """ + if not bot_token or not chat_id: + print("⚠️ Telegram credentials not configured") + return False + + # Build message + text = f"πŸ€– *{title}*\n\n{message}" + + if posts: + text += "\n\nπŸ“ *New Posts:*\n" + for post in posts[:5]: + title_text = post.get('title', 'No Title')[:80] + score = post.get('score', 0) + permalink = post.get('permalink', '') + text += f"\nβ€’ [{title_text}](https://reddit.com{permalink}) (⬆️ {score})" + + url = f"https://api.telegram.org/bot{bot_token}/sendMessage" + payload = { + "chat_id": chat_id, + "text": text, + "parse_mode": "Markdown", + "disable_web_page_preview": True + } + + try: + response = requests.post(url, json=payload, timeout=10) + if response.status_code == 200: + print("βœ… Telegram alert sent!") + return True + else: + print(f"❌ Telegram error: {response.json()}") + return False + except Exception as e: + print(f"❌ Telegram error: {e}") + return False + +def check_keyword_alerts(posts, keywords, webhook_url=None, telegram_token=None, telegram_chat=None): + """ + Check posts for keyword matches and send alerts. + + Args: + posts: List of posts to check + keywords: List of keywords to monitor + webhook_url: Discord webhook URL + telegram_token: Telegram bot token + telegram_chat: Telegram chat ID + + Returns: + List of matching posts + """ + if not keywords: + return [] + + keywords_lower = [k.lower() for k in keywords] + matching_posts = [] + + for post in posts: + text = f"{post.get('title', '')} {post.get('selftext', '')}".lower() + + matched_keywords = [] + for keyword in keywords_lower: + if keyword in text: + matched_keywords.append(keyword) + + if matched_keywords: + post['matched_keywords'] = matched_keywords + matching_posts.append(post) + + if matching_posts: + title = f"Keyword Alert: {len(matching_posts)} matches!" + message = f"Found posts matching: {', '.join(set(k for p in matching_posts for k in p.get('matched_keywords', [])))}" + + if webhook_url: + send_discord_alert(webhook_url, title, message, matching_posts, color=0xFF6B6B) + + if telegram_token and telegram_chat: + send_telegram_alert(telegram_token, telegram_chat, title, message, matching_posts) + + return matching_posts + +def send_scrape_summary(subreddit, stats, webhook_url=None, telegram_token=None, telegram_chat=None): + """ + Send a summary after scraping completes. + + Args: + subreddit: Subreddit name + stats: Dictionary with scrape statistics + webhook_url: Discord webhook URL + telegram_token: Telegram bot token + telegram_chat: Telegram chat ID + """ + title = f"Scrape Complete: r/{subreddit}" + message = f""" +πŸ“Š **Statistics:** +β€’ Posts: {stats.get('posts', 0)} +β€’ Comments: {stats.get('comments', 0)} +β€’ Images: {stats.get('images', 0)} +β€’ Videos: {stats.get('videos', 0)} +β€’ Duration: {stats.get('duration', 'N/A')} + """.strip() + + if webhook_url: + send_discord_alert(webhook_url, title, message, color=0x00D166) + + if telegram_token and telegram_chat: + send_telegram_alert(telegram_token, telegram_chat, title, message) + +class AlertMonitor: + """Monitor for keyword-based alerts.""" + + def __init__(self, keywords, discord_webhook=None, telegram_token=None, telegram_chat=None): + self.keywords = keywords + self.discord_webhook = discord_webhook + self.telegram_token = telegram_token + self.telegram_chat = telegram_chat + self.seen_posts = set() + + def check_posts(self, posts): + """Check new posts for keyword matches.""" + new_posts = [p for p in posts if p.get('id') not in self.seen_posts] + + if not new_posts: + return [] + + # Mark as seen + for p in new_posts: + self.seen_posts.add(p.get('id')) + + # Check for keywords + matches = check_keyword_alerts( + new_posts, + self.keywords, + self.discord_webhook, + self.telegram_token, + self.telegram_chat + ) + + return matches diff --git a/analytics/__init__.py b/analytics/__init__.py new file mode 100644 index 0000000..a00602f --- /dev/null +++ b/analytics/__init__.py @@ -0,0 +1,2 @@ +# Analytics module +from .sentiment import * diff --git a/analytics/sentiment.py b/analytics/sentiment.py new file mode 100644 index 0000000..943c092 --- /dev/null +++ b/analytics/sentiment.py @@ -0,0 +1,236 @@ +""" +Analytics module - Sentiment Analysis, Word Clouds, Statistics +""" +import re +from collections import Counter +from pathlib import Path +import sys + +# Simple sentiment analysis without external dependencies +POSITIVE_WORDS = { + 'good', 'great', 'awesome', 'excellent', 'amazing', 'love', 'best', 'perfect', + 'nice', 'wonderful', 'fantastic', 'brilliant', 'superb', 'outstanding', 'happy', + 'beautiful', 'helpful', 'thanks', 'thank', 'appreciate', 'recommend', 'interesting', + 'useful', 'cool', 'fun', 'enjoy', 'like', 'loved', 'impressive', 'incredible' +} + +NEGATIVE_WORDS = { + 'bad', 'terrible', 'awful', 'horrible', 'hate', 'worst', 'poor', 'disappointing', + 'useless', 'waste', 'annoying', 'boring', 'ugly', 'stupid', 'dumb', 'fail', + 'wrong', 'broken', 'sad', 'angry', 'frustrated', 'scam', 'fake', 'trash', + 'pathetic', 'ridiculous', 'disgusting', 'overpriced', 'avoid', 'never' +} + +INTENSIFIERS = {'very', 'really', 'extremely', 'absolutely', 'totally', 'completely'} + +def analyze_sentiment(text): + """ + Simple sentiment analysis. + Returns: (score, label) + - score: -1.0 to 1.0 + - label: 'positive', 'negative', or 'neutral' + """ + if not text: + return 0.0, 'neutral' + + # Clean and tokenize + words = re.findall(r'\b[a-z]+\b', text.lower()) + + if not words: + return 0.0, 'neutral' + + positive_count = 0 + negative_count = 0 + intensifier_next = False + + for word in words: + multiplier = 1.5 if intensifier_next else 1.0 + + if word in POSITIVE_WORDS: + positive_count += multiplier + elif word in NEGATIVE_WORDS: + negative_count += multiplier + + intensifier_next = word in INTENSIFIERS + + total = positive_count + negative_count + if total == 0: + return 0.0, 'neutral' + + score = (positive_count - negative_count) / len(words) + score = max(-1.0, min(1.0, score * 5)) # Normalize + + if score > 0.1: + label = 'positive' + elif score < -0.1: + label = 'negative' + else: + label = 'neutral' + + return round(score, 3), label + +def analyze_posts_sentiment(posts): + """Analyze sentiment for a list of posts.""" + results = [] + sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0} + + for post in posts: + text = f"{post.get('title', '')} {post.get('selftext', '')}" + score, label = analyze_sentiment(text) + post['sentiment_score'] = score + post['sentiment_label'] = label + sentiment_counts[label] += 1 + results.append(post) + + return results, sentiment_counts + +def analyze_comments_sentiment(comments): + """Analyze sentiment for comments.""" + results = [] + sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0} + + for comment in comments: + score, label = analyze_sentiment(comment.get('body', '')) + comment['sentiment_score'] = score + comment['sentiment_label'] = label + sentiment_counts[label] += 1 + results.append(comment) + + return results, sentiment_counts + +def extract_keywords(texts, top_n=50): + """Extract most common keywords from texts.""" + # Stopwords + stopwords = { + 'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being', + 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could', + 'should', 'may', 'might', 'must', 'shall', 'can', 'to', 'of', 'in', + 'for', 'on', 'with', 'at', 'by', 'from', 'as', 'into', 'through', + 'during', 'before', 'after', 'above', 'below', 'between', 'under', + 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', + 'why', 'how', 'all', 'each', 'few', 'more', 'most', 'other', 'some', + 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', + 'too', 'very', 'just', 'and', 'but', 'if', 'or', 'because', 'until', + 'while', 'this', 'that', 'these', 'those', 'i', 'me', 'my', 'myself', + 'we', 'our', 'you', 'your', 'he', 'she', 'it', 'they', 'them', 'what', + 'which', 'who', 'whom', 'its', 'his', 'her', 'their', 'our', 'up', + 'out', 'about', 'any', 'also', 'get', 'got', 'like', 'one', 'two', + 'know', 'even', 'new', 'want', 'way', 'people', 'time', 'year', 'think', + 'amp', 'http', 'https', 'www', 'com', 'reddit', 'deleted', 'removed', 'nan' + } + + all_words = [] + for text in texts: + if text: + words = re.findall(r'\b[a-z]{3,}\b', text.lower()) + all_words.extend([w for w in words if w not in stopwords]) + + return Counter(all_words).most_common(top_n) + +def generate_wordcloud_data(texts, top_n=100): + """Generate word frequency data for word cloud visualization.""" + keywords = extract_keywords(texts, top_n) + + if not keywords: + return [] + + max_count = keywords[0][1] + + return [ + {"text": word, "value": count, "size": int(10 + (count / max_count) * 90)} + for word, count in keywords + ] + +def calculate_engagement_metrics(posts): + """Calculate engagement metrics for posts.""" + if not posts: + return {} + + total_posts = len(posts) + total_score = sum(p.get('score', 0) for p in posts) + total_comments = sum(p.get('num_comments', 0) for p in posts) + total_awards = sum(p.get('total_awards', 0) for p in posts) + + # Posts with engagement + engaged_posts = [p for p in posts if p.get('score', 0) > 0 or p.get('num_comments', 0) > 0] + + # Top performers + top_by_score = sorted(posts, key=lambda x: x.get('score', 0), reverse=True)[:10] + top_by_comments = sorted(posts, key=lambda x: x.get('num_comments', 0), reverse=True)[:10] + + # Post type performance + type_performance = {} + for post in posts: + ptype = post.get('post_type', 'unknown') + if ptype not in type_performance: + type_performance[ptype] = {'count': 0, 'total_score': 0, 'total_comments': 0} + type_performance[ptype]['count'] += 1 + type_performance[ptype]['total_score'] += post.get('score', 0) + type_performance[ptype]['total_comments'] += post.get('num_comments', 0) + + for ptype in type_performance: + count = type_performance[ptype]['count'] + type_performance[ptype]['avg_score'] = type_performance[ptype]['total_score'] / count + type_performance[ptype]['avg_comments'] = type_performance[ptype]['total_comments'] / count + + return { + 'total_posts': total_posts, + 'total_score': total_score, + 'total_comments': total_comments, + 'total_awards': total_awards, + 'avg_score': total_score / total_posts if total_posts else 0, + 'avg_comments': total_comments / total_posts if total_posts else 0, + 'engagement_rate': len(engaged_posts) / total_posts if total_posts else 0, + 'top_by_score': top_by_score, + 'top_by_comments': top_by_comments, + 'type_performance': type_performance + } + +def find_best_posting_times(posts): + """Analyze best times to post based on engagement.""" + hourly_stats = {} + daily_stats = {} + + for post in posts: + created = post.get('created_utc', '') + if not created: + continue + + try: + # Parse ISO format + from datetime import datetime + dt = datetime.fromisoformat(created.replace('Z', '+00:00')) + hour = dt.hour + day = dt.strftime('%A') + + # Hourly + if hour not in hourly_stats: + hourly_stats[hour] = {'count': 0, 'total_score': 0} + hourly_stats[hour]['count'] += 1 + hourly_stats[hour]['total_score'] += post.get('score', 0) + + # Daily + if day not in daily_stats: + daily_stats[day] = {'count': 0, 'total_score': 0} + daily_stats[day]['count'] += 1 + daily_stats[day]['total_score'] += post.get('score', 0) + except: + continue + + # Calculate averages + for hour in hourly_stats: + hourly_stats[hour]['avg_score'] = hourly_stats[hour]['total_score'] / hourly_stats[hour]['count'] + + for day in daily_stats: + daily_stats[day]['avg_score'] = daily_stats[day]['total_score'] / daily_stats[day]['count'] + + # Find best times + best_hours = sorted(hourly_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:5] + best_days = sorted(daily_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:3] + + return { + 'hourly_stats': hourly_stats, + 'daily_stats': daily_stats, + 'best_hours': [(h, s['avg_score']) for h, s in best_hours], + 'best_days': [(d, s['avg_score']) for d, s in best_days] + } diff --git a/config.py b/config.py new file mode 100644 index 0000000..b07cb43 --- /dev/null +++ b/config.py @@ -0,0 +1,58 @@ +""" +Reddit Scraper Suite - Configuration +""" +import os +from pathlib import Path + +# --- PATHS --- +BASE_DIR = Path(__file__).parent +DATA_DIR = BASE_DIR / "data" +DB_PATH = DATA_DIR / "reddit_scraper.db" + +# --- SCRAPER SETTINGS --- +USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" + +# Sources: old.reddit.com for residential IPs, mirrors for data centers +MIRRORS = [ + "https://old.reddit.com", + "https://redlib.catsarch.com", + "https://redlib.vsls.cz", + "https://r.nf", + "https://libreddit.northboot.xyz", + "https://redlib.tux.pizza" +] + +# Rate limiting +REQUEST_TIMEOUT = 15 +COOLDOWN_SECONDS = 3 +RETRY_WAIT = 30 + +# Media settings +MAX_IMAGES_PER_POST = 10 +MAX_VIDEOS_PER_POST = 2 +MAX_GALLERY_IMAGES = 15 + +# Comment settings +MAX_COMMENT_DEPTH = 5 + +# --- ASYNC SETTINGS --- +ASYNC_MAX_CONCURRENT = 10 +ASYNC_BATCH_SIZE = 50 + +# --- NOTIFICATION SETTINGS --- +DISCORD_WEBHOOK_URL = os.getenv("DISCORD_WEBHOOK_URL", "") +TELEGRAM_BOT_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "") +TELEGRAM_CHAT_ID = os.getenv("TELEGRAM_CHAT_ID", "") + +# --- DASHBOARD SETTINGS --- +DASHBOARD_HOST = "0.0.0.0" +DASHBOARD_PORT = 8501 + +# --- SCHEDULER SETTINGS --- +SCHEDULER_TIMEZONE = "Asia/Kolkata" + +# --- DATABASE SETTINGS --- +DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DB_PATH}") + +# Ensure data directory exists +DATA_DIR.mkdir(exist_ok=True) diff --git a/dashboard/__init__.py b/dashboard/__init__.py new file mode 100644 index 0000000..e44fc89 --- /dev/null +++ b/dashboard/__init__.py @@ -0,0 +1 @@ +# Dashboard module diff --git a/dashboard/app.py b/dashboard/app.py new file mode 100644 index 0000000..8e16830 --- /dev/null +++ b/dashboard/app.py @@ -0,0 +1,364 @@ +""" +Reddit Scraper Dashboard - Streamlit Web UI +Run with: streamlit run dashboard/app.py +""" +import streamlit as st +import pandas as pd +from pathlib import Path +import sys +from datetime import datetime + +# Add parent to path +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from analytics.sentiment import ( + analyze_posts_sentiment, extract_keywords, + calculate_engagement_metrics, find_best_posting_times +) +from search.query import search_all_data, advanced_search, get_top_posts + +# Page config +st.set_page_config( + page_title="Reddit Scraper Dashboard", + page_icon="πŸ€–", + layout="wide", + initial_sidebar_state="expanded" +) + +# Custom CSS +st.markdown(""" + +""", unsafe_allow_html=True) + +def load_subreddit_data(subreddit_path): + """Load all data for a subreddit.""" + data = {} + + posts_file = subreddit_path / 'posts.csv' + if posts_file.exists(): + data['posts'] = pd.read_csv(posts_file) + + comments_file = subreddit_path / 'comments.csv' + if comments_file.exists(): + data['comments'] = pd.read_csv(comments_file) + + return data + +def get_available_subreddits(): + """Get list of scraped subreddits.""" + data_dir = Path(__file__).parent.parent / 'data' + subs = [] + + if data_dir.exists(): + for sub_dir in data_dir.iterdir(): + if sub_dir.is_dir() and (sub_dir / 'posts.csv').exists(): + subs.append(sub_dir.name) + + return sorted(subs) + +def main(): + # Header + st.markdown('

πŸ€– Reddit Scraper Dashboard

', unsafe_allow_html=True) + + # Sidebar + st.sidebar.title("πŸ“Š Navigation") + + # Get available subreddits + subreddits = get_available_subreddits() + + if not subreddits: + st.warning("No scraped data found! Run the scraper first:") + st.code("python main.py --mode full --limit 100") + return + + # Subreddit selector + selected_sub = st.sidebar.selectbox( + "Select Subreddit", + subreddits, + format_func=lambda x: f"πŸ“ {x}" + ) + + # Load data + data_dir = Path(__file__).parent.parent / 'data' + sub_path = data_dir / selected_sub + data = load_subreddit_data(sub_path) + + if 'posts' not in data: + st.error("No posts data found!") + return + + posts_df = data['posts'] + comments_df = data.get('comments', pd.DataFrame()) + + # Main content tabs + tab1, tab2, tab3, tab4, tab5 = st.tabs([ + "πŸ“Š Overview", "πŸ“ˆ Analytics", "πŸ” Search", "πŸ’¬ Comments", "βš™οΈ Scraper" + ]) + + with tab1: + st.header(f"πŸ“Š Overview: {selected_sub}") + + # Metrics row + col1, col2, col3, col4, col5 = st.columns(5) + + with col1: + st.metric("Total Posts", len(posts_df)) + with col2: + st.metric("Total Comments", len(comments_df)) + with col3: + total_score = posts_df['score'].sum() if 'score' in posts_df else 0 + st.metric("Total Score", f"{total_score:,}") + with col4: + avg_score = posts_df['score'].mean() if 'score' in posts_df else 0 + st.metric("Avg Score", f"{avg_score:.1f}") + with col5: + media_count = posts_df['has_media'].sum() if 'has_media' in posts_df else 0 + st.metric("Media Posts", int(media_count)) + + st.divider() + + # Post type distribution + col1, col2 = st.columns(2) + + with col1: + st.subheader("πŸ“ Post Types") + if 'post_type' in posts_df: + type_counts = posts_df['post_type'].value_counts() + st.bar_chart(type_counts) + + with col2: + st.subheader("πŸ“… Posts Over Time") + if 'created_utc' in posts_df: + posts_df['date'] = pd.to_datetime(posts_df['created_utc']).dt.date + daily = posts_df.groupby('date').size() + st.line_chart(daily) + + st.divider() + + # Top posts + st.subheader("πŸ”₯ Top Posts by Score") + if 'score' in posts_df: + top_posts = posts_df.nlargest(10, 'score')[['title', 'score', 'num_comments', 'post_type', 'created_utc']] + st.dataframe(top_posts, use_container_width=True) + + with tab2: + st.header("πŸ“ˆ Analytics") + + # Sentiment Analysis + st.subheader("πŸ˜€ Sentiment Analysis") + + if st.button("Run Sentiment Analysis"): + with st.spinner("Analyzing sentiment..."): + posts_list = posts_df.to_dict('records') + analyzed_posts, sentiment_counts = analyze_posts_sentiment(posts_list) + + col1, col2, col3 = st.columns(3) + col1.metric("Positive", sentiment_counts['positive'], delta=None) + col2.metric("Neutral", sentiment_counts['neutral'], delta=None) + col3.metric("Negative", sentiment_counts['negative'], delta=None) + + # Pie chart + sentiment_df = pd.DataFrame({ + 'Sentiment': ['Positive', 'Neutral', 'Negative'], + 'Count': [sentiment_counts['positive'], sentiment_counts['neutral'], sentiment_counts['negative']] + }) + st.bar_chart(sentiment_df.set_index('Sentiment')) + + st.divider() + + # Keywords + st.subheader("☁️ Top Keywords") + texts = posts_df['title'].tolist() + if 'selftext' in posts_df: + texts.extend(posts_df['selftext'].dropna().tolist()) + + keywords = extract_keywords(texts, top_n=30) + + if keywords: + kw_df = pd.DataFrame(keywords, columns=['Word', 'Count']) + st.bar_chart(kw_df.set_index('Word').head(20)) + + st.divider() + + # Best posting times + st.subheader("⏰ Best Posting Times") + + if 'created_utc' in posts_df: + timing_data = find_best_posting_times(posts_df.to_dict('records')) + + if timing_data['best_hours']: + st.write("**Best Hours to Post:**") + for hour, avg_score in timing_data['best_hours']: + st.write(f"β€’ {hour}:00 - Avg Score: {avg_score:.1f}") + + if timing_data['best_days']: + st.write("**Best Days to Post:**") + for day, avg_score in timing_data['best_days']: + st.write(f"β€’ {day} - Avg Score: {avg_score:.1f}") + + with tab3: + st.header("πŸ” Search Posts") + + # Search form + col1, col2 = st.columns([3, 1]) + + with col1: + search_query = st.text_input("Search query", placeholder="Enter keywords...") + + with col2: + min_score = st.number_input("Min Score", min_value=0, value=0) + + col3, col4, col5 = st.columns(3) + + with col3: + if 'post_type' in posts_df: + post_types = ['All'] + posts_df['post_type'].dropna().unique().tolist() + selected_type = st.selectbox("Post Type", post_types) + + with col4: + if 'author' in posts_df: + authors = ['All'] + posts_df['author'].dropna().unique().tolist()[:50] + selected_author = st.selectbox("Author", authors) + + with col5: + sort_by = st.selectbox("Sort by", ['score', 'num_comments', 'created_utc']) + + # Search button + if st.button("πŸ” Search"): + filtered = posts_df.copy() + + if search_query: + mask = filtered['title'].str.contains(search_query, case=False, na=False) + if 'selftext' in filtered: + mask |= filtered['selftext'].str.contains(search_query, case=False, na=False) + filtered = filtered[mask] + + if min_score > 0: + filtered = filtered[filtered['score'] >= min_score] + + if selected_type != 'All' and 'post_type' in filtered: + filtered = filtered[filtered['post_type'] == selected_type] + + if selected_author != 'All' and 'author' in filtered: + filtered = filtered[filtered['author'] == selected_author] + + filtered = filtered.sort_values(sort_by, ascending=False) + + st.write(f"Found {len(filtered)} results") + st.dataframe(filtered[['title', 'score', 'num_comments', 'post_type', 'author', 'created_utc']].head(50), use_container_width=True) + + with tab4: + st.header("πŸ’¬ Comments Analysis") + + if len(comments_df) == 0: + st.warning("No comments data found for this subreddit") + else: + col1, col2, col3 = st.columns(3) + + with col1: + st.metric("Total Comments", len(comments_df)) + with col2: + avg_score = comments_df['score'].mean() if 'score' in comments_df else 0 + st.metric("Avg Score", f"{avg_score:.1f}") + with col3: + unique_authors = comments_df['author'].nunique() if 'author' in comments_df else 0 + st.metric("Unique Commenters", unique_authors) + + st.divider() + + # Top comments + st.subheader("πŸ”₯ Top Comments by Score") + if 'score' in comments_df: + top_comments = comments_df.nlargest(10, 'score')[['body', 'score', 'author', 'created_utc']] + for _, row in top_comments.iterrows(): + with st.expander(f"⬆️ {row['score']} - by u/{row['author']}"): + st.write(row['body'][:500]) + + st.divider() + + # Top commenters + st.subheader("πŸ‘₯ Top Commenters") + if 'author' in comments_df: + top_authors = comments_df['author'].value_counts().head(10) + st.bar_chart(top_authors) + + with tab5: + st.header("βš™οΈ Scraper Controls") + + st.subheader("πŸš€ Start New Scrape") + + col1, col2 = st.columns(2) + + with col1: + new_sub = st.text_input("Subreddit/User name", placeholder="e.g. python") + is_user = st.checkbox("Is a User (not subreddit)") + + with col2: + limit = st.number_input("Post Limit", min_value=10, max_value=5000, value=100) + mode = st.selectbox("Mode", ['full', 'history']) + + no_media = st.checkbox("Skip media download") + no_comments = st.checkbox("Skip comments") + + if st.button("πŸš€ Start Scraping"): + st.info(f"Run this command in terminal:") + cmd = f"python main.py {new_sub} --mode {mode} --limit {limit}" + if is_user: + cmd += " --user" + if no_media: + cmd += " --no-media" + if no_comments: + cmd += " --no-comments" + st.code(cmd) + + st.divider() + + # Export options + st.subheader("πŸ“€ Export Data") + + export_format = st.selectbox("Format", ['CSV', 'JSON', 'Excel']) + + if st.button("πŸ“₯ Download Posts"): + if export_format == 'CSV': + csv = posts_df.to_csv(index=False) + st.download_button( + "Download CSV", + csv, + f"{selected_sub}_posts.csv", + "text/csv" + ) + elif export_format == 'JSON': + json_data = posts_df.to_json(orient='records', indent=2) + st.download_button( + "Download JSON", + json_data, + f"{selected_sub}_posts.json", + "application/json" + ) + +if __name__ == "__main__": + main() diff --git a/export/__init__.py b/export/__init__.py new file mode 100644 index 0000000..bbc6a13 --- /dev/null +++ b/export/__init__.py @@ -0,0 +1,2 @@ +# Export module +from .database import * diff --git a/export/database.py b/export/database.py new file mode 100644 index 0000000..4a1f504 --- /dev/null +++ b/export/database.py @@ -0,0 +1,389 @@ +""" +Database module - SQLite storage for scraped data +""" +import sqlite3 +from pathlib import Path +from datetime import datetime +import json +import sys +sys.path.insert(0, str(Path(__file__).parent.parent)) +from config import DB_PATH, DATA_DIR + +def get_connection(): + """Get database connection.""" + DATA_DIR.mkdir(exist_ok=True) + conn = sqlite3.connect(DB_PATH) + conn.row_factory = sqlite3.Row + return conn + +def init_database(): + """Initialize database tables.""" + conn = get_connection() + cursor = conn.cursor() + + # Posts table + cursor.execute(""" + CREATE TABLE IF NOT EXISTS posts ( + id TEXT PRIMARY KEY, + subreddit TEXT, + title TEXT, + author TEXT, + created_utc TEXT, + permalink TEXT UNIQUE, + url TEXT, + score INTEGER DEFAULT 0, + upvote_ratio REAL DEFAULT 0, + num_comments INTEGER DEFAULT 0, + num_crossposts INTEGER DEFAULT 0, + selftext TEXT, + post_type TEXT, + is_nsfw BOOLEAN DEFAULT 0, + is_spoiler BOOLEAN DEFAULT 0, + flair TEXT, + total_awards INTEGER DEFAULT 0, + has_media BOOLEAN DEFAULT 0, + media_downloaded BOOLEAN DEFAULT 0, + source TEXT, + scraped_at TEXT DEFAULT CURRENT_TIMESTAMP, + sentiment_score REAL, + sentiment_label TEXT + ) + """) + + # Comments table + cursor.execute(""" + CREATE TABLE IF NOT EXISTS comments ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + comment_id TEXT UNIQUE, + post_id TEXT, + post_permalink TEXT, + parent_id TEXT, + author TEXT, + body TEXT, + score INTEGER DEFAULT 0, + created_utc TEXT, + depth INTEGER DEFAULT 0, + is_submitter BOOLEAN DEFAULT 0, + scraped_at TEXT DEFAULT CURRENT_TIMESTAMP, + sentiment_score REAL, + sentiment_label TEXT, + FOREIGN KEY (post_id) REFERENCES posts(id) + ) + """) + + # Subreddits table (for tracking) + cursor.execute(""" + CREATE TABLE IF NOT EXISTS subreddits ( + name TEXT PRIMARY KEY, + last_scraped TEXT, + total_posts INTEGER DEFAULT 0, + total_comments INTEGER DEFAULT 0, + total_media INTEGER DEFAULT 0 + ) + """) + + # Scheduled jobs table + cursor.execute(""" + CREATE TABLE IF NOT EXISTS scheduled_jobs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + target TEXT, + is_user BOOLEAN DEFAULT 0, + mode TEXT DEFAULT 'full', + limit_posts INTEGER DEFAULT 100, + cron_expression TEXT, + last_run TEXT, + next_run TEXT, + enabled BOOLEAN DEFAULT 1, + created_at TEXT DEFAULT CURRENT_TIMESTAMP + ) + """) + + # Alerts table + cursor.execute(""" + CREATE TABLE IF NOT EXISTS alerts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + keyword TEXT, + subreddit TEXT, + alert_type TEXT DEFAULT 'discord', + webhook_url TEXT, + enabled BOOLEAN DEFAULT 1, + last_triggered TEXT, + created_at TEXT DEFAULT CURRENT_TIMESTAMP + ) + """) + + # Create indexes + cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON posts(subreddit)") + cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_created ON posts(created_utc)") + cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_score ON posts(score)") + cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_post ON comments(post_id)") + cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_author ON comments(author)") + + conn.commit() + conn.close() + print("βœ… Database initialized") + +def save_post(post_data, subreddit): + """Save a single post to database.""" + conn = get_connection() + cursor = conn.cursor() + + try: + cursor.execute(""" + INSERT OR REPLACE INTO posts + (id, subreddit, title, author, created_utc, permalink, url, score, + upvote_ratio, num_comments, num_crossposts, selftext, post_type, + is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, ( + post_data.get('id'), + subreddit, + post_data.get('title'), + post_data.get('author'), + post_data.get('created_utc'), + post_data.get('permalink'), + post_data.get('url'), + post_data.get('score', 0), + post_data.get('upvote_ratio', 0), + post_data.get('num_comments', 0), + post_data.get('num_crossposts', 0), + post_data.get('selftext', ''), + post_data.get('post_type'), + post_data.get('is_nsfw', False), + post_data.get('is_spoiler', False), + post_data.get('flair', ''), + post_data.get('total_awards', 0), + post_data.get('has_media', False), + post_data.get('media_downloaded', False), + post_data.get('source', '') + )) + conn.commit() + return True + except Exception as e: + print(f"DB Error: {e}") + return False + finally: + conn.close() + +def save_posts_batch(posts, subreddit): + """Save multiple posts efficiently.""" + conn = get_connection() + cursor = conn.cursor() + saved = 0 + + for post in posts: + try: + cursor.execute(""" + INSERT OR IGNORE INTO posts + (id, subreddit, title, author, created_utc, permalink, url, score, + upvote_ratio, num_comments, num_crossposts, selftext, post_type, + is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, ( + post.get('id'), + subreddit, + post.get('title'), + post.get('author'), + post.get('created_utc'), + post.get('permalink'), + post.get('url'), + post.get('score', 0), + post.get('upvote_ratio', 0), + post.get('num_comments', 0), + post.get('num_crossposts', 0), + post.get('selftext', ''), + post.get('post_type'), + post.get('is_nsfw', False), + post.get('is_spoiler', False), + post.get('flair', ''), + post.get('total_awards', 0), + post.get('has_media', False), + post.get('media_downloaded', False), + post.get('source', '') + )) + if cursor.rowcount > 0: + saved += 1 + except: + continue + + conn.commit() + conn.close() + return saved + +def save_comments_batch(comments, post_id): + """Save multiple comments efficiently.""" + conn = get_connection() + cursor = conn.cursor() + saved = 0 + + for comment in comments: + try: + cursor.execute(""" + INSERT OR IGNORE INTO comments + (comment_id, post_id, post_permalink, parent_id, author, body, + score, created_utc, depth, is_submitter) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, ( + comment.get('comment_id'), + post_id, + comment.get('post_permalink'), + comment.get('parent_id'), + comment.get('author'), + comment.get('body'), + comment.get('score', 0), + comment.get('created_utc'), + comment.get('depth', 0), + comment.get('is_submitter', False) + )) + if cursor.rowcount > 0: + saved += 1 + except: + continue + + conn.commit() + conn.close() + return saved + +def search_posts(query=None, subreddit=None, author=None, min_score=None, + start_date=None, end_date=None, post_type=None, limit=100): + """Search posts with filters.""" + conn = get_connection() + cursor = conn.cursor() + + sql = "SELECT * FROM posts WHERE 1=1" + params = [] + + if query: + sql += " AND (title LIKE ? OR selftext LIKE ?)" + params.extend([f"%{query}%", f"%{query}%"]) + + if subreddit: + sql += " AND subreddit = ?" + params.append(subreddit) + + if author: + sql += " AND author = ?" + params.append(author) + + if min_score: + sql += " AND score >= ?" + params.append(min_score) + + if start_date: + sql += " AND created_utc >= ?" + params.append(start_date) + + if end_date: + sql += " AND created_utc <= ?" + params.append(end_date) + + if post_type: + sql += " AND post_type = ?" + params.append(post_type) + + sql += " ORDER BY created_utc DESC LIMIT ?" + params.append(limit) + + cursor.execute(sql, params) + results = [dict(row) for row in cursor.fetchall()] + conn.close() + return results + +def search_comments(query=None, post_id=None, author=None, min_score=None, limit=100): + """Search comments with filters.""" + conn = get_connection() + cursor = conn.cursor() + + sql = "SELECT * FROM comments WHERE 1=1" + params = [] + + if query: + sql += " AND body LIKE ?" + params.append(f"%{query}%") + + if post_id: + sql += " AND post_id = ?" + params.append(post_id) + + if author: + sql += " AND author = ?" + params.append(author) + + if min_score: + sql += " AND score >= ?" + params.append(min_score) + + sql += " ORDER BY score DESC LIMIT ?" + params.append(limit) + + cursor.execute(sql, params) + results = [dict(row) for row in cursor.fetchall()] + conn.close() + return results + +def get_subreddit_stats(subreddit): + """Get statistics for a subreddit.""" + conn = get_connection() + cursor = conn.cursor() + + stats = {} + + # Post stats + cursor.execute(""" + SELECT + COUNT(*) as total_posts, + AVG(score) as avg_score, + MAX(score) as max_score, + SUM(num_comments) as total_comments, + AVG(upvote_ratio) as avg_upvote_ratio + FROM posts WHERE subreddit = ? + """, (subreddit,)) + row = cursor.fetchone() + if row: + stats.update(dict(row)) + + # Post type distribution + cursor.execute(""" + SELECT post_type, COUNT(*) as count + FROM posts WHERE subreddit = ? + GROUP BY post_type + """, (subreddit,)) + stats['post_types'] = {row['post_type']: row['count'] for row in cursor.fetchall()} + + # Top authors + cursor.execute(""" + SELECT author, COUNT(*) as post_count, SUM(score) as total_score + FROM posts WHERE subreddit = ? AND author != '[deleted]' + GROUP BY author ORDER BY post_count DESC LIMIT 10 + """, (subreddit,)) + stats['top_authors'] = [dict(row) for row in cursor.fetchall()] + + # Activity by hour + cursor.execute(""" + SELECT strftime('%H', created_utc) as hour, COUNT(*) as count + FROM posts WHERE subreddit = ? + GROUP BY hour ORDER BY hour + """, (subreddit,)) + stats['hourly_activity'] = {row['hour']: row['count'] for row in cursor.fetchall()} + + conn.close() + return stats + +def get_all_subreddits(): + """Get list of all scraped subreddits.""" + conn = get_connection() + cursor = conn.cursor() + + cursor.execute(""" + SELECT subreddit, COUNT(*) as post_count, + MAX(created_utc) as latest_post, + MIN(created_utc) as oldest_post + FROM posts GROUP BY subreddit ORDER BY post_count DESC + """) + + results = [dict(row) for row in cursor.fetchall()] + conn.close() + return results + +# Initialize on import +init_database() diff --git a/main.py b/main.py index 6f314fc..a02f964 100644 --- a/main.py +++ b/main.py @@ -1,3 +1,7 @@ +""" +πŸ€– Universal Reddit Scraper Suite +Full-featured scraper with analytics, dashboard, notifications, and scheduling. +""" import requests import pandas as pd import datetime @@ -8,14 +12,12 @@ import argparse import random import sys import json -import re from urllib.parse import urlparse -from concurrent.futures import ThreadPoolExecutor +from pathlib import Path # --- CONFIGURATION --- USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -# Sources: old.reddit.com for residential IPs, mirrors for data centers MIRRORS = [ "https://old.reddit.com", "https://redlib.catsarch.com", @@ -108,16 +110,13 @@ def get_media_urls(post_data): """Extracts all media URLs from a post.""" media = {"images": [], "videos": [], "galleries": []} - # Direct image link url = post_data.get('url', '') if any(ext in url.lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp']): media["images"].append(url) - # Reddit-hosted image if 'i.redd.it' in url: media["images"].append(url) - # Reddit video if post_data.get('is_video'): reddit_video = post_data.get('media', {}) if reddit_video and 'reddit_video' in reddit_video: @@ -125,17 +124,14 @@ def get_media_urls(post_data): if video_url: media["videos"].append(video_url.split('?')[0]) - # Preview images preview = post_data.get('preview', {}) if preview and 'images' in preview: for img in preview['images']: source = img.get('source', {}) if source.get('url'): - # Unescape HTML entities clean_url = source['url'].replace('&', '&') media["images"].append(clean_url) - # Gallery posts if post_data.get('is_gallery'): gallery_data = post_data.get('gallery_data', {}) media_metadata = post_data.get('media_metadata', {}) @@ -149,7 +145,6 @@ def get_media_urls(post_data): clean_url = meta['s']['u'].replace('&', '&') media["galleries"].append(clean_url) - # External video (YouTube, etc.) if 'youtube.com' in url or 'youtu.be' in url: media["videos"].append(url) @@ -158,7 +153,6 @@ def get_media_urls(post_data): def download_media(url, save_path, media_type="image"): """Downloads a single media file.""" try: - # Skip if already downloaded if os.path.exists(save_path): return True @@ -169,7 +163,7 @@ def download_media(url, save_path, media_type="image"): f.write(chunk) return True except Exception as e: - print(f"⚠️ Failed to download {media_type}: {e}") + pass return False def download_post_media(post_data, dirs, post_id): @@ -177,23 +171,20 @@ def download_post_media(post_data, dirs, post_id): media = get_media_urls(post_data) downloaded = {"images": 0, "videos": 0} - # Download images - for i, img_url in enumerate(media["images"][:5]): # Limit to 5 images per post + for i, img_url in enumerate(media["images"][:5]): ext = os.path.splitext(urlparse(img_url).path)[1] or '.jpg' save_path = os.path.join(dirs["images"], f"{post_id}_{i}{ext}") if download_media(img_url, save_path, "image"): downloaded["images"] += 1 - # Download gallery images - for i, img_url in enumerate(media["galleries"][:10]): # Limit gallery to 10 + for i, img_url in enumerate(media["galleries"][:10]): ext = '.jpg' save_path = os.path.join(dirs["images"], f"{post_id}_gallery_{i}{ext}") if download_media(img_url, save_path, "gallery"): downloaded["images"] += 1 - # Download videos - for i, vid_url in enumerate(media["videos"][:2]): # Limit to 2 videos - if 'youtube' not in vid_url: # Skip YouTube (can't direct download) + for i, vid_url in enumerate(media["videos"][:2]): + if 'youtube' not in vid_url: ext = '.mp4' save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}") if download_media(vid_url, save_path, "video"): @@ -203,11 +194,10 @@ def download_post_media(post_data, dirs, post_id): # --- COMMENT SCRAPING --- def scrape_comments(permalink, max_depth=3): - """Scrapes comments from a post using Reddit JSON endpoint.""" + """Scrapes comments from a post.""" comments = [] try: - # Clean permalink and build URL if not permalink.startswith('http'): url = f"https://old.reddit.com{permalink}.json?limit=100" else: @@ -219,13 +209,12 @@ def scrape_comments(permalink, max_depth=3): data = response.json() - # Comments are in the second element of the response if len(data) > 1: comment_data = data[1]['data']['children'] comments = parse_comments(comment_data, permalink, depth=0, max_depth=max_depth) except Exception as e: - print(f"⚠️ Comment fetch error: {e}") + pass return comments @@ -237,7 +226,7 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3): return comments for item in comment_list: - if item['kind'] != 't1': # Skip non-comment items + if item['kind'] != 't1': continue c = item['data'] @@ -255,7 +244,6 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3): } comments.append(comment) - # Parse replies recursively replies = c.get('replies') if replies and isinstance(replies, dict): reply_children = replies.get('data', {}).get('children', []) @@ -263,12 +251,11 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3): return comments -# --- ENHANCED POST EXTRACTION --- +# --- POST EXTRACTION --- def extract_post_data(post_json): """Extracts comprehensive post data.""" p = post_json - # Determine post type post_type = "text" if p.get('is_video'): post_type = "video" @@ -282,39 +269,28 @@ def extract_post_data(post_json): post_type = "link" return { - # Basic Info "id": p.get('id'), "title": p.get('title'), "author": p.get('author'), "created_utc": datetime.datetime.fromtimestamp(p.get('created_utc', 0)).isoformat(), "permalink": p.get('permalink'), "url": p.get('url_overridden_by_dest', p.get('url')), - - # Engagement "score": p.get('score', 0), "upvote_ratio": p.get('upvote_ratio', 0), "num_comments": p.get('num_comments', 0), "num_crossposts": p.get('num_crossposts', 0), - - # Content "selftext": p.get('selftext', ''), "post_type": post_type, "is_nsfw": p.get('over_18', False), "is_spoiler": p.get('spoiler', False), - - # Flair & Awards "flair": p.get('link_flair_text', ''), "total_awards": p.get('total_awards_received', 0), - - # Media flags "has_media": p.get('is_video', False) or p.get('is_gallery', False) or 'i.redd.it' in p.get('url', ''), "media_downloaded": False, - - # Source tracking "source": "History-Full" } -# --- MODE 2: FULL HISTORY SCRAPE --- +# --- FULL HISTORY SCRAPE --- def run_full_history(target, limit, is_user=False, download_media_flag=True, scrape_comments_flag=True): """Full scrape with images, videos, and comments.""" prefix = "u" if is_user else "r" @@ -331,6 +307,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr total_posts = 0 total_media = {"images": 0, "videos": 0} total_comments = 0 + start_time = time.time() while total_posts < limit: random.shuffle(MIRRORS) @@ -362,11 +339,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr p = child['data'] post = extract_post_data(p) - # Skip if already seen if post['permalink'] in SEEN_URLS: continue - # Download media if download_media_flag: downloaded = download_post_media(p, dirs, post['id']) post['media_downloaded'] = downloaded['images'] > 0 or downloaded['videos'] > 0 @@ -375,22 +350,19 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr posts.append(post) - # Scrape comments if scrape_comments_flag and post['num_comments'] > 0: print(f" πŸ’¬ Fetching comments for: {post['title'][:40]}...") comments = scrape_comments(post['permalink']) all_comments.extend(comments) total_comments += len(comments) - time.sleep(1) # Rate limiting for comment fetches + time.sleep(1) - # Save data saved = save_posts_csv(posts, dirs["posts"]) total_posts += saved if all_comments: save_comments_csv(all_comments, dirs["comments"]) - # Progress update print(f"\nπŸ“Š Progress: {total_posts}/{limit} posts") print(f" πŸ–ΌοΈ Images: {total_media['images']} | 🎬 Videos: {total_media['videos']}") print(f" πŸ’¬ Comments: {total_comments}") @@ -398,7 +370,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr after = data['data'].get('after') if not after: print("\n🏁 Reached end of available history.") - return + break success = True break @@ -407,6 +379,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr print(f" ⚠️ Error with {base_url}: {e}") continue + if not after: + break + if not success: print("\n❌ All sources failed. Waiting 30s...") time.sleep(30) @@ -414,6 +389,8 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr print(f"\n⏸️ Cooling down (3s)...") time.sleep(3) + duration = time.time() - start_time + print("\n" + "=" * 50) print("βœ… SCRAPE COMPLETE!") print(f" πŸ“ Data saved to: {dirs['base']}") @@ -421,8 +398,17 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr print(f" πŸ–ΌοΈ Total images: {total_media['images']}") print(f" 🎬 Total videos: {total_media['videos']}") print(f" πŸ’¬ Total comments: {total_comments}") + print(f" ⏱️ Duration: {duration:.1f}s") + + return { + 'posts': total_posts, + 'images': total_media['images'], + 'videos': total_media['videos'], + 'comments': total_comments, + 'duration': f"{duration:.1f}s" + } -# --- MODE 1: LIVE MONITOR (RSS) - Legacy --- +# --- MONITOR MODE --- def run_monitor(target, is_user=False): prefix = "u" if is_user else "r" if is_user: @@ -437,7 +423,6 @@ def run_monitor(target, is_user=False): if response.status_code != 200: print(f"❌ RSS blocked (Status {response.status_code}), trying JSON...") - # Fallback to JSON run_full_history(target, 25, is_user, download_media_flag=False, scrape_comments_flag=False) return @@ -474,33 +459,144 @@ def run_monitor(target, is_user=False): except Exception as e: print(f"❌ Monitor Error: {e}") -# --- CLI ARGS --- -if __name__ == "__main__": +# --- CLI --- +def main(): parser = argparse.ArgumentParser( - description="πŸ€– Universal Reddit Scraper - Full Media & Comments Support", + description="πŸ€– Universal Reddit Scraper Suite", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" -Examples: - python main.py delhi --mode full --limit 100 - python main.py spez --user --mode full --limit 50 - python main.py python --mode full --no-media --limit 200 - python main.py india --mode monitor +Commands: + SCRAPING: + python main.py --mode full --limit 100 + python main.py --mode history --limit 500 + python main.py --mode monitor + + SEARCH: + python main.py --search "keyword" --subreddit delhi + python main.py --search "keyword" --min-score 100 + + DASHBOARD: + python main.py --dashboard + + SCHEDULE: + python main.py --schedule delhi --every 60 + + ANALYTICS: + python main.py --analyze delhi --sentiment + python main.py --analyze delhi --keywords """ ) - parser.add_argument("target", help="Subreddit name (e.g. 'delhi') or Username (e.g. 'spez')") - parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full", - help="monitor=live RSS, history=posts only, full=posts+media+comments") - parser.add_argument("--user", action="store_true", help="Target is a User, not Subreddit") + + # Scraping args + parser.add_argument("target", nargs='?', help="Subreddit or username to scrape") + parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full") + parser.add_argument("--user", action="store_true", help="Target is a user") parser.add_argument("--limit", type=int, default=100, help="Max posts to scrape") - parser.add_argument("--no-media", action="store_true", help="Skip downloading images/videos") - parser.add_argument("--no-comments", action="store_true", help="Skip scraping comments") + parser.add_argument("--no-media", action="store_true", help="Skip media download") + parser.add_argument("--no-comments", action="store_true", help="Skip comments") + + # Dashboard + parser.add_argument("--dashboard", action="store_true", help="Launch web dashboard") + + # Search + parser.add_argument("--search", type=str, help="Search scraped data") + parser.add_argument("--subreddit", type=str, help="Filter by subreddit") + parser.add_argument("--min-score", type=int, help="Filter by minimum score") + parser.add_argument("--author", type=str, help="Filter by author") + + # Analytics + parser.add_argument("--analyze", type=str, help="Run analytics on subreddit") + parser.add_argument("--sentiment", action="store_true", help="Run sentiment analysis") + parser.add_argument("--keywords", action="store_true", help="Extract keywords") + + # Schedule + parser.add_argument("--schedule", type=str, help="Schedule scraping for target") + parser.add_argument("--every", type=int, help="Interval in minutes") + + # Alerts + parser.add_argument("--alert", type=str, help="Set keyword alert") + parser.add_argument("--discord-webhook", type=str, help="Discord webhook URL") + parser.add_argument("--telegram-token", type=str, help="Telegram bot token") + parser.add_argument("--telegram-chat", type=str, help="Telegram chat ID") args = parser.parse_args() print("=" * 50) - print("πŸ€– UNIVERSAL REDDIT SCRAPER") + print("πŸ€– UNIVERSAL REDDIT SCRAPER SUITE") print("=" * 50) + # Dashboard mode + if args.dashboard: + print("\n🌐 Launching Dashboard...") + print(" Open: http://localhost:8501") + os.system("streamlit run dashboard/app.py") + return + + # Search mode + if args.search: + print(f"\nπŸ” Searching for: {args.search}") + from search.query import search_all_data, print_search_results + + results = search_all_data( + query=args.search, + min_score=args.min_score, + author=args.author + ) + print_search_results(results) + return + + # Analytics mode + if args.analyze: + print(f"\nπŸ“Š Analyzing: {args.analyze}") + + # Load data + data_dir = Path(f"data/r_{args.analyze}") + if not data_dir.exists(): + print(f"❌ No data found for r/{args.analyze}") + return + + posts_file = data_dir / "posts.csv" + if not posts_file.exists(): + print(f"❌ No posts data found") + return + + import pandas as pd + df = pd.read_csv(posts_file) + posts = df.to_dict('records') + + if args.sentiment: + from analytics.sentiment import analyze_posts_sentiment + analyzed, counts = analyze_posts_sentiment(posts) + print(f"\nπŸ˜€ Sentiment Analysis:") + print(f" Positive: {counts['positive']}") + print(f" Neutral: {counts['neutral']}") + print(f" Negative: {counts['negative']}") + + if args.keywords: + from analytics.sentiment import extract_keywords + texts = [str(p.get('title', '') or '') + ' ' + str(p.get('selftext', '') or '') for p in posts] + keywords = extract_keywords(texts, top_n=20) + print(f"\n☁️ Top Keywords:") + for word, count in keywords: + print(f" {word}: {count}") + + return + + # Schedule mode + if args.schedule: + if not args.every: + print("❌ Please specify --every ") + return + + from scheduler.cron import run_scheduled + run_scheduled(args.schedule, args.every, args.mode, args.limit, args.user) + return + + # Regular scraping mode + if not args.target: + parser.print_help() + return + if args.mode == "monitor": prefix = "u" if args.user else "r" dirs = setup_directories(args.target, prefix) @@ -510,10 +606,12 @@ Examples: run_monitor(args.target, args.user) time.sleep(300) elif args.mode == "history": - # Legacy mode - posts only run_full_history(args.target, args.limit, args.user, download_media_flag=False, scrape_comments_flag=False) - else: # full mode + else: run_full_history(args.target, args.limit, args.user, download_media_flag=not args.no_media, scrape_comments_flag=not args.no_comments) + +if __name__ == "__main__": + main() diff --git a/requirements.txt b/requirements.txt index 69de461..a713acf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,9 @@ +# Core pandas requests + +# Dashboard +streamlit + +# Export +openpyxl diff --git a/scheduler/__init__.py b/scheduler/__init__.py new file mode 100644 index 0000000..1652503 --- /dev/null +++ b/scheduler/__init__.py @@ -0,0 +1,2 @@ +# Scheduler module +from .cron import * diff --git a/scheduler/cron.py b/scheduler/cron.py new file mode 100644 index 0000000..328ebd3 --- /dev/null +++ b/scheduler/cron.py @@ -0,0 +1,215 @@ +""" +Scheduler module - Cron-style scheduling for scrape jobs +""" +import time +import threading +from datetime import datetime, timedelta +import json +from pathlib import Path +import sys + +class CronScheduler: + """Simple cron-style scheduler for Reddit scraping jobs.""" + + def __init__(self): + self.jobs = [] + self.running = False + self.thread = None + + def add_job(self, target, mode='full', limit=100, is_user=False, + interval_minutes=60, run_at_start=True): + """ + Add a scheduled scraping job. + + Args: + target: Subreddit or username + mode: 'full', 'history', or 'monitor' + limit: Post limit per run + is_user: True if target is a user + interval_minutes: Minutes between runs + run_at_start: Run immediately when scheduler starts + """ + job = { + 'id': len(self.jobs) + 1, + 'target': target, + 'mode': mode, + 'limit': limit, + 'is_user': is_user, + 'interval_minutes': interval_minutes, + 'run_at_start': run_at_start, + 'last_run': None, + 'next_run': datetime.now() if run_at_start else datetime.now() + timedelta(minutes=interval_minutes), + 'enabled': True, + 'run_count': 0 + } + self.jobs.append(job) + print(f"πŸ“… Added job #{job['id']}: {'u/' if is_user else 'r/'}{target} every {interval_minutes}min") + return job['id'] + + def remove_job(self, job_id): + """Remove a scheduled job.""" + self.jobs = [j for j in self.jobs if j['id'] != job_id] + print(f"πŸ—‘οΈ Removed job #{job_id}") + + def disable_job(self, job_id): + """Temporarily disable a job.""" + for job in self.jobs: + if job['id'] == job_id: + job['enabled'] = False + print(f"⏸️ Disabled job #{job_id}") + + def enable_job(self, job_id): + """Enable a disabled job.""" + for job in self.jobs: + if job['id'] == job_id: + job['enabled'] = True + print(f"▢️ Enabled job #{job_id}") + + def list_jobs(self): + """List all scheduled jobs.""" + print("\nπŸ“‹ Scheduled Jobs:") + print("-" * 60) + for job in self.jobs: + status = "βœ…" if job['enabled'] else "⏸️" + prefix = "u/" if job['is_user'] else "r/" + next_run = job['next_run'].strftime("%H:%M:%S") if job['next_run'] else "Never" + print(f"{status} #{job['id']} | {prefix}{job['target']} | " + f"Every {job['interval_minutes']}min | Next: {next_run} | " + f"Runs: {job['run_count']}") + print() + return self.jobs + + def _run_job(self, job): + """Execute a single job.""" + # Import here to avoid circular imports + try: + from main import run_full_history + + prefix = "u/" if job['is_user'] else "r/" + print(f"\nπŸš€ Running scheduled job: {prefix}{job['target']}") + + run_full_history( + job['target'], + job['limit'], + job['is_user'], + download_media_flag=(job['mode'] == 'full'), + scrape_comments_flag=(job['mode'] == 'full') + ) + + job['last_run'] = datetime.now() + job['run_count'] += 1 + print(f"βœ… Job completed: {prefix}{job['target']}") + + except Exception as e: + print(f"❌ Job failed: {e}") + + def _scheduler_loop(self): + """Main scheduler loop.""" + print("πŸ”„ Scheduler started") + + while self.running: + now = datetime.now() + + for job in self.jobs: + if not job['enabled']: + continue + + if job['next_run'] and now >= job['next_run']: + self._run_job(job) + job['next_run'] = now + timedelta(minutes=job['interval_minutes']) + + # Check every 30 seconds + time.sleep(30) + + print("πŸ›‘ Scheduler stopped") + + def start(self): + """Start the scheduler in background.""" + if self.running: + print("⚠️ Scheduler already running") + return + + self.running = True + self.thread = threading.Thread(target=self._scheduler_loop, daemon=True) + self.thread.start() + print("βœ… Scheduler started in background") + + def stop(self): + """Stop the scheduler.""" + self.running = False + if self.thread: + self.thread.join(timeout=5) + print("πŸ›‘ Scheduler stopped") + + def save_jobs(self, filepath='scheduler_jobs.json'): + """Save jobs to file.""" + jobs_data = [] + for job in self.jobs: + job_copy = job.copy() + job_copy['last_run'] = job_copy['last_run'].isoformat() if job_copy['last_run'] else None + job_copy['next_run'] = job_copy['next_run'].isoformat() if job_copy['next_run'] else None + jobs_data.append(job_copy) + + with open(filepath, 'w') as f: + json.dump(jobs_data, f, indent=2) + print(f"πŸ’Ύ Saved {len(self.jobs)} jobs to {filepath}") + + def load_jobs(self, filepath='scheduler_jobs.json'): + """Load jobs from file.""" + if not Path(filepath).exists(): + print("⚠️ No saved jobs found") + return + + with open(filepath, 'r') as f: + jobs_data = json.load(f) + + for job_data in jobs_data: + if job_data['last_run']: + job_data['last_run'] = datetime.fromisoformat(job_data['last_run']) + if job_data['next_run']: + job_data['next_run'] = datetime.fromisoformat(job_data['next_run']) + self.jobs.append(job_data) + + print(f"πŸ“‚ Loaded {len(jobs_data)} jobs from {filepath}") + + +# Simple interval-based scheduler for CLI +def run_scheduled(target, interval_minutes, mode='full', limit=100, is_user=False): + """ + Run a scrape job on a schedule. + + Args: + target: Subreddit or username + interval_minutes: Minutes between runs + mode: 'full', 'history', or 'monitor' + limit: Post limit per run + is_user: True if target is a user + """ + from main import run_full_history + + prefix = "u/" if is_user else "r/" + print(f"πŸ“… Scheduled: {prefix}{target} every {interval_minutes} minutes") + print("Press Ctrl+C to stop\n") + + run_count = 0 + + try: + while True: + run_count += 1 + print(f"\n{'='*50}") + print(f"πŸ”„ Run #{run_count} - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") + print(f"{'='*50}") + + run_full_history( + target, + limit, + is_user, + download_media_flag=(mode == 'full'), + scrape_comments_flag=(mode == 'full') + ) + + print(f"\n⏰ Next run in {interval_minutes} minutes...") + time.sleep(interval_minutes * 60) + + except KeyboardInterrupt: + print(f"\n\nπŸ›‘ Scheduler stopped after {run_count} runs") diff --git a/search/__init__.py b/search/__init__.py new file mode 100644 index 0000000..8d82644 --- /dev/null +++ b/search/__init__.py @@ -0,0 +1,2 @@ +# Search module +from .query import * diff --git a/search/query.py b/search/query.py new file mode 100644 index 0000000..6017706 --- /dev/null +++ b/search/query.py @@ -0,0 +1,220 @@ +""" +Search & Query module - Search and filter scraped data +""" +import pandas as pd +from pathlib import Path +from datetime import datetime +import re + +def search_csv(filepath, query=None, column=None, min_score=None, max_score=None, + start_date=None, end_date=None, post_type=None, author=None, limit=50): + """ + Search within a CSV file with various filters. + + Args: + filepath: Path to CSV file + query: Text to search for + column: Specific column to search in (default: all text columns) + min_score: Minimum score filter + max_score: Maximum score filter + start_date: Start date (YYYY-MM-DD) + end_date: End date (YYYY-MM-DD) + post_type: Filter by post type (image, video, text, etc.) + author: Filter by author + limit: Maximum results to return + + Returns: + DataFrame with matching results + """ + if not Path(filepath).exists(): + print(f"❌ File not found: {filepath}") + return pd.DataFrame() + + df = pd.read_csv(filepath) + + # Text search + if query: + if column and column in df.columns: + mask = df[column].astype(str).str.contains(query, case=False, na=False) + else: + # Search in all text columns + text_cols = ['title', 'selftext', 'body'] + mask = pd.Series([False] * len(df)) + for col in text_cols: + if col in df.columns: + mask |= df[col].astype(str).str.contains(query, case=False, na=False) + df = df[mask] + + # Score filter + if min_score is not None and 'score' in df.columns: + df = df[df['score'] >= min_score] + if max_score is not None and 'score' in df.columns: + df = df[df['score'] <= max_score] + + # Date filter + if 'created_utc' in df.columns: + if start_date: + df = df[df['created_utc'] >= start_date] + if end_date: + df = df[df['created_utc'] <= end_date] + + # Post type filter + if post_type and 'post_type' in df.columns: + df = df[df['post_type'] == post_type] + + # Author filter + if author and 'author' in df.columns: + df = df[df['author'] == author] + + return df.head(limit) + +def search_all_data(data_dir='data', query=None, **kwargs): + """ + Search across all scraped data. + + Args: + data_dir: Data directory path + query: Text to search for + **kwargs: Additional filters passed to search_csv + + Returns: + Dictionary with results from each subreddit + """ + results = {} + data_path = Path(data_dir) + + if not data_path.exists(): + print(f"❌ Data directory not found: {data_dir}") + return results + + # Find all posts.csv files + for sub_dir in data_path.iterdir(): + if sub_dir.is_dir(): + posts_file = sub_dir / 'posts.csv' + if posts_file.exists(): + df = search_csv(str(posts_file), query=query, **kwargs) + if len(df) > 0: + results[sub_dir.name] = df + + # Also check legacy format + for csv_file in data_path.glob('*.csv'): + if csv_file.stem not in [r.replace('r_', '').replace('u_', '') for r in results.keys()]: + df = search_csv(str(csv_file), query=query, **kwargs) + if len(df) > 0: + results[csv_file.stem] = df + + return results + +def print_search_results(results, show_preview=True): + """Pretty print search results.""" + total = sum(len(df) for df in results.values()) + + print(f"\nπŸ” Found {total} results across {len(results)} sources\n") + print("=" * 70) + + for source, df in results.items(): + print(f"\nπŸ“ {source} ({len(df)} matches)") + print("-" * 50) + + for _, row in df.iterrows(): + title = str(row.get('title', row.get('body', 'N/A')))[:60] + score = row.get('score', 0) + date = str(row.get('created_utc', ''))[:10] + + print(f" [{score:>4}⬆] {title}...") + if show_preview and 'selftext' in row and row['selftext']: + preview = str(row['selftext'])[:100].replace('\n', ' ') + print(f" └─ {preview}...") + print() + +def advanced_search(data_dir='data', query=None, regex=False, sort_by='score', + ascending=False, **kwargs): + """ + Advanced search with regex support and sorting. + + Args: + data_dir: Data directory path + query: Search query (text or regex pattern) + regex: Treat query as regex pattern + sort_by: Column to sort results by + ascending: Sort ascending (default: descending) + **kwargs: Additional filters + + Returns: + Combined DataFrame of all results + """ + all_results = [] + data_path = Path(data_dir) + + for sub_dir in data_path.iterdir(): + if sub_dir.is_dir(): + posts_file = sub_dir / 'posts.csv' + if posts_file.exists(): + df = pd.read_csv(posts_file) + df['source'] = sub_dir.name + all_results.append(df) + + if not all_results: + return pd.DataFrame() + + combined = pd.concat(all_results, ignore_index=True) + + # Apply query + if query: + if regex: + pattern = query + else: + pattern = re.escape(query) + + mask = pd.Series([False] * len(combined)) + for col in ['title', 'selftext']: + if col in combined.columns: + mask |= combined[col].astype(str).str.contains(pattern, case=False, na=False, regex=True) + combined = combined[mask] + + # Apply other filters + if kwargs.get('min_score') and 'score' in combined.columns: + combined = combined[combined['score'] >= kwargs['min_score']] + + if kwargs.get('author') and 'author' in combined.columns: + combined = combined[combined['author'] == kwargs['author']] + + if kwargs.get('post_type') and 'post_type' in combined.columns: + combined = combined[combined['post_type'] == kwargs['post_type']] + + # Sort + if sort_by in combined.columns: + combined = combined.sort_values(sort_by, ascending=ascending) + + limit = kwargs.get('limit', 100) + return combined.head(limit) + +def get_top_posts(data_dir='data', n=10, by='score'): + """Get top N posts across all scraped data.""" + df = advanced_search(data_dir, sort_by=by, ascending=False, limit=n) + return df + +def get_recent_posts(data_dir='data', n=10): + """Get most recent posts across all scraped data.""" + df = advanced_search(data_dir, sort_by='created_utc', ascending=False, limit=n) + return df + +def find_author_posts(data_dir='data', author=None): + """Find all posts by a specific author.""" + return advanced_search(data_dir, author=author, limit=1000) + +def export_search_results(results, output_path, format='csv'): + """Export search results to file.""" + if isinstance(results, dict): + combined = pd.concat(results.values(), ignore_index=True) + else: + combined = results + + if format == 'csv': + combined.to_csv(output_path, index=False) + elif format == 'json': + combined.to_json(output_path, orient='records', indent=2) + elif format == 'excel': + combined.to_excel(output_path, index=False) + + print(f"πŸ’Ύ Exported {len(combined)} results to {output_path}")