v3.0: Full suite - Dashboard, Analytics, Scheduling, Notifications, Search

README.md

@@ -1,90 +1,148 @@
-# 🤖 Universal Reddit Scraper
+# 🤖 Universal Reddit Scraper Suite
 
 [![Docker Build & Publish](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml/badge.svg)](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml)
 
-A robust, full-featured Reddit scraper that downloads **posts, images, videos, galleries, and comments**. Designed to run on low-resource servers (like AWS Free Tier).
+A **full-featured** Reddit scraper suite with analytics dashboard, sentiment analysis, scheduled scraping, notifications, and more!
-
-## 🐳 Quick Start (No Installation Needed!)
-```bash
-docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100
-```
 
 ## ✨ Features
 
 | Feature | Description |
 |---------|-------------|
-| 📊 **Full Metadata** | Title, author, score, upvotes, awards, flair, NSFW flags |
+| 📊 **Full Scraping** | Posts, comments, images, videos, galleries |
-| 🖼️ **Image Download** | Automatically downloads all images from posts |
+| 📈 **Analytics Dashboard** | Beautiful Streamlit web UI |
-| 🎬 **Video Download** | Downloads Reddit-hosted videos |
+| 😀 **Sentiment Analysis** | Analyze post/comment sentiment |
-| 🖼️ **Gallery Support** | Extracts and downloads all images from gallery posts |
+| ☁️ **Keyword Extraction** | Generate word clouds |
-| 💬 **Comment Scraping** | Recursively scrapes all comments with threading info |
+| 🔍 **Search & Filter** | Query scraped data with filters |
-| 🔄 **Dual Sources** | Uses old.reddit.com + Redlib mirrors for reliability |
+| 📅 **Scheduled Scraping** | Cron-style job scheduling |
-| 📁 **Organized Output** | Clean folder structure per subreddit |
+| 📧 **Notifications** | Discord & Telegram alerts |
+| 🗄️ **SQLite Database** | Structured data storage |
+| 📤 **Multiple Exports** | CSV, JSON, Excel |
 
-## 📁 Output Structure
+## 🚀 Quick Start
 
-```
-data/
-└── r_delhi/
-    ├── posts.csv           # All post metadata
-    ├── comments.csv        # All comments with threading
-    └── media/
-        ├── images/         # Downloaded images & galleries
-        │   ├── abc123_0.jpg
-        │   ├── abc123_gallery_0.jpg
-        │   └── ...
-        └── videos/         # Downloaded videos
-            └── xyz789_0.mp4
-```
-
-## 🚀 Usage
-
-### Full Scrape (Posts + Media + Comments)
 ```bash
-# Scrape r/delhi with everything
+# Install dependencies
+pip install -r requirements.txt
+
+# Scrape a subreddit (posts + media + comments)
 python main.py delhi --mode full --limit 100
 
+# Launch analytics dashboard
+python main.py --dashboard
+```
+
+## 📖 Usage Guide
+
+### 🔄 Scraping Modes
+
+```bash
+# Full scrape with everything
+python main.py delhi --mode full --limit 100
+
+# History only (no media/comments - faster)
+python main.py delhi --mode history --limit 500
+
+# Live monitor (checks every 5 min)
+python main.py delhi --mode monitor
+
 # Scrape a user's posts
 python main.py spez --user --mode full --limit 50
+
+# Skip media or comments
+python main.py delhi --mode full --no-media --limit 200
+python main.py delhi --mode full --no-comments --limit 200
 ```
 
-### Posts Only (No Media Download)
+### 📊 Analytics Dashboard
-```bash
-python main.py python --mode full --no-media --limit 200
-```
-
-### Posts Only (No Comments)
-```bash
-python main.py india --mode full --no-comments --limit 100
-```
-
-### Live Monitor Mode
-```bash
-python main.py delhi --mode monitor
-```
-
-### Legacy History Mode (Posts Only, No Media)
-```bash
-python main.py delhi --mode history --limit 500
-```
-
-## 🐳 Docker Usage
 
 ```bash
-# Build the image
+# Launch the web dashboard
-docker build -t reddit-scraper .
+python main.py --dashboard
 
-# Full scrape with media
+# Opens at http://localhost:8501
-docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100
-
-# Scrape without media (faster)
-docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500
-
-# Monitor mode (runs continuously)
-docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
 ```
 
-## 📊 CSV Output Format
+**Dashboard Features:**
+- 📈 Post statistics & charts
+- 😀 Sentiment analysis
+- ☁️ Keyword extraction
+- 🔍 Search & filter interface
+- 📤 Export data
+
+### 🔍 Search Data
+
+```bash
+# Search all scraped data
+python main.py --search "credit card"
+
+# Search with filters
+python main.py --search "laptop" --min-score 100
+python main.py --search "advice" --author username
+python main.py --search "help" --subreddit delhi
+```
+
+### 😀 Analytics
+
+```bash
+# Run sentiment analysis
+python main.py --analyze delhi --sentiment
+
+# Extract top keywords
+python main.py --analyze delhi --keywords
+```
+
+### 📅 Scheduled Scraping
+
+```bash
+# Scrape every 60 minutes
+python main.py --schedule delhi --every 60
+
+# Scrape with options
+python main.py --schedule delhi --every 30 --mode full --limit 50
+```
+
+### 📧 Notifications (Discord/Telegram)
+
+**Discord:**
+```bash
+python main.py delhi --mode monitor --discord-webhook "YOUR_WEBHOOK_URL"
+```
+
+**Telegram:**
+```bash
+python main.py delhi --mode monitor \
+  --telegram-token "YOUR_BOT_TOKEN" \
+  --telegram-chat "YOUR_CHAT_ID"
+```
+
+## 📁 Project Structure
+
+```
+reddit-scraper/
+├── main.py              # Main CLI entry point
+├── config.py            # Configuration settings
+├── analytics/           # Sentiment & keyword analysis
+│   └── sentiment.py
+├── alerts/              # Discord & Telegram notifications
+│   └── notifications.py
+├── dashboard/           # Streamlit web UI
+│   └── app.py
+├── export/              # Database & export functions
+│   └── database.py
+├── scheduler/           # Cron-style scheduling
+│   └── cron.py
+├── search/              # Search & filter engine
+│   └── query.py
+└── data/                # Scraped data
+    └── r_subreddit/
+        ├── posts.csv
+        ├── comments.csv
+        └── media/
+            ├── images/
+            └── videos/
+```
+
+## 📊 Data Output
 
 ### posts.csv
 | Column | Description |
@@ -92,50 +150,51 @@ docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
 | id | Reddit post ID |
 | title | Post title |
 | author | Username |
-| created_utc | Timestamp (ISO format) |
-| permalink | Reddit URL path |
-| url | External/media URL |
 | score | Net upvotes |
-| upvote_ratio | Percentage upvoted |
 | num_comments | Comment count |
-| selftext | Post body text |
+| post_type | text/image/video/gallery |
-| post_type | text/image/video/gallery/link |
+| selftext | Post body |
-| flair | Post flair text |
+| flair | Post flair |
-| has_media | Boolean |
+| is_nsfw | NSFW flag |
-| media_downloaded | Boolean |
+| created_utc | Timestamp |
 
 ### comments.csv
 | Column | Description |
 |--------|-------------|
-| post_permalink | Parent post URL |
+| comment_id | Comment ID |
-| comment_id | Reddit comment ID |
+| post_permalink | Parent post |
-| parent_id | Parent comment/post ID |
 | author | Username |
 | body | Comment text |
-| score | Net upvotes |
+| score | Upvotes |
-| created_utc | Timestamp |
+| depth | Nesting level |
-| depth | Nesting level (0 = top-level) |
-| is_submitter | Is the post author |
 
-## ⚙️ Command Line Options
+## 🐳 Docker
-
-| Option | Description | Default |
-|--------|-------------|---------|
-| `target` | Subreddit or username | Required |
-| `--mode` | `full`, `history`, or `monitor` | `full` |
-| `--user` | Target is a user, not subreddit | `false` |
-| `--limit` | Max posts to scrape | `100` |
-| `--no-media` | Skip downloading images/videos | `false` |
-| `--no-comments` | Skip scraping comments | `false` |
-
-## 🛠️ Requirements
 
 ```bash
-pip install pandas requests
+# Build
+docker build -t reddit-scraper .
+
+# Full scrape
+docker run -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100
+
+# Monitor mode
+docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
+```
+
+## ⚙️ Configuration
+
+Edit `config.py` or use environment variables:
+
+```bash
+export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
+export TELEGRAM_BOT_TOKEN="123456:ABC..."
+export TELEGRAM_CHAT_ID="987654321"
 ```
 
 ## 📜 License
+
 MIT License - Feel free to use, modify, and distribute.
 
 ## 🤝 Contributing
-Pull requests are welcome! For major changes, please open an issue first.
+
+Pull requests welcome! For major changes, please open an issue first.

alerts/__init__.py

@@ -0,0 +1,2 @@
+# Alerts module
+from .notifications import *

alerts/notifications.py

@@ -0,0 +1,208 @@
+"""
+Notification module - Discord & Telegram alerts
+"""
+import requests
+import json
+from datetime import datetime
+
+def send_discord_alert(webhook_url, title, message, posts=None, color=0x5865F2):
+    """
+    Send alert to Discord via webhook.
+    
+    Args:
+        webhook_url: Discord webhook URL
+        title: Alert title
+        message: Alert message
+        posts: Optional list of posts to include
+        color: Embed color (default: Discord blue)
+    """
+    if not webhook_url:
+        print("⚠️ Discord webhook URL not configured")
+        return False
+    
+    embeds = [{
+        "title": f"🤖 {title}",
+        "description": message,
+        "color": color,
+        "timestamp": datetime.utcnow().isoformat(),
+        "footer": {"text": "Reddit Scraper Alert"}
+    }]
+    
+    # Add post previews
+    if posts:
+        fields = []
+        for post in posts[:5]:  # Max 5 posts
+            fields.append({
+                "name": post.get('title', 'No Title')[:100],
+                "value": f"Score: {post.get('score', 0)} | Comments: {post.get('num_comments', 0)}\n[View Post](https://reddit.com{post.get('permalink', '')})",
+                "inline": False
+            })
+        embeds[0]["fields"] = fields
+    
+    payload = {"embeds": embeds}
+    
+    try:
+        response = requests.post(
+            webhook_url,
+            json=payload,
+            headers={"Content-Type": "application/json"},
+            timeout=10
+        )
+        if response.status_code == 204:
+            print("✅ Discord alert sent!")
+            return True
+        else:
+            print(f"❌ Discord error: {response.status_code}")
+            return False
+    except Exception as e:
+        print(f"❌ Discord error: {e}")
+        return False
+
+def send_telegram_alert(bot_token, chat_id, title, message, posts=None):
+    """
+    Send alert to Telegram via bot.
+    
+    Args:
+        bot_token: Telegram bot token
+        chat_id: Chat/Channel ID to send to
+        title: Alert title
+        message: Alert message
+        posts: Optional list of posts to include
+    """
+    if not bot_token or not chat_id:
+        print("⚠️ Telegram credentials not configured")
+        return False
+    
+    # Build message
+    text = f"🤖 *{title}*\n\n{message}"
+    
+    if posts:
+        text += "\n\n📝 *New Posts:*\n"
+        for post in posts[:5]:
+            title_text = post.get('title', 'No Title')[:80]
+            score = post.get('score', 0)
+            permalink = post.get('permalink', '')
+            text += f"\n• [{title_text}](https://reddit.com{permalink}) (⬆️ {score})"
+    
+    url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
+    payload = {
+        "chat_id": chat_id,
+        "text": text,
+        "parse_mode": "Markdown",
+        "disable_web_page_preview": True
+    }
+    
+    try:
+        response = requests.post(url, json=payload, timeout=10)
+        if response.status_code == 200:
+            print("✅ Telegram alert sent!")
+            return True
+        else:
+            print(f"❌ Telegram error: {response.json()}")
+            return False
+    except Exception as e:
+        print(f"❌ Telegram error: {e}")
+        return False
+
+def check_keyword_alerts(posts, keywords, webhook_url=None, telegram_token=None, telegram_chat=None):
+    """
+    Check posts for keyword matches and send alerts.
+    
+    Args:
+        posts: List of posts to check
+        keywords: List of keywords to monitor
+        webhook_url: Discord webhook URL
+        telegram_token: Telegram bot token
+        telegram_chat: Telegram chat ID
+    
+    Returns:
+        List of matching posts
+    """
+    if not keywords:
+        return []
+    
+    keywords_lower = [k.lower() for k in keywords]
+    matching_posts = []
+    
+    for post in posts:
+        text = f"{post.get('title', '')} {post.get('selftext', '')}".lower()
+        
+        matched_keywords = []
+        for keyword in keywords_lower:
+            if keyword in text:
+                matched_keywords.append(keyword)
+        
+        if matched_keywords:
+            post['matched_keywords'] = matched_keywords
+            matching_posts.append(post)
+    
+    if matching_posts:
+        title = f"Keyword Alert: {len(matching_posts)} matches!"
+        message = f"Found posts matching: {', '.join(set(k for p in matching_posts for k in p.get('matched_keywords', [])))}"
+        
+        if webhook_url:
+            send_discord_alert(webhook_url, title, message, matching_posts, color=0xFF6B6B)
+        
+        if telegram_token and telegram_chat:
+            send_telegram_alert(telegram_token, telegram_chat, title, message, matching_posts)
+    
+    return matching_posts
+
+def send_scrape_summary(subreddit, stats, webhook_url=None, telegram_token=None, telegram_chat=None):
+    """
+    Send a summary after scraping completes.
+    
+    Args:
+        subreddit: Subreddit name
+        stats: Dictionary with scrape statistics
+        webhook_url: Discord webhook URL
+        telegram_token: Telegram bot token
+        telegram_chat: Telegram chat ID
+    """
+    title = f"Scrape Complete: r/{subreddit}"
+    message = f"""
+📊 **Statistics:**
+• Posts: {stats.get('posts', 0)}
+• Comments: {stats.get('comments', 0)}
+• Images: {stats.get('images', 0)}
+• Videos: {stats.get('videos', 0)}
+• Duration: {stats.get('duration', 'N/A')}
+    """.strip()
+    
+    if webhook_url:
+        send_discord_alert(webhook_url, title, message, color=0x00D166)
+    
+    if telegram_token and telegram_chat:
+        send_telegram_alert(telegram_token, telegram_chat, title, message)
+
+class AlertMonitor:
+    """Monitor for keyword-based alerts."""
+    
+    def __init__(self, keywords, discord_webhook=None, telegram_token=None, telegram_chat=None):
+        self.keywords = keywords
+        self.discord_webhook = discord_webhook
+        self.telegram_token = telegram_token
+        self.telegram_chat = telegram_chat
+        self.seen_posts = set()
+    
+    def check_posts(self, posts):
+        """Check new posts for keyword matches."""
+        new_posts = [p for p in posts if p.get('id') not in self.seen_posts]
+        
+        if not new_posts:
+            return []
+        
+        # Mark as seen
+        for p in new_posts:
+            self.seen_posts.add(p.get('id'))
+        
+        # Check for keywords
+        matches = check_keyword_alerts(
+            new_posts, 
+            self.keywords,
+            self.discord_webhook,
+            self.telegram_token,
+            self.telegram_chat
+        )
+        
+        return matches

analytics/__init__.py

@@ -0,0 +1,2 @@
+# Analytics module
+from .sentiment import *

analytics/sentiment.py

@@ -0,0 +1,236 @@
+"""
+Analytics module - Sentiment Analysis, Word Clouds, Statistics
+"""
+import re
+from collections import Counter
+from pathlib import Path
+import sys
+
+# Simple sentiment analysis without external dependencies
+POSITIVE_WORDS = {
+    'good', 'great', 'awesome', 'excellent', 'amazing', 'love', 'best', 'perfect',
+    'nice', 'wonderful', 'fantastic', 'brilliant', 'superb', 'outstanding', 'happy',
+    'beautiful', 'helpful', 'thanks', 'thank', 'appreciate', 'recommend', 'interesting',
+    'useful', 'cool', 'fun', 'enjoy', 'like', 'loved', 'impressive', 'incredible'
+}
+
+NEGATIVE_WORDS = {
+    'bad', 'terrible', 'awful', 'horrible', 'hate', 'worst', 'poor', 'disappointing',
+    'useless', 'waste', 'annoying', 'boring', 'ugly', 'stupid', 'dumb', 'fail',
+    'wrong', 'broken', 'sad', 'angry', 'frustrated', 'scam', 'fake', 'trash',
+    'pathetic', 'ridiculous', 'disgusting', 'overpriced', 'avoid', 'never'
+}
+
+INTENSIFIERS = {'very', 'really', 'extremely', 'absolutely', 'totally', 'completely'}
+
+def analyze_sentiment(text):
+    """
+    Simple sentiment analysis.
+    Returns: (score, label)
+    - score: -1.0 to 1.0
+    - label: 'positive', 'negative', or 'neutral'
+    """
+    if not text:
+        return 0.0, 'neutral'
+    
+    # Clean and tokenize
+    words = re.findall(r'\b[a-z]+\b', text.lower())
+    
+    if not words:
+        return 0.0, 'neutral'
+    
+    positive_count = 0
+    negative_count = 0
+    intensifier_next = False
+    
+    for word in words:
+        multiplier = 1.5 if intensifier_next else 1.0
+        
+        if word in POSITIVE_WORDS:
+            positive_count += multiplier
+        elif word in NEGATIVE_WORDS:
+            negative_count += multiplier
+        
+        intensifier_next = word in INTENSIFIERS
+    
+    total = positive_count + negative_count
+    if total == 0:
+        return 0.0, 'neutral'
+    
+    score = (positive_count - negative_count) / len(words)
+    score = max(-1.0, min(1.0, score * 5))  # Normalize
+    
+    if score > 0.1:
+        label = 'positive'
+    elif score < -0.1:
+        label = 'negative'
+    else:
+        label = 'neutral'
+    
+    return round(score, 3), label
+
+def analyze_posts_sentiment(posts):
+    """Analyze sentiment for a list of posts."""
+    results = []
+    sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0}
+    
+    for post in posts:
+        text = f"{post.get('title', '')} {post.get('selftext', '')}"
+        score, label = analyze_sentiment(text)
+        post['sentiment_score'] = score
+        post['sentiment_label'] = label
+        sentiment_counts[label] += 1
+        results.append(post)
+    
+    return results, sentiment_counts
+
+def analyze_comments_sentiment(comments):
+    """Analyze sentiment for comments."""
+    results = []
+    sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0}
+    
+    for comment in comments:
+        score, label = analyze_sentiment(comment.get('body', ''))
+        comment['sentiment_score'] = score
+        comment['sentiment_label'] = label
+        sentiment_counts[label] += 1
+        results.append(comment)
+    
+    return results, sentiment_counts
+
+def extract_keywords(texts, top_n=50):
+    """Extract most common keywords from texts."""
+    # Stopwords
+    stopwords = {
+        'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
+        'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
+        'should', 'may', 'might', 'must', 'shall', 'can', 'to', 'of', 'in',
+        'for', 'on', 'with', 'at', 'by', 'from', 'as', 'into', 'through',
+        'during', 'before', 'after', 'above', 'below', 'between', 'under',
+        'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where',
+        'why', 'how', 'all', 'each', 'few', 'more', 'most', 'other', 'some',
+        'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than',
+        'too', 'very', 'just', 'and', 'but', 'if', 'or', 'because', 'until',
+        'while', 'this', 'that', 'these', 'those', 'i', 'me', 'my', 'myself',
+        'we', 'our', 'you', 'your', 'he', 'she', 'it', 'they', 'them', 'what',
+        'which', 'who', 'whom', 'its', 'his', 'her', 'their', 'our', 'up',
+        'out', 'about', 'any', 'also', 'get', 'got', 'like', 'one', 'two',
+        'know', 'even', 'new', 'want', 'way', 'people', 'time', 'year', 'think',
+        'amp', 'http', 'https', 'www', 'com', 'reddit', 'deleted', 'removed', 'nan'
+    }
+    
+    all_words = []
+    for text in texts:
+        if text:
+            words = re.findall(r'\b[a-z]{3,}\b', text.lower())
+            all_words.extend([w for w in words if w not in stopwords])
+    
+    return Counter(all_words).most_common(top_n)
+
+def generate_wordcloud_data(texts, top_n=100):
+    """Generate word frequency data for word cloud visualization."""
+    keywords = extract_keywords(texts, top_n)
+    
+    if not keywords:
+        return []
+    
+    max_count = keywords[0][1]
+    
+    return [
+        {"text": word, "value": count, "size": int(10 + (count / max_count) * 90)}
+        for word, count in keywords
+    ]
+
+def calculate_engagement_metrics(posts):
+    """Calculate engagement metrics for posts."""
+    if not posts:
+        return {}
+    
+    total_posts = len(posts)
+    total_score = sum(p.get('score', 0) for p in posts)
+    total_comments = sum(p.get('num_comments', 0) for p in posts)
+    total_awards = sum(p.get('total_awards', 0) for p in posts)
+    
+    # Posts with engagement
+    engaged_posts = [p for p in posts if p.get('score', 0) > 0 or p.get('num_comments', 0) > 0]
+    
+    # Top performers
+    top_by_score = sorted(posts, key=lambda x: x.get('score', 0), reverse=True)[:10]
+    top_by_comments = sorted(posts, key=lambda x: x.get('num_comments', 0), reverse=True)[:10]
+    
+    # Post type performance
+    type_performance = {}
+    for post in posts:
+        ptype = post.get('post_type', 'unknown')
+        if ptype not in type_performance:
+            type_performance[ptype] = {'count': 0, 'total_score': 0, 'total_comments': 0}
+        type_performance[ptype]['count'] += 1
+        type_performance[ptype]['total_score'] += post.get('score', 0)
+        type_performance[ptype]['total_comments'] += post.get('num_comments', 0)
+    
+    for ptype in type_performance:
+        count = type_performance[ptype]['count']
+        type_performance[ptype]['avg_score'] = type_performance[ptype]['total_score'] / count
+        type_performance[ptype]['avg_comments'] = type_performance[ptype]['total_comments'] / count
+    
+    return {
+        'total_posts': total_posts,
+        'total_score': total_score,
+        'total_comments': total_comments,
+        'total_awards': total_awards,
+        'avg_score': total_score / total_posts if total_posts else 0,
+        'avg_comments': total_comments / total_posts if total_posts else 0,
+        'engagement_rate': len(engaged_posts) / total_posts if total_posts else 0,
+        'top_by_score': top_by_score,
+        'top_by_comments': top_by_comments,
+        'type_performance': type_performance
+    }
+
+def find_best_posting_times(posts):
+    """Analyze best times to post based on engagement."""
+    hourly_stats = {}
+    daily_stats = {}
+    
+    for post in posts:
+        created = post.get('created_utc', '')
+        if not created:
+            continue
+        
+        try:
+            # Parse ISO format
+            from datetime import datetime
+            dt = datetime.fromisoformat(created.replace('Z', '+00:00'))
+            hour = dt.hour
+            day = dt.strftime('%A')
+            
+            # Hourly
+            if hour not in hourly_stats:
+                hourly_stats[hour] = {'count': 0, 'total_score': 0}
+            hourly_stats[hour]['count'] += 1
+            hourly_stats[hour]['total_score'] += post.get('score', 0)
+            
+            # Daily
+            if day not in daily_stats:
+                daily_stats[day] = {'count': 0, 'total_score': 0}
+            daily_stats[day]['count'] += 1
+            daily_stats[day]['total_score'] += post.get('score', 0)
+        except:
+            continue
+    
+    # Calculate averages
+    for hour in hourly_stats:
+        hourly_stats[hour]['avg_score'] = hourly_stats[hour]['total_score'] / hourly_stats[hour]['count']
+    
+    for day in daily_stats:
+        daily_stats[day]['avg_score'] = daily_stats[day]['total_score'] / daily_stats[day]['count']
+    
+    # Find best times
+    best_hours = sorted(hourly_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:5]
+    best_days = sorted(daily_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:3]
+    
+    return {
+        'hourly_stats': hourly_stats,
+        'daily_stats': daily_stats,
+        'best_hours': [(h, s['avg_score']) for h, s in best_hours],
+        'best_days': [(d, s['avg_score']) for d, s in best_days]
+    }

config.py

@@ -0,0 +1,58 @@
+"""
+Reddit Scraper Suite - Configuration
+"""
+import os
+from pathlib import Path
+
+# --- PATHS ---
+BASE_DIR = Path(__file__).parent
+DATA_DIR = BASE_DIR / "data"
+DB_PATH = DATA_DIR / "reddit_scraper.db"
+
+# --- SCRAPER SETTINGS ---
+USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+
+# Sources: old.reddit.com for residential IPs, mirrors for data centers
+MIRRORS = [
+    "https://old.reddit.com",
+    "https://redlib.catsarch.com",
+    "https://redlib.vsls.cz",
+    "https://r.nf",
+    "https://libreddit.northboot.xyz",
+    "https://redlib.tux.pizza"
+]
+
+# Rate limiting
+REQUEST_TIMEOUT = 15
+COOLDOWN_SECONDS = 3
+RETRY_WAIT = 30
+
+# Media settings
+MAX_IMAGES_PER_POST = 10
+MAX_VIDEOS_PER_POST = 2
+MAX_GALLERY_IMAGES = 15
+
+# Comment settings
+MAX_COMMENT_DEPTH = 5
+
+# --- ASYNC SETTINGS ---
+ASYNC_MAX_CONCURRENT = 10
+ASYNC_BATCH_SIZE = 50
+
+# --- NOTIFICATION SETTINGS ---
+DISCORD_WEBHOOK_URL = os.getenv("DISCORD_WEBHOOK_URL", "")
+TELEGRAM_BOT_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "")
+TELEGRAM_CHAT_ID = os.getenv("TELEGRAM_CHAT_ID", "")
+
+# --- DASHBOARD SETTINGS ---
+DASHBOARD_HOST = "0.0.0.0"
+DASHBOARD_PORT = 8501
+
+# --- SCHEDULER SETTINGS ---
+SCHEDULER_TIMEZONE = "Asia/Kolkata"
+
+# --- DATABASE SETTINGS ---
+DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DB_PATH}")
+
+# Ensure data directory exists
+DATA_DIR.mkdir(exist_ok=True)

dashboard/__init__.py

@@ -0,0 +1 @@
+# Dashboard module

dashboard/app.py

@@ -0,0 +1,364 @@
+"""
+Reddit Scraper Dashboard - Streamlit Web UI
+Run with: streamlit run dashboard/app.py
+"""
+import streamlit as st
+import pandas as pd
+from pathlib import Path
+import sys
+from datetime import datetime
+
+# Add parent to path
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from analytics.sentiment import (
+    analyze_posts_sentiment, extract_keywords, 
+    calculate_engagement_metrics, find_best_posting_times
+)
+from search.query import search_all_data, advanced_search, get_top_posts
+
+# Page config
+st.set_page_config(
+    page_title="Reddit Scraper Dashboard",
+    page_icon="🤖",
+    layout="wide",
+    initial_sidebar_state="expanded"
+)
+
+# Custom CSS
+st.markdown("""
+<style>
+    .main-header {
+        font-size: 2.5rem;
+        font-weight: 700;
+        background: linear-gradient(90deg, #FF4500, #FF6B6B);
+        -webkit-background-clip: text;
+        -webkit-text-fill-color: transparent;
+        margin-bottom: 1rem;
+    }
+    .metric-card {
+        background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
+        padding: 1rem;
+        border-radius: 10px;
+        color: white;
+    }
+    .stTabs [data-baseweb="tab-list"] {
+        gap: 24px;
+    }
+    .stTabs [data-baseweb="tab"] {
+        height: 50px;
+        padding: 10px 20px;
+        background-color: #262730;
+        border-radius: 5px;
+    }
+</style>
+""", unsafe_allow_html=True)
+
+def load_subreddit_data(subreddit_path):
+    """Load all data for a subreddit."""
+    data = {}
+    
+    posts_file = subreddit_path / 'posts.csv'
+    if posts_file.exists():
+        data['posts'] = pd.read_csv(posts_file)
+    
+    comments_file = subreddit_path / 'comments.csv'
+    if comments_file.exists():
+        data['comments'] = pd.read_csv(comments_file)
+    
+    return data
+
+def get_available_subreddits():
+    """Get list of scraped subreddits."""
+    data_dir = Path(__file__).parent.parent / 'data'
+    subs = []
+    
+    if data_dir.exists():
+        for sub_dir in data_dir.iterdir():
+            if sub_dir.is_dir() and (sub_dir / 'posts.csv').exists():
+                subs.append(sub_dir.name)
+    
+    return sorted(subs)
+
+def main():
+    # Header
+    st.markdown('<h1 class="main-header">🤖 Reddit Scraper Dashboard</h1>', unsafe_allow_html=True)
+    
+    # Sidebar
+    st.sidebar.title("📊 Navigation")
+    
+    # Get available subreddits
+    subreddits = get_available_subreddits()
+    
+    if not subreddits:
+        st.warning("No scraped data found! Run the scraper first:")
+        st.code("python main.py <subreddit> --mode full --limit 100")
+        return
+    
+    # Subreddit selector
+    selected_sub = st.sidebar.selectbox(
+        "Select Subreddit",
+        subreddits,
+        format_func=lambda x: f"📁 {x}"
+    )
+    
+    # Load data
+    data_dir = Path(__file__).parent.parent / 'data'
+    sub_path = data_dir / selected_sub
+    data = load_subreddit_data(sub_path)
+    
+    if 'posts' not in data:
+        st.error("No posts data found!")
+        return
+    
+    posts_df = data['posts']
+    comments_df = data.get('comments', pd.DataFrame())
+    
+    # Main content tabs
+    tab1, tab2, tab3, tab4, tab5 = st.tabs([
+        "📊 Overview", "📈 Analytics", "🔍 Search", "💬 Comments", "⚙️ Scraper"
+    ])
+    
+    with tab1:
+        st.header(f"📊 Overview: {selected_sub}")
+        
+        # Metrics row
+        col1, col2, col3, col4, col5 = st.columns(5)
+        
+        with col1:
+            st.metric("Total Posts", len(posts_df))
+        with col2:
+            st.metric("Total Comments", len(comments_df))
+        with col3:
+            total_score = posts_df['score'].sum() if 'score' in posts_df else 0
+            st.metric("Total Score", f"{total_score:,}")
+        with col4:
+            avg_score = posts_df['score'].mean() if 'score' in posts_df else 0
+            st.metric("Avg Score", f"{avg_score:.1f}")
+        with col5:
+            media_count = posts_df['has_media'].sum() if 'has_media' in posts_df else 0
+            st.metric("Media Posts", int(media_count))
+        
+        st.divider()
+        
+        # Post type distribution
+        col1, col2 = st.columns(2)
+        
+        with col1:
+            st.subheader("📝 Post Types")
+            if 'post_type' in posts_df:
+                type_counts = posts_df['post_type'].value_counts()
+                st.bar_chart(type_counts)
+        
+        with col2:
+            st.subheader("📅 Posts Over Time")
+            if 'created_utc' in posts_df:
+                posts_df['date'] = pd.to_datetime(posts_df['created_utc']).dt.date
+                daily = posts_df.groupby('date').size()
+                st.line_chart(daily)
+        
+        st.divider()
+        
+        # Top posts
+        st.subheader("🔥 Top Posts by Score")
+        if 'score' in posts_df:
+            top_posts = posts_df.nlargest(10, 'score')[['title', 'score', 'num_comments', 'post_type', 'created_utc']]
+            st.dataframe(top_posts, use_container_width=True)
+    
+    with tab2:
+        st.header("📈 Analytics")
+        
+        # Sentiment Analysis
+        st.subheader("😀 Sentiment Analysis")
+        
+        if st.button("Run Sentiment Analysis"):
+            with st.spinner("Analyzing sentiment..."):
+                posts_list = posts_df.to_dict('records')
+                analyzed_posts, sentiment_counts = analyze_posts_sentiment(posts_list)
+                
+                col1, col2, col3 = st.columns(3)
+                col1.metric("Positive", sentiment_counts['positive'], delta=None)
+                col2.metric("Neutral", sentiment_counts['neutral'], delta=None)
+                col3.metric("Negative", sentiment_counts['negative'], delta=None)
+                
+                # Pie chart
+                sentiment_df = pd.DataFrame({
+                    'Sentiment': ['Positive', 'Neutral', 'Negative'],
+                    'Count': [sentiment_counts['positive'], sentiment_counts['neutral'], sentiment_counts['negative']]
+                })
+                st.bar_chart(sentiment_df.set_index('Sentiment'))
+        
+        st.divider()
+        
+        # Keywords
+        st.subheader("☁️ Top Keywords")
+        texts = posts_df['title'].tolist()
+        if 'selftext' in posts_df:
+            texts.extend(posts_df['selftext'].dropna().tolist())
+        
+        keywords = extract_keywords(texts, top_n=30)
+        
+        if keywords:
+            kw_df = pd.DataFrame(keywords, columns=['Word', 'Count'])
+            st.bar_chart(kw_df.set_index('Word').head(20))
+        
+        st.divider()
+        
+        # Best posting times
+        st.subheader("⏰ Best Posting Times")
+        
+        if 'created_utc' in posts_df:
+            timing_data = find_best_posting_times(posts_df.to_dict('records'))
+            
+            if timing_data['best_hours']:
+                st.write("**Best Hours to Post:**")
+                for hour, avg_score in timing_data['best_hours']:
+                    st.write(f"• {hour}:00 - Avg Score: {avg_score:.1f}")
+            
+            if timing_data['best_days']:
+                st.write("**Best Days to Post:**")
+                for day, avg_score in timing_data['best_days']:
+                    st.write(f"• {day} - Avg Score: {avg_score:.1f}")
+    
+    with tab3:
+        st.header("🔍 Search Posts")
+        
+        # Search form
+        col1, col2 = st.columns([3, 1])
+        
+        with col1:
+            search_query = st.text_input("Search query", placeholder="Enter keywords...")
+        
+        with col2:
+            min_score = st.number_input("Min Score", min_value=0, value=0)
+        
+        col3, col4, col5 = st.columns(3)
+        
+        with col3:
+            if 'post_type' in posts_df:
+                post_types = ['All'] + posts_df['post_type'].dropna().unique().tolist()
+                selected_type = st.selectbox("Post Type", post_types)
+        
+        with col4:
+            if 'author' in posts_df:
+                authors = ['All'] + posts_df['author'].dropna().unique().tolist()[:50]
+                selected_author = st.selectbox("Author", authors)
+        
+        with col5:
+            sort_by = st.selectbox("Sort by", ['score', 'num_comments', 'created_utc'])
+        
+        # Search button
+        if st.button("🔍 Search"):
+            filtered = posts_df.copy()
+            
+            if search_query:
+                mask = filtered['title'].str.contains(search_query, case=False, na=False)
+                if 'selftext' in filtered:
+                    mask |= filtered['selftext'].str.contains(search_query, case=False, na=False)
+                filtered = filtered[mask]
+            
+            if min_score > 0:
+                filtered = filtered[filtered['score'] >= min_score]
+            
+            if selected_type != 'All' and 'post_type' in filtered:
+                filtered = filtered[filtered['post_type'] == selected_type]
+            
+            if selected_author != 'All' and 'author' in filtered:
+                filtered = filtered[filtered['author'] == selected_author]
+            
+            filtered = filtered.sort_values(sort_by, ascending=False)
+            
+            st.write(f"Found {len(filtered)} results")
+            st.dataframe(filtered[['title', 'score', 'num_comments', 'post_type', 'author', 'created_utc']].head(50), use_container_width=True)
+    
+    with tab4:
+        st.header("💬 Comments Analysis")
+        
+        if len(comments_df) == 0:
+            st.warning("No comments data found for this subreddit")
+        else:
+            col1, col2, col3 = st.columns(3)
+            
+            with col1:
+                st.metric("Total Comments", len(comments_df))
+            with col2:
+                avg_score = comments_df['score'].mean() if 'score' in comments_df else 0
+                st.metric("Avg Score", f"{avg_score:.1f}")
+            with col3:
+                unique_authors = comments_df['author'].nunique() if 'author' in comments_df else 0
+                st.metric("Unique Commenters", unique_authors)
+            
+            st.divider()
+            
+            # Top comments
+            st.subheader("🔥 Top Comments by Score")
+            if 'score' in comments_df:
+                top_comments = comments_df.nlargest(10, 'score')[['body', 'score', 'author', 'created_utc']]
+                for _, row in top_comments.iterrows():
+                    with st.expander(f"⬆️ {row['score']} - by u/{row['author']}"):
+                        st.write(row['body'][:500])
+            
+            st.divider()
+            
+            # Top commenters
+            st.subheader("👥 Top Commenters")
+            if 'author' in comments_df:
+                top_authors = comments_df['author'].value_counts().head(10)
+                st.bar_chart(top_authors)
+    
+    with tab5:
+        st.header("⚙️ Scraper Controls")
+        
+        st.subheader("🚀 Start New Scrape")
+        
+        col1, col2 = st.columns(2)
+        
+        with col1:
+            new_sub = st.text_input("Subreddit/User name", placeholder="e.g. python")
+            is_user = st.checkbox("Is a User (not subreddit)")
+        
+        with col2:
+            limit = st.number_input("Post Limit", min_value=10, max_value=5000, value=100)
+            mode = st.selectbox("Mode", ['full', 'history'])
+        
+        no_media = st.checkbox("Skip media download")
+        no_comments = st.checkbox("Skip comments")
+        
+        if st.button("🚀 Start Scraping"):
+            st.info(f"Run this command in terminal:")
+            cmd = f"python main.py {new_sub} --mode {mode} --limit {limit}"
+            if is_user:
+                cmd += " --user"
+            if no_media:
+                cmd += " --no-media"
+            if no_comments:
+                cmd += " --no-comments"
+            st.code(cmd)
+        
+        st.divider()
+        
+        # Export options
+        st.subheader("📤 Export Data")
+        
+        export_format = st.selectbox("Format", ['CSV', 'JSON', 'Excel'])
+        
+        if st.button("📥 Download Posts"):
+            if export_format == 'CSV':
+                csv = posts_df.to_csv(index=False)
+                st.download_button(
+                    "Download CSV",
+                    csv,
+                    f"{selected_sub}_posts.csv",
+                    "text/csv"
+                )
+            elif export_format == 'JSON':
+                json_data = posts_df.to_json(orient='records', indent=2)
+                st.download_button(
+                    "Download JSON",
+                    json_data,
+                    f"{selected_sub}_posts.json",
+                    "application/json"
+                )
+
+if __name__ == "__main__":
+    main()

export/__init__.py

@@ -0,0 +1,2 @@
+# Export module
+from .database import *

export/database.py

@@ -0,0 +1,389 @@
+"""
+Database module - SQLite storage for scraped data
+"""
+import sqlite3
+from pathlib import Path
+from datetime import datetime
+import json
+import sys
+sys.path.insert(0, str(Path(__file__).parent.parent))
+from config import DB_PATH, DATA_DIR
+
+def get_connection():
+    """Get database connection."""
+    DATA_DIR.mkdir(exist_ok=True)
+    conn = sqlite3.connect(DB_PATH)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+def init_database():
+    """Initialize database tables."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    # Posts table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS posts (
+            id TEXT PRIMARY KEY,
+            subreddit TEXT,
+            title TEXT,
+            author TEXT,
+            created_utc TEXT,
+            permalink TEXT UNIQUE,
+            url TEXT,
+            score INTEGER DEFAULT 0,
+            upvote_ratio REAL DEFAULT 0,
+            num_comments INTEGER DEFAULT 0,
+            num_crossposts INTEGER DEFAULT 0,
+            selftext TEXT,
+            post_type TEXT,
+            is_nsfw BOOLEAN DEFAULT 0,
+            is_spoiler BOOLEAN DEFAULT 0,
+            flair TEXT,
+            total_awards INTEGER DEFAULT 0,
+            has_media BOOLEAN DEFAULT 0,
+            media_downloaded BOOLEAN DEFAULT 0,
+            source TEXT,
+            scraped_at TEXT DEFAULT CURRENT_TIMESTAMP,
+            sentiment_score REAL,
+            sentiment_label TEXT
+        )
+    """)
+    
+    # Comments table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS comments (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            comment_id TEXT UNIQUE,
+            post_id TEXT,
+            post_permalink TEXT,
+            parent_id TEXT,
+            author TEXT,
+            body TEXT,
+            score INTEGER DEFAULT 0,
+            created_utc TEXT,
+            depth INTEGER DEFAULT 0,
+            is_submitter BOOLEAN DEFAULT 0,
+            scraped_at TEXT DEFAULT CURRENT_TIMESTAMP,
+            sentiment_score REAL,
+            sentiment_label TEXT,
+            FOREIGN KEY (post_id) REFERENCES posts(id)
+        )
+    """)
+    
+    # Subreddits table (for tracking)
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS subreddits (
+            name TEXT PRIMARY KEY,
+            last_scraped TEXT,
+            total_posts INTEGER DEFAULT 0,
+            total_comments INTEGER DEFAULT 0,
+            total_media INTEGER DEFAULT 0
+        )
+    """)
+    
+    # Scheduled jobs table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS scheduled_jobs (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            target TEXT,
+            is_user BOOLEAN DEFAULT 0,
+            mode TEXT DEFAULT 'full',
+            limit_posts INTEGER DEFAULT 100,
+            cron_expression TEXT,
+            last_run TEXT,
+            next_run TEXT,
+            enabled BOOLEAN DEFAULT 1,
+            created_at TEXT DEFAULT CURRENT_TIMESTAMP
+        )
+    """)
+    
+    # Alerts table
+    cursor.execute("""
+        CREATE TABLE IF NOT EXISTS alerts (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            keyword TEXT,
+            subreddit TEXT,
+            alert_type TEXT DEFAULT 'discord',
+            webhook_url TEXT,
+            enabled BOOLEAN DEFAULT 1,
+            last_triggered TEXT,
+            created_at TEXT DEFAULT CURRENT_TIMESTAMP
+        )
+    """)
+    
+    # Create indexes
+    cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON posts(subreddit)")
+    cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_created ON posts(created_utc)")
+    cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_score ON posts(score)")
+    cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_post ON comments(post_id)")
+    cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_author ON comments(author)")
+    
+    conn.commit()
+    conn.close()
+    print("✅ Database initialized")
+
+def save_post(post_data, subreddit):
+    """Save a single post to database."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    try:
+        cursor.execute("""
+            INSERT OR REPLACE INTO posts 
+            (id, subreddit, title, author, created_utc, permalink, url, score, 
+             upvote_ratio, num_comments, num_crossposts, selftext, post_type,
+             is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source)
+            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+        """, (
+            post_data.get('id'),
+            subreddit,
+            post_data.get('title'),
+            post_data.get('author'),
+            post_data.get('created_utc'),
+            post_data.get('permalink'),
+            post_data.get('url'),
+            post_data.get('score', 0),
+            post_data.get('upvote_ratio', 0),
+            post_data.get('num_comments', 0),
+            post_data.get('num_crossposts', 0),
+            post_data.get('selftext', ''),
+            post_data.get('post_type'),
+            post_data.get('is_nsfw', False),
+            post_data.get('is_spoiler', False),
+            post_data.get('flair', ''),
+            post_data.get('total_awards', 0),
+            post_data.get('has_media', False),
+            post_data.get('media_downloaded', False),
+            post_data.get('source', '')
+        ))
+        conn.commit()
+        return True
+    except Exception as e:
+        print(f"DB Error: {e}")
+        return False
+    finally:
+        conn.close()
+
+def save_posts_batch(posts, subreddit):
+    """Save multiple posts efficiently."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    saved = 0
+    
+    for post in posts:
+        try:
+            cursor.execute("""
+                INSERT OR IGNORE INTO posts 
+                (id, subreddit, title, author, created_utc, permalink, url, score, 
+                 upvote_ratio, num_comments, num_crossposts, selftext, post_type,
+                 is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+            """, (
+                post.get('id'),
+                subreddit,
+                post.get('title'),
+                post.get('author'),
+                post.get('created_utc'),
+                post.get('permalink'),
+                post.get('url'),
+                post.get('score', 0),
+                post.get('upvote_ratio', 0),
+                post.get('num_comments', 0),
+                post.get('num_crossposts', 0),
+                post.get('selftext', ''),
+                post.get('post_type'),
+                post.get('is_nsfw', False),
+                post.get('is_spoiler', False),
+                post.get('flair', ''),
+                post.get('total_awards', 0),
+                post.get('has_media', False),
+                post.get('media_downloaded', False),
+                post.get('source', '')
+            ))
+            if cursor.rowcount > 0:
+                saved += 1
+        except:
+            continue
+    
+    conn.commit()
+    conn.close()
+    return saved
+
+def save_comments_batch(comments, post_id):
+    """Save multiple comments efficiently."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    saved = 0
+    
+    for comment in comments:
+        try:
+            cursor.execute("""
+                INSERT OR IGNORE INTO comments 
+                (comment_id, post_id, post_permalink, parent_id, author, body, 
+                 score, created_utc, depth, is_submitter)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+            """, (
+                comment.get('comment_id'),
+                post_id,
+                comment.get('post_permalink'),
+                comment.get('parent_id'),
+                comment.get('author'),
+                comment.get('body'),
+                comment.get('score', 0),
+                comment.get('created_utc'),
+                comment.get('depth', 0),
+                comment.get('is_submitter', False)
+            ))
+            if cursor.rowcount > 0:
+                saved += 1
+        except:
+            continue
+    
+    conn.commit()
+    conn.close()
+    return saved
+
+def search_posts(query=None, subreddit=None, author=None, min_score=None, 
+                 start_date=None, end_date=None, post_type=None, limit=100):
+    """Search posts with filters."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    sql = "SELECT * FROM posts WHERE 1=1"
+    params = []
+    
+    if query:
+        sql += " AND (title LIKE ? OR selftext LIKE ?)"
+        params.extend([f"%{query}%", f"%{query}%"])
+    
+    if subreddit:
+        sql += " AND subreddit = ?"
+        params.append(subreddit)
+    
+    if author:
+        sql += " AND author = ?"
+        params.append(author)
+    
+    if min_score:
+        sql += " AND score >= ?"
+        params.append(min_score)
+    
+    if start_date:
+        sql += " AND created_utc >= ?"
+        params.append(start_date)
+    
+    if end_date:
+        sql += " AND created_utc <= ?"
+        params.append(end_date)
+    
+    if post_type:
+        sql += " AND post_type = ?"
+        params.append(post_type)
+    
+    sql += " ORDER BY created_utc DESC LIMIT ?"
+    params.append(limit)
+    
+    cursor.execute(sql, params)
+    results = [dict(row) for row in cursor.fetchall()]
+    conn.close()
+    return results
+
+def search_comments(query=None, post_id=None, author=None, min_score=None, limit=100):
+    """Search comments with filters."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    sql = "SELECT * FROM comments WHERE 1=1"
+    params = []
+    
+    if query:
+        sql += " AND body LIKE ?"
+        params.append(f"%{query}%")
+    
+    if post_id:
+        sql += " AND post_id = ?"
+        params.append(post_id)
+    
+    if author:
+        sql += " AND author = ?"
+        params.append(author)
+    
+    if min_score:
+        sql += " AND score >= ?"
+        params.append(min_score)
+    
+    sql += " ORDER BY score DESC LIMIT ?"
+    params.append(limit)
+    
+    cursor.execute(sql, params)
+    results = [dict(row) for row in cursor.fetchall()]
+    conn.close()
+    return results
+
+def get_subreddit_stats(subreddit):
+    """Get statistics for a subreddit."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    stats = {}
+    
+    # Post stats
+    cursor.execute("""
+        SELECT 
+            COUNT(*) as total_posts,
+            AVG(score) as avg_score,
+            MAX(score) as max_score,
+            SUM(num_comments) as total_comments,
+            AVG(upvote_ratio) as avg_upvote_ratio
+        FROM posts WHERE subreddit = ?
+    """, (subreddit,))
+    row = cursor.fetchone()
+    if row:
+        stats.update(dict(row))
+    
+    # Post type distribution
+    cursor.execute("""
+        SELECT post_type, COUNT(*) as count 
+        FROM posts WHERE subreddit = ? 
+        GROUP BY post_type
+    """, (subreddit,))
+    stats['post_types'] = {row['post_type']: row['count'] for row in cursor.fetchall()}
+    
+    # Top authors
+    cursor.execute("""
+        SELECT author, COUNT(*) as post_count, SUM(score) as total_score
+        FROM posts WHERE subreddit = ? AND author != '[deleted]'
+        GROUP BY author ORDER BY post_count DESC LIMIT 10
+    """, (subreddit,))
+    stats['top_authors'] = [dict(row) for row in cursor.fetchall()]
+    
+    # Activity by hour
+    cursor.execute("""
+        SELECT strftime('%H', created_utc) as hour, COUNT(*) as count
+        FROM posts WHERE subreddit = ?
+        GROUP BY hour ORDER BY hour
+    """, (subreddit,))
+    stats['hourly_activity'] = {row['hour']: row['count'] for row in cursor.fetchall()}
+    
+    conn.close()
+    return stats
+
+def get_all_subreddits():
+    """Get list of all scraped subreddits."""
+    conn = get_connection()
+    cursor = conn.cursor()
+    
+    cursor.execute("""
+        SELECT subreddit, COUNT(*) as post_count, 
+               MAX(created_utc) as latest_post,
+               MIN(created_utc) as oldest_post
+        FROM posts GROUP BY subreddit ORDER BY post_count DESC
+    """)
+    
+    results = [dict(row) for row in cursor.fetchall()]
+    conn.close()
+    return results
+
+# Initialize on import
+init_database()

main.py

@@ -1,3 +1,7 @@
+"""
+🤖 Universal Reddit Scraper Suite
+Full-featured scraper with analytics, dashboard, notifications, and scheduling.
+"""
 import requests
 import pandas as pd
 import datetime
@@ -8,14 +12,12 @@ import argparse
 import random
 import sys
 import json
-import re
 from urllib.parse import urlparse
-from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
 
 # --- CONFIGURATION ---
 USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
 
-# Sources: old.reddit.com for residential IPs, mirrors for data centers
 MIRRORS = [
     "https://old.reddit.com",
     "https://redlib.catsarch.com",
@@ -108,16 +110,13 @@ def get_media_urls(post_data):
     """Extracts all media URLs from a post."""
     media = {"images": [], "videos": [], "galleries": []}
     
-    # Direct image link
     url = post_data.get('url', '')
     if any(ext in url.lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp']):
         media["images"].append(url)
     
-    # Reddit-hosted image
     if 'i.redd.it' in url:
         media["images"].append(url)
     
-    # Reddit video
     if post_data.get('is_video'):
         reddit_video = post_data.get('media', {})
         if reddit_video and 'reddit_video' in reddit_video:
@@ -125,17 +124,14 @@ def get_media_urls(post_data):
             if video_url:
                 media["videos"].append(video_url.split('?')[0])
     
-    # Preview images
     preview = post_data.get('preview', {})
     if preview and 'images' in preview:
         for img in preview['images']:
             source = img.get('source', {})
             if source.get('url'):
-                # Unescape HTML entities
                 clean_url = source['url'].replace('&', '&')
                 media["images"].append(clean_url)
     
-    # Gallery posts
     if post_data.get('is_gallery'):
         gallery_data = post_data.get('gallery_data', {})
         media_metadata = post_data.get('media_metadata', {})
@@ -149,7 +145,6 @@ def get_media_urls(post_data):
                         clean_url = meta['s']['u'].replace('&', '&')
                         media["galleries"].append(clean_url)
     
-    # External video (YouTube, etc.)
     if 'youtube.com' in url or 'youtu.be' in url:
         media["videos"].append(url)
     
@@ -158,7 +153,6 @@ def get_media_urls(post_data):
 def download_media(url, save_path, media_type="image"):
     """Downloads a single media file."""
     try:
-        # Skip if already downloaded
         if os.path.exists(save_path):
             return True
         
@@ -169,7 +163,7 @@ def download_media(url, save_path, media_type="image"):
                     f.write(chunk)
             return True
     except Exception as e:
-        print(f"⚠️ Failed to download {media_type}: {e}")
+        pass
     return False
 
 def download_post_media(post_data, dirs, post_id):
@@ -177,23 +171,20 @@ def download_post_media(post_data, dirs, post_id):
     media = get_media_urls(post_data)
     downloaded = {"images": 0, "videos": 0}
     
-    # Download images
+    for i, img_url in enumerate(media["images"][:5]):
-    for i, img_url in enumerate(media["images"][:5]):  # Limit to 5 images per post
         ext = os.path.splitext(urlparse(img_url).path)[1] or '.jpg'
         save_path = os.path.join(dirs["images"], f"{post_id}_{i}{ext}")
         if download_media(img_url, save_path, "image"):
             downloaded["images"] += 1
     
-    # Download gallery images
+    for i, img_url in enumerate(media["galleries"][:10]):
-    for i, img_url in enumerate(media["galleries"][:10]):  # Limit gallery to 10
         ext = '.jpg'
         save_path = os.path.join(dirs["images"], f"{post_id}_gallery_{i}{ext}")
         if download_media(img_url, save_path, "gallery"):
             downloaded["images"] += 1
     
-    # Download videos
+    for i, vid_url in enumerate(media["videos"][:2]):
-    for i, vid_url in enumerate(media["videos"][:2]):  # Limit to 2 videos
+        if 'youtube' not in vid_url:
-        if 'youtube' not in vid_url:  # Skip YouTube (can't direct download)
             ext = '.mp4'
             save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}")
             if download_media(vid_url, save_path, "video"):
@@ -203,11 +194,10 @@ def download_post_media(post_data, dirs, post_id):
 
 # --- COMMENT SCRAPING ---
 def scrape_comments(permalink, max_depth=3):
-    """Scrapes comments from a post using Reddit JSON endpoint."""
+    """Scrapes comments from a post."""
     comments = []
     
     try:
-        # Clean permalink and build URL
         if not permalink.startswith('http'):
             url = f"https://old.reddit.com{permalink}.json?limit=100"
         else:
@@ -219,13 +209,12 @@ def scrape_comments(permalink, max_depth=3):
         
         data = response.json()
         
-        # Comments are in the second element of the response
         if len(data) > 1:
             comment_data = data[1]['data']['children']
             comments = parse_comments(comment_data, permalink, depth=0, max_depth=max_depth)
     
     except Exception as e:
-        print(f"⚠️ Comment fetch error: {e}")
+        pass
     
     return comments
 
@@ -237,7 +226,7 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
         return comments
     
     for item in comment_list:
-        if item['kind'] != 't1':  # Skip non-comment items
+        if item['kind'] != 't1':
             continue
         
         c = item['data']
@@ -255,7 +244,6 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
         }
         comments.append(comment)
         
-        # Parse replies recursively
         replies = c.get('replies')
         if replies and isinstance(replies, dict):
             reply_children = replies.get('data', {}).get('children', [])
@@ -263,12 +251,11 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
     
     return comments
 
-# --- ENHANCED POST EXTRACTION ---
+# --- POST EXTRACTION ---
 def extract_post_data(post_json):
     """Extracts comprehensive post data."""
     p = post_json
     
-    # Determine post type
     post_type = "text"
     if p.get('is_video'):
         post_type = "video"
@@ -282,39 +269,28 @@ def extract_post_data(post_json):
         post_type = "link"
     
     return {
-        # Basic Info
         "id": p.get('id'),
         "title": p.get('title'),
         "author": p.get('author'),
         "created_utc": datetime.datetime.fromtimestamp(p.get('created_utc', 0)).isoformat(),
         "permalink": p.get('permalink'),
         "url": p.get('url_overridden_by_dest', p.get('url')),
-        
-        # Engagement
         "score": p.get('score', 0),
         "upvote_ratio": p.get('upvote_ratio', 0),
         "num_comments": p.get('num_comments', 0),
         "num_crossposts": p.get('num_crossposts', 0),
-        
-        # Content
         "selftext": p.get('selftext', ''),
         "post_type": post_type,
         "is_nsfw": p.get('over_18', False),
         "is_spoiler": p.get('spoiler', False),
-        
-        # Flair & Awards
         "flair": p.get('link_flair_text', ''),
         "total_awards": p.get('total_awards_received', 0),
-        
-        # Media flags
         "has_media": p.get('is_video', False) or p.get('is_gallery', False) or 'i.redd.it' in p.get('url', ''),
         "media_downloaded": False,
-        
-        # Source tracking
         "source": "History-Full"
     }
 
-# --- MODE 2: FULL HISTORY SCRAPE ---
+# --- FULL HISTORY SCRAPE ---
 def run_full_history(target, limit, is_user=False, download_media_flag=True, scrape_comments_flag=True):
     """Full scrape with images, videos, and comments."""
     prefix = "u" if is_user else "r"
@@ -331,6 +307,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
     total_posts = 0
     total_media = {"images": 0, "videos": 0}
     total_comments = 0
+    start_time = time.time()
     
     while total_posts < limit:
         random.shuffle(MIRRORS)
@@ -362,11 +339,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
                         p = child['data']
                         post = extract_post_data(p)
                         
-                        # Skip if already seen
                         if post['permalink'] in SEEN_URLS:
                             continue
                         
-                        # Download media
                         if download_media_flag:
                             downloaded = download_post_media(p, dirs, post['id'])
                             post['media_downloaded'] = downloaded['images'] > 0 or downloaded['videos'] > 0
@@ -375,22 +350,19 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
                         
                         posts.append(post)
                         
-                        # Scrape comments
                         if scrape_comments_flag and post['num_comments'] > 0:
                             print(f"   💬 Fetching comments for: {post['title'][:40]}...")
                             comments = scrape_comments(post['permalink'])
                             all_comments.extend(comments)
                             total_comments += len(comments)
-                            time.sleep(1)  # Rate limiting for comment fetches
+                            time.sleep(1)
                     
-                    # Save data
                     saved = save_posts_csv(posts, dirs["posts"])
                     total_posts += saved
                     
                     if all_comments:
                         save_comments_csv(all_comments, dirs["comments"])
                     
-                    # Progress update
                     print(f"\n📊 Progress: {total_posts}/{limit} posts")
                     print(f"   🖼️  Images: {total_media['images']} | 🎬 Videos: {total_media['videos']}")
                     print(f"   💬 Comments: {total_comments}")
@@ -398,7 +370,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
                     after = data['data'].get('after')
                     if not after:
                         print("\n🏁 Reached end of available history.")
-                        return
+                        break
                     
                     success = True
                     break
@@ -407,6 +379,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
                 print(f"   ⚠️ Error with {base_url}: {e}")
                 continue
         
+        if not after:
+            break
+            
         if not success:
             print("\n❌ All sources failed. Waiting 30s...")
             time.sleep(30)
@@ -414,6 +389,8 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
             print(f"\n⏸️ Cooling down (3s)...")
             time.sleep(3)
     
+    duration = time.time() - start_time
+    
     print("\n" + "=" * 50)
     print("✅ SCRAPE COMPLETE!")
     print(f"   📁 Data saved to: {dirs['base']}")
@@ -421,8 +398,17 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
     print(f"   🖼️  Total images: {total_media['images']}")
     print(f"   🎬 Total videos: {total_media['videos']}")
     print(f"   💬 Total comments: {total_comments}")
+    print(f"   ⏱️  Duration: {duration:.1f}s")
+    
+    return {
+        'posts': total_posts,
+        'images': total_media['images'],
+        'videos': total_media['videos'],
+        'comments': total_comments,
+        'duration': f"{duration:.1f}s"
+    }
 
-# --- MODE 1: LIVE MONITOR (RSS) - Legacy ---
+# --- MONITOR MODE ---
 def run_monitor(target, is_user=False):
     prefix = "u" if is_user else "r"
     if is_user:
@@ -437,7 +423,6 @@ def run_monitor(target, is_user=False):
         
         if response.status_code != 200:
             print(f"❌ RSS blocked (Status {response.status_code}), trying JSON...")
-            # Fallback to JSON
             run_full_history(target, 25, is_user, download_media_flag=False, scrape_comments_flag=False)
             return
 
@@ -474,33 +459,144 @@ def run_monitor(target, is_user=False):
     except Exception as e:
         print(f"❌ Monitor Error: {e}")
 
-# --- CLI ARGS ---
+# --- CLI ---
-if __name__ == "__main__":
+def main():
     parser = argparse.ArgumentParser(
-        description="🤖 Universal Reddit Scraper - Full Media & Comments Support",
+        description="🤖 Universal Reddit Scraper Suite",
         formatter_class=argparse.RawDescriptionHelpFormatter,
         epilog="""
-Examples:
+Commands:
-  python main.py delhi --mode full --limit 100
+  SCRAPING:
-  python main.py spez --user --mode full --limit 50
+    python main.py <target> --mode full --limit 100
-  python main.py python --mode full --no-media --limit 200
+    python main.py <target> --mode history --limit 500
-  python main.py india --mode monitor
+    python main.py <target> --mode monitor
+    
+  SEARCH:
+    python main.py --search "keyword" --subreddit delhi
+    python main.py --search "keyword" --min-score 100
+    
+  DASHBOARD:
+    python main.py --dashboard
+    
+  SCHEDULE:
+    python main.py --schedule delhi --every 60
+    
+  ANALYTICS:
+    python main.py --analyze delhi --sentiment
+    python main.py --analyze delhi --keywords
         """
     )
-    parser.add_argument("target", help="Subreddit name (e.g. 'delhi') or Username (e.g. 'spez')")
+    
-    parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full", 
+    # Scraping args
-                        help="monitor=live RSS, history=posts only, full=posts+media+comments")
+    parser.add_argument("target", nargs='?', help="Subreddit or username to scrape")
-    parser.add_argument("--user", action="store_true", help="Target is a User, not Subreddit")
+    parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full")
+    parser.add_argument("--user", action="store_true", help="Target is a user")
     parser.add_argument("--limit", type=int, default=100, help="Max posts to scrape")
-    parser.add_argument("--no-media", action="store_true", help="Skip downloading images/videos")
+    parser.add_argument("--no-media", action="store_true", help="Skip media download")
-    parser.add_argument("--no-comments", action="store_true", help="Skip scraping comments")
+    parser.add_argument("--no-comments", action="store_true", help="Skip comments")
+    
+    # Dashboard
+    parser.add_argument("--dashboard", action="store_true", help="Launch web dashboard")
+    
+    # Search
+    parser.add_argument("--search", type=str, help="Search scraped data")
+    parser.add_argument("--subreddit", type=str, help="Filter by subreddit")
+    parser.add_argument("--min-score", type=int, help="Filter by minimum score")
+    parser.add_argument("--author", type=str, help="Filter by author")
+    
+    # Analytics
+    parser.add_argument("--analyze", type=str, help="Run analytics on subreddit")
+    parser.add_argument("--sentiment", action="store_true", help="Run sentiment analysis")
+    parser.add_argument("--keywords", action="store_true", help="Extract keywords")
+    
+    # Schedule
+    parser.add_argument("--schedule", type=str, help="Schedule scraping for target")
+    parser.add_argument("--every", type=int, help="Interval in minutes")
+    
+    # Alerts
+    parser.add_argument("--alert", type=str, help="Set keyword alert")
+    parser.add_argument("--discord-webhook", type=str, help="Discord webhook URL")
+    parser.add_argument("--telegram-token", type=str, help="Telegram bot token")
+    parser.add_argument("--telegram-chat", type=str, help="Telegram chat ID")
     
     args = parser.parse_args()
     
     print("=" * 50)
-    print("🤖 UNIVERSAL REDDIT SCRAPER")
+    print("🤖 UNIVERSAL REDDIT SCRAPER SUITE")
     print("=" * 50)
     
+    # Dashboard mode
+    if args.dashboard:
+        print("\n🌐 Launching Dashboard...")
+        print("   Open: http://localhost:8501")
+        os.system("streamlit run dashboard/app.py")
+        return
+    
+    # Search mode
+    if args.search:
+        print(f"\n🔍 Searching for: {args.search}")
+        from search.query import search_all_data, print_search_results
+        
+        results = search_all_data(
+            query=args.search,
+            min_score=args.min_score,
+            author=args.author
+        )
+        print_search_results(results)
+        return
+    
+    # Analytics mode
+    if args.analyze:
+        print(f"\n📊 Analyzing: {args.analyze}")
+        
+        # Load data
+        data_dir = Path(f"data/r_{args.analyze}")
+        if not data_dir.exists():
+            print(f"❌ No data found for r/{args.analyze}")
+            return
+        
+        posts_file = data_dir / "posts.csv"
+        if not posts_file.exists():
+            print(f"❌ No posts data found")
+            return
+        
+        import pandas as pd
+        df = pd.read_csv(posts_file)
+        posts = df.to_dict('records')
+        
+        if args.sentiment:
+            from analytics.sentiment import analyze_posts_sentiment
+            analyzed, counts = analyze_posts_sentiment(posts)
+            print(f"\n😀 Sentiment Analysis:")
+            print(f"   Positive: {counts['positive']}")
+            print(f"   Neutral:  {counts['neutral']}")
+            print(f"   Negative: {counts['negative']}")
+        
+        if args.keywords:
+            from analytics.sentiment import extract_keywords
+            texts = [str(p.get('title', '') or '') + ' ' + str(p.get('selftext', '') or '') for p in posts]
+            keywords = extract_keywords(texts, top_n=20)
+            print(f"\n☁️ Top Keywords:")
+            for word, count in keywords:
+                print(f"   {word}: {count}")
+        
+        return
+    
+    # Schedule mode
+    if args.schedule:
+        if not args.every:
+            print("❌ Please specify --every <minutes>")
+            return
+        
+        from scheduler.cron import run_scheduled
+        run_scheduled(args.schedule, args.every, args.mode, args.limit, args.user)
+        return
+    
+    # Regular scraping mode
+    if not args.target:
+        parser.print_help()
+        return
+    
     if args.mode == "monitor":
         prefix = "u" if args.user else "r"
         dirs = setup_directories(args.target, prefix)
@@ -510,10 +606,12 @@ Examples:
             run_monitor(args.target, args.user)
             time.sleep(300)
     elif args.mode == "history":
-        # Legacy mode - posts only
         run_full_history(args.target, args.limit, args.user, 
                         download_media_flag=False, scrape_comments_flag=False)
-    else:  # full mode
+    else:
         run_full_history(args.target, args.limit, args.user,
                         download_media_flag=not args.no_media,
                         scrape_comments_flag=not args.no_comments)
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -1,2 +1,9 @@
+# Core
 pandas
 requests
+
+# Dashboard
+streamlit
+
+# Export
+openpyxl

scheduler/__init__.py

@@ -0,0 +1,2 @@
+# Scheduler module
+from .cron import *

scheduler/cron.py

@@ -0,0 +1,215 @@
+"""
+Scheduler module - Cron-style scheduling for scrape jobs
+"""
+import time
+import threading
+from datetime import datetime, timedelta
+import json
+from pathlib import Path
+import sys
+
+class CronScheduler:
+    """Simple cron-style scheduler for Reddit scraping jobs."""
+    
+    def __init__(self):
+        self.jobs = []
+        self.running = False
+        self.thread = None
+    
+    def add_job(self, target, mode='full', limit=100, is_user=False, 
+                interval_minutes=60, run_at_start=True):
+        """
+        Add a scheduled scraping job.
+        
+        Args:
+            target: Subreddit or username
+            mode: 'full', 'history', or 'monitor'
+            limit: Post limit per run
+            is_user: True if target is a user
+            interval_minutes: Minutes between runs
+            run_at_start: Run immediately when scheduler starts
+        """
+        job = {
+            'id': len(self.jobs) + 1,
+            'target': target,
+            'mode': mode,
+            'limit': limit,
+            'is_user': is_user,
+            'interval_minutes': interval_minutes,
+            'run_at_start': run_at_start,
+            'last_run': None,
+            'next_run': datetime.now() if run_at_start else datetime.now() + timedelta(minutes=interval_minutes),
+            'enabled': True,
+            'run_count': 0
+        }
+        self.jobs.append(job)
+        print(f"📅 Added job #{job['id']}: {'u/' if is_user else 'r/'}{target} every {interval_minutes}min")
+        return job['id']
+    
+    def remove_job(self, job_id):
+        """Remove a scheduled job."""
+        self.jobs = [j for j in self.jobs if j['id'] != job_id]
+        print(f"🗑️ Removed job #{job_id}")
+    
+    def disable_job(self, job_id):
+        """Temporarily disable a job."""
+        for job in self.jobs:
+            if job['id'] == job_id:
+                job['enabled'] = False
+                print(f"⏸️ Disabled job #{job_id}")
+    
+    def enable_job(self, job_id):
+        """Enable a disabled job."""
+        for job in self.jobs:
+            if job['id'] == job_id:
+                job['enabled'] = True
+                print(f"▶️ Enabled job #{job_id}")
+    
+    def list_jobs(self):
+        """List all scheduled jobs."""
+        print("\n📋 Scheduled Jobs:")
+        print("-" * 60)
+        for job in self.jobs:
+            status = "✅" if job['enabled'] else "⏸️"
+            prefix = "u/" if job['is_user'] else "r/"
+            next_run = job['next_run'].strftime("%H:%M:%S") if job['next_run'] else "Never"
+            print(f"{status} #{job['id']} | {prefix}{job['target']} | "
+                  f"Every {job['interval_minutes']}min | Next: {next_run} | "
+                  f"Runs: {job['run_count']}")
+        print()
+        return self.jobs
+    
+    def _run_job(self, job):
+        """Execute a single job."""
+        # Import here to avoid circular imports
+        try:
+            from main import run_full_history
+            
+            prefix = "u/" if job['is_user'] else "r/"
+            print(f"\n🚀 Running scheduled job: {prefix}{job['target']}")
+            
+            run_full_history(
+                job['target'],
+                job['limit'],
+                job['is_user'],
+                download_media_flag=(job['mode'] == 'full'),
+                scrape_comments_flag=(job['mode'] == 'full')
+            )
+            
+            job['last_run'] = datetime.now()
+            job['run_count'] += 1
+            print(f"✅ Job completed: {prefix}{job['target']}")
+            
+        except Exception as e:
+            print(f"❌ Job failed: {e}")
+    
+    def _scheduler_loop(self):
+        """Main scheduler loop."""
+        print("🔄 Scheduler started")
+        
+        while self.running:
+            now = datetime.now()
+            
+            for job in self.jobs:
+                if not job['enabled']:
+                    continue
+                
+                if job['next_run'] and now >= job['next_run']:
+                    self._run_job(job)
+                    job['next_run'] = now + timedelta(minutes=job['interval_minutes'])
+            
+            # Check every 30 seconds
+            time.sleep(30)
+        
+        print("🛑 Scheduler stopped")
+    
+    def start(self):
+        """Start the scheduler in background."""
+        if self.running:
+            print("⚠️ Scheduler already running")
+            return
+        
+        self.running = True
+        self.thread = threading.Thread(target=self._scheduler_loop, daemon=True)
+        self.thread.start()
+        print("✅ Scheduler started in background")
+    
+    def stop(self):
+        """Stop the scheduler."""
+        self.running = False
+        if self.thread:
+            self.thread.join(timeout=5)
+        print("🛑 Scheduler stopped")
+    
+    def save_jobs(self, filepath='scheduler_jobs.json'):
+        """Save jobs to file."""
+        jobs_data = []
+        for job in self.jobs:
+            job_copy = job.copy()
+            job_copy['last_run'] = job_copy['last_run'].isoformat() if job_copy['last_run'] else None
+            job_copy['next_run'] = job_copy['next_run'].isoformat() if job_copy['next_run'] else None
+            jobs_data.append(job_copy)
+        
+        with open(filepath, 'w') as f:
+            json.dump(jobs_data, f, indent=2)
+        print(f"💾 Saved {len(self.jobs)} jobs to {filepath}")
+    
+    def load_jobs(self, filepath='scheduler_jobs.json'):
+        """Load jobs from file."""
+        if not Path(filepath).exists():
+            print("⚠️ No saved jobs found")
+            return
+        
+        with open(filepath, 'r') as f:
+            jobs_data = json.load(f)
+        
+        for job_data in jobs_data:
+            if job_data['last_run']:
+                job_data['last_run'] = datetime.fromisoformat(job_data['last_run'])
+            if job_data['next_run']:
+                job_data['next_run'] = datetime.fromisoformat(job_data['next_run'])
+            self.jobs.append(job_data)
+        
+        print(f"📂 Loaded {len(jobs_data)} jobs from {filepath}")
+
+
+# Simple interval-based scheduler for CLI
+def run_scheduled(target, interval_minutes, mode='full', limit=100, is_user=False):
+    """
+    Run a scrape job on a schedule.
+    
+    Args:
+        target: Subreddit or username
+        interval_minutes: Minutes between runs
+        mode: 'full', 'history', or 'monitor'
+        limit: Post limit per run
+        is_user: True if target is a user
+    """
+    from main import run_full_history
+    
+    prefix = "u/" if is_user else "r/"
+    print(f"📅 Scheduled: {prefix}{target} every {interval_minutes} minutes")
+    print("Press Ctrl+C to stop\n")
+    
+    run_count = 0
+    
+    try:
+        while True:
+            run_count += 1
+            print(f"\n{'='*50}")
+            print(f"🔄 Run #{run_count} - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
+            print(f"{'='*50}")
+            
+            run_full_history(
+                target,
+                limit,
+                is_user,
+                download_media_flag=(mode == 'full'),
+                scrape_comments_flag=(mode == 'full')
+            )
+            
+            print(f"\n⏰ Next run in {interval_minutes} minutes...")
+            time.sleep(interval_minutes * 60)
+            
+    except KeyboardInterrupt:
+        print(f"\n\n🛑 Scheduler stopped after {run_count} runs")

search/__init__.py

@@ -0,0 +1,2 @@
+# Search module
+from .query import *

search/query.py

@@ -0,0 +1,220 @@
+"""
+Search & Query module - Search and filter scraped data
+"""
+import pandas as pd
+from pathlib import Path
+from datetime import datetime
+import re
+
+def search_csv(filepath, query=None, column=None, min_score=None, max_score=None,
+               start_date=None, end_date=None, post_type=None, author=None, limit=50):
+    """
+    Search within a CSV file with various filters.
+    
+    Args:
+        filepath: Path to CSV file
+        query: Text to search for
+        column: Specific column to search in (default: all text columns)
+        min_score: Minimum score filter
+        max_score: Maximum score filter
+        start_date: Start date (YYYY-MM-DD)
+        end_date: End date (YYYY-MM-DD)
+        post_type: Filter by post type (image, video, text, etc.)
+        author: Filter by author
+        limit: Maximum results to return
+    
+    Returns:
+        DataFrame with matching results
+    """
+    if not Path(filepath).exists():
+        print(f"❌ File not found: {filepath}")
+        return pd.DataFrame()
+    
+    df = pd.read_csv(filepath)
+    
+    # Text search
+    if query:
+        if column and column in df.columns:
+            mask = df[column].astype(str).str.contains(query, case=False, na=False)
+        else:
+            # Search in all text columns
+            text_cols = ['title', 'selftext', 'body']
+            mask = pd.Series([False] * len(df))
+            for col in text_cols:
+                if col in df.columns:
+                    mask |= df[col].astype(str).str.contains(query, case=False, na=False)
+        df = df[mask]
+    
+    # Score filter
+    if min_score is not None and 'score' in df.columns:
+        df = df[df['score'] >= min_score]
+    if max_score is not None and 'score' in df.columns:
+        df = df[df['score'] <= max_score]
+    
+    # Date filter
+    if 'created_utc' in df.columns:
+        if start_date:
+            df = df[df['created_utc'] >= start_date]
+        if end_date:
+            df = df[df['created_utc'] <= end_date]
+    
+    # Post type filter
+    if post_type and 'post_type' in df.columns:
+        df = df[df['post_type'] == post_type]
+    
+    # Author filter
+    if author and 'author' in df.columns:
+        df = df[df['author'] == author]
+    
+    return df.head(limit)
+
+def search_all_data(data_dir='data', query=None, **kwargs):
+    """
+    Search across all scraped data.
+    
+    Args:
+        data_dir: Data directory path
+        query: Text to search for
+        **kwargs: Additional filters passed to search_csv
+    
+    Returns:
+        Dictionary with results from each subreddit
+    """
+    results = {}
+    data_path = Path(data_dir)
+    
+    if not data_path.exists():
+        print(f"❌ Data directory not found: {data_dir}")
+        return results
+    
+    # Find all posts.csv files
+    for sub_dir in data_path.iterdir():
+        if sub_dir.is_dir():
+            posts_file = sub_dir / 'posts.csv'
+            if posts_file.exists():
+                df = search_csv(str(posts_file), query=query, **kwargs)
+                if len(df) > 0:
+                    results[sub_dir.name] = df
+    
+    # Also check legacy format
+    for csv_file in data_path.glob('*.csv'):
+        if csv_file.stem not in [r.replace('r_', '').replace('u_', '') for r in results.keys()]:
+            df = search_csv(str(csv_file), query=query, **kwargs)
+            if len(df) > 0:
+                results[csv_file.stem] = df
+    
+    return results
+
+def print_search_results(results, show_preview=True):
+    """Pretty print search results."""
+    total = sum(len(df) for df in results.values())
+    
+    print(f"\n🔍 Found {total} results across {len(results)} sources\n")
+    print("=" * 70)
+    
+    for source, df in results.items():
+        print(f"\n📁 {source} ({len(df)} matches)")
+        print("-" * 50)
+        
+        for _, row in df.iterrows():
+            title = str(row.get('title', row.get('body', 'N/A')))[:60]
+            score = row.get('score', 0)
+            date = str(row.get('created_utc', ''))[:10]
+            
+            print(f"  [{score:>4}⬆] {title}...")
+            if show_preview and 'selftext' in row and row['selftext']:
+                preview = str(row['selftext'])[:100].replace('\n', ' ')
+                print(f"         └─ {preview}...")
+            print()
+
+def advanced_search(data_dir='data', query=None, regex=False, sort_by='score', 
+                   ascending=False, **kwargs):
+    """
+    Advanced search with regex support and sorting.
+    
+    Args:
+        data_dir: Data directory path
+        query: Search query (text or regex pattern)
+        regex: Treat query as regex pattern
+        sort_by: Column to sort results by
+        ascending: Sort ascending (default: descending)
+        **kwargs: Additional filters
+    
+    Returns:
+        Combined DataFrame of all results
+    """
+    all_results = []
+    data_path = Path(data_dir)
+    
+    for sub_dir in data_path.iterdir():
+        if sub_dir.is_dir():
+            posts_file = sub_dir / 'posts.csv'
+            if posts_file.exists():
+                df = pd.read_csv(posts_file)
+                df['source'] = sub_dir.name
+                all_results.append(df)
+    
+    if not all_results:
+        return pd.DataFrame()
+    
+    combined = pd.concat(all_results, ignore_index=True)
+    
+    # Apply query
+    if query:
+        if regex:
+            pattern = query
+        else:
+            pattern = re.escape(query)
+        
+        mask = pd.Series([False] * len(combined))
+        for col in ['title', 'selftext']:
+            if col in combined.columns:
+                mask |= combined[col].astype(str).str.contains(pattern, case=False, na=False, regex=True)
+        combined = combined[mask]
+    
+    # Apply other filters
+    if kwargs.get('min_score') and 'score' in combined.columns:
+        combined = combined[combined['score'] >= kwargs['min_score']]
+    
+    if kwargs.get('author') and 'author' in combined.columns:
+        combined = combined[combined['author'] == kwargs['author']]
+    
+    if kwargs.get('post_type') and 'post_type' in combined.columns:
+        combined = combined[combined['post_type'] == kwargs['post_type']]
+    
+    # Sort
+    if sort_by in combined.columns:
+        combined = combined.sort_values(sort_by, ascending=ascending)
+    
+    limit = kwargs.get('limit', 100)
+    return combined.head(limit)
+
+def get_top_posts(data_dir='data', n=10, by='score'):
+    """Get top N posts across all scraped data."""
+    df = advanced_search(data_dir, sort_by=by, ascending=False, limit=n)
+    return df
+
+def get_recent_posts(data_dir='data', n=10):
+    """Get most recent posts across all scraped data."""
+    df = advanced_search(data_dir, sort_by='created_utc', ascending=False, limit=n)
+    return df
+
+def find_author_posts(data_dir='data', author=None):
+    """Find all posts by a specific author."""
+    return advanced_search(data_dir, author=author, limit=1000)
+
+def export_search_results(results, output_path, format='csv'):
+    """Export search results to file."""
+    if isinstance(results, dict):
+        combined = pd.concat(results.values(), ignore_index=True)
+    else:
+        combined = results
+    
+    if format == 'csv':
+        combined.to_csv(output_path, index=False)
+    elif format == 'json':
+        combined.to_json(output_path, orient='records', indent=2)
+    elif format == 'excel':
+        combined.to_excel(output_path, index=False)
+    
+    print(f"💾 Exported {len(combined)} results to {output_path}")