v3.0: Full suite - Dashboard, Analytics, Scheduling, Notifications, Search
This commit is contained in:
Sanjeev Kumar
parent
2eb9ef9a54
commit
0a1a604f49
16 changed files with 2025 additions and 160 deletions
249
README.md
249
README.md
|
|
@ -1,90 +1,148 @@
|
|||
# 🤖 Universal Reddit Scraper
|
||||
# 🤖 Universal Reddit Scraper Suite
|
||||
|
||||
[](https://github.com/ksanjeev284/reddit-universal-scraper/actions/workflows/docker-publish.yml)
|
||||
|
||||
A robust, full-featured Reddit scraper that downloads **posts, images, videos, galleries, and comments**. Designed to run on low-resource servers (like AWS Free Tier).
|
||||
|
||||
## 🐳 Quick Start (No Installation Needed!)
|
||||
```bash
|
||||
docker run -d -v $(pwd)/data:/app/data ghcr.io/ksanjeev284/reddit-universal-scraper:latest delhi --mode full --limit 100
|
||||
```
|
||||
A **full-featured** Reddit scraper suite with analytics dashboard, sentiment analysis, scheduled scraping, notifications, and more!
|
||||
|
||||
## ✨ Features
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| 📊 **Full Metadata** | Title, author, score, upvotes, awards, flair, NSFW flags |
|
||||
| 🖼️ **Image Download** | Automatically downloads all images from posts |
|
||||
| 🎬 **Video Download** | Downloads Reddit-hosted videos |
|
||||
| 🖼️ **Gallery Support** | Extracts and downloads all images from gallery posts |
|
||||
| 💬 **Comment Scraping** | Recursively scrapes all comments with threading info |
|
||||
| 🔄 **Dual Sources** | Uses old.reddit.com + Redlib mirrors for reliability |
|
||||
| 📁 **Organized Output** | Clean folder structure per subreddit |
|
||||
| 📊 **Full Scraping** | Posts, comments, images, videos, galleries |
|
||||
| 📈 **Analytics Dashboard** | Beautiful Streamlit web UI |
|
||||
| 😀 **Sentiment Analysis** | Analyze post/comment sentiment |
|
||||
| ☁️ **Keyword Extraction** | Generate word clouds |
|
||||
| 🔍 **Search & Filter** | Query scraped data with filters |
|
||||
| 📅 **Scheduled Scraping** | Cron-style job scheduling |
|
||||
| 📧 **Notifications** | Discord & Telegram alerts |
|
||||
| 🗄️ **SQLite Database** | Structured data storage |
|
||||
| 📤 **Multiple Exports** | CSV, JSON, Excel |
|
||||
|
||||
## 📁 Output Structure
|
||||
## 🚀 Quick Start
|
||||
|
||||
```
|
||||
data/
|
||||
└── r_delhi/
|
||||
├── posts.csv # All post metadata
|
||||
├── comments.csv # All comments with threading
|
||||
└── media/
|
||||
├── images/ # Downloaded images & galleries
|
||||
│ ├── abc123_0.jpg
|
||||
│ ├── abc123_gallery_0.jpg
|
||||
│ └── ...
|
||||
└── videos/ # Downloaded videos
|
||||
└── xyz789_0.mp4
|
||||
```
|
||||
|
||||
## 🚀 Usage
|
||||
|
||||
### Full Scrape (Posts + Media + Comments)
|
||||
```bash
|
||||
# Scrape r/delhi with everything
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Scrape a subreddit (posts + media + comments)
|
||||
python main.py delhi --mode full --limit 100
|
||||
|
||||
# Launch analytics dashboard
|
||||
python main.py --dashboard
|
||||
```
|
||||
|
||||
## 📖 Usage Guide
|
||||
|
||||
### 🔄 Scraping Modes
|
||||
|
||||
```bash
|
||||
# Full scrape with everything
|
||||
python main.py delhi --mode full --limit 100
|
||||
|
||||
# History only (no media/comments - faster)
|
||||
python main.py delhi --mode history --limit 500
|
||||
|
||||
# Live monitor (checks every 5 min)
|
||||
python main.py delhi --mode monitor
|
||||
|
||||
# Scrape a user's posts
|
||||
python main.py spez --user --mode full --limit 50
|
||||
|
||||
# Skip media or comments
|
||||
python main.py delhi --mode full --no-media --limit 200
|
||||
python main.py delhi --mode full --no-comments --limit 200
|
||||
```
|
||||
|
||||
### Posts Only (No Media Download)
|
||||
```bash
|
||||
python main.py python --mode full --no-media --limit 200
|
||||
```
|
||||
|
||||
### Posts Only (No Comments)
|
||||
```bash
|
||||
python main.py india --mode full --no-comments --limit 100
|
||||
```
|
||||
|
||||
### Live Monitor Mode
|
||||
```bash
|
||||
python main.py delhi --mode monitor
|
||||
```
|
||||
|
||||
### Legacy History Mode (Posts Only, No Media)
|
||||
```bash
|
||||
python main.py delhi --mode history --limit 500
|
||||
```
|
||||
|
||||
## 🐳 Docker Usage
|
||||
### 📊 Analytics Dashboard
|
||||
|
||||
```bash
|
||||
# Build the image
|
||||
docker build -t reddit-scraper .
|
||||
# Launch the web dashboard
|
||||
python main.py --dashboard
|
||||
|
||||
# Full scrape with media
|
||||
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100
|
||||
|
||||
# Scrape without media (faster)
|
||||
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --no-media --limit 500
|
||||
|
||||
# Monitor mode (runs continuously)
|
||||
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
|
||||
# Opens at http://localhost:8501
|
||||
```
|
||||
|
||||
## 📊 CSV Output Format
|
||||
**Dashboard Features:**
|
||||
- 📈 Post statistics & charts
|
||||
- 😀 Sentiment analysis
|
||||
- ☁️ Keyword extraction
|
||||
- 🔍 Search & filter interface
|
||||
- 📤 Export data
|
||||
|
||||
### 🔍 Search Data
|
||||
|
||||
```bash
|
||||
# Search all scraped data
|
||||
python main.py --search "credit card"
|
||||
|
||||
# Search with filters
|
||||
python main.py --search "laptop" --min-score 100
|
||||
python main.py --search "advice" --author username
|
||||
python main.py --search "help" --subreddit delhi
|
||||
```
|
||||
|
||||
### 😀 Analytics
|
||||
|
||||
```bash
|
||||
# Run sentiment analysis
|
||||
python main.py --analyze delhi --sentiment
|
||||
|
||||
# Extract top keywords
|
||||
python main.py --analyze delhi --keywords
|
||||
```
|
||||
|
||||
### 📅 Scheduled Scraping
|
||||
|
||||
```bash
|
||||
# Scrape every 60 minutes
|
||||
python main.py --schedule delhi --every 60
|
||||
|
||||
# Scrape with options
|
||||
python main.py --schedule delhi --every 30 --mode full --limit 50
|
||||
```
|
||||
|
||||
### 📧 Notifications (Discord/Telegram)
|
||||
|
||||
**Discord:**
|
||||
```bash
|
||||
python main.py delhi --mode monitor --discord-webhook "YOUR_WEBHOOK_URL"
|
||||
```
|
||||
|
||||
**Telegram:**
|
||||
```bash
|
||||
python main.py delhi --mode monitor \
|
||||
--telegram-token "YOUR_BOT_TOKEN" \
|
||||
--telegram-chat "YOUR_CHAT_ID"
|
||||
```
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
reddit-scraper/
|
||||
├── main.py # Main CLI entry point
|
||||
├── config.py # Configuration settings
|
||||
├── analytics/ # Sentiment & keyword analysis
|
||||
│ └── sentiment.py
|
||||
├── alerts/ # Discord & Telegram notifications
|
||||
│ └── notifications.py
|
||||
├── dashboard/ # Streamlit web UI
|
||||
│ └── app.py
|
||||
├── export/ # Database & export functions
|
||||
│ └── database.py
|
||||
├── scheduler/ # Cron-style scheduling
|
||||
│ └── cron.py
|
||||
├── search/ # Search & filter engine
|
||||
│ └── query.py
|
||||
└── data/ # Scraped data
|
||||
└── r_subreddit/
|
||||
├── posts.csv
|
||||
├── comments.csv
|
||||
└── media/
|
||||
├── images/
|
||||
└── videos/
|
||||
```
|
||||
|
||||
## 📊 Data Output
|
||||
|
||||
### posts.csv
|
||||
| Column | Description |
|
||||
|
|
@ -92,50 +150,51 @@ docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
|
|||
| id | Reddit post ID |
|
||||
| title | Post title |
|
||||
| author | Username |
|
||||
| created_utc | Timestamp (ISO format) |
|
||||
| permalink | Reddit URL path |
|
||||
| url | External/media URL |
|
||||
| score | Net upvotes |
|
||||
| upvote_ratio | Percentage upvoted |
|
||||
| num_comments | Comment count |
|
||||
| selftext | Post body text |
|
||||
| post_type | text/image/video/gallery/link |
|
||||
| flair | Post flair text |
|
||||
| has_media | Boolean |
|
||||
| media_downloaded | Boolean |
|
||||
| post_type | text/image/video/gallery |
|
||||
| selftext | Post body |
|
||||
| flair | Post flair |
|
||||
| is_nsfw | NSFW flag |
|
||||
| created_utc | Timestamp |
|
||||
|
||||
### comments.csv
|
||||
| Column | Description |
|
||||
|--------|-------------|
|
||||
| post_permalink | Parent post URL |
|
||||
| comment_id | Reddit comment ID |
|
||||
| parent_id | Parent comment/post ID |
|
||||
| comment_id | Comment ID |
|
||||
| post_permalink | Parent post |
|
||||
| author | Username |
|
||||
| body | Comment text |
|
||||
| score | Net upvotes |
|
||||
| created_utc | Timestamp |
|
||||
| depth | Nesting level (0 = top-level) |
|
||||
| is_submitter | Is the post author |
|
||||
| score | Upvotes |
|
||||
| depth | Nesting level |
|
||||
|
||||
## ⚙️ Command Line Options
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `target` | Subreddit or username | Required |
|
||||
| `--mode` | `full`, `history`, or `monitor` | `full` |
|
||||
| `--user` | Target is a user, not subreddit | `false` |
|
||||
| `--limit` | Max posts to scrape | `100` |
|
||||
| `--no-media` | Skip downloading images/videos | `false` |
|
||||
| `--no-comments` | Skip scraping comments | `false` |
|
||||
|
||||
## 🛠️ Requirements
|
||||
## 🐳 Docker
|
||||
|
||||
```bash
|
||||
pip install pandas requests
|
||||
# Build
|
||||
docker build -t reddit-scraper .
|
||||
|
||||
# Full scrape
|
||||
docker run -v $(pwd)/data:/app/data reddit-scraper delhi --mode full --limit 100
|
||||
|
||||
# Monitor mode
|
||||
docker run -d -v $(pwd)/data:/app/data reddit-scraper delhi --mode monitor
|
||||
```
|
||||
|
||||
## ⚙️ Configuration
|
||||
|
||||
Edit `config.py` or use environment variables:
|
||||
|
||||
```bash
|
||||
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
|
||||
export TELEGRAM_BOT_TOKEN="123456:ABC..."
|
||||
export TELEGRAM_CHAT_ID="987654321"
|
||||
```
|
||||
|
||||
## 📜 License
|
||||
|
||||
MIT License - Feel free to use, modify, and distribute.
|
||||
|
||||
## 🤝 Contributing
|
||||
Pull requests are welcome! For major changes, please open an issue first.
|
||||
|
||||
Pull requests welcome! For major changes, please open an issue first.
|
||||
|
|
|
|||
2
alerts/__init__.py
Normal file
2
alerts/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# Alerts module
|
||||
from .notifications import *
|
||||
208
alerts/notifications.py
Normal file
208
alerts/notifications.py
Normal file
|
|
@ -0,0 +1,208 @@
|
|||
"""
|
||||
Notification module - Discord & Telegram alerts
|
||||
"""
|
||||
import requests
|
||||
import json
|
||||
from datetime import datetime
|
||||
|
||||
def send_discord_alert(webhook_url, title, message, posts=None, color=0x5865F2):
|
||||
"""
|
||||
Send alert to Discord via webhook.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL
|
||||
title: Alert title
|
||||
message: Alert message
|
||||
posts: Optional list of posts to include
|
||||
color: Embed color (default: Discord blue)
|
||||
"""
|
||||
if not webhook_url:
|
||||
print("⚠️ Discord webhook URL not configured")
|
||||
return False
|
||||
|
||||
embeds = [{
|
||||
"title": f"🤖 {title}",
|
||||
"description": message,
|
||||
"color": color,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"footer": {"text": "Reddit Scraper Alert"}
|
||||
}]
|
||||
|
||||
# Add post previews
|
||||
if posts:
|
||||
fields = []
|
||||
for post in posts[:5]: # Max 5 posts
|
||||
fields.append({
|
||||
"name": post.get('title', 'No Title')[:100],
|
||||
"value": f"Score: {post.get('score', 0)} | Comments: {post.get('num_comments', 0)}\n[View Post](https://reddit.com{post.get('permalink', '')})",
|
||||
"inline": False
|
||||
})
|
||||
embeds[0]["fields"] = fields
|
||||
|
||||
payload = {"embeds": embeds}
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
webhook_url,
|
||||
json=payload,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=10
|
||||
)
|
||||
if response.status_code == 204:
|
||||
print("✅ Discord alert sent!")
|
||||
return True
|
||||
else:
|
||||
print(f"❌ Discord error: {response.status_code}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"❌ Discord error: {e}")
|
||||
return False
|
||||
|
||||
def send_telegram_alert(bot_token, chat_id, title, message, posts=None):
|
||||
"""
|
||||
Send alert to Telegram via bot.
|
||||
|
||||
Args:
|
||||
bot_token: Telegram bot token
|
||||
chat_id: Chat/Channel ID to send to
|
||||
title: Alert title
|
||||
message: Alert message
|
||||
posts: Optional list of posts to include
|
||||
"""
|
||||
if not bot_token or not chat_id:
|
||||
print("⚠️ Telegram credentials not configured")
|
||||
return False
|
||||
|
||||
# Build message
|
||||
text = f"🤖 *{title}*\n\n{message}"
|
||||
|
||||
if posts:
|
||||
text += "\n\n📝 *New Posts:*\n"
|
||||
for post in posts[:5]:
|
||||
title_text = post.get('title', 'No Title')[:80]
|
||||
score = post.get('score', 0)
|
||||
permalink = post.get('permalink', '')
|
||||
text += f"\n• [{title_text}](https://reddit.com{permalink}) (⬆️ {score})"
|
||||
|
||||
url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
|
||||
payload = {
|
||||
"chat_id": chat_id,
|
||||
"text": text,
|
||||
"parse_mode": "Markdown",
|
||||
"disable_web_page_preview": True
|
||||
}
|
||||
|
||||
try:
|
||||
response = requests.post(url, json=payload, timeout=10)
|
||||
if response.status_code == 200:
|
||||
print("✅ Telegram alert sent!")
|
||||
return True
|
||||
else:
|
||||
print(f"❌ Telegram error: {response.json()}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f"❌ Telegram error: {e}")
|
||||
return False
|
||||
|
||||
def check_keyword_alerts(posts, keywords, webhook_url=None, telegram_token=None, telegram_chat=None):
|
||||
"""
|
||||
Check posts for keyword matches and send alerts.
|
||||
|
||||
Args:
|
||||
posts: List of posts to check
|
||||
keywords: List of keywords to monitor
|
||||
webhook_url: Discord webhook URL
|
||||
telegram_token: Telegram bot token
|
||||
telegram_chat: Telegram chat ID
|
||||
|
||||
Returns:
|
||||
List of matching posts
|
||||
"""
|
||||
if not keywords:
|
||||
return []
|
||||
|
||||
keywords_lower = [k.lower() for k in keywords]
|
||||
matching_posts = []
|
||||
|
||||
for post in posts:
|
||||
text = f"{post.get('title', '')} {post.get('selftext', '')}".lower()
|
||||
|
||||
matched_keywords = []
|
||||
for keyword in keywords_lower:
|
||||
if keyword in text:
|
||||
matched_keywords.append(keyword)
|
||||
|
||||
if matched_keywords:
|
||||
post['matched_keywords'] = matched_keywords
|
||||
matching_posts.append(post)
|
||||
|
||||
if matching_posts:
|
||||
title = f"Keyword Alert: {len(matching_posts)} matches!"
|
||||
message = f"Found posts matching: {', '.join(set(k for p in matching_posts for k in p.get('matched_keywords', [])))}"
|
||||
|
||||
if webhook_url:
|
||||
send_discord_alert(webhook_url, title, message, matching_posts, color=0xFF6B6B)
|
||||
|
||||
if telegram_token and telegram_chat:
|
||||
send_telegram_alert(telegram_token, telegram_chat, title, message, matching_posts)
|
||||
|
||||
return matching_posts
|
||||
|
||||
def send_scrape_summary(subreddit, stats, webhook_url=None, telegram_token=None, telegram_chat=None):
|
||||
"""
|
||||
Send a summary after scraping completes.
|
||||
|
||||
Args:
|
||||
subreddit: Subreddit name
|
||||
stats: Dictionary with scrape statistics
|
||||
webhook_url: Discord webhook URL
|
||||
telegram_token: Telegram bot token
|
||||
telegram_chat: Telegram chat ID
|
||||
"""
|
||||
title = f"Scrape Complete: r/{subreddit}"
|
||||
message = f"""
|
||||
📊 **Statistics:**
|
||||
• Posts: {stats.get('posts', 0)}
|
||||
• Comments: {stats.get('comments', 0)}
|
||||
• Images: {stats.get('images', 0)}
|
||||
• Videos: {stats.get('videos', 0)}
|
||||
• Duration: {stats.get('duration', 'N/A')}
|
||||
""".strip()
|
||||
|
||||
if webhook_url:
|
||||
send_discord_alert(webhook_url, title, message, color=0x00D166)
|
||||
|
||||
if telegram_token and telegram_chat:
|
||||
send_telegram_alert(telegram_token, telegram_chat, title, message)
|
||||
|
||||
class AlertMonitor:
|
||||
"""Monitor for keyword-based alerts."""
|
||||
|
||||
def __init__(self, keywords, discord_webhook=None, telegram_token=None, telegram_chat=None):
|
||||
self.keywords = keywords
|
||||
self.discord_webhook = discord_webhook
|
||||
self.telegram_token = telegram_token
|
||||
self.telegram_chat = telegram_chat
|
||||
self.seen_posts = set()
|
||||
|
||||
def check_posts(self, posts):
|
||||
"""Check new posts for keyword matches."""
|
||||
new_posts = [p for p in posts if p.get('id') not in self.seen_posts]
|
||||
|
||||
if not new_posts:
|
||||
return []
|
||||
|
||||
# Mark as seen
|
||||
for p in new_posts:
|
||||
self.seen_posts.add(p.get('id'))
|
||||
|
||||
# Check for keywords
|
||||
matches = check_keyword_alerts(
|
||||
new_posts,
|
||||
self.keywords,
|
||||
self.discord_webhook,
|
||||
self.telegram_token,
|
||||
self.telegram_chat
|
||||
)
|
||||
|
||||
return matches
|
||||
2
analytics/__init__.py
Normal file
2
analytics/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# Analytics module
|
||||
from .sentiment import *
|
||||
236
analytics/sentiment.py
Normal file
236
analytics/sentiment.py
Normal file
|
|
@ -0,0 +1,236 @@
|
|||
"""
|
||||
Analytics module - Sentiment Analysis, Word Clouds, Statistics
|
||||
"""
|
||||
import re
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
# Simple sentiment analysis without external dependencies
|
||||
POSITIVE_WORDS = {
|
||||
'good', 'great', 'awesome', 'excellent', 'amazing', 'love', 'best', 'perfect',
|
||||
'nice', 'wonderful', 'fantastic', 'brilliant', 'superb', 'outstanding', 'happy',
|
||||
'beautiful', 'helpful', 'thanks', 'thank', 'appreciate', 'recommend', 'interesting',
|
||||
'useful', 'cool', 'fun', 'enjoy', 'like', 'loved', 'impressive', 'incredible'
|
||||
}
|
||||
|
||||
NEGATIVE_WORDS = {
|
||||
'bad', 'terrible', 'awful', 'horrible', 'hate', 'worst', 'poor', 'disappointing',
|
||||
'useless', 'waste', 'annoying', 'boring', 'ugly', 'stupid', 'dumb', 'fail',
|
||||
'wrong', 'broken', 'sad', 'angry', 'frustrated', 'scam', 'fake', 'trash',
|
||||
'pathetic', 'ridiculous', 'disgusting', 'overpriced', 'avoid', 'never'
|
||||
}
|
||||
|
||||
INTENSIFIERS = {'very', 'really', 'extremely', 'absolutely', 'totally', 'completely'}
|
||||
|
||||
def analyze_sentiment(text):
|
||||
"""
|
||||
Simple sentiment analysis.
|
||||
Returns: (score, label)
|
||||
- score: -1.0 to 1.0
|
||||
- label: 'positive', 'negative', or 'neutral'
|
||||
"""
|
||||
if not text:
|
||||
return 0.0, 'neutral'
|
||||
|
||||
# Clean and tokenize
|
||||
words = re.findall(r'\b[a-z]+\b', text.lower())
|
||||
|
||||
if not words:
|
||||
return 0.0, 'neutral'
|
||||
|
||||
positive_count = 0
|
||||
negative_count = 0
|
||||
intensifier_next = False
|
||||
|
||||
for word in words:
|
||||
multiplier = 1.5 if intensifier_next else 1.0
|
||||
|
||||
if word in POSITIVE_WORDS:
|
||||
positive_count += multiplier
|
||||
elif word in NEGATIVE_WORDS:
|
||||
negative_count += multiplier
|
||||
|
||||
intensifier_next = word in INTENSIFIERS
|
||||
|
||||
total = positive_count + negative_count
|
||||
if total == 0:
|
||||
return 0.0, 'neutral'
|
||||
|
||||
score = (positive_count - negative_count) / len(words)
|
||||
score = max(-1.0, min(1.0, score * 5)) # Normalize
|
||||
|
||||
if score > 0.1:
|
||||
label = 'positive'
|
||||
elif score < -0.1:
|
||||
label = 'negative'
|
||||
else:
|
||||
label = 'neutral'
|
||||
|
||||
return round(score, 3), label
|
||||
|
||||
def analyze_posts_sentiment(posts):
|
||||
"""Analyze sentiment for a list of posts."""
|
||||
results = []
|
||||
sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0}
|
||||
|
||||
for post in posts:
|
||||
text = f"{post.get('title', '')} {post.get('selftext', '')}"
|
||||
score, label = analyze_sentiment(text)
|
||||
post['sentiment_score'] = score
|
||||
post['sentiment_label'] = label
|
||||
sentiment_counts[label] += 1
|
||||
results.append(post)
|
||||
|
||||
return results, sentiment_counts
|
||||
|
||||
def analyze_comments_sentiment(comments):
|
||||
"""Analyze sentiment for comments."""
|
||||
results = []
|
||||
sentiment_counts = {'positive': 0, 'negative': 0, 'neutral': 0}
|
||||
|
||||
for comment in comments:
|
||||
score, label = analyze_sentiment(comment.get('body', ''))
|
||||
comment['sentiment_score'] = score
|
||||
comment['sentiment_label'] = label
|
||||
sentiment_counts[label] += 1
|
||||
results.append(comment)
|
||||
|
||||
return results, sentiment_counts
|
||||
|
||||
def extract_keywords(texts, top_n=50):
|
||||
"""Extract most common keywords from texts."""
|
||||
# Stopwords
|
||||
stopwords = {
|
||||
'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
|
||||
'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
|
||||
'should', 'may', 'might', 'must', 'shall', 'can', 'to', 'of', 'in',
|
||||
'for', 'on', 'with', 'at', 'by', 'from', 'as', 'into', 'through',
|
||||
'during', 'before', 'after', 'above', 'below', 'between', 'under',
|
||||
'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where',
|
||||
'why', 'how', 'all', 'each', 'few', 'more', 'most', 'other', 'some',
|
||||
'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than',
|
||||
'too', 'very', 'just', 'and', 'but', 'if', 'or', 'because', 'until',
|
||||
'while', 'this', 'that', 'these', 'those', 'i', 'me', 'my', 'myself',
|
||||
'we', 'our', 'you', 'your', 'he', 'she', 'it', 'they', 'them', 'what',
|
||||
'which', 'who', 'whom', 'its', 'his', 'her', 'their', 'our', 'up',
|
||||
'out', 'about', 'any', 'also', 'get', 'got', 'like', 'one', 'two',
|
||||
'know', 'even', 'new', 'want', 'way', 'people', 'time', 'year', 'think',
|
||||
'amp', 'http', 'https', 'www', 'com', 'reddit', 'deleted', 'removed', 'nan'
|
||||
}
|
||||
|
||||
all_words = []
|
||||
for text in texts:
|
||||
if text:
|
||||
words = re.findall(r'\b[a-z]{3,}\b', text.lower())
|
||||
all_words.extend([w for w in words if w not in stopwords])
|
||||
|
||||
return Counter(all_words).most_common(top_n)
|
||||
|
||||
def generate_wordcloud_data(texts, top_n=100):
|
||||
"""Generate word frequency data for word cloud visualization."""
|
||||
keywords = extract_keywords(texts, top_n)
|
||||
|
||||
if not keywords:
|
||||
return []
|
||||
|
||||
max_count = keywords[0][1]
|
||||
|
||||
return [
|
||||
{"text": word, "value": count, "size": int(10 + (count / max_count) * 90)}
|
||||
for word, count in keywords
|
||||
]
|
||||
|
||||
def calculate_engagement_metrics(posts):
|
||||
"""Calculate engagement metrics for posts."""
|
||||
if not posts:
|
||||
return {}
|
||||
|
||||
total_posts = len(posts)
|
||||
total_score = sum(p.get('score', 0) for p in posts)
|
||||
total_comments = sum(p.get('num_comments', 0) for p in posts)
|
||||
total_awards = sum(p.get('total_awards', 0) for p in posts)
|
||||
|
||||
# Posts with engagement
|
||||
engaged_posts = [p for p in posts if p.get('score', 0) > 0 or p.get('num_comments', 0) > 0]
|
||||
|
||||
# Top performers
|
||||
top_by_score = sorted(posts, key=lambda x: x.get('score', 0), reverse=True)[:10]
|
||||
top_by_comments = sorted(posts, key=lambda x: x.get('num_comments', 0), reverse=True)[:10]
|
||||
|
||||
# Post type performance
|
||||
type_performance = {}
|
||||
for post in posts:
|
||||
ptype = post.get('post_type', 'unknown')
|
||||
if ptype not in type_performance:
|
||||
type_performance[ptype] = {'count': 0, 'total_score': 0, 'total_comments': 0}
|
||||
type_performance[ptype]['count'] += 1
|
||||
type_performance[ptype]['total_score'] += post.get('score', 0)
|
||||
type_performance[ptype]['total_comments'] += post.get('num_comments', 0)
|
||||
|
||||
for ptype in type_performance:
|
||||
count = type_performance[ptype]['count']
|
||||
type_performance[ptype]['avg_score'] = type_performance[ptype]['total_score'] / count
|
||||
type_performance[ptype]['avg_comments'] = type_performance[ptype]['total_comments'] / count
|
||||
|
||||
return {
|
||||
'total_posts': total_posts,
|
||||
'total_score': total_score,
|
||||
'total_comments': total_comments,
|
||||
'total_awards': total_awards,
|
||||
'avg_score': total_score / total_posts if total_posts else 0,
|
||||
'avg_comments': total_comments / total_posts if total_posts else 0,
|
||||
'engagement_rate': len(engaged_posts) / total_posts if total_posts else 0,
|
||||
'top_by_score': top_by_score,
|
||||
'top_by_comments': top_by_comments,
|
||||
'type_performance': type_performance
|
||||
}
|
||||
|
||||
def find_best_posting_times(posts):
|
||||
"""Analyze best times to post based on engagement."""
|
||||
hourly_stats = {}
|
||||
daily_stats = {}
|
||||
|
||||
for post in posts:
|
||||
created = post.get('created_utc', '')
|
||||
if not created:
|
||||
continue
|
||||
|
||||
try:
|
||||
# Parse ISO format
|
||||
from datetime import datetime
|
||||
dt = datetime.fromisoformat(created.replace('Z', '+00:00'))
|
||||
hour = dt.hour
|
||||
day = dt.strftime('%A')
|
||||
|
||||
# Hourly
|
||||
if hour not in hourly_stats:
|
||||
hourly_stats[hour] = {'count': 0, 'total_score': 0}
|
||||
hourly_stats[hour]['count'] += 1
|
||||
hourly_stats[hour]['total_score'] += post.get('score', 0)
|
||||
|
||||
# Daily
|
||||
if day not in daily_stats:
|
||||
daily_stats[day] = {'count': 0, 'total_score': 0}
|
||||
daily_stats[day]['count'] += 1
|
||||
daily_stats[day]['total_score'] += post.get('score', 0)
|
||||
except:
|
||||
continue
|
||||
|
||||
# Calculate averages
|
||||
for hour in hourly_stats:
|
||||
hourly_stats[hour]['avg_score'] = hourly_stats[hour]['total_score'] / hourly_stats[hour]['count']
|
||||
|
||||
for day in daily_stats:
|
||||
daily_stats[day]['avg_score'] = daily_stats[day]['total_score'] / daily_stats[day]['count']
|
||||
|
||||
# Find best times
|
||||
best_hours = sorted(hourly_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:5]
|
||||
best_days = sorted(daily_stats.items(), key=lambda x: x[1]['avg_score'], reverse=True)[:3]
|
||||
|
||||
return {
|
||||
'hourly_stats': hourly_stats,
|
||||
'daily_stats': daily_stats,
|
||||
'best_hours': [(h, s['avg_score']) for h, s in best_hours],
|
||||
'best_days': [(d, s['avg_score']) for d, s in best_days]
|
||||
}
|
||||
58
config.py
Normal file
58
config.py
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
"""
|
||||
Reddit Scraper Suite - Configuration
|
||||
"""
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# --- PATHS ---
|
||||
BASE_DIR = Path(__file__).parent
|
||||
DATA_DIR = BASE_DIR / "data"
|
||||
DB_PATH = DATA_DIR / "reddit_scraper.db"
|
||||
|
||||
# --- SCRAPER SETTINGS ---
|
||||
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
|
||||
# Sources: old.reddit.com for residential IPs, mirrors for data centers
|
||||
MIRRORS = [
|
||||
"https://old.reddit.com",
|
||||
"https://redlib.catsarch.com",
|
||||
"https://redlib.vsls.cz",
|
||||
"https://r.nf",
|
||||
"https://libreddit.northboot.xyz",
|
||||
"https://redlib.tux.pizza"
|
||||
]
|
||||
|
||||
# Rate limiting
|
||||
REQUEST_TIMEOUT = 15
|
||||
COOLDOWN_SECONDS = 3
|
||||
RETRY_WAIT = 30
|
||||
|
||||
# Media settings
|
||||
MAX_IMAGES_PER_POST = 10
|
||||
MAX_VIDEOS_PER_POST = 2
|
||||
MAX_GALLERY_IMAGES = 15
|
||||
|
||||
# Comment settings
|
||||
MAX_COMMENT_DEPTH = 5
|
||||
|
||||
# --- ASYNC SETTINGS ---
|
||||
ASYNC_MAX_CONCURRENT = 10
|
||||
ASYNC_BATCH_SIZE = 50
|
||||
|
||||
# --- NOTIFICATION SETTINGS ---
|
||||
DISCORD_WEBHOOK_URL = os.getenv("DISCORD_WEBHOOK_URL", "")
|
||||
TELEGRAM_BOT_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "")
|
||||
TELEGRAM_CHAT_ID = os.getenv("TELEGRAM_CHAT_ID", "")
|
||||
|
||||
# --- DASHBOARD SETTINGS ---
|
||||
DASHBOARD_HOST = "0.0.0.0"
|
||||
DASHBOARD_PORT = 8501
|
||||
|
||||
# --- SCHEDULER SETTINGS ---
|
||||
SCHEDULER_TIMEZONE = "Asia/Kolkata"
|
||||
|
||||
# --- DATABASE SETTINGS ---
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", f"sqlite:///{DB_PATH}")
|
||||
|
||||
# Ensure data directory exists
|
||||
DATA_DIR.mkdir(exist_ok=True)
|
||||
1
dashboard/__init__.py
Normal file
1
dashboard/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
# Dashboard module
|
||||
364
dashboard/app.py
Normal file
364
dashboard/app.py
Normal file
|
|
@ -0,0 +1,364 @@
|
|||
"""
|
||||
Reddit Scraper Dashboard - Streamlit Web UI
|
||||
Run with: streamlit run dashboard/app.py
|
||||
"""
|
||||
import streamlit as st
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
# Add parent to path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from analytics.sentiment import (
|
||||
analyze_posts_sentiment, extract_keywords,
|
||||
calculate_engagement_metrics, find_best_posting_times
|
||||
)
|
||||
from search.query import search_all_data, advanced_search, get_top_posts
|
||||
|
||||
# Page config
|
||||
st.set_page_config(
|
||||
page_title="Reddit Scraper Dashboard",
|
||||
page_icon="🤖",
|
||||
layout="wide",
|
||||
initial_sidebar_state="expanded"
|
||||
)
|
||||
|
||||
# Custom CSS
|
||||
st.markdown("""
|
||||
<style>
|
||||
.main-header {
|
||||
font-size: 2.5rem;
|
||||
font-weight: 700;
|
||||
background: linear-gradient(90deg, #FF4500, #FF6B6B);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
.metric-card {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
padding: 1rem;
|
||||
border-radius: 10px;
|
||||
color: white;
|
||||
}
|
||||
.stTabs [data-baseweb="tab-list"] {
|
||||
gap: 24px;
|
||||
}
|
||||
.stTabs [data-baseweb="tab"] {
|
||||
height: 50px;
|
||||
padding: 10px 20px;
|
||||
background-color: #262730;
|
||||
border-radius: 5px;
|
||||
}
|
||||
</style>
|
||||
""", unsafe_allow_html=True)
|
||||
|
||||
def load_subreddit_data(subreddit_path):
|
||||
"""Load all data for a subreddit."""
|
||||
data = {}
|
||||
|
||||
posts_file = subreddit_path / 'posts.csv'
|
||||
if posts_file.exists():
|
||||
data['posts'] = pd.read_csv(posts_file)
|
||||
|
||||
comments_file = subreddit_path / 'comments.csv'
|
||||
if comments_file.exists():
|
||||
data['comments'] = pd.read_csv(comments_file)
|
||||
|
||||
return data
|
||||
|
||||
def get_available_subreddits():
|
||||
"""Get list of scraped subreddits."""
|
||||
data_dir = Path(__file__).parent.parent / 'data'
|
||||
subs = []
|
||||
|
||||
if data_dir.exists():
|
||||
for sub_dir in data_dir.iterdir():
|
||||
if sub_dir.is_dir() and (sub_dir / 'posts.csv').exists():
|
||||
subs.append(sub_dir.name)
|
||||
|
||||
return sorted(subs)
|
||||
|
||||
def main():
|
||||
# Header
|
||||
st.markdown('<h1 class="main-header">🤖 Reddit Scraper Dashboard</h1>', unsafe_allow_html=True)
|
||||
|
||||
# Sidebar
|
||||
st.sidebar.title("📊 Navigation")
|
||||
|
||||
# Get available subreddits
|
||||
subreddits = get_available_subreddits()
|
||||
|
||||
if not subreddits:
|
||||
st.warning("No scraped data found! Run the scraper first:")
|
||||
st.code("python main.py <subreddit> --mode full --limit 100")
|
||||
return
|
||||
|
||||
# Subreddit selector
|
||||
selected_sub = st.sidebar.selectbox(
|
||||
"Select Subreddit",
|
||||
subreddits,
|
||||
format_func=lambda x: f"📁 {x}"
|
||||
)
|
||||
|
||||
# Load data
|
||||
data_dir = Path(__file__).parent.parent / 'data'
|
||||
sub_path = data_dir / selected_sub
|
||||
data = load_subreddit_data(sub_path)
|
||||
|
||||
if 'posts' not in data:
|
||||
st.error("No posts data found!")
|
||||
return
|
||||
|
||||
posts_df = data['posts']
|
||||
comments_df = data.get('comments', pd.DataFrame())
|
||||
|
||||
# Main content tabs
|
||||
tab1, tab2, tab3, tab4, tab5 = st.tabs([
|
||||
"📊 Overview", "📈 Analytics", "🔍 Search", "💬 Comments", "⚙️ Scraper"
|
||||
])
|
||||
|
||||
with tab1:
|
||||
st.header(f"📊 Overview: {selected_sub}")
|
||||
|
||||
# Metrics row
|
||||
col1, col2, col3, col4, col5 = st.columns(5)
|
||||
|
||||
with col1:
|
||||
st.metric("Total Posts", len(posts_df))
|
||||
with col2:
|
||||
st.metric("Total Comments", len(comments_df))
|
||||
with col3:
|
||||
total_score = posts_df['score'].sum() if 'score' in posts_df else 0
|
||||
st.metric("Total Score", f"{total_score:,}")
|
||||
with col4:
|
||||
avg_score = posts_df['score'].mean() if 'score' in posts_df else 0
|
||||
st.metric("Avg Score", f"{avg_score:.1f}")
|
||||
with col5:
|
||||
media_count = posts_df['has_media'].sum() if 'has_media' in posts_df else 0
|
||||
st.metric("Media Posts", int(media_count))
|
||||
|
||||
st.divider()
|
||||
|
||||
# Post type distribution
|
||||
col1, col2 = st.columns(2)
|
||||
|
||||
with col1:
|
||||
st.subheader("📝 Post Types")
|
||||
if 'post_type' in posts_df:
|
||||
type_counts = posts_df['post_type'].value_counts()
|
||||
st.bar_chart(type_counts)
|
||||
|
||||
with col2:
|
||||
st.subheader("📅 Posts Over Time")
|
||||
if 'created_utc' in posts_df:
|
||||
posts_df['date'] = pd.to_datetime(posts_df['created_utc']).dt.date
|
||||
daily = posts_df.groupby('date').size()
|
||||
st.line_chart(daily)
|
||||
|
||||
st.divider()
|
||||
|
||||
# Top posts
|
||||
st.subheader("🔥 Top Posts by Score")
|
||||
if 'score' in posts_df:
|
||||
top_posts = posts_df.nlargest(10, 'score')[['title', 'score', 'num_comments', 'post_type', 'created_utc']]
|
||||
st.dataframe(top_posts, use_container_width=True)
|
||||
|
||||
with tab2:
|
||||
st.header("📈 Analytics")
|
||||
|
||||
# Sentiment Analysis
|
||||
st.subheader("😀 Sentiment Analysis")
|
||||
|
||||
if st.button("Run Sentiment Analysis"):
|
||||
with st.spinner("Analyzing sentiment..."):
|
||||
posts_list = posts_df.to_dict('records')
|
||||
analyzed_posts, sentiment_counts = analyze_posts_sentiment(posts_list)
|
||||
|
||||
col1, col2, col3 = st.columns(3)
|
||||
col1.metric("Positive", sentiment_counts['positive'], delta=None)
|
||||
col2.metric("Neutral", sentiment_counts['neutral'], delta=None)
|
||||
col3.metric("Negative", sentiment_counts['negative'], delta=None)
|
||||
|
||||
# Pie chart
|
||||
sentiment_df = pd.DataFrame({
|
||||
'Sentiment': ['Positive', 'Neutral', 'Negative'],
|
||||
'Count': [sentiment_counts['positive'], sentiment_counts['neutral'], sentiment_counts['negative']]
|
||||
})
|
||||
st.bar_chart(sentiment_df.set_index('Sentiment'))
|
||||
|
||||
st.divider()
|
||||
|
||||
# Keywords
|
||||
st.subheader("☁️ Top Keywords")
|
||||
texts = posts_df['title'].tolist()
|
||||
if 'selftext' in posts_df:
|
||||
texts.extend(posts_df['selftext'].dropna().tolist())
|
||||
|
||||
keywords = extract_keywords(texts, top_n=30)
|
||||
|
||||
if keywords:
|
||||
kw_df = pd.DataFrame(keywords, columns=['Word', 'Count'])
|
||||
st.bar_chart(kw_df.set_index('Word').head(20))
|
||||
|
||||
st.divider()
|
||||
|
||||
# Best posting times
|
||||
st.subheader("⏰ Best Posting Times")
|
||||
|
||||
if 'created_utc' in posts_df:
|
||||
timing_data = find_best_posting_times(posts_df.to_dict('records'))
|
||||
|
||||
if timing_data['best_hours']:
|
||||
st.write("**Best Hours to Post:**")
|
||||
for hour, avg_score in timing_data['best_hours']:
|
||||
st.write(f"• {hour}:00 - Avg Score: {avg_score:.1f}")
|
||||
|
||||
if timing_data['best_days']:
|
||||
st.write("**Best Days to Post:**")
|
||||
for day, avg_score in timing_data['best_days']:
|
||||
st.write(f"• {day} - Avg Score: {avg_score:.1f}")
|
||||
|
||||
with tab3:
|
||||
st.header("🔍 Search Posts")
|
||||
|
||||
# Search form
|
||||
col1, col2 = st.columns([3, 1])
|
||||
|
||||
with col1:
|
||||
search_query = st.text_input("Search query", placeholder="Enter keywords...")
|
||||
|
||||
with col2:
|
||||
min_score = st.number_input("Min Score", min_value=0, value=0)
|
||||
|
||||
col3, col4, col5 = st.columns(3)
|
||||
|
||||
with col3:
|
||||
if 'post_type' in posts_df:
|
||||
post_types = ['All'] + posts_df['post_type'].dropna().unique().tolist()
|
||||
selected_type = st.selectbox("Post Type", post_types)
|
||||
|
||||
with col4:
|
||||
if 'author' in posts_df:
|
||||
authors = ['All'] + posts_df['author'].dropna().unique().tolist()[:50]
|
||||
selected_author = st.selectbox("Author", authors)
|
||||
|
||||
with col5:
|
||||
sort_by = st.selectbox("Sort by", ['score', 'num_comments', 'created_utc'])
|
||||
|
||||
# Search button
|
||||
if st.button("🔍 Search"):
|
||||
filtered = posts_df.copy()
|
||||
|
||||
if search_query:
|
||||
mask = filtered['title'].str.contains(search_query, case=False, na=False)
|
||||
if 'selftext' in filtered:
|
||||
mask |= filtered['selftext'].str.contains(search_query, case=False, na=False)
|
||||
filtered = filtered[mask]
|
||||
|
||||
if min_score > 0:
|
||||
filtered = filtered[filtered['score'] >= min_score]
|
||||
|
||||
if selected_type != 'All' and 'post_type' in filtered:
|
||||
filtered = filtered[filtered['post_type'] == selected_type]
|
||||
|
||||
if selected_author != 'All' and 'author' in filtered:
|
||||
filtered = filtered[filtered['author'] == selected_author]
|
||||
|
||||
filtered = filtered.sort_values(sort_by, ascending=False)
|
||||
|
||||
st.write(f"Found {len(filtered)} results")
|
||||
st.dataframe(filtered[['title', 'score', 'num_comments', 'post_type', 'author', 'created_utc']].head(50), use_container_width=True)
|
||||
|
||||
with tab4:
|
||||
st.header("💬 Comments Analysis")
|
||||
|
||||
if len(comments_df) == 0:
|
||||
st.warning("No comments data found for this subreddit")
|
||||
else:
|
||||
col1, col2, col3 = st.columns(3)
|
||||
|
||||
with col1:
|
||||
st.metric("Total Comments", len(comments_df))
|
||||
with col2:
|
||||
avg_score = comments_df['score'].mean() if 'score' in comments_df else 0
|
||||
st.metric("Avg Score", f"{avg_score:.1f}")
|
||||
with col3:
|
||||
unique_authors = comments_df['author'].nunique() if 'author' in comments_df else 0
|
||||
st.metric("Unique Commenters", unique_authors)
|
||||
|
||||
st.divider()
|
||||
|
||||
# Top comments
|
||||
st.subheader("🔥 Top Comments by Score")
|
||||
if 'score' in comments_df:
|
||||
top_comments = comments_df.nlargest(10, 'score')[['body', 'score', 'author', 'created_utc']]
|
||||
for _, row in top_comments.iterrows():
|
||||
with st.expander(f"⬆️ {row['score']} - by u/{row['author']}"):
|
||||
st.write(row['body'][:500])
|
||||
|
||||
st.divider()
|
||||
|
||||
# Top commenters
|
||||
st.subheader("👥 Top Commenters")
|
||||
if 'author' in comments_df:
|
||||
top_authors = comments_df['author'].value_counts().head(10)
|
||||
st.bar_chart(top_authors)
|
||||
|
||||
with tab5:
|
||||
st.header("⚙️ Scraper Controls")
|
||||
|
||||
st.subheader("🚀 Start New Scrape")
|
||||
|
||||
col1, col2 = st.columns(2)
|
||||
|
||||
with col1:
|
||||
new_sub = st.text_input("Subreddit/User name", placeholder="e.g. python")
|
||||
is_user = st.checkbox("Is a User (not subreddit)")
|
||||
|
||||
with col2:
|
||||
limit = st.number_input("Post Limit", min_value=10, max_value=5000, value=100)
|
||||
mode = st.selectbox("Mode", ['full', 'history'])
|
||||
|
||||
no_media = st.checkbox("Skip media download")
|
||||
no_comments = st.checkbox("Skip comments")
|
||||
|
||||
if st.button("🚀 Start Scraping"):
|
||||
st.info(f"Run this command in terminal:")
|
||||
cmd = f"python main.py {new_sub} --mode {mode} --limit {limit}"
|
||||
if is_user:
|
||||
cmd += " --user"
|
||||
if no_media:
|
||||
cmd += " --no-media"
|
||||
if no_comments:
|
||||
cmd += " --no-comments"
|
||||
st.code(cmd)
|
||||
|
||||
st.divider()
|
||||
|
||||
# Export options
|
||||
st.subheader("📤 Export Data")
|
||||
|
||||
export_format = st.selectbox("Format", ['CSV', 'JSON', 'Excel'])
|
||||
|
||||
if st.button("📥 Download Posts"):
|
||||
if export_format == 'CSV':
|
||||
csv = posts_df.to_csv(index=False)
|
||||
st.download_button(
|
||||
"Download CSV",
|
||||
csv,
|
||||
f"{selected_sub}_posts.csv",
|
||||
"text/csv"
|
||||
)
|
||||
elif export_format == 'JSON':
|
||||
json_data = posts_df.to_json(orient='records', indent=2)
|
||||
st.download_button(
|
||||
"Download JSON",
|
||||
json_data,
|
||||
f"{selected_sub}_posts.json",
|
||||
"application/json"
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
2
export/__init__.py
Normal file
2
export/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# Export module
|
||||
from .database import *
|
||||
389
export/database.py
Normal file
389
export/database.py
Normal file
|
|
@ -0,0 +1,389 @@
|
|||
"""
|
||||
Database module - SQLite storage for scraped data
|
||||
"""
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
import json
|
||||
import sys
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
from config import DB_PATH, DATA_DIR
|
||||
|
||||
def get_connection():
|
||||
"""Get database connection."""
|
||||
DATA_DIR.mkdir(exist_ok=True)
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
def init_database():
|
||||
"""Initialize database tables."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Posts table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS posts (
|
||||
id TEXT PRIMARY KEY,
|
||||
subreddit TEXT,
|
||||
title TEXT,
|
||||
author TEXT,
|
||||
created_utc TEXT,
|
||||
permalink TEXT UNIQUE,
|
||||
url TEXT,
|
||||
score INTEGER DEFAULT 0,
|
||||
upvote_ratio REAL DEFAULT 0,
|
||||
num_comments INTEGER DEFAULT 0,
|
||||
num_crossposts INTEGER DEFAULT 0,
|
||||
selftext TEXT,
|
||||
post_type TEXT,
|
||||
is_nsfw BOOLEAN DEFAULT 0,
|
||||
is_spoiler BOOLEAN DEFAULT 0,
|
||||
flair TEXT,
|
||||
total_awards INTEGER DEFAULT 0,
|
||||
has_media BOOLEAN DEFAULT 0,
|
||||
media_downloaded BOOLEAN DEFAULT 0,
|
||||
source TEXT,
|
||||
scraped_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
sentiment_score REAL,
|
||||
sentiment_label TEXT
|
||||
)
|
||||
""")
|
||||
|
||||
# Comments table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS comments (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
comment_id TEXT UNIQUE,
|
||||
post_id TEXT,
|
||||
post_permalink TEXT,
|
||||
parent_id TEXT,
|
||||
author TEXT,
|
||||
body TEXT,
|
||||
score INTEGER DEFAULT 0,
|
||||
created_utc TEXT,
|
||||
depth INTEGER DEFAULT 0,
|
||||
is_submitter BOOLEAN DEFAULT 0,
|
||||
scraped_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
sentiment_score REAL,
|
||||
sentiment_label TEXT,
|
||||
FOREIGN KEY (post_id) REFERENCES posts(id)
|
||||
)
|
||||
""")
|
||||
|
||||
# Subreddits table (for tracking)
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS subreddits (
|
||||
name TEXT PRIMARY KEY,
|
||||
last_scraped TEXT,
|
||||
total_posts INTEGER DEFAULT 0,
|
||||
total_comments INTEGER DEFAULT 0,
|
||||
total_media INTEGER DEFAULT 0
|
||||
)
|
||||
""")
|
||||
|
||||
# Scheduled jobs table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS scheduled_jobs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
target TEXT,
|
||||
is_user BOOLEAN DEFAULT 0,
|
||||
mode TEXT DEFAULT 'full',
|
||||
limit_posts INTEGER DEFAULT 100,
|
||||
cron_expression TEXT,
|
||||
last_run TEXT,
|
||||
next_run TEXT,
|
||||
enabled BOOLEAN DEFAULT 1,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
# Alerts table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS alerts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
keyword TEXT,
|
||||
subreddit TEXT,
|
||||
alert_type TEXT DEFAULT 'discord',
|
||||
webhook_url TEXT,
|
||||
enabled BOOLEAN DEFAULT 1,
|
||||
last_triggered TEXT,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
# Create indexes
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON posts(subreddit)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_created ON posts(created_utc)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_posts_score ON posts(score)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_post ON comments(post_id)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS idx_comments_author ON comments(author)")
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print("✅ Database initialized")
|
||||
|
||||
def save_post(post_data, subreddit):
|
||||
"""Save a single post to database."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
cursor.execute("""
|
||||
INSERT OR REPLACE INTO posts
|
||||
(id, subreddit, title, author, created_utc, permalink, url, score,
|
||||
upvote_ratio, num_comments, num_crossposts, selftext, post_type,
|
||||
is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
post_data.get('id'),
|
||||
subreddit,
|
||||
post_data.get('title'),
|
||||
post_data.get('author'),
|
||||
post_data.get('created_utc'),
|
||||
post_data.get('permalink'),
|
||||
post_data.get('url'),
|
||||
post_data.get('score', 0),
|
||||
post_data.get('upvote_ratio', 0),
|
||||
post_data.get('num_comments', 0),
|
||||
post_data.get('num_crossposts', 0),
|
||||
post_data.get('selftext', ''),
|
||||
post_data.get('post_type'),
|
||||
post_data.get('is_nsfw', False),
|
||||
post_data.get('is_spoiler', False),
|
||||
post_data.get('flair', ''),
|
||||
post_data.get('total_awards', 0),
|
||||
post_data.get('has_media', False),
|
||||
post_data.get('media_downloaded', False),
|
||||
post_data.get('source', '')
|
||||
))
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"DB Error: {e}")
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save_posts_batch(posts, subreddit):
|
||||
"""Save multiple posts efficiently."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
saved = 0
|
||||
|
||||
for post in posts:
|
||||
try:
|
||||
cursor.execute("""
|
||||
INSERT OR IGNORE INTO posts
|
||||
(id, subreddit, title, author, created_utc, permalink, url, score,
|
||||
upvote_ratio, num_comments, num_crossposts, selftext, post_type,
|
||||
is_nsfw, is_spoiler, flair, total_awards, has_media, media_downloaded, source)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
post.get('id'),
|
||||
subreddit,
|
||||
post.get('title'),
|
||||
post.get('author'),
|
||||
post.get('created_utc'),
|
||||
post.get('permalink'),
|
||||
post.get('url'),
|
||||
post.get('score', 0),
|
||||
post.get('upvote_ratio', 0),
|
||||
post.get('num_comments', 0),
|
||||
post.get('num_crossposts', 0),
|
||||
post.get('selftext', ''),
|
||||
post.get('post_type'),
|
||||
post.get('is_nsfw', False),
|
||||
post.get('is_spoiler', False),
|
||||
post.get('flair', ''),
|
||||
post.get('total_awards', 0),
|
||||
post.get('has_media', False),
|
||||
post.get('media_downloaded', False),
|
||||
post.get('source', '')
|
||||
))
|
||||
if cursor.rowcount > 0:
|
||||
saved += 1
|
||||
except:
|
||||
continue
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return saved
|
||||
|
||||
def save_comments_batch(comments, post_id):
|
||||
"""Save multiple comments efficiently."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
saved = 0
|
||||
|
||||
for comment in comments:
|
||||
try:
|
||||
cursor.execute("""
|
||||
INSERT OR IGNORE INTO comments
|
||||
(comment_id, post_id, post_permalink, parent_id, author, body,
|
||||
score, created_utc, depth, is_submitter)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
comment.get('comment_id'),
|
||||
post_id,
|
||||
comment.get('post_permalink'),
|
||||
comment.get('parent_id'),
|
||||
comment.get('author'),
|
||||
comment.get('body'),
|
||||
comment.get('score', 0),
|
||||
comment.get('created_utc'),
|
||||
comment.get('depth', 0),
|
||||
comment.get('is_submitter', False)
|
||||
))
|
||||
if cursor.rowcount > 0:
|
||||
saved += 1
|
||||
except:
|
||||
continue
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
return saved
|
||||
|
||||
def search_posts(query=None, subreddit=None, author=None, min_score=None,
|
||||
start_date=None, end_date=None, post_type=None, limit=100):
|
||||
"""Search posts with filters."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
sql = "SELECT * FROM posts WHERE 1=1"
|
||||
params = []
|
||||
|
||||
if query:
|
||||
sql += " AND (title LIKE ? OR selftext LIKE ?)"
|
||||
params.extend([f"%{query}%", f"%{query}%"])
|
||||
|
||||
if subreddit:
|
||||
sql += " AND subreddit = ?"
|
||||
params.append(subreddit)
|
||||
|
||||
if author:
|
||||
sql += " AND author = ?"
|
||||
params.append(author)
|
||||
|
||||
if min_score:
|
||||
sql += " AND score >= ?"
|
||||
params.append(min_score)
|
||||
|
||||
if start_date:
|
||||
sql += " AND created_utc >= ?"
|
||||
params.append(start_date)
|
||||
|
||||
if end_date:
|
||||
sql += " AND created_utc <= ?"
|
||||
params.append(end_date)
|
||||
|
||||
if post_type:
|
||||
sql += " AND post_type = ?"
|
||||
params.append(post_type)
|
||||
|
||||
sql += " ORDER BY created_utc DESC LIMIT ?"
|
||||
params.append(limit)
|
||||
|
||||
cursor.execute(sql, params)
|
||||
results = [dict(row) for row in cursor.fetchall()]
|
||||
conn.close()
|
||||
return results
|
||||
|
||||
def search_comments(query=None, post_id=None, author=None, min_score=None, limit=100):
|
||||
"""Search comments with filters."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
sql = "SELECT * FROM comments WHERE 1=1"
|
||||
params = []
|
||||
|
||||
if query:
|
||||
sql += " AND body LIKE ?"
|
||||
params.append(f"%{query}%")
|
||||
|
||||
if post_id:
|
||||
sql += " AND post_id = ?"
|
||||
params.append(post_id)
|
||||
|
||||
if author:
|
||||
sql += " AND author = ?"
|
||||
params.append(author)
|
||||
|
||||
if min_score:
|
||||
sql += " AND score >= ?"
|
||||
params.append(min_score)
|
||||
|
||||
sql += " ORDER BY score DESC LIMIT ?"
|
||||
params.append(limit)
|
||||
|
||||
cursor.execute(sql, params)
|
||||
results = [dict(row) for row in cursor.fetchall()]
|
||||
conn.close()
|
||||
return results
|
||||
|
||||
def get_subreddit_stats(subreddit):
|
||||
"""Get statistics for a subreddit."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
stats = {}
|
||||
|
||||
# Post stats
|
||||
cursor.execute("""
|
||||
SELECT
|
||||
COUNT(*) as total_posts,
|
||||
AVG(score) as avg_score,
|
||||
MAX(score) as max_score,
|
||||
SUM(num_comments) as total_comments,
|
||||
AVG(upvote_ratio) as avg_upvote_ratio
|
||||
FROM posts WHERE subreddit = ?
|
||||
""", (subreddit,))
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
stats.update(dict(row))
|
||||
|
||||
# Post type distribution
|
||||
cursor.execute("""
|
||||
SELECT post_type, COUNT(*) as count
|
||||
FROM posts WHERE subreddit = ?
|
||||
GROUP BY post_type
|
||||
""", (subreddit,))
|
||||
stats['post_types'] = {row['post_type']: row['count'] for row in cursor.fetchall()}
|
||||
|
||||
# Top authors
|
||||
cursor.execute("""
|
||||
SELECT author, COUNT(*) as post_count, SUM(score) as total_score
|
||||
FROM posts WHERE subreddit = ? AND author != '[deleted]'
|
||||
GROUP BY author ORDER BY post_count DESC LIMIT 10
|
||||
""", (subreddit,))
|
||||
stats['top_authors'] = [dict(row) for row in cursor.fetchall()]
|
||||
|
||||
# Activity by hour
|
||||
cursor.execute("""
|
||||
SELECT strftime('%H', created_utc) as hour, COUNT(*) as count
|
||||
FROM posts WHERE subreddit = ?
|
||||
GROUP BY hour ORDER BY hour
|
||||
""", (subreddit,))
|
||||
stats['hourly_activity'] = {row['hour']: row['count'] for row in cursor.fetchall()}
|
||||
|
||||
conn.close()
|
||||
return stats
|
||||
|
||||
def get_all_subreddits():
|
||||
"""Get list of all scraped subreddits."""
|
||||
conn = get_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute("""
|
||||
SELECT subreddit, COUNT(*) as post_count,
|
||||
MAX(created_utc) as latest_post,
|
||||
MIN(created_utc) as oldest_post
|
||||
FROM posts GROUP BY subreddit ORDER BY post_count DESC
|
||||
""")
|
||||
|
||||
results = [dict(row) for row in cursor.fetchall()]
|
||||
conn.close()
|
||||
return results
|
||||
|
||||
# Initialize on import
|
||||
init_database()
|
||||
228
main.py
228
main.py
|
|
@ -1,3 +1,7 @@
|
|||
"""
|
||||
🤖 Universal Reddit Scraper Suite
|
||||
Full-featured scraper with analytics, dashboard, notifications, and scheduling.
|
||||
"""
|
||||
import requests
|
||||
import pandas as pd
|
||||
import datetime
|
||||
|
|
@ -8,14 +12,12 @@ import argparse
|
|||
import random
|
||||
import sys
|
||||
import json
|
||||
import re
|
||||
from urllib.parse import urlparse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
# --- CONFIGURATION ---
|
||||
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
|
||||
# Sources: old.reddit.com for residential IPs, mirrors for data centers
|
||||
MIRRORS = [
|
||||
"https://old.reddit.com",
|
||||
"https://redlib.catsarch.com",
|
||||
|
|
@ -108,16 +110,13 @@ def get_media_urls(post_data):
|
|||
"""Extracts all media URLs from a post."""
|
||||
media = {"images": [], "videos": [], "galleries": []}
|
||||
|
||||
# Direct image link
|
||||
url = post_data.get('url', '')
|
||||
if any(ext in url.lower() for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp']):
|
||||
media["images"].append(url)
|
||||
|
||||
# Reddit-hosted image
|
||||
if 'i.redd.it' in url:
|
||||
media["images"].append(url)
|
||||
|
||||
# Reddit video
|
||||
if post_data.get('is_video'):
|
||||
reddit_video = post_data.get('media', {})
|
||||
if reddit_video and 'reddit_video' in reddit_video:
|
||||
|
|
@ -125,17 +124,14 @@ def get_media_urls(post_data):
|
|||
if video_url:
|
||||
media["videos"].append(video_url.split('?')[0])
|
||||
|
||||
# Preview images
|
||||
preview = post_data.get('preview', {})
|
||||
if preview and 'images' in preview:
|
||||
for img in preview['images']:
|
||||
source = img.get('source', {})
|
||||
if source.get('url'):
|
||||
# Unescape HTML entities
|
||||
clean_url = source['url'].replace('&', '&')
|
||||
media["images"].append(clean_url)
|
||||
|
||||
# Gallery posts
|
||||
if post_data.get('is_gallery'):
|
||||
gallery_data = post_data.get('gallery_data', {})
|
||||
media_metadata = post_data.get('media_metadata', {})
|
||||
|
|
@ -149,7 +145,6 @@ def get_media_urls(post_data):
|
|||
clean_url = meta['s']['u'].replace('&', '&')
|
||||
media["galleries"].append(clean_url)
|
||||
|
||||
# External video (YouTube, etc.)
|
||||
if 'youtube.com' in url or 'youtu.be' in url:
|
||||
media["videos"].append(url)
|
||||
|
||||
|
|
@ -158,7 +153,6 @@ def get_media_urls(post_data):
|
|||
def download_media(url, save_path, media_type="image"):
|
||||
"""Downloads a single media file."""
|
||||
try:
|
||||
# Skip if already downloaded
|
||||
if os.path.exists(save_path):
|
||||
return True
|
||||
|
||||
|
|
@ -169,7 +163,7 @@ def download_media(url, save_path, media_type="image"):
|
|||
f.write(chunk)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"⚠️ Failed to download {media_type}: {e}")
|
||||
pass
|
||||
return False
|
||||
|
||||
def download_post_media(post_data, dirs, post_id):
|
||||
|
|
@ -177,23 +171,20 @@ def download_post_media(post_data, dirs, post_id):
|
|||
media = get_media_urls(post_data)
|
||||
downloaded = {"images": 0, "videos": 0}
|
||||
|
||||
# Download images
|
||||
for i, img_url in enumerate(media["images"][:5]): # Limit to 5 images per post
|
||||
for i, img_url in enumerate(media["images"][:5]):
|
||||
ext = os.path.splitext(urlparse(img_url).path)[1] or '.jpg'
|
||||
save_path = os.path.join(dirs["images"], f"{post_id}_{i}{ext}")
|
||||
if download_media(img_url, save_path, "image"):
|
||||
downloaded["images"] += 1
|
||||
|
||||
# Download gallery images
|
||||
for i, img_url in enumerate(media["galleries"][:10]): # Limit gallery to 10
|
||||
for i, img_url in enumerate(media["galleries"][:10]):
|
||||
ext = '.jpg'
|
||||
save_path = os.path.join(dirs["images"], f"{post_id}_gallery_{i}{ext}")
|
||||
if download_media(img_url, save_path, "gallery"):
|
||||
downloaded["images"] += 1
|
||||
|
||||
# Download videos
|
||||
for i, vid_url in enumerate(media["videos"][:2]): # Limit to 2 videos
|
||||
if 'youtube' not in vid_url: # Skip YouTube (can't direct download)
|
||||
for i, vid_url in enumerate(media["videos"][:2]):
|
||||
if 'youtube' not in vid_url:
|
||||
ext = '.mp4'
|
||||
save_path = os.path.join(dirs["videos"], f"{post_id}_{i}{ext}")
|
||||
if download_media(vid_url, save_path, "video"):
|
||||
|
|
@ -203,11 +194,10 @@ def download_post_media(post_data, dirs, post_id):
|
|||
|
||||
# --- COMMENT SCRAPING ---
|
||||
def scrape_comments(permalink, max_depth=3):
|
||||
"""Scrapes comments from a post using Reddit JSON endpoint."""
|
||||
"""Scrapes comments from a post."""
|
||||
comments = []
|
||||
|
||||
try:
|
||||
# Clean permalink and build URL
|
||||
if not permalink.startswith('http'):
|
||||
url = f"https://old.reddit.com{permalink}.json?limit=100"
|
||||
else:
|
||||
|
|
@ -219,13 +209,12 @@ def scrape_comments(permalink, max_depth=3):
|
|||
|
||||
data = response.json()
|
||||
|
||||
# Comments are in the second element of the response
|
||||
if len(data) > 1:
|
||||
comment_data = data[1]['data']['children']
|
||||
comments = parse_comments(comment_data, permalink, depth=0, max_depth=max_depth)
|
||||
|
||||
except Exception as e:
|
||||
print(f"⚠️ Comment fetch error: {e}")
|
||||
pass
|
||||
|
||||
return comments
|
||||
|
||||
|
|
@ -237,7 +226,7 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
|
|||
return comments
|
||||
|
||||
for item in comment_list:
|
||||
if item['kind'] != 't1': # Skip non-comment items
|
||||
if item['kind'] != 't1':
|
||||
continue
|
||||
|
||||
c = item['data']
|
||||
|
|
@ -255,7 +244,6 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
|
|||
}
|
||||
comments.append(comment)
|
||||
|
||||
# Parse replies recursively
|
||||
replies = c.get('replies')
|
||||
if replies and isinstance(replies, dict):
|
||||
reply_children = replies.get('data', {}).get('children', [])
|
||||
|
|
@ -263,12 +251,11 @@ def parse_comments(comment_list, post_permalink, depth=0, max_depth=3):
|
|||
|
||||
return comments
|
||||
|
||||
# --- ENHANCED POST EXTRACTION ---
|
||||
# --- POST EXTRACTION ---
|
||||
def extract_post_data(post_json):
|
||||
"""Extracts comprehensive post data."""
|
||||
p = post_json
|
||||
|
||||
# Determine post type
|
||||
post_type = "text"
|
||||
if p.get('is_video'):
|
||||
post_type = "video"
|
||||
|
|
@ -282,39 +269,28 @@ def extract_post_data(post_json):
|
|||
post_type = "link"
|
||||
|
||||
return {
|
||||
# Basic Info
|
||||
"id": p.get('id'),
|
||||
"title": p.get('title'),
|
||||
"author": p.get('author'),
|
||||
"created_utc": datetime.datetime.fromtimestamp(p.get('created_utc', 0)).isoformat(),
|
||||
"permalink": p.get('permalink'),
|
||||
"url": p.get('url_overridden_by_dest', p.get('url')),
|
||||
|
||||
# Engagement
|
||||
"score": p.get('score', 0),
|
||||
"upvote_ratio": p.get('upvote_ratio', 0),
|
||||
"num_comments": p.get('num_comments', 0),
|
||||
"num_crossposts": p.get('num_crossposts', 0),
|
||||
|
||||
# Content
|
||||
"selftext": p.get('selftext', ''),
|
||||
"post_type": post_type,
|
||||
"is_nsfw": p.get('over_18', False),
|
||||
"is_spoiler": p.get('spoiler', False),
|
||||
|
||||
# Flair & Awards
|
||||
"flair": p.get('link_flair_text', ''),
|
||||
"total_awards": p.get('total_awards_received', 0),
|
||||
|
||||
# Media flags
|
||||
"has_media": p.get('is_video', False) or p.get('is_gallery', False) or 'i.redd.it' in p.get('url', ''),
|
||||
"media_downloaded": False,
|
||||
|
||||
# Source tracking
|
||||
"source": "History-Full"
|
||||
}
|
||||
|
||||
# --- MODE 2: FULL HISTORY SCRAPE ---
|
||||
# --- FULL HISTORY SCRAPE ---
|
||||
def run_full_history(target, limit, is_user=False, download_media_flag=True, scrape_comments_flag=True):
|
||||
"""Full scrape with images, videos, and comments."""
|
||||
prefix = "u" if is_user else "r"
|
||||
|
|
@ -331,6 +307,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
total_posts = 0
|
||||
total_media = {"images": 0, "videos": 0}
|
||||
total_comments = 0
|
||||
start_time = time.time()
|
||||
|
||||
while total_posts < limit:
|
||||
random.shuffle(MIRRORS)
|
||||
|
|
@ -362,11 +339,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
p = child['data']
|
||||
post = extract_post_data(p)
|
||||
|
||||
# Skip if already seen
|
||||
if post['permalink'] in SEEN_URLS:
|
||||
continue
|
||||
|
||||
# Download media
|
||||
if download_media_flag:
|
||||
downloaded = download_post_media(p, dirs, post['id'])
|
||||
post['media_downloaded'] = downloaded['images'] > 0 or downloaded['videos'] > 0
|
||||
|
|
@ -375,22 +350,19 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
|
||||
posts.append(post)
|
||||
|
||||
# Scrape comments
|
||||
if scrape_comments_flag and post['num_comments'] > 0:
|
||||
print(f" 💬 Fetching comments for: {post['title'][:40]}...")
|
||||
comments = scrape_comments(post['permalink'])
|
||||
all_comments.extend(comments)
|
||||
total_comments += len(comments)
|
||||
time.sleep(1) # Rate limiting for comment fetches
|
||||
time.sleep(1)
|
||||
|
||||
# Save data
|
||||
saved = save_posts_csv(posts, dirs["posts"])
|
||||
total_posts += saved
|
||||
|
||||
if all_comments:
|
||||
save_comments_csv(all_comments, dirs["comments"])
|
||||
|
||||
# Progress update
|
||||
print(f"\n📊 Progress: {total_posts}/{limit} posts")
|
||||
print(f" 🖼️ Images: {total_media['images']} | 🎬 Videos: {total_media['videos']}")
|
||||
print(f" 💬 Comments: {total_comments}")
|
||||
|
|
@ -398,7 +370,7 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
after = data['data'].get('after')
|
||||
if not after:
|
||||
print("\n🏁 Reached end of available history.")
|
||||
return
|
||||
break
|
||||
|
||||
success = True
|
||||
break
|
||||
|
|
@ -407,6 +379,9 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
print(f" ⚠️ Error with {base_url}: {e}")
|
||||
continue
|
||||
|
||||
if not after:
|
||||
break
|
||||
|
||||
if not success:
|
||||
print("\n❌ All sources failed. Waiting 30s...")
|
||||
time.sleep(30)
|
||||
|
|
@ -414,6 +389,8 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
print(f"\n⏸️ Cooling down (3s)...")
|
||||
time.sleep(3)
|
||||
|
||||
duration = time.time() - start_time
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("✅ SCRAPE COMPLETE!")
|
||||
print(f" 📁 Data saved to: {dirs['base']}")
|
||||
|
|
@ -421,8 +398,17 @@ def run_full_history(target, limit, is_user=False, download_media_flag=True, scr
|
|||
print(f" 🖼️ Total images: {total_media['images']}")
|
||||
print(f" 🎬 Total videos: {total_media['videos']}")
|
||||
print(f" 💬 Total comments: {total_comments}")
|
||||
print(f" ⏱️ Duration: {duration:.1f}s")
|
||||
|
||||
return {
|
||||
'posts': total_posts,
|
||||
'images': total_media['images'],
|
||||
'videos': total_media['videos'],
|
||||
'comments': total_comments,
|
||||
'duration': f"{duration:.1f}s"
|
||||
}
|
||||
|
||||
# --- MODE 1: LIVE MONITOR (RSS) - Legacy ---
|
||||
# --- MONITOR MODE ---
|
||||
def run_monitor(target, is_user=False):
|
||||
prefix = "u" if is_user else "r"
|
||||
if is_user:
|
||||
|
|
@ -437,7 +423,6 @@ def run_monitor(target, is_user=False):
|
|||
|
||||
if response.status_code != 200:
|
||||
print(f"❌ RSS blocked (Status {response.status_code}), trying JSON...")
|
||||
# Fallback to JSON
|
||||
run_full_history(target, 25, is_user, download_media_flag=False, scrape_comments_flag=False)
|
||||
return
|
||||
|
||||
|
|
@ -474,33 +459,144 @@ def run_monitor(target, is_user=False):
|
|||
except Exception as e:
|
||||
print(f"❌ Monitor Error: {e}")
|
||||
|
||||
# --- CLI ARGS ---
|
||||
if __name__ == "__main__":
|
||||
# --- CLI ---
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="🤖 Universal Reddit Scraper - Full Media & Comments Support",
|
||||
description="🤖 Universal Reddit Scraper Suite",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
Examples:
|
||||
python main.py delhi --mode full --limit 100
|
||||
python main.py spez --user --mode full --limit 50
|
||||
python main.py python --mode full --no-media --limit 200
|
||||
python main.py india --mode monitor
|
||||
Commands:
|
||||
SCRAPING:
|
||||
python main.py <target> --mode full --limit 100
|
||||
python main.py <target> --mode history --limit 500
|
||||
python main.py <target> --mode monitor
|
||||
|
||||
SEARCH:
|
||||
python main.py --search "keyword" --subreddit delhi
|
||||
python main.py --search "keyword" --min-score 100
|
||||
|
||||
DASHBOARD:
|
||||
python main.py --dashboard
|
||||
|
||||
SCHEDULE:
|
||||
python main.py --schedule delhi --every 60
|
||||
|
||||
ANALYTICS:
|
||||
python main.py --analyze delhi --sentiment
|
||||
python main.py --analyze delhi --keywords
|
||||
"""
|
||||
)
|
||||
parser.add_argument("target", help="Subreddit name (e.g. 'delhi') or Username (e.g. 'spez')")
|
||||
parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full",
|
||||
help="monitor=live RSS, history=posts only, full=posts+media+comments")
|
||||
parser.add_argument("--user", action="store_true", help="Target is a User, not Subreddit")
|
||||
|
||||
# Scraping args
|
||||
parser.add_argument("target", nargs='?', help="Subreddit or username to scrape")
|
||||
parser.add_argument("--mode", choices=["monitor", "history", "full"], default="full")
|
||||
parser.add_argument("--user", action="store_true", help="Target is a user")
|
||||
parser.add_argument("--limit", type=int, default=100, help="Max posts to scrape")
|
||||
parser.add_argument("--no-media", action="store_true", help="Skip downloading images/videos")
|
||||
parser.add_argument("--no-comments", action="store_true", help="Skip scraping comments")
|
||||
parser.add_argument("--no-media", action="store_true", help="Skip media download")
|
||||
parser.add_argument("--no-comments", action="store_true", help="Skip comments")
|
||||
|
||||
# Dashboard
|
||||
parser.add_argument("--dashboard", action="store_true", help="Launch web dashboard")
|
||||
|
||||
# Search
|
||||
parser.add_argument("--search", type=str, help="Search scraped data")
|
||||
parser.add_argument("--subreddit", type=str, help="Filter by subreddit")
|
||||
parser.add_argument("--min-score", type=int, help="Filter by minimum score")
|
||||
parser.add_argument("--author", type=str, help="Filter by author")
|
||||
|
||||
# Analytics
|
||||
parser.add_argument("--analyze", type=str, help="Run analytics on subreddit")
|
||||
parser.add_argument("--sentiment", action="store_true", help="Run sentiment analysis")
|
||||
parser.add_argument("--keywords", action="store_true", help="Extract keywords")
|
||||
|
||||
# Schedule
|
||||
parser.add_argument("--schedule", type=str, help="Schedule scraping for target")
|
||||
parser.add_argument("--every", type=int, help="Interval in minutes")
|
||||
|
||||
# Alerts
|
||||
parser.add_argument("--alert", type=str, help="Set keyword alert")
|
||||
parser.add_argument("--discord-webhook", type=str, help="Discord webhook URL")
|
||||
parser.add_argument("--telegram-token", type=str, help="Telegram bot token")
|
||||
parser.add_argument("--telegram-chat", type=str, help="Telegram chat ID")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
print("=" * 50)
|
||||
print("🤖 UNIVERSAL REDDIT SCRAPER")
|
||||
print("🤖 UNIVERSAL REDDIT SCRAPER SUITE")
|
||||
print("=" * 50)
|
||||
|
||||
# Dashboard mode
|
||||
if args.dashboard:
|
||||
print("\n🌐 Launching Dashboard...")
|
||||
print(" Open: http://localhost:8501")
|
||||
os.system("streamlit run dashboard/app.py")
|
||||
return
|
||||
|
||||
# Search mode
|
||||
if args.search:
|
||||
print(f"\n🔍 Searching for: {args.search}")
|
||||
from search.query import search_all_data, print_search_results
|
||||
|
||||
results = search_all_data(
|
||||
query=args.search,
|
||||
min_score=args.min_score,
|
||||
author=args.author
|
||||
)
|
||||
print_search_results(results)
|
||||
return
|
||||
|
||||
# Analytics mode
|
||||
if args.analyze:
|
||||
print(f"\n📊 Analyzing: {args.analyze}")
|
||||
|
||||
# Load data
|
||||
data_dir = Path(f"data/r_{args.analyze}")
|
||||
if not data_dir.exists():
|
||||
print(f"❌ No data found for r/{args.analyze}")
|
||||
return
|
||||
|
||||
posts_file = data_dir / "posts.csv"
|
||||
if not posts_file.exists():
|
||||
print(f"❌ No posts data found")
|
||||
return
|
||||
|
||||
import pandas as pd
|
||||
df = pd.read_csv(posts_file)
|
||||
posts = df.to_dict('records')
|
||||
|
||||
if args.sentiment:
|
||||
from analytics.sentiment import analyze_posts_sentiment
|
||||
analyzed, counts = analyze_posts_sentiment(posts)
|
||||
print(f"\n😀 Sentiment Analysis:")
|
||||
print(f" Positive: {counts['positive']}")
|
||||
print(f" Neutral: {counts['neutral']}")
|
||||
print(f" Negative: {counts['negative']}")
|
||||
|
||||
if args.keywords:
|
||||
from analytics.sentiment import extract_keywords
|
||||
texts = [str(p.get('title', '') or '') + ' ' + str(p.get('selftext', '') or '') for p in posts]
|
||||
keywords = extract_keywords(texts, top_n=20)
|
||||
print(f"\n☁️ Top Keywords:")
|
||||
for word, count in keywords:
|
||||
print(f" {word}: {count}")
|
||||
|
||||
return
|
||||
|
||||
# Schedule mode
|
||||
if args.schedule:
|
||||
if not args.every:
|
||||
print("❌ Please specify --every <minutes>")
|
||||
return
|
||||
|
||||
from scheduler.cron import run_scheduled
|
||||
run_scheduled(args.schedule, args.every, args.mode, args.limit, args.user)
|
||||
return
|
||||
|
||||
# Regular scraping mode
|
||||
if not args.target:
|
||||
parser.print_help()
|
||||
return
|
||||
|
||||
if args.mode == "monitor":
|
||||
prefix = "u" if args.user else "r"
|
||||
dirs = setup_directories(args.target, prefix)
|
||||
|
|
@ -510,10 +606,12 @@ Examples:
|
|||
run_monitor(args.target, args.user)
|
||||
time.sleep(300)
|
||||
elif args.mode == "history":
|
||||
# Legacy mode - posts only
|
||||
run_full_history(args.target, args.limit, args.user,
|
||||
download_media_flag=False, scrape_comments_flag=False)
|
||||
else: # full mode
|
||||
else:
|
||||
run_full_history(args.target, args.limit, args.user,
|
||||
download_media_flag=not args.no_media,
|
||||
scrape_comments_flag=not args.no_comments)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
|
|||
|
|
@ -1,2 +1,9 @@
|
|||
# Core
|
||||
pandas
|
||||
requests
|
||||
|
||||
# Dashboard
|
||||
streamlit
|
||||
|
||||
# Export
|
||||
openpyxl
|
||||
|
|
|
|||
2
scheduler/__init__.py
Normal file
2
scheduler/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# Scheduler module
|
||||
from .cron import *
|
||||
215
scheduler/cron.py
Normal file
215
scheduler/cron.py
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
"""
|
||||
Scheduler module - Cron-style scheduling for scrape jobs
|
||||
"""
|
||||
import time
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
class CronScheduler:
|
||||
"""Simple cron-style scheduler for Reddit scraping jobs."""
|
||||
|
||||
def __init__(self):
|
||||
self.jobs = []
|
||||
self.running = False
|
||||
self.thread = None
|
||||
|
||||
def add_job(self, target, mode='full', limit=100, is_user=False,
|
||||
interval_minutes=60, run_at_start=True):
|
||||
"""
|
||||
Add a scheduled scraping job.
|
||||
|
||||
Args:
|
||||
target: Subreddit or username
|
||||
mode: 'full', 'history', or 'monitor'
|
||||
limit: Post limit per run
|
||||
is_user: True if target is a user
|
||||
interval_minutes: Minutes between runs
|
||||
run_at_start: Run immediately when scheduler starts
|
||||
"""
|
||||
job = {
|
||||
'id': len(self.jobs) + 1,
|
||||
'target': target,
|
||||
'mode': mode,
|
||||
'limit': limit,
|
||||
'is_user': is_user,
|
||||
'interval_minutes': interval_minutes,
|
||||
'run_at_start': run_at_start,
|
||||
'last_run': None,
|
||||
'next_run': datetime.now() if run_at_start else datetime.now() + timedelta(minutes=interval_minutes),
|
||||
'enabled': True,
|
||||
'run_count': 0
|
||||
}
|
||||
self.jobs.append(job)
|
||||
print(f"📅 Added job #{job['id']}: {'u/' if is_user else 'r/'}{target} every {interval_minutes}min")
|
||||
return job['id']
|
||||
|
||||
def remove_job(self, job_id):
|
||||
"""Remove a scheduled job."""
|
||||
self.jobs = [j for j in self.jobs if j['id'] != job_id]
|
||||
print(f"🗑️ Removed job #{job_id}")
|
||||
|
||||
def disable_job(self, job_id):
|
||||
"""Temporarily disable a job."""
|
||||
for job in self.jobs:
|
||||
if job['id'] == job_id:
|
||||
job['enabled'] = False
|
||||
print(f"⏸️ Disabled job #{job_id}")
|
||||
|
||||
def enable_job(self, job_id):
|
||||
"""Enable a disabled job."""
|
||||
for job in self.jobs:
|
||||
if job['id'] == job_id:
|
||||
job['enabled'] = True
|
||||
print(f"▶️ Enabled job #{job_id}")
|
||||
|
||||
def list_jobs(self):
|
||||
"""List all scheduled jobs."""
|
||||
print("\n📋 Scheduled Jobs:")
|
||||
print("-" * 60)
|
||||
for job in self.jobs:
|
||||
status = "✅" if job['enabled'] else "⏸️"
|
||||
prefix = "u/" if job['is_user'] else "r/"
|
||||
next_run = job['next_run'].strftime("%H:%M:%S") if job['next_run'] else "Never"
|
||||
print(f"{status} #{job['id']} | {prefix}{job['target']} | "
|
||||
f"Every {job['interval_minutes']}min | Next: {next_run} | "
|
||||
f"Runs: {job['run_count']}")
|
||||
print()
|
||||
return self.jobs
|
||||
|
||||
def _run_job(self, job):
|
||||
"""Execute a single job."""
|
||||
# Import here to avoid circular imports
|
||||
try:
|
||||
from main import run_full_history
|
||||
|
||||
prefix = "u/" if job['is_user'] else "r/"
|
||||
print(f"\n🚀 Running scheduled job: {prefix}{job['target']}")
|
||||
|
||||
run_full_history(
|
||||
job['target'],
|
||||
job['limit'],
|
||||
job['is_user'],
|
||||
download_media_flag=(job['mode'] == 'full'),
|
||||
scrape_comments_flag=(job['mode'] == 'full')
|
||||
)
|
||||
|
||||
job['last_run'] = datetime.now()
|
||||
job['run_count'] += 1
|
||||
print(f"✅ Job completed: {prefix}{job['target']}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Job failed: {e}")
|
||||
|
||||
def _scheduler_loop(self):
|
||||
"""Main scheduler loop."""
|
||||
print("🔄 Scheduler started")
|
||||
|
||||
while self.running:
|
||||
now = datetime.now()
|
||||
|
||||
for job in self.jobs:
|
||||
if not job['enabled']:
|
||||
continue
|
||||
|
||||
if job['next_run'] and now >= job['next_run']:
|
||||
self._run_job(job)
|
||||
job['next_run'] = now + timedelta(minutes=job['interval_minutes'])
|
||||
|
||||
# Check every 30 seconds
|
||||
time.sleep(30)
|
||||
|
||||
print("🛑 Scheduler stopped")
|
||||
|
||||
def start(self):
|
||||
"""Start the scheduler in background."""
|
||||
if self.running:
|
||||
print("⚠️ Scheduler already running")
|
||||
return
|
||||
|
||||
self.running = True
|
||||
self.thread = threading.Thread(target=self._scheduler_loop, daemon=True)
|
||||
self.thread.start()
|
||||
print("✅ Scheduler started in background")
|
||||
|
||||
def stop(self):
|
||||
"""Stop the scheduler."""
|
||||
self.running = False
|
||||
if self.thread:
|
||||
self.thread.join(timeout=5)
|
||||
print("🛑 Scheduler stopped")
|
||||
|
||||
def save_jobs(self, filepath='scheduler_jobs.json'):
|
||||
"""Save jobs to file."""
|
||||
jobs_data = []
|
||||
for job in self.jobs:
|
||||
job_copy = job.copy()
|
||||
job_copy['last_run'] = job_copy['last_run'].isoformat() if job_copy['last_run'] else None
|
||||
job_copy['next_run'] = job_copy['next_run'].isoformat() if job_copy['next_run'] else None
|
||||
jobs_data.append(job_copy)
|
||||
|
||||
with open(filepath, 'w') as f:
|
||||
json.dump(jobs_data, f, indent=2)
|
||||
print(f"💾 Saved {len(self.jobs)} jobs to {filepath}")
|
||||
|
||||
def load_jobs(self, filepath='scheduler_jobs.json'):
|
||||
"""Load jobs from file."""
|
||||
if not Path(filepath).exists():
|
||||
print("⚠️ No saved jobs found")
|
||||
return
|
||||
|
||||
with open(filepath, 'r') as f:
|
||||
jobs_data = json.load(f)
|
||||
|
||||
for job_data in jobs_data:
|
||||
if job_data['last_run']:
|
||||
job_data['last_run'] = datetime.fromisoformat(job_data['last_run'])
|
||||
if job_data['next_run']:
|
||||
job_data['next_run'] = datetime.fromisoformat(job_data['next_run'])
|
||||
self.jobs.append(job_data)
|
||||
|
||||
print(f"📂 Loaded {len(jobs_data)} jobs from {filepath}")
|
||||
|
||||
|
||||
# Simple interval-based scheduler for CLI
|
||||
def run_scheduled(target, interval_minutes, mode='full', limit=100, is_user=False):
|
||||
"""
|
||||
Run a scrape job on a schedule.
|
||||
|
||||
Args:
|
||||
target: Subreddit or username
|
||||
interval_minutes: Minutes between runs
|
||||
mode: 'full', 'history', or 'monitor'
|
||||
limit: Post limit per run
|
||||
is_user: True if target is a user
|
||||
"""
|
||||
from main import run_full_history
|
||||
|
||||
prefix = "u/" if is_user else "r/"
|
||||
print(f"📅 Scheduled: {prefix}{target} every {interval_minutes} minutes")
|
||||
print("Press Ctrl+C to stop\n")
|
||||
|
||||
run_count = 0
|
||||
|
||||
try:
|
||||
while True:
|
||||
run_count += 1
|
||||
print(f"\n{'='*50}")
|
||||
print(f"🔄 Run #{run_count} - {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")
|
||||
print(f"{'='*50}")
|
||||
|
||||
run_full_history(
|
||||
target,
|
||||
limit,
|
||||
is_user,
|
||||
download_media_flag=(mode == 'full'),
|
||||
scrape_comments_flag=(mode == 'full')
|
||||
)
|
||||
|
||||
print(f"\n⏰ Next run in {interval_minutes} minutes...")
|
||||
time.sleep(interval_minutes * 60)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print(f"\n\n🛑 Scheduler stopped after {run_count} runs")
|
||||
2
search/__init__.py
Normal file
2
search/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# Search module
|
||||
from .query import *
|
||||
220
search/query.py
Normal file
220
search/query.py
Normal file
|
|
@ -0,0 +1,220 @@
|
|||
"""
|
||||
Search & Query module - Search and filter scraped data
|
||||
"""
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
import re
|
||||
|
||||
def search_csv(filepath, query=None, column=None, min_score=None, max_score=None,
|
||||
start_date=None, end_date=None, post_type=None, author=None, limit=50):
|
||||
"""
|
||||
Search within a CSV file with various filters.
|
||||
|
||||
Args:
|
||||
filepath: Path to CSV file
|
||||
query: Text to search for
|
||||
column: Specific column to search in (default: all text columns)
|
||||
min_score: Minimum score filter
|
||||
max_score: Maximum score filter
|
||||
start_date: Start date (YYYY-MM-DD)
|
||||
end_date: End date (YYYY-MM-DD)
|
||||
post_type: Filter by post type (image, video, text, etc.)
|
||||
author: Filter by author
|
||||
limit: Maximum results to return
|
||||
|
||||
Returns:
|
||||
DataFrame with matching results
|
||||
"""
|
||||
if not Path(filepath).exists():
|
||||
print(f"❌ File not found: {filepath}")
|
||||
return pd.DataFrame()
|
||||
|
||||
df = pd.read_csv(filepath)
|
||||
|
||||
# Text search
|
||||
if query:
|
||||
if column and column in df.columns:
|
||||
mask = df[column].astype(str).str.contains(query, case=False, na=False)
|
||||
else:
|
||||
# Search in all text columns
|
||||
text_cols = ['title', 'selftext', 'body']
|
||||
mask = pd.Series([False] * len(df))
|
||||
for col in text_cols:
|
||||
if col in df.columns:
|
||||
mask |= df[col].astype(str).str.contains(query, case=False, na=False)
|
||||
df = df[mask]
|
||||
|
||||
# Score filter
|
||||
if min_score is not None and 'score' in df.columns:
|
||||
df = df[df['score'] >= min_score]
|
||||
if max_score is not None and 'score' in df.columns:
|
||||
df = df[df['score'] <= max_score]
|
||||
|
||||
# Date filter
|
||||
if 'created_utc' in df.columns:
|
||||
if start_date:
|
||||
df = df[df['created_utc'] >= start_date]
|
||||
if end_date:
|
||||
df = df[df['created_utc'] <= end_date]
|
||||
|
||||
# Post type filter
|
||||
if post_type and 'post_type' in df.columns:
|
||||
df = df[df['post_type'] == post_type]
|
||||
|
||||
# Author filter
|
||||
if author and 'author' in df.columns:
|
||||
df = df[df['author'] == author]
|
||||
|
||||
return df.head(limit)
|
||||
|
||||
def search_all_data(data_dir='data', query=None, **kwargs):
|
||||
"""
|
||||
Search across all scraped data.
|
||||
|
||||
Args:
|
||||
data_dir: Data directory path
|
||||
query: Text to search for
|
||||
**kwargs: Additional filters passed to search_csv
|
||||
|
||||
Returns:
|
||||
Dictionary with results from each subreddit
|
||||
"""
|
||||
results = {}
|
||||
data_path = Path(data_dir)
|
||||
|
||||
if not data_path.exists():
|
||||
print(f"❌ Data directory not found: {data_dir}")
|
||||
return results
|
||||
|
||||
# Find all posts.csv files
|
||||
for sub_dir in data_path.iterdir():
|
||||
if sub_dir.is_dir():
|
||||
posts_file = sub_dir / 'posts.csv'
|
||||
if posts_file.exists():
|
||||
df = search_csv(str(posts_file), query=query, **kwargs)
|
||||
if len(df) > 0:
|
||||
results[sub_dir.name] = df
|
||||
|
||||
# Also check legacy format
|
||||
for csv_file in data_path.glob('*.csv'):
|
||||
if csv_file.stem not in [r.replace('r_', '').replace('u_', '') for r in results.keys()]:
|
||||
df = search_csv(str(csv_file), query=query, **kwargs)
|
||||
if len(df) > 0:
|
||||
results[csv_file.stem] = df
|
||||
|
||||
return results
|
||||
|
||||
def print_search_results(results, show_preview=True):
|
||||
"""Pretty print search results."""
|
||||
total = sum(len(df) for df in results.values())
|
||||
|
||||
print(f"\n🔍 Found {total} results across {len(results)} sources\n")
|
||||
print("=" * 70)
|
||||
|
||||
for source, df in results.items():
|
||||
print(f"\n📁 {source} ({len(df)} matches)")
|
||||
print("-" * 50)
|
||||
|
||||
for _, row in df.iterrows():
|
||||
title = str(row.get('title', row.get('body', 'N/A')))[:60]
|
||||
score = row.get('score', 0)
|
||||
date = str(row.get('created_utc', ''))[:10]
|
||||
|
||||
print(f" [{score:>4}⬆] {title}...")
|
||||
if show_preview and 'selftext' in row and row['selftext']:
|
||||
preview = str(row['selftext'])[:100].replace('\n', ' ')
|
||||
print(f" └─ {preview}...")
|
||||
print()
|
||||
|
||||
def advanced_search(data_dir='data', query=None, regex=False, sort_by='score',
|
||||
ascending=False, **kwargs):
|
||||
"""
|
||||
Advanced search with regex support and sorting.
|
||||
|
||||
Args:
|
||||
data_dir: Data directory path
|
||||
query: Search query (text or regex pattern)
|
||||
regex: Treat query as regex pattern
|
||||
sort_by: Column to sort results by
|
||||
ascending: Sort ascending (default: descending)
|
||||
**kwargs: Additional filters
|
||||
|
||||
Returns:
|
||||
Combined DataFrame of all results
|
||||
"""
|
||||
all_results = []
|
||||
data_path = Path(data_dir)
|
||||
|
||||
for sub_dir in data_path.iterdir():
|
||||
if sub_dir.is_dir():
|
||||
posts_file = sub_dir / 'posts.csv'
|
||||
if posts_file.exists():
|
||||
df = pd.read_csv(posts_file)
|
||||
df['source'] = sub_dir.name
|
||||
all_results.append(df)
|
||||
|
||||
if not all_results:
|
||||
return pd.DataFrame()
|
||||
|
||||
combined = pd.concat(all_results, ignore_index=True)
|
||||
|
||||
# Apply query
|
||||
if query:
|
||||
if regex:
|
||||
pattern = query
|
||||
else:
|
||||
pattern = re.escape(query)
|
||||
|
||||
mask = pd.Series([False] * len(combined))
|
||||
for col in ['title', 'selftext']:
|
||||
if col in combined.columns:
|
||||
mask |= combined[col].astype(str).str.contains(pattern, case=False, na=False, regex=True)
|
||||
combined = combined[mask]
|
||||
|
||||
# Apply other filters
|
||||
if kwargs.get('min_score') and 'score' in combined.columns:
|
||||
combined = combined[combined['score'] >= kwargs['min_score']]
|
||||
|
||||
if kwargs.get('author') and 'author' in combined.columns:
|
||||
combined = combined[combined['author'] == kwargs['author']]
|
||||
|
||||
if kwargs.get('post_type') and 'post_type' in combined.columns:
|
||||
combined = combined[combined['post_type'] == kwargs['post_type']]
|
||||
|
||||
# Sort
|
||||
if sort_by in combined.columns:
|
||||
combined = combined.sort_values(sort_by, ascending=ascending)
|
||||
|
||||
limit = kwargs.get('limit', 100)
|
||||
return combined.head(limit)
|
||||
|
||||
def get_top_posts(data_dir='data', n=10, by='score'):
|
||||
"""Get top N posts across all scraped data."""
|
||||
df = advanced_search(data_dir, sort_by=by, ascending=False, limit=n)
|
||||
return df
|
||||
|
||||
def get_recent_posts(data_dir='data', n=10):
|
||||
"""Get most recent posts across all scraped data."""
|
||||
df = advanced_search(data_dir, sort_by='created_utc', ascending=False, limit=n)
|
||||
return df
|
||||
|
||||
def find_author_posts(data_dir='data', author=None):
|
||||
"""Find all posts by a specific author."""
|
||||
return advanced_search(data_dir, author=author, limit=1000)
|
||||
|
||||
def export_search_results(results, output_path, format='csv'):
|
||||
"""Export search results to file."""
|
||||
if isinstance(results, dict):
|
||||
combined = pd.concat(results.values(), ignore_index=True)
|
||||
else:
|
||||
combined = results
|
||||
|
||||
if format == 'csv':
|
||||
combined.to_csv(output_path, index=False)
|
||||
elif format == 'json':
|
||||
combined.to_json(output_path, orient='records', indent=2)
|
||||
elif format == 'excel':
|
||||
combined.to_excel(output_path, index=False)
|
||||
|
||||
print(f"💾 Exported {len(combined)} results to {output_path}")
|
||||
Loading…
Reference in a new issue