Commit graph

7 commits

Author SHA1 Message Date
Sanjeev Kumar
8ffb53eae3 feat: real-time progress for posts processing and comments 2025-12-14 07:47:51 +05:30
Sanjeev Kumar
076404906d fix: --limit flag now works correctly + video download with audio
- Fixed --limit flag being ignored (was always requesting 100 posts)
- Added ffmpeg-based audio merging for Reddit videos
- Updated README with ffmpeg requirement
- Applied fixes to both main.py and async_scraper.py
2025-12-14 07:09:19 +05:30
Sanjeev Kumar
f65b35f881 feat: Add long-running observability and integration features
- Job history tracking with SQLite table
- Dry-run mode (--dry-run) to test scrape rules
- Plugin system with 3 built-in plugins (sentiment, dedupe, keywords)
- REST API server (--api) for Metabase/Grafana integration
- Parquet export (--export-parquet) for DuckDB/warehouses
- SQLite maintenance (--backup, --vacuum)
- Dashboard Integrations tab with external tools guides
- Updated Dockerfile and docker-compose.yml for cloud deployment
- Comprehensive README documentation
2025-12-14 03:42:24 +05:30
Sanjeev Kumar
0a1a604f49 v3.0: Full suite - Dashboard, Analytics, Scheduling, Notifications, Search 2025-12-13 22:17:32 +05:30
Sanjeev Kumar
2eb9ef9a54 v2.0: Full media & comment scraping support 2025-12-13 22:06:03 +05:30
Sanjeev Kumar
24e6646dff Add old.reddit.com as primary source for residential IPs 2025-12-13 21:58:32 +05:30
Sanjeev Kumar
db4e363130 Initial release of Universal Scraper 2025-12-13 21:51:02 +05:30