Sanjeev Kumar
4a2e5666d3
fix(dashboard): auto-detect and cleanup dead scraper processes
Docker Build & Publish / build (push) Has been cancelled
2025-12-14 08:04:06 +05:30
Sanjeev Kumar
73735b0669
fix(dashboard): resolved NameError and handled NoneType for media export
2025-12-14 07:55:29 +05:30
Sanjeev Kumar
604a50ce17
fix(dashboard): enabled scraper access without existing data
2025-12-14 07:52:23 +05:30
Sanjeev Kumar
8ce832010f
feat(dashboard): improved navigation with separate user/subreddit lists
2025-12-14 07:49:58 +05:30
Sanjeev Kumar
8ffb53eae3
feat: real-time progress for posts processing and comments
2025-12-14 07:47:51 +05:30
Sanjeev Kumar
3c6f5415cb
fix(dashboard): add missing time and os imports
2025-12-14 07:44:01 +05:30
Sanjeev Kumar
a731a9aeb5
feat: persistent background scraping jobs with file-based logging
2025-12-14 07:42:30 +05:30
Sanjeev Kumar
96246c4778
fix: add missing except clause
2025-12-14 07:38:25 +05:30
Sanjeev Kumar
5f0a5ba0db
fix: improved regex parsing for real-time metrics extraction
2025-12-14 07:37:22 +05:30
Sanjeev Kumar
9703075561
fix: unbuffered subprocess output for real-time dashboard updates
2025-12-14 07:34:47 +05:30
Sanjeev Kumar
4c4b8cb04f
fix: remove deprecated use_container_width warnings
2025-12-14 07:32:24 +05:30
Sanjeev Kumar
b240d714a0
feat: improved dashboard with real-time metrics (posts, comments, images, videos, rate)
2025-12-14 07:25:35 +05:30
Sanjeev Kumar
05b1186c6a
fix: Windows Unicode encoding for subprocess output
2025-12-14 07:21:51 +05:30
Sanjeev Kumar
70e72c1c09
feat: enable direct execution for all dashboard actions (Parquet export, backup, vacuum, API start)
2025-12-14 07:19:53 +05:30
Sanjeev Kumar
9a77d1cb95
feat: add real-time scraping from dashboard UI
2025-12-14 07:17:44 +05:30
Sanjeev Kumar
d73cdfc004
Merge feature branch: fix limit flag and add video audio support
2025-12-14 07:12:22 +05:30
Sanjeev Kumar
0d15941d45
fix: add ffmpeg to Docker image for video audio support
2025-12-14 07:10:58 +05:30
Sanjeev Kumar
89991bcc4a
Merge pull request #4 from ksanjeev284/feature/observability-enhancements
...
fix: --limit flag now works correctly + video download with audio
2025-12-14 07:10:00 +05:30
Sanjeev Kumar
076404906d
fix: --limit flag now works correctly + video download with audio
...
- Fixed --limit flag being ignored (was always requesting 100 posts)
- Added ffmpeg-based audio merging for Reddit videos
- Updated README with ffmpeg requirement
- Applied fixes to both main.py and async_scraper.py
2025-12-14 07:09:19 +05:30
Sanjeev Kumar
989e16e9ac
Merge pull request #3 from ksanjeev284/feature/observability-enhancements
...
feat: Add media export option to dashboard with ZIP download
2025-12-14 04:03:46 +05:30
Sanjeev Kumar
bf84b7ca64
feat: Add media export option to dashboard with ZIP download
2025-12-14 04:02:58 +05:30
Sanjeev Kumar
fbfc562767
Merge pull request #2 from ksanjeev284/feature/observability-enhancements
...
fix: Remove redundant Path import causing UnboundLocalError
2025-12-14 03:55:26 +05:30
Sanjeev Kumar
515fff94b0
fix: Remove redundant Path import causing UnboundLocalError
2025-12-14 03:54:46 +05:30
Sanjeev Kumar
a3f8030934
Merge pull request #1 from ksanjeev284/feature/observability-enhancements
...
feat: Add long-running observability and integration features
2025-12-14 03:43:01 +05:30
Sanjeev Kumar
f65b35f881
feat: Add long-running observability and integration features
...
- Job history tracking with SQLite table
- Dry-run mode (--dry-run) to test scrape rules
- Plugin system with 3 built-in plugins (sentiment, dedupe, keywords)
- REST API server (--api) for Metabase/Grafana integration
- Parquet export (--export-parquet) for DuckDB/warehouses
- SQLite maintenance (--backup, --vacuum)
- Dashboard Integrations tab with external tools guides
- Updated Dockerfile and docker-compose.yml for cloud deployment
- Comprehensive README documentation
2025-12-14 03:42:24 +05:30
Sanjeev Kumar
facd3a50af
Update README.md
2025-12-13 22:58:54 +05:30
Sanjeev Kumar
a623e5c12d
feat: Async scraper, cloud upload, subreddit stats
2025-12-13 22:35:09 +05:30
Sanjeev Kumar
0a1a604f49
v3.0: Full suite - Dashboard, Analytics, Scheduling, Notifications, Search
2025-12-13 22:17:32 +05:30
Sanjeev Kumar
2eb9ef9a54
v2.0: Full media & comment scraping support
2025-12-13 22:06:03 +05:30
Sanjeev Kumar
24e6646dff
Add old.reddit.com as primary source for residential IPs
2025-12-13 21:58:32 +05:30
Sanjeev Kumar
d1a475a5dc
Add GitHub Actions CI/CD pipeline with build badge
2025-12-13 21:55:06 +05:30
Sanjeev Kumar
6f301e7f4c
Add MIT License
2025-12-13 21:52:06 +05:30
Sanjeev Kumar
db4e363130
Initial release of Universal Scraper
2025-12-13 21:51:02 +05:30
