A file deleted between os.walk() and path.stat() raises
FileNotFoundError, which propagated uncaught and failed the entire
discover() sweep. With enough failures this trips the circuit
breaker, silencing the poller.
Catch FileNotFoundError around the stat() call and skip the file.
The next sweep (or watchfiles) will emit the DELETE event.
All pollers sharing the same poll_interval_s previously woke up and
swept at exactly the same moment after startup. With 10+ sources
this causes a coordinated spike in listing traffic (S3 LIST, WebDAV
PROPFIND, HTTP HEAD) every interval.
Add a random initial delay of 0-25% of the poll interval after the
first sweep, applied to both PeriodicPoller and FSPoller's sweep
loop. Subsequent sweeps run on the normal fixed interval, now
staggered across sources.
HTTPSource and WebDAVSource previously created a new AsyncClient for
every head(), fetch(), and discover() call — no connection reuse, TLS
renegotiation on every request, and connection pool churn at scale.
Create the client once in __init__ and reuse it for the lifetime of
the source. Add aclose() to both sources, called by PollerManager on
shutdown to cleanly close the connection pool.
has_pending() scans jobs WHERE source_id=? AND status IN
('queued','claimed') on every poller sweep — without an index this
is a full table scan as the jobs table grows. Similarly,
count_succeeded_since() scans by completed_at for the dashboard's
rolling-throughput display.
Add two partial indexes:
- idx_jobs_pending_by_source: covers has_pending() lookups
- idx_jobs_succeeded_completed: covers count_succeeded_since()
Both use CREATE INDEX IF NOT EXISTS so they're idempotent on
existing databases.
Workers previously polled the queue with a fixed 1s sleep between
claim attempts, adding ~500ms average latency to job pickup. Now
JobRepo.job_available (an asyncio.Condition) is notified on every
successful enqueue, waking idle workers immediately. The poll
interval remains as a timeout fallback for stop signals and breaker
state changes.
