Make ApiCallTracker.save() atomic to prevent corrupt history files
Cin observed that database/api_call_history.json was occasionally landing on disk truncated mid-write — `_load()` would log `History file is not valid JSON, starting fresh` and 24h of metrics would be lost. Root cause: `save()` opened the file in 'w' mode (which truncates to 0 bytes immediately) and then streamed JSON via `json.dump`. Any SIGINT/SIGTERM/crash between truncate and final write left the file half-formed — exactly Cin's symptom of the JSON cutting off mid-array. Switch to the standard atomic pattern: write to a sibling .tmp file, flush + fsync, then `os.replace` (atomic on every platform we run on). Failed writes also clean up the leftover .tmp file. The canonical file is now either the previous good copy or the new good copy — never a partial one.
This commit is contained in:
parent
e66af77ff6
commit
02f26bf338
1 changed files with 20 additions and 3 deletions
|
|
@ -267,7 +267,12 @@ class ApiCallTracker:
|
|||
|
||||
|
||||
def save(self):
|
||||
"""Persist 24h minute history to disk. Call on shutdown."""
|
||||
"""Persist 24h minute history to disk. Call on shutdown.
|
||||
|
||||
Uses an atomic write (write to a sibling .tmp file, fsync, rename) so
|
||||
a SIGINT/SIGTERM/crash mid-write can't leave the JSON file truncated.
|
||||
Without this, an interrupted shutdown corrupts the history file and
|
||||
the next startup loses 24h of metrics."""
|
||||
try:
|
||||
now = time.time()
|
||||
cutoff = now - 86400
|
||||
|
|
@ -283,10 +288,22 @@ class ApiCallTracker:
|
|||
if entries:
|
||||
data[key] = entries
|
||||
events = [dict(e) for e in self._events if e['ts'] >= cutoff]
|
||||
with open(_PERSIST_PATH, 'w') as f:
|
||||
json.dump({'ts': now, 'history': data, 'events': events}, f)
|
||||
|
||||
payload = {'ts': now, 'history': data, 'events': events}
|
||||
tmp_path = _PERSIST_PATH + '.tmp'
|
||||
with open(tmp_path, 'w') as f:
|
||||
json.dump(payload, f)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp_path, _PERSIST_PATH)
|
||||
except Exception as e:
|
||||
logger.error(f"[ApiCallTracker] Failed to save history: {e}")
|
||||
# Best-effort cleanup of stale tmp file from a failed write.
|
||||
try:
|
||||
if os.path.exists(_PERSIST_PATH + '.tmp'):
|
||||
os.remove(_PERSIST_PATH + '.tmp')
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _load(self):
|
||||
"""Restore 24h minute history from disk. Called on init."""
|
||||
|
|
|
|||
Loading…
Reference in a new issue