Skip to content

Development#200

Merged
viktorbeck98 merged 50 commits into
mainfrom
development
Jul 6, 2026
Merged

Development#200
viktorbeck98 merged 50 commits into
mainfrom
development

Conversation

@ipmach

@ipmach ipmach commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Merge with main

viktorbeck98 and others added 30 commits June 3, 2026 14:34
…metadata

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…, fix load() thread safety

- Add module-level _load helper that mirrors _save
- Restore event_data_class from metadata on load
- PersistencySaver.load() now acquires self._lock before mutating persistency
- Use fs.pipe() for atomic file writes in _save to eliminate read/write race
- Add test_load_restores_event_data_class and TestPersistencySaverThreadSafety

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ctor

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
save(ep) returns a zip archive as bytes; load(ep, bytes) restores from
it. export_state() and import_state() on CoreDetector gain the same
overload, so callers can transfer state over a network API without
touching the filesystem.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Thorina Boenke <68156005+thorinaboenke@users.noreply.github.com>
Co-authored-by: Thorina Boenke <68156005+thorinaboenke@users.noreply.github.com>
Co-authored-by: Thorina Boenke <68156005+thorinaboenke@users.noreply.github.com>
Co-authored-by: Thorina Boenke <68156005+thorinaboenke@users.noreply.github.com>
Co-authored-by: Thorina Boenke <68156005+thorinaboenke@users.noreply.github.com>
viktorbeck98 and others added 19 commits June 21, 2026 17:52
Relocate export_state/import_state from CoreDetector to CoreComponent so
parsers and aggregators can reuse them. Return None / no-op instead of
raising when no persistency is configured, and keep the saver lock that
guards against the background save timer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
auto_load true start without saved state on first boot
…sistency_saver

Feature/get status of persistency saver
ingest_event held no lock, so the background save timer could iterate
events_data while ingest mutated it (silently-skipped saves), and /load
could replace state mid-ingest (torn state). Give EventPersistency an
RLock, acquire it in ingest_event, and have PersistencySaver share it so
ingest/save/load are mutually exclusive. RLock (not Lock) because ingest
can re-enter save() via the events_until_save callback on the same thread.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses PR review: ingest_event held the shared lock across save
serialization AND file writes, and export_state ran unlocked.

- ingest_event fires on-ingest callbacks outside the lock, so a
  count-triggered save no longer holds the ingest lock during I/O.
- Split the save path into _serialize (CPU, under lock) and _write
  (I/O, no lock); PersistencySaver.save() serializes+resets the counter
  under the lock, then writes outside it.
- export_state acquires saver.locked(), symmetric with import_state.
  Counter reset moved into PersistencySaver.save(), so export no longer
  zeroes events_since_save.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Code review flagged the unsynchronized read of _events_since_save as
looking like a bug. It is intentional: firing the on-ingest callback
outside the ingest lock is what keeps save()'s file I/O off that lock.
Comment names the TOCTOU (late/redundant save, never a missed save or
lost event) and warns against re-adding a lock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread src/detectmatelibrary/utils/persistency/persistency_saver.py Fixed
Comment thread tests/detectmatelibrary/test_utils/test_persistency_saver.py Dismissed
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@viktorbeck98 viktorbeck98 self-requested a review July 6, 2026 11:03
@viktorbeck98 viktorbeck98 merged commit e2292e1 into main Jul 6, 2026
7 checks passed
@viktorbeck98 viktorbeck98 deleted the development branch July 6, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants