Add daily registration digest email (QUI-785) by kleczkowski · Pull Request #10 · QuipNetwork/qubitrefill

kleczkowski · 2026-07-02T18:52:53Z

Summary

Adds a daily email digest of the registered-contact list so registration stats can be handed off without the operator acting each time. An in-app scheduler emails the list to a fixed recipient once per UTC day.

reporting/registrations.py — read-only projection of the agents table (modeled on persistence/leaderboard.py) with an explicit column allowlist, so the agent id (the secret API key) is never exported. CSV cells are sanitized against spreadsheet formula injection.
orchestration/digest_scheduler.py — hourly loop plus a testable run_once; idempotent via a one-row digest_state table. Fail-closed: inert unless both DIGEST_RECIPIENTS and SMTP are set, so the contact list never reaches the logs.
email/sender.py — send_digest on the Protocol; SMTP attaches the CSV, Console/Fake redact.
cli.py — qtw send-digest [--dry-run] for manual send / verification.
Recipient — deployment defaults DIGEST_RECIPIENTS to the BD alias bd@postquant.xyz in docker-compose.yml; config.py's library default stays empty (fail-closed). Uses ${VAR-default} so an explicit empty value still disables the digest.

Adds no HTTP route and no MCP tool.

Config

DIGEST_RECIPIENTS — comma-separated; deployment default bd@postquant.xyz.
DIGEST_HOUR_UTC — hour at/after which the daily digest may send (default 7).

Testing

pytest tests/test_digest.py — 13 passed. Covers the API-key-leak guard, column allowlist, CSV injection neutralization, stats, idempotency, and the recipient/SMTP enable gate.

Refs QUI-785

An in-app scheduler emails the registered-contact list to a fixed recipient set once per UTC day, so registration stats can be handed off without the operator acting each time. - reporting/registrations.py: read-only projection of the agents table (modeled on persistence/leaderboard.py) with an explicit column allowlist, so the agent id (the secret API key) is never exported. CSV cells are sanitized against spreadsheet formula injection. - orchestration/digest_scheduler.py: hourly loop + testable run_once; idempotent via a one-row digest_state table. Fail-closed — inert unless both DIGEST_RECIPIENTS and SMTP are set, so the contact list never reaches the logs. - email/sender.py: send_digest on the Protocol; SMTP attaches the CSV, Console/Fake redact. - cli.py: `qtw send-digest [--dry-run]` for manual send / verification. - config: DIGEST_RECIPIENTS, DIGEST_HOUR_UTC (+ docker-compose, AGENTS.md). Adds no HTTP route and no MCP tool. Tests cover the key-leak guard, column allowlist, CSV injection, stats, and idempotency. Refs QUI-785

Point the daily digest at bd@postquant.xyz instead of a list of personal inboxes, centralizing the PII handoff to one BD address. - docker-compose.yml: default DIGEST_RECIPIENTS to bd@postquant.xyz. Use ${VAR-default} (no colon) so an explicit empty value still disables the digest — the fail-closed escape hatch is preserved. - config.py: note that the library default stays empty and the deployment default lives in docker-compose.yml. - AGENTS.md: document the default recipient and override path. Refs QUI-785

linear · 2026-07-02T18:52:56Z

QUI-785

augmentcode · 2026-07-02T18:56:42Z

🤖 Augment PR Summary

Summary: Adds an in-app, once-per-UTC-day email digest of registration/contact data to support operational handoff (QUI-785).

Changes:

Introduces a read-only registrations projection with an explicit export allowlist and CSV formula-injection sanitization (backend/src/backend/reporting/registrations.py).
Adds a DigestState single-row marker/table to persist last successful send time and drive idempotency (backend/src/backend/db/models.py).
Implements a lifespan-scoped background scheduler that wakes hourly and sends at/after DIGEST_HOUR_UTC, guarded by a Postgres advisory lock to prevent double-sends across workers (backend/src/backend/orchestration/digest_scheduler.py).
Extends the email sender protocol with send_digest and adds SMTP/console/fake implementations (CSV attachment; console redacts) (backend/src/backend/email/sender.py).
Registers scheduler startup/shutdown in the FastAPI lifespan (backend/src/backend/api/app.py).
Adds a CLI command (qtw send-digest [--dry-run]) for manual/verification sends (backend/src/backend/cli.py).
Adds config/env wiring and docker-compose defaults (DIGEST_RECIPIENTS, DIGEST_HOUR_UTC) (backend/src/backend/config.py, docker-compose.yml).
Adds comprehensive unit/regression tests for allowlisting, CSV sanitization, stats, idempotency, and concurrency (backend/tests/test_digest.py).

Technical Notes: Scheduler is fail-closed (inert unless recipients + real SMTP are configured) and advances the send marker only after a successful send+commit.

_{🤖 Was this summary useful? React with 👍 or 👎}

augmentcode

Review completed. 2 suggestions posted.

Comment augment review to trigger a new review at any time.

augmentcode · 2026-07-02T18:56:43Z

+    """
+    today = now.date().isoformat()
+    state = await _get_state(session)
+    if state.last_sent_date == today:


backend/src/backend/orchestration/digest_scheduler.py:77 — The last_sent_date guard and the later marker update/commit() aren’t atomic, so if the app runs with multiple Uvicorn workers (--workers / WEB_CONCURRENCY) two processes can both pass this check and send duplicate digests for the same UTC day. That can result in duplicate outbound PII emails even though the digest_state table exists.

Severity: high

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Fixed in b8f7a91. Confirmed the race: start() runs per uvicorn worker, so WEB_CONCURRENCY>1 spawns one scheduler per process and the in-memory guard cannot serialize them. Wrapped the check→send→mark in a transaction-scoped Postgres advisory lock (pg_advisory_xact_lock), which is cluster-global and auto-releases on commit/rollback, so the "marker advances only on success" retry semantics are preserved. Added a regression test that races two run_once calls and asserts exactly one send (it fails on the pre-fix code with a duplicate-key violation / double send).

augmentcode · 2026-07-02T18:56:43Z

+    new_since = (
+        total
+        if since_date is None
+        else sum(1 for r in rows if r.created_at[:10] > since_date)


backend/src/backend/reporting/registrations.py:95 — Because since_date is only a YYYY-MM-DD date string, r.created_at[:10] > since_date will not count registrants created later on the same day as the last send, even though the rendered copy says “New since last digest”. If the digest can send at/after DIGEST_HOUR_UTC (not necessarily at 00:00), this can make new_since misleading.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Fixed in b8f7a91. The marker stored only YYYY-MM-DD, so a registrant created later on the send's own UTC day was permanently excluded from new_since. Changed DigestState.last_sent_date → last_sent_at (full ISO-8601 timestamp) and compare full timestamps; the date prefix still drives once-per-day idempotency. Added test_compute_stats_counts_same_day_after_send covering the same-day case.

Addresses the two PR #10 review findings. Finding 1 (high) — concurrent workers could double-send. `start()` runs in every uvicorn worker's lifespan, so WEB_CONCURRENCY>1 spawns one scheduler per process; the in-memory `last_sent_date` guard and the marker commit weren't atomic, so two workers could both pass the guard and send duplicate PII digests. Serialize the check→send→mark with a transaction-scoped Postgres advisory lock (cluster-global, auto-released on commit/rollback), which preserves the "marker advances only on success" retry semantics. Regression test races two run_once calls and asserts exactly one send. Finding 2 (medium) — `new_since` undercounted same-day registrations. The marker stored only a date, so anyone registering later on a send's own UTC day was permanently excluded from "New since last digest" even though the CSV listed them. Store the full send timestamp (DigestState.last_sent_date → last_sent_at) and compare full ISO timestamps; the date prefix still drives once-per-day idempotency. Note: `create_all` does not alter existing tables, so the digest_state table must be dropped once for the column change to take effect. Refs QUI-785

kleczkowski · 2026-07-02T19:28:48Z

augment review

Please focus this pass on substantive issues only: correctness, concurrency/atomicity, security, PII handling, and data integrity. Skip style, naming, formatting, and docstring-wording nits — ruff enforces style and those are not actionable here. If the two prior findings (advisory-lock double-send, same-day new_since) are resolved and nothing substantive remains, say so explicitly rather than surfacing minor nitpicks.

augmentcode

Review completed. 4 suggestions posted.

Comment augment review to trigger a new review at any time.

augmentcode · 2026-07-02T19:32:27Z

+        text = "; ".join(str(v) for v in value)
+    else:
+        text = str(value)
+    if text.startswith(_FORMULA_TRIGGERS):


_sanitize_cell() only guards when the raw string starts with a trigger; spreadsheet apps often ignore leading spaces, so values like ' =HYPERLINK(...)' may still be interpreted as formulas. Consider hardening the check to account for leading whitespace so CSV injection can’t bypass the guard.

Severity: medium

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Fixed in 5ca2184. The guard now also checks the lstripped string, so a trigger behind leading whitespace/tabs (" =HYPERLINK(...)") is quoted. Added test_csv_injection_behind_leading_whitespace_is_neutralized.

augmentcode · 2026-07-02T19:32:27Z

+        if not config.SMTP_PASSWORD:
+            raise SystemExit("SMTP is not configured — set SMTP_PASSWORD (or use --dry-run)")
+
+        await digest_scheduler.send_now(session, get_email_sender(), now)


_send_digest_async() computes rows/stats before sending, but then ignores the RegistrationStats returned by send_now() and prints stats.total from the pre-send snapshot. Under concurrent registrations this can make the CLI output disagree with what was actually emailed.

Severity: low

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Fixed in 5ca2184. The CLI now prints the RegistrationStats returned by send_now (the emailed snapshot) instead of the pre-send one.

augmentcode · 2026-07-02T19:32:27Z

+]
+# Hour of day (UTC) at/after which the daily digest may send. The scheduler wakes
+# hourly and sends the first time it is past this hour with no send recorded today.
+DIGEST_HOUR_UTC: int = int(os.environ.get("DIGEST_HOUR_UTC", "7"))


DIGEST_HOUR_UTC is parsed as an int but isn’t validated to a 0–23 range, so misconfiguration can silently prevent sends or change the schedule in surprising ways. Consider validating/clamping early so ops failures are loud and diagnosable.

Severity: low

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Fixed in 5ca2184. config now rejects DIGEST_HOUR_UTC outside 0–23 at import with a clear ValueError.

augmentcode · 2026-07-02T19:32:27Z

+    __tablename__ = "digest_state"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True)  # always 1
+    last_sent_at: Mapped[str | None] = mapped_column(String(32), nullable=True)


DigestState.last_sent_at is String(32), which is exactly the length of some datetime.isoformat() outputs (with microseconds + offset) and leaves little room for format variation. If this ever truncates, it could break idempotency / new_since calculations, so consider a larger column size to avoid data loss.

Severity: low

_{🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.}

Declining this one. last_sent_at is always datetime.now(UTC).isoformat(), which is deterministically <=32 chars ("YYYY-MM-DDTHH:MM:SS.ffffff+00:00" = exactly 32), so String(32) fits by construction — there is no format variation to truncate. Enlarging it would also require another manual prod migration (create_all does not ALTER existing tables) for no functional gain.

Second-round PR #10 review fixes (substantive only). - registrations: quote a formula trigger hidden behind leading whitespace (" =HYPERLINK(...)") — spreadsheets strip leading spaces before evaluating, so the CSV-injection guard must too. - cli: print the RegistrationStats returned by send_now (what was actually emailed) instead of a pre-send snapshot. - config: reject DIGEST_HOUR_UTC outside 0–23 at startup; the schedule guard is `now.hour < DIGEST_HOUR_UTC`, so a bad value silently never fires. Left DigestState.last_sent_at at String(32): the stored value is always datetime.now(UTC).isoformat(), which is exactly <=32 chars, so it fits by construction. Refs QUI-785

kleczkowski added 2 commits July 2, 2026 13:22

augmentcode Bot reviewed Jul 2, 2026

View reviewed changes

kleczkowski merged commit 057e6e5 into main Jul 2, 2026
1 check passed

kleczkowski deleted the feature/qui-785-registration-digest branch July 2, 2026 19:41

Uh oh!

Conversation

kleczkowski commented Jul 2, 2026

Summary

Config

Testing

Uh oh!

linear Bot commented Jul 2, 2026

Uh oh!

augmentcode Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

kleczkowski commented Jul 2, 2026

Uh oh!

augmentcode Bot left a comment

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

augmentcode Bot Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kleczkowski Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

augmentcode Bot commented Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading

augmentcode Bot Jul 2, 2026 •

edited

Loading