Skip to content

[Feature] Add SDK methods for persistent Sets (incl. dm.save_set()) #16

Description

@gtchax

Summary

The DataMaker API now exposes a persistent, team/project-scoped Sets resource (/sets CRUD), but the Python SDK has no way to read or write it. A Set is a durable snapshot of generated/fetched rows (a regression dataset, shared fixture, or audit-ready export). The web UI's empty state and the Set model docstring both advertise dm.save_set() as the scenario-facing entry point — but that method does not exist yet. This issue tracks adding a SetsClient route module plus delegated DataMaker methods, mirroring the existing TemplatesClient pattern.

Motivation

Scenarios run inside datamaker-runner and talk to the API through this SDK (DATAMAKER_API_KEY / DATAMAKER_API_URL). Users are told they can persist a regression set from a scenario via dm.save_set(...), but there is no SDK surface for it, so the documented flow dead-ends. Sets need first-class SDK methods like every other entity (templates, connections, projects).

API surface to wrap

The API (datamaker-api) exposes these routes, all authMiddleware + scopeFilter scoped to the caller's team/project:

Method Path Notes
GET /sets?projectId= List sets for the active project scope
GET /sets/:id Fetch one
POST /sets Create. Body: { name, description?, data?, rowCount?, projectId? }. rowCount is derived from data (array length) when omitted; teamId/projectId are resolved server-side from the validated scope.
PATCH /sets/:id Partial update (name?, description?, data?, rowCount?)
DELETE /sets/:id Delete

data is arbitrary JSON; the common case is an array of row objects.

Proposed implementation (mirror routes/templates.py)

1. New route module src/datamaker/routes/sets.py

class SetsClient(BaseClient):
    """Client for saved Set operations."""

    def get_sets(self, project_id: str | None = None) -> List[Dict]: ...
    def get_set(self, set_id: str) -> Dict: ...
    def create_set(self, name, data=None, description=None, row_count=None, project_id=None) -> Dict: ...
    def update_set(self, set_id: str, set_data: Dict) -> Dict: ...
    def delete_set(self, set_id: str) -> Dict: ...

2. Register in main.py — instantiate self._sets = SetsClient(...) and add delegated methods on DataMaker:

def get_sets(self, project_id=None): return self._sets.get_sets(project_id)
def get_set(self, set_id): return self._sets.get_set(set_id)
def create_set(self, name, data=None, description=None, row_count=None, project_id=None): ...
def update_set(self, set_id, set_data): return self._sets.update_set(set_id, set_data)
def delete_set(self, set_id): return self._sets.delete_set(set_id)

3. save_set() convenience helper — the documented scenario-facing entry point. Thin wrapper over create_set (or upsert by name if we want re-running a scenario to overwrite rather than duplicate):

def save_set(self, name, data, description=None, project_id=None) -> Dict:
    """Persist rows as a saved Set. Used from scenarios: dm.save_set("orders-regression", rows)."""
    return self._sets.create_set(name=name, data=data, description=description, project_id=project_id)

Decide whether save_set should be create-only or upsert-by-name. Upsert matches the "re-run a scenario, refresh the regression set" mental model, but the API has no upsert endpoint yet — if we want it, either add GET /sets?name= filtering + client-side find-then-PATCH, or a server-side upsert. Flag for discussion.

Acceptance criteria

  • SetsClient added under src/datamaker/routes/sets.py with get/get-one/create/update/delete.
  • DataMaker exposes delegated get_sets/get_set/create_set/update_set/delete_set + save_set.
  • save_set(name, data, ...) persists rows and the result shows on the Sets page.
  • data array length auto-populates rowCount (parity with the API's derivation) when not passed.
  • Tests under tests/ covering the new methods (follow existing route tests).
  • README / llms.txt updated to document dm.save_set() and the Sets methods.

Reference

  • src/datamaker/routes/templates.py — route-module + CRUD pattern
  • src/datamaker/main.py — client composition + delegated methods
  • API side (in automators-com/datamaker): apps/datamaker-api/src/routes/sets.ts, packages/database/prisma/schema.prisma (Set model). Tracked there by automators-com/datamaker#2443.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions