Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,21 @@ iii trigger migrate::codegen --json '{}'
# -> ./db.types.ts — one interface per table
```

**Payloads with quotes.** SQL regularly contains single quotes and `$$`
blocks, which fight the shell when inlined into `--json '…'` (the iii CLI
has no `@file`/stdin form). Put the payload in a file and let the shell do
the reading — no escaping needed:

```sh
cat > /tmp/payload.json <<'EOF'
{ "db": "primary", "sql": "UPDATE users SET motto = 'don''t panic' WHERE id = $1", "params": [1] }
EOF
iii trigger database::execute --json "$(cat /tmp/payload.json)"
```

Simple scalar fields can also skip JSON entirely: `iii trigger
migrate::create name=add_users` (key=value pairs merge over `--json`).

Three rules keep it simple and safe:

- **Forward-only.** Never edit an applied file — write a new migration to
Expand Down
35 changes: 35 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
# Changelog

## Unreleased

- README: how to pass payloads containing quotes to the iii CLI (read the
JSON from a file with `--json "$(cat …)"`; key=value pairs for scalars) —
the CLI has no native `@file`/stdin form.
- Skill: states that `future_dated` entries never block schema work — the
resolution is automatic (`migrate::create` timestamps are monotonic), so
an agent must not stall on an anomaly that is already handled.
- Skill: documents that `database::query` is read-only (writes fail with
SQLSTATE 25006) — ad-hoc writes go through `database::execute` or
`database::transaction`.
- Docs: canonical Postgres recreate-pattern recipe (column reorder,
incompatible type change) in the workflows, with the checklist of classic
omissions — incoming foreign keys, the serial sequence's `OWNED BY`,
indexes, triggers, defaults.
- New function `migrate::schema`: read-only structured report of the live
schema — ordered columns with defaults, primary keys, foreign keys
(multi-column pairs included), indexes, triggers, and Postgres enums —
for verifying what a migration actually did without hand-writing
`information_schema` queries. Optional `table` payload filter.
- New function `migrate::check`: static validation of the migrations
directory — statement splitting, empty files, naming scheme, checksum
drift — without executing any SQL. Reports every problem at once instead
of failing on the first, and keeps working while the database worker is
down (`db_checked: false`). Forward-only migrations cannot be rolled
back, so validate before you apply.
- Breaking (0.x): `migrate::status`'s `future_dated` entries are now objects
`{ name, applied, hint }` instead of bare names — the hint states the
remediation explicitly (applied: harmless; pending: re-scaffold via
`migrate::create`, whose timestamps are monotonic).
- Database errors now name the driver-native code in their message —
`… (SQLSTATE 42703)` on Postgres, `… (sqlite error code 1555)` on SQLite —
instead of burying it inside `database_error`. The full structured body is
still attached unchanged.

## 0.1.3 — 2026-07-23

- `migrate::create` keeps timestamps monotonic with the files already on
Expand Down
2 changes: 1 addition & 1 deletion docs/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Every handler error body carries a stable `code` field plus context fields.
| Code | Meaning | Recovery |
|---|---|---|
| `CHECKSUM_MISMATCH` | An applied migration's file changed on disk. Nothing was applied. | Restore the original file content (checksums normalize CRLF, so line endings are never the cause). To change the schema, write a new migration instead. |
| `MIGRATION_FAILED` | A migration's SQL failed. The batch rolled back; earlier migrations in the same run stay applied. | Read `name`, `statement_index` (0-based), and the database worker's structured error under `database_error`. Fix the SQL in the failing file — it was never recorded as applied, so a plain re-run of `migrate::up` picks it up again. |
| `MIGRATION_FAILED` | A migration's SQL failed. The batch rolled back; earlier migrations in the same run stay applied. | Read `name`, `statement_index` (0-based), and the database worker's structured error under `database_error`. When the driver reported a native code, `message` names it — `… (SQLSTATE 42703)` on Postgres — so the cause is searchable without digging into the body. Fix the SQL in the failing file — it was never recorded as applied, so a plain re-run of `migrate::up` picks it up again. |
| `CONFIG_ERROR` | Bad or missing configuration — unknown `db`, codegen without `types_out` or `out`, etc. | Fix the `configuration` entry `miiigrate` and restart the worker (config is read once at startup). |
| `DIR_NOT_FOUND` | The `dir` folder does not exist. | Only `migrate::create` creates the directory; run it once, or create the folder. |
| `INVALID_MIGRATION_NAME` | A `.sql` file does not match `YYYYMMDDHHMMSS_slug.sql`, or a `create` slug is not `[A-Za-z0-9_-]+`. | Rename the file or fix the slug. |
Expand Down
78 changes: 74 additions & 4 deletions docs/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ time, including while `migrate::up` is failing.
"pending": [{ "name": "…", "checksum": "…" }],
"mismatched": [{ "name": "…", "applied_checksum": "…", "file_checksum": "…", "applied_at": "…" }],
"missing": [{ "name": "…", "checksum": "…", "applied_at": "…" }],
"future_dated": ["…"]
"future_dated": [{ "name": "…", "applied": false, "hint": "pending: recreate it via migrate::create …" }]
}
```

Expand All @@ -69,9 +69,10 @@ time, including while `migrate::up` is failing.
`migrate::up` refuses to run while this list is non-empty.
- `missing` — recorded as applied but the file no longer exists on disk.
- `future_dated` — files on disk whose timestamp prefix is ahead of the wall
clock beyond a 5-minute skew tolerance: hand-written names. Harmless once
applied (`migrate::create` keeps new timestamps monotonic with them), but a
pending one should be renamed before apply.
clock beyond a 5-minute skew tolerance: hand-written names. Each entry says
whether the file is `applied` and carries an explicit `hint`: applied ones
are harmless (`migrate::create` keeps new timestamps monotonic with them),
pending ones should be re-scaffolded via `migrate::create` before apply.

## `migrate::create`

Expand Down Expand Up @@ -102,6 +103,75 @@ One `export interface` per table, plus an aggregate `Database` type. The
`_iii_migrations` tracking table is excluded. See [codegen.md](codegen.md)
for the exact type mapping.

## `migrate::check`

Statically validate the migrations directory — "would `migrate::up`
succeed?" — without executing any SQL. Read-only; the only network call is
the tracking-table read, and it degrades gracefully.

- **Payload**: `{}`
- **Returns**:

```json
{
"ok": true,
"dir": "./migrations",
"pending_checked": [{ "name": "…", "ok": true, "statements": 3 }],
"invalid_names": [{ "file": "notes.sql", "reason": "…" }],
"future_dated": [{ "name": "…", "applied": false, "hint": "…" }],
"mismatched": [],
"missing": [],
"db_checked": true
}
```

- Every pending file is parsed with the same splitter as `migrate::up`;
`pending_checked[].error` carries the exact message `up` would fail with
(unsplittable SQL, empty file).
- Unlike `migrate::up`/`status`, an ill-named `.sql` file does not abort the
report — all problems are listed at once under `invalid_names`.
- `ok` is false when anything would block or break `migrate::up`: an invalid
name, a broken pending file, or checksum drift. `future_dated` and
`missing` are warnings and do not flip it.
- Works while the database worker is down: `db_checked: false`, drift lists
are then unknown (empty).

## `migrate::schema`

Read-only structured report of the live schema — the verification companion
of `migrate::up`. Everything an agent or human needs to confirm what a
migration actually did, without hand-writing `information_schema` queries.

- **Payload**: `{}` or `{ "table": "users" }` (exact name; an unknown table
yields an empty `tables` list, not an error)
- **Returns**:

```json
{
"db": "primary",
"dialect": "postgres",
"tables": [{
"name": "reservations",
"columns": [{ "name": "id", "data_type": "bigint", "nullable": false,
"default": "nextval('…')", "position": 1 }],
"primary_key": ["id"],
"foreign_keys": [{ "name": "…_fkey", "columns": ["event_id"],
"references_table": "events", "references_columns": ["id"],
"on_delete": "CASCADE", "on_update": "NO ACTION" }],
"indexes": [{ "name": "…", "unique": true, "columns": ["…"], "definition": "CREATE …" }],
"triggers": [{ "name": "…", "timing": "AFTER", "events": ["INSERT", "UPDATE"] }]
}],
"enums": [{ "name": "mood", "labels": ["happy", "sad", "curious"] }]
}
```

- Columns come in ordinal (DDL) order with defaults; multi-column foreign
keys are paired column-by-column; `enums` is Postgres-only.
- `indexes[].columns` is best-effort: empty for expression indexes — read
`definition` there. SQLite auto-indexes backing PRIMARY KEY / UNIQUE have
no `definition`.
- The `_iii_migrations` tracking table is excluded.

## Checksums

Checksums are SHA-256 over the file content with line endings normalized
Expand Down
56 changes: 56 additions & 0 deletions docs/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,59 @@ INSERT INTO users_new (id, name, score) SELECT id, name, CAST(score AS REAL) FRO
DROP TABLE users;
ALTER TABLE users_new RENAME TO users;
```

### Postgres recreate pattern (column reorder, incompatible type change)

Postgres has no `ALTER TABLE … REORDER`; when the column order matters or a
type change is beyond `ALTER COLUMN TYPE`, recreate the table **inside one
migration** (single atomic batch — the swap is all-or-nothing). The DDL is
straightforward; what bites are the objects attached to the *old* table
that silently die with it. Canonical example, reordering `users`:

```sql
-- 1. New shape, everything declared up front (defaults, constraints).
CREATE TABLE users_new (
id bigint PRIMARY KEY DEFAULT nextval('users_id_seq'),
first_name text,
last_name text,
email text UNIQUE,
created_at timestamptz NOT NULL DEFAULT now()
);

-- 2. Copy the data, naming columns explicitly on both sides.
INSERT INTO users_new (id, first_name, last_name, email, created_at)
SELECT id, first_name, last_name, email, created_at FROM users;

-- 3. Re-point the sequence: it is OWNED BY the old table and would be
-- dropped with it otherwise.
ALTER SEQUENCE users_id_seq OWNED BY users_new.id;

-- 4. Incoming foreign keys (other tables referencing users) must be
-- dropped before the old table can go, and recreated against the new.
ALTER TABLE reservations DROP CONSTRAINT reservations_user_id_fkey;

-- 5. Swap.
DROP TABLE users;
ALTER TABLE users_new RENAME TO users;

-- 6. Recreate what lived on (or pointed at) the old table:
ALTER TABLE reservations
ADD CONSTRAINT reservations_user_id_fkey
FOREIGN KEY (user_id) REFERENCES users (id) ON DELETE CASCADE;
CREATE INDEX idx_users_mood ON users (mood);
CREATE TRIGGER users_audit AFTER INSERT OR UPDATE ON users
FOR EACH ROW EXECUTE FUNCTION audit_users();
```

Checklist of the classic omissions — each dies silently with `DROP TABLE`:

- **Incoming foreign keys** from other tables (step 4/6) — list them first:
they do not show up on the table you are recreating.
- **The sequence** behind `bigserial`/`serial` (step 3) — without
`OWNED BY` it is dropped and inserts start failing.
- **Indexes** beyond the primary key, **triggers**, and non-inline
**constraints** (step 6).
- **Column defaults** — declare them in step 1, they are not copied.

Verify the result with `migrate::schema` (columns in order, foreign keys,
indexes, triggers) before writing the next migration.
25 changes: 23 additions & 2 deletions skills/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,12 @@ migration as skipped, not failed.
- Apply with `migrate::up`, then check the response: when `types_path` is
present the TypeScript types were already regenerated; when it is absent
and the project has a `types_out`, run `migrate::codegen` yourself.
- Before any schema work, call `migrate::status` and resolve `mismatched`,
`missing`, or `future_dated` entries before adding new migrations.
- Before any schema work, call `migrate::status`. `mismatched` and
`missing` entries must be resolved first. `future_dated` entries do NOT
block you: an applied one is harmless (it expires once the clock catches
up), and a pending one is fixed by scaffolding through `migrate::create`
as usual — its timestamps are monotonic with existing files, so the new
file lands after the future-dated one. Never stall on `future_dated`.

## When to Use

Expand All @@ -50,6 +54,12 @@ migration as skipped, not failed.
`migrate::up`.
- You need the current schema state: applied, pending, drifted, or
future-dated files (`migrate::status`, read-only, safe anytime).
- You wrote migration SQL and want it validated before applying —
`migrate::check` runs the exact `migrate::up` parsing without executing
anything, and lists every problem at once.
- You need to verify what a migration actually did — `migrate::schema`
reports ordered columns, primary keys, foreign keys, indexes, and
triggers; never hand-write `information_schema` queries for that.
- TypeScript code reads rows from `database::query` and needs types that
match the JSON wire format (`migrate::codegen`).
- A `migrate::up` run failed and you need the failing file and statement
Expand All @@ -63,6 +73,9 @@ migration as skipped, not failed.
- Dialect-specific SQL is your responsibility (e.g. SQLite cannot
`ALTER COLUMN TYPE`; use the table-recreate pattern inside one migration —
the file runs as a single atomic batch).
- `database::query` is read-only — a write through it fails with SQLSTATE
25006. Route any ad-hoc write through `database::execute` or
`database::transaction`; schema changes always go through migrations.
- For ad-hoc queries or data edits, call the `database` worker directly; for
file or shell operations, use the `shell` worker.

Expand All @@ -74,13 +87,21 @@ migration as skipped, not failed.
`types_out` is set), regenerates types after a run that applied anything.
- `migrate::status` — read-only report `{ applied, pending, mismatched,
missing, future_dated }` with names, checksums, and apply dates.
- `migrate::check` — static validation of pending migrations (splitting,
empty files, naming, checksum drift) without executing any SQL; returns
`{ ok, pending_checked, invalid_names, mismatched, … }`. Run it after
writing SQL, before `migrate::up`.
- `migrate::create` — payload `{ name: "add_users" }`; creates
`<dir>/<UTC timestamp>_add_users.sql` with a unique, monotonic timestamp
and returns `{ name, path }`. Write the SQL into that file before calling
`migrate::up`.
- `migrate::codegen` — introspect the live schema through `database::query`
and write TypeScript types to the configured `types_out` (or payload
`out`); returns `{ path, tables, enums }`.
- `migrate::schema` — read-only structured report of the live schema
(ordered columns with defaults, primary keys, foreign keys, indexes,
triggers, Postgres enums); payload `{}` or `{ table: "users" }`. Use it
to verify a migration's effect instead of raw `information_schema` SQL.

Configuration (`db`, `dir`, `auto`, `types_out`, `codegen_on_up`, `dialect`)
lives in the `configuration` worker under id `miiigrate` and is read once at
Expand Down
6 changes: 4 additions & 2 deletions src/codegen.rs
Original file line number Diff line number Diff line change
Expand Up @@ -335,14 +335,16 @@ fn ts_property(name: &str) -> String {
}
}

fn str_field(row: &Map<String, Value>, key: &str) -> Result<String, String> {
pub(crate) fn str_field(row: &Map<String, Value>, key: &str) -> Result<String, String> {
row.get(key)
.and_then(Value::as_str)
.map(String::from)
.ok_or_else(|| format!("introspection row missing string field `{key}`: {row:?}"))
}

fn int_field(row: &Map<String, Value>, key: &str) -> Option<i64> {
/// Integer field tolerant of the wire format: int8/cardinal values may
/// arrive as JSON strings through `database::query`.
pub(crate) fn int_field(row: &Map<String, Value>, key: &str) -> Option<i64> {
let v = row.get(key)?;
v.as_i64()
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
Expand Down
Loading
Loading