diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 319a9473bd..33fba48069 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -46,7 +46,7 @@ jobs: # ponytail: STT is handled by the bundled whisper-stt-server (whisper.cpp # with native DTW token timestamps). No separate VAD model is fetched - # here. See docs/engineering/stt-spec.md for the full architecture. + # here. See technical-documentation/architecture/transcription-and-captions.md for the full architecture. - name: Build Windows app run: npm run build:win -- --publish never @@ -77,7 +77,7 @@ jobs: # ponytail: STT is handled by the bundled whisper-stt-server (whisper.cpp # with native DTW token timestamps). No separate VAD model is fetched - # here. See docs/engineering/stt-spec.md for the full architecture. + # here. See technical-documentation/architecture/transcription-and-captions.md for the full architecture. - name: Build Windows Store package run: npm run build:win:store -- --publish never diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3c65e41b6b..95d9d0c224 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,10 +1,13 @@ name: CI +# feat/ai-edition is a long-lived integration branch that PRs land on for +# months at a time. Without it listed here, every one of those PRs merged with +# no lint, no typecheck, no tests and no PR-title check. on: pull_request: - branches: [main] + branches: [main, feat/ai-edition] push: - branches: [main] + branches: [main, feat/ai-edition] jobs: lint: @@ -23,6 +26,19 @@ jobs: - uses: ./.github/actions/setup - run: npx tsc --noEmit + docs: + name: Docs + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + # No ./.github/actions/setup: check-docs.mjs imports only node builtins, + # so npm ci would be a minute of install for nothing. Node 22 is here for + # import.meta.dirname (needs >= 20.11). + - uses: actions/setup-node@v4 + with: + node-version: 22 + - run: npm run docs:check + test: name: Test runs-on: ubuntu-latest diff --git a/.gitignore b/.gitignore index ccc431231d..82bb66e0d1 100644 --- a/.gitignore +++ b/.gitignore @@ -84,7 +84,7 @@ result-* # whisper.cpp DTW POC (tools/stt-eval/whispercpp-dtw-poc) — vendored engine # clone, downloaded GGML models, and generated build/results artifacts. -# See docs/engineering/stt-whispercpp-dtw-poc-plan.md. +# See technical-documentation/architecture/transcription-and-captions.md. /tools/stt-eval/whispercpp-dtw-poc/whisper.cpp/ /tools/stt-eval/whispercpp-dtw-poc/build-cpu/ /tools/stt-eval/whispercpp-dtw-poc/build-vulkan/ diff --git a/.harness/docs/architecture-overview.md b/.harness/docs/architecture-overview.md index 373f9618e6..dbc2e4df54 100644 --- a/.harness/docs/architecture-overview.md +++ b/.harness/docs/architecture-overview.md @@ -1,6 +1,6 @@ # OpenScreen Architecture Notes -Quick map of how the app fits together, for the Mavis reins. For deeper details, see `../docs/architecture/native-bridge.md` and `../docs/engineering/`. +Quick map of how the app fits together, for the Mavis reins. For deeper details, see `../technical-documentation/architecture/native-bridge.md` and `../technical-documentation/engineering/`. ## Process layout diff --git a/.harness/docs/git-workflow.md b/.harness/docs/git-workflow.md index cd0895fdc3..68f2486452 100644 --- a/.harness/docs/git-workflow.md +++ b/.harness/docs/git-workflow.md @@ -41,7 +41,7 @@ All five must be green before merge. Native helper code is NOT covered by CI — ## Release flow -Two `workflow_dispatch` workflows cut a release. Trunk-based on `main`, but **release branches freeze the RC codebase between cut and promote** (see § Release branches below). Both require the `OPENSCREEN_RELEASE_TOKEN` secret — see `docs/secrets.md`. +Two `workflow_dispatch` workflows cut a release. Trunk-based on `main`, but **release branches freeze the RC codebase between cut and promote** (see § Release branches below). Both require the `OPENSCREEN_RELEASE_TOKEN` secret — see `technical-documentation/engineering/release-and-secrets.md`. ### Step 1: cut a release candidate diff --git a/.harness/reins/openscreen-dev/agent.md b/.harness/reins/openscreen-dev/agent.md index 33c4a28a18..4bd9ce150e 100644 --- a/.harness/reins/openscreen-dev/agent.md +++ b/.harness/reins/openscreen-dev/agent.md @@ -15,7 +15,7 @@ You are the generalist implementer for the OpenScreen project — a free, open-s ## How you work - Read `AGENTS.md` at the repo root before touching anything — it has the canonical commands, layout, and conventions. -- When the change touches recording, IPC, or the native bridge, read `.harness/docs/architecture-overview.md` (start here), `docs/architecture/native-bridge.md` (deeper dive), and `docs/engineering/` (native helper roadmaps). +- When the change touches recording, IPC, or the native bridge, read `.harness/docs/architecture-overview.md` (start here), `technical-documentation/architecture/native-bridge.md` (deeper dive), and `technical-documentation/engineering/` (native helper roadmaps). - TypeScript strict mode, Biome format (tabs, double quotes, 100-col). Run `npm run lint:fix` before committing. - For renderer-only iteration use `npm run build-vite`. For full packaging use `npm run build` (electron-builder, slow). - Native helpers require a real platform to test — don't claim "done" on macOS/Windows native code without a manual smoke test. diff --git a/.harness/reins/openscreen-tester/agent.md b/.harness/reins/openscreen-tester/agent.md index cbf3afdc6c..bd4658ce4a 100644 --- a/.harness/reins/openscreen-tester/agent.md +++ b/.harness/reins/openscreen-tester/agent.md @@ -15,7 +15,7 @@ You are the test specialist for the OpenScreen project — a free, open-source s ## How you work - Read `AGENTS.md` at the repo root for commands and conventions. -- Read `docs/tests/writing-tests.md` for the project's test style guide. +- Read `technical-documentation/testing/writing-tests.md` for the project's test style guide. - Match the style of neighboring `*.test.` files in the same package — don't invent new patterns. - Unit tests: `npm run test` (Vitest, jsdom). Browser tests: `npm run test:browser` (needs `npm run test:browser:install` once). E2E: `npm run test:e2e` (Playwright). - E2E specs in `tests/e2e/windows-native-checklist.spec.ts` are Windows-only — gate with `test.skip` for other platforms rather than deleting. diff --git a/.worktrees/wt-9ce78f24/HANDOFF.md b/.worktrees/wt-9ce78f24/HANDOFF.md deleted file mode 100644 index 9ab72d3187..0000000000 --- a/.worktrees/wt-9ce78f24/HANDOFF.md +++ /dev/null @@ -1,299 +0,0 @@ -# AI-Edition Implementation Handoff - -**Branch**: `docs/ai-edition-plan` (commit `cf25858`, pushed to `origin`) -**Worktree**: `G:\repos\openscreen\.worktrees\wt-9ce78f24` -**Dev server**: `http://localhost:5173/?windowType=editor` (browser mode with shim) - ---- - -## 1. Context - -The user (Etienne Lescot, repo owner) was working through the implementation of the **OpenScreen x Axcut AI-edition merge**. The original PR #35 (commit `1e9db17` on the same branch) introduced the planning docs only: - -- `docs/architecture/ai-edition-merge-plan.md` — the 10-phase merge plan -- `docs/architecture/axcut-inventory.md` — catalog of the axcut codebase -- `docs/architecture/openscreen-inventory.md` — catalog of the OpenScreen codebase -- `docs/architecture/ai-edition-collision-analysis.md` — collision analysis - -This implementation PR (`cf25858`) delivers the **code** for that plan — all phases 0, 1, 3, 4, 6-8, and partial 9, plus a developer-convenience browser shim and spec updates that changed the framing. - -The plan was re-framed mid-implementation. The user clarified: -- **New editing model** (multi-asset, clips, skips, transcript, virtual-time preview) = **default for all users**, not opt-in -- **AI features** (LLM provider config, chat) = **opt-in** behind `AI_FEATURES_ENABLED` -- **Local Whisper** = **privacy-safe, not gated** (runs in-browser, never calls out) - -This is the spec's `§0 Framing` section. See `docs/architecture/ai-edition-merge-plan.md` lines ~13-65. - ---- - -## 2. What was built (file by file) - -### 2.1 Schema & migration (`src/lib/ai-edition/`) - -| File | Purpose | -|------|---------| -| `schema/index.ts` | Vendored axcut v2 schema + v3 additions (`annotations[]`, `zoomRanges[]`, `legacyEditor` envelope, `transcripts[]`). `axcutSchemaVersion = 3`. `clip.sourceEndSec` made optional (duration unknown at migration time). | -| `schema/index.test.ts` | 15 schema tests (version enforcement, optional clip duration, envelope passthrough, etc.) | -| `document/timeline.ts` | Pure interval math: `normalizeIntervals`, `subtractInterval`, `invertIntervals`, `buildTimelineFromIntervals`, `replaceTimeline`, `restoreFullTimeline`. Ported from axcut `apps/server/src/lib/timeline.ts` (no event bridge, no agent — just the math). | -| `document/timeline.test.ts` | 14 tests covering all the above. | -| `document/migrate.ts` | Bidirectional `EditorProjectData` (v2) ↔ `AxcutDocument` (v3). Notes: `zoomRanges`/`annotations` use **ms** units to mirror the legacy types; timeline ops use **sec** units. The migration is lossless in both directions thanks to the `legacyEditor` passthrough. | -| `document/migrate.test.ts` | 14 tests including round-trip, v1 legacy, focus clamping. | -| `document/transcribe.ts` | `transcribeAsset(document, assetId)` wraps the existing `extractMono16kFromVideoUrl` + `transcribeMono16kToSegments` (from `src/lib/captioning/`). Returns an `AxcutTranscript`. `withTranscript` writes it back to the document. | -| `document/ids.ts` | `createId(prefix)` using `uuid.v4()`. | -| `timeline/virtual-preview.ts` | Pure time-mapping: `totalVirtualDuration`, `clampVirtualTime`, `locateVirtualPosition`, `locateSourcePosition`, `keptWordIdSet`, `formatSeconds`. | -| `timeline/virtual-preview.test.ts` | 8 tests. | -| `store/projectStore.ts` | Zustand store: `projectId`, `document`, `revision`, `status`, `error`, `sourceDurationSec`, `currentTimeSec`. Actions: `loadProject`, `createProject`, `addAsset`, `removeAsset`, `replaceTimeline`, `restoreFullTimeline`, `setTranscript`, `setSourceDuration`, `setCurrentTime`, `saveDocument`, `setDocument`, `clear`. | -| `store/projectStore.test.ts` | 5 tests with `nativeBridgeClient.aiEdition` mocked. | -| `exporter/documentExporter.ts` | Adapter: maps `AxcutDocument` → `VideoExporterConfig` / `GifExporterConfig`. Clips → `trimRegions` (inverse). Reads `legacyEditor` for wallpaper, cursor, webcam, etc. `sourceWidth`/`sourceHeight` come from caller. | - -### 2.2 Main-process services (`electron/ai-edition/`) - -| File | Purpose | -|------|---------| -| `document-service.ts` | `DocumentService(projectsRoot)`: `listProjects`, `getProject(projectId)`, `createProject(title)`, `saveProject(doc)`, `deleteProject(projectId)`, `addAsset(projectId, {path, label?})`, `removeAsset(projectId, assetId)`. One `.axcut` JSON file per project under `app.getPath('userData')/projects/`. Validates paths against an allowlist of video extensions. Cascades clips + skipRanges on asset removal. | -| `document-service.test.ts` | 16 tests (CRUD, path traversal, cascade, primary-asset reassignment). | -| `provider-registry.ts` | 8 provider definitions (anthropic, openai, google, mistral, openrouter, openai-compatible, openai-oauth, copilot-proxy) with `authKind`, `supportsReasoningEffort`, `envKeys`, `baseUrl`. Ported from axcut `provider-registry.ts`. | -| `llm-config-store.ts` | `LlmConfigStore(userDataPath)`: config in `llm-config.json` plain JSON, **credentials in `safeStorage`-encrypted bytes** at `llm-credentials.enc`. Env vars override stored keys (same precedence as axcut). | -| `chat-service.ts` | `runChat(projectId, message, llmConfig)`: validates config + API key, stores messages in a `Map`, **returns a stub assistant message** (LLM call needs `@langchain/*` deps). `getChatHistory(projectId)` returns the in-memory list. | -| `native-bridge/services/aiEditionService.ts` | Adapter to the existing `native-bridge` envelope: wraps `DocumentService`, `LlmConfigStore`, and the chat stubs into the `domain: "aiEdition"` IPC contract. | - -### 2.3 IPC bridge extensions - -- `electron/ipc/nativeBridge.ts` — added the `aiEdition` domain case. Each action calls into `AiEditionService` (`document.listProjects`, `document.get`, `document.create`, `document.save`, `document.delete`, `document.addAsset`, `document.removeAsset`, `llm.getSnapshot`, `llm.setConfig`, `llm.setApiKey`, `llm.removeApiKey`, `chat.run`, `chat.history`). -- `electron/ipc/handlers.ts` — wires `DocumentService` + `LlmConfigStore` + chat functions into the `NativeBridgeContext`. -- `src/native/contracts.ts` — adds `AiEditionLlmConfig`, `AiEditionLlmSnapshot`, `AiEditionChatMessage`, `AiEditionChatResult` types and the new `aiEdition` action cases to the `NativeBridgeRequest` union. -- `src/native/client.ts` — adds the `nativeBridgeClient.aiEdition` namespace with `listProjects`, `get`, `create`, `save`, `delete`, `addAsset`, `removeAsset`, `llmGetSnapshot`, `llmSetConfig`, `llmSetApiKey`, `llmRemoveApiKey`, `chatRun`, `chatHistory`. -- `src/native/browserShim.ts` — **new**. Browser-mode shim that: - - Stubs `window.electronAPI` (no-op `openVideoFilePicker`, `pickExportSavePath`, etc.) - - Overrides `nativeBridgeClient` methods to return mock data - - Persists projects/docs in `localStorage` (`browser-shim-projects`, `browser-shim-document`) - - Auto-installs when running in a plain browser at `http://localhost:5173/?windowType=editor` - - Detected via absence of `window.electronAPI` - -### 2.4 Renderer UI (`src/components/ai-edition/`) - -| File | Purpose | -|------|---------| -| `IconRail.tsx` | Vertical 36-44px icon rail with collapse/expand chevron. Used for both left and right rails. Tooltip on hover. | -| `NewEditorShell.tsx` | **The default editor** for all users (replaces legacy `VideoEditor`). Layout: top header (project title + 3 toggle buttons) + body with left rail | left content (Project/Chat) | center (video + timeline) | right content (Transcript/Background/Video effects/Camera/Cursor/Crop/Export) | right rail. Recording → asset on editor open (auto-creates project + adds asset). Legacy `.openscreen` loading via the "Open" header button (migrates v2 → v3). | -| `AiEditionShell.tsx` | Re-exports `AiEditionOrLegacy` which delegates to `NewEditorShell` (legacy VideoEditor is now unused but kept for rollback). | -| `ProjectPanel.tsx` | Left content: project list + create input + assets list. Uses raw Tailwind matching OpenScreen's dark surface. | -| `TimelinePane.tsx` + `.module.css` | Ported from axcut `apps/web/src/components/TimelinePane.tsx` (~837 lines). Ruler, kept/cut segments, playhead, zoom (Ctrl+wheel), pan (Alt+drag), add cut, delete cut, resize cut handles, fit button, navigator overview. | -| `VirtualPreview.tsx` + `.module.css` | Ported from axcut `apps/web/src/components/VirtualPreview.tsx`. Single-video element with virtual-time seeking; seeks across clip boundaries; reports metadata via `onLoadedMetadata`; exposes video element via `onVideoElement` callback. | -| `TranscriptEditor.tsx` + `.module.css` | Click word / shift-click word → range → "Cut" button → `dropWordRange` op. Kept words = default, skipped = red strikethrough. | -| `ChatPanel.tsx` | Right content when `leftTab === "chat"`. Messages list + input + send. In-memory history. | -| `EditorSettings.tsx` | Bridge that wraps the **original `SettingsPanel`** (from `src/components/video-editor/SettingsPanel.tsx`, unchanged). Reads from `AxcutDocument.legacyEditor` (wallpaper, cursor, webcam, shadow, etc.), `document.zoomRanges`, `document.annotations`. Writes back through `setDocument` / `saveDocument`. Maps `activeTab` to `SettingsPanelMode` (background/effects/layout/cursor/timeline/export). Calls `SettingsPanel` with `hideInternalRail` so the right rail is the only navigation. | - -### 2.5 App-level wiring - -- `src/App.tsx` — imports and calls `installBrowserShims()` before render. The `editor` windowType still lazy-loads the `AiEditionShell` (which now renders `NewEditorShell`). -- `src/components/video-editor/featureFlags.ts` — renamed `AI_EDITION_ENABLED` → `AI_FEATURES_ENABLED`, default `false`. The flag now **only** gates the LLM/agent UI (chat panel). The new editor is the default for everyone. -- `package.json` — added `zod: ^3.23.8` and `zustand: ^5.0.8`. - -### 2.6 Documentation - -- `docs/architecture/ai-edition-merge-plan.md` — **major rewrite**: - - **New §0 Framing** — two layers (new editor = default, AI features = opt-in) - - **§5.8 locked decision** updated: flag now gates only LLM/agent UI - - **§10 cut-over** — no editor cut-over (new editor is default); only AI features opt-in - - Locked decisions list re-ordered: framing change recorded - ---- - -## 3. What was tested - -- **`npx tsc --noEmit`**: clean (no errors) -- **`npm run lint`**: clean (1 warning, not error — `useExhaustiveDependencies` in TimelinePane, pre-existing pattern) -- **`npm run test`**: **313 / 313 tests pass** across 39 test files - - 16 `document-service.test.ts` - - 15 `schema/index.test.ts` - - 14 `timeline.test.ts` - - 14 `migrate.test.ts` - - 8 `virtual-preview.test.ts` - - 5 `projectStore.test.ts` - - + 239 pre-existing tests (all still passing) -- **Browser smoke test**: `http://localhost:5173/?windowType=editor` renders the editor with shim data, project create/select works, asset add works (mocked), transcript/chat panels render, settings panel shows correct view per right-rail tab. - ---- - -## 4. Key decisions and rationale - -### 4.1 The framing change (user-driven) - -The original plan treated "AI-edition" as a single opt-in feature. Mid-implementation the user said: *multi-asset/clips/etc. is valid outside of user opt-in. It is valid outside of user opt-in. The opt-in should be limited to llm/conversation.* This led to: -- `AI_EDITION_ENABLED` → `AI_FEATURES_ENABLED` (the rename makes the semantic explicit) -- New editor ships to all users by default (kill-switch removed) -- The right rail's chat / LLM config is the only gated surface -- Local Whisper stays ungated (privacy-safe by construction) - -### 4.2 Why the new editor ships as the default despite incomplete feature parity - -The spec calls for full feature parity (annotations, zoom, cursor, webcam, blur, crop, export, legacy `.openscreen` loading). The implementation delivers the **architecture** and the **export, legacy loading, transcript, transcription, settings panel** integrations, but the new editor's UI is intentionally simpler than the legacy `VideoEditor` for some affordances (no annotations/zoom UI for adding new ones, just editing existing ones from the `SettingsPanel`). This is acceptable for a first cut because: -- The legacy `VideoEditor` is still on disk and reachable via git (rollback path) -- Adding the remaining UI affordances is incremental (no new architecture needed) -- The `SettingsPanel` integration already lets users edit every field that exists in their v3 document - -### 4.3 Why the AI runtime is stubbed - -Phases 6-8 require `@langchain/openai`, `@langchain/anthropic`, `deepagents`, `better-sqlite3`. These are heavy (multi-MB native modules, OAuth flows, langgraph runtime). The implementation: -- Ships the IPC contracts, provider registry, LLM config store (with `safeStorage`), chat history -- Stubs the actual LLM call (returns a fixed message reminding the user to install deps) -- The 8 providers, OAuth flow, reasoning effort mapping, and the chat-service scaffolding are all in place — adding the real `@langchain/*` calls is a focused follow-up - -### 4.4 Why ms for `annotations[]` / `zoomRanges[]` but sec for timeline - -`AxcutDocument.annotations` and `AxcutDocument.zoomRanges` mirror the legacy `ProjectEditorState.annotationRegions` / `.zoomRegions` which use **ms**. The timeline ops (`skipRanges`, `clips.sourceStartSec`, etc.) follow axcut's convention of **sec** because axcut's `clips` are authored from the agent/runtime where the second-based model is canonical. This dual-unit is contained to the document schema and handled by the `document/timeline.ts` math + `migrate.ts` conversion. The renderer reads `document.zoomRanges` directly as ms. - -### 4.5 Why `safeStorage` for credentials (not plain JSON) - -Per locked decision 4 in the spec: LLM credentials are stored in `safeStorage`-encrypted bytes (OS keychain on macOS, libsecret on Linux, DPAPI on Windows). Config (provider, model, baseUrl, reasoningEffort) is plain JSON. This matches axcut's security improvement over their original plain-JSON approach. - ---- - -## 5. What's NOT in this PR (deferred work) - -These are deliberate deferrals, not oversights: - -1. **Full feature parity UI** — adding new annotations/zoom regions from the new shell (the SettingsPanel can only edit existing ones). Follow-up: port the legacy `VideoEditor`'s annotation/zoom add flows to `NewEditorShell`. -2. **Real LLM calls** — `@langchain/*` deps not installed. Follow-up: `npm i @langchain/openai @langchain/anthropic deepagents` and replace the stub in `chat-service.ts:runChat`. -3. **SQLite for sessions/checkpoints** — `better-sqlite3` not installed. Follow-up: port axcut's `DatabaseService` and `PersistentFileCheckpointSaver`. -4. **Webcam real-time preview** in `VirtualPreview` — current is a single-video component; axcut has a two-layer crossfade. Follow-up for a richer preview experience. -5. **13-locale i18n** — the new components use hardcoded English strings ("Transcribe", "Remove cuts", "Export", etc.). Follow-up: add to `src/i18n/locales//*.json`. -6. **Settings sync to `userPreferences.ts`** — `AI_FEATURES_ENABLED` toggle is a constant, not user-toggleable. Follow-up: wire to the existing settings sync. -7. **Export dialog integration** — the new editor's "Export" button shows a toast. Follow-up: wire the `ExportDialog` component with the full options. -8. **The legacy `VideoEditor.tsx`** (2961 lines) is unchanged on disk. It can be deleted in a follow-up once confidence is high. - ---- - -## 6. How to continue - -### 6.1 Resume this branch - -```bash -cd G:\repos\openscreen\.worktrees\wt-9ce78f24 -git status # should be clean -git log --oneline -3 -npm run dev # already running, port 5173 -# Open http://localhost:5173/?windowType=editor -``` - -### 6.2 Add a real LLM provider - -1. `npm i @langchain/openai @langchain/anthropic deepagents better-sqlite3` -2. In `electron/ai-edition/chat-service.ts:runChat`, replace the stub with a real call: - ```ts - import { ChatOpenAI } from "@langchain/openai"; - const model = new ChatOpenAI({ model: config.model, apiKey }); - const result = await model.invoke(message); - ``` -3. Add the corresponding provider in `provider-registry.ts` if it's not already there. - -### 6.3 Add full feature parity (annotations/zoom creation UI) - -1. Port the legacy `VideoEditor`'s annotation-add flow (around line 2500+) to a new component. -2. Mount it in `NewEditorShell` alongside `SettingsPanel`. -3. Wire it to `documentStore.setDocument` (already wired through `EditorSettings`). - -### 6.4 Open a PR - -```bash -git push origin docs/ai-edition-plan # already pushed -gh pr create \ - --base main \ - --head docs/ai-edition-plan \ - --title "feat(ai-edition): implement v3 editor model + AI features scaffold" \ - --body-file PR_BODY.md -``` - -### 6.5 Delete the legacy `VideoEditor` when ready - -The file `src/components/video-editor/VideoEditor.tsx` (2961 lines) is now unused in the default flow. `grep -r "from.*VideoEditor" src/` to confirm. Then delete and remove from `App.tsx` lazy import. - ---- - -## 7. File map (where to look) - -``` -G:\repos\openscreen\.worktrees\wt-9ce78f24\ -├── docs/architecture/ -│ └── ai-edition-merge-plan.md # updated §0, §5.9, §10 -├── electron/ -│ ├── ai-edition/ -│ │ ├── document-service.ts # CRUD on .axcut files -│ │ ├── document-service.test.ts -│ │ ├── llm-config-store.ts # safeStorage credentials -│ │ ├── provider-registry.ts # 8 providers, static -│ │ └── chat-service.ts # in-memory, LLM stub -│ ├── ipc/ -│ │ ├── handlers.ts # wires services to bridge -│ │ └── nativeBridge.ts # adds aiEdition domain -│ └── native-bridge/services/ -│ └── aiEditionService.ts # bridge adapter -├── src/ -│ ├── App.tsx # installs browser shim -│ ├── native/ -│ │ ├── browserShim.ts # NEW - browser-mode stubs -│ │ ├── client.ts # adds aiEdition namespace -│ │ └── contracts.ts # adds aiEdition types -│ ├── components/ -│ │ ├── video-editor/ -│ │ │ ├── SettingsPanel.tsx # + hideInternalRail prop -│ │ │ └── featureFlags.ts # AI_EDITION_ENABLED → AI_FEATURES_ENABLED -│ │ └── ai-edition/ # NEW directory -│ │ ├── AiEditionShell.tsx # kill-switch removed -│ │ ├── NewEditorShell.tsx # main shell, the default -│ │ ├── IconRail.tsx -│ │ ├── ProjectPanel.tsx -│ │ ├── TimelinePane.tsx + .module.css -│ │ ├── VirtualPreview.tsx + .module.css -│ │ ├── TranscriptEditor.tsx + .module.css -│ │ ├── ChatPanel.tsx -│ │ └── EditorSettings.tsx # bridge → SettingsPanel -│ └── lib/ai-edition/ # NEW directory -│ ├── schema/index.ts + .test.ts -│ ├── document/ -│ │ ├── timeline.ts + .test.ts -│ │ ├── migrate.ts + .test.ts -│ │ ├── transcribe.ts -│ │ └── ids.ts -│ ├── timeline/ -│ │ └── virtual-preview.ts + .test.ts -│ ├── store/ -│ │ └── projectStore.ts + .test.ts -│ └── exporter/ -│ └── documentExporter.ts -└── package.json # +zod, +zustand -``` - ---- - -## 8. The conversation arc (for context) - -1. User asked to check PR #35 and start implementation per its plan. -2. Implemented Phase 0 (schema, migration, feature flag) — 29 tests pass, human-testable via dev server. -3. Implemented PR 1.1 (project panel, document service, IPC bridge) — human-testable. -4. User asked for total spec completion. Implemented PR 1.2 + 1.3 (timeline port, preview port, new editor shell with kill-switch). -5. User said "implement 1, 2, and 3" with the axcut `\\wsl.localhost\Ubuntu\home\etienne\repos\axcut\` path. Implemented: - - Export (Phase 3) — adapter to existing VideoExporter - - Legacy `.openscreen` loading — migrate v2 → v3 - - Settings panel (annotations, zoom, cursor, webcam, wallpaper) — bridge to `SettingsPanel` -6. User said "go on → full implementation". Implemented Phases 6-8 scaffolding (provider registry, LLM config store with safeStorage, chat service stub, IPC contracts) and Phase 9 partial (settings toggle, i18n deferred). -7. User asked to relaunch in browser. Added `browserShim.ts` for `http://localhost:5173/?windowType=editor`. -8. User asked for UI redesign to match original OpenScreen + axcut layout. Implemented: - - Left icon rail (Project / Chat) - - Right icon rail (Transcript / Background / Video effects / Camera / Cursor / Crop / Export) - - Top header (project title + PanelLeft / PanelRight / Download) - - NewEditorShell with full-height columns - - Removed the chevron collapse buttons (user requested) - - Used original OpenScreen SettingsPanel icons - - Added `hideInternalRail` prop so the right rail is the only navigation -9. User asked about worktree, branch, commit, push. Confirmed branch (`docs/ai-edition-plan`), committed and pushed. -10. User asked for a handoff summary in English for a coding agent. - ---- - -**The next coding agent should**: -- Open `http://localhost:5173/?windowType=editor` to see the current state -- Read `docs/architecture/ai-edition-merge-plan.md` for the plan -- Pick up from §5 (deferred work) — most impactful next steps are real LLM calls (#2) and feature parity UI (#1) -- All architecture is in place; the remaining work is wiring and UI polish, not new design diff --git a/AGENTS.md b/AGENTS.md index 417f937e5c..babd479cb7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,7 +20,7 @@ OpenScreen is a free, open-source screen recorder and video editor (Electron + R - `src/` — React app: UI, editor components, timeline, i18n, captioning/cursor/exporter libs - `electron/` — main process, IPC, recording orchestration - `electron/native/` — **native** capture helpers: `screencapturekit/` (Swift, macOS) and `wgc-capture/` (C++/Win32, Windows). These are built and shipped with the app, not loaded from npm -- `docs/` — architecture, engineering roadmaps, testing guides +- `technical-documentation/` — architecture, engineering and testing reference (start at its README) - `tests/` — Playwright e2e specs + fixtures - `scripts/` — native build scripts, diagnostic tools - `nix/`, `flake.nix` — Linux packaging @@ -98,7 +98,7 @@ Two `workflow_dispatch` workflows cut a release with a pre-release candidate (RC - **Promote RC**: Actions → "Promote RC to stable release" → Run workflow. Input: `rc_tag` (e.g. `v1.5.0-rc.2`), optional `release_notes_extra`. Closes the `vX.Y.Z` milestone, strips `-rc.N` from `package.json`, pushes `vX.Y.Z` tag, which triggers `build.yml` to publish a stable release (full notarization, Tier 3 homebrew/winget/nix/aur fires). Notifies `#announcements` on Discord. - **Manual fallback**: `git tag vX.Y.Z-rc.N && git push origin vX.Y.Z-rc.N` does the same as Cut RC (minus the milestone migration and Discord announce) — useful for emergency cuts. -Both workflows require the `OPENSCREEN_RELEASE_TOKEN` secret (a fine-grained PAT with `contents: write` + `issues: write`). This is the standard fix for `release: published` not triggering downstream workflows when the release is created by `GITHUB_TOKEN`. See `docs/secrets.md`. +Both workflows require the `OPENSCREEN_RELEASE_TOKEN` secret (a fine-grained PAT with `contents: write` + `issues: write`). This is the standard fix for `release: published` not triggering downstream workflows when the release is created by `GITHUB_TOKEN`. See `technical-documentation/engineering/release-and-secrets.md`. **Release branches freeze the build between cut and promote.** Every RC cut creates `release/vX.Y.Z-rc.N`. The branch is *not* merged into `main` until the stable tag is published; only cherry-picks of bugfixes land on the release branch during the RC window. The stable tag points at the branch tip (RC + cherry-picks), then `promote.yml` opens a `release/vX.Y.Z-sync → main` PR to bring main into line. This contract exists because of the v1.6.0 incident (2026-07-05) where the original promote workflow tagged `main` instead of the RC snapshot, causing 23 unreleased commits to ship in `v1.6.0`. Full rules in `.harness/docs/git-workflow.md` § Release branches. diff --git a/docs/ai-edition/v4-design-parity.md b/docs/ai-edition/v4-design-parity.md deleted file mode 100644 index 77be26337d..0000000000 --- a/docs/ai-edition/v4-design-parity.md +++ /dev/null @@ -1,76 +0,0 @@ -# OpenScreen Editor v4 — design parity & implementation notes - -Implementation of the **OpenScreen Editor v4** design -(`claude.ai/design` project `54649d16-a5bd-4352-bed6-a62d4c2dff2e`, -`OpenScreen Editor v4 - standalone.html`), started from -`feat/native-stt-whispercpp`. - -## How the design was imported - -The design lives in a claude.ai/design **design-system** project, read -through the design MCP (`DesignSync`, authenticated via `/design-login` to -`https://api.anthropic.com/v1/design/mcp`). Method calls used: - -- `list_files` on project `54649d16…` — full file inventory. -- `get_file` on the v4 design sources. - -**On `OpenScreen Editor v4 - standalone.html` specifically:** that file is a -self-extracting gzip **bundle** (an unpacker script + a `__bundler/manifest` -of base64/gzip assets + a `__bundler/template` of the page markup). Retrieved -via `get_file`, it comes back **exactly 262 144 bytes = 256 KiB** — the tool's -hard response cap — and is truncated there: the manifest (253 KB, images only — -the C2PA-signed logo PNG + SVGs) fits, but the `__bundler/template` markup and -closing tags sit past the cap and are unreachable. `get_file` has no -offset/paging parameter, so the *full* bundle cannot be pulled through the MCP. - -The bundle is a build artifact **generated from** the project's editable -ground-truth source, which was retrieved **complete** and audited in full: - -- `OpenScreen Editor v4.dc.html` — 134 799 bytes, ends in ``. The - `` template + the `DCLogic` controller (`state`, `renderVals()`), - covering the `stageMode` media/preview/rec branches, the 8 inspector facets, - every slider/toggle, the timeline lanes/clips/navigator logic, and the - light+dark token maps. -- `ui_kits/editor/index.html` — the React recreation of the same editor. -- `tokens/*.css` — the color/type/spacing/radii/elevation tokens. - -The standalone renders exactly this design; the truncated tail contains no -design information not present in the fully-retrieved `.dc.html`. - -## Design → component map - -| Design element | Implementation | -|---|---| -| App grid, rows `58px 1fr {timeline}` per mode | `NewEditorShell.tsx` + `v4/EditorShellV4.module.css` | -| Top bar + **Media / Edit / Rec** switch | `v4/EditorTopBar.tsx` | -| Agent panel (392px, Edit only) | reuses `LeftPanel` (`ChatStripPanel`) in the `.agent` column | -| Stage — Edit preview | `Preview.tsx` inside `.stage` | -| Stage — Media browser (search / grid / detail) | `v4/MediaStage.tsx` | -| Stage — Rec (webcam + glass source bar + big REC) | `v4/RecStage.tsx` | -| Floating transport | `v4/FloatingTransport.tsx` | -| Floating inspector + 8-facet rail | `v4/FloatingInspector.tsx` → reuses `RightPanes` (Background/Effects/Layout/Cursor/Transcript) + Captions/Crop/Chapters | -| Timeline footer (tools, ruler, lanes, clips, navigator) | `Bottombar.tsx` + `TimelinePane.tsx` | -| Media timeline "Arrange clips" header | `Bottombar` `timelineVariant="media"` | -| Region pills (flat wash + 1.5px frame, per-kind color) | `NewEditorShell.module.css` `.zoomPill/.speedPill/.annotationPill/.skipPill` | -| Lane watermark hides when populated | `RegionTimeline.tsx` (`Children.count`) | -| Clip card (--surface-1 / 1.5px / 11px) + floating blur label chip | `TimelinePane.module.css` `.trackBlock/.trackVisual/.trackInfo` | -| Navigator (grey track + --surface-hi window + grips) | `TimelinePane.module.css` `.timelineNavigator*` | -| Tokens (incl. added `--surface-hi`, dark `--fg-emphasis`, `--accent-border/ring/stripe`, `*-wash`) | `src/styles/design-tokens.css` | - -## Verification - -`tsc --noEmit`, `biome check`, and full `vite build` (renderer + Electron main -+ preload) all pass. All three modes and the populated Edit-mode timeline were -verified in a real browser render (dark hero theme) via a browser-only preview -config (`vite.v4preview.config.ts`) with a seeded document — see -[verify commands memory] / `NewEditorShell` dev store exposure. - -## Deliberate engineering decision - -The timeline's **visual** matches the design (verified with live data). Its -**interaction logic** — clip drag/reorder, region resize, place-skip, -zoom-pan navigator — is the existing, tested machinery, restyled rather than -rewritten. Re-deriving that logic from scratch was intentionally avoided: it -carries real regression risk to functional editing for no visual gain. If a -ground-up timeline reimplementation is desired, it should be scoped as its own -effort with that risk acknowledged. diff --git a/docs/architecture/ai-edition-collision-analysis.md b/docs/architecture/ai-edition-collision-analysis.md deleted file mode 100644 index 1828fe33ea..0000000000 --- a/docs/architecture/ai-edition-collision-analysis.md +++ /dev/null @@ -1,355 +0,0 @@ -# AI Edition Merge — Collision Analysis & Edge Cases - -> Cross-references `ai-edition-roadmap.md`, `openscreen-inventory.md`, `axcut-inventory.md`. Every collision listed here must be resolved before its owning phase starts. -> -> **Severity legend:** 🔴 breaks the merge · 🟠 will cause user-visible regressions · 🟡 polish / nice-to-have. - ---- - -## 1. Schema collisions (Phase 0) - -### 1.1 Both projects have `version: 2` but mean different things 🔴 -- OpenScreen: `projectPersistence.ts:65` — `PROJECT_VERSION = 2`, denotes the `EditorProjectData` envelope. -- axcut: `axcut-schema/src/index.ts:1` — `axcutSchemaVersion = 2`, denotes the `AxcutDocument` shape. -- **Resolution:** bump everything to v3. Single `SCHEMA_VERSION = 3` constant lives in `src/lib/ai-edition/schema/`. Migration is bidirectional: v2 OpenScreen ↔ v3 ↔ v2 axcut. Document the migration paths in `migrateDocument(value, fromVersion)`: - - `fromVersion = 2, source = 'openscreen'` → wrap `EditorProjectData` into `AxcutDocument` (single asset, single clip spanning full duration, trimRegions inverted to skipRanges). - - `fromVersion = 2, source = 'axcut'` → pass-through (already valid). - - `fromVersion = 3` → no-op. -- Auto-detect source by sniffing for `media.screenVideoPath` (OpenScreen) vs `assets[]` (axcut). No source-of-origin field needed in v3. - -### 1.2 Axcut's merged primary `transcript` vs per-asset `transcripts[]` 🟠 -- `axcut-schema/src/index.ts:96-100` — `documentSchema` carries **both** a single merged `transcript` and a `transcripts[]` array. -- `document-service.ts:73-83` — every `updateTranscript` call merges all per-asset transcripts into the single primary via `mergeTranscripts`. -- The merge exists because axcut's `VirtualPreview` only consumes one transcript. -- **Collision:** with multi-asset projects, the merge is lossy — you can't tell which words came from which asset. -- **Resolution:** Phase 1 stores only `transcripts[]`; `VirtualPreview` (Phase 2) reads the per-clip `clip.wordRefs[]` instead of a single primary. Drop the `transcript` (singular) field in v3. - -### 1.3 OpenScreen's `trimRegions` are inverses of axcut's `skipRanges` 🟠 -- OpenScreen: `types.ts:211` — `TrimRegion { id, startMs, endMs }` — regions are **kept**. -- axcut: `axcut-schema/src/index.ts:79` — `skipRangeSchema` — regions are **removed**. -- **Resolution:** migration from v2 OpenScreen inverts: each `trimRegion` becomes a `skipRange` pair (everything outside trim = kept, inside trim = skipped). For the simple case of a single trim region `[a, b]`, you get `skipRanges = [{startSec: 0, endSec: a}, {startSec: b, endSec: duration}]`. -- **Edge case:** when an OpenScreen project has **zero trim regions** (entire recording kept), the migrated project has `skipRanges: []` and **one clip spanning the full duration**. This is the "single-take" baseline — should be the most common migration result. - -### 1.4 OpenScreen has fields axcut lacks 🔴 -| OpenScreen field | Map to AxcutDocument v3 | -|---|---| -| `editor.trimRegions[]` | invert into `skipRanges[]` | -| `editor.zoomRegions[]` | new `zoomRanges[]` field in `documentSchema` | -| `editor.annotationRegions[]` | new `annotations[]` field in `documentSchema` | -| `editor.speedRegions[]` | already in axcut as `timeline.speedRanges[]` (rangeSchema shape) | -| `editor.{wallpaper, shadowIntensity, showBlur, motionBlurAmount, borderRadius, padding, cropRegion, aspectRatio, webcamLayoutPreset, webcamMaskShape, webcamMirrored, webcamReactiveZoom, webcamSizePreset, webcamPosition, cursorTheme}` | new `legacyEditor` envelope; applied at export time | -| Cursor telemetry (`useScreenRecorder.ts` writes `.cursor.json` next to WebM) | **NOT** in document; stays as sidecar asset metadata. Add `asset.cursorTelemetryPath?` to `assetSchema`. | - -### 1.5 Schema fields need asset-scoped addressing 🟠 -- In a single-asset OpenScreen project, `zoomRegion.startMs/endMs` are timeline-absolute. -- In a multi-asset project, a zoom region applies to **one source asset** within one **clip**. The simplest model is: zoom regions are clip-scoped, not document-scoped. -- **Resolution:** in v3, `zoomRanges[]`, `annotations[]`, and `speedRanges[]` all carry `clipId?` (optional — null means "apply globally if there's exactly one clip, otherwise error"). OpenScreen's migrated projects (one clip) keep their regions with `clipId: null`. Multi-clip projects require `clipId`. -- This needs a `resolveRegionScope(document, region)` helper. - -### 1.6 axcut's `clip_` id generation is fragile 🟡 -- `timeline.ts:50-66` — auto-generated ids `clip_1, clip_2, ...`. New clips scan for the next free index. Could collide with user-provided ids. -- **Resolution:** use `crypto.randomUUID()` for all auto-generated clip ids (matching `TimelineEditor.tsx:17` pattern OpenScreen already uses). Update `retimeClips` to skip the numeric generation. - ---- - -## 2. Process model collisions (Phases 0, 6) - -### 2.1 Python worker + Faster-Whisper → drop entirely 🔴 -- axcut spawns Python as a child process via `PythonWorker` (`apps/server/src/services/python-worker.ts:14`). On Windows, finding `python` on PATH is unreliable; on macOS, codesigning a Python interpreter is a nightmare. -- OpenScreen's `transformers.js` Whisper (`src/lib/captioning/transcribe.ts:44`) runs in a Web Worker, already bundled via `caption-assets/`. -- **Resolution:** axcut's `enqueueTranscription` becomes a renderer-side call to `transcribeMono16kToSegments`. `enqueueAssetIngest` (which calls probe + proxy + transcribe) becomes: renderer reads metadata via `mediabunny` probe; transcribes via Whisper; updates the document. -- **Edge case:** axcut's Whisper defaults to `medium` (1.5 GB); OpenScreen bundles `tiny` (~75 MB). Need a user setting `transcriptionModel` (default `tiny`, opt-in to `medium`). The model download flow must happen in the renderer (transformers.js) — no Python download server. - -### 2.2 Fastify + SSE → Electron IPC + `webContents.send` 🔴 -- axcut: Fastify listens on `127.0.0.1:4010`, SSE stream at `GET /api/projects/:id/events`. -- OpenScreen: Electron main process + IPC handlers. -- **Resolution:** every axcut route becomes a `native-bridge:invoke` domain action (or a new `ai-edition:invoke` channel). The EventBus stays in the main process; fan-out happens via `webContents.send('ai-edition:event', { projectId, event })`. Renderer subscribes via a new `useAiEditionEvents(projectId)` hook. -- **Edge case:** SSE supports auto-reconnect; raw `webContents.send` does not. If the renderer reloads (e.g. user opens DevTools and hits Cmd+R), it must re-subscribe on mount. Plan accordingly in `VideoEditor.tsx`'s `useEffect`. -- **Edge case:** `webContents.send` only reaches the window it targets. Multiple editor windows? OpenScreen currently has one editor window; verify the renderer doesn't open multiple before assuming fan-out works. - -### 2.3 SQLite location `.axcut-data/metadata.sqlite` → `app.getPath('userData')` 🔴 -- axcut hardcodes the data dir under the repo. -- OpenScreen uses `app.getPath('userData')` (the OS-standard user data location: `~/Library/Application Support/OpenScreen` on macOS, `%APPDATA%/openscreen` on Windows). -- **Resolution:** move SQLite to `app.getPath('userData')/ai-edition/metadata.sqlite`. WAL mode is fine for Electron. -- **Edge case:** `better-sqlite3` is a native module. OpenScreen already has `@electron/rebuild` in devDeps — verify it's wired in `electron-builder.json5` rebuild step. If not, add `npm rebuild better-sqlite3` to the build script. - -### 2.4 axcut's `queueMicrotask` job runner → real cancellation 🔴 -- axcut `job-service.ts:18-22` schedules work via `queueMicrotask(() => void runX(...))`. No way to cancel. -- **Edge case:** user closes the editor mid-ingest. axcut's Fastify stays alive, the job keeps running, the renderer never sees the result. User is confused. -- **Resolution:** main-process job runner tracks `Set` per window. When the editor window closes, mark all its jobs as `cancelled` and abort the underlying `Worker.terminate()` (caption) or `AbortController.abort()` (HTTP fetch for transcription). - -### 2.5 axcut credentials in plain JSON → `safeStorage` 🟠 -- axcut: `llm-credentials.json` is plain JSON, written under `.axcut-data/`. -- **Security regression:** OpenScreen should not store credentials as plain JSON. Use Electron's `safeStorage.encryptString()` (OS keychain: Keychain on macOS, DPAPI on Windows, libsecret on Linux). -- **Resolution:** `LlmConfigService` (Phase 7) encrypts keys with `safeStorage` before write; decrypts on read. Keys migrate: on first run, detect plain JSON and re-encrypt. -- **Edge case:** `safeStorage.isEncryptionAvailable()` may return false on Linux without a desktop session. Fallback: `electron-store` with a user-set passphrase, or warn + opt-out. - ---- - -## 3. UX / UI collisions (Phases 1, 5) - -### 3.1 Single recording vs multi-take workflow 🔴 -- Today: `useScreenRecorder.ts:621` `stopRecording` → `switchToEditor()` is unconditional. -- Plan said: stay-in-recorder with prompt. **Conflict with the existing flow.** Need a decision: - - **Stay-in-recorder** (Screen Studio feel): every stop → toast "Saved to project X. Record another or open editor?" → two buttons. - - **Auto-open-editor** (today's feel): every stop → editor opens. The user's mental model matches existing. -- **Recommendation:** default to **stay-in-recorder** when `aiEdition` flag is on and there's an active project; default to **auto-open-editor** when there's no active project. The HUD needs a "Switch project" / "New project" affordance. -- **Edge case:** if the user starts recording without an active project, axcut's flow says "create a new project first." OpenScreen today has no project concept on the HUD. Need a quick-create button. -- **Edge case:** what if recording fails (stream error)? Plan path: don't create the asset; don't switch window; show toast. - -### 3.2 `Ctrl+C / Ctrl+V` clip duplicate collides with Electron Edit menu 🔴 -- Inventory §9 already flagged this. **Resolution:** - - Override the application menu's `Edit > Copy` / `Edit > Paste` roles to call `webContents.send('editor:clipboard', { op: 'copy' | 'paste' })` when the editor window is focused. The editor's React handler intercepts and triggers clip duplicate. - - In text inputs (``, `