Downloader analytics v2: multi-plot workspace, state timeline, fault decoding, and docs

server/installer/data-downloader/README.md

@@ -78,3 +78,263 @@ sequenceDiagram
 Set `POSTGRES_DSN` and `DEFAULT_SEASON_TABLE` to match your deployment so the SQL sent from `backend/server_scanner.py` and `backend/sql.py` queries the correct season table.
 
 All services mount `./data` inside the container and the FastAPI layer manages file I/O with atomic writes to keep data consistent between the worker and UI actions. If the rolling lookback produces no sensors, the collector now falls back to the oldest/newest run windows discovered by the date scanner, so no manual date tuning is required.
+
+---
+
+## Analysis workspace
+
+The Analysis tab is a post-session analytics workspace that lives alongside the existing Past Runs tab in the frontend. It queries TimescaleDB for signal time-series and vehicle state data, renders interactive multi-plot Plotly charts, and provides CSV export.
+
+### What it does
+
+- Plots one or more CAN signals on interactive Plotly charts within a run time window.
+- Adapts query density to the time range: raw samples for short windows, time-bucketed min/max/avg envelopes for wide windows.
+- Groups signals into separate plots with independent dual Y-axes, persisted per season in `localStorage`.
+- Renders a vehicle state timeline below the plots showing VCU state, inverter VSM state, and PM100 fault intervals decoded from the CAN bitfields.
+- Exports visible series data to CSV.
+
+### Architecture
+
+```mermaid
+flowchart LR
+  subgraph Backend ["FastAPI (data-downloader-api)"]
+    SERIES["/api/series"]
+    STATES["/api/states"]
+    TDB[("TimescaleDB")]
+    SERIES --> TDB
+    STATES --> TDB
+  end
+
+  subgraph Frontend ["React / Vite"]
+    TOOLBAR["AnalysisToolbar\n(run picker, custom range)"]
+    PICKER["AnalysisSignalPicker\n(DBC-grouped, drag and drop)"]
+    PLOTS["AnalysisPlotStack\n(Plotly multi-plot)"]
+    TIMELINE["AnalysisStateTimeline\n(state and fault lanes)"]
+    HOOK["useSeriesData\n(debounce, cache)"]
+    WORKSPACE["AnalysisWorkspace\n(orchestrator)"]
+
+    WORKSPACE --> TOOLBAR
+    WORKSPACE --> PICKER
+    WORKSPACE --> PLOTS
+    WORKSPACE --> TIMELINE
+    WORKSPACE --> HOOK
+  end
+
+  HOOK -->|"POST /api/series"| SERIES
+  WORKSPACE -->|"POST /api/states"| STATES
+```
+
+### Query pipeline
+
+1. The user selects a run (or enters a custom range) in the toolbar and picks signals from the signal picker.
+2. `AnalysisWorkspace` flattens the current plot layout into a deduplicated signal list and passes the range and signals to `useSeriesData`.
+3. `useSeriesData` debounces requests by 300 ms, checks a client-side LRU cache (1 M point budget), and calls `POST /api/series`.
+4. The backend validates the request, estimates total row count via TimescaleDB `approximate_row_count` or `COUNT(*)`, then runs a raw query or a `time_bucket` envelope query depending on density. A `SET LOCAL statement_timeout = 15000` guard prevents runaway queries.
+5. The frontend renders one Plotly chart per plot group, with linked x-axes so zoom on one chart zooms all charts.
+6. State timeline data is fetched once per run window (not refetched on zoom) via `POST /api/states`.
+
+### Density modes
+
+| Mode | Condition | Columns returned |
+|------|-----------|------------------|
+| `raw` | estimated rows per signal < 100 000 | `t`, `v` |
+| `envelope` | estimated rows per signal >= 100 000 | `t`, `min`, `max`, `avg` |
+
+The `target_points` parameter (default 4 000, max 20 000) controls the bucket count in envelope mode. The client auto-scales `target_points` based on the number of selected signals to stay within a shared 1 M point budget.
+
+---
+
+## Analytics API endpoints
+
+These are the endpoints added by the analytics feature. The existing downloader endpoints (`/api/runs`, `/api/sensors`, `/api/query`, etc.) are unchanged.
+
+### POST /api/series
+
+Returns time-series data for one or more signals within a time window.
+
+Request body:
+
+```json
+{
+  "season": "wfr26",
+  "signals": ["Motor_Speed", "Motor_Temp"],
+  "start": "2026-05-10T14:00:00Z",
+  "end": "2026-05-10T14:30:00Z",
+  "target_points": 4000
+}
+```
+
+Response shape:
+
+```json
+{
+  "season": "wfr26",
+  "start": "2026-05-10T14:00:00+00:00",
+  "end": "2026-05-10T14:30:00+00:00",
+  "series": {
+    "Motor_Speed": {
+      "mode": "raw",
+      "resolution_ms": null,
+      "point_count": 1200,
+      "t": [1715349600000],
+      "v": [3200.5]
+    }
+  }
+}
+```
+
+Envelope mode returns `min`, `max`, `avg` arrays instead of `v`.
+
+Limits:
+
+| Parameter | Constraint |
+|-----------|------------|
+| `signals` | 1 to 12, unique, valid identifiers |
+| `target_points` | 1 to 20 000 |
+| Time window | max 7 days |
+| Projected total points | max 1 500 000 (rejected before fetch) |
+
+### POST /api/states
+
+Returns vehicle state lane segments and decoded PM100 fault intervals.
+
+Request body:
+
+```json
+{
+  "season": "wfr26",
+  "start": "2026-05-10T14:00:00Z",
+  "end": "2026-05-10T14:30:00Z"
+}
+```
+
+Response shape:
+
+```json
+{
+  "season": "wfr26",
+  "start": "...",
+  "end": "...",
+  "lanes": [
+    {
+      "id": "car",
+      "signal": "State",
+      "label": "Car",
+      "segments": [
+        { "start_ms": 0, "end_ms": 5000, "value": 4, "label": "DRIVE" }
+      ]
+    }
+  ],
+  "faults": [
+    {
+      "name": "Over-current Fault",
+      "source": "run",
+      "segments": [{ "start_ms": 4000, "end_ms": 4100 }]
+    }
+  ]
+}
+```
+
+State lanes are derived by detecting value transitions with a 5-second gap tolerance. Labels come from the DBC `VAL_` definitions when available. Fault segments are decoded from the PM100 `INV_Post_Fault_Lo/Hi` and `INV_Run_Fault_Lo/Hi` bitfields using the Cascadia Motion CAN protocol fault name tables.
+
+---
+
+## Analytics frontend structure
+
+```
+frontend/src/
+├── analysis/
+│   ├── analysis-range.ts        Run-to-millisecond conversion, cross-season overlap
+│   ├── export-csv.ts            Series-to-CSV conversion and download trigger
+│   ├── plot-layout.ts           Pure model: groups, signal toggling, assignment, dual axis
+│   ├── plot-traces.ts           Plotly trace and layout builders for raw/envelope data
+│   ├── series-cache.ts          LRU cache with a shared 1M point budget
+│   ├── state-timeline.ts        Severity classification, segment box geometry, formatting
+│   └── use-series-data.ts       React hook: debounced fetch, cache, retry, clear
+├── components/
+│   ├── AnalysisWorkspace.tsx     Top-level orchestrator (range, signals, fetch, state)
+│   ├── AnalysisToolbar.tsx       Run picker, custom range inputs, export button
+│   ├── AnalysisSignalPicker.tsx  DBC-grouped signal list, click/drag to assign
+│   ├── AnalysisPlotStack.tsx     Plotly multi-plot renderer with drop targets
+│   ├── AnalysisStateTimeline.tsx Collapsible state and fault lane visualization
+│   ├── PlotAssignMenu.tsx        Accessible combobox for plot reassignment
+│   └── sensor-palette.ts        DJB2-hashed subsystem color palette
+└── types.ts                     Shared TypeScript interfaces
+```
+
+All pure logic modules under `analysis/` are side-effect-free and fully unit-tested. Component modules handle rendering, user interaction, and DOM integration.
+
+### State management
+
+Plot layout (which signals belong to which plot, right-axis assignments) is persisted to `localStorage` keyed by season name. On mount, persisted signals are pruned against the season's known signal list to avoid stale references. The layout is serialized after every toggle, assignment, or drag-and-drop.
+
+The state timeline collapse flag is also stored in `localStorage`.
+
+---
+
+## Analytics backend structure
+
+```
+backend/
+├── series_queries.py      Validation, density estimation, raw/envelope SQL, point budget
+├── state_queries.py        State lane assembly, PM100 fault bitfield decoding
+├── dbc_utils.py            DBC loading, signal grouping, signal_choices() for VAL_ labels
+├── app.py                  FastAPI routes including /api/series and /api/states
+└── tests/
+    ├── test_series_queries.py             12 tests: validation, SQL, envelope shaping
+    ├── test_series_endpoint.py            4 tests: happy path, errors, 503
+    ├── test_state_queries.py              20 tests: segments, gaps, seeds, fault decode
+    ├── test_states_endpoint.py            4 tests: happy path, unknown table, bad window
+    └── test_compose_api_env_allowlist.py  Compose env hygiene
+```
+
+### State query internals
+
+The state query pipeline avoids returning one row per CAN sample. Instead it:
+
+1. Queries value transitions using `LAG()` window functions with a configurable gap threshold (5 seconds).
+2. Seeds the initial state from a 1-hour lookback before the window start.
+3. Assembles transitions into contiguous segments, splitting on value changes and time gaps.
+4. Labels segments using DBC `VAL_` choice maps when available, falling back to numeric values.
+5. Decodes PM100 fault registers by scanning the 32-bit `Post_Fault` and `Run_Fault` words for set bits and mapping them to named faults via the Cascadia Motion fault tables.
+6. Merges overlapping fault intervals from the same named fault across multiple registers.
+
+The result is a few kilobytes per lane regardless of window length.
+
+---
+
+## Testing
+
+### Backend
+
+```bash
+cd server/installer/data-downloader
+.venv/bin/python -m pytest backend/tests/ -v
+```
+
+48 tests covering series query validation, SQL generation, envelope shaping, state segment assembly (value changes, gap detection, seed injection), PM100 fault decoding (hi/lo word, reserved bits), interval merging, and endpoint integration.
+
+### Frontend
+
+```bash
+cd server/installer/data-downloader/frontend
+npx vitest run
+```
+
+130 tests across 14 files covering all pure logic modules and every analysis component. Key coverage areas:
+
+- **AnalysisWorkspace** (21 tests) -- tab switching, season remount, persistence/pruning, refresh overlays, export guards, state timeline wiring, zoom interaction, error recovery
+- **useSeriesData** (13 tests) -- debounce, cache hits, error handling, retry, clear
+- **plot-traces** (12 tests) -- raw and envelope trace building, relayout parsing, axis configuration
+- **series-cache** (8 tests) -- LRU eviction, budget enforcement, oversize rejection
+- **state-timeline** (7 tests) -- severity classification, segment box geometry, formatting, fault overlap
+- **PlotAssignMenu** (6 tests) -- keyboard navigation, selection, accessibility
+
+---
+
+## Limitations
+
+- The state timeline tracks only `VCU_State_Info.State` and `M170_Internal_States.INV_VSM_State`. Adding more lanes requires updating `STATE_LANES` in `state_queries.py` and the severity maps in `state-timeline.ts`.
+- The PM100 fault tables are hardcoded from the Cascadia Motion CAN protocol. If the inverter firmware changes the fault bit layout, the tables in `state_queries.py` must be updated.
+- Signal names must match valid SQL identifiers (`[A-Za-z_][A-Za-z0-9_]*`). Signals with non-identifier names cannot be queried.
+- The client-side series cache is in-memory and shared across hook instances. It does not persist across page reloads.

server/installer/data-downloader/backend/app.py

@@ -15,9 +15,14 @@
 from fastapi.responses import HTMLResponse
 from pydantic import BaseModel
 
-from backend import series_queries
+from backend import series_queries, state_queries
 from backend.config import get_settings
-from backend.dbc_utils import group_sensors_by_message, load_dbc_db, refresh_dbc
+from backend.dbc_utils import (
+    group_sensors_by_message,
+    load_dbc_db,
+    refresh_dbc,
+    signal_choices,
+)
 from backend.services import DataDownloaderService
 
 
@@ -53,6 +58,12 @@ class SeriesPayload(BaseModel):
     target_points: int = series_queries.DEFAULT_TARGET_POINTS
 
 
+class StatesPayload(BaseModel):
+    season: str
+    start: datetime
+    end: datetime
+
+
 settings = get_settings()
 service = DataDownloaderService(settings)
 logger = logging.getLogger(__name__)
@@ -242,6 +253,37 @@ def query_series(payload: SeriesPayload) -> dict:
         raise HTTPException(status_code=503, detail=f"Database query failed: {exc}")
 
 
+@app.post("/api/states")
+def query_states(payload: StatesPayload) -> dict:
+    try:
+        state_queries.validate_states_request(payload.start, payload.end)
+        if payload.season not in {item["table"] for item in service.get_seasons()}:
+            raise ValueError(f"unknown season table: {payload.season}")
+        db, _source = load_dbc_db(settings)
+        choices = {
+            lane["signal"]: signal_choices(db, lane["signal"])
+            for lane in state_queries.STATE_LANES
+        }
+        result = state_queries.execute_states_query(
+            postgres_dsn=settings.postgres_dsn,
+            table=payload.season,
+            start=payload.start,
+            end=payload.end,
+            choices_by_signal=choices,
+        )
+        return {
+            "season": payload.season,
+            "start": series_queries.normalize_utc(payload.start).isoformat(),
+            "end": series_queries.normalize_utc(payload.end).isoformat(),
+            **result,
+        }
+    except ValueError as exc:
+        raise HTTPException(status_code=400, detail=str(exc))
+    except Exception as exc:
+        logger.exception("States query failed for %s", payload.season)
+        raise HTTPException(status_code=503, detail=f"Database query failed: {exc}")
+
+
 # ── CAN frames batch ingest (for flight-recorder sync) ─────────────────────────
 
 # Cache of known signal columns per table, thread-safe

server/installer/data-downloader/backend/dbc_utils.py

@@ -260,3 +260,14 @@ def group_sensors_by_message(sensor_names: list[str], db) -> dict:
         m["signals"].sort()
 
     return {"messages": messages, "ungrouped": sorted(ungrouped)}
+
+
+def signal_choices(db, signal_name: str) -> Optional[dict]:
+    """Return the DBC VAL_ label map for a signal as {int: str}, or None."""
+    if db is None:
+        return None
+    for message in db.messages:
+        for signal in message.signals:
+            if signal.name == signal_name and signal.choices:
+                return {int(k): str(v) for k, v in signal.choices.items()}
+    return None