Skip to content

Add the functionality to automatically register images in the "Camera Registration" tab#1769

Open
romleiaj wants to merge 14 commits into
mainfrom
dev/auto-image-registration
Open

Add the functionality to automatically register images in the "Camera Registration" tab#1769
romleiaj wants to merge 14 commits into
mainfrom
dev/auto-image-registration

Conversation

@romleiaj

Copy link
Copy Markdown
Collaborator

Uses MINIMA-LoFTR under the hood to compute features and match points - a model tuned to work across modalities (EO/IR)

romleiaj and others added 6 commits July 20, 2026 15:38
New 'Auto Align (current frame)' button computes the camera-pair
alignment automatically using the MINIMA-LoFTR matcher hosted by the
VIAME interactive service (companion VIAME branch dev/auto-align), a
LoFTR fine-tuned for cross-modality matching (EO<->IR).

Flow: the panel asks the platform bridge for an alignment of the
selected pair on the current frame; desktop ViewerLoader resolves each
camera's image path (same per-camera getters stereo uses) and sends
both to the interactive service, which returns a homography plus ~24
spatially-spread inlier correspondences in native pixels. Those are
injected as ordinary point pairs (CameraRegistrationStore.
applyAutoAlignment): they appear in the correspondences table, are
hand-refinable like picked points, fit through the normal homography
path, and persist through the existing save and Export-menu downloads.
Provenance (autoAlignModel, autoAlignInlierRatio) is stamped into the
registration source.

Desktop-only, mirroring the text-query wiring end to end:
interactive.ts autoAlign -> ipcService alignment-auto-align /
alignment-available (matcher weights present in the VIAME install) ->
frontend api -> Viewer's provide/inject auto-align bridge
(dive-common/use/useAutoAlign) -> RegistrationTools. Web (and installs
without the weights) never see the button. Unmatchable scenes surface
the service's actionable error (e.g. 'try a frame with more
structure') instead of applying a bad warp.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
handleAutoAlign gated on loadStereoMetadata(), which rejects anything that
isn't a stereoscopic (subType === 'stereo') dataset. A plain multicam rig --
e.g. an EO/IR pair (subType === 'multicam') -- therefore failed with
"Auto-align requires a multi-camera dataset" even though deep-feature
alignment needs only two cameras' images for the current frame, not a
calibrated stereo pair.

Factor the (stereo-agnostic) per-camera image-path-getter population out of
loadStereoMetadata into populateMultiCamImagePathGetters, and add
loadMultiCamMetadata that gates on isMultiCamDatasetMeta (stereo OR multicam).
handleAutoAlign now uses the latter. loadStereoMetadata keeps its
isStereoscopicDatasetMeta gate unchanged, so the stereo service still never
loads on a non-stereo dataset; the getter-building code is moved verbatim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The button creates a camera-pair REGISTRATION -- it automates exactly
what the Camera Registration panel does by hand -- so name it with the
registration vocabulary (registration = creating the transform, Align
View = consuming it). Renames the UI strings, the store method
(applyAutoRegistration), the provide/inject bridge
(dive-common/use/useAutoRegister), the apispec types
(AutoRegisterRequest/Response), the desktop API and IPC channels
(auto-register / auto-register-available), the provenance keys stamped
into the registration source (autoRegisterModel,
autoRegisterInlierRatio), and the feature-plan doc.

The interactive service's wire name (command: 'auto_align', VIAME
branch dev/auto-align) is deliberately unchanged -- it renames in step
with the service side.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@romleiaj
romleiaj marked this pull request as draft July 20, 2026 19:57
@romleiaj

Copy link
Copy Markdown
Collaborator Author

See VIAME/VIAME#272

romleiaj and others added 4 commits July 20, 2026 16:06
The injected correspondences were invisible after a successful Auto
Register unless the user separately flipped the "Pick points" toggle,
because the keypoint layer draws nothing while picking is off. Enable
picking in applyAutoRegistration so the points appear immediately for
review, matching the summary text's "Review the points" instruction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
Since Auto Register and registration files can both populate the
correspondence table, the toggle's job is broader than placing new
points: it gates seeing, dragging, and deleting them too. Comments that
referenced the toggle by name follow suit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
applyAutoRegistration merged its matcher provenance (model, inlier
ratio) into cameraRegistrationSource, but that stamp is rig-global and
written into EVERY per-camera registration file. Auto-registering (or
then refining) one pair therefore changed every other camera's file
body too, so saving after touching only UV points warned it would
overwrite the IR file as well, and the Source readout blended the
producer's lineage with matcher details.

Drop the stamp: the pair's divergence from a loaded producer
registration already surfaces through the refined-from-source warning,
like any in-app refit. Persisted per-pair matcher provenance is
deferred until the file format has a pair-level source field; the plan
doc now records that decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
The divergence note is a persistent status by design (the pair really
has diverged from the producer registration, before and after saving),
but its copy always opened with "Save, then download...", which reads
as a pending to-do even right after a save. Key the action hint on the
dirty flag: while unsaved it still says save first; once saved it goes
straight to the export instruction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
romleiaj and others added 4 commits July 21, 2026 14:30
Its design rationale now lives in code comments and the PR description;
the status/history sections were already stale.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
Replace the Promise<any> return and duplicated inline request shapes with
AutoRegisterRequest/AutoRegisterResponse from apispec, plus a local
interface for the service's snake_case wire payload.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The current-frame qualifier moves to a tooltip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The count includes the reference camera, which is ready by definition but
was never registered to anything -- a fresh 3-camera rig read "1/3
cameras registered" before any work happened. Applies to the panel
summary, the Align View tooltip, and the import menu hint, which share
the same registrationProgress semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
@romleiaj

Copy link
Copy Markdown
Collaborator Author

Branch summary: dev/auto-image-registration

What it adds: an "Auto Register" button in the Camera Registration panel that computes a cross-modality (e.g. EO↔IR) homography between two cameras automatically, using a deep feature matcher (MINIMA-LoFTR) hosted in the VIAME desktop interactive service. ~550 added lines over 14 commits.

How it workds

  • Interactive service / IPC (interactive.ts, ipcService.ts): a new register_images wire command sends two image paths (plus optional RANSAC/matching thresholds) to the Python service, which returns a 3×3 homography, the inlier correspondences, and quality stats. The matcher model loads lazily on first use and stays resident. A companion register-images-available handler probes whether the matcher weights exist in the VIAME install; without them the button is hidden entirely. The boundary is typed end to end: both the IPC handler and the service method use the shared AutoRegisterRequest/AutoRegisterResponse types, with a local interface for the service's snake_case wire payload.
  • API contract (apispec.ts): new AutoRegisterRequest/AutoRegisterResponse types, including machine-readable failure codes (insufficient_matches, low_confidence, degenerate_homography).
  • Platform bridge (useAutoRegister.ts, Viewer.vue, ViewerLoader.vue): a provide/inject service keeps the shared UI platform-agnostic — desktop's ViewerLoader passes a handler prop that resolves each camera's image path for the current frame and calls the service; on web nothing is provided, so the panel shows no button. Along the way, the stereo-only image-path-getter setup in ViewerLoader was refactored into a shared populateMultiCamImagePathGetters, because Auto Register works on any multi-camera dataset, not just stereoscopic ones (this does touch stereo code).
  • UI + store (RegistrationTools.vue, CameraRegistrationStore.ts): the button runs the matcher on the current frame's pair, confirms before replacing any existing points/transform, then injects the returned inliers as ordinary correspondence points — so the user can inspect, drag, or delete them exactly like hand-picked ones — switches the pair to a homography fit, and refits. Point editing is auto-enabled so the injected points are immediately visible. Results surface as a success summary (point count + match consensus) or an actionable error.
  • Copy changes: the feature was renamed Auto Align → Auto Register (UI, IPC, and wire names all updated), and the "Pick points" toggle became "Edit points." The button reads "Auto Register" with the current-frame qualifier in a tooltip. The rig status now says "N/M cameras ready" instead of "registered" (the count includes the reference camera, which is ready by definition but was never registered to anything) — this touches the panel summary, the Align View tooltip, and the import menu hint.

Tests

CameraRegistrationStore.spec.ts gains coverage for applyAutoRegistration (point injection, homography switch, source non-restamping), and alignedViewTooltip.spec.ts is updated for the "cameras ready" wording. The service/IPC layer and Vue wiring are untested — the Python matcher side lives in the separate VIAME/service repo, not this diff.

@romleiaj
romleiaj marked this pull request as ready for review July 21, 2026 19:05
@romleiaj
romleiaj requested a review from BryonLewis July 21, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant