Add the functionality to automatically register images in the "Camera Registration" tab#1769
Open
romleiaj wants to merge 14 commits into
Open
Add the functionality to automatically register images in the "Camera Registration" tab#1769romleiaj wants to merge 14 commits into
romleiaj wants to merge 14 commits into
Conversation
New 'Auto Align (current frame)' button computes the camera-pair alignment automatically using the MINIMA-LoFTR matcher hosted by the VIAME interactive service (companion VIAME branch dev/auto-align), a LoFTR fine-tuned for cross-modality matching (EO<->IR). Flow: the panel asks the platform bridge for an alignment of the selected pair on the current frame; desktop ViewerLoader resolves each camera's image path (same per-camera getters stereo uses) and sends both to the interactive service, which returns a homography plus ~24 spatially-spread inlier correspondences in native pixels. Those are injected as ordinary point pairs (CameraRegistrationStore. applyAutoAlignment): they appear in the correspondences table, are hand-refinable like picked points, fit through the normal homography path, and persist through the existing save and Export-menu downloads. Provenance (autoAlignModel, autoAlignInlierRatio) is stamped into the registration source. Desktop-only, mirroring the text-query wiring end to end: interactive.ts autoAlign -> ipcService alignment-auto-align / alignment-available (matcher weights present in the VIAME install) -> frontend api -> Viewer's provide/inject auto-align bridge (dive-common/use/useAutoAlign) -> RegistrationTools. Web (and installs without the weights) never see the button. Unmatchable scenes surface the service's actionable error (e.g. 'try a frame with more structure') instead of applying a bad warp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
handleAutoAlign gated on loadStereoMetadata(), which rejects anything that isn't a stereoscopic (subType === 'stereo') dataset. A plain multicam rig -- e.g. an EO/IR pair (subType === 'multicam') -- therefore failed with "Auto-align requires a multi-camera dataset" even though deep-feature alignment needs only two cameras' images for the current frame, not a calibrated stereo pair. Factor the (stereo-agnostic) per-camera image-path-getter population out of loadStereoMetadata into populateMultiCamImagePathGetters, and add loadMultiCamMetadata that gates on isMultiCamDatasetMeta (stereo OR multicam). handleAutoAlign now uses the latter. loadStereoMetadata keeps its isStereoscopicDatasetMeta gate unchanged, so the stereo service still never loads on a non-stereo dataset; the getter-building code is moved verbatim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The button creates a camera-pair REGISTRATION -- it automates exactly what the Camera Registration panel does by hand -- so name it with the registration vocabulary (registration = creating the transform, Align View = consuming it). Renames the UI strings, the store method (applyAutoRegistration), the provide/inject bridge (dive-common/use/useAutoRegister), the apispec types (AutoRegisterRequest/Response), the desktop API and IPC channels (auto-register / auto-register-available), the provenance keys stamped into the registration source (autoRegisterModel, autoRegisterInlierRatio), and the feature-plan doc. The interactive service's wire name (command: 'auto_align', VIAME branch dev/auto-align) is deliberately unchanged -- it renames in step with the service side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
romleiaj
marked this pull request as draft
July 20, 2026 19:57
Collaborator
Author
|
See VIAME/VIAME#272 |
The injected correspondences were invisible after a successful Auto Register unless the user separately flipped the "Pick points" toggle, because the keypoint layer draws nothing while picking is off. Enable picking in applyAutoRegistration so the points appear immediately for review, matching the summary text's "Review the points" instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
Since Auto Register and registration files can both populate the correspondence table, the toggle's job is broader than placing new points: it gates seeing, dragging, and deleting them too. Comments that referenced the toggle by name follow suit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
applyAutoRegistration merged its matcher provenance (model, inlier ratio) into cameraRegistrationSource, but that stamp is rig-global and written into EVERY per-camera registration file. Auto-registering (or then refining) one pair therefore changed every other camera's file body too, so saving after touching only UV points warned it would overwrite the IR file as well, and the Source readout blended the producer's lineage with matcher details. Drop the stamp: the pair's divergence from a loaded producer registration already surfaces through the refined-from-source warning, like any in-app refit. Persisted per-pair matcher provenance is deferred until the file format has a pair-level source field; the plan doc now records that decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
The divergence note is a persistent status by design (the pair really has diverged from the producer registration, before and after saving), but its copy always opened with "Save, then download...", which reads as a pending to-do even right after a save. Key the action hint on the dirty flag: while unsaved it still says save first; once saved it goes straight to the export instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4dW5stcXQBj89d5Kc35bw
Its design rationale now lives in code comments and the PR description; the status/history sections were already stale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
Replace the Promise<any> return and duplicated inline request shapes with AutoRegisterRequest/AutoRegisterResponse from apispec, plus a local interface for the service's snake_case wire payload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The current-frame qualifier moves to a tooltip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
The count includes the reference camera, which is ready by definition but was never registered to anything -- a fresh 3-camera rig read "1/3 cameras registered" before any work happened. Applies to the panel summary, the Align View tooltip, and the import menu hint, which share the same registrationProgress semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013MgbsJaWzbWbntMwAt47Wa
Collaborator
Author
Branch summary:
|
romleiaj
marked this pull request as ready for review
July 21, 2026 19:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uses MINIMA-LoFTR under the hood to compute features and match points - a model tuned to work across modalities (EO/IR)