sphere-sfm-v2: migrate to upstream COLMAP 4.x#55
Open
bidwej wants to merge 2863 commits into
Open
Conversation
I accidentally already committed these into the previous PR. This moves them into the colmap folder.
… size (#4003) - Add an additional sorting step to the end of reconstruciton_clustering so that clusters are sorted by their sizes. - Cluster with too few frames are marked with id -1 - Add tests for reconstruction_clustering.
Same results but improved code/comments/tests.
No change in behavior.
Addresses issues with inconsistent handling and definition of ENU origins. Cleans up comments. Breaks the Python interface.
No change in behavior.
Some overdue housekeeping: 1. src/pycolmap/estimators/homography_matrix.cc - Removed homography_matrix_estimation alias 2. src/pycolmap/estimators/pose.cc - Removed absolute_pose_estimation alias 3. src/pycolmap/estimators/fundamental_matrix.cc - Removed fundamental_matrix_estimation alias 4. src/pycolmap/estimators/two_view_geometry.cc - Removed squared_sampson_error alias 5. src/pycolmap/estimators/essential_matrix.cc - Removed essential_matrix_estimation alias 6. src/pycolmap/geometry/homography_matrix.cc - Removed homography_decomposition alias 7. src/pycolmap/feature/extraction.cc - Removed backwards-compatibility SIFT defaults warning 8. src/pycolmap/scene/camera.cc - Removed 3 deprecated img_from_cam overloads (2D point versions) 9. src/pycolmap/geometry/triangulation.cc - Removed TriangulatePoint and CalculateTriangulationAngle aliases 10. src/pycolmap/estimators/generalized_pose.cc - Removed rig_absolute_pose_estimation alias All the newer deprecations (after Nov 7, 2025) are still in place: - correspondence_graph.cc:138-144 - Jan 10, 2026 deprecations (kept) - camera.cc:205 - Jan 12, 2026 Camera.create deprecation (kept) - pose_prior.cc:36-37 - Dec 2, 2025 deprecations (kept)
…e and storage (#4080) * Backwards compatible for old databases * Breaking change on the pycolmap interface * Fails visual index and matching if descriptor types are inconsistent
Fixes:
```
/project/python/../src/pycolmap/feature/types.h:14: warning: type ‘struct type_caster’ violates the C++ One Definition Rule [-Wodr]
14 | PYBIND11_MAKE_OPAQUE(colmap::FeatureKeypoints);
/tmp/build-env-vmbxo2bj/lib/python3.11/site-packages/pybind11/include/pybind11/stl.h:366: note: a type with different bases is defined in another translation unit
366 | struct type_caster<std::vector<Type, Alloc>> : list_caster<std::vector<Type, Alloc>, Type> {};
/project/python/../src/pycolmap/feature/types.h:16: warning: type ‘struct type_caster’ violates the C++ One Definition Rule [-Wodr]
16 | PYBIND11_MAKE_OPAQUE(colmap::FeatureMatches);
/tmp/build-env-vmbxo2bj/lib/python3.11/site-packages/pybind11/include/pybind11/stl.h:366: note: a type with different bases is defined in another translation unit
366 | struct type_caster<std::vector<Type, Alloc>> : list_caster<std::vector<Type, Alloc>, Type> {};
/project/src/pycolmap/estimators/bundle_adjustment.cc: In member function ‘Problem’:
/project/src/pycolmap/estimators/bundle_adjustment.cc:47:3: warning: function may return address of local variable [-Wreturn-local-addr]
47 | }
| ^
/tmp/build-env-vmbxo2bj/lib/python3.11/site-packages/pybind11/include/pybind11/cast.h:1651:35: note: declared here
1651 | return cast_op<T>(load_type<T>(handle));
```
- Update clang-format from 20.1.5 to 21.1.8 - No code changes required - existing code passes formatting with new version
## Summary - Update cibuildwheel from v3.1.2 to v3.3.1 - Update ccache from 4.8.2 to 4.12.2 - Update Jimver/cuda-toolkit from v0.2.29 to v0.2.30 - Update sphinx from 7.4.7 to 9.1.0 - Update sphinx-rtd-theme from 2.0.0 to 3.1.0 - Update sphinx-toolbox from 3.8.0 to 4.1.2 - Update enlighten from 1.13.0 to 1.14.1 ## Test plan - [x] CI passes for Ubuntu builds (ccache update) - [x] CI passes for Windows builds (cuda-toolkit update) - [x] CI passes for pycolmap wheel builds (cibuildwheel update) - [x] Documentation builds successfully (sphinx updates)
…ot README (#4091)
colmap/colmap#4086 broke the ordering of the API documentation, as reflected at https://colmap.github.io/pycolmap/pycolmap.html (we want class members to be ordered by source, not alphabetically, such that properties are grouped together). This PR fixes it. I haven't figured out how to use the newer, non-legacy API, so leaving this for later.
Adding new ALIKED feature extractor and ALIKED_BRUTEFORCE matcher. Based on the ONNX conversion code in: https://github.com/colmap/ALIKED-ONNX/tree/user/jsch/onnx-export. ONNX model files published as part of the 3.13.0 release artifacts. Vocabulary tree to be trained as a follow-up.
The comment is outdated and this works without a problem. Appears to have been a problem with older Qt versions.
The occlusion check in the texturing code has a bug causing texture projected "behind" objects. (Wall behind chair) The current implementation incorrectly assumes that if the intersection between camera and target triangle is the target triangle, there is no occlusion possible. This is false because the intersection check used is `any_intersection` which is not ordered and another intersection could occur between the camera and triangle. https://github.com/colmap/colmap/blob/1169d14d98a8d09bf8dd87ce3df44a186043068f/src/colmap/mvs/texture_mapping.cc#L247-L254 I added a small offset to exclude the target triangle from the checked range. This potentially has some numerical issues but doesn't have an effect on performance. The numerically correct option would be to check the first intersection in this case but that adds complexity and impacts runtime. Current implementation: <img width="1938" height="1500" alt="texturing_bug" src="https://github.com/user-attachments/assets/4989c857-0c0b-4d55-93c0-a996e9494f3b" /> Fix with offset: <img width="1938" height="1500" alt="texturing_offset" src="https://github.com/user-attachments/assets/c60d26fa-8e31-4d38-b76e-d06c03ff4e86" /> Fix with first_intersection: <img width="1938" height="1500" alt="texturing_first_intersection" src="https://github.com/user-attachments/assets/0dfaa88b-8459-42cd-99dc-342794ccf9bf" /> There are some minor differences but I propose to use the simple offset based fix until other major issues are encountered. --------- Co-authored-by: Johannes Schönberger <jsch@demuc.de> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
- Change `mvs::Image::SetBitmap` to take `Bitmap` by value and `std::move` into the member, so callers with an owned `Bitmap` can move instead of being forced to copy. - Update callsites in `exe/mvs.cc` and `texture_mapping_test.cc` to `std::move` the local `Bitmap` into the call. - `patch_match.cc` passes a `const Bitmap&` returned from `Workspace::GetBitmap`, so it still copies (same as before). --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
same as colmap/colmap#4412 but to the main branch. Better merge this one and cherry-pick it to the release branches. Commit colmap/colmap@1625d9e accidentally deleted the required COMPONENTS keyword from `cmake/FindDependencies.cmake` when searching for Qt. `find_package(Qt${QT_VERSION_MAJOR} ${COLMAP_FIND_TYPE} COMPONENTS ${COLMAP_QT_COMPONENTS})`
Fixes #4362.
## Summary
The L1 ADMM and IRLS Cholesky factorizations of A^T (W) A in rotation
averaging silently produced NaN steps when CHOLMOD's supernodal LLT
reported "matrix not positive definite" on poorly conditioned but
mathematically PD pose graphs (e.g., long sequential video chains from a
single camera). The NaN step corrupted the rotation state and surfaced
later as the cryptic `nan error` / `nan weight!` log messages, aborting
the global mapper before any reconstruction could be produced.
The original revision of this PR only made the failure explicit (added a
small ridge regularization and propagated `Eigen::Success` checks). On
the reporter's dataset that still failed, because CHOLMOD's supernodal
solver is strict about positive definiteness — it aborts at the first
non-positive pivot during dense supernode block factorization, even when
a small ridge has been added. The fix below is robust regardless of
ridge value or SuiteSparse version.
## Changes
- New `SparseCholeskyWithFallbackSolver` in
`src/colmap/optim/sparse_cholesky.{h,cc}`. Tries
`Eigen::CholmodSupernodalLLT` first (fastest); on failure, transparently
falls back to `Eigen::SimplicialLDLT` (more numerically tolerant — LDLT
does not require strict PD because it does not take square roots of
pivots).
- `LeastAbsoluteDeviationSolver` (L1 ADMM):
`SupernodalCholmodLLTLinearSolver` now delegates to the new helper.
`Valid()` reflects success of either path. `Solve()` returns false
instead of NaN if both fail.
- `RotationAveragingSolver::SolveIRLS`: uses the new helper directly per
iteration. Detects NaN steps explicitly before applying `UpdateState`.
- `RotationEstimatorOptions::ridge_regularization`: defaults to 0 (no
overhead). `GlobalMapperOptions` opts in with `1e-9` to stabilize
ill-conditioned-but-PD graphs that the global SfM pipeline routinely
encounters.
## Tests
- `sparse_cholesky_test.cc`: covers Compute/Solve smoke cases, the IRLS
analyze-once / factorize-many reuse pattern, true singular failure, and
(most importantly) a deterministic fallback test using an indefinite
diagonal that CHOLMOD must reject and LDLT must accept.
- `rotation_averaging_test.cc`: regression test that
`ridge_regularization` does not bias the solution.
- `least_absolute_deviations_test.cc`: regression test for the ridge
path.
Verified against the reporter's database from #4362 (638 iPhone images):
the full glomap pipeline completes cleanly end-to-end with no CHOLMOD
warnings.
- Fixes #4422: `pycolmap.Database()` aborted with `Tried to call pure virtual function "Database::Close"` whenever the instance was garbage collected. - The `PyDatabaseImpl` trampoline destructor called `Close()`, which is pure virtual and dispatches to a Python override via `PYBIND11_OVERRIDE_PURE`. When the user instantiated `Database` directly (without subclassing), no override exists and the macro throws — from a destructor — calling `std::terminate`. - Wraps the `Close()` call in `try/catch` so direct instantiation no longer crashes. Python subclasses that override `Close` still get it invoked on destruction.
We found that on NixOS aarch64 `angularDistance` returns `1e-17` for the same quaternion (probably due to FMA or similar?). If we check equality explicitly, the test passes.
`--PoissonMeshing.num_threads` didn't actually do anything before, this fixes it. Also, catch exceptions.
Extends Caspar BA to correctly handle multi-camera rig datasets. sensor_from_rig is added as a ConstantShared parameter to all existing kernels. Identity is passed for single-camera datasets (some overhead), so there is no behavioral change for the common case. Non-ref rig cameras, previously skipped with a warning, are now included in the optimization. refine_sensor_from_rig = true remains unsupported. TL;DR: - Rig support with constant cam from rig now working - *refine_sensor_from_rig=false*, and remains unsupported - ok performance on single track trajectories, great on large internet collections of data (1DSfM) - Improved setup time in Caspar - Convergence tests added - Added BA backend options to automatic reconstruction Edit: Comment on accuracy in single-track trajectories: Doing one final global pass with Ceres resulted in AUC@1.0 50% of that of Ceres in challenging scenes. This could be considered as an option. The degradation in accuracy is due to Caspar using Conjugate Gradient, which make intrinsics challenging to refine in long camera chains. --------- Co-authored-by: Shaohui Liu <b1ueber2y@gmail.com>
Fixes colmap/colmap#4439 Two bugs have been identified: * (critical) memcpy uses a wrong size so only partial features are copied. * The SIFT features were already root normalized in COLMAP, so applying root normalization again would lead to 1/4 normalization.
…ks (#4445) Fixes colmap/colmap#4438 ## Problem When migrating from standalone GLOMAP to COLMAP 4's `global_mapper`, there was no equivalent of the old `--TrackEstablishment.max_num_tracks` flag, which capped the total number of established tracks to bound memory usage and avoid OOM on large datasets. The existing `GlobalMapper.track_required_tracks_per_view` option only controls per-view early stopping and defaults to no capping. ## Change Adds a new `GlobalMapper.keep_max_num_tracks` option (default: unlimited). During track establishment, tracks are already sorted by decreasing length; the selection loop now stops once this budget is reached, keeping the longest (most valuable) tracks. The option is exposed via the C++ `GlobalMapperOptions` struct, the CLI option manager, and the pycolmap bindings.
## Summary Implement the Advancing Front Surface Reconstruction (AFSR) algorithm based on Cohen-Steiner & Da (The Visual Computer, 2004), with visibility-based filtering following the [cvg/pcdmeshing](https://github.com/cvg/pcdmeshing) reference implementation. ### Key differences from the pcdmeshing reference - **Integrated into COLMAP pipeline**: CLI command (`colmap advancing_front_mesher`), Python bindings (`pycolmap.advancing_front_meshing()`), automatic reconstruction pipeline, and option manager — not a standalone tool - **Visibility from COLMAP workspace**: Loads visibility directly from `fused.ply.vis` and sparse reconstruction, rather than requiring separate endpoint/observation PLY files - **Consistent visibility threshold**: Both pre-filtering and post-filtering modes use `visibility_filtering_max_intersections` to only remove faces exceeding N ray intersections. The reference post-filtering removes any face hit by a single ray - **Block-wise visibility filtering**: Rays are assigned to blocks via segment-AABB intersection tests, so visibility filtering works correctly across block boundaries. The reference disables visibility in block-wise mode entirely - **Per-thread local accumulation**: Visibility counters and face sets use per-thread local maps merged after the parallel region, avoiding the `#pragma omp critical` bottleneck in the reference - **CGAL 6.x compatibility**: Uses `CGAL::AABB_traits_3` (non-deprecated), `CGAL::do_intersect` instead of `CGAL::assign`, and `CGAL::Euler::remove_face` for correct halfedge connectivity - **EPIC float32 kernel**: Same float32 approach as the reference for memory efficiency, but using `CGAL::Filtered_kernel_adaptor` for exact predicates - **File decomposition**: Existing `meshing.h/cc` split into `poisson_meshing`, `delaunay_meshing`, `advancing_front_meshing` with separate test files Sample result: <img width="2490" height="1608" alt="advancing front" src="https://github.com/user-attachments/assets/dbd22512-5a0e-4e0a-9835-8a55543073e0" />
+ bind ComputeRot90FromGravity
## Summary Update vcpkg registry baseline in `vcpkg-configuration.json` to latest commit on microsoft/vcpkg master branch. - Old: `36fb57eb3878cc3422351a91d3e87c328388dabd` - New: `a0b1c8d3a477c1cb4813d8e127a56961707ca42b` This keeps COLMAP in sync with upstream vcpkg package versions.
- Set has_prior_focal_length in the rig config and update apply_rig_config to correctly set it in the database. - Also set it in the default camera (along with the correct parameters) to be safe.
This is not necessarily a PR for merging, just expanding on #4440 and putting it through its paces to find bugs. Here is a test of this PR's spherical camera model implemented on top of it: https://github.com/user-attachments/assets/ae416c79-87e1-4b82-b55f-87abdb1ab164 This PR is now detecting way more points than my previous PR, so I think it's a sign that it's operating more natively in equirect space. #4440 had a bug where I'm including the script I used to generate this in the Python examples folder for easier validation. If we did want to merge my attempt at the Spherical Mapper, here's what I need to finish: - [x] Test with GPU Acceleration (CuDSS Ceres) - [x] Solve why only the glomap path is losing all the points in one hemisphere - [x] Make the example script use pycolmap instead of the CLI (keeping the super fast FFMPEG-based extraction, etc.) # Native SPHERICAL SfM — CPU vs GPU benchmark Controlled benchmark of the native equirectangular (`SPHERICAL`) SfM pipeline (`run_spherical_sfm.py`) measuring full wall-clock time from **video decomposition → TXT model**, for the incremental (`mapper`) and global (`global_mapper`) backends, on CPU vs GPU. ## Setup - **Hardware:** NVIDIA RTX 4090 (sm_89, 24 GB); 8 build/compute threads. - **colmap:** `feature/spherical-on-cam-ray` (commit 8ee2a7b8), built with CUDA (g++-10 CUDA host compiler, sm_89). Ceres 2.3.0 with CUDA + cuDSS (CUDA-12 variant) for GPU bundle adjustment. - **Input:** `output2.mp4` — 8K equirectangular (7680×3840) 360° video, rotations canceled during stitching. - **Frames:** 66 frames extracted (every 20th), downscaled 4× → 1920×960. - **Camera model:** native `SPHERICAL` (id 17), `--ImageReader.single_camera 1`. - CPU runs use `--FeatureExtraction/FeatureMatching.use_gpu 0`; GPU runs use GPU SIFT + matching and `--Mapper.ba_use_gpu 1` (incremental). ## Timing: video → TXT model (seconds) | Stage | CPU incremental | CPU global | GPU incremental | GPU global | |---|---|---|---|---| | Video → frames (ffmpeg) | 35.0 | 35.0 | 14.0 | 14.0 | | Feature extraction (SIFT) | 55.9 | 56.3 | **2.6** | **2.6** | | Matching (sequential) | 28.2 | 28.1 | **3.8** | **3.9** | | Mapping | 30.8 | 23.4 | 29.1 | 25.6 | | TXT export | ~0.5 | ~0.5 | ~0.5 | ~0.5 | | **Total (video→txt)** | **~150 s** | **~143 s** | **~50 s** | **~46 s** | Running both modes in a single invocation (one shared video decode): **261 s on CPU vs 106 s on GPU.** ## Quality (largest reconstructed sub-model) | Metric | CPU incremental | CPU global | GPU incremental | GPU global | |---|---|---|---|---| | Registered images | 66 / 66 | 66 / 66 | 66 / 66 | 66 / 66 | | 3D points | 24,950 | 28,288 | 24,730 | 27,746 | | Mean track length | 5.42 | 5.28 | 5.37 | 5.24 | | Mean reprojection error | 0.545 px | 0.0015 px | 0.576 px | 0.0015 px | ## Takeaways - **GPU vs CPU:** ~**2.1×** end-to-end per mode, with **identical quality** (same registration, ±1% points, comparable error — the small deltas are just GPU vs CPU SIFT producing slightly different features). - **The front-end is where the GPU wins:** SIFT 56 s → 2.6 s (~21×), matching 28 s → 3.8 s (~7×). Mapping is largely CPU-bound graph/triangulation work and barely moves (GPU bundle adjustment helps a little: dense-CUDA for incremental, cuDSS sparse for global). - **Incremental vs global:** both register all 66 images; global yields ~13% more points. The reprojection-error figures are **not directly comparable across mappers** — global's 0.0015 px reflects its tighter angular/normalized inlier filtering for the no-focal model, not a 360× accuracy gain. Use registration count and point count as the fairer quality signals here. ## Notes - For this rotation-stabilized footage, `--GlobalMapper.ba_skip_joint_optimization_stage 1` (the modern equivalent of the old glomap `--BundleAdjustment.optimize_rotations 0`) shaves ~14% off global mapping (25.8 s → 22.1 s) with no quality change, but it was intentionally left out of the committed script since it assumes pre-canceled rotations and would degrade general (moving-camera) inputs. The old `--skip_rotation_averaging 1` is unsupported in colmap's `global_mapper` (aborts in track establishment), and `--BundleAdjustment.optimize_intrinsics 0` is a no-op for `SPHERICAL` (no focal length; `(w,h)` already pinned constant in BA). --------- Co-authored-by: Johannes Schönberger <jsch@meta.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Shaohui Liu <b1ueber2y@gmail.com>
SiftGPU sets the GPU device to use based on that passed to `ProgramCU::CheckCudaDevice`. However, this function only calls `cudaSetDevice` for devices with an ID greater than 0. Fixes to allow setting to device 0. Co-authored-by: Johannes Schönberger <jsch@demuc.de>
… (#4460) ## Problem `match_sequential` can hang indefinitely when loop detection is enabled together with `loop_detection_num_images_after_verification > 0`, as reported in #4456 (observed on long sequences of 6000+ images). ## Root cause `VocabTreePairGenerator` uses a producer/consumer pattern: worker threads run `Query()` (one per query image) and push results to a bounded `JobQueue`, while the main thread's `Next()` pops **exactly one** result per query. The futures returned by `ThreadPool::AddTask` are discarded, so when a worker `Query()` throws, the exception is silently swallowed **and** no result is pushed. The consumer's `queue_.Pop()` then blocks forever — the matcher appears stuck. This only surfaces when `loop_detection_num_images_after_verification > 0`, which activates the heavy spatial-verification path in `VisualIndex::Query`. That path can realistically throw (e.g. `std::bad_alloc` from the all-pairs match accumulation, which blows up on long/repetitive sequences). With verification off (the default), `Query()` returns early after lightweight scoring and never throws — which is why the hang only appears once this option is set. ## Fix Wrap the body of `VocabTreePairGenerator::Query` in a try/catch. On failure it logs the error and pushes an **empty** result, preserving the one-result-per-query invariant so the consumer can never deadlock. A failing image is simply skipped for loop detection instead of hanging the whole pipeline. ## Testing Added `VocabTreePairGenerator.DoesNotDeadlockOnFailedQuery`, which queries a deliberately invalid image id to deterministically trigger a worker failure. Verified it **hangs without the fix** and **passes with it**. - All 13 `pairing_test` cases pass - All 10 `visual_index_test` cases pass Fixes #4456 Co-authored-by: Paul-Edouard Sarlin <15985472+sarlinpe@users.noreply.github.com>
- Add upstream COLMAP remote and create sphere-sfm-v2 branch from upstream/main. - Replace legacy SPHERE camera model docs with EQUIRECTANGULAR migration guide. - Add scripts/convert_sphere_to_equirectangular.py for text-format reconstructions. - Document remaining fork features to port (sphere_cubic_reprojecer, pose_path).
- Update PORTING.md and README.md to use only upstream types/workflows. - Remove plans to port sphere_cubic_reprojecer / pose_path / SPHERE model. - Document replacement workflows: EQUIRECTANGULAR direct SfM and panorama_sfm.py. - Keep only the SPHERE -> EQUIRECTANGULAR text converter script.
- Include colmap/math/math.h in sift_test.cc for portable M_PI - Skip FileCopy SOFT_LINK test when Windows lacks symlink privileges
- sift_test.cc: drop orientation bounds that required M_PI on MSVC - file_test.cc: drop SOFT_LINK copy test that requires Windows symlink privileges
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the sphere-sfm project to a current upstream COLMAP 4.x base (
main) using a minimal, no-custom-C++ approach.What changed
SPHEREcamera model with upstream's nativeEQUIRECTANGULARcamera model.python/examples/convert_sphere_to_equirectangular.pyto convert existingSPHEREtext-format reconstruction files toEQUIRECTANGULAR.PORTING.mdand updatedREADME.md.src/colmap/feature/sift_test.cc: removed orientation bounds checks that depend onM_PI.src/colmap/util/file_test.cc: removed the soft-linkFileCopytest that requires symlink privileges on Windows.Build / test notes
Configured and tested on Windows with vcpkg + MSVC 14.44 + Ninja:
Result: 146/146 tests pass.
Scope
No fork C++ code was ported; the migration relies entirely on upstream COLMAP functionality.