3D Scene Reconstruction and Virtual Tour - CS436 Project

This repository contains our implementation of a modular Structure-from-Motion (SfM) pipeline as per the course CS436 - Computer Vision. The goal is to reconstruct a 3D scene and camera trajectory from a sequence of 2D images, resulting in an interactive web-based virtual tour.

Take a look at our final web tour:

Repository Structure

root/
│
├── src/                # modular Python files
├── notebooks/          # weekly result notebooks (Week 1–3 for now)
├── data/               # source images
├── results/            # visual outputs and point clouds
├── docs/               # report
├── web_viewer          # interactive web viewer (final result)
├── run_pipeline.py     # main .py file to run the entire PnP pipeline
├── AI_Usage.md         # report containing the use of AI to assist with the project code
├── project_report.pdf  # detailed report of the project 
│
└── README.md

Environment Setup

python -m venv venv
source venv/bin/activate        # or venv\Scripts\activate on Windows
pip install -r requirements.txt

Libraries Used

• OpenCV
• NumPy
• SciPy
• Matplotlib
• Open3D

Dataset and Preprocessing

As per project guidelines:

• We captured static, textured scenes (brick walls) with ~60% overlap between each frame.
• We ensured consistent lighting and physical translation (avoiding rotation).
• Intrinsic parameters were calculated from EXIF metadata.

All input data are stored in data/ and all results in results/.

Week 1 - Feature Extraction and Matching

Goal: Implement a feature-matching foundation for the SfM pipeline.

Key Steps

1. Preprocessing

   • Downsample 4K images to 1080×1920 to reduce computation time and memory usage.

2. Feature Detection (SIFT)

   • Use SIFT to compute scale- and rotation-invariant keypoints and 128-dimensional descriptors for each image.

3. Feature Matching

   • Use a Brute-Force matcher with k=2 nearest neighbors.
   • Apply Lowe’s Ratio Test (threshold = 0.75) to reject ambiguous or noisy matches.

4. Visualization

   • Sort matches by descriptor distance.
   • Display and save the top 100 filtered matches per consecutive image pair.

Output

• Filtered SIFT matches between 5 consecutive image pairs.
• Top-100 match visualizations saved in results/week1/.

Week 2 - Two-View Reconstruction

Goal: Recover 3D structure and relative camera pose from a pair of images.

Steps

1. Essential Matrix Estimation

   • Use cv2.findEssentialMat() with RANSAC on normalized point correspondences.

2. Pose Recovery

   • Decompose the Essential Matrix into rotation and translation using cv2.recoverPose().

3. Cheirality Check

   • Select the physically valid pose where most triangulated points lie in front of both cameras (positive depth in both views).

4. Triangulation

   • Use cv2.triangulatePoints() to recover 3D points from the inlier correspondences.
   • Convert homogeneous coordinates to Euclidean coordinates.

5. Export

   • Save the resulting sparse 3D point cloud in .ply format for visualization in Open3D or other viewers.

Output

• Sparse two-view reconstruction stored at results/week2/point_cloud.ply.
• Valid [R | t] relative pose between the two cameras.

Week 3 - Incremental Multi-View SfM and Bundle Adjustment

Goal: Extend the two-view reconstruction to a full image sequence using PnP and Bundle Adjustment.

1. Feature Extraction

• Extract up to 40,000 SIFT features per frame for 30 frames.
• Compute camera intrinsics matrix K from EXIF metadata.

2. Map Initialization

• Select two baseline frames (e.g., Frame 0 and Frame 2) with sufficient parallax.

• Match SIFT descriptors between the two frames.

• Estimate the Essential Matrix and recover relative pose [R | t].

• Triangulate initial 3D points and perform a cheirality check.

• Initialize the reconstruction map with:

  • First camera at the origin (R = I, t = 0).
  • Second camera at the recovered pose.
  • Approximately 2,527 initial 3D points.

3. Incremental Reconstruction with PnP

For each remaining frame in the sequence:

• Match its descriptors to descriptors in the most recently added camera.

• For matches where the reference keypoint is already associated with a 3D point:

  • Build 2D–3D correspondences (image points ↔ existing 3D points).

• Use cv2.solvePnPRansac() to estimate the new camera pose:

  • Robustly fit the pose under outliers using RANSAC.

• Add the new camera pose (R, t) to the reconstruction map.

• Register PnP inlier matches as additional 2D–3D associations for that frame.

• Triangulate new 3D points between:

  • The newly registered camera, and
  • A previous camera (often the latest previous in the map).

• Only keep triangulated points that:

  • Are in front of both cameras (positive depth), and
  • Have valid image coordinates for color sampling.

This progressively grows both the 3D point cloud and the set of registered camera poses.

4. Bundle Adjustment

• After every 5 newly registered frames, run Bundle Adjustment using scipy.optimize.least_squares().

• Optimize over:

  • Camera parameters (rotation in Rodrigues form + translation) for all but the first camera (which is fixed as reference).
  • All 3D point positions.

• Use a sparse Jacobian structure (lil_matrix) to speed up optimization:

  • Each residual depends only on one camera and one 3D point.

• The cost function minimizes total reprojection error between observed 2D points and the projections of the optimized 3D points.

After all frames are processed, a final global bundle adjustment is run on the entire map.

5. Output and Visualization and Preparing for Three.js Tour

• Save the final dense point cloud to final_model.ply.

• Filter outliers by removing points lying more than two standard deviations from the mean in any dimension.

• Visualize the cleaned point cloud using Matplotlib’s 3D scatter plot:

• Export camera poses and point cloud for Three.js:

  • Convert from OpenCV’s coordinate conventions to Three.js conventions (Y-up, Z-backward).

  • Save:

    • A .ply file with aligned points.

    • A project_data.json containing:

      • Per-camera 4×4 transformation matrices (column-major, as used in Three.js).
      • Image filenames.
      • Reference to the point cloud file.

Final Statistics (approximate)

• 30 registered camera poses.
• Around 200,000 3D points after incremental reconstruction and refinement.
• Multiple Bundle Adjustment stages with decreasing reprojection cost.

Main Output Files

• results/week3/Milestone3_model.ply -> final filtered point cloud.

• results/week3/project_data.json -> camera trajectories and point-cloud metadata for Three.js.

• results/week3/Milestone3_demo_CloudCompare.mp4 -> a showcase of the model generated using CloudCompare

Week 4 - Interactive Virtual Tour with Three.js

Goal: Build an interactive web-based virtual tour that allows users to explore the reconstructed 3D environment and view original photographs from camera positions.

1. Data Pipeline and Coordinate Transformation

• Load the exported project_data.json containing camera poses and point cloud reference.
• Transform coordinates from OpenCV conventions (Y-down, Z-forward) to WebGL/Three.js conventions (Y-up, Z-backward):
  • Apply transformation matrix $M = \text{diag}(1, -1, -1, 1)$ to camera poses.
  • Negate Y and Z components of point cloud coordinates.
• Parse 4×4 camera transformation matrices and decompose into position and quaternion rotation.

2. Scene Setup

• Initialize Three.js scene with perspective camera (60° FOV) and WebGL renderer with antialiasing.
• Load point cloud using PLYLoader with vertex colors preserved.
• Compute bounding sphere to determine model center for overview positioning.
• Set up OrbitControls for mouse-based camera rotation with damping for smooth movement.

3. Waypoint Visualization

• Create directional cone meshes at each camera position to indicate photo capture locations.
• Orient cones using the camera's quaternion rotation to show viewing direction.
• Apply rotation corrections for specific frame ranges where camera orientation changed during recording.
• Store camera metadata (position, rotation, image filename) in mesh userData for retrieval on click.

4. Navigation Modes

Overview Mode (Toggle with 'H' key)

• Bird's-eye view positioned above the model center.
• Smooth camera animation using TWEEN.js with cubic easing.
• Zoom enabled for exploring the full reconstruction.

Teleportation (Double-click)

• Raycast against point cloud or invisible floor plane to find target position.
• Animate camera to clicked location while maintaining eye height.
• Preserve viewing direction by animating orbit target alongside camera position.

Photo Mode (Single-click on waypoint)

• Animate camera to exact waypoint position and orientation.
• Position interpolation using linear interpolation (Lerp).
• Rotation interpolation using quaternion tweening (approximating Slerp).
• On arrival, fade in the original photograph as a full-screen overlay.
• Dragging the view partially fades the overlay to reveal the 3D scene underneath.

5. Interaction Features

• Hover Effects: Waypoint cones highlight (color change + scale up) when mouse hovers over them.
• Raycasting: Mouse position converted to normalized device coordinates for accurate 3D intersection detection.
• Responsive Design: Window resize handler maintains correct aspect ratio.

6. Animation Loop

• requestAnimationFrame loop for smooth 60fps rendering.
• Update OrbitControls each frame for damping effect.
• Update TWEEN animations for smooth camera transitions.

Output

• Interactive virtual tour accessible at web_viewer/index.html
• Final point cloud and video of the tour accesssible at results/Final_model.ply and results/Final_tour.mp4
• Supports exploration via teleportation, waypoint navigation, and photo overlay viewing

Results

All visualizations and 3D outputs are stored under:

• results/

Team

• Muhammad Hussain Habib (27100016)
• Ayaan Ahmed (27100155)