Photogrammetry and Structure from Motion (SfM) - thomaspingel/advanced_remote_sensing GitHub Wiki
Structure from Motion (SfM) is the process of taking sequential images, usually captured from an aircraft or drone, and using a comparison of keypoints within images to digitally reconstruct the environment in 3D. Inputs to this process are generally georeferenced images, and outputs are generally point clouds, orthophotos and DEMs/DTMs/DSMs. The technique has its basis in stereo-photogrammetry in which paired photographs were used to measure the environment and create topographic maps. Sometimes multiple cameras are still used, but the motion of the camera essentially replicates the multiple views and so single cameras are often used to simplify the process and/or reduce carrying weight the drone. The same technique can be applied to thermal imaging and to photographs taken terrestrially.
Background Information
- As an overview, you may want to watch Dr. Pingel's lectures on photogrammetry and structure from motion.
- Here is a quick refresher on LiDAR, photogrammetry, and how they relate.
- Two good academic articles to get started are this one by Westoby et al. (2012) and this one by Elnter et al. (2016).
Software Overview
- We use Pix4Dmapper for most of our processing. Pix4D has other products, including Pix4DMatic, Pix4Dcloud, Pix4Dcapture for drone flight planning, and Pix4Dcatch for smartphone/tablet capture. Register for a free two-week account. We are looking to expand student access to this software.
- Binghamton has a license for Drone2Map, so students are encouraged to use it for SfM.
- WebODM is a good, open-source SfM product. See Dr. Pingel for software.
- RealityCapture is owned by Epic Games. Use these tutorials to get started.
- A full list of software is below.
Esri's Drone2Map
Esri's Drone2Map is a powerful software application designed to transform raw drone-captured imagery into high-quality geographic products, such as 2D orthomosaics, 3D meshes, digital elevation models (DEMs), and more. Built on Esri’s ArcGIS platform, it enables users to easily visualize and analyze drone data within the broader context of geographic information systems (GIS). With its intuitive interface, Drone2Map simplifies photogrammetry workflows, making it accessible to both beginners and professionals. The software supports real-time image processing, allowing users to assess the quality of their data while still in the field, ensuring that missions are completed efficiently. Its integration with ArcGIS Online and ArcGIS Pro makes it particularly suited for users seeking seamless data sharing and advanced spatial analysis capabilities.
WebODM
WebODM (Web OpenDroneMap) is an open-source software platform designed for processing aerial imagery captured by drones into maps, 3D models, and other geographic data. It allows users to upload drone photos, which are then processed using photogrammetry techniques to generate orthophotos, point clouds, digital surface models (DSM), and 3D meshes. The platform is accessible via a web interface, making it user-friendly and easy to integrate into various workflows, whether for agriculture, construction, environmental monitoring, or other industries. WebODM also supports various plugins and integrates well with other GIS software, offering a flexible, scalable solution for drone-based mapping projects.
Getting Started with Pix4D
- We generally use Pix4D in the lab for reconstructions. Other popular options are Agisoft and OpenDroneMap. We've recently begun experimenting with RealityCapture and nFrames' SURE, both of which will additionally fuse photogrammetry with lidar. A complete list of software is given below.
- Pix4D is available for download as a trial, but is also installed on all lab machines and in the Wallace 101 lab. Since Pix4D is a resource-intensive application, you'll generally want to have a good multi-core processor, at least 32 GB of RAM, and an Nvidia graphics card. The trial is fully featured and lasts for two weeks. Otherwise, you'll need a Pix4D account. CNRE hosts a license (to which NEIL contributes), and has a login and password, but this can't be shared with students directly for security reasons.
- Pix4D has a number of excellent sample datasets and tutorials for getting started available on YouTube. This playlist for learning Pix4D Desktop is a good place to start.
- As a first project, use Pix4D to make a reconstruction of a sample dataset like the quarry or a house outside of Blacksburg. Open and visualize the point cloud in CloudCompare, the mesh in one of the Windows 3D viewers, and the orthophoto in ArcGIS. Watch these video tutorials to help you along: [Part 1] [Part 2] [Part 3].
Sample Datasets
Dataset | # | Size | Vehicle | Description |
---|---|---|---|---|
Pix4D Quarry | 127 | 0.5 GB | Pix4D | A sample dataset provided by Pix4D for demonstrations and benchmarking. |
Binghamton East Gym Soccer Field | 48 | 815 MB | Autel Evo II Pro V3 | Single grid nadir + GCPs |
Binghamton East Gym Soccer Field | 142 | 855 MB | Mavic 3T | Single grid nadir + GCPs |
Bishop House 1 | 172 | 2.2 GB | Mavic 2 | A double-grid mission at property near Blacksburg, VA. |
Bishop House 2 | 430 | 1.0 GB | Iris+ / GoPro 4 | A double-grid mission flown at a property near Blacksburg, VA. |
Capers Island | 179 | 0.9 GB | Mavic 2 | A single-grid flight over a beach in South Carolina |
Binghamton East Gym Soccer Field - Alana | 161 | 2.5 GB | Autel Evo II V3 | Double grid + GCPs, 15m vertical error |
Franklin County Tornado | 123 | 1.2 GB | Mavic 2 | A small section of tornado damage from the Franklin County Tornado (April 2019) |
NIU | 293 | 0.5 GB | Iris+ / GoPro 4 | A small section of the NIU campus, flown with the Iris+ and GoPro Hero 4 |
Old Middle School Tree | 116 | 2.0 GB | Mavic 2 | A detailed scan of an oak tree at the old Blacksburg Middle School |
VT - Major Williams | 298 | 2.2 GB | Mavic 2 | NE section of Virginia Tech Campus near Major Williams Hall |
Owens Park Trail | 799 | 1.8 GB | POAS / GoPro 8 | Section of paved walking trail in Owens Park. |
Old Johnson | 668 | 3.81 GB | Mavic 3T | Double grid nadir and 8-point obliques of area around Old Johnson at Binghamton University |
GoPro 11 Dataset | 111 | 600 MB | GoPro 11 | Single grid nadir |
Multispectral Dataset | Parrot Sequoia | U of Idaho |
Bad Camera Models
- Many Pix4D camera models are bad, and we've compiled a list of corrected models and a calibration procedure here.
Pix4D Settings
- There really isn't a single group of settings that will work in all cases. Pix4D has templates for Mapping and Modeling that are often good starts. You can also create your own.
- We've found that for many application, the 3D Models template works best for Step 1, but with Geometrically Verified Matching enabled (2nd tab) and Rematching enabled (3rd tab). An image of these settings is here.
- We've found that the 3D Maps template works best for Step 2, but with exporting a LAZ file instead of LAS (files merged to one), Classify Point Cloud enabled, Image Scale set to Full, 3D Textured Mesh disabled (unless needed). An image of these settings is here.
- For Step 3, you may want to specify the resolution, rather than just accepting the default of 1 GSD, especially if your camera is terrestrial (POAS, rover, etc.).
- However, a deeper understanding of what these settings are, what they do, what they're for, how they affect output, and how they affect processing time will serve you well! Ask questions! Try combinations and note the differences! Your improved knowledge of these settings will help you process data faster and better than just using even these defaults.
- This page gives a good overview of different settings. This applies to Pix4Dmatic, which as far as we can tell is mostly just a faster implementation.
Collective Wisdom
- Good data collection is key to an easy reconstruction. Plan your mission carefully to collect a double-grid if possible, ideally at more than one height and at different camera angles. Conventional wisdom suggests that putting the camera at slightly off-nadir (10-20 degrees) is preferable. Sources: 1, 2. People generally recommend to collect data at around noon to minimize shadows and shadow movement, or under overcast conditions.
- A good resource when planning collection is the Drone Mapping Worksheet by Nathan Eick, often posted to the Drone Mapping Group on Facebook.
- Plan to collect several Ground Control Points (GCPs) over the study area with RTK GPS. Either place targets, or collect points in the natural environment. If the latter, manholes generally make readily identifiable features in the air, but any small, high-contrast, flat object can work. Be sure to take careful notes about the position of the GCP to avoid misidentification later. A photo with a camera phone makes for a good backup. If placing target or making marks, make sure this is done prior to image collection.
- Make sure camera settings are set for correct exposure, speed, etc. (3DR recommendations for GoPro at 1 and 2)
- Make sure the clock is set precisely before collection. If using a GoPro, take a photo of a time source (e.g., time.gov, or time.is/just and use period to get fractional seconds) with the camera for easy validation of timestamps later. Triple check the camera is running before launch. When pulling data, copy from the SD card directly rather than from the USB connection. The new GoPro Labs GPS Sync works great for Hero 9 and 10s.
- Collect with RTK GPS if possible. This should reduce or eliminate the need for GCPs.
- When loading images into Pix4D, it's generally best to have images in EGM 96 altitudes already. We often use UNAVCO to do the conversion from the WGS84 datum. It's unclear how well Pix4D handles the vertical datum conversions. Any attribute can be edited here, and CSV files can be read in to define or replace existing values. To edit all values, right-click and set all values (sometimes the first attempt doesn't "take"). Horizontal and vertical accuracies are often far too high (5m and 10m default). These values represent the standard deviation, so are better set to something in the 2-5 m range. Omega, Phi, and Kappa are not yaw/pitch/roll, but instead include the heading of the craft. Neilpy has a converter for YPR2OPK, and these values can be inferred from the GPS track if necessary (e.g, if using a GoPro). Be sure the camera model is set correctly, as this is a critical series of parameters. Wrong values here will cause the reconstruction to fail. We've seen items in the database with incorrect values.
- We've had best results with model-like settings for Step 1 and map-like settings for Steps 2 and 3 in Pix4D.
- For Step 1, Keypoints Image Scale can often be set to 1/2 or 1/4 if the scene is well differentiated, but for optimal results Full is recommended for final processing. Free Flight / Terrestrial generally makes for better results, but takes 5x times longer. We recommend using Geometrically Verified Matching and Calibration set to Rematch. Sometimes Internal Parameters can be set to All Prior or None if the camera model is known to be good and reconstruction is problematic.
- For Step 2, you'll generally want to set the Image Scale to Full unless doing exploratory analysis. Export as LAZ and check Merge Tiles into One File. Point Density should generally be set to Optimal. "High" produces very dense point clouds, but can take a very long time to run (e.g., a week!). We generally don't produce meshes, and leave the Matching Window Size to 7x7.'
- For Step 3, default settings are generally fine, but if using photogrammetry on a stick, you may want to set the resolution to a fixed value rather than use the defaults, as the default GSD value could be under 1 cm.
- It's kind to turn resources down a few clicks if you're using a shared computer, or if you want your computer to be responsive during the run.
Troubleshooting
- Doming or curvature in a reconstruction usually happens when the camera model is wrong, and can be addressed by fixing the camera model, adding GCPs or MTPs, and setting the vertical height accuracy to a precise (low) value. This is a complex issue and we have a whole wiki page about this.
- Ghosting happens when keypoints break into sets of matching images, with the result being several "versions" of the surface. We've found this is most easily addressed with the addition of Manual Tie Points (MTPs). A few MTPs spread out over your project as a "check" is often a good idea, even if you're using RTK GPS. When MTPs are useful, you'll notice that the placement of them is difficult - images are poorly registered, so the "suggested" locations of MTPs in yet-unmarked images is often significantly off. As the reconstruction improves, the suggested locations should be much closer or even exactly on target. For this reason, you should chose MTP locations in images where you'll be able to 100% correctly identify them in other images. [Video example]
Pix4D Resources
Pix4D has a wide variety of resources to support its use, including a YouTube Channel, excellent documentation, and an active knowledge base / forum. Some individual videos and articles we've found useful:
- Accurate Georeferencing in Pix4D
- Do more GCPs equal more accurate drone maps?.
- Getting started with your first Pix4DMapper project.
- How to Export Pix4D projects for 3D Printing
- Improved accuracy for rolling shutter cameras.
- Measuring distances, areas, and volumes in 3D
- Planning a flight
- Using the Mosaic Editor for tweaking orthophoto output.
- Webinar on Indoor Mapping. Selecting-the-Image-Acquisition-Plan-Type)
- https://community.pix4d.com/t/phantom-4-pro-rolling-shutter-effect/3555/15
- https://support.pix4d.com/hc/en-us/articles/360015702911-FAQ-on-the-accurate-geolocation-pipeline
- https://support.pix4d.com/hc/en-us/articles/205327965-Menu-Process-Processing-Options-1-Initial-Processing-Calibration#label2
- Designing the Image Acquisition Plan
SfM Software
- Historical
- Photosynth (2006)
- Commercial
- Agisoft Metashape / PhotoScan (2010)
- Canvas.io
- Cupix
- DroneDeploy (2013)
- Embark for turning scans into game-ready meshes
- ESRI's Drone2Map. Tutorials 1, 2, and 3.
- nFrames SURE Code: CD6X-TAMV-DUJA-KRV4-I6IQ-KI98-KEXT
- NVIDIA's NeRFs
- Pix4D / Drone2Map (2011)
- polycam for iPhones
- Qimera - SfM tools for bathymetry
- RealityCapture (2016)
- Sitescape
- Uzuverse and article that describes it.
- Open Source / Computational Toolkits
- AliceVision/Meshroom
- Matlab
- OpenCV SfM
- OpenDroneMap (ODM): VT has WebODM running on a campus server. To use, request access here and then access the service itself at uavmapping.gis.vt.edu.
- Python VisualSFM
- TeleSculptor and Examples
Non-Academic Articles
- https://hackaday.com/2022/10/02/in-a-way-3d-scanning-is-over-a-century-old/
- https://towardsdatascience.com/the-ultimate-guide-to-3d-reconstruction-with-photogrammetry-56155516ddc4
- Video: The Chips that See: Rise of the Image Sensor
References
- Alexiou et al. 2021. Comparing high accuracy t-lidar and UAV-SfM derived point clouds for geomorphological change detection. [archive]
- Alfio et al. 2021. Influence of Image TIFF Format and JPEG Compression Level in the Accuracy of the 3D Model and Quality of the Orthophoto in UAV Photogrammetry.
- Barba et al. 2019. Accuracy Assessment of 3D Photogrammetric Models from an Unmanned Aerial Vehicle.
- Bianco et al. 2018. Evaluating the performance of Structure from Motion pipelines. [archive]
- Burdziakowski and Bobkowska. 2021. UAV photogrammetry under poor lighting conditions - accuracy considerations.
- Brunier et al. 2016. Close range airborne SfM for high-resolution morphometric surveys - examples from an embayed rotating beach.
- Cabo et al. 2021. Influence of the Number and Spatial Distribution of Ground Control Points in the Accuracy of UAV-SfM DEMs: An Approach Based on Generalized Additive Models.
- Perez-Alvarez et al. 2019. A protocol for mapping archaeological sites through aerial 4k videos. [Academia.edu]
- Chirico et al. 2020. Evaluating Elevation Change Thresholds between Structure-from-Motion DEMs Derived from Historical Aerial Photos and 3DEP LiDAR Data.
- Eltner et al. 2016. Image-based surface reconstruction in geomorphometry – merits, limits and developments. [archive]
- Fernandez. 2018. Monitoring snow depth change across a range of landscapes with ephemeral snowpacks using SfM.
- Flener et al. 2013. Seamless mapping of river channels at high resolution using mobile lidar and UAV-photography.
- Fonstad et al. 2013. Topographic structure from motion - a new development in photogrammetric measurement.
- Holata et al. 2018. Integration of Low-Resolution ALS and Ground-Based SfM Photogrammetry Data. A Cost-Effective Approach Providing an ‘Enhanced 3D Model’ of the Hound Tor Archaeological Landscapes (Dartmoor, South-West England).
- Iheaturu et al. 2020. An Assessment of the accuracy of Structure-from-Motion (SfM) Photogrammetry for 3D terrain mapping. [[academia.edu](An Assessment of the accuracy of Structure-from-Motion (SfM) Photogrammetry for 3D terrain mapping)]
- Isibue and Pingel. 2020. Unmanned aerial vehicle based measurement of urban forests.
- James and Robson. 2014. Mitigating systematic error in topographic models derived from UAV and ground‐based image networks. [archive]
- James et al. 2020. Mitigating systematic error in topographic models for geomorphic change detection: accuracy, precision and considerations beyond off‐nadir imagery.
- Lovitt, Rahmann, and McDermid. 2017. Assessing the value of UAV photogrammetry for characterizing terrain in complex peatlands.
- Lucieer, de Jong, and Turner. 2013. Mapping landslide displacements using SfM.
- Luetzenburg et a. 2021. Evaluation of the Apple iPhone 12 Pro LiDAR for an Application in Geosciences.
- Mancini et al. 2013. Using UAV for high-resolution reconstruction of topography.
- Manfreda et al. 2019. Assessing the Accuracy of Digital Surface Models Derived from Optical Imagery Acquired with Unmanned Aerial Systems.
- Meyer and Skiles. 2019. Assessing the ability of structure from motion to map high resolution snow surface elevations in complex terrain.
- Meyer et al. 2021. Mapping snow depth and volume at the alpine watershed scale from aerial imagery using Structure from Motion.
- Mohamad et al. 2019. Evaluating Water Level Changes at Different Tidal Phases Using UAV Photogrammetry and GNSS Vertical Data.
- Nel. 2015. Post-processing of UAV-captured images for enhanced mapping by image stitching. [archive]
- Obanawa and Sakanoue. 2021. Conditions of Aerial Photography to Reduce Doming Effect.
- Pingel et al. 2021. Deriving Land and Water Surface Elevations in the Northeastern Yucatán Peninsula Using PPK GPS and UAV-Based Structure from Motion.
- Prior et al. 2022. Estimation of mean dominant height using NAIP digital aerial photogrammetry and lidar over mixed deciduous forest in the southeastern USA.
- Tan et al. 2017. Automatic Camera Calibration Using Active Displays of a Virtual Pattern.
- Vautherin et al. 2016. Photogrammetric accuracy and modeling of rolling shutter cameras.
- Watson. 2012. [An Automated Technique for Generating Georectified Mosaicsfrom Ultra-High Resolution Unmanned Aerial Vehicle (UAV) Imagery, Based on Structure from Motion (SfM) Point Clouds])(http://doi.org/10.3390/rs4051392).
- Westoby et al. 2012. Structure-from-Motion photogrammetry: A low-cost, effective tool for geoscience applications. [archive]
- Wang and Watanabe. 2019. Impact of Recreational Activities on an Unmanaged Alpine Campsite: The Case of Kuro-Dake Campsite, Daisetsuzan National Park, Japan.
- Xie et al. 2022. Optimal Self-Calibration Strategies in the Combined Bundle Adjustment of Aerial–Terrestrial Integrated Images.
- Zaksek et al. 2015. Cloud Photogrammetry from Space.
- Zhang et al. 2000. A flexible new technique for camera calibration.