Photogrammetry and Structure from Motion (SfM) - thomaspingel/advanced_remote_sensing GitHub Wiki

Structure from Motion (SfM) is the process of taking sequential images, usually captured from an aircraft or drone, and using a comparison of keypoints within images to digitally reconstruct the environment in 3D. Inputs to this process are generally georeferenced images, and outputs are generally point clouds, orthophotos and DEMs/DTMs/DSMs. The technique has its basis in stereo-photogrammetry in which paired photographs were used to measure the environment and create topographic maps. Sometimes multiple cameras are still used, but the motion of the camera essentially replicates the multiple views and so single cameras are often used to simplify the process and/or reduce carrying weight the drone. The same technique can be applied to thermal imaging and to photographs taken terrestrially.

Background Information

  • As an overview, you may want to watch Dr. Pingel's lectures on photogrammetry and structure from motion.
  • Here is a quick refresher on LiDAR, photogrammetry, and how they relate.
  • Two good academic articles to get started are this one by Westoby et al. (2012) and this one by Elnter et al. (2016).

Software Overview

  • We use Pix4Dmapper for most of our processing. Pix4D has other products, including Pix4DMatic, Pix4Dcloud, Pix4Dcapture for drone flight planning, and Pix4Dcatch for smartphone/tablet capture. Register for a free two-week account. We are looking to expand student access to this software.
  • Binghamton has a license for Drone2Map, so students are encouraged to use it for SfM.
  • WebODM is a good, open-source SfM product. See Dr. Pingel for software.
  • RealityCapture is owned by Epic Games. Use these tutorials to get started.
  • A full list of software is below.

Esri's Drone2Map

Esri's Drone2Map is a powerful software application designed to transform raw drone-captured imagery into high-quality geographic products, such as 2D orthomosaics, 3D meshes, digital elevation models (DEMs), and more. Built on Esri’s ArcGIS platform, it enables users to easily visualize and analyze drone data within the broader context of geographic information systems (GIS). With its intuitive interface, Drone2Map simplifies photogrammetry workflows, making it accessible to both beginners and professionals. The software supports real-time image processing, allowing users to assess the quality of their data while still in the field, ensuring that missions are completed efficiently. Its integration with ArcGIS Online and ArcGIS Pro makes it particularly suited for users seeking seamless data sharing and advanced spatial analysis capabilities.

WebODM

WebODM (Web OpenDroneMap) is an open-source software platform designed for processing aerial imagery captured by drones into maps, 3D models, and other geographic data. It allows users to upload drone photos, which are then processed using photogrammetry techniques to generate orthophotos, point clouds, digital surface models (DSM), and 3D meshes. The platform is accessible via a web interface, making it user-friendly and easy to integrate into various workflows, whether for agriculture, construction, environmental monitoring, or other industries. WebODM also supports various plugins and integrates well with other GIS software, offering a flexible, scalable solution for drone-based mapping projects.

Getting Started with Pix4D

  • We generally use Pix4D in the lab for reconstructions. Other popular options are Agisoft and OpenDroneMap. We've recently begun experimenting with RealityCapture and nFrames' SURE, both of which will additionally fuse photogrammetry with lidar. A complete list of software is given below.
  • Pix4D is available for download as a trial, but is also installed on all lab machines and in the Wallace 101 lab. Since Pix4D is a resource-intensive application, you'll generally want to have a good multi-core processor, at least 32 GB of RAM, and an Nvidia graphics card. The trial is fully featured and lasts for two weeks. Otherwise, you'll need a Pix4D account. CNRE hosts a license (to which NEIL contributes), and has a login and password, but this can't be shared with students directly for security reasons.
  • Pix4D has a number of excellent sample datasets and tutorials for getting started available on YouTube. This playlist for learning Pix4D Desktop is a good place to start.
  • As a first project, use Pix4D to make a reconstruction of a sample dataset like the quarry or a house outside of Blacksburg. Open and visualize the point cloud in CloudCompare, the mesh in one of the Windows 3D viewers, and the orthophoto in ArcGIS. Watch these video tutorials to help you along: [Part 1] [Part 2] [Part 3].

Sample Datasets

Dataset # Size Vehicle Description
Pix4D Quarry 127 0.5 GB Pix4D A sample dataset provided by Pix4D for demonstrations and benchmarking.
Binghamton East Gym Soccer Field 48 815 MB Autel Evo II Pro V3 Single grid nadir + GCPs
Binghamton East Gym Soccer Field 142 855 MB Mavic 3T Single grid nadir + GCPs
Bishop House 1 172 2.2 GB Mavic 2 A double-grid mission at property near Blacksburg, VA.
Bishop House 2 430 1.0 GB Iris+ / GoPro 4 A double-grid mission flown at a property near Blacksburg, VA.
Capers Island 179 0.9 GB Mavic 2 A single-grid flight over a beach in South Carolina
Binghamton East Gym Soccer Field - Alana 161 2.5 GB Autel Evo II V3 Double grid + GCPs, 15m vertical error
Franklin County Tornado 123 1.2 GB Mavic 2 A small section of tornado damage from the Franklin County Tornado (April 2019)
NIU 293 0.5 GB Iris+ / GoPro 4 A small section of the NIU campus, flown with the Iris+ and GoPro Hero 4
Old Middle School Tree 116 2.0 GB Mavic 2 A detailed scan of an oak tree at the old Blacksburg Middle School
VT - Major Williams 298 2.2 GB Mavic 2 NE section of Virginia Tech Campus near Major Williams Hall
Owens Park Trail 799 1.8 GB POAS / GoPro 8 Section of paved walking trail in Owens Park.
Old Johnson 668 3.81 GB Mavic 3T Double grid nadir and 8-point obliques of area around Old Johnson at Binghamton University
GoPro 11 Dataset 111 600 MB GoPro 11 Single grid nadir
Multispectral Dataset Parrot Sequoia U of Idaho

Bad Camera Models

  • Many Pix4D camera models are bad, and we've compiled a list of corrected models and a calibration procedure here.

Pix4D Settings

  • There really isn't a single group of settings that will work in all cases. Pix4D has templates for Mapping and Modeling that are often good starts. You can also create your own.
  • We've found that for many application, the 3D Models template works best for Step 1, but with Geometrically Verified Matching enabled (2nd tab) and Rematching enabled (3rd tab). An image of these settings is here.
  • We've found that the 3D Maps template works best for Step 2, but with exporting a LAZ file instead of LAS (files merged to one), Classify Point Cloud enabled, Image Scale set to Full, 3D Textured Mesh disabled (unless needed). An image of these settings is here.
  • For Step 3, you may want to specify the resolution, rather than just accepting the default of 1 GSD, especially if your camera is terrestrial (POAS, rover, etc.).
  • However, a deeper understanding of what these settings are, what they do, what they're for, how they affect output, and how they affect processing time will serve you well! Ask questions! Try combinations and note the differences! Your improved knowledge of these settings will help you process data faster and better than just using even these defaults.
  • This page gives a good overview of different settings. This applies to Pix4Dmatic, which as far as we can tell is mostly just a faster implementation.

Collective Wisdom

  • Good data collection is key to an easy reconstruction. Plan your mission carefully to collect a double-grid if possible, ideally at more than one height and at different camera angles. Conventional wisdom suggests that putting the camera at slightly off-nadir (10-20 degrees) is preferable. Sources: 1, 2. People generally recommend to collect data at around noon to minimize shadows and shadow movement, or under overcast conditions.
  • A good resource when planning collection is the Drone Mapping Worksheet by Nathan Eick, often posted to the Drone Mapping Group on Facebook.
  • Plan to collect several Ground Control Points (GCPs) over the study area with RTK GPS. Either place targets, or collect points in the natural environment. If the latter, manholes generally make readily identifiable features in the air, but any small, high-contrast, flat object can work. Be sure to take careful notes about the position of the GCP to avoid misidentification later. A photo with a camera phone makes for a good backup. If placing target or making marks, make sure this is done prior to image collection.
  • Make sure camera settings are set for correct exposure, speed, etc. (3DR recommendations for GoPro at 1 and 2)
  • Make sure the clock is set precisely before collection. If using a GoPro, take a photo of a time source (e.g., time.gov, or time.is/just and use period to get fractional seconds) with the camera for easy validation of timestamps later. Triple check the camera is running before launch. When pulling data, copy from the SD card directly rather than from the USB connection. The new GoPro Labs GPS Sync works great for Hero 9 and 10s.
  • Collect with RTK GPS if possible. This should reduce or eliminate the need for GCPs.
  • When loading images into Pix4D, it's generally best to have images in EGM 96 altitudes already. We often use UNAVCO to do the conversion from the WGS84 datum. It's unclear how well Pix4D handles the vertical datum conversions. Any attribute can be edited here, and CSV files can be read in to define or replace existing values. To edit all values, right-click and set all values (sometimes the first attempt doesn't "take"). Horizontal and vertical accuracies are often far too high (5m and 10m default). These values represent the standard deviation, so are better set to something in the 2-5 m range. Omega, Phi, and Kappa are not yaw/pitch/roll, but instead include the heading of the craft. Neilpy has a converter for YPR2OPK, and these values can be inferred from the GPS track if necessary (e.g, if using a GoPro). Be sure the camera model is set correctly, as this is a critical series of parameters. Wrong values here will cause the reconstruction to fail. We've seen items in the database with incorrect values.
  • We've had best results with model-like settings for Step 1 and map-like settings for Steps 2 and 3 in Pix4D.
    • For Step 1, Keypoints Image Scale can often be set to 1/2 or 1/4 if the scene is well differentiated, but for optimal results Full is recommended for final processing. Free Flight / Terrestrial generally makes for better results, but takes 5x times longer. We recommend using Geometrically Verified Matching and Calibration set to Rematch. Sometimes Internal Parameters can be set to All Prior or None if the camera model is known to be good and reconstruction is problematic.
    • For Step 2, you'll generally want to set the Image Scale to Full unless doing exploratory analysis. Export as LAZ and check Merge Tiles into One File. Point Density should generally be set to Optimal. "High" produces very dense point clouds, but can take a very long time to run (e.g., a week!). We generally don't produce meshes, and leave the Matching Window Size to 7x7.'
    • For Step 3, default settings are generally fine, but if using photogrammetry on a stick, you may want to set the resolution to a fixed value rather than use the defaults, as the default GSD value could be under 1 cm.
    • It's kind to turn resources down a few clicks if you're using a shared computer, or if you want your computer to be responsive during the run.

Troubleshooting

  • Doming or curvature in a reconstruction usually happens when the camera model is wrong, and can be addressed by fixing the camera model, adding GCPs or MTPs, and setting the vertical height accuracy to a precise (low) value. This is a complex issue and we have a whole wiki page about this.
  • Ghosting happens when keypoints break into sets of matching images, with the result being several "versions" of the surface. We've found this is most easily addressed with the addition of Manual Tie Points (MTPs). A few MTPs spread out over your project as a "check" is often a good idea, even if you're using RTK GPS. When MTPs are useful, you'll notice that the placement of them is difficult - images are poorly registered, so the "suggested" locations of MTPs in yet-unmarked images is often significantly off. As the reconstruction improves, the suggested locations should be much closer or even exactly on target. For this reason, you should chose MTP locations in images where you'll be able to 100% correctly identify them in other images. [Video example]

Pix4D Resources

Pix4D has a wide variety of resources to support its use, including a YouTube Channel, excellent documentation, and an active knowledge base / forum. Some individual videos and articles we've found useful:

SfM Software

Non-Academic Articles

References