Deep Learning is an approach often used in remote sensing for classification. Deep Learning works by using training data to automatically detect patterns in the data using an Artificial Neural Network (ANN). Compared to other machine learning approaches, deep learning requires a very large amount of input data and computing power, generally via a Graphics Processing Unit (GPU). Some deep learning frameworks include TensorFlow, PyTorch, Keras, and many more. Deep Learning approaches to classification work well provided good training data, but unlike regression- or algorithmic-based approaches, they are "black boxes", in which precisely how the problem is solved remains hidden.

ArcGIS includes Deep Learning support via the ArcGIS Image Analyst extension and arcgis.learn. A recent ArcGIS article by Rohit Singh gives a general overview. New tools let you perform a Deep Learning classification on point clouds as well. ArcGIS includes alternatives to Deep Learning, including a more general machine learning approach using the Classification Wizard and Classification Toolkit, including this example on calculating Surface Imperviousness.

Many technologies are powered by Deep Learning or Deep-Learning like approaches, including ChatCPT, DALL-E, Point-E, Shap-E, ViperGPT, and other tech to turn text into meshes. More: 1 2 3

A list of pre-trained models is here: https://livingatlas.arcgis.com/en/browse/?q=pretrained%20models#q=pretrained+models&d=2

The Basic Process

1. Make sure the Deep Learning Framework is installed.
2. Create a training dataset, either vector or raster, that corresponds to the object you want to detect in imagery. Use the Imagery -> Classification Tools -> Label Objects for Deep Learning tool to do this, which has some digitization tools to speed this process along.
3. Use the Export Training Data for Deep Learning tool to create "image chips" from from the training data. Image chips are used to train the deep learning model. Use PASCAL Visual Object Classes for object detection and Classified Tiles for pixel classification.
4. Use the Train Deep Learning Model to train the model. This step is time consuming, and generally requires a good GPU.
5. Use either the Detect Objects Using Deep Learning tool or the Classify Pixels Using Deep Learning tool to apply the model, depending on your goal.

Getting Started

There are many excellent tutorials available:

• Read this article talking about how great Deep Learning in ArcGIS is.
• Use deep learning to assess palm tree health is a good starting point. It provides current instructions for getting the Deep Learning Framework installed and running into ArcGIS Pro, and runs you through a task which extracts trees based on training data.
• Deep Learning Using ArcGIS
• Classify a point cloud using machine learning
• Classify Power Lines Using Deep Learning

See also

• Sam Carani's Thesis on using Deep Learning to classify tornado damage
• The OpenAerialMap repository for drone imagery may be good for various projects.
• West Virginia's course on Geospatial Deep Learning
• https://www.esri.com/arcgis-blog/products/arcgis-pro/geoai/enhanced-object-detection-using-drones-and-ai/
• https://samgeo.gishub.org/examples/automatic_mask_generator/
• Text SAM in ArcGIS Pro: https://www.youtube.com/watch?v=hfMin59puXA, and DLPK Download, and Esri blog post
• Pretrained deep learning models: https://www.esri.com/en-us/arcgis/deep-learning-models

Example Workflow

import arcpy
arcpy.CheckOutExtension("ImageAnalyst")
from arcpy.ia import *
import os


# Set local variables

in_raster = "data/eBee_transparent_mosaic_group1.tif"
in_training = "data/damage.shp"

chip_folder = "chips"
model_folder = "model"
out_folder = "out"
classified_raster_fn = out_folder + '\\classified.tif'

# We are training on only the northern portion of the dataset
training_extent = "data/bounds_north.shp"


# Set Environments and folders

cwd = os.getcwd() + '\\'
print(cwd)

if not os.path.exists(chip_folder):
    os.makedirs(chip_folder)
if not os.path.exists(model_folder):
    os.makedirs(model_folder)
if not os.path.exists(out_folder):
    os.makedirs(out_folder)

arcpy.env.overwriteOutput = True
try:
    scratch_db = arcpy.CreateFileGDB_management('.', 'scratch')
except:
    print('Warning, could not create scratch directory.  Maybe it already exists.')
arcpy.env.workspace = cwd + 'scratch.gdb'


# [Create Training Data](https://pro.arcgis.com/en/pro-app/latest/help/analysis/image-analyst/label-objects-for-deep-learning.htm)

image_chip_format = "TIFF"
tile_size = "512"
stride= "256"
output_nofeature_tiles= "ONLY_TILES_WITH_FEATURES"
metadata_format= "Classified_Tiles"
start_index = 0
classvalue_field = "Classvalue"
buffer_radius = 0
in_mask_polygons = ""
rotation_angle = 0
reference_system = "MAP_SPACE"
processing_mode = "PROCESS_AS_MOSAICKED_IMAGE"
blacken_around_feature = "NO_BLACKEN"
crop_mode = "FIXED_SIZE"

arcpy.env.extent = training_extent

ExportTrainingDataForDeepLearning(in_raster, chip_folder, in_training, image_chip_format, 
                                  tile_size, tile_size, stride, stride,
                                  output_nofeature_tiles, metadata_format, start_index, classvalue_field, 
                                  buffer_radius, in_mask_polygons, rotation_angle, reference_system, processing_mode, 
                                  blacken_around_feature, crop_mode)


# [Train Deep Learning Model](https://pro.arcgis.com/en/pro-app/latest/tool-reference/image-analyst/train-deep-learning-model.htm)

max_epochs = 25
model_type = "UNET"
batch_size = 12
arg = ""
learning_rate = ""
backbone_model = "RESNET34" 
pretrained_model = ""
validation_percent = 10
stop_training = "STOP_TRAINING"
freeze = "FREEZE_MODEL"


# Execute
TrainDeepLearningModel(cwd + chip_folder, cwd + model_folder, max_epochs, model_type)

"""
TrainDeepLearningModel(chip_folder, model_folder, max_epochs, model_type,
                       batch_size, arg, learning_rate, backbone_model, pretrained_model,
                       validation_percent, stop_training, freeze)
                       """


# [Apply Deep Learning Model](https://pro.arcgis.com/en/pro-app/latest/tool-reference/image-analyst/classify-pixels-using-deep-learning.htm)

in_model_definition = model_folder + '/' + model_folder + '.emd'

model_arguments = "padding 56; batch_size 4; predict_background True; test_time_augmentation False; tile_size 224"
processing_mode = "PROCESS_AS_MOSAICKED_IMAGE"

classified_raster = ClassifyPixelsUsingDeepLearning(in_raster, in_model_definition)

classified_raster.save(classified_raster_fn)

Example Workflow 2

import arcpy

from arcpy.ia import *  

# To allow overwriting outputs change overwriteOutput option to True.
arcpy.env.overwriteOutput = True

# create training data
input_data = "D:/ml_final_project/data/ca-camp_fire-20181226/camp-752-8bit.tif"
training = "D:/ml_final_project/data/v3/training"
labeled_input = "D:/ml_final_project/data/ca-camp_fire-20181226/burned_area/camp_bp_70-100_polylabled_2.shp"
#%%
# Process: Export Training Data For Deep Learning (Export Training Data For Deep Learning) 
ExportTrainingDataForDeepLearning(in_raster=input_data,
                                                out_folder=training,
                                                in_class_data=labeled_input,
                                                image_chip_format="TIFF",
                                                tile_size_x=256, tile_size_y=256,
                                                stride_x=128, stride_y=128,
                                                output_nofeature_tiles="ALL_TILES",
                                                metadata_format="Classified_Tiles",
                                                start_index=0,
                                                class_value_field="Classvalue",
                                                buffer_radius=0,
                                                in_mask_polygons="",
                                                rotation_angle=0,
                                                reference_system="MAP_SPACE",
                                                processing_mode="PROCESS_AS_MOSAICKED_IMAGE",
                                                blacken_around_feature="NO_BLACKEN",
                                                crop_mode="FIXED_SIZE")

# build model
model_output = "D:/ml_final_project/data/v3/model"

# Process: Train Deep Learning Model (Train Deep Learning Model) 

DL = TrainDeepLearningModel(in_folder=training,
                                 out_folder=model_output, 
                                 max_epochs=20,
                                 model_type="UNET", 
                                 batch_size=10, 
                                 learning_rate=1e-05,
                                 arguments=["", ""](/thomaspingel/advanced_remote_sensing/wiki/"",-""), 
                                 backbone_model="RESNET34", 
                                 validation_percentage=20,
                                 stop_training="STOP_TRAINING")

del DL

# classify raster
raster = "camp-752-8bit.tif"
model = "D:/ml_final_project/data/v3/model/model.emd"
output = "D:/ml_final_project/data/v3/output"

env = arcpy.EnvManager(outputCoordinateSystem="PROJCS['WGS_1984_UTM_Zone_10N',GEOGCS['GCS_WGS_1984',DATUM['D_WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Transverse_Mercator'],PARAMETER['False_Easting',500000.0],PARAMETER['False_Northing',0.0],PARAMETER['Central_Meridian',-123.0],PARAMETER['Scale_Factor',0.9996],PARAMETER['Latitude_Of_Origin',0.0],UNIT['Meter',1.0]]",
                       processorType="GPU")


with env:
    ClassifyPixelsUsingDeepLearning(in_raster=raster, 
                                             out_classified_raster=output, 
                                             in_model_definition=model, 
                                             arguments=[["padding", "56"], 
                                                        ["batch_size", "4"], 
                                                        ["predict_background", "True"]], 
                                             processing_mode="PROCESS_AS_MOSAICKED_IMAGE")

Troubleshooting

A lot of errors in training can be connected back to the training data.

• Arcpy does not tell you this. It also does not fail when creating chips, you need to research your problem and find the right classifier.

Setting up the right conda environment… [setup]

• Follow the online guide.
• Do not upgrade anything! The DL tools only work on very specific versions of each of those packages.

TIFF errors [creating training] [training]

• Check that you are not giving the tools more than 3 bands of imagery. One of the core packages, Pillow (PIL), can only support 3 channels. It was created for plain rgb image classification (ex: cat vs dog).
• Check that your TIFF is an 8bit Unsigned. This usually involves a conversion and can be done when you are exporting the imagery from Arc. Also supports 16bit Unsigned.

Divided by Zero error [training]

• Double check your batch size and learning rate. I need to investigate more, but it seems that too large of a batch size, and/or learning rate, will cause this error.

List index error [training]

• This appears to be an bug (still trying to track down the arc issue).
• When exporting training data, if there are tiles that do not have a classification, no label is created for that tile so the arcpy module fails when trying to find the file.
• Solution: In our fire polygon feature class, we added a polygon that identified the rest of the image as no fire.
• This is an open issue with ESRI

OSERROR(-2) [training]

• I have no idea what that means...
• It appears to be from a libtiff module.
• I think this is ultimately a problem with the training data, i.e. too many channels, not 8bit unsigned, bad labeling system.

Other References

• https://machinelearningmastery.com/autokeras-for-classification-and-regression/
• https://www.kdnuggets.com/2020/11/building-complete-artificial-neural-network.html