For a human, it's relatively easy to understand what's in an image—it's simple to find an object, like a car or a face; to classify a structure as damaged or undamaged; or to visually identify different landcover types. For machines, the task is much more difficult. However, it's critical to be able to use and automate machine-based feature extraction to solve real-world problems. To accomplish this, ArcGIS implements deep learning technology to detect and classify objects in imagery.
Deep learning is a type of machine learning that can be used to detect features in imagery. It uses a neural network—a computer system designed to work like a human brain—with multiple layers; each layer can extract one or more unique features in the image. Processing is often distributed to perform analysis in a timely manner.
Deep learning workflows for feature extraction can be performed directly in ArcGIS Pro, or processing can be distributed using ArcGIS Image Server as a part of ArcGIS Enterprise. ArcGIS integrates with third-party deep learning frameworks, including TensorFlow, PyTorch, CNTK, and Keras, to extract features from single images, imagery collections, or video. The arcgis.learn module in the ArcGIS API for Python can also be used to train deep learning models with an intuitive API. It integrates with the ArcGIS platform by consuming the exported training samples directly, and the models that it creates can be used directly for object detection in ArcGIS Pro and ArcGIS Image Server.
Deep learning workflows in ArcGIS follow these steps:
- Generate training samples of features or objects of interest in ArcGIS Pro using the classification and deep learning tools. These tools take advantage of GPU processing to perform analysis in a timely manner.
- Use those training samples to train a deep learning model using a third-party deep learning framework or the arcgis.learn module. Once the model has been trained, the resulting model definition file can be used multiple times as input to the geoprocessing tools to assess multiple images over different locations and time periods.
- Using the resulting deep learning model definition file, run the inference geoprocessing tools in ArcGIS Pro (or distribute processing using ArcGIS Image Server) to extract specific features in your imagery.
Explore the following resources to learn more about object detection using deep learning in ArcGIS. (Not sure where to start? Look for the star by Esri's most helpful resources.)
Note:
To perform object detection, you will need the ArcGIS Image Analyst extension for ArcGIS Pro. To perform object detection using distributed processing, you'll need ArcGIS Enterprise with ArcGIS Image Server.Imagery Workflows resources
Community-supported tools and best practices for working with imagery and automating workflows:
- Explore a collection of deep learning samples (with test images and trained models), including object detection samples using TensorFlow, CNTK, and PyTorch.
ArcGIS Help
Reference material for ArcGIS Pro, ArcGIS Online, and ArcGIS Enterprise:
- Read an overview of the Deep Learning toolset in ArcGIS Pro.
- Read how to configure and deploy ArcGIS Enterprise for deep learning.
- Learn more about the Object Detection geoprocessing tool in ArcGIS Pro.
- Find information on setting up the TensorFlow deep learning framework in ArcGIS.
ArcGIS blogs, articles, story maps, and white papers
Supplemental guidance about concepts, software functionality, and workflows:
- Read about how deep learning in ArcGIS was used for post-fire damage assessment.
- Learn when to use ArcGIS Pro and when to use ArcGIS Enterprise to perform object detection.
- Learn more about building footprint detection using deep learning in ArcGIS.
- Read a story map about how deep learning in ArcGIS can be used to detect and monitor encroaching structures along a pipeline corridor.
- (via Medium.com) Learn more about how deep learning in ArcGIS was used to quantify parking lot utilization and identify the different types of cars.
- (via Medium.com) Learn more about how deep learning in ArcGIS can be used for asset health inspection.
Videos
Esri-produced videos that clarify and demonstrate concepts, software functionality, and workflows:
- See how Esri detected and classified pools using deep learning in ArcGIS (6 mins).
- Watch how deep learning was used to identify well pads by integrating Jupyter notebooks and ArcGIS Pro, and to detect pavement cracks from video by integrating ArcGIS Pro with third-party frameworks (12 mins).
- Watch how the ArcGIS API for Python and Jupyter Notebooks are leveraged to perform deep learning for asset health inspection (3 mins).
Learn Lessons
Guided, hands-on lessons based on real-world problems:
- In this hands-on exercise, try using deep learning in ArcGIS to assess palm tree health (1 hr 40 mins).*
Developer resources
Resources and support for automating and customizing workflows:
- Explore the raster-deep-learning GitHub repo to learn more about Python raster functions to get started with deep learning in ArcGIS Pro.*
- Explore this GitHub repo containing code for creating a swimming pool detector using deep learning in ArcGIS as presented at the Esri UC 2018 Plenary.
- Find information on using the REST-based geoprocessing services in ArcGIS Enterprise, which can be used to automate object detection workflows.
- Read about the arcgis.learn module in the ArcGIS Python API, which can be used to call the Deep Learning tools.
Esri Training
Authoritative learning resources focusing on key ArcGIS skills:
- Watch a live training seminar discussing the deep learning tools in ArcGIS. (1 hr) *
* Esri's top picks
Geonet
Online places for the Esri community to connect, collaborate, and share experiences:
- Ask the imagery community questions about deep learning.
* Esri's top picks