For a human, it's relatively easy to understand what's in an image—it's simple to find an object, like a car or a face; to classify a structure as damaged or undamaged; or to visually identify different land cover types. For machines, the task is much more difficult. However, it's critical to be able to use and automate machine-based feature extraction to solve real-world problems. To accomplish this, ArcGIS implements deep learning technology to detect and classify objects in imagery.
Deep learning is a type of machine learning that can be used to detect features in imagery. It uses a neural network—a computer system designed to work like a human brain—with multiple layers; each layer can extract one or more unique features in the image. Processing is often distributed to perform analysis in a timely manner.
Deep learning workflows for feature extraction can be performed directly in ArcGIS Pro, or processing can be distributed using ArcGIS Image Server as a part of ArcGIS Enterprise. ArcGIS integrates with third-party deep learning frameworks, including TensorFlow, PyTorch, CNTK, and Keras, to extract features from single images, imagery collections, or video. The arcgis.learn module in the ArcGIS API for Python can also be used to train deep learning models with an intuitive API. It integrates with the ArcGIS platform by consuming the exported training samples directly, and the models that it creates can be used directly for object detection in ArcGIS Pro and ArcGIS Image Server.
Deep learning workflows in ArcGIS follow these steps:
- Generate training samples of features or objects of interest in ArcGIS Pro using the classification and deep learning tools. These tools take advantage of GPU processing to perform analysis in a timely manner.
- Use those training samples to train a deep learning model using a third-party deep learning framework or the arcgis.learn module. Once the model has been trained, the resulting model definition file can be used multiple times as input to the geoprocessing tools to assess multiple images over different locations and time periods.
- Using the resulting deep learning model definition file, run the inference geoprocessing tools in ArcGIS Pro (or distribute processing using ArcGIS Image Server) to extract specific features in your imagery.
Explore the following resources to learn more about object detection using deep learning in ArcGIS. (Not sure where to start? Look for the star by Esri's most helpful resources.)
Note:To perform object detection, you will need the ArcGIS Image Analyst extension for ArcGIS Pro. To perform object detection using distributed processing, you'll need ArcGIS Enterprise with ArcGIS Image Server configured for raster analytics.
Imagery Workflows resources
Community-supported tools and best practices for working with imagery and automating workflows:
Reference material for ArcGIS Pro, ArcGIS Online, and ArcGIS Enterprise:
ArcGIS blogs, articles, story maps, and more
Supplemental guidance about concepts, software functionality, and workflows:
- Visit the GeoAI Hub for industry- and technology-specific samples of deep learning workflows in ArcGIS, including sample projects utilizing object detection.*
- Learn how to quickly label deep learning samples using a configurable app for imagery.
- Learn when to use ArcGIS Pro and when to use ArcGIS Enterprise to perform object detection.
- Read about deep learning models in the arcgis.learn module of the ArcGIS API for Python.
- Read about a variety of deep learning applications in ArcGIS:
- Improving disaster response using automated damage detection
- Post-fire damage assessment
- Building footprint detection
- Detecting and monitoring encroaching structures along a pipeline corridor (story map)
- Quantifying parking lot utilization and identifying the different types of cars (via Medium.com)
- Asset health inspection (via Medium.com)
Esri-produced videos that clarify and demonstrate concepts, software functionality, and workflows:
- See how Esri detected and classified pools using deep learning in ArcGIS (6 mins).
- Watch how deep learning was used to identify well pads by integrating Jupyter notebooks and ArcGIS Pro, and to detect pavement cracks from video by integrating ArcGIS Pro with third-party frameworks (12 mins).
- Watch how the ArcGIS API for Python and Jupyter Notebooks are leveraged to perform deep learning for asset health inspection (3 mins).
Guided, hands-on lessons based on real-world problems:
- In this hands-on exercise, try using deep learning in ArcGIS to assess palm tree health (1 hr 40 mins).*
Industry-specific configurations for ArcGIS:
- The 3D Basemaps solution, which streamlines the creation and maintenance of a 3D basemap, now includes a deep learning model for tree point classification from lidar.
Resources and support for automating and customizing workflows:
- Download Deep Learning Libraries installers for ArcGIS Pro and ArcGIS Server.
- Review these sample notebooks to see how to use the ArcGIS API for Python for deep learning:
- Classifying land cover using satellite imagery
- Classifying land cover using sparse training data
- Detecting swimming pools using satellite imagery
- Identifying plant species using a TensorFlow-lite model on a mobile device
- Extracting building footprints from drone data
- Detecting super blooms using satellite imagery
- Categorizing features using satellite imagery
- Automating road surface inspection
- Detecting and categorizing brick kilns
- Detecting slums in satellite imagery
- Reconstructing 3D buildings from aerial lidar
- Detecting settlements using supervised classification and deep learning
- Detecting and tracking vehicles
- Detecting impervious surfaces using multispectral imagery
- Explore an interactive dashboard showing the results of parking lot occupancy detection using deep learning.
- Explore the raster-deep-learning GitHub repo to learn more about Python raster functions to get started with deep learning in ArcGIS Pro.
- Explore this GitHub repo containing code for creating a swimming pool detector using deep learning in ArcGIS as presented at the Esri UC 2018 plenary.
- Find information on using the REST-based geoprocessing services in ArcGIS Enterprise, which can be used to automate object detection workflows.
- Read Help documentation and a blog about the arcgis.learn module in the ArcGIS API for Python, which can be used to call the deep learning tools.
Authoritative learning resources focusing on key ArcGIS skills:
- Watch a live training seminar discussing the deep learning tools in ArcGIS. (1 hr) *
* Esri's top picks
Online places for the Esri community to connect, collaborate, and share experiences:
- See what the imagery community is saying about deep learning.
* Esri's top picks