Use deep learning for feature extraction and classification

For a human, it's relatively easy to understand what's in an image—it's simple to find an object, such as a car or a face; to classify a structure as damaged or undamaged; or to visually identify different land-cover types. However, it can become tedious to do this at scale. This is where machine learning comes in. With machine learning, you can use and automate this task to solve real-world problems. To accomplish this, ArcGIS implements deep learning technology to extract features in imagery to understand patterns—such as detecting objects, classifying pixels, or detecting change—in different data types and modalities.

Deep learning is a type of machine learning that can be used to detect features in imagery. It uses a neural network—a computer system designed to work like a human brain—with multiple layers. Each layer can extract one or more unique features in an image. Processing is often distributed to perform analysis in a timely manner.

To implement deep learning in your workflow, you can use or fine-tune pretrained deep learning packages or train a deep learning model using various ArcGIS deployments. ArcGIS also integrates with third-party deep learning frameworks to extract features from single images, imagery collections, point clouds, or videos.

Deep learning workflows in ArcGIS follow these steps:

1. Generate training samples of features using editing tools in ArcGIS. These tools use GPU processing to perform analysis promptly.
2. Use those training samples to train a deep learning model using ArcGIS Pro, ArcGIS Image Server for ArcGIS Enterprise, or ArcGIS API for Python.
3. Using the resulting deep learning model, run the inferencing tools in ArcGIS Pro (in a desktop environment), ArcGIS Image Server for ArcGIS Enterprise (to scale the processing), ArcGIS Online (in a SaaS environment), or ArcGIS API for Python (in a developer environment) to extract specific features in the imagery.

Explore the following resources to learn more about feature extraction using deep learning in ArcGIS. (Not sure where to start? Look for the star by Esri's most helpful resources.)

Note:

To perform deep learning using feature extraction, you need the ArcGIS Image Analyst extension for ArcGIS Pro. To use the pretrained deep learning models online, you need ArcGIS Image for ArcGIS Online. To perform deep learning using distributed processing, you need ArcGIS Enterprise with ArcGIS Image Server configured for raster analytics.

Imagery Workflows resources

Review the community-supported tools and best practices for working with and automating imagery and remote sensing workflows:

• Learn about Esri's collection of ready-to-use pretrained deep learning models, then try some of them:
  • Building footprint extraction from high-resolution satellite imagery
  • Tree point classification from point cloud datasets
  • Land cover classification from Landsat 8 imagery
  • Road extraction from satellite imagery
  • High-resolution land cover classification
  • Power line classification
  • Human settlement classification using Sentinel-2 imagery

ArcGIS Help

Review the following links on reference materials for ArcGIS products:

• Learn how to use ArcGIS pretrained deep learning models that range from arctic seal detection to window and door extraction.*
• Learn how to install deep learning frameworks for ArcGIS.
• Read an overview of the Deep Learning toolset in ArcGIS Pro.
• Read how to configure and deploy ArcGIS Enterprise for deep learning.
• Learn about Deep Learning Studio, a web app available in ArcGIS Enterprise.
• Learn about the Object Detection geoprocessing tool in ArcGIS Pro.
• Learn about the Detect Objects Using Deep Learning tool in ArcGIS Online.

ArcGIS blogs, articles, story maps, and technical papers

Review the following supplemental guidance about concepts, software functionality, and workflows:

• Visit the GeoAI Hub for industry- and technology-specific samples of deep learning workflows in ArcGIS, including sample projects using object detection.*
• Review a three-part blog series that reveals deep learning tips: Part 1, Part 2, Part 3.*
• Read about the time-saving benefits of pretrained geospatial deep learning models.
• Learn about pretrained AI models in a story format.
• Learn how to fine-tune a pretrained deep learning model for your training data.
• Read about the different applications of imagery deep learning models in ArcGIS.
• Learn how to quickly label deep learning samples using a configurable app for imagery.
• Read a blog that introduces Deep Learning Studio, a web app that streamlines the deep learning workflow in ArcGIS Image Server.
• Learn when to use ArcGIS Pro and when to use ArcGIS Enterprise to perform object detection.
• Read about deep learning models in the arcgis.learn module of ArcGIS API for Python.
• Review a story from ArcGIS StoryMaps that highlights geospatial deep learning with ArcGIS API for Python.
• Read about the following deep learning applications in ArcGIS:
  • Improving disaster response using automated damage detection
  • Post-fire damage assessment
  • Building footprint detection
  • Flood inundation mapping using radar & AI
  • Site facilities and resource access in a refugee camp
  • Identifying plant species
  • Detecting and monitoring encroaching structures along a pipeline corridor (story)
  • Quantifying parking lot utilization and identifying the different types of cars (on Medium.com)
  • Asset health inspection (on Medium.com)

Videos

Review the following Esri-produced videos that clarify and demonstrate concepts, software functionality, and workflows:

• See how Esri detected and classified pools (6 minutes), and extracted building footprints (5 minutes) and residential parcels (5 minutes) using deep learning in ArcGIS.
• Learn about deep learning with a demo session about the arcgis.learn module, ArcGIS Pro, and more (1 hour).
• Watch how deep learning was used to identify well pads by integrating Jupyter notebooks and ArcGIS Pro, and to detect pavement cracks from video by integrating ArcGIS Pro with third-party frameworks (12 minutes).
• See how you can use the arcgis.learn module to perform deep learning on full-motion video (6 minutes).
• Watch how ArcGIS API for Python and Jupyter Notebook are used to perform deep learning for asset health inspection (3 minutes).
• Review deep learning workflows (4 minutes) and motion imagery enhancements (2 minutes) created in ArcGIS Pro 2.9.
• Watch a recorded webinar that reviews deep learning analysis in higher education (57 minutes).

Training and tutorials

Review the following guided lessons and tutorials based on real-world problems and key ArcGIS skills:

• In this hands-on exercise, use deep learning in ArcGIS to assess palm tree health (1 hour, 40 minutes).*
• Learn how to use deep learning to identify infrastructure at risk of landslides (1 hour).
• Use deep learning to classify power lines using lidar point cloud data (1 hour, 30 minutes), mangroves using Landsat 8 imagery (1 hour, 15 minutes), and damage to buildings after a wildfire using aerial imagery (1 hour, 15 minutes).
• Watch a training seminar discussing the deep learning tools in ArcGIS. (1 hour).*
• Watch a web course discussing how to use pretrained deep learning models with ArcGIS Image for ArcGIS Online (1 hour, 45 minutes).
• Learn about using deep learning workflows in ArcGIS Pro in a web course format (2 hours, 15 minutes).

ArcGIS Solutions

Review the following industry-specific configurations for ArcGIS:

• The 3D Basemaps solution, which streamlines the creation and maintenance of a 3D basemap, now includes a deep learning model for tree point classification from lidar.

Developer resources

Review the following resources and support for automating and customizing workflows:

• Download Deep Learning Libraries installers for ArcGIS Pro and ArcGIS Server.
• Explore an interactive dashboard showing the results of parking lot occupancy detection using deep learning.
• Explore the raster-deep-learning GitHub repository to learn about Python raster functions to get started with deep learning in ArcGIS Pro.
• Explore this GitHub repository containing code for creating a swimming pool detector using deep learning in ArcGIS as presented at the 2018 Esri User Conference plenary.
• Find information about using the REST-based geoprocessing services in ArcGIS Enterprise, which can be used to automate object detection workflows.
• Read help documentation and a blog about the arcgis.learn module in ArcGIS API for Python, which can invoke the deep learning capabilities.
• Review sample notebooks to see how to use ArcGIS API for Python for deep learning:
  • Detect objects such as swimming pools, utility features with encroaching vegetation, palm trees, multiclass land cover changes, lunar craters, traffic lights with non-nadir oriented imagery, shipwrecks, deforestation, wind turbines, road surface deterioration, and brick kilns.
  • Classify land cover with sparse training data, hyperspectral imagery, or satellite imagery.
  • Classify and detect settlements, extract building footprints with drone data, detect mussel farms, and extract glacial calving fronts with HRNet.
  • Extract features such as coastlines (using multispectral imagery), land cover with Panoptic Segmentation, flooding, streams, road networks, land parcels (with the Edge Detection model), informal settlements, and building footprints.
  • Classify objects such as image scenes, coastlines, satellite imagery features (for categorization), wildlife species (with Camera Trap Images), and plant species (using TensorFlow Lite model).

Esri Community

Use the online imagery community to connect, collaborate, and share experiences:

• Visit the Esri Imagery and Remote Sensing community to connect, collaborate, and share experiences regarding deep learning.