Detect Objects From Point Cloud Using Trained Model (3D Analyst)—ArcGIS AllSource

Summary

Detects objects captured in a point cloud using a deep learning model.

Illustration

Usage

This tool requires the installation of Deep Learning Essentials, which provides multiple neural network solutions that include neural architectures for classifying point clouds.
To set up your machine to use deep learning frameworks in ArcGIS AllSource, see Install deep learning frameworks for ArcGIS.
Object detection models do not require the input point cloud to be classified. However, the point cloud that will be evaluated should have similar characteristics to the point cloud data used for training the model. For example, an object detection model trained on a terrestrial point cloud to classify cars would be suitable for another terrestrial point cloud, but it may not work well on a point cloud collected from an aerial lidar survey.
The model identifies a variety of objects, some of which may overlap. To refine these detections, two key thresholds are used: the confidence score threshold and the overlap threshold. The confidence score is a numerical value between 0.0 and 1.0 that indicates how likely an object is correctly identified. A higher score implies greater certainty, and a threshold can be set to filter out detections below a certain level of confidence.
In cases in which multiple similar objects have overlapping bounding boxes, the overlap threshold becomes even more important. This measure uses the Intersection over Union (IoU) ratio, which is calculated by dividing the volume of the intersection by the volume of the union for the overlapping bounding boxes. For the objects whose IoU exceeds the overlap threshold, only the one with the highest confidence score is retained. The Non-Maximum Suppression (NMS) algorithm is used to ensure that only the most relevant detections are part of the final output.
The Reference Surface parameter is required when the input model was trained with relative height attributes. The raster surface is used as a reference height from which relative heights are interpolated for each point. This provides additional information for the model that can be used to more readily differentiate objects. The raster surface provided for this parameter should represent the same type of data as the raster that was used in the training data that created the model. In most cases, this will be a raster created from ground classified points. A raster surface can be generated from the ground classified points in the LAS dataset by applying a ground filter and using the LAS Dataset To Raster tool. A ground surface can also be generated from a point cloud scene layer using the Point Cloud To Raster tool. Raster surfaces that are not sourced from the input point cloud can also be used, but you must ensure that the z-values in the raster correspond appropriately with the z-values in the point cloud.
When the input model was trained with points from certain classes that were excluded from the training data, use the Excluded Class Codes parameter to ensure that those points are omitted from the set of points evaluated by this model. Excluding classes that do not provide a useful context for the objectives of a given model will reduce the number of points that are evaluated, which will improve the speed of training and applying the model. For example, points representing buildings will usually have no relevance for points that represents objects such as traffic lights, power lines, or cars. Building points can also be reliably classified using the Classify LAS Building tool. If points with class 6, which represents buildings, were excluded in the training data that was used to create the model, the input point cloud must also classify building points and have them excluded in this tool.

Parameters

Label	Explanation	Data Type
Input Point Cloud	The point cloud that will be used to detect objects.	LAS Dataset Layer
Input Model Definition	The object detection model that will be used. An Esri model definition file (.emd), deep learning package (.dlpk), or published object detection model from ArcGIS Online or Portal for ArcGIS can be specified.	File; String
Target Objects	The objects that will be identified in the input point cloud, along with the confidence and overlap threshold values that will be used to accept the detected objects. Object Code—The codes that represent the objects the model was trained to identify. Confidence—The confidence threshold for object recognition operates on a scale from 0.0 to 1.0. A higher value means that fewer objects will meet the criteria for being positively identified. Setting the threshold to 1.0 requires 100 percent certainty in the object's detection, effectively making it improbable for any object to be recognized. If you want to exclude a particular object from the output, set its confidence threshold to 1.0. Overlap—The overlap threshold provides a way to select the object that will be retained when multiple overlapping objects have been identified. The overlap threshold describes the ratio between the intersection and the union of overlapping bounding boxes. When multiple objects overlap and exceed the overlap threshold, the object with the higher confidence will be retained.	Value Table
Output Object Features	The output multipatch features that will contain the bounding boxes surrounding the objects detected from the input point cloud.	Feature Class
Batch Size (Optional)	The number of blocks from the input point cloud that will be simultaneously processed. When no value is provided, one block will be processed at a time.	Long
Processing Boundary	A polygon feature that will define the area of interest to be processed.	Feature Layer
Reference Surface	The raster surface that will be used to calculate relative heights for each point. This parameter is required when the input model contains the relative height attribute, which indicates it was trained using a reference height raster surface.	Raster Layer
Excluded Classes	The class codes from the input point cloud that will be excluded from the points that are processed to detect objects. If the model was trained with points from certain class codes being omitted, the points in the input point cloud should have the same class of objects identified and excluded in order to obtain the best results.	Long

arcpy.ddd.DetectObjectsFromPointCloudUsingTrainedModel(in_point_cloud, in_trained_model, target_objects, out_features, {batch_size}, boundary, reference_height, excluded_class_codes)

Name	Explanation	Data Type
in_point_cloud	The point cloud that will be used to detect objects.	LAS Dataset Layer
in_trained_model	The object detection model that will be used. An Esri model definition file (.emd), deep learning package (.dlpk), or published object detection model from ArcGIS Online or Portal for ArcGIS can be specified.	File; String
target_objects [target_objects,...]	The objects that will be identified in the input point cloud, along with the confidence and overlap threshold values that will be used to accept the detected objects. Object Code—The codes that represent the objects the model was trained to identify. Confidence—The confidence threshold for object recognition operates on a scale from 0.0 to 1.0. A higher value means that fewer objects will meet the criteria for being positively identified. Setting the threshold to 1.0 requires 100 percent certainty in the object's detection, effectively making it improbable for any object to be recognized. If you want to exclude a particular object from the output, set its confidence threshold to 1.0. Overlap—The overlap threshold provides a way to select the object that will be retained when multiple overlapping objects have been identified. The overlap threshold describes the ratio between the intersection and the union of overlapping bounding boxes. When multiple objects overlap and exceed the overlap threshold, the object with the higher confidence will be retained.	Value Table
out_features	The output multipatch features that will contain the bounding boxes surrounding the objects detected from the input point cloud.	Feature Class
batch_size (Optional)	The number of blocks from the input point cloud that will be simultaneously processed. When no value is provided, one block will be processed at a time.	Long
boundary	A polygon feature that will define the area of interest to be processed.	Feature Layer
reference_height	The raster surface that will be used to calculate relative heights for each point. This parameter is required when the input model contains the relative height attribute, which indicates it was trained using a reference height raster surface.	Raster Layer
excluded_class_codes [excluded_class_codes,...]	The class codes from the input point cloud that will be excluded from the points that are processed to detect objects. If the model was trained with points from certain class codes being omitted, the points in the input point cloud should have the same class of objects identified and excluded in order to obtain the best results.	Long

Code sample

DetectObjectsFromPointCloudUsingTrainedModel example (Python window)

The following sample demonstrates the use of this tool in the Python window.

import arcpy
arcpy.env.workspace = 'C:/data/detect_cars'
arcpy.ddd.DetectObjectsFromPointCloudUsingTrainedModel('2018_survey.lasd', 'cars.emd',
                                                       [(1, 0.7, 0.4)], 'Cars_in_Point_Cloud.shp',
                                                       10, 'study_area_boundary.shp',
                                                       'dem.tif', [2, 6, 7, 18])

Environments

Extent, Processor Type, GPU ID, Output Coordinate System, Geographic Transformations

Feedback on this topic?