Evaluate Point Cloud Classification Model (3D Analyst)

Summary

Evaluates the quality of one or more point cloud classification models using a well-classified point cloud as a baseline for comparing the classification results obtained from each model.

Usage

• To obtain the best evaluation results, the reference point cloud must be well classified for the objects of interest that are being evaluated.

• All input models being evaluated must have the same classification codes. If the reference point cloud does not have class codes whose values or meaning match the classes in the models being evaluated, use the Point Cloud Class Remap parameter to designate the reference point cloud's classification with the codes in the trained models.

• The PointCNN deep learning architecture is used to create the point cloud classification model. This architecture may yield slightly different results when processing the same set of input data. You can specify the same model as an input more than once to evaluate the consistency of the classification result from the point cloud. See PointCNN: Convolution On X-Transformed Points for more information on the potential for variability in the classification results.

• The evaluation process will create several output files in the Target Folder parameter value. The name of each file will begin with the text specified in the Base Name parameter value.

  • <base name>_ModelStatistics.csv—A table listing each model's overall accuracy, precision, recall, and F1 score.
  • <base name>_ClassCodeStatistics.csv—A table summarizing the evaluation results for each class code's accuracy, precision, recall, and F1 score.
  • <base name>_ConfusionMatrices.csv—A table summarizing how each point from each class code in the reference point cloud was classified by the input models. In addition to the number of true positives and false positives, this table also identifies the classes that the false positives were occurring in.
  • <base name>_ConfusionMatrix_<model number>.png—An image showing a chart that provides an easy to read depiction of the confusion matrix for a given model.

• The Reference Surface parameter is required when the input model was trained with relative height attributes. The raster surface is used as a reference height from which relative heights are interpolated for each point. This provides additional information for the model that can be used to more readily differentiate objects. The raster surface provided for this parameter should represent the same type of data as the raster that was used in the training data that created the model. In most cases, this will be a raster created from ground classified points. A raster surface can be generated from the ground classified points in the LAS dataset by applying a ground filter and using the LAS Dataset To Raster tool. A ground surface can also be generated from a point cloud scene layer using the Point Cloud To Raster tool. Raster surfaces that are not sourced from the input point cloud can also be used, but you must ensure that the z-values in the raster correspond appropriately with the z-values in the point cloud.

• When the input model was trained with points from certain classes that were excluded from the training data, use the Excluded Class Codes parameter to ensure that those points are omitted from the set of points evaluated by this model. Excluding classes that do not provide a useful context for the objectives of a given model will reduce the number of points that are evaluated, which will improve the speed of training and applying the model. For example, points representing buildings will usually have no relevance for points that represents objects such as traffic lights, power lines, or cars. Building points can also be reliably classified using the Classify LAS Building tool. If points with class 6, which represents buildings, were excluded in the training data that was used to create the model, the input point cloud must also classify building points and have them excluded in this tool.

Parameters

Dialog

Label	Explanation	Data Type
Input Model Definition	The point cloud classification models and batch sizes that will be used during the evaluation process.	Value Table
Reference Point Cloud	The point cloud that will be used to evaluate the classification models.	LAS Dataset Layer; File
Target Folder	The directory that will store the files which summarize the evaluation results.	Folder
Base Name	The file name prefix that will be used for each of the output files summarizing the evaluation results.	String
Processing Boundary (Optional)	The polygon feature that delineates the portions of the reference point cloud that will be used for evaluating the classification models.	Feature Layer
Point Cloud Class Remapping (Optional)	The class codes from the reference point cloud must match the class codes in the models being evaluated. When the class codes do not match, use this parameter to associate the differing class codes in the point cloud with the classes that are supported in the models being evaluated.	Value Table
Reference Surface (Optional)	The raster surface that will be used to provide relative height values for each point in the point cloud data. Points that do not overlap with the raster will be omitted from the analysis.	Raster Layer
Excluded Class Codes (Optional)	The class codes that will be excluded from processing. Any value in the range of 0 to 255 can be specified.	Long

Derived Output

Label	Explanation	Data Type
Output Confusion Matrices	The CSV format table storing the confusion matrix for each class code in each input model.	Text File
Output Model Statistics	The CSV format table that summarizes the overall statistics of the input models.	Text File
Output Class Code Statistics	The CSV format table that summarizes the statistics for each class code in each input model.	Text File

Python

arcpy.ddd.EvaluatePointCloudClassificationModel(in_trained_model, in_point_cloud, target_folder, base_name, {boundary}, {class_remap}, {reference_height}, {excluded_class_codes})

Name	Explanation	Data Type
in_trained_model [in_trained_model,...]	The point cloud classification models and batch sizes that will be used during the evaluation process.	Value Table
in_point_cloud	The point cloud that will be used to evaluate the classification models.	LAS Dataset Layer; File
target_folder	The directory that will store the files which summarize the evaluation results.	Folder
base_name	The file name prefix that will be used for each of the output files summarizing the evaluation results.	String
boundary (Optional)	The polygon feature that delineates the portions of the reference point cloud that will be used for evaluating the classification models.	Feature Layer
class_remap [class_remap,...] (Optional)	The class codes from the reference point cloud must match the class codes in the models being evaluated. When the class codes do not match, use this parameter to associate the differing class codes in the point cloud with the classes that are supported in the models being evaluated.	Value Table
reference_height (Optional)	The raster surface that will be used to provide relative height values for each point in the point cloud data. Points that do not overlap with the raster will be omitted from the analysis.	Raster Layer
excluded_class_codes [excluded_class_codes,...] (Optional)	The class codes that will be excluded from processing. Any value in the range of 0 to 255 can be specified.	Long

Derived Output

Name	Explanation	Data Type
out_confusion_matrices	The CSV format table storing the confusion matrix for each class code in each input model.	Text File
out_model_statistics	The CSV format table that summarizes the overall statistics of the input models.	Text File
out_class_code_statistics	The CSV format table that summarizes the statistics for each class code in each input model.	Text File

Code sample

EvaluatePointCloudUsingTrainedModel example (Python window)

The following sample demonstrates the use of this tool in the Python window:

import arcpy
arcpy.env.workspace = 'C:/data'
arcpy.ddd.EvaluatePointCloudUsingTrainedModel(
        ['Transmission_Power_Lines.dlpk', 'Distribution_Power_Lines.dlpk'], 
        'Classified_Power_Lines.lasd', 'D:/Evaluate_PointCNN_Models', 
        'Power_Line_Results_', 'test_boundary.shp', [[18, 14], [20, 14]])

Environments

Extent, Processor Type, GPU ID, Output Coordinate System, Geographic Transformations