Available with Image Analyst license.
One capability of motion imagery is tracking specific objects in video data while it plays. These objects can be stationary or moving, may become obscured and re-emerge, or change shape (such as a person entering a vehicle). The object tracking capability in full-motion video (FMV) provides automated and computer-assisted tools to address a variety of situations when tracking objects in video imagery. It relies on deep learning and computer vision technology to assist in object tracking, feature extraction, and matching. You can build a deep learning object tracking model, and use the suite of tools to select and track an object of interest. The centroids corresponding to the object's identification rectangles can be digitized and saved as a point class in the project's geodatabase. The saved points can then be displayed as the archived video plays.
Requirements
Object tracking capability in FMV is available in AllSource with the ArcGIS Image Analyst extension.
Note:
Ensure that your video card drivers are current.
Deep learning model
Tracking objects in a video requires one or more trained deep learning models. The effectiveness of tracking depends on the quality of the deep learning training sample data and how closely your object of interest is associated with the training data. For example, to track a truck moving along a highway, you must have labeled training samples of trucks from many angles. The source of the training samples (annotated images) must be motion imagery. The labeled training samples are used to train the deep learning model to track objects, for instance, trucks in this case. The model will have limited ability to track objects with different appearance, such as cars, but may have success in tracking larger recreational vehicles or buses.
You must install deep learning framework packages to perform deep learning workflows in AllSource. Use a variety of tools to prepare video and still imagery training data, label objects, create deep learning models, inference, and review results. For information about how to install these packages, see Install deep learning frameworks for ArcGIS.
Deep learning is computationally intensive, and a powerful GPU is recommended with CUDA Compute Capability support, version 6.0 or later.
For details about deep learning and deep learning workflows, see Introduction to deep learning and Deep learning in ArcGIS Pro. For more information about the suite of deep learning tools in ArcGIS, see An overview of the Deep Learning toolset.
Tracking tab
The Tracking contextual tab is enabled when you select a video in the Contents pane.
Note:
The Tracking tab is available once the deep learning packages are installed and enabled in AllSource.The tools in the Object Tracking group allow you to identify and manage object tracking in your video data:
- Tracking Pane—Open the Object Tracking Display pane, where you can specify the object tracking deep learning model and configure parameter settings.
- Enable—Activate the interactive object tracking tools.
- Add Object—Add an object to perform object tracking by interactively drawing a rectangle around the object.
- Reposition Object—Click an existing object's tracking rectangle to select it, and redraw the rectangle around the object's updated position.
- Remove Object—Click an existing object's tracking rectangle to remove it from being actively tracked.
- Object(s) to Feature—Save the centroids of the object detection rectangles as a new feature class.
Tracking Pane
Click the Tracking Pane button to open the Object Tracking Display pane. The pane contains the Settings and Tracked Objects tabs.
Settings
On the Settings tab, you can choose the deep learning model and set parameters for tracking objects.
Click the Add object tracking model using local path or URL button 
to open the Add Deep Learning Model From Path dialog box. Specify the path to your deep learning model package file (.dlpk) by providing a URL or by browsing to the file in a local directory. You can assign an alias for the model's name in the Model text box. Click Add to load the model and close the dialog box. The model name appears in, and is selected from, the Model drop-down list.
The Settings tab contains options that help control the object tracking: Detect Track Failure, Recover track, and Auto Detector.
- Detect Track Failure—Specifies whether the object is successfully tracked based on changes in appearance. The default is checked.
- Interval (frames)—The interval, in number of frames, at which the application checks for object appearance changes. The default is 5 frames.
- Minimum Object Size (pixels)—Object tracking will cease if an object is smaller than the specified size. The default is 10 pixels.
- Max Search Interval—When an object is lost, specifies the maximum search interval defined in units of video frames. The default value is 60.
- Status Queue Size—The number of frames for which an object status is maintained when the object is lost before the search interval expires.
- Recover Track—Specifies whether an attempt is made to find an object after the track has been lost. The default is checked.
- Confidence Threshold (0-1)—The minimum ratio between matched source image features and searched object features for successful recovery. The confidence threshold is a number between 0 and 1. The default value is 0.1.
- Minimum Overlap Threshold (0-1)—The minimum overlap ratio between the detected object and searched object for successful recovery. The threshold is a number between 0 and 1. The default value is 0.1.
- Auto Detector—Specifies whether the detection and identification of target objects will be performed automatically using a deep learning-based detector model. The default is unchecked.
Specify the path to your deep learning model file (.emd) by providing a URL, or by browsing to the file in a local directory.
- Auto Detector Frequency—The interval between successive detector runs to reidentify and realign target objects in the scene. Lower values result in greater accuracy with reduced performance. The default value is 1.
- Minimum Confidence Threshold—The minimum confidence value for filtering out detections. The threshold is a number between 0 and 1. The default value is 0.4.
Tracked Objects
The Tracked Objects tab is where you can view and manage tracked objects. The ID, Source, and Status values of every tracked object are listed in the table.
- ID—The unique identifier for each tracked object.
- Source—The source video file in which the object is identified.
- Status—The status of each tracked object: actively tracked, lost, or in a search.
The Tracked Objects tab also contains the following five action buttons for managing tracked objects:
- Add—Add one or more objects to track by drawing a rectangle around each object of interest.
- Reposition—Click an existing object's tracking rectangle to select it, and reposition the rectangle around the object's updated position.
- Remove—Click an existing object's tracking rectangle to remove it from being actively tracked.
- Reassociate—Reassociate the listed object with the Lost status by redrawing the object’s updated position.
- Delete—Delete the selected object from the tracked object list.
Object tracking workflow
A summary of the object tracking workflow is as follows:
- Load a deep learning model and set the tracking parameters on the Settings tab in the Object Tracking Display pane.
- Click Enable to activate the object tracking tools.
- Click Add Object to draw a rectangle around the object you want to track in the video player. The object will be tracked in every video frame.
Note:
This step is not required in Auto Detector mode.
- On the Tracked Objects tab, view the status of the tracked objects.
- If the object becomes obscured and tracking is lost, click Reassociate, and redraw a rectangle around the object's updated position to re-engage tracking.
Note:
This step is not required in Auto Detector mode.
- If a tracking rectangle becomes misaligned with an object being tracked, click Reposition, select its tracking rectangle, and reposition it around the object.
Note:
This step is not required in Auto Detector mode.
- Optionally, click Add to add an object to be tracked.
Note:
This step is not required in Auto Detector mode.
- Optionally, click Remove to remove an object from active tracking.
Note:
This step is not required in Auto Detector mode.
- On the Tracking tab, in the Object Tracking group, click Object(s) to Feature and specify the output location and prefix name to use to store the object centroids to a feature class. Optionally, specify whether the feature class will be added to the map, as well as the frequency interval in seconds by which the centroids will be saved.