Essential analytic concepts

Real-time analysisBig data analysisAnalytic optionsShare analyticsView analytic details and statusGeofence analysisSample real-time data

Analyze patterns

Calculate DensityCalculate JourneysFind Hot SpotsFind Point ClustersGeneralized Linear RegressionForest-based Classification and Regression

Enrich data

Calculate Motion Statistics

Find locations

Detect IncidentsDetect GapsFind Dwell LocationsFind Similar Locations

Manage data

Calculate FieldControl Event VolumeDissolve BoundariesFilter by ExpressionFilter by Time RangeFilter by GeometryMap FieldsMergeOverlay LayersProjectSelect Fields

Summarize data

Aggregate PointsJoin FeaturesReconstruct TracksSummarize AttributesSummarize Within

Use proximity

Create BuffersCreate Range FansCalculate DistanceSnap to Network

Schedule recurring analysis

Schedule recurring big data analysisNear real-time analysisGenerate up-to-date informational products

Arcade expressions

Use Arcade expressionsUse track functionsBest practices for Arcade expressions

Troubleshoot

Common analysis issues and solutions
Back to Top

Detect Gaps

In this topic

  1. Example
  2. Usage notes
  3. Parameters
  4. Output layer: real-time analytics
  5. Output layer: big data analytics
  6. Considerations and limitations

Tool icon Available in real-time and big data analytics.

The Detect Gaps tool Detect Gaps tool is available in both real-time and big data analytics, however, the capabilities and specifics of how this tool operates are different between analytic types. In real-time analytics, the Detect Gaps tool monitors every incoming observation and reports whenever any track or tracks have exceeded the specified gap duration. Gap detection can be set to a specified interval and a gap notification is created as an output event. Alternatively, in big data analytics, the Detect Gaps tool processes the complete set of features provided to the tool and based on the values in the key fields set for the Track ID and Start Time options, identifies the starting, ending, or ongoing gap features according to the parameters that you define for each track.

Example

The following are example uses of the Detect Gaps tool:

  • In a real-time analytic, a network of smart pressure gauges reports the level of pressure every 10 seconds. A maintenance crew wants to know when any of the pressure gauges are offline and failing to report pressure information. The Detect Gaps tool can be configured with a gap duration of 30 seconds and a gap detection interval of 5 seconds. With this configuration, the Detect Gaps tool checks all known gauges every 5 seconds to determine whether any have gone more than 30 seconds without reporting pressure information.
  • In a big data analytic, a fleet manager is reviewing their automatic vehicle location (AVL) data reporting consistency over a multiweek period to ensure the vehicle sensors and their sensor vendor API are consistently providing features. The Detect Gaps tool can be configured to identify feature tracks where they are missing features greater than the gap duration specified such as 15 minutes.

Usage notes

Keep the following in mind when working with the Detect Gaps tool:

  • A Gap Duration parameter value equal to or greater than the expected interval between observations is recommended. If the gap duration is set to less than the normal expected interval, it frequently detects false gaps.
  • This tool only processes events with a unique track ID. A key field must be specified in the input dataset Track ID option to use this tool.
  • Real-time analytic-specific usage notes:
    • The Detect Gaps tool runs at the rate specified by the Gap Detection Interval parameter. This means that if the gap detection interval is set to 20 minutes, the Detect Gaps tool reviews all known tracks every 20 minutes to determine whether any have exceeded the gap duration.
    • The Detect Gaps tool for real-time analytics provides Continuous or On Change notification modes for when a gap is detected.
      • Continuous—A gap event is created when a gap is first detected for a given track ID. An additional gap event is created at the rate specified in the Gap Detection Interval parameter for as long as there is an ongoing gap. A final gap event is issued when the same track ID has been observed again.
      • On Change—A gap event is created when a gap is first detected for a given track ID. A second and final gap event is issued when the same track ID has been observed again.
    • A Gap Duration and Gap Detection Interval must be specified to identify gaps. Features missing for equal to or greater than the specified gap duration are identified as gaps. Gap detection occurs at the rate specified by the Gap Detection Interval parameter.
    • In real-time analytics, the Detect Gaps tool maintains a state store of the current observation for each track ID. When a new observation is received, it is compared to the current observation of the same track ID, if present, to determine if a gap has occurred. If the new observation has a more recent timestamp than the current observation in the state store, the current observation is replaced. If it has an identical timestamp to the current observation for the same track ID, the tool does not distinguish between the two observations. This means the tool does not replace the current observation in the state store with the new observation, the new observation is not added to the state store and future observations for this track ID are compared to the current observation that remained in the state store until a new observation arrives with a more recent timestamp. A warning is logged by the tool when this occurs.
  • Big data analytics-specific usage notes
    • The Detect Gaps tool for big data analytics provides Individual or Cumulative methods for gap detection.
      • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given track ID exceeds the gap duration.
      • Cumulative—Identifies consecutive differences in time between two consecutive features for a given track ID exceeding the gap duration as a single gap.
    • A value must be specified in the Gap Duration parameter to identify gaps. Features with time gaps that are equal to or greater than the gap duration are identified as gaps.
    • Choose whether to output all gap feature types (started, ongoing, and ending features) or only the ending gaps (ending features).
    • Only one or more fields can be specified to identify tracks in addition to the field specified as the track ID. Tracks are represented by the unique combination of one or more track fields.

Parameters

The following are the parameters for the Detect Gaps tool:

ParameterDescriptionData type

Input Layer

The tracks that are being watched for gaps in a data stream.

Features

Gap Duration

Specifies the time window in which an observation must be received for a track.

String

Notification Mode / Method

Specifies gap notification or gap detection methods. The options available for this parameter differ between real-time and big data analytics.

  • Real-time analytics:
    • Continuous—Issue a gap event at the rate specified in the Gap Detection Interval parameter until the track ID has been observed and the gap is no longer ongoing.
    • On Change—Issue one gap event when a gap is first detected and a second final gap event when the track ID has been observed and the gap is no longer ongoing.
  • Big data analytics:
    • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given track ID exceeds the gap duration.
    • Cumulative—Identifies consecutive differences in time between two consecutive features for a given track ID exceeding the gap duration as a single gap.

String

Gap Detection Interval (real-time analytics only)

Specifies how frequently events should be checked for exceeding the value specified in the Gap Duration parameter. This parameter only applies to real-time analytics.

String

Ended Gaps Only (big data analytics only)

Specifies whether to output all features participating in a gap (started, ongoing, and ending features) or only the ending features for each gap. This parameter only applies to big data analytics.

Boolean

Track ID Fields

Choose the field or fields to uniquely identify features to be processed. The parameter is auto-populated if a key field is set for the Track ID option on the input.

Fields

Output layer: real-time analytics

The output layer from this tool in real-time analytics has the fields and values of the incoming feature and also contains three more fields including lastReceived, IsGap, and GapDuration.

Field nameDescriptionField type

LastReceived

The time stamp of the most recent observation received for the track.

Date

IsGap

The current gap status: true if a gap condition exists and false if not.

Boolean

GapDuration

The total elapsed time of the ongoing gap.

Integer

Output layer: big data analytics

The output layer from this tool in big data analytics has the fields and values of the incoming feature and also contains six more fields including GapFeatureType, GapId, GapStartTime, GapEndTime, GapCurrentDuration, and GapOverallDuration.

Field nameDescriptionField type

GapFeatureType

The part of the identified gap a feature represents. The possible values are Started, OnGoing, and Ended.

String

GapId

A unique ID generated to represent each independent gap.

String

GapStartTime

The datetime when the gap event began.

Date

GapEndTime

The datetime when the gap event ended.

Date

GapCurrentDuration

The current elapsed amount of time ongoing in the current gap for the gap feature. Gap feature types of Started always have a GapCurrentDuration of 0. Gap feature types of Ended have the same value for GapCurrentDuration and GapOverallDuration.

Float

GapOverallDuration

The overall elapsed amount of time for the gap to which this feature belongs.

Float

Considerations and limitations

There are several considerations to keep in mind when using the Detect Gaps tool:

  • The Gap Duration parameter (real-time analytics only) cannot exceed 24 hours. Consider using a big data analytic to monitor features when the gap duration is expected to be greater.
  • The Detect Gaps tool (real-time analytics only) maintains the state of a gap for a maximum of seven days.

Feedback on this topic?

In this topic
  1. Example
  2. Usage notes
  3. Parameters
  4. Output layer: real-time analytics
  5. Output layer: big data analytics
  6. Considerations and limitations