Essential analytic concepts

Real-time analysisBig data analysisAnalytic optionsShare analyticsView analytic details and statusGeofence analysisSample real-time data

Analyze patterns

Calculate DensityCalculate JourneysFind Hot SpotsFind Point ClustersGeneralized Linear RegressionForest-based Classification and Regression

Enrich data

Calculate Motion Statistics

Find locations

Detect IncidentsDetect GapsFind Dwell LocationsFind Similar Locations

Manage data

Calculate FieldControl Event VolumeDissolve BoundariesFilter by ExpressionFilter by Time RangeFilter by GeometryMap FieldsMerge LayersOverlay LayersProjectSelect Fields

Summarize data

Aggregate PointsJoin FeaturesReconstruct TracksSummarize AttributesSummarize Within

Use proximity

Create BuffersCreate Range FansCalculate DistanceSnap to Network

Schedule recurring analysis

Schedule recurring big data analysisNear real-time analysisGenerate up-to-date informational products

Arcade expressions

Use Arcade expressionsUse track functionsBest practices for Arcade expressions

Troubleshoot

Common analysis issues and solutions
Back to Top

Detect Gaps

In this topic

  1. Examples
  2. Usage notes
  3. Parameters
  4. Output layer: real-time analytics
  5. Output layer: big data analytics
  6. Considerations and limitations

Tool icon Available in real-time and big data analytics.

The Detect Gaps tool Detect Gaps tool is available in both real-time and big data analytics, however, the capabilities and specifics of how this tool operates are different between analytic types. In real-time analytics, the Detect Gaps tool monitors every incoming observation and reports whenever any track or tracks have exceeded the specified Gap Duration. Gap detection happens at the rate of the Gap Detection Interval and a gap notification is created as an output event. Alternatively, in big data analytics, the Detect Gaps tool processes the complete set of features provided to the tool and based on Track ID and Start Time key field values, identifies starting, ending, or ongoing gap features according to user-defined parameters for each track.

Examples

  • Real-time analytics
    • A network of smart pressure gauges reports the level of pressure every 10 seconds. A maintenance crew wants to know when any of the pressure gauges have gone offline and are failing to report pressure information. The Detect Gaps tool can be configured with a Gap Duration of 30 seconds and a Gap Detection Interval of 5 seconds. With this configuration, the Detect Gaps tool will check all known gauges every 5 seconds to determine whether any have gone more than 30 seconds without reporting pressure information.
  • Big data analytics
    • A fleet manager is reviewing their automatic vehicle location (AVL) data reporting consistency over a multiweek period to ensure the vehicle sensors and their sensor vendor API are consistently providing features. The Detect Gaps tool can be configured to identify feature tracks where they are missing features greater than the Gap Duration specified such as 15 minutes.

Usage notes

  • Gap Duration should be set to a value equal to or greater than the expected interval between observations. If Gap Duration is set to less than the normal expected interval, it will frequently detect false gaps.
  • This tool only processes events with a unique Track ID. A Track ID key field must be identified on the input dataset to use this tool.
  • Real-time analytic-specific usage notes.
    • The Detect Gaps tool runs at the rate specified by the Gap Detection Interval. This means that if the Gap Detection Interval is set to 20 minutes, the Detect Gaps tool will review all known tracks every 20 minutes to determine whether any have exceeded the Gap Duration.
    • The Detect Gaps tool for real-time analytics provides Continuous or On Change notification modes for when a gap is detected.
      • Continuous—A gap event will be created when a gap is first detected for a given Track ID. An additional gap event will be created at the rate of the Gap Detection Interval for as long as there is an ongoing gap. A final gap event will be issued when the same Track ID has been observed again.
      • On Change—A gap event will be created when a gap is first detected for a given Track ID. A second and final gap event will be issued when the same Track ID has been observed again.
    • A Gap Duration and Gap Detection Interval must be specified to identify gaps. Features missing for equal to or greater than the specified gap duration will be identified as gaps. Gap detection occurs at the rate specified by the Gap Detection Interval.
    • In real-time analytics, the Detect Gaps tool maintains a state store of the current observation for each Track ID. When a new observation is received it is compared to the current observation of the same Track ID, if present, to determine if a gap has occurred. If the new observation has a newer timestamp than the current observation in the state store, the current observation is replaced. If it has an identical timestamp to the current observation for the same Track ID, the tool does not distinguish between the two observations. This means the tool will not replace the current observation in the state store with the new observation, the new observation will not be added to the state store and future observations for this Track ID will be compared to the current observation that remained in the state store until a new observation arrives with a later timestamp. A warning will be logged by the tool when this occurs.
  • Big data analytics-specific usage notes
    • The Detect Gaps tool for big data analytics provides methods of Individual or Cumulative gap detection.
      • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given Track ID exceeds the gap duration.
      • Cumulative—Identifies consecutive differences in time between two consecutive features for a given Track ID exceeding the gap duration as a single gap.
    • A Gap Duration must be specified to identify gaps. Features with time gaps that are equal to or greater than the gap duration will be identified as gaps.
    • Choose whether to output all gap feature types (started, ongoing, and ending features) or only the ending gaps (ending features).
    • Only one or more fields can be specified to identify tracks in addition to the Track ID field. Tracks are represented by the unique combination of one or more track fields.

Parameters

ParameterDescriptionData type

Input Layer

The tracks that will be watched for gaps in a data stream.

Features

Gap Duration

Specifies the time window in which an observation must be received for a track.

String

Notification Mode / Method

Specifies gap notification or gap detection methods. The options available for this parameter differ between real-time and big data analytics:

  • Real-time analytics.
    • Continuous—Issue a gap event at the rate of the Gap Detection Interval until the Track ID has been observed and the gap is no longer ongoing.
    • On Change—Issue one gap event when a gap is first detected and a second final gap event when the Track ID has been observed and the gap is no longer ongoing.
  • Big data analytics.
    • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given Track ID exceeds the gap duration.
    • Cumulative—Identifies consecutive differences in time between two consecutive features for a given Track ID exceeding the gap duration as a single gap.

String

Gap Detection Interval (real-time analytics only)

Specifies how frequently events should be checked for exceeding the Gap Duration. This parameter only applies to real-time analytics.

String

Ended Gaps Only (big data analytics only)

Specifies whether to output all features participating in a gap (started, ongoing, and ending features) or only the ending features for each gap. This parameter only applies to big data analytics.

Boolean

Track ID Fields (big data analytics only)

Choose the field or fields to uniquely identify features to be processed. The parameter will be autopopulated if a TRACK_ID key field is set on the input.

Fields

Output layer: real-time analytics

The output layer from this tool in real-time analytics will have the fields and values of the incoming feature and will also contain three additional fields including lastReceived,IsGap, and GapDuration.

Field nameDescriptionField type

LastReceived

The time stamp of the most recent observation received for the track.

Date

IsGap

The current gap status: true if a gap condition exists and false if not.

Boolean

GapDuration

The total elapsed time of the ongoing gap.

Integer

Output layer: big data analytics

The output layer from this tool in big data analytics will have the fields and values of the incoming feature and will also contain six additional fields including GapFeatureType, GapId, GapStartTime, GapEndTime, GapCurrentDuration, and GapOverallDuration.

Field nameDescriptionField type

GapFeatureType

The part of the identified gap a feature represents. The possible values are Started, OnGoing, and Ended.

String

GapId

A unique ID generated to represent each independent gap.

String

GapStartTime

The datetime when the gap event began.

Date

GapEndTime

The datetime when the gap event ended.

Date

GapCurrentDuration

The current elapsed amount of time ongoing in the current gap for the gap feature. Gap feature types of Started will always have a GapCurrentDuration of 0. Gap feature types of Ended will have the same value for GapCurrentDuration and GapOverallDuration.

Float

GapOverallDuration

The overall elapsed amount of time for the gap to which this feature belongs.

Float

Considerations and limitations

  • The Gap Duration parameter (real-time analytics only) cannot exceed 24 hours. Consider using a big data analytic to monitor features when the gap duration is expected to be greater.
  • The Detect Gaps tool (real-time analytics only) will maintain the state of a gap for a maximum of seven days.

Feedback on this topic?

In this topic
  1. Examples
  2. Usage notes
  3. Parameters
  4. Output layer: real-time analytics
  5. Output layer: big data analytics
  6. Considerations and limitations