Detect Gaps

Detect Gaps icon The Detect Gaps tool is available in both real-time and big data analytics, however the capabilities and specifics of how this tool operates is different between analytic types. In real-time analytics, the Detect Gaps tool monitors every incoming observation and reports whenever any track(s) have exceeded the specified Gap Duration. Gap detection happens at the rate of the Gap Detection Interval and a gap notification is created as an output event. Alternatively, in big data analytics, the Detect Gaps tool processes the complete set of features provided to the tool and based on Track ID and Start Time key field values, identifies starting, ending, or ongoing gap features according to user-defined parameters for each track.

Examples

  • Real-time analytics:
    • A network of smart pressure gauges report the level of pressure every 10 seconds. A maintenance crew wants to know when any of the pressure gauges have gone offline and are failing to report pressure information. The Detect Gaps tool can be configured with a Gap Duration of 30 seconds and a Gap Detection Interval of 5 seconds. With this configuration, the Detect Gaps tool will check all known gauges every 5 seconds see if any have gone more than 30 seconds without reporting pressure information.
  • Big data analytics:
    • A fleet manager is reviewing their automatic vehicle location (AVL) data reporting consistency over a multi-week period to ensure the vehicle sensors and their sensor vendor API are consistently providing features. The Detect Gaps tool can be configured to identify feature tracks where they are missing features greater than the Gap Duration specified such as 15 minutes.

Usage notes

  • Gap Duration should be set to a value equal to or greater than the expected interval between observations. If Gap Duration is set to less than the normal expected interval, it will frequently detect false gaps.
  • This tool only processes events with a unique Track ID. A Track ID must be identified on the input dataset to use this tool.
  • Real-time analytic specific usage notes:
    • The Detect Gaps tool runs at the rate specified by the Gap Detection Interval. This means that if the Gap Detection Interval is set to 20 minutes, the Detect Gaps tool will review all known tracks every 20 minutes to see if any have exceeded the Gap Duration.
    • The Detect Gaps tool for real-time analytics offers Continuous or On Change notification modes for when a gap is detected.
      • Continuous—A gap event will be created when a gap is first detected for a given Track ID. An additional gap event will be created at the rate of the Gap Detection Interval for as long as there is an ongoing gap. A final gap event will be issued when the same Track ID has been observed again.
      • On Change—A gap event will be created when a gap is first detected for a given Track ID. A second and final gap event will be issued when the same Track ID has been observed again.
    • A Gap Duration and Gap Detection Interval must be specified to identify gaps. Features missing for equal to or greater than the specified gap duration will be identified as gaps. Gap detection occurs at the rate specified by the Gap Detection Interval.
  • Big data analytics specific usage notes:
    • The Detect Gaps tool for big data analytics offers methods of Individual or Cumulative gap detection.
      • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given Track ID exceeds the gap duration.
      • Cumulative—Identifies consecutive differences in time between two consecutive features for a given Track ID exceeding the gap duration as a single gap.
    • A Gap Duration must be specified to identify gaps. Features with time gaps that are equal to or greater than the gap duration will be identified as gaps.
    • Choose whether to output all gap feature types (started, ongoing, and ending features) or only the ending gaps (ending features).
    • Only one or more fields can be specified to identify tracks in addition to the Track ID field. Tracks are represented by the unique combination of one or more track fields.

Parameters

ParameterDescriptionData Type

Input Layer

The tracks which will be watched for gaps in a data stream.

Features

Gap Duration

Specifies the time window in which an observation must be received for a track.

String

Notification Mode / Method

Specifies gap notification or gap detection methods. The options available for this parameter differ between real-time and big data analytics:

  • Real-time analytics:
    • Continuous—Issue a gap event at the rate of the Gap Detection Interval until the Track ID has been observed and the gap is no longer ongoing.
    • On Change—Issue one gap event when a gap is first detected and a second final gap event when the Track ID has been observed and the gap is no longer ongoing.
  • Big data analytics:
    • Individual—Identifies a unique gap when the difference in time between two consecutive features for a given Track ID exceeds the gap duration.
    • Cumulative—Identifies consecutive differences in time between two consecutive features for a given Track ID exceeding the gap duration as a single gap.

String

Gap Detection Interval (real-time analytics only)

Specifies how frequently events should be checked for exceeding the Gap Duration. This parameter only applies to real-time analytics.

String

Ended Gaps Only (big data analytics only)

Specifies whether or not to output all features participating in a gap (started, ongoing, and ending features) or only the ending features for each gap. This parameter only applies to big data analytics.

Boolean

Track ID Fields (big data analytics only)

Choose the field(s) to uniquely identify features to be processed. The parameter will be auto-populated if a TRACK_ID key field is set on the input.

Fields

Output layer: real-time analytics

The output layer from this tool in real-time analytics will have the fields and values of the incoming feature and will also contain three additional fields including lastReceived,IsGap, and GapDuration.

Field NameDescriptionField Type

LastReceived

The timestamp of the most recent observation received for the track.

Date

IsGap

The current gap status: true if a gap condition exists and false if not.

Boolean

GapDuration

The total elapsed time of the ongoing gap.

Integer

Output layer: big data analytics

The output layer from this tool in big data analytics will have the fields and values of the incoming feature and will also contain six additional fields including GapFeatureType, GapId, GapStartTime, GapEndTime, GapCurrentDuration, and GapOverallDuration.

Field NameDescriptionField Type

GapFeatureType

The part of the identified gap a feature represents. The possible values are Started, OnGoing, and Ended.

String

GapId

A unique ID generated to represent each independent gap.

String

GapStartTime

The datetime when the gap event began.

Date

GapEndTime

The datetime when the gap event ended.

Date

GapCurrentDuration

The current elapsed amount of time ongoing in the current gap for the gap feature. Gap feature types of Started will always have a GapCurrentDuration of 0. Gap feature types of Ended will have the same value for GapCurrentDuration and GapOverallDuration.

Float

GapOverallDuration

The overall elapsed amount of time for the gap that this feature belongs to.

Float

Considerations and limitations

  • Gap Duration (real-time analytics only) cannot exceed 24 hours. Consider using a big data analytic to monitor features when the gap duration is expected to be greater.
  • The Detect Gaps tool (real-time analytics only) will maintain the state of a gap for a maximum of 7 days.